Calico Emgu.CV

From IPRE Wiki
Jump to: navigation, search

This page documents the use of the Emgu.CV library for accessing computer vision functions and a general interface to using webcams. It uses OpenCV.

You will need Calico 2.3.1 or better to use these examples.

These instructions are ALPHA, meaning that this is all experimental, and not well tested or debugged.

Overview

For those of you interested in integrating Computer Vision into your courses and research, this documents a bit of the development and plans for this.

First, we plan on having an interface to the Robot Operating System (ROS) in Calico eventually. By doing so, we hope to leverage a large amount of previous and sophisticated work in ROS, with the ease of Calico. However, currently ROS is really only supported on Linux, so this may not be a viable alternative for all users.

In the meantime, we are working towards making the OpenCV library available in Calico, through the Emgu.CV API. This has two levels: a language-neutral level (dlls in Calico/modules/), and then the low-level binaries (dlls in Calico/bin/).

Perhaps someday, we can just make Emgu.CV a standard part of Calico. For the time being, you'll have to download some additional components.

So far, everything we have tested on Windows7 is working ok... not perfectly yet. For example, not all webcams on all platforms have been tested.

What can you do with Emgu.CV? Find faces in your Fluke images, sophisticated object recognition (say, based on shape), find your location, motion detection, feature detection, machine learning, tracking, panoramic creation, and much more. Search the web for "opencv" for more information.

We'll be continuing to work on this over time.

Installation

From Calico

  1. Copy the emgu zip file for your Operating System to your computer from here:
    1. http://myro.roboteducation.org/~dblank/download/addons/calico2/
  2. Unzip the file to a temporary location
  3. Run setup.py from inside Calico
    1. If you want to re-install (after it has been installed before), you can't directly on Windows, because you can't overwrite a file in use. But you can do this:
    2. StartCalico --nomodules --repl --nographics path\to\setup.py

Manual, Non-Calico version

The manual install of factory software will work, to a point. But there are additions that the Calico version (above) contains.

  1. Download Emgu.CV
    1. http://sourceforge.net/projects/emgucv/?source=dlp
  2. Copy the low-level (machine-dependent) opencv_*.dll libraries to Calico's bin folder.
    1. On Windows, use the 32-bit (x86) version even if you are on a 64-bit computer
    2. For example copy "C:\Emgu\emgucv-windows-universal-gpu 2.4.9.1847\bin\x86\*.*" to C:\Calico\bin
  3. Copy the high-level (managed) libraries to Calico's module folder
    1. For example copy "C:\Emgu\emgucv-windows-universal-gpu 2.4.9.1847\bin\*.dll" to C:\Calico\modules

Here is the code added to Image.cs, if you want to add that yourself:

        public class ImageUtils {
		public static Image<Structure.Gray,Byte> MakeImageGray(System.Drawing.Bitmap bitmap) {
			return new Image<Structure.Gray,Byte>(bitmap);
		}
		public static Image<Structure.Rgb,Byte> MakeImageRgb(System.Drawing.Bitmap bitmap) {
			return new Image<Structure.Rgb,Byte>(bitmap);
		}
		public static Image<Structure.Rgba,Byte> MakeImageRgba(System.Drawing.Bitmap bitmap) {
			return new Image<Structure.Rgba,Byte>(bitmap);
		}
		public static Image<Structure.Bgr,Byte> MakeImageBgr(System.Drawing.Bitmap bitmap) {
			return new Image<Structure.Bgr,Byte>(bitmap);
		}
		public static Image<Structure.Bgra,Byte> MakeImageBgra(System.Drawing.Bitmap bitmap) {
			return new Image<Structure.Bgra,Byte>(bitmap);
		}
	}

Testing

In running these interactions, it is probably a good idea to NOT reset the shell when running a new Calico script. In Calico when you have a program script (say, Python) open, go to menu -> Script -> Script Options and UNCHECK "Reset Shell on Run". This will not clear variables, and will allow the webcamera connection to remain continuous through your session.

After installation, you should be able to run the following:

import Emgu
import Myro
import Graphics

capture = Emgu.CV.Capture()
image = capture.QueryFrame() ## draw the image obtained from camera

Myro.show(Graphics.makePicture(image.Bitmap))

If that fails, perhaps you have the wrong architecture (32-bits vs. 64-bits). Although, the 32-bit version should work for all Windows versions.

Windows7: ok, so far

Linux: looks like Emgu.CV needs to be compiled on a 32-bit computer

Mac: not tested

Examples

Images in CV

To do any computer vision (CV) functions, you'll need to get an image into CV's Image format. Normally, one would:

pic = Myro.takePicture()
bitmap = pic.toBitmap()
image = Emgu.CV.Image[Emgu.CV.Structure.Gray,System.Byte](bitmap)

However, something is wrong, and IronPython cannot find the interfaces for Image. So, we have added some static methods to our own version of Emgu.CV that will allow you to make images given a bitmap:

  • Emgu.CV.ImageUtils.MakeImageBgr(bitmap)
  • Emgu.CV.ImageUtils.MakeImageBgra(bitmap)
  • Emgu.CV.ImageUtils.MakeImageGray(bitmap)
  • Emgu.CV.ImageUtils.MakeImageRgb(bitmap)
  • Emgu.CV.ImageUtils.MakeImageRgba(bitmap)

So, you can take a picture, and get a CV image:

pic = Myro.takePicture()
bitmap = pic.toBitmap()
image = Emgu.CV.ImageUtils.MakeImageGray(bitmap)

Filtering

Myro.init("sim")
bitmap = Myro.takePicture().toBitmap()
image = Emgu.CV.ImageUtils.MakeImageGray(bitmap) ## get the image obtained from camera
filtered = image.PyrDown().PyrUp()
pic = Myro.makePicture(filtered.Bitmap) ## and back again, just testing
window = Myro.makeWindow("Emgu.CV", pic.width, pic.height)
pic.draw(window)

Filtered.gif

Face Detection from Fluke/Simulated Fluke

Here is the code:

import Emgu
import Myro
import System
import Graphics

Myro.init("sim") # or real robot, eg "com40"

pic = Myro.takePicture()
bitmap = pic.toBitmap()
image = Emgu.CV.ImageUtils.MakeImageBgr(bitmap)
pic = Myro.makePicture(image.Bitmap) ## display it, as an example of going back to Myro
Myro.show(pic, "Emgu.CV")

grayFrame = image.Convert[Emgu.CV.Structure.Gray,System.Byte]()
filename = calico.relativePath("../modules/haarcascade_frontalface_default.xml")
haarCascade = Emgu.CV.HaarCascade(filename)
detectedFaces = grayFrame.DetectHaarCascade(haarCascade)[0]

for face in detectedFaces:
    print("Found a face at ", face.rect)
    rect = Graphics.Rectangle((face.rect.X,
                               face.rect.Y),
                              (face.rect.X + face.rect.Width,
                               face.rect.Y + face.rect.Height))
    rect.fill = None
    rect.outline = Myro.Color("red")
    rect.border = 4
    rect.draw(Myro.getWindow())
Myro.show(pic, "Emgu.CV, Face Detection")

Face.gif

Face Detection from Webcam

NOTE: the format

image.Convert[Emgu.CV.Structure.Gray,System.Byte]()

is the Python way of converting C# generics:

image.Convert<Emgu.CV.Structure.Gray,System.Byte>()

Here is the code:

import Emgu
import Myro
import System
import Graphics

capture = Emgu.CV.Capture() ## Sometimes, the first image is black
image = capture.QueryFrame() ## get the image obtained from camera

pic = Myro.makePicture(image.Bitmap) ## display it
Myro.show(pic, "Emgu.CV")

grayFrame = image.Convert[Emgu.CV.Structure.Gray,System.Byte]()
filename = "C:/Emgu/emgucv-windows-universal-gpu 2.4.9.1847/bin/haarcascade_frontalface_default.xml"""
haarCascade = Emgu.CV.HaarCascade(filename)
detectedFaces = grayFrame.DetectHaarCascade(haarCascade)[0]

for face in detectedFaces:
    print("Found a face at ", face.rect)
    rect = Graphics.Rectangle((face.rect.X,
                               face.rect.Y),
                              (face.rect.X + face.rect.Width,
                               face.rect.Y + face.rect.Height))
    rect.fill = None
    rect.outline = Myro.Color("red")
    rect.border = 4
    rect.draw(Myro.getWindow())
Myro.show(pic, "Emgu.CV, Face Detection")

Continuously looking for faces:

import Emgu
import Myro
import System
import Graphics

capture = Emgu.CV.Capture()
image = capture.QueryFrame() ## draw the image obtained from camera
pic = Myro.makePicture(image.Bitmap)
window = Myro.makeWindow("Emgu.CV", pic.width, pic.height)

filename = "C:/Emgu/emgucv-windows-universal-gpu 2.4.9.1847/bin/haarcascade_frontalface_default.xml"""
haarCascade = Emgu.CV.HaarCascade(filename)

while True:
    print("Taking a picture...")
    image = capture.QueryFrame() ## draw the image obtained from camera
    if image is None:
        continue
    try:
        pic = Myro.makePicture(image.Bitmap)
    except:
        print("Error!")
        capture = Emgu.CV.Capture()
        continue
    window.clear()
    pic.draw(window)
    grayFrame = image.Convert[Emgu.CV.Structure.Gray,System.Byte]()
    detectedFaces = grayFrame.DetectHaarCascade(haarCascade)[0]

    for face in detectedFaces:
        print("Found a face at ", face.rect)
        rect = Graphics.Rectangle((face.rect.X,
                                   face.rect.Y),
                                  (face.rect.X + face.rect.Width,
                                   face.rect.Y + face.rect.Height))
        rect.fill = None
        rect.outline = Myro.Color("red")
        rect.border = 4
        rect.draw(window)
    Myro.wait(.1)