Error code: 127
First, we need to grab some pictures from the Fluke's camera. We can rotate the scribbler approximately 360 degrees and grab our set of images:
from myro import * import sys init() manualCamera() # this turns off auto-gain, auto-exposure, auto-* for i in range(60): p = takePicture() name = "%03d.jpg" % i savePicture(p, name) turnLeft(.6, .03)
Here you see the last 4 pictures:
As a Movie
We could, of course, always put those pictures into a list and create an animated gif, like so:
from myro import * movie =  for i in range(60, 0, -1): this = "%.03d.jpg" % i p1= makePicture(this) movie.append(p1) savePicture(movie, 'movie.gif')
But we want to create one big picture - a panorama!
Let's use an off-the-shelf tool to turn these images into a panorama. We'll use Hugin and supporting tools.
I had trouble using the SIFT features to grab "control points" via the GUI - it didn't find enough features in the images to stitch them all together.
Instead, I first used the command line tool align_image_stack that finds plenty of matching features:
./align_image_stack -p out.pto *.jpg
This creates a Hugin project file called out.pto, let's open that in the Hugin application:
Hugin out.pto (on linux) open out.pto (on mac)
- Then click the "2. Align.." images button.
- Then click the "3. Create panorama..." button.
Error code: 127
There are a variety of options in Hugin to play with (e.g. cylindrical, vs. rectilinear projections). You can also manually place or fix image correspondances, etc. As you can see it isn't perfect. There is a giant hole in my chair!
Although we leave it as future work to implement SIFT feature detection and the homography optimization in myro ;-) (some of the code (like autopano-sift) is actually written in C# so maybe in Myro 3.0 ...), we do present here a somewhat simpler way to do a panorama in myro.
The following code assumes that the robot was spinning in place to the left when it was collecting images (using the above #Collecting_Images code, for example).
The myro panorama program looks at images in pairs, and searches for the best alignment in the horizontal direction between the two images. It assumes no movement in the vertical direction, this makes the search easier, but if there are bumps in your carpet could result in weird artifacts. You can think of it as sliding the 2nd image across the 1st, until they "click". We slide, or shift, the second image, by one pixel and then compare each overlapping pixel. We try to find the offset with minimum error. We define the error to be the difference in intensity between aligned pixels. We also normalize the sum of errors by the number of pixels in the intersection. That way bigger overlaps aren't penalized for larger errors - since there are more pixels to contribute to the error. The actual program below will hopefully make this clearer.
The only clever thing done is a little bit of divide and conquer. Instead of exhaustively searching each possible alignment, we do this in a hierarchical fashion. We start at a coarse level, and hone in on more fine grained alignments. This assumes the images are relatively smooth, which isn't always the case, but speeds things up a bit.
Also, there isn't any post-processing, like smoothing, which would make the final product look a lot better.
So here is the code. Of course, the standard:
from myro import *
First, let's write a function that converts each pixel to black and white. We'll be comparing each pixel based on intensity rather than RGB so we need to find the intensity of each pixel.
def getIntensity(pixel): """ Returns the intensity of pixel """ return (getRed(pixel) + getGreen(pixel) + getBlue(pixel)) / 3
Next, we'll write a function computerError() that takes 2 images, and an offset and return the error between the two pictures aligned at the specified offset (relative to p1). We also provide a resolution parameter that tells the function if we should look at each pixel or skip some.
def computeError(p1, p2, hshift, resolution): """ Computes the sum squared error between the differences of the images p1 and a shifted p2. 'hshift' determines the shift in pixels. 'resolution' is the step size for computing the error in both the horizontal and vertical directions. Returns a tuple: the sum squared error, the number of pixels in the intersection, and the total error divided by the number of pixels. """ error = 0.0 pixs = 0 for i in range(0, getWidth(p1) - hshift, resolution): for j in range(0, getHeight(p1), resolution): px1 = getPixel(p1, i+ hshift, j) px2 = getPixel(p2, i, j) error += (getIntensity(px1) - getIntensity(px2))**2 pixs += 1 return error, pixs, error/pixs
findMinError() will, given 2 pictures, try to find the best offset to register the two images. It does it in a hierarchical fashion to speed things up a bit, but might result in sub-optimal alignments.
def findMinError(p1, p2): """ Uses a divide and conquer technique (ala image pyramids) to find the minimum horizontal shift of p2 to line up with p1. Returns a tuple: offset with minimum error, the normalized error """ minx= 0 maxx= 256 for res in [16, 8, 4, 2, 1]: minerr = 1e7 minidx = 0 for i in range(minx, maxx, res): terror, pixs, error = computeError(p1, p2, i, res) if error < minerr: minidx = i minerr = error minx = minidx maxx = minidx + res print res, minidx, minerr #print "new bounds", minx, maxx return minidx, minerr
The patch() function takes 2 pictures and offsets. It combines the pictures based on the offsets and return the merged result. Because the images have a black bar in the left, we have to ignore those bits, that is what the 'fudge' factor is about.
def patch(p1, p2, offset1, offset2, fudge=8): """ Returns a new image p3, composed of p1 and p2 composed such that up to the offset column p3 is p1, and after p2. There is a fudge factor so that the left columns of p2 are ignored since that is a black stripe. """ p3 = makePicture(getWidth(p1) + offset1, getHeight(p1)) if fudge > offset1: fudge = offset1 for i in range(0, getWidth(p3)): for j in range(0, getHeight(p3)): if i < (offset2 + fudge): px = getPixel(p1, i, j) else: px = getPixel(p2, i - offset2, j) px2 = getPixel(p3, i, j) setColor(px2, getColor(px)) return p3
Finally, we are going through each pair of images, find the offset with minimum error and combine them. The tricky bit is we are keeping a new image 'panorama' that needs to be augmented after each iteration. Also, we go through the images in reverse order, since we are patching from left to right. We took the images right to left ...
panorama = None for i in range(60, 1, -1): this = "%.03d.jpg" % i p1= makePicture(this) next = "%.03d.jpg" % (i-1) p2 = makePicture(next) idx, err = findMinError(p1, p2) print "error between (", i, i-1, ") was at pos = ", idx, "with error =", err if (idx >= 255): print "discarding this image" continue if panorama: panorama = patch(panorama, p2, idx, getWidth(panorama) - getWidth(p1) + idx) else: panorama = patch(p1, p2, idx, idx) print "New panorama size = ", getWidth(panorama) savePicture(panorama, 'out.jpg')
The new manualCamera() disables autogain, autoexposure, autowhitebalance. It takes 3 parameters, the gain, brightness, and exposure. They default to reasonable values, but depending on lighting, need to be tweeked. Also added support so if you have the MYROROBOT env. variable set init() uses that robot serial port instead of prompting ... both changes in SVN.