RobotPanoramas

From IPRE Wiki
Jump to: navigation, search
Error creating thumbnail: /bin/bash: /usr/bin/convert: No such file or directory

Error code: 127

Learn how to use your IPRE robot, myro, and possibly a tool called Hugin to create panoramas.

Collecting Images

First, we need to grab some pictures from the Fluke's camera. We can rotate the scribbler approximately 360 degrees and grab our set of images:

 from myro import *
 import sys
 
 init()
 
 manualCamera() # this turns off auto-gain, auto-exposure, auto-*
 
 for i in range(60):
    p = takePicture()
    name = "%03d.jpg" % i
    savePicture(p, name)
    turnLeft(.6, .03)

Here you see the last 4 pictures:

Pan052.jpg Pan051.jpg Pan050.jpgPan049.jpg

As a Movie

We could, of course, always put those pictures into a list and create an animated gif, like so:

from myro import *

movie = []

for i in range(60, 0, -1):    
   this = "%.03d.jpg" % i
   p1= makePicture(this)
   movie.append(p1)

savePicture(movie, 'movie.gif')


But we want to create one big picture - a panorama!

Hugin Panorama

Let's use an off-the-shelf tool to turn these images into a panorama. We'll use Hugin and supporting tools.

I had trouble using the SIFT features to grab "control points" via the GUI - it didn't find enough features in the images to stitch them all together.

Instead, I first used the command line tool align_image_stack that finds plenty of matching features:

  ./align_image_stack -p out.pto *.jpg

This creates a Hugin project file called out.pto, let's open that in the Hugin application:

  Hugin out.pto (on linux)
  open out.pto (on mac)
  1. Then click the "2. Align.." images button.
  2. Then click the "3. Create panorama..." button.
  3. Viola!
Error creating thumbnail: /bin/bash: /usr/bin/convert: No such file or directory

Error code: 127

There are a variety of options in Hugin to play with (e.g. cylindrical, vs. rectilinear projections). You can also manually place or fix image correspondances, etc. As you can see it isn't perfect. There is a giant hole in my chair!

Myro Panorama

Myropan.jpg

Although we leave it as future work to implement SIFT feature detection and the homography optimization in myro ;-) (some of the code (like autopano-sift) is actually written in C# so maybe in Myro 3.0 ...), we do present here a somewhat simpler way to do a panorama in myro.

The following code assumes that the robot was spinning in place to the left when it was collecting images (using the above #Collecting_Images code, for example).

The myro panorama program looks at images in pairs, and searches for the best alignment in the horizontal direction between the two images. It assumes no movement in the vertical direction, this makes the search easier, but if there are bumps in your carpet could result in weird artifacts. You can think of it as sliding the 2nd image across the 1st, until they "click". We slide, or shift, the second image, by one pixel and then compare each overlapping pixel. We try to find the offset with minimum error. We define the error to be the difference in intensity between aligned pixels. We also normalize the sum of errors by the number of pixels in the intersection. That way bigger overlaps aren't penalized for larger errors - since there are more pixels to contribute to the error. The actual program below will hopefully make this clearer.

The only clever thing done is a little bit of divide and conquer. Instead of exhaustively searching each possible alignment, we do this in a hierarchical fashion. We start at a coarse level, and hone in on more fine grained alignments. This assumes the images are relatively smooth, which isn't always the case, but speeds things up a bit.

Also, there isn't any post-processing, like smoothing, which would make the final product look a lot better.

So here is the code. Of course, the standard:

 from myro import *

First, let's write a function that converts each pixel to black and white. We'll be comparing each pixel based on intensity rather than RGB so we need to find the intensity of each pixel.

 def getIntensity(pixel):
   """
   Returns the intensity of pixel
   """
   return (getRed(pixel) + getGreen(pixel) + getBlue(pixel)) / 3


Next, we'll write a function computerError() that takes 2 images, and an offset and return the error between the two pictures aligned at the specified offset (relative to p1). We also provide a resolution parameter that tells the function if we should look at each pixel or skip some.

 def computeError(p1, p2, hshift, resolution):
   """
   Computes the sum squared error between the differences of the
   images p1 and a shifted p2. 'hshift' determines the shift in
   pixels.  'resolution' is the step size for computing the error in
   both the horizontal and vertical directions.  Returns a tuple: the
   sum squared error, the number of pixels in the
   intersection, and the total error divided by the number of pixels.
   """
   error = 0.0
   pixs = 0
   for i in range(0, getWidth(p1) - hshift, resolution):
       for j in range(0, getHeight(p1), resolution):
           px1 = getPixel(p1, i+ hshift, j)
           px2 = getPixel(p2, i, j)
           error += (getIntensity(px1) - getIntensity(px2))**2
           pixs += 1
 
   return error, pixs, error/pixs

findMinError() will, given 2 pictures, try to find the best offset to register the two images. It does it in a hierarchical fashion to speed things up a bit, but might result in sub-optimal alignments.

 def findMinError(p1, p2):
   """
   Uses a divide and conquer technique (ala image pyramids) to find
   the minimum horizontal shift of p2 to line up with p1. Returns
   a tuple: offset with minimum error, the normalized error
   """
   minx= 0
   maxx= 256
    
   for res in [16, 8, 4, 2, 1]:
       minerr = 1e7
       minidx = 0
       for i in range(minx, maxx, res):        
           terror, pixs, error = computeError(p1, p2, i, res)
           if error < minerr:
               minidx = i
               minerr = error
                
       minx = minidx
       maxx = minidx + res
 
       print res, minidx, minerr
       #print "new bounds", minx, maxx
        
   return minidx, minerr

The patch() function takes 2 pictures and offsets. It combines the pictures based on the offsets and return the merged result. Because the images have a black bar in the left, we have to ignore those bits, that is what the 'fudge' factor is about.

 def patch(p1, p2, offset1, offset2, fudge=8):
   """
   Returns a new image p3, composed of p1 and p2 composed such that
   up to the offset column p3 is p1, and after p2. There is a fudge
   factor so that the left columns of p2 are ignored since that is a
   black stripe.
   """
   p3 = makePicture(getWidth(p1) + offset1, getHeight(p1))

   if fudge > offset1:
       fudge = offset1
       
   for i in range(0, getWidth(p3)):
       for j in range(0, getHeight(p3)):
 
           if i < (offset2 + fudge):
               px = getPixel(p1, i, j)
           else:
               px = getPixel(p2, i - offset2, j)
 
           px2 = getPixel(p3, i, j)
           setColor(px2, getColor(px))
 
   return p3


Finally, we are going through each pair of images, find the offset with minimum error and combine them. The tricky bit is we are keeping a new image 'panorama' that needs to be augmented after each iteration. Also, we go through the images in reverse order, since we are patching from left to right. We took the images right to left ...

 panorama = None 
 
 for i in range(60, 1, -1): 
   this = "%.03d.jpg" % i
   p1= makePicture(this)
 
   next = "%.03d.jpg" % (i-1)
   p2 = makePicture(next)
 
   idx, err = findMinError(p1, p2)
     
   print "error between (", i, i-1, ") was at pos = ", idx, "with error =", err
 
   if (idx >= 255):
       print "discarding this image"
       continue
 
   if panorama:
       panorama = patch(panorama, p2, idx, getWidth(panorama) - getWidth(p1) + idx)
   else:
       panorama = patch(p1, p2, idx, idx)
 
   print "New panorama size = ", getWidth(panorama)
 
   savePicture(panorama, 'out.jpg')


Misc.

The new manualCamera() disables autogain, autoexposure, autowhitebalance. It takes 3 parameters, the gain, brightness, and exposure. They default to reasonable values, but depending on lighting, need to be tweeked. Also added support so if you have the MYROROBOT env. variable set init() uses that robot serial port instead of prompting ... both changes in SVN.

Future Improvements

  1. Do some local adjustments in the tilt direction. Currently we only register in the horizontal, but if the surface is bumpy, you might want to do some small adjustments in the vertical direction.
  2. Use OpenCV python bindings to create a panorama.
  3. Implement SIFT or maybe SURF in myro.