Lab 05: Introduction to Image Processing and Vision

Objectives

Learn the basics of image processing and robot vision
Manipulate pixels in an image
Make your robot locate objects of different colors in its environment

To Do

Read Chapter 9 of your text on Image Processing and Perception. If you don't have your text, you can find a copy of chapter 9 here: | Chapter 9
Submit the 3 exercises you will do in lab to me at the end of the lab session

Basic Components of an Image

You can think of an image or a picture as a rectangular object which has a specific size determined by its width and its height. Each picture is composed of tiny components called pixels. You can calculate the number of pixels in an image by using the following formula:

number of pixels in an image = width of image * height of image

Each pixel has a red, green and blue component (an RGB value). Each of these colors has a value that ranges from 0 to 255. For instance, a completely red pixel would be represented as (255,0,0), or full red, and zero green and zero blue components. Similarly, a green pixel would be represented as (0,255,0) and a blue pixel would be represented as (0,0,255). Other colors are made by mixing various values of red, green, and blue. For example, white is made by combining all of the colors (255,255,255), and black is made by an absence of any color (0,0,0). If you combine green and blue with no red (0,255,255) you would get a teal color, while a combination of red and green with no blue (255,255,0) would produce yellow. Also, each pixel has a specific (x,y) location in an image.

Taking Pictures

The fluke on your robot has a small digital camera that it can use to take pictures. Once, you've imported the myro library, you can use the takePicture function to make your robot take a picture. You can then assign this picture to a variable, and then use show function to display it.

Try the following:

>>> pic = takePicture()
>>> show(pic)

You can determine the size of the image by using the getWidth() and getHeight() functions.

Try the following:

picWidth = getWidth(pic)
picHeight = getHeight(pic)
print "The Picture is", picWidth , "pixels wide and", picHeight, "pixels high."

Exercise 1

Exercise 1: If your picture is 256 pixels wide, and 192 pixels high, how many total pixels does it have? Write a function to calculate this and to RETURN the computed value. MAKE SURE IT CALCULATES FOR CASES OTHER THAN ONE FIXED VALUE.

Saving and Loading Pictures

You can save the pictures your robot takes by using the savePicture function.

myPicture = takePicture()
savePicture(myPicture,"myPic.jpg")

You can give your picture any name you like. Just make sure it ends with .jpg to tell Myro (and your computer) that it is a JPEG image file. Later, you can load the picture with the makePicture() function:

mySavedPicture = makePicture("myPic.jpg")
show(mySavedPicture)

If you've previously captured an image that is saved on your hard drive, and you want to reload it, you can use the following commands to navigate and then select the image to load:

mySavedPicture = makePicture(pickAFile())
show(mySavedPicture)

Drawing a Line

NOTE: If you would like to use a copy of the apple picture rather than using your own picture in the examples below, replace:

takePicture()

with:

makePicture("http://wiki.roboteducation.org/wiki/images/2/2b/Apple.jpg")

If you want to change a lot of pixels all at once, you can use a loop. For example, the following loop will change all pixels that have an X value of 10 and a Y value anywhere between 0 and 100 (but not including 100) to be red:

for yValue in range(0,100):
  aPixel = getPixel(newPic, 10, yValue)
  setRed(aPixel,255)
  setGreen(aPixel,0)
  setBlue(aPixel,0)

show(newPic)

The result is a vertical red line (at X position 10). Note that this piece of code will only work correctly for pictures that are exactly 100 pixels high (because you loop from zero to 100). But you can generalize this code to work on pictures of any size by replacing the 100 with a function call that tells us the actual height of the picture as follows:

for yValue in range(0, getHeight(newPic) ):
  aPixel = getPixel(newPic, 10, yValue)
  setRed(aPixel,255)
  setGreen(aPixel,0)
  setBlue(aPixel,0)

Drawing a line could be a useful function to use later, so you should prepare to re-use the code above by putting it into a function. But since you don't know the exact X position that the user will want to draw their line, you should make that into a parameter that the user can specify. Also, since you don't know the name of the variable that will hold the picture, that should also be a parameter:

def drawVerticalRedLine( picture, xPos ):
  for yPos in range(0, getHeight(picture) ):
     aPixel = getPixel( picture, xPos, yPos)
     setRed(aPixel,255)
     setGreen(aPixel,0)
     setBlue(aPixel,0)

Now you have a function that will draw a vertical red line on a picture of any height. Note that your function does NOT show the image, so a user would have to call our function, and then call the show() function to display the line on screen:

pic = takePicture()
show(pic)
wait(1)
drawVerticalRedLine(pic, 25)
show(pic)

Exercise 2

Exercise 2: Write a function to draw a horizontal red line. ** Hint: Your function should resemble the drawVerticalRedLine function above.

Manipulating Pixels

Operating on a single pixel in an image

Myro has a set of functions that allow you to get the individual red, green, and blue values from a pixel, as well as set the values to any number you want. But before you get or set the value of a pixel, you need to select a specific pixel to change. To demonstrate this, you will start out by making a blank picture that is 100 pixels high by 100 pixels wide, using the makePicture() function:

picWidth = picHeight = 100
newPic = makePicture(picWidth,picWidth)
show(newPic)

Note that all the pixels in the picture you made are colored pure black. If you want all your pixels to be a different color, you can change the RGB values in the following code:

newPic = makePicture(W, H, makeColor(R,G,B))

You can also use a for loop to do the same thing. For instance, suppose you wanted to change all the pixels in an image to white, you could do the following:

for x in range(picWidth)
   for y in range(picHeight):
       pixel = getPixel(newPic, x, y)
setColor(pixel, white)
repaint(newPic)

In the code above, the getPixel command returns the pixel at the specified x and y locations in the picture. setColor sets the given pixel’s color to any specified color. Above you’re using the predefined color white. You can create a new color by specifying its RGB values in the command:

myRed = makeColor(255, 0, 0)

To visually select a color and its corresponding RGB values, you can use the command:

myColor = pickAColor()

A color palette will be displayed from which you can select a color of your choosing. The palette also shows you the chosen color’s RGB values. After you select a color and press OK, the value of myColor will be the selected color. The repaint command refreshes the displayed image so you can view the changes you made.

Now that you have a desired image, let us try to change the color of a single pixel. You can select a specific pixel (for example, the one at X location 10 and Y location 5) using the getPixel() function. After you select a specific pixel, you can then set its red value to full on (255) with the setRed() function.

onePixel = getPixel(newPic,10,5)
print getRed(onePixel)          #initial red value
setRed(onePixel,255)
print getRed(onePixel)          #new red value
show(newPic)

Note that the red value of the pixel at (10,5) was initially zero, but after you called setRed() the pixel value has changed to 255. Notice also that you had to call show() again before your change was displayed on the screen. If you look at the picture very carefully, you will see that one pixel near the top left corner is now red, instead of black. If you want to make that pixel white, you have to assign 255 to the green and blue components as well:

setGreen(onePixel,255)
setBlue(onePixel,255)
show(newPic)

Now the pixel at location (10,5) is pure white.

Operating on all pixels in an image

The getPixel() function returns a pixel at a specific location in the picture. However, sometimes you want to do something to all pixels in the picture. In such cases you can use the getPixels() method which returns a a list of all the pixels. You can use the getPixels() method in a for loop to perform an operation on all pixels in the image. For example, you can turn the red color of all pixels all the way on with the following code:

myPic = takePicture()
show(myPic)

for eachPixel in getPixels(myPic):
  setRed(eachPixel,255)

show(myPic)

Because the above code does not change the green or blue values of the pixels, the picture is still recognizable, but all pixels have a reddish tint.

Robot Vision

Locate Bright Areas in an image

By making a decision about each pixel in an image, you can locate specific areas in an image. For example, by looking for pixels that have a large value, you can locate bright areas in the image. You can even modify the image to outline bright areas. For example:

myPicture = takePicture()
show(myPicture)

for pixel in getPixels(myPicture):
  redValue = getRed(pixel)
  greenValue = getGreen(pixel)
  blueValue = getBlue(pixel)
  averageValue = ( redValue + greenValue + blueValue) / 3.0

  if averageValue > 175 :
    #Turn the pixel white
    setRed(pixel,255)
    setGreen(pixel,255)
    setBlue(pixel,255)
  else:
    #Otherwise, turn it black.
    setRed(pixel,0)
    setGreen(pixel,0)
    setBlue(pixel,0)

show(myPicture)

Locate Red Areas in an image

By changing the conditional test, you can instead look for areas that have a large amount of the color red. Red areas are characterized by having large red values, but smaller blue and green values.

myPicture = takePicture()
show(myPicture)

for pixel in getPixels(myPicture):
  redValue = getRed(pixel)
  greenValue = getGreen(pixel)
  blueValue = getBlue(pixel)

  if redValue > 175 and greenValue < 175 and blueValue < 175 :
    #Turn the pixel white
    setRed(pixel,255)
    setGreen(pixel,255)
    setBlue(pixel,255)
  else:
    #Otherwise, turn it black.
    setRed(pixel,0)
    setGreen(pixel,0)
    setBlue(pixel,0)

show(myPicture)

Original Picture.

Pixels with red values larger than 175 highlighted in white.

Pixels with green values lower than 175 highlighted in white.

Pixels that have both red values larger than 175 and green and blue values lower than 175 highlighted in white.

Represent Red Areas in an image as White Areas

You can then define a function to find red pixels and return a black and white picture with white pixels representing "red" areas.

def findRedAreas(picture):

  for pixel in getPixels(picture):
    redValue = getRed(pixel)
    greenValue = getGreen(pixel)
    blueValue = getBlue(pixel)

    if redValue > 175 and greenValue < 175 and blueValue < 175 :
      #Turn the pixel white
      setRed(pixel,255)
      setGreen(pixel,255)
      setBlue(pixel,255)
    else:
      #Otherwise, turn it black.
      setRed(pixel,0)
      setGreen(pixel,0)
      setBlue(pixel,0)

  return picture

The result of testing for "red areas" is an image where most of the apple has been detected, but a lot of other pixels scattered around the image are also somewhat "reddish". If you want the robot to turn towards the direction with the most red pixels, you need to calculate the average X (horizontal) location of the pixels that have been marked (by turning them white).

Drawing a Line in the Red Area

If you examine every pixel in the image, and average the X coordinates of all the "detected" (or white) pixels, you can determine the middle of the red areas. To do this, you use two loops (one for the Y positions, and one for the X positions) to look at every pixel, and add up the X positions of all pixels that are turned "on" (or white). Because a white pixel has values of (255,255,255) and an "off" or black pixel has values of (0,0,0) you can take a shortcut and only test the value of one of the three colors.

def AverageXofWhitePixels(picture):
   sumX = 0.0           # Use floating point values
   counter = 0.0        # Use floating point values
   for xPos in range(0, getWidth(picture) ):
      for yPos in range(0, getHeight(picture) ):
          pixel = getPixel(picture,xPos,yPos)
          value = getGreen(pixel)
          if value > 0 :
             sumX = sumX + xPos
             counter = counter + 1

   averageX = sumX / counter
   return int(averageX)         #Return an Integer

Now, you can use the functions you have defined so far to locate the average X location of red pixels, and draw a line at that position:

myPicture = takePicture()
picture = copyPicture(myPicture)
picture = findRedAreas(picture)
XposAvg = AverageXofWhitePixels(picture)
drawVerticalRedLine(myPicture,XposAvg)
show(myPicture)

Because of all the other "red" pixels detected that were not on the apple, the red line is not exactly centered on the apple, but it is close enough that you could use the value in the XposAvg variable to figure out which way to turn the robot to face the apple.

Exercise 3

Exercise 3: Write a program that will turn your robot towards red objects. **Hint: Read the Image Understanding and Robot Vision section of Chapter 9 in your text to get a better understanding of how to do this.

Assignment 05

Part I: Lab Exercises - Make sure you have submitted exercises 1, 2 and 3 that you did in the lab session

Links

Back to Lab Home Page
Back to Course Home Page

CS110:Lab05

Contents