Intro to Robots

9 Robot Vision

Intro to Robots

Processing Pictures:

• Without the fluke board we can’t process pictures taken by the robot.

• However we can process regular .jpg files.• Caution: Processing can take a long time so keep your

pictures to 600 x 600 pixels. If you are just experimenting try to keep your pictures even smaller – 200x200 pixels.

Intro to Robots

Picture Functions:

from myro import *

picture = makePicture(“somefile.jpg”) # read from a .jpg fileshow(picture)

newPic = pictureCopy(picture)

picWidth = getWidth(picture); picHeight = getHeight(picture)

<list of pixels> = getPixels(picture)<some Pixel> = getPixel(picture,x,y)

setRed(pix, <n>) # 0 <= n <= 255, ‘red’ == 255setGreen(pix, <n>) # 0 <= n <= 255, ‘red’ == 0setBlue(pix, <n>) # 0 <= n <= 255, ‘red’ == 0

redValue = getRed(pix); greenValue = getGreen(pix); blueValue = getBlue(pix)x = getX(pix); y = getY(pix)

savePicture(picture,”anotherfile.jpg”) # save to a .jpg file

Intro to Robots

Robot Vision:

• Without the Fluke board, Scribbler can not take pictures.• However we can explore how we would process

photographs taken by Scribbler and to use results of that analysis.

Intro to Robots

Making your own Picture:

newPic = makePicture(100,100)show(newPic)

onePixel = getPixel(newPic,10,5)print getRed(onePixel) #initial red valuesetRed(onePixel,255)print getRed(onePixel) #new red valueshow(newPic)

.

black by default

Intro to Robots

Making your own Picture:

for yValue in range(0, getHeight(newPic)): aPixel = getPixel(newPic, 10, yValue) setRed(aPixel,255) setGreen(aPixel,0) setBlue(aPixel,0)show(newPic)

# or make it into a function

def drawVerticalRedLine( picture, xPos ): for yPos in range(0, getHeight(picture) ): aPixel = getPixel( picture, xPos, yPos) setRed(aPixel,255) setGreen(aPixel,0) setBlue(aPixel,0)

drawVerticalRedLine(newPic,10)show(newPic)

height of picture

Intro to Robots

Real Photograph

• Photo taken by Scribbler

applePic = takePicture() ORapplePic = loadPicture(‘apple.jpg’)

drawVerticalRedLine(applePic,10)show(applePic)

Intro to Robots

Scan Real Photograph

• If we draw the vertical red line continuously across the photo we will cover up the photo.

• We would rather have the following behaviour.• What we are doing is:

– drawing a red line, – waiting a fraction of a second,– putting back the original pixels and – drawing the line again further on.

run

Intro to Robots

Robot/Program Vision:

• The robot and our program don’t see purple, they each see a combination of red, green and blue.

• Colors with low values of red, green and blue are generally dark and with high values, generally light.

r,g,b = getColors(pixel)

width=height=500myCanvas = GraphWin("Circles", width, height)myCanvas.setBackground("white")for i in range(1,25): c = Circle(Point(randrange(0,width), randrange(0,height)),20) red = 10*i c.setFill(color_rgb(red, red/2, red/4)) c.draw(myCanvas) wait(1) wait(5)myCanvas.close()

run

Intro to Robots

Filtering Photographs:

• Have you ever heard the expression, “The world is not black and white, but various shades of gray”.

• Some times seeing things as black and white helps filter out other irrelevant or confusing messages.

• Go back to our original pictureand ask, “What parts of the photo are brightest?”

Intro to Robots

Brightness Filter:

• We know that higher red, green and blue values mean brighter object.

• Brightness: average color value > 175

myPicture = takePicture()show(myPicture)for pixel in getPixels(myPicture): rValue = getRed(pixel) gValue = getGreen(pixel) bValue = getBlue(pixel) avgValue = ( rValue + gValue + bValue) / 3.0 if avgValue > 175 : #Turn the pixel white setRed(pixel,255);setGreen(pixel,255) setBlue(pixel,255) else: #Otherwise, turn it black. setRed(pixel,0);setGreen(pixel,0) setBlue(pixel,0)show(myPicture)

before

after

Intro to Robots

Redness Filter:

• Suppose you want to find the reddest thing in a photo (and if the photo was taken by the robot, have the robot turn towards that thing).

• In the previous example most of the filtered apple was black, why?

• Redness: – high red value (> 175), – low green and blue values (< 175).

Intro to Robots

Redness Filter:

• Program for highlighting red

myPicture = takePicture()show(myPicture)def filterRed(picture): for pixel in getPixels(picture): rValue = getRed(pixel) if rValue > 175: #Turn the pixel white setRed(pixel,255) setGreen(pixel,255) setBlue(pixel,255) else: #Otherwise, turn it black. setRed(pixel,0) setGreen(pixel,0) setBlue(pixel,0)

filterRed(myPicture)show(myPicture)

an awful lot of red

Intro to Robots

Greenness Filter:

• Same program for green

myPicture = takePicture()show(myPicture)for pixel in getPixels(myPicture): gValue = getGreen(pixel) if gValue > 175: #Turn the pixel white setRed(pixel,255) setGreen(pixel,255) setBlue(pixel,255) else: #Otherwise, turn it black. setRed(pixel,0) setGreen(pixel,0) setBlue(pixel,0)

show(myPicture)

funny how a green filter makes a red apple stand out

Intro to Robots

Primarily Red Filter:

myPicture = takePicture()show(myPicture)Def findRedAreas(picture): for pixel in getPixels(myPicture): rValue = getRed(pixel) gValue = getGreen(pixel) bValue = getBlue(pixel) if rValue > 175 and gValue < 175 and bValue < 175: #Turn the pixel white setRed(pixel,255) setGreen(pixel,255) setBlue(pixel,255) else: #Otherwise, turn it black. setRed(pixel,0) setGreen(pixel,0) setBlue(pixel,0)

show(myPicture)more like it; just two red“hot spots”

Intro to Robots

Find the Center of Redness:

• Simple Technique: Find the average x-coordinate of all white pixels in the filtered picture.

• This will point to the center of redness, which may unfortunately be somewhere in between two major red (now white) areas

average x-coordinate

Intro to Robots

Center of Redness of our Photography

def AverageXofWhitePixels(picture): sumX = 0.0 # We use floating point values. counter = 0.0 # We use floating point values. for xPos in range(0, getWidth(picture) ): for yPos in range(0, getHeight(picture) ): pixel = getPixel(picture,xPos,yPos) value = getGreen(pixel) if value > 0 : # add x-coord of every white pixel sumX = sumX + xPos counter = counter + 1 averageX = sumX / counter return int(averageX) #Return an Integer

myPicture = takePicture()picture = copyPicture(myPicture)picture = findRedAreas(picture)XposAvg = AverageXofWhitePixels(picture)drawVerticalRedLine(myPicture,XposAvg)show(myPicture)

Vertical line to one side becauseredness of the apple shadow ispulling the red center to the right.

Intro to Robots

Processing Red Areas Individually:

• Call each red area “a blob”. • Our problem is:

– find a blob– count its pixels– calculate its x-coordinate center

• Easiest way to traverse the picture is for pix in getPixels(picCopy): . . .

two blobs one blob

Intro to Robots

Red Blobs one-at-a-time:

• Problem: Processing the pixels left-to-right, top-to-bottom we won’t know when we process the line indicated by the red arrow if the blobs are two or one.

• Solution: Keep a list of partially-built blobs and as we find that two blobs are really one, combine them into one.

two blobs one blob

processing a new blobcontinue on same blobno longer processing blobprocessing a new blob;or is it the same blob?

Intro to Robots

BlobList Data Structure:

• A BlobList is a data structure that contains a list of blobs and has many methods for accessing that list.

from blob import *

class BlobList: def __init__(self): self.contents = [ ]

def add(self,b): self.contents.append(b)

def findBlob(self,c): for b in self.contents: if b.inBlob(c): return b return None def getIndex(self,b): for i in range(len(self.contents)): if b == self.contents[i]: return i return -1

def dropBlob(self, b): i = self.getIndex(b) del self.contents[i]

def writeOut(self): for b in self.contents: b.writeOut() print ""

def getBiggestBlob(self): maxNdx = -1 maxSz = 0 for i in range(len(self.contents)): b = self.contents[i] size = b.getSize() if size > maxSz: maxSz = size maxNdx = i return self.contents[maxNdx]

Intro to Robots

Blob Data Structure:

• A Blob is a contiguous red section of a filtered photo. All pixels are connected either vertically or horizontally.

• A blob is built by finding a red point, not already part of another blob and adding it and all the connected red points on the same vertical line to the same blob.

.x

As the point, x, is addedto a new blob, we continueto add all the red pixels on the same vertical line asthe point, x, and connectedto x

Intro to Robots

Complicated Blobs

x . these points are part of theinitial blob containing x

these points are not part of the initial blob containing x because although on the same vertical line, the two lines are not connected by red pixels

once the line labeled A is part of the same blob as x, the extension of the vertical line containing x will also become part of the same blob

A

eventually the entire donut will bea single blob

Intro to Robots

How to Build a Blob:

. . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . .

total blob in picture

pixels addedto programblob

1st

2nd

Which is the last pixel to be added to the program blob?

Intro to Robots

Blob Strategy

• Each time we add a pixel to a blob we give it a color that is not red and not black and not used in any other blob.

select a red pixelcreate a new empty blob with new colorwhile pixel is red: add pixel to current blob (change color) if pixel to the left is already in another blob: combine the two blobs if pixel to the right is already in another blob: combine the two blobs get the pixel immediately above the previous pixelselect the pixel immediately below the original red pixelwhile pixel is red: add pixel to current blob (change color) if pixel to the left is already in another blob: combine the two blobs if pixel to the right is already in another blob: combine the two blobs get the pixel immediately below the previous pixel

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

previous blob new blobcombined blob

Intro to Robots

Blob Code:

from myro import *

class Blob:

def __init__(self, c=None,x=None): self.colors = [ ]; self.blobPixelSum = 0 ### blob consists of a list of colors in use self.verticalCounts = { }; self.midPoint = -1 ### and a dictionary of vertical counts if c != None: ### key = xCoord; value = number yPixels self.colors.append(c) if x != None: self.verticalCounts[x] = 0 def incPixelCount(self,xCoordinate,cnt): self.blobPixelSum = self.blobPixelSum + cnt if self.verticalCounts.has_key(xCoordinate): ### xCoord already belongs to blob self.verticalCounts[xCoordinate] = ### handles the donut case self.verticalCounts[xCoordinate] + cnt else: self.verticalCounts[xCoordinate] = cnt

def inBlob(self,c): return c in self.colors ### tests if a color is already in the blob; if the same color ### is found in two blobs, they are the same blob

. . . . .

. . . . .

. . . . .

. . . . .xCoordinates 12 13 14

vertcialCounts = {12:2, 13:3, 14:1}

Intro to Robots

Blob Code:

def mergeDictionaries(self,b): newD = { } d1 = self.verticalCounts ### {x1:c1, x2:c2, …, xn:cn} d2 = b.verticalCounts for d in d1.keys(): newD[d] = d1[d] if d2.has_key(d): newD[d] = newD[d] + d2[d] ### add vertical counts if xCoord in both blobs del d2[d] for d in d2.keys(): ### get the remaining xCoords not gotten in first loop newD[d] = d2[d] return newD

def combine(self,b): newB = Blob() newB.colors = self.colors + b.colors ### combine disjoint color lists newB.blobPixelSum = self.blobPixelSum + b.blobPixelSum newB.verticalCounts = self.mergeDictionaries(b) ### combine vertical counts return newB

Intro to Robots

Blob Code:

def calculateMidPoint(self): sum = 0 for x in self.verticalCounts.keys(): ### same calculation as the single blob problem sum = sum + x*self.verticalCounts[x] self.midPoint = sum/self.blobPixelSum

def getMidPoint(self): if self.midPoint == -1: self.calculateMidPoint() return self.midPoint

def getSize(self): return self.blobPixelSum

def writeOut(self): print self.colors print "sum of pixels: ", self.blobPixelSum self.calculateMidPoint() print 'midPoint ' , self.midPoint print 'color list', self.colors

Intro to Robots

Generate New Colors:

• Our filtered picture is all black (0,0,0) or red (255,0,0).• We have 224 – 2 other colors to choose from.• More than enough for the pictures we will process.

from myro import *

class AvailableColors: def __init__(self): self.g = 100; self.b = 0

def getNext(self): g = self.g; b = self.b self.b = self.b + 1 if self.b > 255: self.g = self.g+1 self.b = 0 return Color(0,g,b)

generates blue-green colorson the fly in the ranges: 100 <= green <= 255 0 <= blue <= 255approximately 37500 differentcolors

Intro to Robots

Alternate AvailableColors Implementation:

from myro import *

class AlternativeAvailableColors: def __init__(self): self.contents = [ ] self.index = 0 for i in range (100,256): for j in range(0,256): self.contents.append(Color(0,i,j))

def getNext(self): c = self.contents[self.index] index = index + 1 return c

creates a complete lost of colorsup front and then returns these colors one at a time as requested

Intro to Robots

Data/Code Comparisons:

• Both AvailableColors and AlternativeAvailableColors have exactly the same interface:

• But their implementations are very different.• AvailableColors has almost no data (variables g and b)

and a complicated algorithm for getNext().• AlternativeAvailableColors has a complicated data

structure (self.contents, a list of all available colors) but a very simple implementation of getNext().

• Lesson to Learn: There is a trade-off between data structure and algorithm; if one is simple the other is usually complex.

ac.getNext()

Intro to Robots

Main Program:

for each pixel if pixel is red: create a new empty blob and new color while pixel is red: add pixel to current blob (change color) if pixel to the left is already in another blob: combine the two blobs if pixel to the right is already in another blob: combine the two blobs get the pixel immediately above the previous pixel select the pixel immediately below the original red pixel while pixel is red: add pixel to current blob (change color) if pixel to the left is already in another blob: combine the two blobs if pixel to the right is already in another blob: combine the two blobs get the pixel immediately below the previous pixel add blob to blob list

ac= AvailableColors() bl = BlobList()

for pix in getPixels(pictureCopy): redValue = getRed(pix) if redValue > maxRed:

clr = ac.getNext()x = getX(pix)y = getY(pix)b = Blob(clr,x)pixel = pix

while yNdx >= 0 and getRed(pixel) > maxRed: setColor(pixel,clr)

leftPixel = getPixel(pictureCopy,x-1,yNdx)if getRed(leftPixel) == 0 and getGreen(leftPixel) > 0: leftColor = getColor(leftPixel) if not b.inBlob(leftColor): oldB = bl.findBlob(leftColor) b = b.combine(oldB) bl.dropBlob(oldB)

rightPixel = getPixel(pictureCopy,x+1,yNdx)if getRed(rightPixel) == 0 and getGreen(rightPixel) > 0: rightColor = getColor(rightPixel) if not b.inBlob(rightColor): oldB = bl.findBlob(rightColor) b = b.combine(oldB) bl.dropBlob(oldB)

yNdx = yNdx - 1pixel = getPixel(pictureCopy,x,yNdx)

yNdx = y+1picHeight = getHeight(pictureCopy)if yNdx < picHeight:

pixel = getPixel(pictureCopy,x,yNdx)

while yNdx < picHeight and getRed(pixel) > maxRed: setColor(pixel,clr)

leftPixel = getPixel(pictureCopy,x-1,yNdx)if getRed(leftPixel) == 0 and getGreen(leftPixel) > 0: leftColor = getColor(leftPixel) if not b.inBlob(leftColor): oldB = bl.findBlob(leftColor) b = b.combine(oldB) bl.dropBlob(oldB)

rightPixel = getPixel(pictureCopy,x+1,yNdx)if getRed(rightPixel) == 0 and getGreen(rightPixel) > 0: rightColor = getColor(rightPixel) if not b.inBlob(rightColor): oldB = bl.findBlob(rightColor) b = b.combine(oldB) bl.dropBlob(oldB)

yNdx = yNdx + 1pixel = getPixel(pictureCopy,x,yNdx)

yCount = yCount + yNdx - y -1b.incPixelCount(x,yCount)bl.add(b)