Robotic Vision

Support page to the MOOC Course by Peter Corke

Further reading

Vision in humans and nature

   Stone, J. V. (2012).Vision and Brain: How We Perceive the World. Cambridge, MA: MIT Press.
   Ings, S. (2007). The Eye: A Natural History. London: Bloomsbury.
   Land, M. & Nilsson, D-E. (2002). Animal eyes. New York: Oxford University Press.
   Hughes, H. (1999). Sensory Exotica: A World beyond Human Experience. Cambridge, MA: MIT Press.

Computer and robot vision in general

   Corke, P. (2013). Robotics, Vision and Control. Berlin: Springer.
   Szeliski, R. (2011). Computer vision: Algorithms and applications. New York: Springer.

Multiview/3D vision specifically

   Hartley, R. & Zisserman, A. (2003). Multiple view geometry in computer vision. Cambridge: University Press.
   Ma, Y., Soatto, S., Košecká, J., & Sastry, S. (2006). An invitation to 3-D vision: From images to geometric models. New York: Springer.

Matlab code

 im = iread('/path/to/image.jpg');
 idisp(im); // special display for the course from the vision package

 im = iread('monalisa.png', 'grey', 'double');  // 0.0 - 1.0 instead of 0 - 255
 about(im);

Compare this

a = 100;
b = 200;

a+b
a-b 
a/b

a = uint8(100);
b = uint8(200);

a+b
a-b 
a/b

Getting an image from videocamera. Grab and display a image from the camera.

 cam = VideoCamera(0);
 im = cam.grab();
 clear cam // In order to turn the camera off, use the clear function.

Getting an image from movie.

cam = Movie('traffic_sequence.mpg');
cam.grab(); // next sequence image

Creating an image from code

Create a 200x200 matrix of zeros (representing black) and set the first (top, left) element to 1 (representing white). Display the image.

 im = zeros(200, 200);
 im(1,1) = 1;
 idisp(im)

 im(150:160,60:70) = 0.5;
 idisp(im)

 circle = kcircle(30);
 idisp(circle)

 im = ipaste(im, circle*0.7, [100 30]);
 idisp(im)

 im = iline(im, [30 40], [150 190], 0.9);
 idisp(im)

 im = testpattern('rampx', 200);
 idisp(im)

 im = testpattern('squares', 200, 50, 25);

 im = iread('monalisa.png', 'grey');
 about im
 idisp(im)

 im(194,276)
 ans = 26

 im2 = im(180:210,260:290);
 idisp(im2)

 lin = im(190,260:290);
 plot(lin)

 im3 = im(1:4:end,1:4:end);
 idisp(im3)

 im4 = im(end:-1:1,1:end);
 idisp(im4)

Thresholding

 im = iread('penguins.png', 'grey', 'double');
 idisp(im)
 figure
 ihist(im)
 b = im > 0.45;
 figure
 idisp(b)

 // Use the slider to adjust the threshold setting.
 ithresh(im)

Diadic Operations

 cam = Movie('traffic_sequence.mpg', 'double', 'grey');  // double is important, because uint8 can't do e.g. 50 - 65

 // Extract the first and second frames from the movie. 
 
 im1 = cam.grab();
 im2 = cam.grab();

 // Display the image difference.
 idisp(im1-im2)

 idisp(im1-im2, 'invsigned')  // colors: red / blue for positive / negative values

Spatial operators: Moving average

  % Load the Mona Lisa image.
  im = iread('monalisa.png', 'grey', 'double');

  %  Perform an average over a 7x7 window by 
  %  correlating the image with a 7x7 matrix of 1's using the 
  %  toolbox function iconv(). Display with the original image.

  s7 = iconv(im, ones(7,7));
  idisp(im)
  figure(2)
  idisp(s7)
 
  % again with 21x21 window. 
  s21 = iconv(im, ones(21,21));
  idisp(s21)

Introducing kernels

Create a Gaussian kernal with a standard deviation of 5 and a half radius of 15 pixels using the kgauss() function.

 K = kgauss(5, 15);
 % Display the kernel a greycale image.
 idisp(K)
 figure
 surf(K)

 % Load the Mona Lisa image and correlate this with the Gaussian kernel. Display the image.
 im = iread('monalisa.png', 'grey', 'double');
 s = iconv(K, im);
 idisp(s)

Edges

 // Load the penguins.jpg image and display.
 im = iread('penguins.jpg','grey','double');
 idisp(im)

 ... zrejme Gaussian and Laplacian kernels application

Template matching

Load the wheres_walle.png image as greyscale and of type double. Display the image. Then load a pattern / template.

Using the function isimilarity() from Toolbox we can perform a template matching using ZNCC similarity function. This will take several minutes.

Similarity function return values between 0 and 1, where values approaching 1 is where the template is a good fit.

We can search for the position of best fit by using the toolbox function peak2() and passing in the similarity image, the pixel distance in which to test (i.e. we want to find the pixel which is brightest compared to the pixels within a distance of it) and let's find the top 5 matches.

The first of the output variables contains the similarity value of the top 5 matches and the second variable contains the related positions for these matches. We can locate and label these positions using the plot_circle() and plot_point() functions. For plot_circle() we want to plot circles of radius 30 pixels, of a blue color, an alpha value of 0.3 making them translucent and with no edge color. For plot_point we want to label them in sequence, with a font size of 36 and in the color yellow.

We can extract the region around the point of highest similarity using the toolbox function iroi() - display RegionOfInterest. We want to do this for a region of 50x50 pixels about the first point of p which was obtained earlier.

  crowd = iread('wheres_walle.png','grey', 'double');
  idisp(crowd)

  bender = iread('bender.png', 'double');
  figure(2)
  idisp(bender)

  S = isimilarity(bender, crowd, @zncc);
  idisp(S)

  [mx, p] = peak2(S, 1, 'npeaks', 5)
  % mx =
  %      0.6107    0.5580    0.5277    0.5092    0.4931
  %
  % p =
  %     331        1165        1164         293        1013
  %     364         852         861         619         789
  % 

  idisp(crowd)
  plot_circle(p, 30, 'fillcolor', 'b', 'alpha', 0.3, 'edgecolor', 'none')
  plot_point(p, 'sequence', 'bold', 'textsize', 36, 'textcolor', 'y')

  iroi(crowd, p(:,1), 50)