Program Your Computer to See
Pages: 1, 2
Almost all cameras suffer from barrel distortion, with wide angle or "fish eye" lenses being the worst. Further, every camera and lens will have slightly different characteristics. The result is that straight lines in the target scene become curves in the captured image, particularly at the edges. The upper-left portion of the image below, captured with an inexpensive B&W security camera, demonstrates this well.
Corrected vs. uncorrected image from small security camera.
In Computer Vision applications, camera calibration is often a critical first step in extracting meaningful data. After all, how is a computer algorithm suppose to understand what it is looking at when straight lines are curves? Calibration, or extracting the camera's "intrinsics," registers the 2D data the camera provides to the 3D world it is viewing and allows the distortion to be removed from the video stream. The OSCVL has been used to undistort the lower-right hand portion of the image.
Once the intrinsics are known, it is also possible to calculate the true (X,Y,Z) coordinates of targets of known characteristics. With two or more calibrated cameras viewing a scene, arbitrary 3D extraction and background elimination can be performed. This leads to applications like inexpensive but complex gesturing functions (e.g., the computer knows where on the screen or your desk you're looking or pointing) and automatic 3D feature extraction of objects in a scene (how big is that vehicle, and how fast is it going).
CamCal application for Linux
The OSCVL package for Windows comes with several sample applications, but they all currently rely on Video for Windows to run. Because of its importance for most further CV work, the sample camera calibration application was the first to be ported to run under Linux, using Video4Linux devices. It is currently distributed separately, as it was not ported by Intel.
To build the package, uncompress it as you did above for the OSCVL, and
cd into the created directory. Execute the
./autogen.sh command, and if there are no warnings about missing components, run
make. Once it's finished building, running
make install (as root) will place the
calib application in
/usr/local/bin/. If you don't want to install right away, you can run the application from the
How the CamCal Program Works - Chris Halsall walks you through the CamCal set-up and calibration in this companion article to "Program Your Computer to See," which is an introduction to the science of Computer Vision.
When the application launches, it tries to open the
/dev/video device and, if successful, starts capturing two frames per second from this source. If unsuccessful, it will complain and wait for you to set the correct device from the Preferences panel, available from the menu. If you continue to have difficulty opening the device, you
might need to set the device's ownership, group, or file-mode permissions to allow access.
It's possible to have multiple copies running at the same time, each
connecting to a different Video4Linux device (
/dev/video1, etc.). If you have more than one camera,
be sure to calibrate each separately.
The calibration process is quite simple. It involves the use of a
chessboard pattern placed on a planar (flat) surface, as seen in the
image below. Several different patterns are included with the package
targets/ directory, appropriate for 8.5 by 11 inch
paper. One of these should be printed out with a high quality laser
printer and mounted on a rigid, flat surface.
Calibration software running under Linux; Tux is helping hold target.
Next, you need to set the "etalon" values for the target you've decided to use. The target that Tux is holding in the image is an 11 by 8 target with squares of 2cm each. The linear size value isn't actually needed for undistortion calculations, as it's only a scaling factor for 3D calculations, but it's a good idea to give the correct values even if you don't plan to use 3D extraction right away.
Once the sizes are known, the program will look for the chessboard pattern in the scene and draw red Xs at all found intersections. If the number of intersections found matches what is expected, and the points appear to be of high quality, the red Xs turn green, and, if appropriate, the found points are stored for processing. It is interesting to note that the library is able to determine the intersection points with an accuracy of 0.1 pixel along each axis.
Several screen shots worth of samples must be acquired, with the target moving around the field of view of the camera and with the plane of the target varying as well; that is, the six degrees of freedom must be exercised. By default, 20 such images are collected, and this can be done sequentially by pressing the "Start" button and moving the target around, or by pressing the "Single" button when the target has been moved to the next location in space.
Once the full set of images have been acquired, a call to the OSCVL is made, which processes the found intersection points and tries to calculate the intrinsics. The processing can take from a few seconds to a minute or so, depending on the speed of the processor. If successful, the calculated intrinsics are displayed in the lower-right hand corner of the window, and the captured video frames are subsequently undistorted to visually show what the effects are.
If the calculations fail (because of interlace shutter noise or the target placements being too coplanar), the process is reset, and another set of images must be captured. You can also restart the calibration by pressing the "Start" button again, to train another camera, or if you're not satisfied with the current results.
Once the intrinsics have been extracted for a camera, they can be written out using the "File->Save As" menu. These files are plain text and can be re-read into the calibration program using the "File->Open" menu, or into other applications which will be working with data from the same camera. Being plain text files, they can be edited by hand. Try adjusting the various values to see what each does.
It is also possible to turn on and off the undistortion by toggling the check box. Interpolation filtering can also be toggled in this way, which results in faster processing but with the risk of "jaggies."
The critical camera calibration functions of the OSCVL, briefly outlined above, are only a small example of the features and abilities delivered. Leveraging on this library, and with only a small amount of research, just about any developer should be able to add sophisticated CV functions to their applications.
Intel plans to continue to add additional functions and abilities to the library over time, with several new features currently being tested internally. Contributions are encouraged, so long as they're free from anything that would prevent them from being placed under the OSCVL license. That means you won't find patented algorithms in the library unless they're released appropriately, or they expire.
Personally, I think this is yet another example of Intel "getting it." While there's no question that applications built with the library will encourage the purchase of faster processors, because of the BSD-like license, there is no lock-in to Intel processors specifically.
For developers and researchers alike, the benefits are huge. Previously, duplication of effort, or lots of hunting around for code snippets, was required to get the basics in place before any real new work could begin. With processors as fast as they are now, and only getting faster by the quarter, ever more advanced real world applications are becoming possible. Cameras also continue to become more sensitive, smaller, and less expensive.
The OSCVL is an excellent example of what makes open source work: You make the substrate, or infrastructure, free and available for everyone to work on, and then just watch as people do amazing things. The time is ripe for CV applications to become common.
There is no doubt: we live in very interesting times.
Discuss this article in the O'Reilly Network Linux Forum.
Return to the Linux DevCenter.