Camera CalibrationCamera Calibrationand Software Image and Software Image

EnhancementEnhancement

Guy Martin BEng MAScMarie Josée Perreault BAASerge Gagné BCom

Digital Imaging and Intelligent SystemsDigital Imaging and Intelligent Systems

Digital imaging is an enabling technology, the key building block for Digital imaging is an enabling technology, the key building block for a broad range of high end, highly automated technologies.a broad range of high end, highly automated technologies.

The use of digital cameras as measuring tools in Intelligent Systems The use of digital cameras as measuring tools in Intelligent Systems (IS) requires the camera to be calibrated.(IS) requires the camera to be calibrated.

We corrected systematic errors found in every camera calibration We corrected systematic errors found in every camera calibration model and technique known.model and technique known.

Using our technique as a software lens correction filter,Using our technique as a software lens correction filter,we demonstrated to National Defence Canadawe demonstrated to National Defence Canada**

8:1 higher measurement accuracy (image or 3D telemetry)8:1 higher measurement accuracy (image or 3D telemetry)4:1 faster computation time4:1 faster computation time30% added lossless video compression30% added lossless video compression… … **(June 2(June 2ndnd 2009) 2009)

Camera CalibrationCamera CalibrationMarket WiseMarket WiseBesides Aerospace and Military applications, the technology can be found applied in a broad range of Intelligent Systems

100%0.14Facial Recognition

20%1.20Biometrics

20%14.00Security

17%40.04Transportation

100%1.0Robot Surgeon

10%95.0Tele-medicine

12.25%13.44Artificial Vision

40-70%0.25Service Robots

7-15%7.37Robotics

Annual Growth

Market billions $US

Major Markets

Digital Imaging is forecasted to become in the present century the next big technological trend setter, as big as the advent of the silicon chip in the 20th century.

Camera CalibrationCamera CalibrationMarket WiseMarket Wise

Our innovation reaches 60% of the optics and photonics market

- 75 billions $ annual sales worldwide

- 2/3 shared between 5,000 US companies

- Satellite imaging: digital compression, infrared fusion, sub pixel image analysis…

- Enhanced night vision: infrared and color image fusion

- Surveillance and object tracking: sub pixel contouring

- Aerial cartography using UAV (Unmanned Airborne Vehicle): 3D from motion and 3D mapping using a camera video feed

- Video feedback and robotics guidance: docking to ISS and Canadarm-Dextre visual feedback, rover exploration…

…

Military and Space Opportunities

Camera CalibrationCamera CalibrationIs knowing how an image projects Is knowing how an image projects through the lens to print on the CCD through the lens to print on the CCD camera surfacecamera surface

We measure the behaviour related toWe measure the behaviour related to

- Camera posture with respect to the sceneCamera posture with respect to the scene- Image deformation introduced by the lensImage deformation introduced by the lens- Image sensor and lens mechanical assembly tolerancesImage sensor and lens mechanical assembly tolerances

Once calibrated, through software we can remove image artefacts: Once calibrated, through software we can remove image artefacts: Errors introduced in the image by the imperfect nature of lenses. Errors introduced in the image by the imperfect nature of lenses. We can then use the corrected image as a feedback sensor for We can then use the corrected image as a feedback sensor for automated tasks in Intelligent Systems.automated tasks in Intelligent Systems.

Camera CalibrationCamera CalibrationWe correct the biggest error found in digital imaging

The distortion introduced by the lens Lens distortion is known to cause

•errors in object measurement and tracking using cameras

•blurring of object edges

•imaging software sensitivity to lighting conditions

•lower digital image and video compression ratios

Using off the shelf cameras and lenses, we demonstrated to National Defense Canada (DND)

•an accuracy increase of 8:1 in 3D measurements using digital images

•a 4:1 speed increase in image geometric distortion correction

•a model extension for using zooming lenses in telemetry, a world first …

We corrected a systematical error found in every camera model and calibration technique published.

We can retrieve the shape of a camera pixel with an accuracy better than 10-10 mm (roughly the size of an hydrogen molecule), removing a bias in the image center as big as 2 pixels.

Every lens parameter is ‘polluted’ by the image center bias.

Our results surpass MIT, NASA, JPL, Stanford, INRIA, INO, Creaform, …

Camera CalibrationCamera Calibration

Geometric distortion curves straight lines and shears the squareness of rectangles with increasing magnitude, going from the image center towards the outside border.Chromatic distortion splits white light as in a prism blurring the rectangle edges in blue on the short side, and in red on the far side with respect to the image center.Chromatic distortion introduces a systematic edge measurement error with varying object color or light spectrum changes.Both distortions are seen by image analysis algorithms as false image point motion while it actually is an imperfection created by the optics.We like to use short focal length lenses to gain image and surface definition, to see more details. Up until our work, we were faced with a trade off between image definition and measurement accuracy because of lens distortion. We are now able to compensate through software.

Lens distortion introduces the biggest Lens distortion introduces the biggest error found in digital imagingerror found in digital imaging

Camera CalibrationCamera CalibrationHow ? By taking the picture of a 3D known How ? By taking the picture of a 3D known structure, and modeling how the image structure, and modeling how the image prints on the camera electronic surface.prints on the camera electronic surface.

Camera CalibrationCamera CalibrationWe corrected everyone’s error in We corrected everyone’s error in

locating the image centerlocating the image centerThe Image center is the center of symmetry needed to model and correct lens distortion, both geometric and chromatic…

Finding the true image center position corrects a systematic bias in all other lens parameters • the lens focal distance• the lens skew compensation• the lens geometric distortion• the lens chromatic distortion• the camera position w.r. to the outside world

and all 3D measurements taken from camera images in telemetry…

Camera CalibrationCamera Calibration3D from a stereo camera pair3D from a stereo camera pair

All advanced imagery techniques require a pinhole ‘errorless’ camera image, where geometric and chromatic distortion are removed, and without bias in the image center. It enables sub pixel edge extraction and increases 3D triangulation accuracy.

Camera CalibrationCamera Calibration3D from a video sequence3D from a video sequence

3D from motion techniques behaves essentially like a camera stereo pair. Two pictures of a still scenery taken from two different vantage points will yield a 3D model. Still, lens geometric and chromatic distortion have to be removed from the images.

This is the technique used by imaging satellites, UAV planes, rovers, panoramic cameras, … to achieve a 3D map of a given area.

In satellite imaging, the earth’s atmosphere behaves like a lens and adds distortion to the image. It can be compensated for only if the camera distortion is accurately known.

Camera CalibrationCamera CalibrationEarth Observation and RovingEarth Observation and RovingA rover relies on its camera to map its surroundings. In aerial photography, we can retrieve edge locations at a fraction of a pixel. In order to make the measurement significant, lens chromatic and geometric distortion have to be removed from the image.

Sub Pixel edge detection is biased by lens distortion.Geometric and chromatic distortion correction require an accurate knowledge of the image center. We removed a 2 pixel image center bias in image calibration using COTS cameras and lenses

Camera CalibrationCamera CalibrationImage fusion and Night VisionImage fusion and Night Vision

Seeing through fog, smoke, snow, camouflage The US Army is spending billions to gain a tactical advantage in night operations. They are seeking the next generation of wide angle imaging to replace the current NVG light amplification devices. Our proposed modification of SWIR (Short Wave Infra Red) cameras to DND Canada provides that kind of performance, and will cut down their supplying cost by 125 to 150$ millions with a smaller/lighter and more accurate night vision camera. Our initial involvement in this venture was to show DND Canada how camera calibration would solve the image fusion problem between SWIR and color images needed for Enhanced Night Vision Goggles (ENVG)We then proceeded to increase the camera sensitivityThe same technology goes for ground or airborne surveillance cameras, soldier helmet, and satellite imaging.

Camera CalibrationCamera CalibrationLanding an AirplaneLanding an Airplane

Civil Aviation produces more CO2 emissions than the entire African continent.At 300,000 $ a piece, with a 2 year payback in fuel economy alone, a cryogenic infrared setup helps to locate the runway in low visibility conditions.The next generation of supersonic airplanes will rely on cameras so no cockpit or hull windows are needed, making the plane lighter. Our proposal to DND is an augmented reality helmet using the human eye as a pointing device to replace the mouse click, a device that doesn’t require cryogenic cooling.

Synthetic vision with Vision Amplification should appear in civilian airline transportation around year 2018

Camera CalibrationCamera CalibrationMedical ImagingMedical Imaging

Historically, imaging technology developed for military uses has seen its way to medical imaging.

The use of robotic surgery shortens the patient’s recovery period in hospital by more than half. Less invasive, it relies on cameras to provide access to the operating area.

The US Army is developing a robot surgeon for NASA’s space program.

Removing lens distortion makes the use of cameras more intuitive and comfortable for the surgeon, increasing his skill.

Used with a robot, lens distortion removal makes measurements 8 times more accurate.

Camera CalibrationCamera CalibrationTelecommunicationsTelecommunications

We demonstrated an added 30% lossless compression over existing codec. Removing lens distortion allowed us to use geometric redundancy to increase the compression ratio of images and video sequences. Video content is therefore faster to transmit. Internet saturates because of the increased use of digital video. The hardware cost for North America is estimated at 55 $billions.In satellite applications, our accuracy gain impacts two ways:

• Smaller image files because of added compression• Less images because larger areas can be scanned

In wireless transmission, added compression means more devices serviced using the same network. Since 75% of all Internet data content is video, a given telco operator could be more profitable than the owner of the network it is roaming on.

Camera CalibrationCamera CalibrationOther market targets include……

•Pharmaceutical packaging•Robotics parcel handling•Highway traffic surveillance•Airport runway traffic management•Border surveillance control…

As part of the Optics and Photonics market segment, computer vision and digital imaging will impact on most of humanity’s economic activities, part of a technological break trough as significant as the advent of the microchip and the electron in the 20th century (according to ISO’s market study).The use of zooming lenses cameras for telemetry was thought to be the only major technological barrier for the explosion of this specific market niche. We showed quite a few more…