LEDGER PRISM PRISM Project List - University of Pretoria · 2015. 6. 26. · 5860-PRISM-800700-01 RPT Rev 1 Order KT521899,Item 7 Reference P13E LEDGER PRISM PRISM Project List by

RESTRICTED

5860-PRISM-800700-01 RPT Rev 1Order KT521899, Item 7, Reference P13E

LEDGER PRISMPRISM Project List

by A Bachoo, B Duvenhage, Jason de Villiers

23 January 2015

c© CSIR 2015. All rights to the intellectual property and/or contents of this document remain vested in theCSIR. This document is issued for the sole purpose for which it is supplied. No part of this publication maybe reproduced, stored in a retrieval system or transmitted, in any form or by means electronic, mechanical,photocopying, recording or otherwise without the express written permission of the CSIR. It may also not be lent,resold, hired out or otherwise disposed of by way of trade in any form of binding or cover other than that in whichit is published.

RESTRICTED

RESTRICTED

APPROVAL PAGE

Project Name LEDGER PRISM

Title PRISM Project List

Prepared for Armscor

By A Bachoo Signature Date

B Duvenhage Signature Date

Jason de Villiers Signature Date

Moderator Z Hattingh Signature Date

Project No GEOPRIS

Approved for CSIR

By Simphiwe Mkwelo Signature Date

Designation Competency Area Manager: Optronic Sensor Systems

Approved for Armscor

By R Calitz Signature Date

Designation Electronics Technology Manager

Distribution Listeb (Electronic Copy)

PDF5860-PRISM-800700-01 RPT Rev 1 Page 1 of 25

RESTRICTED

RESTRICTED

DOCUMENT CHANGE HISTORY

ChangeControl No Date Name Description

This page will only be updated starting from: Rev 2

Filename 5860-PRISM-800700-01RPTRev1.pdf Software DPSS LATEX Style Version 2011-02-03


RESTRICTED

RESTRICTED

CONTENTS

Executive Summary 5

1 Introduction and Background Information 61.1 Potential Problems in the Defence Domain 61.2 Sensors 71.3 Project Guidelines 8

2 Research Topics in Image Processing Software and Hardware 92.1 Image Processing Parallelization 92.2 Image Processing on a Smart Phone 92.3 Image Processing on an Intel Edison System on Chip 9

3 Research Topics in Image Enhancement 103.1 Heat Shimmer (Scintillation) Mitigation 103.2 Night Vision Noise Reduction 103.3 Real Time Colour Balancing 103.4 Optical Fourier processing 11

4 Research Topics in Scene Modeling, Feature Extraction and Segmentation 124.1 Scene Modeling and Segmentation in a Terrestrial Environment 124.2 Maritime Scene Modeling and Segmentation 124.3 Graph Cuts for Image Segmentation 124.4 Generating and Processing Super-Pixel Images 134.5 Motion Detection from a Moving Camera 134.6 An Efficient Implementation of the DPT Transform for Fast Image Processing 134.7 Detection and Segmentation of Salient Objects 13

5 Research Topics in Target Detection and Tracking 14

6 Research Topics in Classification and Identification 15

7 Research Topics in 3D Computer Vision 167.1 Computational Photography Concepts 167.2 Photogrammetric Stereo using a Plane Sweep 167.3 Plenoptic Imaging 16

8 Research Topics in Sensors, Components and Systems 188.1 Integrated Optical Circuits (IOCs), Micro Optic and Electromechanical Systems (MOEMS)

and Photonic Integrated Sensors 188.2 Gated Cameras 198.3 Lasers 208.4 Navigation Sensors and Algorithms 228.5 Optomechatronics Technologies 22


RESTRICTED

RESTRICTED

9 Research Topics in Measurements and Modeling 24


RESTRICTED

RESTRICTED

EXECUTIVE SUMMARY

Title: PRISM Project List

Abstract: This document contains the PRISM project list for students and researchers. Thebasic project guidelines are provided together with other background information.Several research topics are presented in the different chapters and provide insightinto the various image processing, computer vision and photonics problems that areof interest to the South African National Defence Force (SANDF). The proposedtopics are of varying degrees of complexity; where possible, the level of study forstudents (such as MSc or PhD) is indicated.


RESTRICTED

RESTRICTED

1 Introduction and Background Information

The projects under the PRISM program are all intended to increase South Africa’s surveillance capabil-ity in the field one day. A final system, built upon knowledge gained from these projects, will be exposedto harsh operational environments. Some of the aspects pertinent to image processing and relatedfields are described below. Project PRISM currently addresses the following areas of interest, but is notnecessarily restricted to these, as new research tasks can be defined which are of value to the SouthAfrican National Defence Force (SANDF):

• Image processing - image enhancement, target detection, target tracking and classification.

• Solid state laser sources.

• Optronics based navigation sensors and algorithms.

• Surveillance sensors.

• IR detector materials.

• Protection against laser dazzlers and countermeasures.

• Measurements and modeling.

• Optomechatronic technologies.

• 3D vision.

• Micro-Opto-Electro-Mechanical Systems(MOEMS) devices.

The PRISM projects are typically meant to provide a strong research base for the Department of De-fence. The research areas are concerned with defence and safe keeping rather than being on theoffensive.

1.1 Potential Problems in the Defence DomainA majority of the surveillance work is concerned with the maritime environment. For the SAN, there areseveral potential threats that require counteraction:

• Boats and Ships: Typically slow moving watercraft restricted to reside near the water surface.

• Aircraft: Fast moving targets approaching from high elevations.

• Missiles: Extremely fast moving targets.

• Terrestrial: People and vehicles, when in the vicinty of land.

• Submerged: Submarines and mines.

Real-world situations are also composed of a variety of phenomena and events that must be studiedfor maritime (and possibly other) surveillance systems. Some of these are:


RESTRICTED

RESTRICTED

• Dynamic background: This includes both the moving ocean and clouds. Also, whitecaps arepresent on the waves and the varying surface of the sea leads to changes in sun glint reflectedoff of it.

• Platform motion: In operation the cameras that are capturing the data will not be static. It is likelythough that the motion will be known (to some degree of accuracy) using Inertial MeasurementUnits (IMU), Global Position System (GPS) receivers, etc. Some causes of platform motion are:

– Vibration: The engines in the ship cause vibrations which could lead to blurring of imagesfor long exposures.

– Rotation: Even when anchored, a ship experiences roll and pitch disturbances due to thewaves. Whilst moving the ship will experience yaw and lean angles when turning as well aspitch effects as waves are crested.

– Translation: A moving ship will see the same background objects from different angles andat different distances.

– Twisting motion of the platform if it is large in size (e.g. a ship).

• Uncontrolled lighting: The system has to operate in an outside environment. This means thatlighting will change:

– 24 Hour Operation: The level of ambient illumination (in the visible spectrum) changesdrastically throughout the course of a day, varying from a milli-Lux in overcast starlight witha new moon to well over 0.1 Mega-Lux at midday in summer in the tropics.

– Clouds: Passing clouds can cause localised changes in illumination in the shadows theycast.

– Shadows: The direction of shadows of objects in the field of view (FOV) change throughoutthe course of a day.

• Atmospheric effects: Fog and heat shimmer cause both slight perturbations in transmissivityand refraction, adversely affecting the the images acquired.

• Wildlife: Birds and animals could be present in the systems sensory area whilst not posing anactual threat.

• Real-time constraints: It is highly preferable to be able to process images at least as fast asthey are produced by the camera/s. While it is helpful to know, in hindsight, where the killingmissile came from, it is much more useful to be able to take some form of corrective action. Thisalso means that a system must be capable of running continuously (without overflowing buffersor variables etc).

1.2 SensorsMany different types of sensors are available with which data could be collected:

• RADAR: Useful for detecting large objects at a distance.

• Cameras: Different types of cameras exist, in both charged coupled device (CCD) and comple-mentary metal oxide semiconductor (CMOS) variants:


RESTRICTED

RESTRICTED

– Colour: Visual - In either Bayer pattern or true 3 colours per pixel (i.e. 3 CCD cameras).

– Greyscale: Visual (This is what we’re currently using, has some sensitivity in the near in-frared region too.)

– Low light: Visual - Electron multiplying CCD cameras, capable of working in milli-Lux envi-ronments, normally grayscale.

– High Frame Rate: Useful for detecting rapidly moving objects at close to medium range, butproducing vast amounts of data to be processed.

– Infrared: Able to detect the heat signatures, whilst avoiding the problems of refraction, glint,ocean turbulance, clouds, shadows and low light but very expensive and unlikely to be ableto be used to identify targets.

– Gated imaging cameras (range gating).

• Ultraviolet Missile Approach Warning: Detects UV radiated by missile plumes.

• Sonar: To detect submerged threats.

1.3 Project GuidelinesThis document provides a list of research areas and suggested topics for final year (4th year Engineer-ing or BSc Hons), MSc and PhD students. The topics list is not exhaustive and is subject to change asnew research areas and ideas emerge. The research areas are summarized as follows:

a) Image processing software and hardware.

b) Image enhancement.

c) Scene modeling, feature extraction and segmentation.

d) Target detection and tracking.

e) Classification, recognition and identification.

f) 3D vision.

g) Sensors, components and systems.

h) Measurements and modeling.

A large number of research topics are interrelated and, thus, should not be constrained too strictly withinthe above areas. Several topics are defined in great detail; some topics and research areas are generaland require more clarification over time. Thus, in some cases, students are expected to define theirown research topics that fit within the research areas of PRISM. For some of the image processing andcomputer vision research, this process can be simplified by examining the video test data available athttp://prism.csir.co.za (discussed below). It may be required that students do pre-processing on someof the data sets. For example, ground truth (tracking/segmentation) for several of the sequences do notexist and will need to be generated by PRISM students. The new ground truth data will be uploaded tothe PRISM server for use by other/future researchers. Furthermore, students are encouraged to helpextend these datasets through the capture of new video sequences and the generation of appropriateground truth data. The final approval of research grants will be made at the discretion of the PRISMResearch Management Committee (RIMC).


RESTRICTED

RESTRICTED

2 Research Topics in Image Processing Software and Hardware

2.1 Image Processing ParallelizationThe abundance of parallel processing architectures has grown enormously in recent times. Multipleheavy weight processors are available on a CPU, and hundreds of light weight processors are availableon graphics processors. It is often difficult to determine whether time should be spent on parallelizationand whether multi-threaded CPU programs should be written or the program ported to run on the GPU.The student would determine what benefits are obtainable from taking advantage of parallel processingand generate a set of guidelines, backed with experimental proof, as to what types of algorithms shouldbe implemented on a GPU and which on a CPU (multi- or single-threaded). Here is a suggested topicfor a 4th year student:

• Survey of GPU and heterogeneous algorithms implementations for Image Processing (4thYear): Many algorithms have been ported to run on the multi-core processors to take advantageof parallel processing performance benefits. A survey of which algorithms have been imple-mented, what performance gain they obtained as well as the implementation themselves arerequired.

2.2 Image Processing on a Smart PhoneMany new smart phones such as the IPhone and Nokia N-Series have both a powerul CPU and GPU.Combined with their built in camera and battery this makes them an ideal portable image processingplatform. Work in this area, such as augmented reality has already started and is available. It is desiredto set up a framework on a smart phone so that image processing techniques developed under thePRISM program can be implemented on mobile devices. These are, preferably, phones running theopen source Android mobile operating system.

2.3 Image Processing on an Intel Edison System on ChipIntel is making progress in producing low power system on chip (SoC) devices such as the Intel Edison.The potential benefit of using and x86 Intel processor over an Arm based processor, for example, is thatexisting software systems might be more easily deployed on x86. The Edison integrates an Intel dualcore Atom processor, an Intel Quark processor, 1GB RAM, 4GB solid state storage, an WiFi interfaceand a Bluetooth interface into a device roughly the size of an SD card.

The student is expected to test a number of image processing algorithms on the Edison and then reporton the performance and usefulness of such an Intel SoC.


RESTRICTED

RESTRICTED

3 Research Topics in Image Enhancement

3.1 Heat Shimmer (Scintillation) MitigationThis work is concerned with the mitigation of image distortion due to atmospheric effects caused bythermal layers/pockets over land and sea, normally experienced whilst observing objects over greatdistances via a narrow field of view surveillance system. Two approaches exist: a passive approachwhich relies purely on correcting the image distortion based on an understanding of the behaviour ofthe atmosphere, and an active approach which would allow for the testing of the atmospheric transferfunction with a known active (laser) source, and the correcting of the resultant distortion. PreviousPRISM and CSIR work on scintillation mitigation used image processing to reduce image blurring andto stabilise the image.

Future research can examine the problem of a moving camera platform and retaining the motion ofmoving objects. Including lucky region detection and better estimation of the atmospheric point spreadfunction would also be very useful.

3.2 Night Vision Noise ReductionThe cover of darkness is often used as a stealth mechanism. Criminals, the military, police and evenanimals use it with great effect. In order to support the gathering of information or to improve thesituational awareness of the observer, a camera system can be used. A completely passive surveillancesystem has the advantage of not divulging the tactical position of the observer.

Over the past few years emerging Electron Multiplying Charge Coupled Devices (EMCCDs) have ri-valled the performance of traditional image intensifier technology. Although these sensors do producean image in conditions were the human eye cannot, the image is plagued with the problem of noise. Ini-tial laboratory attempts show that image processing can improve the image quality under the conditionof both a stationary camera and scene. If either assumption is violated blurring and ghosting occurs.These phenomena are highly disconcerting to a person or system monitoring the scene. In order torealise a system that can be utilised in the field these shortcomings must be addressed. This projecttherefore is an investigation into the area of low light level surveillance with the result been a method forimproving the surveillance function with respect to detection, recognition and identification. Improve-ments in these categories could be measured in terms of time, probability and range. It is expected thatthe researcher develop an understanding of sensor technology, understand the current state of the art,propose and test various methods and report on their findings.

3.3 Real Time Colour BalancingIn many surveillance applications a wide area is covered. This coverage ma be obtained either bya staring array of cameras or by a rotating or steerable camera. In all of theses situations as theviewing direction of the cameras change relative to the Sun, objects that have the same chromaticcharacteristics appear different in the images. This may be due to part of a scene being shaded bypassing cloud cover, or the exposure and gain of a camera changing as it looks closer toward the Sun.

It is desired to balance the colours in each camera so that the colours remain constant. This will improve


RESTRICTED

RESTRICTED

the accuracy of feature based image processing and the fidelity of stitched panoramas created from theraw scenes.

The student would be required to perform a literature study on available colour balancing/equalisationtechniques and then implement and compare them for real-time application.

3.4 Optical Fourier processingIt is known that for certain lenses objects that are one focal length in front of the lens have their 2DFourier transform created one focal length behind the lens. If a small display is thus placed one focallength in front a lens, the Fourier transform of its image is created one focal length behind the lens.A spatial light modulator such as a high resolution LCD display can be placed at the Fourier imagelocation one focal length behind the lens. This LCD can then in real time convolve any the image withand desired filter, either for image enhancement or object detection. If a second lens is placed one focallength behind the LCD, then one focal length behind that second lens the original image convolved withthe filter response is created. If the original display in front of the first lens is from a live camera, anda second camera records the filtered image created one focal length behind the second lens, then alloptical real-time live processing/filtering system can be created. The filter response can be changed byadapting the pattern on the LCD between the two lenses.

The student would be required to perform a literature study on Fourier optical processing and createand test a working prototype system.


RESTRICTED

RESTRICTED

4 Research Topics in Scene Modeling, Feature Extraction andSegmentation

Scene modeling and image segmentation is an important step for surveillance systems. In this section,some important research areas are highlighted. The first two research areas are general topics forsurveillance systems. The remaining problem descriptions are for particular research areas that aredeemed very useful.

4.1 Scene Modeling and Segmentation in a Terrestrial EnvironmentBackground modelling and scene segmentation in terrestrial environments is a major problem forsurveillance systems. A number of factors impact on the performance of a system such as varyingillumination levels, crowded environments, dynamic backgrounds, birds, smoke and inclement weather.SANDF personnel also experience varying vegetation types from tropical forest to desert. The studentis expected to formulate a topic that addresses one or some of the following:

• Modeling and segmentation of foliage in, for example, savanna environments.

• Background modeling in scenes with fluctuating illumination. A common problem is light levelfluctuations caused by the movement of clouds.

4.2 Maritime Scene Modeling and SegmentationMaritime scenes share some of the same problems as terrestrial scenes. However, a major challengeis the modeling and segmentation of the ocean which is a dynamic region. Several ideas can beexamined:

• Modeling of the wave action of the ocean (in wide and narrow fields of view).

• Ocean segmentation using texture features.

• Computing the optical flow of the ocean as a cue for segmentation.

4.3 Graph Cuts for Image SegmentationGraph partitioning methods can be used to perform image segmentation. For example, the graph cutalgorithm considers a pixel or groups of pixels to be graph vertices. The similarity between thesepixels or groups of pixels is described by edge weights between the pixels. Segmentation by graphcuts is performed by partitioning the vertices into sets such that the weights of the removed edges areminimized. Hence, this can be viewed as an energy minimization problem.

The graphics processing unit (GPU) can be used to process graphs that are represented using textures.For this project, the student will survey the literature on image segmentation on the GPU using graphpartitioning. He will then develop a new algorithm or improve an existing one. The algorithm will thenbe implemented to run on the GPU for a particular application (video processing). Use of the GPU


RESTRICTED

RESTRICTED

for image segmentation implies that the algorithm will run in real time or be significantly faster thantraditional CPU based processing. Alternate topics in energy minimization may be considered if theycan be used for image segmentation on the GPU e.g. active contours and pixel (data) clustering.

4.4 Generating and Processing Super-Pixel ImagesGraph cuts (mentioned above) is one way of generating what is known as super-pixel images. Super-pixels are useful for image segmentation as discussed, but also for potentially reducing the bandwidthof image and video data. The student should investigate different ways of efficiently generating, storingand processing super-pixel images.

4.5 Motion Detection from a Moving CameraA surveillance camera is unlikely to be completely static or experiencing pure rotation. More often thannot, it will be a moving system and background objects will be moving across its field of view at differentrates. The student is expected to model the optical flow field created such that objects of interest can besegmented. Initial work can look at processing video captured in a controlled environment. Interestingsituations for this problem include dynamic backgrounds, noisy data, motion blur and small targets.Particular topics should consider:

• Motion segmentation.

• Detecting of moving targets.

• Dense motion field estimation.

4.6 An Efficient Implementation of the DPT Transform for Fast ImageProcessing

The Discrete Pulse Transform (DPT) is a scale space representation that has recently shown promisingresults for image and video processing.

Some work has been done to create an efficient implementation of the DPT. A possible research projectis to study the DPT further and implement an efficient decomposition algorithm using a multi-coreprocessor.

4.7 Detection and Segmentation of Salient ObjectsSeveral image cues can be combined within a conditional random field (CRF) framework to detect andsegment salient objects. This approach should be considered for object detection; computer visionmodels for particular scenes can be used in a CRF.


RESTRICTED

RESTRICTED

5 Research Topics in Target Detection and Tracking

Target detection and tracking is an important requirement for the South African Navy. Under the PRISMprogram, target tracking in the maritime environment is one of the primary focus areas. Maritime scenesare generally cluttered with flying birds, dynamic backgrounds and sea glint. Furthermore, scenescaptured with a high zoom camera can exhibit low contrast, scintillation and adverse camera motion. Alarge volume of data is available for this detection and tracking problem. There also exists the option ofgenerating synthetic test data by simulating and embedding particular targets and trajectories in video.This option is particularly effective for also generating ground truth data. The tracking problem formaritime scenes will consist of detecting targets of interest and tracking them through the initializationand update of a target signature model within a tracking filter. The target model may incorporate awide range of measurements such as pixel luminance, shape, appearance cues and depth information.These initial functions can be extended to include tagging, classification and identification of the targets.It is assumed that the tracking filter will address occlusions and false alarms. Some of the inputs tothe tracking system will include the video rate and resolution. It is assumed that range information isunavailable. The following topics are suggested:

• Tracking Targets Against a Dynamic Background: Turretted systems with high zoom cam-eras are used to track targets of interest on the ocean. This type of system is able to provide ahigh resolution visual of a target of interest for the purpose of classification. However, the track-ing problem under this scenario is made even more complex by the camera motion, dynamicbackground of the ocean, low contrast and sea state. It is required that the target be effectivelysegmented and tracked to maintain lock-on.

• Target Detection in a Cluttered Environment: Detection of targets under severe sea conditionsgenerate a significant number of false positives. Some of the factors that affect the performanceof a detector are sea glint, white caps (of waves), birds, dynamic nature of the ocean, cameramotion, sensor noise and sea spray. An object detector must be developed to minimize thenumber of false alarms. The problem can be approached, as a start, by first detecting all movingobjects (rather than a particular type of object). It is assumed that the camera is non-stationary.For simplicity, it can also be assumed that a wide field of view camera is available for videoprocessing.

• Small Target Detection: This topic is related to the one above. However, the primary focus is onsmall target detection. In the ideal case, targets that are one pixel in size will be detected. How-ever, a minimum detectable target size can be set at 3x3 pixels for perfect operating conditions.For typical scenes, small targets will be considered to be 5x5 pixels or less. The aforementioneddescription is a general one.


RESTRICTED

RESTRICTED

6 Research Topics in Classification and Identification

The concepts of classification and identification are pertinent to detecting potential threats and reactingto them. They are also relevant for data mining in surveillance video. Current PRISM research work fallsunder the central category of target and weapon classification with the long term intention of developingsystems for intent detection.

It is often required to automatically determine and understand the contents of an image. In the maritimeenvironment, for instance, it is useful to know if a moving object is a boat, ship, yacht, bird, whale or (onthe landward side) a person or vehicle. In high resolution images, aspects such as weapon detectionand classification are possible. It is not necessarily required that one algorithm can detect and classifyall potential targets and weapons. Two research topics are currently relevant:

• Real-time Target Classification: Real-time classification of targets is a basic requirement forassessing potential threats. An initial project will be required to classify targets in a maritimescene. Several datasets are available for common targets such as boats and jetskis. A potentialstudent will be required to develop a detector and classifier for some of these target classes. Thereal-time constraint can be ignored if other complexities are addressed (such as the developmentof online detectors and classifiers).

• Content Based Image Retrieval Concepts: Content based image retrieval (CBIR) uses imageprocessing and computer vision techniques to search for images or video in very large databases.The aim is to process and analyze the actual contents of the images rather than any metadataassociated with the data. A CBIR system is able to avoid having users manually tag data andit improves search results by minimizing the amount of useless data returned. From a defenceperspective, it is obvious that large volumes of video data are generated daily by surveillancesystems. It would be infeasible and exhaustive for a human operator to view all the data andextract useful information. The CBIR system can automate this task and also annotate videoswith image information that is consistent across all data files. This can be achieved, for example,through querying by example images or example image data (such as image patches or blobsof colour and texture). A CBIR system will use pattern classification techniques for retrieval oftargets of interest and other objects in surveillance video.


RESTRICTED

RESTRICTED

7 Research Topics in 3D Computer Vision

3D vision is an emerging research area within the PRISM program. Currently, the focus is at an ex-ploratory level and will lead to more well defined projects in the future. A long term goal concernsbuilding a system that can generate a 3D view of an area under surveillance together with the 3Dtargets and their trajectories.

7.1 Computational Photography ConceptsComputational photography refers to combining computing, digital sensors and modern optics to extendthe capabilities of traditional digital cameras. Through the use of computational photography ideas,techniques are available, for example, for extending the dynamic range and depth of field in an image,image completion and image compression. More complex techniques are available for creating rangedata, 3D models and 4D light fields. Previous PRISM work involved exploration of coded apertureand coded exposure photography. It would be interesting for a student to use commercially availablecameras and optics (such as Nikon or Canon) to examine the concepts of 3D ranging and 4D light fieldsusing computational photography. Another interesting related topic is that of compressed sensing. Thisis the exploitation of the sparsity and incoherence of information in a signal (in this case an image)by sampling below the Nyquist frequency to obtain results of comparable fidelity using much fewersamples. This has application in small target detection and low bandwidth communication scenarios.

7.2 Photogrammetric Stereo using a Plane SweepStandard research on stereo vision focusses on the creation of disparity maps. Much work has goneinto techniques to create these maps and on how to rectify the images to make this as efficient aspossible. There is another technique to perform stereo vision that avoids not only the unnecessarycreation of a disparity map but also directly yields the range to each pixel. This technique is called theplane sweep and uses the camera geometric calibration and relative positions to create a stitch of thescene at a specific distance (in a plane perpendicular to the major axis of the stereo pair). It is thenpossible to see if a point in one of the input images is stitched correctly at the current plane distance.By performing this stitching at several distances (sweeping through the planes) it is then possible todetermine what the distance is to every point observed by both cameras. The student is required tocreate a stereo camera pair, calibrate it, and then implement both the plane sweep method and severaldisparity map based methods to determine which method is more efficient, more GPU friendly, morerobust and more accurate.

7.3 Plenoptic ImagingLight field imaging (sometimes known as plenoptic imaging) is an emerging technology that typicallyuses imaging hardware akin to an insect eye to provide many slightly different perspectives of thesame scene. These different views can be combined computationally to do many different potentiallyuseful things; for example, the distance to various objects in the scene may be computed (passiverange finding), the scene may be converted to a full three-dimensional map, and composite views thatallow us to see around small obstructions (such as foliage) can be created. These sorts of capabilities


RESTRICTED

RESTRICTED

have many potential applications in defence settings, such as for covert target ranging, autonomousnavigation, obstruction mapping, and so on.

The student would be required to perform a literature study on light field imaging and either constructor purchase a light field camera and implement some algorithms to further investigate the suitability ofsuch cameras to surveillance purposes.


RESTRICTED

RESTRICTED

8 Research Topics in Sensors, Components and Systems

The projects under this section are concerned with semiconductor physics, optical circuits, electrome-chanical systems and their applications.

8.1 Integrated Optical Circuits (IOCs), Micro Optic and Electromechan-ical Systems (MOEMS) and Photonic Integrated Sensors

Photonic Integrated Circuits (PICs) are devices that integrate multiple photonic functions. They areanalogous to electronic integrated circuits.

Micro-Opto-Electro-Mechanical Systems (MOEMS) is not a special class of Micro-Electro-MechanicalSystems (MEMS) but in fact it is MEMS merged with Micro-optics which involves sensing or manip-ulating optical signals on a very small size scale using integrated mechanical, optical, and electricalsystems. MOEMS includes a wide variety of devices including optical switch, optical cross-connect,tunable VCSEL, microbolometers amongst others. These devices are usually fabricated using micro-optics and standard micromachining technologies using materials like silicon, silicon dioxide, siliconnitride and gallium arsenide.

Photonic integrated sensors are analogous to photonic integrated circuits. They will enable high perfor-mance, low cost manufacturing and reliability.

Currently, the following topics are suggested for this field:

• Gated Sensor on a Chip (Msc): The design and implementation of integrated circuit photonicsensor on CMOS which will be used together with an active pulsed laser source to establish anactive gated imaging system. The CMOS sensor should have the following broad requirements:

– Initially a small matrix array with fast access to individual all pixel sensors simultaneously.

– Attempt if possible, a few detector concepts to be fabbed and tested e.g. normal PIN diodes(which have no inherent gain), photo avalanche detectors (which have inherent gain), photoHBT’s (which also have inherent gain).

– Each pixel should be as sensitive as possible to detection in the wavelength band of inter-est (i.e. high quantum efficiency); typically visual up to 5µm wavelength (i.e. the mid IRspectrum). The choice of the wavelength band will be determined by what may be feasiblyprocessed on CMOS. Consider the use of InGaAs which will be usable up to 2µm as astarting point, if this is at all possible. However to start with it would be better to concentrateon the fast gating and short integration times in the wavelengths up to approximately 1µm.

– The sensor to be gated electronically as follows ( i.e. time after sending of laser pulse attime T0):

∗ Start of sensing: after T0 to be as short as possible. This will test the design andprocess as to the fastest ON time after T0 to start detection. The maximum delay in thestart of sensing can be to be in the region of 60 µs (equivalent to a round trip of 10kmsfor the laser light), or more if feasible. This is not too critical at this stage and must notlimit the design or unnecessarily introduce design complications. A shorter time can be


RESTRICTED

RESTRICTED

considered initially relating to a few kilometres round trip if this becomes a complicatingfactor.

∗ The period of sensing: to be variable from as short as feasibly possible, (again totest the design in terms of switching times, detector sensitivity, and S/N ratios) to 3µs(equivalent to 1km of light distance travel).

∗ Multi pulse integration: The illuminating laser will have some Pulse Repetitive Fre-quency (PRF) in the region of Hz to KHz. This can be used to do integration of signaldetection at pixel level. Two options exist: Integrate within the pixel for a number ofpulses at the PRF and then readout the analogue value to be buffered and amplifiedfor transmission to the digital CMOS for AD conversion. This will depend on carrierlifetimes within the semiconductor detector. To read out the detector voltage after eachdetection pulse into a closely situated capacitor through analogue switching circuitryand integrate there, as carrier lifetimes will then not play a part, but only capacitorcharge leakage. At a suitable integration period the capacitor voltage can then be readout to the CMOS AD circuitry for further processing.

– Consider the use of an integrated peltier cooler at the detector level to enhance sensitivity,and detection S/N ratio. This should have the option of varying the detector temp fromambient to its lowest feasible temperature, or anything in between.

This project will require inputs and discussions from Denel Dynamics, Detek and Prof Moniko duPlessis (UP).

• Time of flight and velocimetry sensor Time of flight sensors detect the difference in phase be-tween reflected light and the low frequency modulated light that they receive. This is performedindependently for each pixel. It would be desirable to investigate whether these physical architec-tures could detect not only the phase shift but also the frequency shift of either the emmitted lightor the modulation. This would allow velocimetry via the Doppler effect.

• Accelerometers: An accelerometer is an instrument used to measure proper acceleration (thephysical acceleration experienced by an object). Proper acceleration is, thus, acceleration relativeto a freefall observer. Accelerometers are commonly used to sense position and velocity. PRISMresearch can look at accelerometers for use with fiber optic gyros to enable inertial navigation.Some aspects to be considered are improvements in sensitivity and size. The subsection onnavigation sensors and algorithms overlaps with work on accelerometers. For example, fiberoptic gyros are used to sense orientation and this is used in conjunction with accelerometers forparticular applications requiring precise object pose estimation.

• Photonic combiners, processors, phase manipulation modulators and routers.

8.2 Gated CamerasGated imaging requires an independent light source and detector, which operate outside the bandsof solar/ordinary radiation. This makes the detector blind to ordinary radiation/stray light such thatthe independent light source is necessary for scene illumination. Near infrared illumination makes thesystem ideal for night time observation since the emitted light is not visible to an observer without aninfrared detector. Operating the light source at a given frequency/gating also enhances the performanceof the system. Synchronizing the detector with the light source allows the operator to collect images


RESTRICTED

RESTRICTED

from specific distances by collecting the rays which are incident with the object of interest. This functionhas a see through fog/snow capability since some of the scattered light is not detected. Gated camerasare also known as time-of-flight cameras. It is desired to build a 3D map of the surrounding environment,so that objects of interest can be found. In summary, the gating function collects rays which have anidentical path length which is beneficial for see through fog capabilities or observing objects which areat a certain distance from the detector. CSIR has both a commercial gated camera and one that itconstructed. These may be used at the Scientia campus for this work.

Potential work for gated imaging include 3D data processing (for map building), identifying objectsof interest, single emitter multiple receiving aperture investigations, and the testing and analysis ofpenetration through foilage, rain, and fog.

8.3 LasersLaser research under the PRISM program is linked directly to electronic warfare countermeasures,surveillance and (remote) sensing. Some of the research done will focus on new laser sources, tendingto concentrate on fiber lasers due to their compactness, robustness and overall efficiency, althoughother technologies will not be excluded if they show benefits for military application. Work will alsofocus on laser protection concepts, gated laser imaging for remote sensing and laser based proximitysensors. The work on fiber lasers may focus on some of the following aspects:

• Low power fiber lasers for use in fiber optic gyros and gated laser sensors (LADAR).

• High power fiber lasers for use in dazzlers.

• Single fiber AND Multi fiber (steerable, combining in space).

• Frequency agility and flexibility of the fiber lasers.

The following topics are proposed for laser research:

• Video Target Ranging using LIDAR (MSc): LIght Detection And Ranging (LIDAR) is an opticalremote sensing technology that measures properties of scattered light to find range and/or otherinformation of a distant target. The prevalent method to determine distance to an object or surfaceis to use laser pulses. Like the similar radar technology, which uses radio waves, which is lightthat is not in the visible spectrum, the range to an object is determined by measuring the timedelay between transmission of a pulse and detection of the reflected signal. It is desired touse this method to determine the range to one or more horizontal lines visible to a camera todetermine the distance of the points along a scan-line. The end aim is to build a distance map ofthe environment to detect targets of interest.

• Laser Protection Concepts (PhD): For this work, the incumbent is expected to develop materialsor techniques for restricting laser intensity transimission through optics. It will be used for humanself protection against laser dazzlers. The following criteria must be met:

– Tunable or wideband effect: either absorb or reflect.

– Wavelengths of interest: visual up to 5um wavelength.

– It must clamp the energy transmitted through the material above a value which should betunable.


RESTRICTED

RESTRICTED

– It must clamp fast: <ns.

– It must recover quickly: <ns.

– Selective area clamping i.e. It should not display a ’blooming’ effect (as CCDs do).

– Practical material: a manufacturable coating or impregnated film.

– Not damaged under high laser energy exposure.

– Stable under operating environmental conditions and with time.

– No attenuation of low intensity light.

• Laser designation absorption material (MSc/PhD): Lasers are used both for ranging to anddesignating targets. A material that absorbs the common laser wavelengths used for these pur-poses would provide additional protection. Research into such materials is desired.

• Gated Laser Imaging for Remote Sensing (PhD): The project as a high level sensing systemwill allow for the research and development of various concepts and technologies. Its ultimategoal will be to explore technologies which would enhance remote sensing over short and longdistances under adverse environmental conditions and in so doing push the boundaries of thelocal photonics capability. The final application in a demonstrator system would be to:

– To detect and track various targets targets (stationary or slow and fast moving small andlarge targets on ground, sea or air) at short (10’s of meters to large distances (10’s ofkms) using ’pulse doppler’ (or any other concept similar to those used within the radarcommunity), within a high dynamic clutter environment.

– To be eye safe (by choice of wavelength and/or energy) but also allow for high energy outputin a countermeasure mode which would a cater for the overloading of electronic sensors.

– In addition to its target detection and tracking ability, to detect and classify mademade ob-jects ( e.g. optic lenses, eyes, camouflaged objects, metallic objects, people).

– To use novel sensitive and high dynamic range sensors even if this requires specially devel-oped integrated sensors (which could also be done as part of a LEDGER project at CEFIMif required).

– To have the ability to produce a 3D map of the scanned volume of interest, and to classifythe enviroment within this volume.

– To have high resolution in the X, Y and Z axis.

• Laser Based Proximity Sensors for Detecting of Objects of Interest Travelling at HighSpeed (PhD)

• Phased Array Lasers: This refers to hardware technology that uses surface elements to controlthe phase and amplitude of light waves reflecting or transmitting from a 2D surface or from anarray of laser sources, allowing for spatial combining of energy.

• Development of high power mid-infra-red fibre lasers Fibre lasers are currently receivingworldwide interest due to its good beam quality, its high efficiency, compactness and lower cost.Major advances have been made in recent years and they are fast becoming a major competitorto the more established solid-state lasers. However, there is a lack of knowledge on the subjectamongst the South African photonics community. The goal is to do research in high averagepower fibre lasers and to develop record breaking, high power fibre lasers in the mid-infraredregion.


RESTRICTED

RESTRICTED

• Development of high power optical parametric oscillators in the mid- and far- infrared re-gion This project proposes research in nonlinear optical conversion techniques in the middle-and far-infrared region, particularly the study of Optical Parametric Oscillators (OPO). Coher-ent sources in the mid-infrared are particularly attractive where efficient propagation through theatmosphere is desired. Three main atmospheric transmission windows are present in the mid-infrared region, namely the 2µm, 3-5µm and 8-12µm windows. The problem however, is thathigh-power mid-IR laser sources are limited, especially when compared to other wavelength re-gions. The aim is to demonstrate efficient nonlinear conversion in the mid- and far-IR region;illustrating alternative sources of coherent light in this region in addition to creating local exper-tise and establishing the foundation for larger projects in the future.

• Secure optical communications Secure long distance quantum communication, with quantumentanglement as the core resource, is mooted as a future technology and is the subject of intenseresearch the world over. Before quantum communication can become a fully commercializedtechnology which enables telecommunication secure against eavesdropping, there are still twomain challenges to be overcome: increased bandwidths and distances between communicationnodes have to be achieved. It is the central aim of this project to develop the enabling photonictechnologies necessary for the implementation of a secure quantum communication system thatcan perform quantum key distribution over long distance telecommunication networks.

8.4 Navigation Sensors and AlgorithmsThis is a general research area and topics should look at fiber optic gyros, accelerometers, GPS, imag-ing data and navigation algorithms. The use of a variety of sensors (and fusion of sensor information)is encouraged in order to improve the prediction of positional information. Typical applications includenavigation for man and autonomous robotic platforms underground, undersea, on land and in the air.These topics are also related to work on accelerometers (see an earlier subsection on accelerometers).

8.5 Optomechatronics Technologies• Optical Aperture Synthesis (OAS) (PhD): Research the OAS as a possible technique for narrow-

field surveillance through turbulent atmosphere using the closure phase technique. Problemsinclude aperture selection and long staring times to get sufficient data.

• Colour Sensor for Surveillance Alerting (4th Year): Study the use of a low cost multi-spectralsensor to detect colour anomalies in the environment.

• Characterisation of Light-to-Frequency Converters (4th Year): Research the use of light-to-frequency circuits for offering large dynamic range and simple interfacing. Light sensitivity andnoise performance are critical for precision applications. Correction for dark level and characteri-sation of linearity and dynamic range are also important issues. The absolute calibration of suchsensors would be a very useful capability arising out of the project.

• Construction and Characterisation of a High Power LED Illuminator (4th Year): High powerLEDs (e.g. Osram Oslon or LumiLEDs Luxeon) offer quasi-monochromatic optical power outputat low cost per watt. This project would involve construction and characterisation of a fan beamLED illuminator using red or blue spectrum, high power LEDs mounted in a line and collimated


RESTRICTED

RESTRICTED

using a parabolic trough constructed with mirror film from 3M or ReflecTech. The far-field spatialirradiance would have to be predicted and measured.

• Rayleigh Interferometer (MSc/PhD): Design and construct a Rayliegh interferometer for analy-sis of wavefronts distorted by atmospheric turbulence.

• Hypersensitive Temperature Sensor (MSc/PhD): A hypersensitive temperature sensor air mea-surement system which can measure temperature variations down to 1.0e-6 K and at frequenciesup to 10 kHz must be designed and constructed. Such an instrument can measure turbulencevariations directly. This instrument would problably be based on interferometric measurementconcepts, since a physical temperature measurement would be too slow and insensitive.

• Atmospheric Measurement (4th Year): Design and construct a data capture unit that enablesthe simultaneous measurement of temperature, humidity, wind speed and wind direction at aminimum of 5 points.

• SFTIRS: SFTIRS for UAV systems (MSc/PhD): A Static Fourier Transform Infrared Spectrom-eter (SFTIRS) can be used in a low-flying UAV to detect persons in remote areas. This hasapplication in border-crossing and poaching. It is a simple design for night time detection of peo-ple in remote areas and is lightweight and cheap. It can also be used on mutiple UAVs and worksat night (when visible sensors can’t). This project will study and present results for the aboveconcept.

• Fiber Based Sensors: The use of optical fiber or optical micro devices as a sensor for tempera-ture, orientation, acceleration and vibration strain should be examined for this topic.


RESTRICTED

RESTRICTED

9 Research Topics in Measurements and Modeling

The following topics are currently available for measurements and modeling:

• Measurement of Solar Concentrating Mirror Shape using DLP Projector (MSc/PhD): Inves-tigate the possibility of using a DLP projector system to measure the shape of a solar concentrat-ing mirror. Analyse the error sources. Construct a test mirror of known shape and implement themeasurement. The implementation would be similar to a scanning shack-hartmann arrangement,but without the mechanical scanning system.

• Solar Blind UV Sensing (MSc/PhD): Hamamatsu makes a device called a UVTRON (since theearly 60’s). This is a low cost sensor that operates in the solar-blind UV spectrum. The principleof operation is the photoelectric effect, with the work function of the photocathode manipulateddown into the UV. The angular spatial response is kind of lambertian, but the spatial response atthe detector is not given in the datasheet. Hamamatsu may have mapped it but don’t publish itopenly as far as one can tell. A few important questions should be tackled for this project:

a) How sensitive is the UVTRON ? Could it be used as a Missile Approach Warning (MAW)device?

b) If used in conjunction with concentrating optics to manipulate the angular spatial responseor increase the sensitivity in a particular direction, what does the angular response look like.How could efficient optics (including homogenisers such as light pipes and CPCs) be usedin conjunction with the UVTRON to get a uniform and well-defined angular response as wellas increase the range?

• 3D Atmospheric Radiative Transfer (MSc/PhD): There are several public codes for 3D radiativetransfer calculations, including Mystic, Grimaldi and the community code. The task would be toget one of the public 3D codes operational on a cluster computer and compare plane parallelresults to one of the existing 1D codes (libRadtran, MODTRAN etc.). Computation of 3D radiantenvironment maps would be a very useful adjunct.

• Instrument for Measurement of Aerosol Phase Functions (MSc/PhD): Measurement of aerosolphase functions is usually performed with a spectro-gonoimetric nephelometer. This task wouldinvolve the investigation of the architecture of these instruments and the construction of a simplephase function measurement instrument, along with characterisation. A possible light source for3-wavelength measurement is the PicoP MEMS steerable laser device.

• Long-Range Free-Space Optical Communications (FSOC) (MSc/PhD): Investigate issues re-lated to long-range FSOC, particularly effects of atmospheric turbulence and the state-of-the-artwith respect to turbulence mitigation.

• Time-Resolved Radiative Transfer using Monte Carlo Techniques (MSc/PhD): There areexisting community codes for solution of time-resolved radiative transfer problems. Applicationsinclude laser range-gated imaging and Optical Coherence Tomography (OCT) through scatteringmedia. This task would involve investigation of GPU-based photon transport codes and solutionof a selected problem.


RESTRICTED

RESTRICTED

• Atmospheric Path Geometry Predictor (MSc/PhD): Modtran server that predict future atmo-spheric path geometries, based on current flight paths, and then calculate the future atmosphericproperties such that it is immediately available when needed. A general interface can be devel-oped for such a server making it available over the network for virtually any purpose or project.

• Spatial wavelength converter (PhD): Derive a mathematical model for the absorption and emis-sion of photons and thermal distribution for various materials, surface finishes and microstruc-tures. The materials will be unknown. One will be expected to engineer test cases. It is expectedthat the input will be 1 micron light and output 1.5-5.5 micron light.

• Finite Element Modelling of Perturbed Lenses and Mirrors (MSc/PhD): The initial goal of theFinite Element Method (FEM) project is to produce a first order estimation of the Optical PathDifference (OPD) and ray deflections caused by therm-optical-elastic effects in axisymmetricallenses and mirror surfaces. The specific effects that will be estimated by the software are theOPD and ray deflections caused by:

– Stress-optical effects due to mechanical stresses introduced by mounting.

– Deflections of optical surfaces due to mounting stresses and gravity.

– Deflection of optical surfaces due to non-uniform temperature distributions.

– Changes in material refractive index due to temperature deviations and non-uniform tem-perature distributions.

– Stress-optical effects caused by non-uniform temperature distributions.


RESTRICTED

Documents

LEDGER PRISM PRISM Project List - University of Pretoria · 2015. 6. 26. · 5860-PRISM-800700-01 RPT Rev 1 Order KT521899,Item 7 Reference P13E LEDGER PRISM PRISM Project List by