Finger Detection for Multi-Touch Tabletop Display System

Finger Detection for Multi-Touch Tabletop Display System

Su-ting, Chuang2010/8/21OutlineIntroductionRelated WorkSystem and MethodExperimentConclusion & Future Work22OutlineIntroductionRelated WorkSystem and MethodExperimentsConclusion & Future Work33IntroductionNon-uniform lighting problemVarious finger touch response among different positionLow computation efficiencyNo such tool that helps users determine parameters automatically

44OutlineIntroductionRelated WorkSystem and MethodExperimentsConclusion & Future Work55Related WorkFTIR (Frustrated Total Internal Reflection)

J. Y. Han, Low-cost multi-touch sensing through frustrated total internal reflection," in Proceedings of the 18th annual ACM symposium on User interface software and technology (UIST '05). New York, NY, USA: ACM Press, 2005, pp. 115-118.

66Related WorkDI (Diffused Illumination)J. Rekimoto and N. Matsushita, Perceptual surfaces: Towards a human and object sensitive interactive display," Workshop on Perceptural User Interfaces (PUI'97), 1997.

7An IR camera with IR illuminators to observe hands

7Related WorkTouchLibA multi-touch development kit

Finger detection processing flow chart8

BackgroundSubtractionSimpleHighpass

ScaleThresholdFinger Analysis background (edge)Scale thresholdthreshold8Related WorkDirectShowFilter-based framework GShowGPU-accelerated frameworkCombination of DirectX and DirectShow

9Dshow: [media-streaming] Provide high-quality capture and playback of multimedia streams can perform high-quality video and audio playback or captureIt supports capture from digital and analog devices based on the Windows Driver Model (WDM) or Video for Windows. It automatically detects and uses video and audio acceleration hardware when available, but also supports systems without acceleration hardware.

Gshow: combination of directX and directShow

9OutlineIntroductionRelated WorkSystem and MethodExperimentsConclusion & Future Work1010Hardware Configuration

(2) IR Camera(3) IR Illuminator(1) Peripheral Projector1111Hardware ConfigurationOrder of diffuser layer and touch-glass layer12

Diffuser layerIR illuminatorIR cameraspotIR illuminatorIR cameraTouch-glass layerIR cameraspotIR camera

2143The surface of this kind of tabletop system is usually composed of a diffuser layer and a touch-glass layer.12Hardware ConfigurationProblem:IR rays reflected by the touch-glass will result in hot spot regions in camera viewsSolution:Use other cameras to recover the regions which are sheltered by IR spots

13

camera13Software ArchitectureDetection systemImage StitchingFinger DetectionFinger Tracking

Parameter determination14ImageStichingFingerDetectionFinger TrackingImage StitchingCombine multi-camera view into a virtual camera viewFinger DetectionRecognize touched fingertipFinger TrackingSmooth the trajectory of finger and fix lost results14Software Architecture15

ImageStichingFingerDetectionFinger Tracking15Image StitchingGoalCombine multi-camera view into a virtual camera view

16

Image StitchingAdvantagesRemove IR spot effectUnify finger size among different position of tableReduce matching problemBe compatible with existent finger detection system

17

Image Stitching18ImageBlendingIR Camera(L)IR camera(R)UndistortionUndistortionHomoWarpHomoWarpImage StitchingHomoWarp19

521625431436

2519Image StitchingImage Blending

20

20Finger DetectionTouchLib

Our method21NormalizationDifference of GaussianBackground SubtractionBinaryFinger AnalysisSimple HighpassScaleBackground SubtractionBinaryFinger Analysis

Finger DetectionNormalizationMethodModel distribution of IR illuminationUse specific material to simulate foregroundConstruct normalization map

Normalize foreground image

ResultBefore normalization: mean = 75, standard variation = 30After normalization: mean = 255, standard variation = 322

Construct normalization mapCalculate each pixels dynamic range Stretch dynamic range to 0-255Normalize foreground imageMultiply normalization ma

22Finger DetectionDifference of Gaussian (DoG)Modified from simple highpass in TouchLib

23

Simple highpass -blur, noise, touchLibmedian filter , midian filter time complexity

sigma = (n/2 - 1)*0.3 + 0.8 , n=kernel size23Fingertip TrackingGoalSmooth the trajectory of finger Fix lost resultsMethodKalman filterSmooth the path

Predict the new state and its uncertaintyCorrect the tracker with its new measurementAssume white noise and uniform velocityOriginalAfter Kalman filter2424Parameter DeterminationRequirements of ideal finger detection systemSensitive missNoise-free false alarm GoalFind an applicable set of parameters for finger detection system fulfilling the requirements25Parameter Determination26Parameters DeterminatorParameter CombinationDetection ResultApplicable set of ParametersTestSetTouch DataGround Truth(Trace)Detection SystemParameters Determinator : generate parameter combinations & evaluate parameter combinations by calculating miss and false alarm

26Parameter DeterminationEvaluation of parametersData CollectionDepict traceMeasurementMinimize # of miss and false alarm27

requirmentsmiss & false alarm27Parameter DeterminationIdeal finger detection Only one fingertip landing on traceContinuity among frames28

Good frameOnly one finger landing on traceContinuity

28OutlineIntroductionRelated WorkSystem and MethodExperimentsConclusion & Future Work2929ExperimentsPerformance evaluation30

30ExperimentsParameter determinationDecide parameters in our systemAdopt sampling-based parameter search technique

31NormalizationDifference of GaussianBackground SubtractionBinaryFinger AnalysisSubtract valueSmoothkernelThresholdFingerSize31ExperimentsParameter determinationExhaustive searchParameter combination5 (step) *5 (step) *5 (step) *5 (step) = 625Applicable parameter num16/625 = 2.56%

32Subtract valueSmooth kernelThresholdFinger sizeLow bound051010Step55510High bound2025305032ExperimentsParameter determinationParticle filtering33SamplingMeasure

Initialize particlesParticle generationWeight Computation12M12MNormalize WeightsMore iterationOver limited iterationResamplingoutputExitNoYesYesNoThe key idea is to represent the required posterior function by a set of random samples with associated weights and to compute estimates based on these samples and weights.

33OutlineIntroductionRelated WorkSystem and MethodExperimentsConclusion & Future Work34343535