(Active) 3D Scanning(Active) 3D Scanning

Theory and Case StudiesTheory and Case Studies

Range Acquisition TaxonomyRange Acquisition Taxonomy

RangeRangeacquisitionacquisition

ContactContact

TransmissiveTransmissive

ReflectiveReflectiveNon-opticalNon-optical

OpticalOptical

Industrial CTIndustrial CT

Mechanical Mechanical (CMM, jointed arm)(CMM, jointed arm)

RadarRadar

SonarSonar

UltrasoundUltrasound

MRIMRI

Ultrasonic trackersUltrasonic trackersMagnetic trackersMagnetic trackers

Inertial Inertial (gyroscope, accelerometer)(gyroscope, accelerometer)

Range Acquisition TaxonomyRange Acquisition Taxonomy

OpticalOpticalmethodsmethods

PassivePassive

ActiveActive

Shape from X:Shape from X:stereostereomotionmotionshadingshadingtexturetexturefocusfocusdefocusdefocus

Active variants of passive methodsActive variants of passive methodsStereo w. projected textureStereo w. projected textureActive depth from defocusActive depth from defocusPhotometric stereoPhotometric stereo

Time of flightTime of flight

TriangulationTriangulation

Active Optical MethodsActive Optical Methods

• Advantages:Advantages:– Usually can get dense dataUsually can get dense data– Usually much more robust and Usually much more robust and

accurate than passive techniquesaccurate than passive techniques

• Disadvantages:Disadvantages:– Introduces light into scene (distracting, Introduces light into scene (distracting,

etc.)etc.)– Not motivated by human visionNot motivated by human vision

Active Variants of Passive Active Variants of Passive TechniquesTechniques

• Regular stereo with projected textureRegular stereo with projected texture– Provides features for correspondenceProvides features for correspondence

• Active depth from defocusActive depth from defocus– Known pattern helps to estimate Known pattern helps to estimate

defocusdefocus

• Photometric stereoPhotometric stereo– Shape from shading with multiple Shape from shading with multiple

known lightsknown lights

Pulsed Time of FlightPulsed Time of Flight

• Basic idea: send out pulse of light Basic idea: send out pulse of light (usually laser), time how long it takes (usually laser), time how long it takes to returnto return tcd

2

1tcd

2

1

Pulsed Time of FlightPulsed Time of Flight

• Advantages:Advantages:– Large working volume (up to 100 m.)Large working volume (up to 100 m.)

• Disadvantages:Disadvantages:– Not-so-great accuracy (at best ~5 mm.)Not-so-great accuracy (at best ~5 mm.)

• Requires getting timing to ~30 picosecondsRequires getting timing to ~30 picoseconds• Does not scale with working volumeDoes not scale with working volume

• Often used for scanning buildings, Often used for scanning buildings, rooms, archeological sites, etc.rooms, archeological sites, etc.

AM Modulation Time of FlightAM Modulation Time of Flight

• Modulate a laser at frequencyModulate a laser at frequencym m ,, it it

returns with a phase shift returns with a phase shift

• Note the ambiguity in the measured Note the ambiguity in the measured phase!phase! Range ambiguity of Range ambiguity of 11//22mmnn

2

2

2

1 n

ν

cd

m

2

2

2

1 n

ν

cd

m

AM Modulation Time of FlightAM Modulation Time of Flight

• Accuracy / working volume tradeoffAccuracy / working volume tradeoff(e.g., noise ~ (e.g., noise ~ 11//500 500 working volume)working volume)

• In practice, often used for room-sized In practice, often used for room-sized environments (cheaper, more environments (cheaper, more accurate than pulsed time of flight)accurate than pulsed time of flight)

TriangulationTriangulation

Triangulation: Moving theTriangulation: Moving theCamera and IlluminationCamera and Illumination

• Moving independently leads to Moving independently leads to problems with focus, resolutionproblems with focus, resolution

• Most scanners mount camera and Most scanners mount camera and light source rigidly, move them as a light source rigidly, move them as a unitunit

Triangulation: Moving theTriangulation: Moving theCamera and IlluminationCamera and Illumination

Triangulation: Moving theTriangulation: Moving theCamera and IlluminationCamera and Illumination

Triangulation: Extending to 3DTriangulation: Extending to 3D

• Possibility #1: add another mirror (flying Possibility #1: add another mirror (flying spot)spot)

• Possibility #2: project a stripe, not a dotPossibility #2: project a stripe, not a dot

ObjectObject

LaserLaser

CameraCameraCameraCamera

Triangulation Scanner IssuesTriangulation Scanner Issues

• Accuracy proportional to working volumeAccuracy proportional to working volume(typical is ~1000:1)(typical is ~1000:1)

• Scales down to small working volumeScales down to small working volume(e.g. 5 cm. working volume, 50 (e.g. 5 cm. working volume, 50 m. accuracy)m. accuracy)

• Does not scale up (baseline too large…)Does not scale up (baseline too large…)

• Two-line-of-sight problem (shadowing from Two-line-of-sight problem (shadowing from either camera or laser)either camera or laser)

• Triangulation angle: non-uniform resolution if Triangulation angle: non-uniform resolution if too small, shadowing if too big (useful range: too small, shadowing if too big (useful range: 1515-30-30))

Triangulation Scanner IssuesTriangulation Scanner Issues

• Material properties (dark, specular)Material properties (dark, specular)

• Subsurface scatteringSubsurface scattering

• Laser speckleLaser speckle

• Edge curlEdge curl

• Texture embossingTexture embossing

Multi-Stripe TriangulationMulti-Stripe Triangulation

• To go faster, project multiple stripesTo go faster, project multiple stripes

• But which stripe is which?But which stripe is which?

• Answer #1: assume surface Answer #1: assume surface continuitycontinuity

Multi-Stripe TriangulationMulti-Stripe Triangulation

• To go faster, project multiple stripesTo go faster, project multiple stripes

• But which stripe is which?But which stripe is which?

• Answer #2: colored stripes (or dots)Answer #2: colored stripes (or dots)

Multi-Stripe TriangulationMulti-Stripe Triangulation

• To go faster, project multiple stripesTo go faster, project multiple stripes

• But which stripe is which?But which stripe is which?

• Answer #3: time-coded stripesAnswer #3: time-coded stripes

Time-Coded Light PatternsTime-Coded Light Patterns

• Assign each stripe a unique illumination Assign each stripe a unique illumination codecodeover time [Posdamer 82]over time [Posdamer 82]

SpaceSpace

TimeTime

Applications of 3D Scanning Applications of 3D Scanning – Scanning Sculptures– Scanning Sculptures

• The Pietà ProjectThe Pietà ProjectIBM ResearchIBM Research

• The Digital Michelangelo ProjectThe Digital Michelangelo ProjectStanford UniversityStanford University

• The Great Buddha ProjectThe Great Buddha ProjectUniversity of TokyoUniversity of Tokyo

Why Scan Sculptures?Why Scan Sculptures?

• Sculptures interesting objects to look Sculptures interesting objects to look atat

• Introduce scanning to new disciplinesIntroduce scanning to new disciplines– Art: studying working techniquesArt: studying working techniques– Art historyArt history– Cultural heritage preservationCultural heritage preservation– ArcheologyArcheology

• High-visibility projectsHigh-visibility projects

Why Scan Sculptures?Why Scan Sculptures?

• ChallengingChallenging– High detail, large areasHigh detail, large areas– Large data setsLarge data sets– Field conditionsField conditions– Pushing hardware, software technologyPushing hardware, software technology

• But not too challengingBut not too challenging– Simple topologySimple topology– Possible to scan most of surfacePossible to scan most of surface

Issues AddressedIssues Addressed

• ResolutionResolution

• CoverageCoverage– Theoretical: limits of scanning technologiesTheoretical: limits of scanning technologies– Practical: physical access, timePractical: physical access, time

• Type of dataType of data– High-res 3D data vs. coarse 3D + normal High-res 3D data vs. coarse 3D + normal

mapsmaps– Influenced by eventual applicationInfluenced by eventual application

• Intellectual PropertyIntellectual Property

IBM’s Pietà ProjectIBM’s Pietà Project

• Michelangelo’s Michelangelo’s “Florentine Pietà”“Florentine Pietà”

• Late work (1550s)Late work (1550s)

• Partially destroyed by Partially destroyed by Michelangelo, Michelangelo, recreated by his recreated by his studentstudent

• Currently in the Museo Currently in the Museo dell’Opera del Duomodell’Opera del Duomoin Florencein Florence

Who?Who?

• Dr. Jack Wasserman, professor Dr. Jack Wasserman, professor emeritus of art history at Temple emeritus of art history at Temple UniversityUniversity

• Visual and Geometric Computing Visual and Geometric Computing groupgroup@ IBM Research:@ IBM Research:

Fausto BernardiniHolly RushmeierIoana MartinJoshua Mittleman

Gabriel TaubinAndre GueziecClaudio Silva

ScannerScanner

• Visual Interface “Virtuoso”Visual Interface “Virtuoso”

• Active multibaseline stereoActive multibaseline stereo

• Projector (stripe pattern),Projector (stripe pattern),6 B&W cameras, 1 color camera6 B&W cameras, 1 color camera

• Augmented with 5 extraAugmented with 5 extra“point” light sources for“point” light sources forphotometric stereophotometric stereo(active shape from shading)(active shape from shading)

DataData

• Range data has 2 mm spacing, Range data has 2 mm spacing, 0.1mm noise0.1mm noise

• Each range image: 10,000 points, Each range image: 10,000 points, 202020 cm20 cm

• Color data: 5 images with controlled Color data: 5 images with controlled lighting, 1280lighting, 1280960, 0.5 mm 960, 0.5 mm resolutionresolution

• Total of 770 scans, 7.2 million pointsTotal of 770 scans, 7.2 million points

ScanningScanning

• Final scan June Final scan June 1998, completed 1998, completed July 1999July 1999

• Total scanning time: Total scanning time: 90 hours over 14 90 hours over 14 daysdays(includes equipment (includes equipment setup time)setup time)

PostprocessingPostprocessing

• Use 11Use 1111 grid of projected laser 11 grid of projected laser dots to help with pairwise alignmentdots to help with pairwise alignment

• Align all scans to each other, then Align all scans to each other, then apply nonrigid “conformance apply nonrigid “conformance smoothing”smoothing”

• Reconstruct surface using BPAReconstruct surface using BPA

• Compute normal and albedo maps,Compute normal and albedo maps,align to geometryalign to geometry

ResultsResults

The Digital Michelangelo The Digital Michelangelo ProjectProject

GoalsGoals

• Scan 10 sculptures by MichelangeloScan 10 sculptures by Michelangelo

• High-resolution (“quarter-millimeter”) High-resolution (“quarter-millimeter”) geometrygeometry

• Side projects: architectural scanning Side projects: architectural scanning (Accademia and Medici chapel), (Accademia and Medici chapel), scanning fragments of Forma Urbis scanning fragments of Forma Urbis RomaeRomae

Why Capture Chisel Marks?Why Capture Chisel Marks?

Atlas Atlas (Accademia)(Accademia)

ugnettougnettougnettougnetto

??

Why Capture Chisel MarksWhy Capture Chisel Marksas Geometry?as Geometry?

Day (Medici Day (Medici Chapel)Chapel)

2 mm2 mm

Who?Who?

Faculty and staffFaculty and staffProf. Brian CurlessProf. Brian Curless John GerthJohn GerthJelena JovanovicJelena Jovanovic Prof. Marc LevoyProf. Marc LevoyLisa PacelleLisa Pacelle Domi Pitturo Domi Pitturo Dr. Kari PulliDr. Kari Pulli

Graduate studentsGraduate studentsSean AndersonSean Anderson Barbara CaputoBarbara CaputoJames DavisJames Davis Dave KollerDave KollerLucas PereiraLucas Pereira Szymon RusinkiewiczSzymon RusinkiewiczJonathan ShadeJonathan Shade Marco TariniMarco TariniDaniel WoodDaniel Wood

UndergraduatesUndergraduatesAlana ChanAlana Chan Kathryn ChinnKathryn ChinnJeremy GinsbergJeremy Ginsberg Matt GinztonMatt GinztonUnnur GretarsdottirUnnur Gretarsdottir Rahul GuptaRahul GuptaWallace HuangWallace HuangDana KatterDana KatterEphraim LuftEphraim Luft Dan PerkelDan PerkelSemira RahemtullaSemira Rahemtulla Alex RoetterAlex RoetterJoshua SchroederJoshua Schroeder Maisie TsuiMaisie TsuiDavid WeeklyDavid Weekly

In FlorenceIn FlorenceDottssa Cristina Acidini Dottssa Cristina Acidini Dottssa Franca FallettiDottssa Franca FallettiDottssa Licia BertaniDottssa Licia Bertani Alessandra MarinoAlessandra MarinoMatti AuvinenMatti Auvinen

In RomeIn RomeProf. Eugenio La RoccaProf. Eugenio La Rocca Dottssa Susanna Le PeraDottssa Susanna Le PeraDottssa Anna SomellaDottssa Anna Somella Dottssa Laura FerreaDottssa Laura Ferrea

In PisaIn PisaRoberto ScopignoRoberto Scopigno

SponsorsSponsorsInterval ResearchInterval Research Paul G. Allen Foundation for the Paul G. Allen Foundation for the

ArtsArtsStanford UniversityStanford University

Equipment donorsEquipment donorsCyberwareCyberware Cyra TechnologiesCyra TechnologiesFaro TechnologiesFaro Technologies IntelIntelSilicon GraphicsSilicon Graphics SonySony3D Scanners3D Scanners

Scanner DesignScanner Design

• FlexibilityFlexibility– outward-lookingoutward-looking

rotational rotational scanningscanning

– 16 ways to mount 16 ways to mount scan head on armscan head on arm

• AccuracyAccuracy– center of gravitycenter of gravity

kept stationarykept stationaryduring motionsduring motions

– precision drives,precision drives,vernier homing,vernier homing,stiff trussesstiff trusses

4 motorized axes4 motorized axes

laser, range camera,laser, range camera,white light, and color camerawhite light, and color camera

Scanning a Large ObjectScanning a Large Object

• Calibrated motionsCalibrated motions– pitch pitch (yellow)(yellow)– pan pan (blue)(blue)– horizontal translation horizontal translation (orange)(orange)

• Uncalibrated motionsUncalibrated motions– vertical translationvertical translation– rolling the gantryrolling the gantry– remounting the scan headremounting the scan head

Single Scan of St. Single Scan of St. MatthewMatthew 1 mm1 mm1 mm1 mm

PostprocessingPostprocessing

• Manual initial alignmentManual initial alignment

• Pairwise ICP, then global registrationPairwise ICP, then global registration

• VRIP (parallelized across subvolumes)VRIP (parallelized across subvolumes)

• Use high-res geometry to discard bad Use high-res geometry to discard bad color data, perform inverse lighting color data, perform inverse lighting calculationscalculations

Statistics About the Scan of Statistics About the Scan of DavidDavid

• 480 individually aimed 480 individually aimed scansscans

• 0.3 mm sample spacing0.3 mm sample spacing

• 2 billion polygons2 billion polygons

• 7,000 color images7,000 color images

• 32 gigabytes32 gigabytes

• 30 nights of scanning30 nights of scanning

• 22 people22 people

Head of Michelangelo’s Head of Michelangelo’s DavidDavid

PhotographPhotograph 1.0 mm computer model1.0 mm computer model

Side project:Side project:The Forma Urbis RomaeThe Forma Urbis Romae

Forma Urbis Romae Forma Urbis Romae FragmentFragment

side face

Hard ProblemsHard Problems

• Keeping scanner calibrated is hard in Keeping scanner calibrated is hard in the lab, the lab, reallyreally hard in the museum hard in the museum

• Dealing with large data sets is painfulDealing with large data sets is painful

• Filling all the holes converges only Filling all the holes converges only asymptotically (if it converges at asymptotically (if it converges at all…)all…)

The Great Buddha ProjectThe Great Buddha Project

• Great Buddha of KamakuraGreat Buddha of Kamakura

• Original made of wood, completed Original made of wood, completed 12431243

• Covered in bronze and gold leaf, 1267Covered in bronze and gold leaf, 1267

• Approx. 15 m tallApprox. 15 m tall

• Goal: preservation ofGoal: preservation ofcultural heritagecultural heritage

Who?Who?

• Institute of Industrial Science,Institute of Industrial Science,University of TokyoUniversity of Tokyo

Daisuke MiyazakiDaisuke MiyazakiTakeshi OoishiTakeshi OoishiTaku NishikawaTaku NishikawaRyusuke SagawaRyusuke Sagawa

Ko NishinoKo NishinoTakashi TomomatsuTakashi TomomatsuYutaka TakaseYutaka TakaseKatsushi IkeuchiKatsushi Ikeuchi

ScannerScanner

• Cyrax range scannerCyrax range scannerby Cyra Technologiesby Cyra Technologies

• Laser pulse time-of-flightLaser pulse time-of-flight

• Accuracy: 4 mmAccuracy: 4 mm

• Range: 100 mRange: 100 m

ProcessingProcessing

• 20 range images (a few million 20 range images (a few million points)points)

• Simultaneous all-to-all ICPSimultaneous all-to-all ICP

• Variant of volumetric merging Variant of volumetric merging (parallelized)(parallelized)

ResultsResults