Upload
pete
View
94
Download
5
Embed Size (px)
DESCRIPTION
Data Conversion & Integration. Data Conversion/Integration Process. Data Inventory Existing hard-copy maps / digital data Data Collection (additional ) Satellite Imagery, Aerial Photo, etc. Field Collection (hand-held devices-GPS, etc.) Data Input/Conversion - PowerPoint PPT Presentation
Citation preview
Workshop on Census Cartography and Management, Lusaka, Zambia, 08–12 October 2007
Data Conversion &
Integration
Workshop on Census Cartography and Management, Bangkok, Thailand, 15–19 October 2007
Data Conversion/Integration Process
• Data Inventory• Existing hard-copy maps / digital data
• Data Collection (additional )• Satellite Imagery, Aerial Photo, etc.• Field Collection (hand-held devices-GPS,
etc.)• Data Input/Conversion
• Keyboard entry of coordinates• Digitizing/Scanning/Raster-to-Vector• Editing/Building Topology
• Data Integration• Georeferencing/Geocoding
Workshop on Census Cartography and Management, Bangkok, Thailand, 15–19 October 2007
About Geographic Data
• Conversion of hardcopy to digital maps is the most time-consuming task in GIS
• Up to 80% of project costs
• Example: estimated to be a US $10 billion annual market
• Labor intensive, tedious and error-prone
Workshop on Census Cartography and Management, Bangkok, Thailand, 15–19 October 2007
Data Inventory
• National overview maps
• 1:250,000 and 1:5,000,000 (small scale)
• show major civil divisions, urban areas, physical features such as roads, rivers, lakes, elevation, etc.
• used for planning purposes
Workshop on Census Cartography and Management, Bangkok, Thailand, 15–19 October 2007
• Topographic maps- scales range from 1:25,000 to 250,000 (mid-scale)
• Town and city maps at large cartographic scales, showing roads, city blocks, parks, etc. (1:1,000 to 1:5,000)
• Maps of administrative units at all levels of civil division
• Thematic maps showing population distribution for previous census dates, or any features that may be useful for census mapping
Data Inventory (cont.)
Workshop on Census Cartography and Management, Bangkok, Thailand, 15–19 October 2007
Existing Digital Data
• Digital maps• Satellite imagery• GPS coordinates• Etc.
Workshop on Census Cartography and Management, Bangkok, Thailand, 15–19 October 2007
GPS
Remote SensingAerial Photography
Maps
Census & Surveys
Data Collection
GDB
Surveying.
GIS
Capture
Management
Workshop on Census Cartography and Management, Bangkok, Thailand, 15–19 October 2007
Aerial photography
• Aerial photography is obtained using specialized cameras on-board low-flying planes. The camera captures the image digitally or on photographic film.
• Aerial photography is the method of choice for mapping applications that require high accuracy and a fast completion of the tasks.
• Photogrammetry—the science of obtaining measurements from photographic images.
Workshop on Census Cartography and Management, Bangkok, Thailand, 15–19 October 2007
• Traditional end product: printed photos• Today: digital image (scanned from photo) in standard graphics
format (TIFF, JPEG) that can be integrated in a GIS or desktop mapping package
• Trend: fully digital process• digital orthophotos
• corrected for camera angle, atmospheric distortions and terrain elevation
• georeferenced in a standard projection (e.g. UTM)• geometric accuracy of a map• large detail of a photograph
Aerial photography (cont.)
Workshop on Census Cartography and Management, Bangkok, Thailand, 15–19 October 2007
Workshop on Census Cartography and Management, Bangkok, Thailand, 15–19 October 2007
Remote sensing process
Earth Surface
Sources of Energy
Sensing System
Receiving station
Workshop on Census Cartography and Management, Bangkok, Thailand, 15–19 October 2007
GPS• Collection of point data• Stored as “waypoints”• Accuracy dependent on device and environmental variables
Surveying
• Paper Based• Manual recording of information
• Electronic Based• Handheld device
Workshop on Census Cartography and Management, Bangkok, Thailand, 15–19 October 2007
Geographic data input/conversion
• Keyboard entry of coordinates• Digitizing• Scanning and raster to vector conversion• Field work data collection using• Global positioning systems• Air photos and remote sensing
Workshop on Census Cartography and Management, Bangkok, Thailand, 15–19 October 2007
Keyboard entry
• keyboard entry of coordinate data• e.g., point lat/long coordinates
• from a gazetteer (a listing of place names and their coordinates)
• from locations recorded on a map
Workshop on Census Cartography and Management, Bangkok, Thailand, 15–19 October 2007
Latitude/Longitude coordinate conversion• Latitude is y-coo, Longitude is x-coo
• Common format is degrees, minutes, seconds 113º 15’ 23” W 21º 56’ 07” N
• To represent lat/long in a GIS, we need to convert to decimal degrees
-113.25639 21.93528
• DD = D + (M + S / 60) / 60
Workshop on Census Cartography and Management, Bangkok, Thailand, 15–19 October 2007
Data Conversion
• Conversion is often the easiest form to import digital spatial data into a GIS
• Data transfer often rely on the exchange of data in mostly proprietary file formats using the import/export functions of commercial GIS packages
• Open source data Conversion software becoming widely available
Workshop on Census Cartography and Management, Bangkok, Thailand, 15–19 October 2007
Conversion of hardcopy maps to digital data
• Turning features that are visible on a hardcopy map into digital point, line, polygon, and attribute information
• In many GIS projects this is the step that requires by far the largest time and resources
• Newer methods are arising to minimize this arduous step
Workshop on Census Cartography and Management, Bangkok, Thailand, 15–19 October 2007
• Digitizing• Manual digitizing• Heads-up digitizing
• Scanning
• Raster-to-Vector
Conversion of hardcopy maps to digital data (cont.)
Workshop on Census Cartography and Management, Bangkok, Thailand, 15–19 October 2007
Manual DigitizingMost common form of coordinate
data input
• Requires a digitizing table • Ranging in size (25x25 cm to
150x200cm)
• Ideally the map should be flat and not torn or folded
• Cost: hundreds (300) to thousands (5000)
Workshop on Census Cartography and Management, Bangkok, Thailand, 15–19 October 2007
Digitizing steps (how points are recorded)
• trace features to be digitized with pointing device (cursor)
• point mode: click at positions where direction changes
• stream mode: digitizer automatically records position at regular intervals or when cursor moved a fixed distance
Workshop on Census Cartography and Management, Bangkok, Thailand, 15–19 October 2007
Control Points
• If a large map is digitized in several stages and the map has to be removed from the digitizing table occasionally, the control points allow the exact re-registration of the map on the digitizing board.
• Control points are chosen for which the real-world coordinates in the base map’s projection system are known.
Workshop on Census Cartography and Management, Bangkok, Thailand, 15–19 October 2007
Digitizing table
y
x
• Grid of wires in the table creates a magnetic field which is detected by the cursor
• X/Y coordinates in digitizing units are• fed directly into GIS
• High precision in coordinate recording
Workshop on Census Cartography and Management, Bangkok, Thailand, 15–19 October 2007
Heads-Up Digitizing I
• Features are traced from a map drawn on a transparent sheet attached to the screen
• Option, if no digitizer is available; but: accuracy very low
Workshop on Census Cartography and Management, Bangkok, Thailand, 15–19 October 2007
Heads-Up Digitizing II
• Common today is heads-up digitizing, where the operator uses a scanned map, air photo or satellite image as a backdrop and traces features with a mouse
• This method yields more accurate results
• Quicker and easier to retrace and save steps
Workshop on Census Cartography and Management, Bangkok, Thailand, 15–19 October 2007
• Raster-scanned image on the computer screen• Operator follows lines on-screen in vector mode
Heads-Up Digitizing II
Workshop on Census Cartography and Management, Bangkok, Thailand, 15–19 October 2007
Digitizing Errors
• Undershoots
• Dangles
• Spurious Polygons
Workshop on Census Cartography and Management, Bangkok, Thailand, 15–19 October 2007
Digitizing errors
• Any digitized map requires considerable post-processing• Check for missing features• Connect lines• Remove spurious polygons• Some of these steps can be automated
Workshop on Census Cartography and Management, Bangkok, Thailand, 15–19 October 2007
Fixing Errors
• Some of the common digitizing errors shown in the figure can be avoided by using the digitizing software’s snap tolerances that are defined by the user
• For example, the user might specify that all endpoints of a line that are closer than 1 mm from another line will automatically be connected (snapped) to that line
• Small sliver polygons that are created when a line is digitized twice can also be automatically removed
Workshop on Census Cartography and Management, Bangkok, Thailand, 15–19 October 2007
Advantages and Disadvantages of DigitizingAdvantages• It is easy to learn and thus does not require expensive
skilled labor
• Attribute information can be added during digitizing process
• High accuracy can be achieved through manual digitizing; i.e., there is usually minimal loss of accuracy compared to the source map
Workshop on Census Cartography and Management, Bangkok, Thailand, 15–19 October 2007
Advantages and Disadvantages of Digitizing
Disadvantages• It is a tedious activity, possibly leading to operator
fatigue and resulting quality problems which may require considerable post-processing
• It is slow. Large-scale data conversion projects may thus require a large number of operators and digitizing tables
• The accuracy of digitized maps is limited by the quality of the source material
Workshop on Census Cartography and Management, Bangkok, Thailand, 15–19 October 2007
Scanning A viable alternative to digitizing• The map is placed onto the scanning surface where light
is directed at the map at an angle• A photosensitive device records the intensity of light
reflected for each cell or pixel in a very fine raster grid• In gray scale mode, the light intensity is converted
directly into a numeric value, for example into a number between 0 (black) and 255 (white)
• In binary mode, the light intensity is converted into white or black (0/1) cell values according to a threshold light intensity
Workshop on Census Cartography and Management, Bangkok, Thailand, 15–19 October 2007
Scanning• Electronic detector moves across map and records light
intensity for regularly shaped pixels• Flat-bed scanner• Drum-scanner (pictured)
Workshop on Census Cartography and Management, Bangkok, Thailand, 15–19 October 2007
Scanning (cont.)
Types of scanners• Flat
• small format, low cost, good for small tasks
• Drum• high precision but expensive and slow
• Feed• fast, good precision, lower cost than
drum
R G B
pixelwidth
opticalsensor
colorsplicing
computer
Workshop on Census Cartography and Management, Bangkok, Thailand, 15–19 October 2007
• direct use of scanned images
• e.g., scanned air-photos
• digital topographic maps in raster format
Scanning (cont.)
Workshop on Census Cartography and Management, Bangkok, Thailand, 15–19 October 2007
• Scanner output is a raster data set usually needs to be converted into a
• Vector representation- manually (on-screen digitizing)- automated (raster-vector conversion)
line-tracing - e.g., MapScan
• Often requires considerable editing
Scanning (cont.)
Workshop on Census Cartography and Management, Bangkok, Thailand, 15–19 October 2007
Advantages and Disadvantages of Scanning
Advantages• Scanned maps can be used as image backdrops for
vector information• Scanned topographic maps can be used in combination
with digitized EA boundaries for the production of enumerator maps
• Clear base maps or original color separations can be vectorized relatively easily using raster-to-vector conversion software
• Small-format scanners are relatively inexpensive and provide quick data capture
Workshop on Census Cartography and Management, Bangkok, Thailand, 15–19 October 2007
Advantages and Disadvantages of Scanning
Disadvantages• Converting large maps with a small format scanners
requires tedious re-assembly of the individual parts
• Large format, high-throughput scanners are expensive
• Despite recent advances in vectorization software associated with scanning, considerable manual editing and attribute labeling may still be required
Workshop on Census Cartography and Management, Bangkok, Thailand, 15–19 October 2007
Raster to Vector ConversionGets scanned/image data into vector format
• Automatic mode: the system converts all lines on the raster image into sequences of coordinates automatically. automated raster to vector process starts with a line thinning algorithm
• Semi-automatic mode, the operator clicks on each line that needs to be converted; system then traces that line to the nearest intersections and converts it into a vector representation
Workshop on Census Cartography and Management, Bangkok, Thailand, 15–19 October 2007
OBIA Raster to Vector Conversion• Object-Based Image Analysis (OBIA) is a tentative name
for a sub-discipline of GIScience devoted to partitioning remote sensing (RS) imagery into meaningful image-objects, and assessing their characteristics through spatial, spectral and temporal scale. At its most fundamental level, OBIA requires image segmentation,
• attribution, classification and the ability to query and link individual objects (a.k.a. segments) in space and time. In order to achieve this, OBIA incorporates knowledge from a vast array of disciplines involved in the generation and use of geographic information (GI).
Workshop on Census Cartography and Management, Bangkok, Thailand, 15–19 October 2007
Object-Based Image Analysis
Workshop on Census Cartography and Management, Bangkok, Thailand, 15–19 October 2007
OBIA Dwelling Identification
• Segmentation Segmentation basedbased
• Pixel basedPixel based
• Automated Automated DigitizingDigitizing
Workshop on Census Cartography and Management, Bangkok, Thailand, 15–19 October 2007
Object-Based Image Analysis• Increasing demand for updated geo-spatial information,
rapid information extraction• Complex image content of VHSR data needs to be
structured and understood• Huge amount of data can only be utilized by automated
analysis and interpretation• New target classes and high variety of instances• Monitoring systems and update cycles• Transferability, objectivity, transparency, flexibility
Workshop on Census Cartography and Management, Bangkok, Thailand, 15–19 October 2007
Editing• Manual digitizing is error prone• Objective is to produce an accurate representation of the
original map data• This means that all lines that connect on the map must
also connect in the digital database• There should be no missing features and no duplicate
lines• The most common types of errors
• Reconnect disconnected line segments, etc
Workshop on Census Cartography and Management, Bangkok, Thailand, 15–19 October 2007
Some common digitizing errors
overshoot
line digitizedtwice
missingline
spike
undershoot
Workshop on Census Cartography and Management, Bangkok, Thailand, 15–19 October 2007
Building Topology• GIS determines relationships between features in the database
• System will determine intersections between two or more roads and will create nodes
• For polygon data, the system will determine which lines define the border of each polygon
• After the completed digital database has been verified to be error-free
• The final step is adding additional attributes
Workshop on Census Cartography and Management, Bangkok, Thailand, 15–19 October 2007
• The building of relationships between objects• Feature topology describes the spatial relationships between
connecting or adjacent geographic features such as roads connecting at intersections
• The user typically does not have to worry about how the GIS stores topological information
• Feature topology describes the spatial relationships between connecting or adjacent geographic features such as roads connecting at intersections
• The user typically does not have to worry about how the GIS stores topological information
Building Topology
Workshop on Census Cartography and Management, Bangkok, Thailand, 15–19 October 2007
Converting Between Different Digital Formats • All software systems provide links to other formats
• But the number and functionality of import routines varies between packages
• Problems often occur because software developers are reluctant to publish the exact file formats that their systems use -> instability of information (ex. file-geodatabase [.gdb])
• Option of using a third data format• Example: Autocad’s DXF format
Workshop on Census Cartography and Management, Bangkok, Thailand, 15–19 October 2007
Georeferencing/Geocoding • Georeferencing
• Converting map coordinates to the real world coordinates corresponding to the source map’s cartographic projection.
• Attaching codes to the digitized features (geocoded feature)
• each line representing a road would obtain a code that refers to the road status (dirt road, one lane road, two lane highway, etc.)
• Or a unique code that can be linked to a list of street names.
Workshop on Census Cartography and Management, Bangkok, Thailand, 15–19 October 2007
For attribute data:
• spreadsheets
• links to external database
• management systems (DBMS)
• tabulation programs (IMPS, Redatam)
Workshop on Census Cartography and Management, Bangkok, Thailand, 15–19 October 2007
Sample components of a digital EA map
1
2
3
4 56
7
12 13
891011
14
151617
18
19
20
2122
23
2425
26
272829
30
3132
33
34
35 36
37
38
39
4041
42
43
44
454647
48 49 50 51 52
5354
55
56
5758
59
6061
62
636465
66
67
68
69
70
71
7273
74 75
76
777879
80
81 82
83
848586
8788
1 2 3 4
9 10
1 2
7
891011
12
13
14
15
16
21
22
23
24
25 26 27
323334
19
20
21
2223
28
2930
3132
33
41
42
43
50
51
5254
58
59
27
28
31
37
38
42
43
4445
51
4340
4142
61 57
585960
65 62
6364
19
2021
22 31
32
3334
35
41424344
1
2
34
5
6
10
11
12 13
18
1920 21
2728
29
358
Enumeration Area Map Symbols
National Statistical Office -July 1998
Census2000
EA
Locality
District EA-Code
Hospital
Church
School
Buildingnumber
Province:District:Locality:EA-Code:
CartaniaChartesMaptown
14032
022100361
Approximate scale
N
45
17
Street Network Buildings
Boundaries Annotation and symbols
Building numbers Neatlines and legend
361
378
374
349350
358
377
362
358
Lambe r t Av en u e
Mercator Avenue
Cassini DriveCassini Drive
Im h of Drive
Eckert Drive
Miller
DriveBon n e Stree t
M o llw e id e S tre et
Grinten S
treet
Good e
Str e et
Bes se l S tre et
Stree t
Ro bins on S treet
Tisso t S treet
Ga ll S tree tPt olem
y Stree t
Orte l ius S
tr eet
Clar k e S
tr eet
To b le r Stre etSnyder Stre et
Krassowsk ij S
treet361
378
374
349 350
358
377
362
Enum eration Area Map Sym bols
Nationa l S tatistical O ffice - Ju ly 1998Census 2000
EA
Locality
D istrict EA-Code
Hospital
Church
School
Buildingnum ber
Province:D istrict:Locality:EA-Code:
CartaniaChartesMaptown
14032
022100361
500 100Approxim ate scale
200m
N
4 5
1
2
3
4 56
7
1 2 1 3
891011
1 4
1 51 61 7
1 81 9
2 0
2 12 2
2 3
2 4 2 5 2 6
2 72 82 9
3 0
3 1 3 233
3 4
35 3 6
37
3 8
3 9
4 04 1
4 2
43
44
454 647
48 4 9 5 0 51 52
5 35 4
55
5 6
5 75 8
5 9
6 06 1
6 2
6 36 46 5
6 6
6 7
6 86 9
7 0
71
7 273
74 7 5
7 6
7 77879
8 081 8 2
8 3
848 58 6
8 78 8
1 2 3 4
9 1 0
1 2
7
891 011
1 2
1 3
14
1 5
1 6
21
2 2
2 3
2 4
2 5 26 2 7
3 23 33 4
1 92 0
2 1
2 22 3
2 82 9
3 0
3 13 2
3 3
4 1
4 24 3
50
5 1
525 4
5 85 9
2 7
2 8
3 1
3 7
3 8
4 2
4 3
4 44 5
5 1
434 0
4 14 2
6 1 57
58596 0
6 5 62
6 364
1 9
2 02 1
22 3 1
3 23 33 4
35
4 14 24 34 4
1
2
34
5
6
10
11
12 1 3
1 8
1 9 20 2 1
2 728
2 9
17
Workshop on Census Cartography and Management, Bangkok, Thailand, 15–19 October 2007
A Simpler Alternative• In many countries, EA map design may be simpler than in
this example
• Instead of a fully integrated digital base map in vector format, rasterized images of topographic maps may be used as a backdrop for EA boundaries
• What is available already!
Workshop on Census Cartography and Management, Bangkok, Thailand, 15–19 October 2007
• In some instances, map features may be more generalized, for instance by using only the centerlines for the streets and polygons for entire city blocks rather than for individual houses
• This can include the use of free data as a baseline or starting point in the creation or updating of census related maps
A Simpler Alternative
Workshop on Census Cartography and Management, Bangkok, Thailand, 15–19 October 2007
Agencies to contact• National geographic institute / mapping agency• Military mapping services• Province, district and municipal governments• Various government or private organizations dealing with
spatial data• Geological or hydrological survey• Environmental protection authority• Transport authority• Utility and communication sector companies• Land titling & surveying agencies• Academic institutions• Donor activities
Workshop on Census Cartography and Management, Bangkok, Thailand, 15–19 October 2007
GPS coordinatecollection
Existing digitalmaps
Scanning
Raster to vectorconversion(automated or semi-automated)
Editinggeographicfeatures
Dataconversion
Sources ofgeographicinformation
Digital mapdata integration
Digitizing
Combine andintegrate digitalmap sheets
Coding (labelling)of digitalgeographicfeatures
Georeferencing(coordinate trans-formation andprojection change)
Constructtopology forgeographicfeatures
Generate linesand polygons
Additionalgeographic datacollection
Identify existingdata sources
Paper maps,existing printedair photos andsatellite images
Field mappingproductssuch assketch maps
Digital air photosand satelliteimages
Additionaldelineation ofEA boundaries
Parallel activity
Developgeographicattributesdatabase