1

Build Your Own Data.gov GEO Viewer in the Cloud

Brand Niemann

http://semanticommunity.net

January 20, 2011

Overview• 1. Background• 2. Data.gov GEO Viewer• 3. USGS OpenView Map Viewer• 4. Google Earth• 5. US EPA National Geospatial Program• 6. Geospatial OneStop• 7. Spotfire Silver• 8. Geospatial Platform• 9. Sample Applications– 9.1 US EPA National Geospatial Program– 9.2 Geospatial One Stop– 9.3 Places and Spaces Atlas of Science– 9.4 NGA-NCOIC– 9.5 Build Your Own Data.gov GEO Viewer in the Cloud• 10. Some Next Steps

2

1. Background

• At the Web Site: http://data.gov– There is the Catalog: Raw Data, Tools, and

Geodata.• My Suggestion: What About All Three

Integrated?!

– There are Multiple File Types: XML, CVS/Text, KML/KMZ, Shapefile, Other?

• My Suggestion: Why Not Try Using Spotfire for all of these!

3

1. Background

• At the Web Site: http://datagov.ideascale.com/

– There is the Suggestion: http://datagov.ideascale.com/a/pmd/27611-6440

• developer wiki by michael.daconta

– His Suggestion: I think it is important to have a separate wiki set up for developers as there are many developer topics where collaboration is key.

• At the Wiki Site: http://semanticommunity.info/

– There is the Work: http://semanticommunity.info/Data.gov/GEOViewer

• Build Your Own Data.gov GEO Viewer in the Cloud by myself (and actually many others that I have collaborated with as you will see)

– My Suggestion: Why not try doing this with the new MindTouch Technical Documentation Suite (Wiki) and TIBCO Spotfire Silver, both cloud computing tools!

4

1. Background• 2011 Geospatial Committee, Emerging Technology SIG, ACT/IAC:

– Geospatial Layer of any FEA Implementation:• Where are we today?

– “How do we bring potential users — those we don’t think of as conventional users of geospatial data such as program or acquisition officers — and help them make better decisions, using tools that help visualize data geographically.” (Karen Siderelis, DoI GIO, Government Computer News, March 5, 2010).

• Where do standards come in?– Metadata, Web services, and other standards are important.

• Ontology and its role.– See http://cegis.usgs.gov/ontology.html

• Semantic interoperability and its potential.– See http://www.ogcnetwork.net/geosemantics

• Outcome – operating model?– “Groups have their own infrastructures and technologies. We really need cloud computing, it

provides a unique opportunity to leverage our technology investment to build a shared infrastructure.” (Jerry Johnson, EPA GIO, Government Computer News, March 5, 2010).

• Next steps.– A geospatial platform in the cloud that includes the above!

5

1. Background• 2011 Geospatial Committee, Emerging Technology SIG, ACT/IAC:

– Geospatial Layer of any FEA Implementation:• How best to engage with the FEA Geospatial Profile?

– My suggestion: Add the Geospatial Cloud Platform to the FEA Geospatial Profile.

• Where can emerging geospatial technologies and practices benefit the FEA?

– My suggestion: By supporting the Geospatial Cloud Platform.

• Develop a mission statement for the subcommittee.

– My suggestion: Applying emerging geospatial technologies to the Geospatial Cloud Platform.

• Produce a graphic of geospatial in enterprise architecture.

– Use Geospatial Platform Conceptual Model and Geospatial Platform Conceptual Architecture and Role of FEA (See slides 25 and 26).

• Develop workshop for the March meeting.

– My suggestion: Invite Jerry Johnson, EPA GIO, to provide an update on the Geospatial Platform & then have a discussion on how the ETSIG can support that.

• Look at semantics in the FEA and Geospatial Layer.

– http://www.ogcnetwork.net/geosemantics

• Look for opportunities for collaboration with SOCoP.

– http://semanticommunity.info/Spatial_Ontology_Community_of_Practice

6

2. Data.gov GEO Viewer

7http://www.data.gov/whatsnew/geoviewer

2. Data.gov GEO Viewer• We are pleased to announce the availability of the Data.gov GEO Viewer, an

interactive mapping tool designed to let users preview geospatial data available through the Data.gov catalogs. Using this tool, you can immediately view datasets on an interactive map, overlay them with other datasets, and investigate the underlying data. While not a full featured Geographic Information System, this new capability allows users to quickly determine whether datasets in the Data.gov catalogs are suitable for their needs before taking the time to download them for further exploration and use. Note: This was not “immediately” at least for the first two.

• To get started, simply click on the "Preview" link where it appears on a dataset's metadata page. Please note that not all geospatial data in the Data.gov catalog is available for preview at this time. The Preview link is included for those datasets that are currently supported. Note: See next slide.

• We hope that you find the Data.gov GEO Viewer to be a useful tool in helping you search for, use, and understand the geospatial data holdings of Data.gov. Please give us feedback on the Viewer, and let us know what new features and tools you would like to see in the future. Note: Do not find section for Data.gov GEO Viewer!

8

Comment: I made the suggestion about a year ago that Data.gov try Spotfire for viewing and working with geospatial data and now I am going to show what I have done with it.

2. Data.gov GEO Viewer

9EPA Region 1 Environmentally Sensitive Areas

3. USGS OpenView Map Viewer

10http://mapdss1.er.usgs.gov/openview/

Note: I Put the Map Layer Metadata in a Spreadsheet and in Spotfire.

4. Google Earth

11Request Access: http://csms.terradex.com/PublicPages/Application.aspx

5. US EPA National Geospatial Program

12http://www.epa.gov/enviro/geo_data.html

Note: I used these in Section 9.

5. US EPA National Geospatial Program

• Keyhole Markup Language (KML) is an XML schema for expressing geographic annotation and visualization within Internet-based, two-dimensional maps and three-dimensional Earth browsers. KML was developed for use with Google Earth, which was originally named Keyhole Earth Viewer. It was created by Keyhole, Inc, which was acquired by Google in 2004. The name "Keyhole" is an homage to the KH reconnaissance satellites, the original eye-in-the-sky military reconnaissance system first launched in 1976. KML is an international standard of the Open Geospatial Consortium. Google Earth was the first program able to view and graphically edit KML files.

• The KML file specifies a set of features (place marks, images, polygons, 3D models, textual descriptions, etc.) for display in Google Earth, Maps and Mobile, or any other 3D Earth browser (geobrowser) implementing the KML encoding. Each place always has a longitude and a latitude.

• KML files are very often distributed in KMZ files, which are zipped files with a .kmz extension.

• Note: Need to convert KML/KMZ files to CSV for use in Spotfire.– David Smith @ EPA recommends: http://choonchernlim.com/kmlcsv/

13Source: http://en.wikipedia.org/wiki/Keyhole_Markup_Language

6. Geospatial OneStop

14http://gos2.geodata.gov/wps/portal/gos

Note: I put this in Spotfire (325 MB).See slide 16.

6. Geospatial OneStop• Geospatial One-Stop makes it easier, faster, and less expensive for all levels of

government and the public to access geospatial information. The geodata.gov portal is designed to facilitate communication and sharing of geographic data and resources to enhance government efficiency and improve citizen services.

– Statistics (I put these in Spotfire)– Publish– Community– Marketplace– Kinds of Data– Personalization– Benefits

• The portal is a catalog of geospatial information containing thousands of metadata records (information about the data) and links to live maps, features, and catalog services, downloadable data sets, images, clearinghouses, map files, and more. The metadata records were submitted to the portal by government agencies, individuals, and companies, or by harvesting the data from geospatial clearinghouses.

15Sources: http://gos2.geodata.gov/gos/static/default/faq.htm and Help Center.

6. Geospatial OneStop

16Web Player

7. Spotfire Silver

17http://goto.spotfire.com/g/?KXLMSEWRM9

See Terms of Use.Limited to ThreeSpreadsheets of Less Than 10 MB.

7. Spotfire Silver

• Demo Files:– An Example to Get Started!– Map - Europe Departments– Map - Latin American States– Map - Middle Eastern States– Map - SE Asia Provinces– Map - US States and Counties (see next slide)– Map – World Counties and Cities (see slide 21)

• Private Analysis: Two Applications Allowed

18

7. Spotfire Silver

19PC Desktop

7. Spotfire Silver

20Web Player

7. Spotfire Silver

21http://semanticommunity.info/Sustainable_Society_Foundation_Index_2008

Note: This combines mapping and statistical analyses and visualizations!

7. Spotfire Silver• Example of Spotfire Community Help that I needed for the previous application where the country

names in the Sustainable Society Foundation Index did not exactly match the country names in the World County Boundary File database.

– How to combine two data sets from different sources via a matching table?• We have two tables with information about companies we would like to combine. Unfortunately the company

name column does not match 100%. We have to create a matching table to be able to combine both datasets. It is easy in Spotfire to merge these 3 tables into one. You can follow these simple steps and re-apply it to your own data.

• First load both data tables into Spotfire (e.g. TOP100-companies-Germany and DAX-companies)• Create a Matching Table out of the fields "Firmenname" and "Company" and load this into Spotfire as well.• Add a new table from Existing and duplicate the TOP100 table, call it TOP100-companies-Germany and DAX• Insert Column from External Data and chose the Matching Table, add the "Company" name in a full outer join• Again Insert Column from External Data and choose the DAX-companies table.• You have now added the information if the TOP100 company is a DAX company or not. Keep the final table

linked to its source tables. You can also link the source data tables to their data sources, but embed the matching table. When you do this, the resulting table will be refreshed whenever the Spotfire files is opened or refreshed/reloaded.

• Source: http://spotfire.tibco.com/community/forums/t/1384.aspx

• Spotfire File in Web Player: http://ondemand.spotfire.com/public/ViewAnalysis.aspx?file=/Users/TIBCEVL-52522/Public/DAX-and-TOP100

22

8. Geospatial Platform

23http://www.geoplatform.gov/index.html

Note: I put the Key Documents inMindTouch (seenext slide).

8. Geospatial Platform

24http://semanticommunity.info/Data.gov/GEOViewer/Geospatial_Platform_Key_Documents

8. Geospatial Platform

25Source: http://semanticommunity.info/Data.gov/GEOViewer/Geospatial_Platform_Key_Documents#Figure_1._Geospatial_Platform_Conceptual_Model

8. Geospatial Platform

26

Source: http://semanticommunity.info/Data.gov/GEOViewer/Geospatial_Platform_Key_Documents#Figure_C.1._Geospatial_Platform_conceptual_architecture_and_role_of_FEA

8. Geospatial Platform

27http://www.geoplatform.gov/gulfresponse/

Note: I put this in a table in Spotfire.

8. Geospatial Platform

28Web Player

Mashup of GeoPlatform for Gulf Response and EPA Monitoring Data

8. Geospatial Platform

29

Submitted Comment: Use tools like Spotfire that supportboth geospatial and statistical visualizations and Open Data where you can easily get the data into and back out.

http://geoplatform.ideascale.com/

9. Sample Applications

• 9.1 US EPA National Geospatial Program• 9.2 Geospatial One Stop• 9.3 Places and Spaces Atlas of Science• 9.4 National Geospatial-Intelligence Agency (NGA) -

Network Centric Operations Industry Consortium (NCOIC)

• 9.5 Build Your Own Data.gov GEO Viewer in the Cloud

30

9.1 US EPA National Geospatial Program

31http://semanticommunity.info/Data.gov/GEOViewer

The "map" looks okay, except for the Guam Brownfields site way to the east. The Longitude should be negative?

9.2 Geospatial One Stop

32PC Desktop (Web Player)

Statistics from Web Site)

9.3 Places and Spaces Atlas of Science

33http://semanticommunity.info/Atlas_of_Science

9.4 NGA-NCOIC

34Source: http://semanticommunity.info/@api/deki/files/1200/=NGA_to_NCOIC_Brief_Final_Slides_21Jun10dlb.pdf

9.4 NGA-NCOIC

• Tuesday, March 1, 2011:– 0800-1000 Focused Session - NGA/NCOIC Cloud Computing

Proposal.– Description: NCOIC has been asked to submit a formal proposal

to the National Geospatial-Intelligence Agency (NGA) to design, build and present a demonstration on how cloud computing technologies and/or techniques could be used to enhance interoperability between geospatial information systems. This session will review the draft proposal for NGA -- all are encouraged to attend and participate.

35Source: https://www.ncoic.org/home/

9.5 Build Your Own Data.gov GEO Viewer in the Cloud

36

PC Desktop (Web Player)

USGS Openview Map Viewer Map Layer Metadata as of Novmber 17, 2010.

9.5 Build Your Own Data.gov GEO Viewer in the Cloud

37

http://semanticommunity.info/Data.gov/GEOViewer

10. Some Next Steps• Work with the USGS Spatial Ontology (RDF/OWL):

– http://cegis.usgs.gov/ontology.html• SICoP Monthly Webinars – To Be Announced Soon.• January 27, 2011: ET SIG Meeting.• January 30, 2011: Atlas of Science Final Submission

due for March 25-26, 2011, VIVO Workshop.• March 1, 2011: Focused Session on NGA/NCOIC Cloud

Computing Proposal.• March 16-17, 2011: European Spotfire Conference.• April 12, 2011: 11th SOA for E-Gov Conference.• June 5-9, 2011: 2011 SemTech Conference.

38