Open Data Trentino - Seminar at Universidad Simon Bolivar - 15th October 2013

11/04/231 Lorenzino Vaccari, Juan Panehttp://dati.trentino.it

Open Government DataSeminar @USB*

*This presentation is taken from the “Open Government Data Tutorial” presented at CLEI2013

Lorenzino Vaccari1, Juan Pane2

1Autonomous Province of Trento, Trento, Italy lorenzino.vaccari@provincia.tn.it

2University of Trento, Trento, Italy – Universidad Nacional de Asuncion, Asuncion, Paraguay pane@disi.unitn.it

11/04/23 Lorenzino Vaccari, Juan Pane2

Goal of the Seminar• Introduce Open Government Data

• Intro, Issues (Part 1)

• If you need it, how can you organize it?• Real experience (Part 2)

• Methods for opening data• Applications (Part 3)• Semantic Issues (Part 4)

11/04/23 Lorenzino Vaccari, Juan Pane3 15/10/2013Juan Pane, Lorenzino Vaccari3http://www.point-fort.com/index.php?2012/01/25/805-why-how-what

http://www.point-fort.com/index.php?2012/01/25/805-why-how-what

“is data that can be freely used, reused and redistributed by anyone – subject only, at most, to the requirement to attribute and

sharealike.” *

*(Source: )

http://www.opendefinition.org

usereuse

“open” = redistributioncommercial reusederivative works

BUT, may require:- attribution- share alike

http://myfbcovers.com/uploads/covers/2012/06/09/16628a1094aa012f7c6e0025902480d2/watermarked_cover.jpg

J. Gray (OKF): http://www.slideshare.net/jwyg/open-government-data-what-why-how

The value is in its use

11/04/23 Lorenzino Vaccari, Juan Pane7 Maurizio Napolitano: http://www.youtube.com/watch?v=YlkjrVAW43Q

Is open data useful?

Maurizio Napolitano: http://www.youtube.com/watch?v=YlkjrVAW43Q

Open Data Benefits The Open data are the knowledge base to:

Improve the economic grow and the entrepreneurship based on the development of digital services reusing Public Sector Information

Answer to social needs through the publication of innovative services and applications

Aims at reducing the cost of the public administrative activities within Public – Private Partnerships (PPP)

Improve the transparency of the activities of the public institutions and the participation of the citizens to these activities

Principles

Tim Berners-Lee (5-Stars of Linked Open Data)Vs.Tim Davis (5-Stars of Open Data Engagement)

http://5stardata.info/

http://www.timdavies.org.uk/2012/01/21/5-stars-of-open-data-engagement/

5 Starts Linked Open DataTim Berners-Lee

http://5stardata.info

5-Stars of Open Data Engagement

* Be demand driven * * Provide context * * * Support conversation * * * * Build capacity & skills* * * * * Collaborate with the community

Tim Davis

http://www.timdavies.org.uk/2012/01/21/5-stars-of-open-data-engagement/

Create Communityhttp://msnbcmedia.msn.com/j/MSNBC/Components/Photo/_new/pb-121007-spain-tarragona-pyramid-nj-02.photoblog900.jpg

Open Government Data

State of the ArtWhat is happening around us?-Globally-Europe-Latin America

Open Data Charter - G8The principles are:Open Data by DefaultQuality and QuantityUseable by AllReleasing Data for Improved GovernanceReleasing Data for Innovation

http://opensource.com/government/13/7/open-data-charter-g8

https://www.gov.uk/government/publications/open-data-charter/g8-open-data-charter-and-technical-annex

11/04/23 Lorenzino Vaccari, Juan Pane17http://opensource.com/government/13/7/open-data-charter-g8

http://census.okfn.org/

OGD around the world

11/04/23 Lorenzino Vaccari, Juan Pane18http://opensource.com/government/13/7/open-data-charter-g8

http://census.okfn.org/country/

OGD in Europe

http://open-data.europa.eu/

OGD in Europescreenshots

http://epsiplatform.eu/content/european-psi-scoreboard

OGD in EuropeInsert table

http://epsiplatform.eu/content/european-psi-scoreboard http://epsiplatform.eu/content/psi-scoreboard-indicator-list

OGD in Italy

http://www.dati.gov.it/content/infografica

OGD in Latin America*

*In Venezuela some OD projects have been started by the USB

11/04/2324 Lorenzino Vaccari, Juan Pane

Questions?

OGD: Part 2 - Issues

11/04/2325 Lorenzino Vaccari, Juan Pane 08/10/2013Juan Pane, Lorenzino Vaccari25http://evian-thesource.com/kids-having-fun/http://evian-thesource.com/kids-having-fun/

Open Data. Oh ohh

08/10/2013Juan Pane, Lorenzino Vaccari26

LegalLegalOrganizationalOrganizational TechnicalTechnicalAdoptionAdoptionBarriersBarriers

ContextualContextual

http://www.wallpapermania.eu/wallpaper/trick-or-treat-cute-pumpkins-lanterns-halloween-wallpaper

11/04/23 Lorenzino Vaccari, Juan Pane27http://de.straba.us/wp-content/uploads/2012/08/barrieres_for_implementation_of_ogd.png

Organizational Barriers

Not readyLack of resources

ITHuman

Don’t want to be ready

http://montcomediation.org/images/MCMC_MyWayYourWay.jpg

Legal barriersOpen the Data

All the data that was produced using public money has to be made publicly available (with exceptions)

vs PrivacyYou cannot open data that could allow

correlation of private personal data

Or the complete lack of legislation!

Adoption barriersData is not contextualizedPeople are not informedOpening data is a complex task, opening

cleaned data is even more complex.Unclear licenses

Technical BarriersAccess to data:

OrganizationalTechnical, Downtimes, logins, Payment fees

Fragmentation, incomplete data, scattered

FormatCataloging, indexing, searchLack of explicit semantics, metadataData is not reliableConflicting standards, models,

ontologies

BarriersZuiderwijk et al 2010

Listed 118 socio-technical impediments for opening data in the literature.FindabilityUsabilityUnderstandablityQualityLinkingComparability and compatibilityMetadata….

http://www.ejeg.com/issue/download.html?idArticle=255

Context Barriers

Privileged access to dataOther companies what to avoid legislation

of privacy.Transparency is bad for fraudulent business

http://img.gawkerassets.com/img/182n8vzdlg1iojpg/original.jpg

11/04/23 Lorenzino Vaccari, Juan Pane34http://netdna.webdesignerdepot.com/uploads/photo_manipulation/manipulation-9.jpg

Preguntas?

Part 3 - Real Experience

11/04/23 Lorenzino Vaccari, Juan Pane36http://goo.gl/T2Xp80

The “Open Data in Trentino” project

• The “Open Data in Trentino” project is a 3 years initiative finalized to develop an open data infrastructure to enhance Service Innovation for Trentino following the PAT strategy for services innovation enabled by ICT. The project will be developed within a partnership between Trento RISE and the Autonomous Province of Trento (PAT) according to the innovation PAT model

• Goals• Improved quality of life for citizens• Open Data and local businesses• Transparency• Improved efficiency and productivity

Workplan - Steps

Nome (Acronimo) Descrizione

Tipo di Dato Estensione del file

Comma Separated Value (CSV) Formato testuale per l'interscambio testuale di tabelle, le cui righe corrispondono a linee e i cui valori delle singole colonne sono separati da una virgola (o punto e virgola)

Dato tabellare .csv

Geographic Markup Language (GML) Formato XML utile allo scambio di dati territoriali di tipo vettoriale

Dato geografico vettoriale

Keyhole Markup Language (KML) Formato basato su XML creato per gestire dati territoriali in tre dimensioni nei programmi Google Earth, Google Maps

Open Document Format (ODF) Formato per l'archiviazione e lo scambio di documenti di testo, fogli di calcolo, diagrammi e presentazioni

Dato tabellare .odc

Resource Description Framework (RDF) Basato su XML, e' lo strumento base proposto da World Wide Web Consortium (W3C) per la codifica, lo scambio e il riutilizzo di metadati strutturati e consente l'interoperabilità tra applicazioni che si scambiano informazioni sul Web

Dato strutturato .rdf

ESRI Shapefile (SHP) Lo Shapefile ESRI è un popolare formato vettoriale per sistemi informativi geografici. Il dato geografico viene distribuito normalmente attraverso tre o quattro files (se indicato il sistema di riferimento delle coordinate). Il formato è stato rilasciato da ESRI come formato (quasi) aperto

.shp, .shx, .dbf,

Extensible Markup Language (XML) E' un formato di markup, ovvero basato su un meccanismo che consente di definire e controllare il significato degli elementi contenuti in un documento o in un testo attraverso delle etichette (markup)

Dato strutturato .xml

…MeteoMeteo GeoDatiGeoDati StatisticaStatistica Comune

TrentoComuneTrento TrasportiTrasporti Etc…Etc……

Tecnological platform

Catalog

The Open Knowledge Foundation (OKF) is a non-profit organisation founded in 2004 and dedicated to promoting open data and open content in all their forms – including government data, publicly funded research and public domain cultural content.

(2004)

http://okfn.org

http://dati.trentino.it*

Analysis: http://dati.trentino.it/stats Admin: http://dati.trentino.it/admin Harvesting: http://dati.trentino.it/harvest

* Available for all the data providers of Trentino

Services

Also Trentino is going to launch a challenge to build software applications and creative products (multimedia, audiovisual products, posters, illustrations) based on the datasets published on the http://dati.trentino.it open data catalog.

#ODTChallenge will be the official hashtag for our first open data challenge in Trentino!

7 months until now68.555 visits 7.988 unique visits2.516 downloads

37,36% returning visitors

62,64% new visitors

NOW- ALL the departmnets demand to be involved- Plus other local actors

AgricultureCultureGeographical DataWelfareWeather ForecastSocial policiesStatisticsTransports…MUNICIPALITY OF TRENTO, and

INFORMATICA TRENTINA

567 datasetsprovided by 10 departments of PAT…

20 reporting errors15 asking for new data10 new suggestions6 OD Applications

100% ENTHUSIASTIC REACTIONS

Want to Know & Learn more?

11/04/23 Lorenzino Vaccari, Juan Pane48http://www.theodi.org/

11/04/23 Lorenzino Vaccari, Juan Pane49http://schoolofdata.org/

11/04/23 Lorenzino Vaccari, Juan Pane50http://opendatahandbook.org/pt_BR/

11/04/23 Lorenzino Vaccari, Juan Pane51 http://www.od4d.org/category/open-data/how-to/

11/04/23 Lorenzino Vaccari, Juan Pane52http://schoolofdata.org/online-resources/

Thanks to the project team !!!!• General Manager: Isabella Bressan

• Project coordinator: Lorenzino Vaccari• Organizational/Communication issues: Francesca Gleria,

Roberto Cibin • Data gatherer: Luca Paolazzi • Catalog: Maurizio Napolitano, Samuele Santi• Semantics: Juan Pane, David Leoni, Alberto Zanella• Legal issues: Eleonora Bassi, Stefano Leucci• Communities: Maurizio Napolitano, Francesca De Chiara• System integration: Marco Combetto, Lorenzo Dallapè• Statistical Linked Data: Pavel Shvaiko

Questions?

OGD: Part 4 - Applications

Apps4Italy

Best Application: http://parlamento17.openpolis.it/

Open Bilancio

Best Idea: http://opendata.comune.fi.it/open_bilancio/

DAL America Latina (2012): http://desarrollandoamerica.org/aplicaciones-2012/

DAL America Latina (2013): http://2013.desarrollandoamerica.org/appschallenge/

http://limaio.innovacion.pe/ http://www.limaio.com/demo

11/04/23 Lorenzino Vaccari, Juan Pane60http://www.mysociety.org/2007/more-travel-maps/morehousing

Johann MITTHEISZ (CIO der Stadt Wien)

http://www.slideshare.net/BrigitteLutz/keynote-mittheisz-cio-stadt-wien/16

Total hours to develop 38 applications:around 2.600

City of Wien saved around 208.000 Euro

The Open Data Ecosystem(and the OpenStreetMap case)

OpenStreetMap

OpenStreetMap project creates and provides geographical data, such as road maps, freely available to anyone. Behind the establishment and growth of the project have been restrictions on use or availability of map information across much of the world and the advent of inexpensive portable satellite navigation devices.

OpenStreetMap is a free map of theworld, created by someone like you

11/04/23 Lorenzino Vaccari, Juan Pane65http://tools.geofabrik.de/mc/?mt0=mapnik&mt1=googlemap&lon=11.12042&lat=46.07224&zoom=18

11/04/23 Lorenzino Vaccari, Juan Pane66http://haiti.ushahidi.com

Watercolor maps

http://content.stamen.com/files/cartography/index_watercolor.html#18.00/46.07204/11.12097

From maps to blankets…

http://softcities.net

Sharing Data Globally(the eHabitat example)

21th Century Challenges

Source: http://www.slideshare.net/angeled/geoss © GEO secretariat

The Group of Earth Observation

Source: http://www.slideshare.net/angeled/geoss © GEO secretariat84 GEO members and 61 Participating organizations

GEOSS Data Sharing Principles

• Full and Open Exchange of Data, recognizing Relevant International Instruments and National Policies

• Data and Products at Minimum Time delay and Minimum Cost

• Free of Charge or minimal Cost for Research and Education

http://www.geoportal.org/web/guest/geo_home

“Venezuela is considered a state with extremely high biodiversity, with habitats ranging from the Andes mountains in the west to the Amazon Basin rainforest in the south, via extensive llanos plains and Caribbean coast in the center and the Orinoco River Delta in the east."

Source: Wikipedia

GEOSS for biodiversity

http://www.eurogeoss-broker.eu/

The eHabitat Model

http://ehabitat-wps.jrc.ec.europa.eu/ehabitat/

Questions?

OGD: Part 5 - Semantics

Available

Structured

Open formats

Redefenceable

Linked

Linked Open Data

The best data is an open data

All data must be perfect

Lack of explicit semanticsThe real meaning of the data was kept in the developers mind when creating the data

78http://goo.gl/npEHKr

Lack of explicit semanticsCan lead to things like:

Semantic heterogeneityDifference in the meaning of local data

Issues when Opening Trentino Data

Each department has authority on only some part of the data.

Dataset originally created for internal use only.Dataset created for a specific need.Dataset created with custom format:

For structure (some exceptions)For data

Lack of reuse -> duplication.Lack of programmers.We cannot TELL them what/how to do (always).Data changes

Available

Structured

Open formats

Redefenceable

Linked

Entity CentricSemantic Layer

Data Catalog

Entity centric: Added valueAggregated dataAccurate data, manually curatedUnique identifiers, distributed perspectives

Re-think identifiersSemantified values

name Juan Pane

nationality italian

lives in Trento

affiliation Univ. Trento

name Ignacio P. F.

born in Paraguay

date of birth 1980

affiliation PF-UNA

EntitiesReal world: is something that has a distinct,

separate existence, although it need not be a material (physical) existence. Has a set of properties, which evolve over time. Example:

Mental: personal (local) model created and maintained by a person that references and describes a real world entity.

Digital: capture the semantics of real world entities, provided by people.

Entity Centric Semantic Layer:• Address the integration problems due to

semantic heterogeneity:• Different formats• Different identifiers• Implicit semantics• Homonyms, synonyms, aliases• Partial knowledge• Knowledge evolution

http://www.webfoundation.org/2011/11/5-star-open-data-initiatives/

Entity-based Integration• Focus on entities as first class citizens

• Entities are objects which are so important in our everyday life to be referred with a name

• Each entity has its own metadata (e.g. name, latitude, longitude, …)• Each entity is in relation with many other entities (e.g. Einstein was

born in Ulm, his affiliation was Charles University, Ulm is a city in Germany)

• There are relatively “few” commonsense entity types (person, …, event)

• There are many domain specific entities (bus stops, cycling paths, ..)• All components have explicit semantics: schema, entities, attributes,

values

Importing pipeline, Macro Steps1. Domain analysis

Study the needed entity types, adapt the knowledge base accordingly. First time bootstrapping

2. Import entities Semi-automatic tool.

Domain experts are expensive. Human attention is a scarce resource. Incremental enrichment and aggregation of

entities.

Open Data PeculiaritiesAll data comes from a CKAN repository

(DCAT).Process one data file at a time.Each data file can be represented as a

table.Each row in the table represents a (partial)

entity.The format of the values might not be

enforced in the data files.Not all data is relevant.

Importing tool process

1. Source SelectionImport one data file at a time

2. Schema MatchingSelect a target type of entity -> correspondences between the input columns and the output attributes

nome provincia descrizione funivie lat long

Andalo (1047) Provincia di Trento

Sorge su un'ampia sella prativa al centro...

3 654463 712857

Canazei (1450) Trento Prov. Situato all'estremità settentrionale della...

2 511504 147444

3. Data ValidationApplies format and structure validation and possible automatic transformations needed to have the input data in the expected format.

4. Semantic Enrichment (1/2)Entity disambiguation: Transform text references into links to existing entities.

4. Semantic Enrichment (2/2)Natural Language Processing: Extract concepts and entity references from free-text.

5. ReconciliationRun Identity Management Algorithms to identify each row as a new or existing entity.

Result•No Match•Match•Multiple Matches

Action:•Use ID•New ID•Ignore Row

6. ExportingAt this point:We know what to export.All values for target attributes conform to the expected format.All text has been semantified (NLP).All textual references to entities are converted to linksEach row has an identifier

i i+1v0

7. PublishingPut back the semantified entities into CKAN so that the entities can be Open Data and can be found in the same catalog as the original data.Developers and find the data files of the cleaned, aggregated entitiesBut can also interact with the entities via the Entitypedia APIs

8. VisualizationSearch and Navigation

Semantic Layer: ServicesTool for aiding the “semantification” of the datasets in the catalog based on:

• Schema matching services• Identity Management services

• Entity Matching services• Global Unique Identifier services

• Semantic search and indexing services• Natural Language Processing• Entity store

Our Goal

11/04/23 Lorenzino Vaccari, Juan Pane100 http://www.youtube.com/watch?v=Bq_ZWl1ZXA0

BEYOND

Gracias!

Grazie!

Mercy!

Gràcies!Gratias!

Thanks!

Danke!

Dank u!

Kiitos!

ευχαριστώ

We thank in particular CLEI 2013, Autonomous Province of Trento, TrentoRise association, Universidad Nacional de Asuncion, Universidad Simon Bolivar and University of Trento

Lorenzino Vaccari1, Juan Pane2

1Autonomous Province of Trento, Trento, Italy lorenzino.vaccari@provincia.tn.it

2University of Trento, Trento, Italy – Universidad Nacional de Asuncion, Asuncion, Paraguay

pane@disi.unitn.it

Open Data Trentino - Seminar at Universidad Simon Bolivar - 15th October 2013

Technology

CIUDAD BOLIVAR, MUNICIPIO HERES ESTADO BOLIVAR-VENEZUELA … · CIUDAD BOLIVAR, MUNICIPIO HERES ESTADO BOLIVAR-VENEZUELA INUNDACIONES EXTREMAS: Hacia el entendimiento de los posibles

Transalp Trentino

The Bolivar bulletin. (Bolivar, Tennessee) 1890-05-16 [p ]

a Single Support Desk for Investment ... - Trentino Sviluppo · Trentino Sviluppo, a Single Support Desk for Investment Trentino Sviluppo is a one-stop-shop in Trentino for business

Bolivar Conservacionista

Paolo Bertolini – Trentino Network Srl Trentino Network: strumento della Pubblica Amministrazione del territorio Trentino 14 maggio 2009 - 11.45 FORUM

Trentino Charme

Trentino Digitale S.p.A. - PROVINCIA AUTONOMA DI TRENTO · trentino (SINET): Trentino Digitale S,p.A., derivante dalla incorporazione con effetto dal 1 dicembre 2018 di Trentino Network

LOMBARDIA • TRENTINO

Simon Bolivar - jvlone.comjvlone.com/BolivarUSIA1983.pdf · Simon Bolivar Author: John V. Lombardi Subject: Simon Bolivar; Bolivar Bicentennial Keywords: Simon Bolivar; Bolivar Bicentennial;

Bolivar exportaciones

CEii Trentino

Alcune realtà regionali: TRENTINO ALTOADIGE – Provincia ... · trentino alto adige trentino sÜdtirol supporti legislativi alla pianificazione del rischio idrogeologico in alto

Bolivar simon

Archivio Trentino

Trentino Sviluppo

Book Trentino

Open Data in Trentino - Corso Trentino School of Management (TSM)

Mappa Trentino Orientale Strada Vino Trentino

Trentino speciál