OGI D3.4 - Report on OpenGovIntelligence ICT tools ... - Rep… · D3.4 Report on OpenGovIntelligence ICT tools – 2nd release Page 3 of 25 Abstract: This deliverable provides details

ThisprojecthasbeenfundedwiththesupportoftheH2020ProgrammeoftheEuropeanUnionãCopyrightbytheOpenGovIntelligenceConsortium

.

OpenGovIntelligence

FosteringInnovationandCreativityinEuropethroughPublicAdministrationModernizationtowardsSupplyingandExploiting

LinkedOpenStatisticalData

Deliverable3.4

ReportonOpenGovIntelligenceICTtools–secondrelease

Leadingpartner: NUIG

Participatingpartners: CERTH,TUT,ProXML,SWIRRL,NUIG

Version-Status: 1.0

Disseminationlevel: PU

D3.4ReportonOpenGovIntelligenceICTtools–2ndrelease

Page2of25

DeliverablefactsheetProjectNumber: 693849

ProjectAcronym: OpenGovIntelligence

ProjectTitle:Fostering Innovation and Creativity in Europe through PublicAdministrationModernizationtowardsSupplyingandExploitingLinkedOpenStatisticalData

Deliverabletitle: ReportonOpenGovIntelligenceICTtools–secondrelease

Deliverablenumber: D3.4

Officialsubmissiondate: 31October2017

Actualsubmissiondate: 31October2017

Editor(s):ArkadiuszStasiewicz(NUIG)

MohamedAdel(NUIG)

Author(s):

DimitriosZeginis(CERTH)

AretiKaramanou(CERTH)

EvangelosKalampokis(CERTH)

KonstantinosTarabanis(CERTH)

ArkadiuszStasiewicz(NUIG)

MohamedAdel(NUIG)

PaulHermans(ProXML)

BillRoberts(SWIRRL)

RickMoynihan(SWIRRL)

Reviewer(s) BillRoberts(SWIRRL)


Page3of25

Abstract: Thisdeliverableprovidesdetailsontheprototypesofsoftwarecomponents delivered as a result of the second developmentstageofOpenGovIntelligenceproject.


Page4of25

EffortofParticipatingPartners Name ShortName Role PersonMonths

1. CentreforResearch&Technology-Hellas CERTH Participant 1

2. DelftUniversityofTechnology TUDelft None 0

3. NationalUniversityofIreland,Galway NUIG Leader 1.5

4. TallinnUniversityofTechnology TUT Participant 0.2

5. ProXMLbvba ProXML Participant 0.1

6. SwirrlITLimited SWIRRL Participant 0.5

7. Traffordcouncil TRAF None 0

8. FlemishGovernment VLO None 0

9. MinistryofInteriorandAdministrativeReconstruction MAREG None 0

10. MinistryofEconomicAffairsandCommunication MKM None 0

11. MarineInstitute MI None 0

12. PublicInstitutionEnterpriseLithuania EL None 0


Page5of25

RevisionHistoryVersion Date Revisedby Reason

0.1 18-Sep-2017 A.Stasiewicz(NUIG) FirstDraft

0.2 11-Oct-2017 A.Stasiewicz(NUIG) Toolslistupdate

0.3 19-Oct-2017 A.Stasiewicz(NUIG) Generalupdates

0.4 26-Oct-2017 A.Stasiewicz(NUIG) Generalupdates

0.5 31-Oct-2017 A.Stasiewicz(NUIG) Addressinginternalreviewcomments

1.0 31-Oct-2017 CERTH SubmissiontoEC

Statementoforiginality:This deliverable contains original unpublishedwork exceptwhere clearly indicated otherwise.Acknowledgement of previously publishedmaterial andof thework of others has beenmadethroughappropriatecitation,quotationorboth.


Page6of25

TableofContentsDELIVERABLEFACTSHEET................................................................................................................2

EFFORTOFPARTICIPATINGPARTNERS...........................................................................................4

REVISIONHISTORY.........................................................................................................................5

TABLEOFCONTENTS......................................................................................................................6

LISTOFFIGURES.............................................................................................................................7

LISTOFTABLES...............................................................................................................................8

LISTOFABBREVIATIONS.................................................................................................................9

EXECUTIVESUMMARY..................................................................................................................10

1 INTRODUCTION....................................................................................................................11

1.1 SCOPE...................................................................................................................................................111.2 AUDIENCE..............................................................................................................................................111.3 STRUCTURE............................................................................................................................................111.4 ADDRESSINGREVIEWER’SCOMMENTS..........................................................................................................11

2 OPENGOVINTELLIGENCEICTTOOLS–OVERVIEW..................................................................13

2.1 TOOLSDEVELOPEDDURINGTHESECONDDEVELOPMENTSTAGE.........................................................................13

3 OPENGOVINTELLIGENCEICTTOOLS-SECONDRELEASE.........................................................15

3.1 ASSISTEDCUBESCHEMACREATOR..............................................................................................................153.2 LOSDMACHINELEARNINGCOMPONENT.....................................................................................................173.3 QBMULTI-DIMENSIONALCHARTING..........................................................................................................183.4 RDFDATACUBEGEODATADASHBOARD....................................................................................................203.5 SPARQLCONNECTORFOREXPLORATORY....................................................................................................213.6 JSONAPIFORDATACUBESPECIFICATION–UPDATE.....................................................................................223.7 JSONAPIFORDATACUBEIMPLEMENTATION-UPDATE.................................................................................223.8 TABLE2QBANDGRAFTER-UPDATE............................................................................................................223.9 DATACUBEAGGREGATOR-UPDATE............................................................................................................23

4 CONCLUSION........................................................................................................................24

REFERENCES.................................................................................................................................25


Page7of25

ListofFiguresFIGURE 1. LOSD TOOL ECOSYSTEM ...................................................................................................................... 14FIGURE 2 OGI LOSD CREATION AND USAGE FLOWCHART – RDF DATA CUBE SCHEMA MAPPING STAGE .......... 17FIGURE 3 QB MULTI-DIMENSIONAL CHARTING DASHBOARD ................................................................................ 19FIGURE 4 LITHUANIAN DATA CUBES GEO BASED VISUALIZATION SCREEN SAMPLES – A...................................... 20FIGURE 5 LITHUANIAN DATA CUBES GEO BASED VISUALIZATION SCREEN SAMPLES – B ...................................... 21


Page8of25

ListofTablesTABLE 1 RECOMMENDATIONS CONCERNING FUTURE WORK .................................................................................. 12TABLE 2. OGI ICT TOOLS TECHNICAL STARTING POINT ........................................................................................ 13


Page9of25

ListofAbbreviationsThefollowingtablepresentstheacronymsusedinthedeliverableinalphabeticalorder.

Abbreviation Description

API ApplicationProgrammingInterface

CMS ContentManagementSystem

ICT InformationandCommunicationTechnologies

LOSD LinkedOpenStatisticalData

OLAP OnLineAnalyticalProcessing

RDF ResourceDescriptionFramework

UI UserInterface

URI UniformResourceIdentifier

WP WorkPackage


Page10of25

ExecutiveSummaryThisdocumentisthedeliverable“D3.4–ReportonOpenGovIntelligenceICTtools–secondrelease”(referred to as D3.4). It provides detailed information about the result of the second stageoftheOpenGovIntelligencedevelopmentinWP3.

ThegoalofWP3(“ICTtoolsdevelopment”)istodeveloptheOpenGovIntelligenceICTtoolsasasuiteof open source and commercial tools. The final version of the developed ICT tools will supporttheOpenGovIntelligenceframeworkcreatedinWP2(“Frameworkcreation”)andwillenable:

a) thecreationofLinkedOpenStatisticalData(LOSD)fromvarioussources,b) theexpansionofLOSDwithdatasetsfromexistingsources,c) theexploitationofLOSDfortheco-productionofpublicservices.

With regards to thecreationofLOSD,OpenGovIntelligencedesignedanddeveloped ICT tools thatenable the transformation of public sector data to standardmachine readable forms (specificallytoRDFdatacubes)andthevalidationofthegeneratedRDF.WithregardstotheexploitationofLOSD,OpenGovIntelligence designed and developed ICT tools that enable visualisation and analysisofstatisticaldata.

While the first release of theOpenGovIntelligence ICT toolswas focused on the tools supportingthecreationandexploitationofLOSD,thenextreleasefocusesonexploitationofLOSDbyprovidingnew tools and improvements (functional and performance) to tools arleardy developed andevaluated.

Thetoolspresentedinthisdeliverablewillbeevaluatedduringthepilotimplementationstage.Basedontheoutcomes,theexistingtoolswillbeimproved.MoreovernewtoolswillbedevelopedinordertocoverremainingfeaturesoftheOpenGovIntelligenceFramework.

The second release of the OpenGovIntelligence ICT tools was guided by challenges and needsidentified in WP1 (D1.1 OpenGovIntelligence challenges and needs), the OpenGovIntelligenceframework (D2.2 OpenGovIntelligence framework) as well as the pilots operation and initialOpenGovIntelligenceevaluation(D4.2Evaluationresults-Firstround).

Results presented in this document are the outputs of Task 3.2 OpenGovIntelligence ICT tools –second release. This deliverable will be updated together with the ongoing dvelopment of theOpenGovIntelligence tools as D3.6 Report on OpenGovIntelligence ICT tools. The final releaseisscheduledforOctober2018.


Page11of25

1 IntroductionThissectionintroducesthebackgroundoftheworkcarriedoutinWP3“ICTtoolsdevelopment”.Sub-section1.1presentsthescopeandtheobjectivesofthecurrentdocument,sub-section1.2describesthe intendedaudience for thisdocument, sub-section1.3outlines the structureof thedocument,whilesub-section1.4addressesthecommentsreceivedfromthe1stProjectReview.

1.1 ScopeThisreportdocumentsthe ICTtoolsdevelopedduringthesecondphaseofWP3.Toguidereadersin understanding the context of these tools, Section 2 presents the overviewof the OpenGovIntelligence tools. The Second release provided tools that enables creation andexploitationofLOSD.FinalreleaseofthetoolswillaimtocoverexpansionofexistingdatasetsandexploitationofLOSDforco-productionofpublicservices.

1.2 AudienceTheintendedaudienceforthisdocumentistheOpenGovIntelligenceconsortium,inparticularpartnerorganisationsresponsibleforthedevelopmentofpilottrials,theEuropeanCommission(EC)andthosewhoareinterestedinchallengesandneedsforopening-upandexploitingLOSDfortheco-productionofinnovativedata-drivenservicesongovernments.

1.3 StructureThestructureofthedocumentisasfollows:

• Section2presentstheOpenGovIntelligenceICTtoolsoverview• Section3providesadetaileddescriptionoftheindividualOpenGovIntelligenceICTtools

developedduringtheseconddevelopmentstageofOpenGovIntelligenceproject;• Finally,Section4concludesthereportandoutlinesthefuturedevelopmentplan.

1.4 Addressingreviewer’scommentsThis sub-section address the general comments from the 1st Consolidated Expert Review Report.The Review Report did not include specific issues concerning D3.2. Therefore, in Table 1Recommendationsconcerningfutureworkwepresentrecommendationsoverthefuturework.


Page12of25

Table1Recommendationsconcerningfuturework

No. Comment Addressedin

1

It is recommended that future iterations ofD3.2 should make a clear statement aboutthetechnicalstartingpointoftheprojectandwhat has transferred from the DaPaaS andOpenCubeprojects.

DetailsareprovidedinSection2.

2“FigurethatgivesOGIarchitecture isshownin this document, where the same figure isusedinthedeliverableD.2.1(Figure4).”

While D2.1 provides overview of thearchitecture,D3.2explainindetaileach

oftheelements.


Page13of25

2 OpenGovIntelligenceICTtools–overviewThissectionaimstoprovidebackgroundinformationrelatedtoLOSDandOpenGovIntelligencetools.

Ingeneral,thedevelopedtoolscoverpartsoftheOpenGovIntelligenceArchitectureandarepartofalooselyconnectedecosystemoftools.TheOpenGovIntelligenceArchitecture(presentedinD3.2)forLOSDanddata-drivenpublicservicesenablesstakeholderstocollaboratetowardstheproductionofinnovative data-driven public services by exploiting LinkedOpen Data technologies and statisticaldatasets.TheinitialversionofthearchitecturewasdocumentedindetailinD3.2andwastestedandrefinedduringthefirstroundofpilotevaluations.Thearchitectureisorganisedasfollows:

• Thearchitectureisdividedintofivelayers:(i)DataProvision,(ii)DataPlatform,(iii)ProcessLayer,(iv)ServiceDesign,and(v)ServiceProvision.

• Keymanagementresponsibilitiesaresharedacrossalllayers.• Eachlayerhasasetofcomponentsthatperformstasksspecifictothatlayer.

Thearchitectureguidesthepilotimplementations,aswellastheotherfutureimplementationsoftheOGIsoftwareinotherprojectstogetherwithOpenGovIntelligenceframework(documentedinD2.2).

2.1 ToolsdevelopedduringtheseconddevelopmentstageIn the first implementation stage, the developed tools were mainly supporting data conversionandexploitationofthedata.ThesecondreleaseofOpenGovIntelligenceincludesadditionaltoolsandsupportsalargerpartoftheOpenGovIntelligenceframework.

While the development of majority of the tools were initialted during the OpenGovIntelligenceproject,someofthemwereusingoutputsoftheDaPaaS1andOpenCube2projectsasthetechnicalstarting point, and subsequently improved and extended within OpenGovIntelligence. Details arepresented in Table 2. OGI ICT Tools technical starting point and visualised in Figure 1. LOSD ToolEcosystem.Also,Table2showswhichtoolsarenewcomponents,startedinyear2,andwhichtoolswerefurtherdevelopedinyear2buildingonworkstartedearlier.

Table2.OGIICTToolstechnicalstartingpoint

No. Name StartedinYear2 Foundation

1 JSONAPIForDataCubeSpecification No OpenGovIntelligence

2 JSONAPIForDataCubeImplementation No OpenGovIntelligence

3 Table2QBAndGrafter No DaPaaS

1https://project.dapaas.eu 2http://opencube-project.eu


Page14of25

4 AssistedCubeSchemaCreator Yes OpenGovIntelligence

5 DataCubeBuilder No OpenGovIntelligence

6 DataCubeExplorer No OpenCube

7 DataCubeAggregator No OpenCube

8 LOSDMachineLearningComponent Yes OpenGovIntelligence

9 OLAPBrowser No OpenCube

10 QBMulti-DimensionalCharting Yes OpenGovIntelligence

11 RDFDataCubeGeoDataDashboard Yes OpenGovIntelligence

12 SPARQLConnectorforExploratory Yes OpenGovIntelligence

Section3documentsthetools,whichweredeliveredduringtheseconddevelopmentstageaswellaslinks by which they can be accessed. In general, tools developed during the project lifecycle canbeaccessedattheGitHubrepositoryavailableat:http://github.com/OpenGovIntelligence

Figure1.LOSDToolEcosystem

Note: The OpenGovIntelligence project will not develop software tools for every componentof the architecture or a single integrated system that matches this architecture as a whole.ImplementationsoftheOpenGovIntelligenceapproachwillcombineaselectionoftoolsandprocessesrelevanttotheusecasestheyaddress,followingtheprinciplessetoutinthearchitecture.Moreover,somepartsofthearchitecturemightendupbeinghumanprocessesoruseappropriatepre-existingtoolsfromoutsideoftheOpenGovIntelligenceproject.Existingtoolsavailableforthepurposewillre-usedwhereeverpossibleandnewtoolswillbedevelopedwhenrequired.Wehavealreadyidentifiedanumberoftoolsthatcanbere-usedforexampleincollaborationspace.ThislistoftoolsisincludedindeliverableD2.1–OpenGovIntelligenceframework–firstrelease.


Page15of25

3 OpenGovIntelligenceICTtools-secondreleaseThissectionprovidesadetaileddescriptionoftheindividualOpenGovIntelligenceICTtoolsdevelopedduringtheseconddevelopmentstageof theOpenGovIntelligenceprojectanddetailsof themajorupdates to the tools developed it the first stage of the OpenGovIntelligence project. Previouslydevelopedtoolsdescriptionsareavailableindeliverable“D3.2–ReportonOpenGovIntelligenceICTtools–firstrelease”.ThetoolsdevelopedduringtheprojectlifecyclecanbeaccessedattheGitHubrepositoryavailableat:http://github.com/OpenGovIntelligence.

3.1 AssistedCubeSchemaCreatorLOSD publishing process always starts with the process of mapping of the desired data set. Thisprocess is required to conformwith RDF Data Cube Vocabulary defined concepts and propertiesinordertoenableinteroperability.

In the first stages of LOSD tools experimentation and development schemamapping processwasaccomplishedbyanalysisof theavailabledatasetsandmanualprocessingof themapping.Resultswereusedase.g.forinputsforRDFDataCubeBuildertool.Anychangesofthesourcedatarequiredrepetition of the mapping process. For sustainability reasons, we could not depend on thismethodologyforourLOSDpublishingprocess.

TheanalysisoftheotherLOSDcreationmethodologies345678leadtodevelopmentofAssistedCubeSchemaCreator.Toensuresustainability,thetoolwasdesignedtosupportvarietyofdatasetsanddifferentusecases.Moreover,toolhadtobesimpletousebydifferentusertypes.

3.1.1 FunctionalityDescriptionThispublishingpipelineiscreatedtoeasethemappingprocessofgovernmentstatisticaldatasetsintoRDFaccordingtotheRDFDataCubevocabulary.Thistoolintegratesspreadsheets,OpenRefine9,RDFRefine10andRDFDataCube11toproduceagenericLOSDmappingtoolthatfitsanypilotorusecase.

3.1.2 ImplementationDescription

Thefollowingstepsareincludedinthepublishingpipeline:

3https://github.com/GMDSP-Linked-Data/GenericStatsCube4http://www-etis.ensea.fr/WOD2013/wp-content/uploads/2013/06/Publishing-Census-as-Linked-Open-Data..pdf5http://wifo5-03.informatik.uni-mannheim.de/bizer/r2r/6https://www.w3.org/2001/sw/rdb2rdf/wiki/Implementations7http://wifo5-03.informatik.uni-mannheim.de/latc/toollibrary/screencast.html8https://www.w3.org/TR/r2rml/#definitions 9http://openrefine.org10http://refine.deri.ie/publications11https://www.w3.org/TR/vocab-data-cube


Page16of25

A. First step in themappingprocess is implementedby collectingdata set toRDFDataCubemappingdetailsfromtheuser.Tocollectinformationaboutthemappingdetailsfromusersaspreadsheettemplatewithbuiltinguidelineswasprepared.Thisspreadsheettemplateeasetheinformationcollectionprocessandmakeitsimplerforalltypesofusers.Usersareableto:

a. Startdefiningexistingdatasetheadersandtheirnewnamesifwanted.b. Defining the RDFData Cube types of those headers i.e. dimensions,measures, or

attributes.c. LinkingthoseheaderswithexitinglinkeddataURIthatmatchestheheaderconcept

andproperties.d. Defininglabels,comments,relatedconceptsanddatatypesforthoseelements.e. DefiningtheRDFdataCube’sDataStructureDefinition(DSD)elementbylinkingthe

dimensions,measures,andattributeswithDSD.f. Finally,userdefinetheRDFDataCube’sDataSet (DS)elementthat includeLinked

Dataannotationabilitiesi.e.datasetpublisherURIdefinitions,relatedsubject(s)URIdefinitions,relatedlocation(s)andorganization(s)definitions.Moreover,userdefinesthelabel,descriptionandcommentsfortheDS.

B. ThesecondstepinthemappingprocessiscarriedoutusingOpenRefinewiththeRDFRefineextension.TheRDFDataCubeVocabularystructurewaspredefinedinJSONformatthatcanbeconsumedbyOpenRefinewithassistancefromtheRDFRefineextension.ThisJSONRDFData CubeVocabulary structure is coupledwith the spreadsheet template. The process isdesignedtobeeasytouseforalltypesofusers.ItsimplifiesandminimisesthedatasettoRDF Data Cube mapping and conversion steps. As demonstrated in Figure 2 OGI LOSDCreationandUsageFlowchart–RDFDataCubeSchemaMappingStage, theuserneedstoload,applyandextracttheRDFschemabyfollowingthefollowingsteps:

a. LoadingmappingspreadsheettoOpenRefine+RDFrefineextension:theuserstartsstep2oftheschemacreationprocessbyloadingthespreadsheetcontainingthedatasettoRDFDataCubeVocabularymappingdetailsintotherunningOpenRefineserverasanOpenRefineproject.

b. ApplyingthepredefinedRDFDataCubeSchematotheinputspreadsheet:theusersimply loads the provided JSON file containing the RDF Data Cube VocabularyStructureandcouplesthiswiththespreadsheettemplate.Thisstepisdoneusingthestandard'applyoperations'proceduresavailableonOpenRefine.

c. ExtractingtheschemamappingofthedatasetinRDF/XMLorTTLformats:finally,theuser uses the RDF refine extension to extract the RDF Data Cube schema file inRDF/XMLorTTLformats.ThisfileisproducedbyapplyingtheJSONfileinstructiontothespreadsheet.


Page17of25

Figure2OGILOSDCreationandUsageFlowchart–RDFDataCubeSchemaMappingStage

3.1.3 AvailabilitySourcecodeisavailableatGitHub:https://github.com/OpenGovIntelligence/qb-assisted-schema-creator

3.1.4 LicenseThesoftwareisavailableasopensourceundertheMITLicense.

3.1.5 PilotsinvolvedTheIrishMarineInstitute,andEnterpriseLithuaniaaretheinitialusersofthetool.ThistoolisgenericandcanbeusedacrosstheOpenGovIntelligenceConsortium.

3.1.6 Futuredevelopmentplan

Theplansfornextyeardevelopmentinclude:

1. Updatestothespreadsheetfillingguidelines2. Evaluationandfeedbackcollection3. Updatingthetoolwithuserfeedback,andtooltestingresults.

3.2 LOSDMachineLearningComponentMachinelearningenablesthecreationofpredictivemodelsbasedontheanalysisofhighvolumesofdata. Thewealth of the statistical data that is freely available online can contribute towards thisdirection.Themulti-dimensionalnatureofstatisticaldataenablestheextractionofnumerousfeaturesfroma singledatasetbasedon the valueof themeasured variable for thedifferent valuesof thedimensionsorthedifferentaggregatedfunctions(e.g.min,max,averageetc.)thatcanbeappliedtothemeasure.


Page18of25

3.2.1 FunctionalityDescriptionTheMachineLearningComponentenablestheautomaticextractionofnumerousfeaturesfromLOSDbasedontheneedsoftheusersandthepredictivescenariothatisimplemented.ItalsoenablestheperformanceofdimensionreductionbasedonrelevantalgorithmssuchasForwardSubset,BackwardSubset,andLassoinauser-friendlyapproach.Inthisway,userswillbeabletoincludebigvolumesofLOSDinmachinelearningscenariosandfilteronlythepredictorsthatarerelevanttothescenarioathand.Thesepredictorscanbelaterusedinthecreationofpredictivemodels.


TheimplementationoftheMachineLearningComponentisbasedontheJSON-QBAPIandRserver.

3.2.3 AvailabilitySourcecodeisavailableatGitHub:https://github.com/OpenGovIntelligence/qb-machine-learning-component

3.2.4 LicenseThesoftwareisavailableasopensourceundertheApacheLicense(v2.0).

3.2.5 PilotsinvolvedAtthisstage,thetoolshavenotbeenactivelyusedinthepilotusecases,thusthetoolisgenericandcanbeusedacrosstheOpenGovIntelligenceConsortium.



1. Collectingusersfeedbackonthetool.2. Updatingthetoolbasedontheuserfeedback,andtooltestingresults.

3.3 QBMulti-DimensionalChartingQB Multi-Dimensional Charting tool is a multi-dimensional charting dashboard which is usinga JavaScript charting library as a base. It supports cross filtering and allowing highly efficientexplorationon largemulti-dimensional. It leveragesd3enginetorendercharts inCSSfriendlySVGformat.Chartsrenderedusingdc.jsarenaturallydatadrivenandreactivethereforeprovidinginstantfeedbackonuser'sinteraction.

ThemainobjectiveofthistoolistoprovideaneasyyetpowerfulJavaScriptdashboardwhichcanbeutilizedtoperformdatavisualizationforcubedataandanalysisinbrowseraswellasonmobiledevice.


Page19of25

3.3.1 FunctionalityDescriptionThistool isprovidinganeasytousemulti-dimensionalchartingdashboardwhichcanbeutilizedtoperformdatavisualization,datafiltering,explorationandanalysisforcubedatainbrowseraswellasonmobiledevice.

Figure3QBMulti-DimensionalChartingdashboard


QBMulti-DimensionalChartingdashboardiswritteninJavaScriptontopofanotherJavaScriptlibrarieslikedc.js,Leaflet.jsandcrossfilter.js;dc.jsisaJavaScriptlibrarythatisusedtocreateinteractivechartsandacorrespondingtable.Moreover,itsupportsaninteractivemap.

3.3.3 AvailabilitySourcecodeisavailableatGitHub:https://github.com/OpenGovIntelligence/qb-multi-dimensional-charting

3.3.4 LicenseThesoftwareisavailableasOpenSourceundertheApache2.0License.

3.3.5 PilotsinvolvedTheIrishMarineInstitute,andEnterpriseLithuaniaaretheinitialusersofthetool.


Theplansfornextyeardevelopmentincludes

1. Furtherintegrationofthetools


Page20of25

2. ReleasingthetoolsasaWordPressplugin

3.4 RDFDataCubeGeoDataDashboardTheRDFDataCubeGeoDataDashboardallowsvisualisationanddiscoveryofgeoannotateddata.RDFDataCubeGeoDataDashboarduses the JSONQBAPI toaccess thegeoannotatedRDFDatastoredinaRDFdatabase.Theretrieveddataarevisualizedasmapbaseddashboardtoenableeaseofinformationunderstandingtoalltypesofusers.

3.4.1 FunctionalityDescriptionThisdashboardisdesignedandimplementedtovisualisegeoannotateddataandsupportpilotspecificusecases.


This toolwaswritten in JavaScript using several libraries and tools as follow: Leaflet.js, GeoJSON,OpenStreetMap;Leafletjsisalightweightopen-sourceJavaScriptlibraryforbuildingmobile-friendlyinteractivemapsthankstoitsstrongmappingfeatures;GeoJSONisaformatforencodingavarietyofgeographicdatastructures,itsupportsthedifferentgeometrytypeslikePoint,Polygon,MultiPoint,andMultiPolygon;OpenStreetMap(OSM)isacollaborativeprojecttocreateafreeeditablemapoftheworld.Figure4andFigure5presentsUserInterface:informationfromtheselectedDataCubearevisualisedasataggedmap.

3.4.3 AvailabilitySourcecodeisavailableatGitHub:https://github.com/OpenGovIntelligence/qb-geo-ui

Figure4LithuanianDataCubesGeobasedVisualizationscreensamples–A


Page21of25

Figure5LithuanianDataCubesGeobasedVisualizationscreensamples–B

3.4.4 LicenseThesoftwareisavailableasopensourceundertheMITLicense.

3.4.5 PilotsinvolvedEnterpriseLithuaniaistheinitialuserofthetool.



1. Collectinguserfeedbackonthetool.2. Updatingthetoolbasedontheuserfeedbackandtooltestingresults.

3.5 SPARQLConnectorforExploratorySPARQLconnectorforExploratoryallowstoconnecttheExploratoryDataSciencetooltoaSPARQLendpoint.ThiscomponentisprovidedasRcodetogetherwithaJSONconfigurationfile.Ithasbecomepartofawidelydeployedproduct.

3.5.1 FunctionalityDescriptionThiscomponentenablesimportofdatastoredasRDFintotheExploratorysystemintoExploratory.


ThiscomponentisprovidedasRcodetogetherwithJSONconfigurationfile.

3.5.3 AvailabilityExtensionisavailableat:https://github.com/OpenGovIntelligence/exploratory_sparql_plugin


Page22of25

3.5.4 LicenseForthelicensedetailspleasecheckhttps://exploratory.io/terms

3.5.5 PilotsinvolvedTheFlemishGovernmentistheinitialusersofthetool.


hiscomponenthasbecomepartofawidelydeployedproductandalreadymeetsitsdesignobjectivessonofurtherdevelopmentofthisisrequiredorforeseen.

3.6 JSONAPIforDataCubeSpecification–updateFollowingevaluationofinitialimplementations,itwasdecidedtoimplementasecondversionoftheAPIusingtheGraphQLquerylanguageforAPIs,dueitsfastgrowingadoptioninthewebdevelopercommunityandbecauseitsfeaturesareagoodmatchforourrequirementstosearchforandfilterRDFDataCubes.

ThesetofAPImethodshasbeenre-specifiedusingtheGraphQLapproach,andexpandedasadditionalusefulfunctionshavebeenidentified

TheworkontheAPIisavailableathttps://github.com/Swirrl/graphql-qb

3.7 JSONAPIforDataCubeImplementation-updateDuringyear2oftheproject,theAPIhasbeenimplementedusingGraphQL.Workiscontinuingtoaddnewdataaccessmethods,tooptimiseperformanceandtoimproveuserdocumentation-availableathttps://github.com/Swirrl/graphql-qb

3.8 Table2QBAndGrafter-updateDuringyear2of theproject, further researchhasbeencarriedout into theoptimaldesignof thetable2qbinputformat,tomakeitflexibleenoughfortherangeofusecasestobesupported,butstilleasyforstatisticiansoranalyststoproduce,withoutneedingtechnicalknowledgeofLinkedData.

• Furtherprototypeshavebeenimplementedandtested• ImprovementstotheGrafterlibrarieshavebeenimplementedtoimproveperformanceand

scalabilitywhenprocessinglargedatacubes.

Thetable2qbspecificationcanbefoundathttps://github.com/Swirrl/table2qb

Graftercanbefoundathttps://github.com/Swirrl/grafter


Page23of25

3.9 DataCubeAggregator-updateDuringyear2oftheproject,theDataCubeAggregatorwasextendedtosupportthecomputationofaggregationsfromrawRDFdata.TheextensiontakesasinputrawRDFdataandcreatesacubethatcontainsthecorrespondingaggregatedobservations.ThecurrentversionoftheDataCubeAggregatoris integrated with version 1 of the JSON-QB API available at GitHub repository:https://github.com/OpenGovIntelligence/json-qb-api-implementation

FutureworkisaimingtomigratetheDataCubeAggregatortothenewAPIimplementationbasedonGraphQL.


Page24of25

4 Conclusion

This deliverable provides description of the components delivered in the second phaseof OpenGovIntelligence project (Month 21 of the project). Detailed information about usageofpreviouslyavailabletoolsaredocumentedinDeliverableD3.2–“ReportonOpenGovIntelligenceICTtools–firstrelease”.

AllthetoolsavailableastheresultofOpenGovInteligenceprojectareconstantlyupdatedbasedontheevaluation result. Theplans fornext yeardevelopment include theusability andperformanceimprovements. It is expected that all project pilots will use suitable tool. Moreover, numerousgovernment organisations external to the project are expected to start usingOpenGovInteligencetoolsinyear3oftheproject.

ContinuousJSON-QBAPIdevelopmentisexpectedtoextendtherangeofmethodssupportedbytheAPI,deployinoperationalenvironments.Oneofthemajorfeaturesunderdevelopmentisaggregationacrosshierarchies. In thiscase theobservationsareaggregatedacrossahierarchyofadimension.Forexample, if a cube contains theelection results atmunicipality level, then theAggregator cancompute the results at region and at country level with the prerequisite that the correspondinghierarchy(municipality→region→country)exists.

ThisdeliverablewillbeupdatedtogetherwiththeongoingdvelopmentoftheOpenGovIntelligencetools.ThefinalreleaseofthedevelopedcomponentswillbedocumentedindeliverableD3.6“ReportonOpenGovIntelligenceICTtools”plannedatOctober2018.


Page25of25

References[1]D1.1OpenGovIntelligencechallengesandneeds

[2]D2.1OpenGovIntelligenceframework-firstrelease

[3]D2.2OpenGovIntelligenceframework

[4]D3.1OpenGovIntelligenceICTtools-firstrelease

[5]D3.2ReportonOpenGovIntelligenceICTtools-firstrelease

[6]D4.1PilotsandEvaluationplan

Documents

OGI D3.4 - Report on OpenGovIntelligence ICT tools ... - Rep… · D3.4 Report on OpenGovIntelligence ICT tools – 2nd release Page 3 of 25 Abstract: This deliverable provides details