Visual Statistics Storytelling

Explore, Collaborate and Publish Official Statistics Insight

and Knowledge

Professor and Director Mikael Jern

NComVA – Norrköping Communicative Visual Analytics,

mikael.jern@ncomva.com

http://ncomva.com and http://ncomva.se

The “participative web” is increasingly influenced by improved intelligent web services that

contribute to collaborate and communicate visual web content. NComVA introduces a Geovisual Analytics Statistics framework for communicating and presenting official statistics based on richer and more dynamic visual user interfaces and its increasing role in worldwide communication. The combination of Statistics eXplorer and Statistics Publisher represents the tool for an integrated statistics analysis, collaboration and publication process facilitating storytelling aimed at producing statistical news content in support of an automatic authoring process. The author can simply press a button in Statistics Publisher to publish the gained insight and knowledge from a visual interactive discovery process in Statistics eXplorer.

This process focuses on the most ancient of social rituals “Storytelling” - telling a story about a region’s development over time and shape the measure of economic growth and well-being. Discoveries that more engagingly draw us into reflections about the knowledge on how life is lived - and can be improved – from region to region and in addition let the reader dynamically participate in this process and help advancing research critical to the dissemination of official statistics by means of web-enabled tools. A platform for dissemination of embedded dynamic statistics data visualization with the analytics sense-making metadata (story) based on snapshots, associate text or sound joined together and publishable in any web pages such as blogs, wikis etc. Publishing official statistics through assisted content creation with emphasis on visualization and metadata represents a key advantage of our storytelling and has the potential to change the terms and structures for learning.

Figure: Statistics eXplorer (left) and a “Vislet” produced from an OECD Regional Statistics (eXplorer)

Storytelling practice.

Copyright NComVA AB

Figure 1: The Statistics eXplorer Storytelling loop

The Statistics eXplorer storytelling (figure) starts with the authoring and expert analysis process followed by a publisher process to present gained insight and knowledge:

1) select existing data or import external regional statistical data; 2) explore and make discoveries through trends and patterns and derive insight - gained

knowledge is the foundation for 3) creating a story that can be 4) shared with colleagues and reach consensus and trust. Visual discoveries are captured into

snapshots together with descriptive metadata and hyperlinks in relation to the analytics reasoning. The author gets feedback from colleagues, adopts the story and

5) finally publishes “tell-a-story” to the community using a “Vislet” that is embedded in blogs or Web pages.

The conceptual approach to our concept is based around three complementary characteristics: eXplorer, storytelling and Publisher (Vislet):

Authoring (Statistics eXplorer): data provider and manager, motion visual representations including choropleth map, scatter plot, table lens, parallel axes, time graph, data grid, coordinated views, map layers, analytic tools (dynamic query, filter, regional categorization, profiles, highlight), dynamic colour scale.

Storytelling: snapshot mechanism, metadata with hyperlinks, story and chapters, edit, capture, save, export story, publish story.

Statistics Publisher (create Vislets): Import stories and create HTML code for embeddable interactive motion visual representations including choropleth map, scatter plot, histogram, parallel axes, table lens, time graph and metadata for publishing HTML code in blog, wikis etc.

Snapshot Mechanism in Statistics eXplorer When exploring and making sense of comprehensive statistics data, we need a coherent cognitive

workspace to hang our discoveries on for organizing and navigating our thoughts. The GAV Flash toolkit includes such means by capturing saving and packaging the results of an eXplorer “gain insight” process in a series of “snapshots” that could help the analyst to highlight views of particular interest and subsequently guide other analysts to follow important discoveries. The snapshot tool creates a single or a continuous series (Story) of visualization captures during the exploration process. In a typical scenario the analyst has selected relevant attributes, time step (temporal data), regions-of-interest, colour class values, filter conditions for selected attributes and finally highlights the “discovery” from a certain angle (viewing properties).

Figure 2: The snapshot mechanism integrated with the GAV Flash toolkit is an important feature that allows

sharable stories to be created in Statistics eXplorer and saved as an XML file.

The analyst requests a snapshot with a Capture function that results in a snapshot class operation

scanning through all its connected GAV Flash visualization components for properties to be captured. Each of these properties will then be parsed into XML and written to a file that also contains details on which data (attributes and GIS regions) was used and a unique name for each component. When a snapshot is activated, the saved state of the snapshot class will be read from the XML file (figure 4) and parse its nodes back into component properties again. The previously marked properties will then be applied and set the state of the application.

Storytelling

Storytelling, in our context, is about telling a story on the subject of statistics data and related analytics reasoning about how gained insight and knowledge is achieved. Storytelling within this participative web context, could more engagingly draw the user into exciting reflections and sometimes change a perspective altogether. The story is placed in the hands of those who need it, e.g. policy and decision makers, teachers but also the informed citizens. Dynamic visual storytelling is a way of telling stories through interactive web-enabled visualization. Our novel storytelling technology in Statistics eXplorer could advance research critical to collaboration and dissemination of digital media and enable a leap in understanding by the audience so as to grasp how statistical indicators may influence our society.

Statisticians with diverse expert background and expertise participate in a creative discovery

processes (figure 1) that transforms statistical insight into knowledge. Storytelling tools integrate this

geovisual analytics statistics process with collaborative means that streamline a knowledge exchange

process of developing a shared understanding with other statisticians and after consensus has been

reached can be published. The snapshot mechanism in Statistics eXplorer helps the author of a story

to highlight data views of particular interest and subsequently guide others to important visual

discoveries.

Figure 3: The storytelling mechanism allows a sharable story to be created in Statistics eXplorer and saved

as an XML file. Readers can then import the story into Statistics eXplorer and follow the analyst’s way of

reasoning through descriptive text and hyperlinks that instantiate snapshots in the visual representation.

The author creates a single or a discrete series of captures or snapshots (figure 2) during the explorative process by electing relevant indicators, regions-of-interest, colour schema, filter conditions focusing on the data-of-interest or a time step for temporal statistics.

Hypertext, meaning "more than just text", provides a richer functionality than simple metatext by allowing the reader to click on key words and learn about topics in the story. A story hyperlink is here a reference in the story metatext that links to an external URL web site or a captured snapshot. To insert a hyperlink in the metatext then select the text and a button “Link” is made visible and two options appear:

a. New capture (snapshot) b. Link to an external URL.

Before the actual capture is done, the user navigates, for example, the map view to a particular country, select indicator, select indicators for the scatter plot, select time

step. A new view such as parallel axes can be added to the story etc. A “Capture” is made and all preferred states are saved. When the story later is read, hyperlinks can be initiated and eXplorer will display the state-of-the-snapshots.

Hyperlinks that instantiate an Statistics eXplorer state are a central feature of our storytelling mechanism together with associated descriptive text that could guide the reader in the analyst’s way of thinking. While it’s true that a picture is often worth a thousand words, sometimes a few words and a snapshot provide the difference between a pretty picture and understanding. This focus on publishing through assisted content creation with emphasis on visualization and metadata represents a novel advantage of our storytelling technique.

Statistics Publisher and Vislets

A Vislet is a standalone Flash application (widget) assembled from low-level GAV Flash components in a class library and Adobe Flex GUI tools and is represented by, for example, a single

map view or a composite time-linked map and scatter plot view (figure 4). A Vislet facilitates the transition of selected tedious statistics data into heterogeneous and communicative sense-making news entities with integrated metadata and dynamic embedded animated visualization that could engage the user.

Figure 4: Examples of Vislets. A Vislet can be a standalone visualization component (map, scatter plot etc.) or a composite Vislet with linked visualization views (map + scatter plot), (histogram + scatter plot) or (map + scatter plot + time graph) etc. This means that a Vislet does not have use the same visualization methods as during the storytelling creation. The top left Vislet, for example, represents a time-linked Histogram and scatter plot animation. (with a map).

Figure 5: Statistics Publisher GUI panel for producing the HTML code to be embedded into a blog or web site. Statistics Publisher is an application server tool that imports a story and generates the HTML code

that represents the Vislet and metadata in a blog or web site. First, the user selects appropriate visual representations for the Vislet e.g. map, scatter plot, parallel axes, table lens or time graph. Then the size of the Vislet window with metadata is set and Publisher can generate the HTML code. Notice, that you may select only a standalone Histogram or two linked views Histogram and Scatter Plot (figure This code is manually copied (figure 5) and finally manually (copy/paste) embedded into a web page (see case study below). The Vislet can now be opened in the reader’s Web browser and dynamically communicate the story. A Publisher server maintains the Vislet flash (swf) files together with a story repository, statistical data and regional shape maps. The Vislets run locally in the client’s Flash Player and can thus achieve dynamic interactive performance.

Interactive features in a Vislet are exposed to all visualizations including tooltips, brushing, highlight, filter that can discover outliers and dynamic multiple-linked views. Several specialist colour legend tasks are supported e.g. show outliers based on 5th and 95th percentiles in certain colours or dynamic sliders that control class values etc.

Figure 6: Example of two Vislets with metatext using only a single Histogram (bottom) and a time-linked

Scatter Plot (top) to communicate insight and knowledge. The author can use Statistics eXplorer to explorer

and analyse, writing stories using all visualization methods available in Statistics eXplorer. Later when the

authors want to publish a story, single or any combination of visualization methods can be selected for the

storytelling. Sometimes a single dynamic Histogram showing “Tertiary Education Attainment” for the OECD

TL2 regions can be sufficient to gain insight about where the most educated regions are located in OECD.

Case Study “Storytelling”

Our case study for a storytelling process is based on the OECD regional eXplorer: http://www.oecd.org/gov/regional/statisticsindicators/explorer and is here used to explain the process from creating a Story with snapshots and metatext, save the Story on an external file and finally use Statistics Publisher to load the Story again, select visualization methods and generate the

HTML code that is placed in a Web site - a Vislet is created.

In this case study, we have used OECD regional eXplorer to demonstrate the process from writing a Story to publishing the Story on a Web site. First select indicators and time steps for the story.

Statistics eXplorer provides several visualization methods to explore and gain insight about statistics data. Select only a map or a map linked with a scatter plot. For temporal data, map + scatter plot + time graph is recommended.

Start writing your Story in the “Story Editor Panel”. Provide Story title, name of author, organisation and chapter title. You can type the text direct into the metadata area or copy-and-paste text. When you close the Story Panel the Story will appear in the right view of Statistics eXplorer.

A “capture” is a reference in the Story metatext that links to an external URL web site or a captured snapshot. To insert a hyperlink in the metatext then select the text and a button “Link” is made visible and two options appear:

a. New Capture (snapshot) b. External URL.

The author creates a single or a discrete series of captures during this explorative process by electing relevant indicators, regions-of-interest, colour schema, filter conditions, focusing on the data-of-interest or a time step for temporal statistics etc. The URL can link to any web

resource, such as website with complementary metadata or a sound file where the author tells the story about a snapshot.

The user can also start a new Chapter in the Story with new Indicators and create a new scenario and repeat the process above by pressing the button “New” in the Chapter panel. When the Story is completed then press Close; Use the button Export, give the Story a name "my story nr 2".xml. REMEMBER the prefix .xml. The Story is now saved locally on your computer and can be reused Import or sent to a colleague for review in OECD regional eXplorer.

Set Vislet properties in the Publisher panel e.g. height and width for the graphics layout including visualizations and metadata, background and text colour.

Select the most suitable visual representations (map, scatter plot, histogram, table lens, parallel axes, time graph etc) for your Story and best understandable by a reader. This is done in drop-down menus. For example, a histogram linked with scatter plot:

You can preview the Vislet in Publisher and see the result before you complete the publishing process.

The HTML-code is now automatically generated by Publisher and copied when you press “Copy”. This HTML code, when pasted into a web page or blog, will load your Vislet with associate metadata and hyperlinks using the settings you have chosen (figure 6).

Paste the HTML code either into a web page or directly into a CMS or blog.

The resulting Vislet in a blog: http://visletblog.blogspot.com/2010/08/oecd-explorer-vislet-ageing-population.html

Here is the same Story but without a map view and instead a histogram and scatter plot view.

http://www.ncomva.se/flash/vislets/regional_01.html

Below is a list of Vislet demonstrators analysed with Statistics eXplorer and published with Statistics Publisher: World eXplorer (22 indicators from Worldbank 1960-2008): http://www.ncomva.com/?page_id=297 World eXplorer map+scatter plot+histogram http://www.ncomva.se/flash/vislets/world_01.html World eXplorer in a Statistics Sweden Blog at: http://swedeneurostat.blogspot.com/ OECD eXplorer Histogram and Scatter Plot: http://www.ncomva.se/flash/vislets/regional_01.html OECD regional eXplorer (ageing population in Europe 1990-2008): http://visletblog.blogspot.com/2010/08/oecd-explorer-vislet-ageing-population.html OECD Regional eXplorer: www.oecd.org/innovation/strategy/charting OECD Regional eXplorer: http://vitagate.itn.liu.se/GAV/development/testVislet/ and only histogram and scatter plot: http://www.ncomva.se/flash/vislets/regional_01.html OECD Regional eXplorer (histogram only): http://vitagate.itn.liu.se/GAV/development/3.7/vislets/oecdRegional-barChart.html OECD Factbook: http://www.ncomva.com/?page_id=111 (NComVA Vislet Demo with OECD Factbook – fertility versus women employment rate for 1970-2006) Eurostat eXplorer in SCB Eurostat Blog: http://swedeneurostat.blogspot.com/2010/03/education-in-eu-example-with-vislets.html#links Denmark eXplorer: http://www.dst.dk/OmDS/BagTal/Arkiv/2010-09-06-explorer.aspx Denmark eXplorer: http://www.dst.dk/OmDS/BagTal/Arkiv/2010-05-06-Motorcykler.aspx Goteborg eXplorer: http://www.samhallsutvecklingen.se/413/stora-inkomstskillnader-i-goteborg/ Two Vislets (Eurostat eXplorer and OECD eXplorer) embedded in a Blog: http://visletblog.blogspot.com/2010/08/eurostat-explorer.html http://visletblog.blogspot.com/2010/08/oecd-explorer-vislet-ageing-population.html Sweden eXplorer http://visletblog.blogspot.com/2010/06/aldrande-befolkning-i-manga-svenska_22.html (Ageing population in Swedish municipalities 1990-2008) Or find it here embedded in a Web page: http://www.ncomva.se/?page_id=47 Categorical Test with Swedish Municipalities: Categorical Test: http://www.ncomva.se/?page_id=56 (You can select and filter display of municipalities by clicking in the legend boxes) The Economist Vislet: http://www.economist.com/britain-in-context/

Technical Description

Vislet and story

The story (XML file) must be available on a server for the Vislet to be able to access it. The same goes for static data such as indicators and GIS. However, static data is usually put on the server when the application is installed; stories are created by users at any time after that installation.

To make a story available online you can either put it on a server manually (e.g. with FTP) or you can use some kind of service to upload it (into a CMS for example). Our backend fits the latter

description; it is a service that takes local stories in a form and puts them on the web server. Such a form is integrated into the Vislet Publisher through the “Local story file” step.

If you choose not to install our backend on your server it is the responsibility of the story author to put the story file on a server and provide the URL to the Vislet Publisher so that a Vislet can be generated for you.

Proxy

Due to security limitations in the Flash player, the Vislet cannot communicate with servers on arbitrary domains, usually just the domain where

it remains. The common way to address this is to introduce a policy file on the target server which states if the flash application source domain has access rights. Although it is an efficient way of dealing with cross-domain traffic between trusted servers, there might be issues regarding having such a policy file allowing public connections. More on flash policy files can be found at:

http://www.adobe.com/devnet/flashplayer/articles/cross_domain_policy.html

To address this issue and allow communication with arbitrary domains we introduce a proxy script on the same domain as the Vislet. That way the Vislet can communicate with the proxy and the proxy fetches the story from some other domain and forwards it to the Vislet. The proxy we supply is a simple java and PHP script, but it does require a server that allows and can run it. A proxy can be built in different ways, with different levels of security. Out implementation has an example of a simple domain check, which looks at the target URL and makes sure it is the domain it is designed for.

Vislet Publishing architecture without backend

Vislet Publishing architecture with external Story Repository and Proxy connection

Comparison between the vislet flash application fetching a story in the presence of a

cross domain file (top) and by using a proxy (bottom)

Backend

The Publisher is not dependant on a particular implementation of the backend. Although we only provide a PHP backend; you can implement another backend, e.g. if your server cannot execute PHP scripts. The HTML code generated by the Publisher includes an URL to a story file, this can be a real file as in our PHP backend case where we just store the xml directly, or it can be a reference to a script that connects to a database and returns the correct story. The backend can be placed on a different domain, but in this case there has to be either a cross domain file or a proxy to allow transfers between different domains.

Vislet Publisher Package

Statistics eXplorer

Statistics eXplorer is generally a package containing an eXplorer.swf application file and a configuration. This also contains maps, data and stories referenced by the configuration. The vislet publisher can be used with almost any existing eXplorer capable of generating stories. The exception is that it can only be used with the default configuration of an eXplorer, if a story is created in an alternate configuration, such as a single map version, the vislet will still read data from the original config file.

Vislet (.swf)

The Vislet.swf is a single flash application file with no extra references. All input to the swf comes through the embedding HTML (generated by the Publisher). This input contains references to which story to load and how the panels should be set up. Vislet automatically loads the eXplorer configuration located in the same folder as the swf.

Statistics Publisher (.swf)

Publisher.swf configurable to use either a backend or a proxy to provide the HTML code needed to embed Vislets. The configuration is done through an xml file that is loaded when the swf runs. If a backend is not present, then the backend upload option will not be available. As stated in the Techincal Proxy section, a proxy is needed if a cross domain file is not present.

Proxy (.php)

This simple implementation of a proxy takes an input URL and then returns its content to the sender. The input URL is specified as a variable at the end of the URL named target_url: www.example.com/proy.php?target_url=www.aaa.com

crossdomain.xml

Standard policies file allowing access from a specific domain

eXplorer-php-backend

This folder contains a collection of PHP scripts that enable the Publisher to save xml files to the server. This implementation is very straightforward and writes any input to incrementally named (0.xml, 1.xml etc) files in a specified folder. For a public publication, a more robust implementation is required.

Vislet Publishing architecture with external Story Repository and Proxy connection