Bridging research and collections

Bridging research and

collections

Vyacheslav Tykhonov - Software Developer

[email protected]

http://www.linkedin.com/in/vyacheslavtikhonov

Jerry de Vries - Information Analyst

[email protected]

http://nl.linkedin.com/pub/jerry-de-vries/13/751/537

This presentation

• Mission statement of IISH

• Adjusting ICT-strategy

• Requirements for software development

• Solutions

– Demo / Proof Of Concepts (POC) of projects & tools

• Questions

25/03/2013 2

Bridging research and collections

Mission statement

The IISH conducts historical research on labour relations at a

global scale and to this end collects data, which are made

available to other researchers as well

• Do research

• Search, use, visualize and update data

• Collecting and preserving the data

• Make data available for research and public

25/03/2013 3


What is our data?

• Metadata (describing data and collections)

• Scans / Full-text

• Image, sound, movie, books & serials

• Datasets

• Aggregation (Metadata, full-text papers and datasets)

• Analogue, digitized and digital

25/03/2013 4


What are our target groups?

We are listening to our target groups:

• Researchers

• Collectors

• Public Audience

We are collecting all ideas and requirements from you!

25/03/2013 5


Historical Research Methodology

What is historical research in IISH?

1. Formulation of the research question

2. Data collection and/or literature review

3. Evaluation of materials

4. Data analysis

5. Write and publish articles

6. Sharing datasets

25/03/2013 6


Data Collecting Methodology

What is collecting in IISH?

• Collecting data

• Storing data

• Preservation • Digitization/scanning

• OCR/full text

• Metadata/MARC21/Indexing

• Make data public available in digital infrastructure

25/03/2013 7


Customer Development Methodology

What is software development in IISH?

Our target groups are sharing with us:

• Requirements

• Experiments

• Insights and ideas

1. Create prototype based on requirements and ideas

2. Present prototypes of software tools to our target group

3. Improve software tools after feedback from our target

group

25/03/2013 8


Where? Who?

25/03/2013 9


WE

IISH

Research CODI

General public

Mission statement




Let’s now look into collecting first!

25/03/2013 10


Typical collectors requirements

• Describe, index and store Metadata in digital library

system

• Improve Metadata • Based on computer based analysis and Natural Language Processing tools

• Link Metadata from IISH to other Metadata systems

• Search and discover digitized and digital born materials

• Transform Metadata into research data (datasets)

25/03/2013 11


Indexing

Extract entities and store it as terms in Metadata

25/03/2013 12


Manual Automatic

Metadata Collections Scans

CODI HiTIME

Automatic indexing example

Metadata linked with Evergreen Authorities:

Vladimir;;VladiMir;;566353;;Personal Name

Congress;;Video Congress;;316063;;Meeting Name

Lenin;;Lenin;;570134;;Uniform Title

Switzerland;;Switzerland;;350823;;Geographic Name

Second Congress of the RSDRP;;411162;;Meeting Name

25/03/2013 13


Input from scan:

Founded at the initiative of Vladimir I. Lenin in 1901 in Switzerland after

the Second Congress of the RSDRP in 1903 the League became the

main bulwark of Menshevism abroad until it disbanded in 1905.

Solutions for collectors

• Evergreen Library System Product

• Metadata management Product

• Metadata reports Product

• Evergreen OAI protocol Product

• Text analyzing tools (collectors & researchers) Prototype API

25/03/2013 14


https://195.169.123.134/opac/en-US/skin/default/xml/rresult.xml?rt=keyword&tp=keyword&t=amsterdam&ft=&l=1&d=0&f=

http://evergreen-ils.org/dokuwiki/doku.php?id=evergreen-user:installing_and_running_the_staff_client

http://evergreen.iisg.nl/reports/

http://api.socialhistoryservices.org/evergreen/oai/

http://node-195.dev.socialhistoryservices.org/api/

Project overview: Evergreen Collectors Metadata Storage System

• Perfect library solution to store Metadata in MARC21

standard

• Open-Source License (free of charge for usage)

• Flexible and Powerful solution, works with millions of

MARC records

• Export of all data in OAI-PMH protocol to link data with

other systems

• Visualization tools to present data online

25/03/2013 15


Evergreen Library System

25/03/2013 16


Metadata management

25/03/2013 17


Mission statement

Remember:




Let’s do some research!

25/03/2013 18


What is historical research?

The process of systematically examining past events to give an account of what has happened in the past

Why do we conduct historical research? • To uncover the unknown • To answer questions • To identify the relationship that the past has to the present • To record and evaluate the accomplishments of

individuals, agencies, or institutions • To assist in understanding the culture in which we live And much, much, much more…

25/03/2013 19


Typical research requirements Access to information

• Find digital materials relevant for research

• Search information stored in Metadata • Poor quality of Metadata = Poor quality of research

• Searching, filtering, navigating, summarization of data

• Analyze papers for research online

• Link materials relevant to research from other sources

• Collection descriptions are relevant to the topic of

research, but papers aren't

25/03/2013 20


Typical research requirements Datasets

Store datasets in a digital infrastructure to answer research

questions

• Use best practice for visualization of datasets

• Generate custom datasets for new research

• Combine/compare datasets in time and/or place

• Share datasets with other researchers (collaboration and

crowdsourcing)

25/03/2013 21


General goal of research

25/03/2013 22


All Data

Possibly relevant

Data

Definately relevant

Data

Structured

Knowledge

Sharing your research

• Publish scientific articles on websites relevant to the topic

of research

• Share research datasets with other researchers

• Generate charts and maps in real-time in digital

infrastructure based on live data • Publish in articles and share on Wikipedia and other popular websites

• Make biographies of famous people more attractive with

timelines of visual materials

25/03/2013 23


Indexing (keywords) For researchers

• Researchers publishing keywords in the beginning of

every research paper

• Keyword in research paper = Index term in Metadata

• Keywords from papers stored as Metadata in library

system

• Keywords used in text analyzing systems to create links

with other papers on the same topic

25/03/2013 24


Solutions for researchers

Data:

• Search engines Prototype

• Linked data Prototype

• Timelines Prototype

25/03/2013 25


Datasets:

• Maps Product

• Charts Product

• Visual Library System Prototype

http://node-195.dev.socialhistoryservices.org/cgi-bin/search.cgi?query=marx

http://node-195.dev.socialhistoryservices.org/cgi-bin/linked.cgi?id=ARCH02300

http://api.socialhistoryservices.org/evergreen/timeline/beatrix.html

http://www.clio-infra.eu/datasets/

http://beta.clio-infra.eu:8080/country.html?explore=Netherlands

http://api.socialhistoryservices.org/evergreen/charts/report.1900.html

Datasets visualization tools: Maps

25/03/2013 26


Datasets visualization tools: Charts

25/03/2013 27


Datasets visual library explorer

25/03/2013 28


Linked data for collectors

• 500000+ authority records in IISH collection

• Bibliographic records linked to authorities by collectors

• Link bibliographic records to authorities automatically in

real time with Authority Linking Module

• Import Metadata from other sources (Google Books,

WorldCat, etc) and link with our authorities

25/03/2013 29


Linked data for researchers

Metadata from IISH is available for harvesting:

• Search (search.socialhistory.org)

• OCLC's WorldCat

• Europeana

• Nederlab and other projects

Link authorities from Evergreen automatically to all other

systems to get more data for doing research

25/03/2013 30


Project overview: Clio Infra

• Datasets Storage System

• Online Visualization of Datasets: • maps, charts, timeline

• Tools to compare data for different countries in time

• Export of custom datasets

25/03/2013 31


Project overview: HiTIME Text Analyzing System

• Matching/linking of authority records from other systems: • Locations

• Persons

• Organizations

• Dates

• NLP tools to recognize unknown entities

• Export to library as Metadata

• Visualization of Metadata on timelines, maps, charts

25/03/2013 32


Named

Entities

Workflow

25/03/2013 33


Metadata system

Storage

Presentation Research

What have we seen today?

We are here for you and together we work on: • Search & Discovery

Metadata searching and filtering, Full-Text Search engines, Linked Data tools, Research Indexes (Controlled Vocabularies)

• Visualization

Charts, graphs, timelines, network connections tools • Analysis

Data Mining, Summarization, Topic Modeling, Tools for Datasets

25/03/2013 34


Questions?

• Feel free to ask now

• Ideas and questions can be sent by email to us

25/03/2013 35


Vyacheslav Tykhonov - Software Developer

[email protected]

http://www.linkedin.com/in/vyacheslavtikhonov

Jerry de Vries - Information Analyst

[email protected]

http://nl.linkedin.com/pub/jerry-de-vries/13/751/537

Technology

Bridging research and collections