· Web viewMedia Content Management Systems. Magyar, Gábor. Szűcs, Gábor. Media Content Management Systems. írta Magyar, Gábor és Szűcs, …

Media Content Management Systems



Magyar, Gbor

Szcs, Gbor


rta Magyar, Gbor s Szcs, Gbor

Publication date 2014

Szerzi jog 2014 Magyar Gbor, Szcs Gbor

Created by XMLmind XSL-FO Converter.



Tartalom

Media Content Management Systems 0

1. 1 BASICS 0

1.1. Data, information, knowledge 0

1.2. Data and information 0

1.3. Data management 0

1.4. Information management 0

1.5. Knowledge management 0

1.6. Media value chain 0

1.7. Multiscreen world 0

1.8. Participative media 0

1.9. digital literacy 0

2. 2 METADATA 0

2.1. Metadata, schemes and standards 0

2.2. Metadata 0

2.3. Basic types of metadata 0

2.4. Metadata (structure) Standards 0

2.5. Dublin Core Metadata Initiative (DCMI) 0

2.6. DC Metadata Element Set 0

2.7. Design goals for DCMES 0

2.8. DCMES: 15 elements 0

2.9. Commonly accepted: 0

2.10. Element name semantics 0

2.11. DC qualifiers 0

2.12. DCMI Terms 0

2.13. Dublin Core evaluation 0

2.14. Application profiles 0

2.15. "Levels of interoperability" 0

2.16. Learning Object Metadata (LOM) 0

2.17. IEEE-LOM 0

2.18. IEEE-LOM evaluation 0

2.19. Uses of IEEE-LOM 0

2.20. METS 0

2.21. METS structure 0

2.22. METS metadata 0

2.23. METS Descriptive Metadata 0

2.24. METS Administrative Metadata 0

2.25. METS File Inventory 0

2.26. METS Structural Map 0

2.27. MODS 0

2.28. MODS 0

2.29. MODS evaluation 0

2.30. MODS User Guidelines 0

2.31. element and attribute 0

2.32. element and sub element 0

2.33. MODS Elements 0

2.34. EAD: Encoded Archival Description 0

2.35. EAD features 0

2.36. MPEG standards 0

2.37. MPEG-7 0

2.38. Application model 0

2.39. main elements of MPEG-7 standard 0

2.40. MPEG-7 DDL 0

2.41. MPEG-7 Visual 0

2.42. MPEG-7 Visual examples 0

2.43. MPEG-7 Audio 0

2.44. MPEG-7 and MPEG-21 community 0

3. 3 SEMANTIC CONTENT MANAGEMENT 0

3.1. Semantic content management/1 0

3.2. RDF 0

3.3. Why we need RDF? 0

3.4. URI 0

3.5. RDF 0

3.6. Subject, Predicate, Object 0

3.7. statement 0

3.8. abbreviations 0

3.9. expressive strength 0

3.10. Common concepts 0

3.11. Common concepts 0

3.12. RDF vocabulary 0

3.13. Semantic Graph Data Model 0

3.14. RDF Schema 0

3.15. RDF Schema 0

3.16. Classes 0

3.17. Defining Classes 0

3.18. Defining Instances 0

3.19. rdf:type 0

3.20. Defining Classes (cont'd) 0

3.21. Notation 0

3.22. Defining Subclasses 0

3.23. Classes and Instances 0

3.24. Properties of rdfs:subClassOf 0

3.25. RDF Schema Predefined Classes 0

3.26. Properties 0

3.27. Defining Properties 0

3.28. Defining Properties 0

3.29. Domain and Range 0

3.30. Datatypes for Ranges 0

3.31. The Property rdfs:label 0

3.32. The Property rdfs:comment 0

3.33. The Property rdfs:seeAlso 0

3.34. RDFS vs. Types (cont'd) 0

3.35. Schema Languages 0

3.36. The Semantic Web "Layer Cake" 0

3.37. URI, Unicode layer 0

3.38. XML layer 0

3.39. RDF layer 0

3.40. RDF Scheme 0

3.41. Ontology layer 0

3.42. Controlled vocabularies 0

3.43. Thesaurus 0

3.44. Thesaurus standards 0

3.45. Widely known thesauri 0

4. 4 CMS types 0

4.1. CMS types 0

4.2. DAM: Digital Asset Management 0

4.3. Broad categories of DAM 0

4.4. Subtypes of DAM 0

4.5. DAM file types 0

4.6. Application (areas) of DAM 0

4.7. DM: Document Management 0

4.8. Components of DM 0

4.9. Capture 0

4.10. Functionalities in DM 0

4.11. DM file types 0

4.12. Application (areas) of DM 0

4.13. LMS: Learning Management System 0

4.14. Functionalities in LMS 0

4.15. LMS standards 0

4.16. Most important LMS softwares 0

4.17. Web CMS 0

4.18. Functionalities in WCMS 0

4.19. Most important WCMS softwares 0

4.20. WCMS file types 0

4.21. KM: Knowledge Management 0

4.22. Knowledge Management software 0

4.23. ECM: Enterprise Content Management 0

4.24. Components of ECM 0

4.25. Capture component 0

4.26. Manage component 0

4.27. Store component 0

4.28. Preserve component 0

4.29. Deliver component 0

4.30. Output and distribution media 0

4.31. Most important ECM systems 0

4.32. Axes of Magic Quadrant 0

4.33. CMS plan, design, integration 0

4.34. Plan of functionalities (1) 0



4.37. Analyzing the content lifecycle 0

4.38. Roles 0

4.39. CMS adopting, integration 0

4.40. Content Migration 0

4.41. Two approaches 0

5. 5 Information Retrieval (IR) 0

5.1. Text Databases and IR 0

5.2. Information Retrieval (IR) 0

5.3. IR vs DBMS 0

5.4. Basic Measures in the IR 0

5.5. Keyword-Based Retrieval 0

5.6. Similarity-Based Retrieval in Text Databases 0

5.7. Retrieval : Ad hoc and Filtering 0

5.8. architecture 0

5.9. IR System and Tasks Involved 0

5.10. Models for IR - Taxonomy 0

5.11. Formal Specification of the Task 0

5.12. Formal Specific. of the Task (Cont.) 0

5.13. Boolean Retrieval Model - Queries 0

5.14. Boolean Retr. Model - Definition 0

5.15. Boolean Retrieval Model 0

5.16. Vector Model - Definition 0

5.17. Vector Model - Similarity 0

5.18. Vector Model 0

5.19. Weights 0

5.20. TF-IDF ranking 0

5.21. Classic Probabilistic Model (first) 0

5.22. Classic Probabilistic Model 0






5.28. Estimation of Term Relevance 0

5.29. (Dis) advantages of Classic Probabilistic model (last) 0

5.30. Summary: taxonomy of IR models 0

5.31. Alternative Probabilistic Models 0

5.32. Bayesian Network 0

5.33. Belief Network Model (1/6) 0






5.39. Summary 0

5.40. Alternative Set Theoretic models 0

5.41. Extended Boolean logic 0

5.42. Extended Boolean Model: 0

5.43. Extended Boolean Model: 0

5.44. Extend the idea to m terms 0

5.45. Properties: 0

5.46. Example: 0

6. 6 Information Retrieval on Web 0

6.1. Mining the World-Wide Web 0

6.2. Web search engines 0

6.3. Web Mining: A more challenging task 0

6.4. Web Mining Taxonomy 0







6.11. Multilayered Web Information Base 0

6.12. Multiple Layered Web Architecture 0



6.15. Benefits of Multi-Layer Meta-Web 0

6.16. Web Search Topics 0

6.17. Search on the Web 0

6.18. Classic IR vs. Web Search 0

6.19. General Web Search Engine Architecture 0

6.20. The Indexer Module 0

6.21. Storage Issues 0

6.22. Update Strategies 0

6.23. Batch vs. Steady 0

6.24. Partial vs. Complete Crawls 0

6.25. Generation Web Search Engines 0

6.26. The evolution of search engines 0

6.27. generation web IR ranking criteria 0

6.28. Considering the link structure 0

6.29. Considering the link structure 0

6.30. Simple PageRank - Definition 0

6.31. Simple PageRank - Example 0

6.32. Simple PageRank - Calculation 0

6.33. Problems if graph is not strongly connected 0

6.34. What happens with Leaks? 0

6.35. What happens with Sinks? 0

6.36. Intuitive Interpretation 0

6.37. PageRank - conclusion 0

6.38. PageRank in the Google engine 0

6.39. HITS: (Hyperlink-Induced Topic Search) 0

6.40. HITS 0

6.41. Systems Based on HITS 0

6.42. HITS: Identifying The Focused Subgraph 0

6.43. HITS: Link Analysis 0

6.44. HITS:Link Analysis Computation 0

6.45. Research issues 0

6.46. Summary of previous lessons: taxonomy of IR models 0

6.47. Search Engine Marketing 0

6.48. PPC and SEO 0

6.49. SEO: Search Engine Optimization 0

6.50. PPC vs. "Natural" SEO 0

6.51. What Are Searchers Looking For? 0

6.52. KEI values at different search engines 0

6.53. SEO techniques: White Hat vs. Black Hat 0

6.54. Black Hat (Spamming) 0

6.55. White Hat: The Seven Steps to Higher Rankings 0

6.56. 1 Get Your Site Fully Indexed 0

6.57. 2 Get Your Pages Visible 0

6.58. Dynamic Site Leaves Session IDs in URLs: 0

6.59. Optimize the use of Images 0

6.60. 3 Build Links and PageRank 0

6.61. Ideal internal site linking hierarchies: 0

6.62. 4 Leverage Your PageRank 0

6.63. Avoid PageRank dilution 0

6.64. Write better anchor text 0

6.65. 5) Encourage Clickthrough 0

6.66. 6) Track the Right Metrics 0

6.67. 7) Avoid Worst Practices 0

6.68. Not Spam, But Bad for Rankings 0

7. 7 Search improvement 0

7.1. Search improvement 0

7.2. Metasearch (1) 0

7.3. Metasearch (2) 0

7.4. Merging algorithms: 0

7.5. Re-ranking method based on inter-document distances 0

7.6. Introduce a function 0

7.7. Goal of the estimation 0

7.8. Relationships 0

7.9. Changing matrix 0

7.10. Document Weight Improving (DWI) 0

7.11. Example 0

7.12. Original relevance values 0

7.13. Calculations 0

7.14. Final results 0

7.15. Multimedia retrieval 0

7.16. Multimedia retrieval: Description vs. Content-based retrieval 0

7.17. Queries in Content-Based Retrieval Systems 0

7.18. Approaches Based on Image Signature 0

7.19. Mining Multimedia Databases 0

7.20. Multidimensional Analysis of Multimedia Data 0

7.21. Mining Associations in Multimedia Data 0



7.24. Preludes and application areas of CBIR 0

7.25. Content-based Image Retrieval (CBIR) 0

7.26. Specification of picture attributes 0

7.27. Comparison of images 0

7.28. Feature extraction for images 0

7.29. Shape Based methods for images 0

7.30. Scale-invariant feature transform (SIFT) 0

7.31. Matching by SIFT 0

7.32. Optical Character Recognition (OCR) 0

7.33. Importance of context 0

7.34. Role of the context 0

7.35. Text retrieval vs. Image retrieval 0

7.36. Experiment of concept search in Text retrieval 0

7.37. Trend of CBIR 0

7.38. New in CBIR: context 0

7.39. Video Retrieval 0

7.40. Multimedia Information Retrieval 0

7.41. Dimension reduction 0

7.42. Information Gain (IG) for Text Mining 0

7.43. Chi-square distribution for Text Mining 0

7.44. Principal components analysis 0

7.45. SVD 0

7.46. Latent Semantic Indexing for Text Mining 0

7.47. Latent Semantic Indexing (LSI) 0

7.48. LSI 0

7.49. Text Index Building 0

7.50. Bag of words 0

7.51. Query 0

7.52. Term-document incidence 0

7.53. Incidence vectors 0

7.54. Answers to query 0

7.55. Bigger corpora 0

7.56. Can't build the matrix 0

7.57. Inverted index 0

7.58. Inverted index 0

7.59. Inverted index construction 0

7.60. Indexer steps 0

7.61. The index we just built 0

7.62. Query processing 0

7.63. The merge 0

7.64. Boolean queries: Exact match 0

7.65. More general merges 0

7.66. Merging 0

7.67. Query optimization 0

7.68. Query optimization example 0

7.69. More general optimization 0

7.70. Exercise 0

7.71. Query processing exercises 0

7.72. Skip pointers for faster postings 0

7.73. Recall basic merge 0

7.74. Augment postings with skip pointers (at indexing time) 0

7.75. Query processing with skip pointers 0

7.76. Where do we place skips? 0

7.77. Placing skips 0

7.78. Problems and questions about term search 0

7.79. Detour: problems and questions 0

7.80. Beyond term search 0

7.81. Evidence accumulation 0

7.82. Ranking search results 0

7.83. Structured vs unstructured data 0

7.84. Unstructured data 0

7.85. Semi-structured data 0

7.86. More sophisticated semi-structured search 0

7.87. Tokenization 0


7.89. Parsing a document 0

7.90. Format/language stripping 0


7.92. Language issues 0

7.93. Tokenization: language issues 0

7.94. Normalization 0

7.95. Punctuation 0

7.96. Numbers 0

7.97. Case folding 0

7.98. Lemmatization 0

7.99. Stemming 0

7.100. Stemming 0

7.101. Types of stemmers 0

7.102. Mistakes of stemming 0

7.103. Measuring the stemmers 0

7.104. Evaluation of stemmers 0

7.105. Porter's algorithm 0

7.106. Phases and typical rules in Porter 0

7.107. Examples for Porter algorithm 0

7.108. Snowball 0

7.109. Other stemmers 0

7.110. Exercise for stemming 0

7.111. Exercise for stemming 0

7.112. Dictionary entries - first cut 0

7.113. Language detection 0

7.114. Language detection using -grams 0

7.115. -grams 0

7.116. Start: Zipf's law 0

7.117. Zipf's law: Reuters-21578 0

7.118. Zipf's law: web 2.2 (Hungarian) 0

7.119. Method 0

7.120. Creating profiles 0

7.121. Observations 0

7.122. Comparing profiles 0

7.123. Example 0

7.124. Flowchart 0

7.125. Results 0

7.126. Phrase queries, Positional indexes 0

7.127. Phrase queries 0

7.128. A first attempt: Biword indexes 0

7.129. Longer phrase queries 0

7.130. Extended biwords 0

7.131. Query processing 0

7.132. Other issues 0

7.133. Positional indexes 0

7.134. Positional index example 0

7.135. Processing a phrase query 0

7.136. Rules of thumb 0

8. 8 Exercise 0

8.1. General information 0

8.2. 0

8.3. 0

8.4. Exercise 0


8.6. Task 0

8.7. Exercise 0


8.9. 0

8.10. Exercise 0


8.12. Task 1 0

8.13. Comparison of two stemmers 0

8.14. Task 2 0







1. 1 BASICS

1.1. Data, information, knowledge

Data: facts of the World, mirrors reality(without meaning).Data on its own carries no meaning.

Information: interpreted data(for data to become information, it must be interpreted and take on a meaning)

Knowledge: information in context(by human being; or artificial intelligence?)

Data

+ interpretation

Information

+ context

Knowledge

(facts)

1.2. Data and information

4 dimensions:you should know the subject/phenomenon(the data refers to)in general and in concrete;+ attribute(s) of subject/phenomenon,in general and in concrete;

Example: "X car type achieved in NCAP safety test *****",you should know in general what a car is, in concrete X type;in general what is car safety, in concrete NCAP test

1.3. Data management

Storage, query, manipulation of raw data.

[data model, DB query, data manipulation languages, etc.]

1.4. Information management

interpretation,

systematization,

evaluation,

retrieval

of information

1.5. Knowledge management

Knowledge management: efforts, measures in order to increase corporate knowledge capital(Knowledge capital is part of the corporate assets.)

1.6. Media value chain

DE! 2012 NJSZT

1.7. Multiscreen world

Source: Google: The New Multiscreen World, 2012

1.8. Participative media

What do we need to participate?

access

content

digital literacy

1.9. digital literacy

multi-tasking

preferred: image, graphics, video

linearity vs. navigation

"copy-paste"

real-time communcation

personality (blog, twit)

"shorter user-concentration"

2. 2 METADATA

2.1. Metadata, schemes and standards

2.2. Metadata

Metadata: data from data

(all data, about other data, e.g. catalogue data)

"Metadata articulates a context for objects of interest - "resources" such as MP3 files, library books, or satellite images - in the form of "resource descriptions". As a tradition, resource description dates back to the earliest archives and library catalogs. The modern "metadata" field that gave rise to Dublin Core and other recent standards emerged with the Web revolution of the mid-1990s."

(source: Dublin Core Metadata Initiative)

2.3. Basic types of metadata

descriptive metadata: describes a resource for purposes such as discovery and identification. It can include elements such as title, abstract, author, keywords, date of last modification, etc.

structural metadata: indicates how compound objects are put together, for example, how pages are ordered to form chapters

administrative metadata provides information to help manage a resource, such as when and how it was created, file type and other technical information, and who can access it. There are several subsets of administrative data; two that are sometimes listed as separate metadata types are

Source: NISO (2004) Understanding Metadata

2.4. Metadata (structure) Standards

Dublin Core (http://dublincore.org)

IEEE-LOM (http://ltsc.ieee.org/wg12/)

MODS (www.loc.gov/standards/mods/)

EAD (http://www.loc.gov/ead/)

MPEG-7 (http://www.mpeg.org/)

2.5. Dublin Core Metadata Initiative (DCMI)

fostering the widespread adoption of interoperable metadata standards, development of specialized metadata vocabularies (to enable intelligent resource discovery systems -> make it easier to find resources in the Internet)

2.6. DC Metadata Element Set

15 descriptive semantic definitions

a core set of elements: could be shared across disciplines, any type of organization (to organize and classify information)

version 1.1:

Internet RFC 2413 (1998)

NISO Standard Z39.85-2001 (2001)

ISO Standard 15836-2003 (2003)

2.7. Design goals for DCMES

Simplicity of creation and maintenance

small and simple metadata element set: non-specialists can create descriptive records (for effective retrieval of information resources)

Commonly understood semantics

the semantics of DCMES are universally understood

International scope

standard to the multilingual and multicultural information universe.

Extensibility

mechanisms for extending the DC element set for additional resource discovery needs

2.8. DCMES: 15 elements

Title (Title):

Creator (Creator)

Subject (Subject and Keywords)

Description (Description):

Publisher (Publisher)

Contributor (Contributor)

Date (Date)

Type (Resource Type)

Format (Format)

Identifier (Resource Identifier)

Source (Source)

Language (Language)

Relation (Relation)

Coverage (Coverage)

Rights (Rights Management)

All elements are optional and repeatable

2.9. Commonly accepted:

Elements and Semantics

Definitions for the content of the elements (e.g., what does it mean: 'a title', 'a creator', etc.)

Content Rules

guidelines for inputting the element (what to capitalize, order of elements, etc.)

Syntax

rules for structuring and expressing the elements for machine processing

2.10. Element name semantics

Element Name: Title

Label: Title

Semantics: a name given to the resource.

Comment: typically, Title will be a name by which the resource is formally known.

Element Name: Creator

Label: Creator

Semantics: An entity primarily responsible for making the content of the resource.

Comment: examples of Creator include a person, an organization, or a service. Typically, the name of a Creator should be used to indicate the entity

2.11. DC qualifiers

to extend and refine the 15 DCMES elements

Element Refinement - these qualifiers make the meaning of an element narrower or more specific ("more restricted scope")

Encoding Scheme - these qualifiers identify schemes that aid in the interpretation of an element value (controlled vocabularies, formal notations, parsing rules)

2.12. DCMI Terms

Authoritative specification of all metadata terms related to DC, including elements, element refinements, encoding schemes, vocabulary terms

Maintained by the DC Usage Board

Contained in the DCMI Metadata Registry

2.13. Dublin Core evaluation

International and cross-domain

Increased efficiency of the discovery/retrieval of digital objects

qualified DC element set aids the management of information

easy mapping to other metadata standards

2.14. Application profiles

Consist of data elements drawn from one or more namespace schemas combined together by implementors and optimised for a particular local application.

Application profiles are useful as they allow the implementor to declare how they are using standard schemas

Characteristics:

May draw on one or more existing namespaces

Introduce no new data elements

May specify permitted schemes and values

Can refine standard definitions

Application profiles enable implementors "to share information about their schemas in order to inter-work with wider groupings...Communities can start to align practice and develop common approaches by sharing their application profiles."

2.15. "Levels of interoperability"

Level 1: shared term definitions (interoperability among metadata-using applications is based on shared natural-language definitions) Terms are "hard-wired" into applications.

Level 2: formal semantic interoperability (interoperability among metadata-using applications is based on the shared formal model provided by RDF)

Level 3: description set syntactic interoperability (applications are compatible with the Linked Data model and share an abstract syntax for validatable metadata records, the "description set")

Level 4: description set profile interoperability (records exchanged among metadata-using applications follow a common set of constraints, use the same vocabularies, and reflect a shared model of the world.

2.16. Learning Object Metadata (LOM)

An array of related standards for description of 'learning objects' or 'learning resources'

IEEE LTSC (Institute of Electrical and Electronics Engineers Learning Technology Standards Committee)

IMS Global Learning Consortium

Dublin Core Metadata Initiative

http://www.dublincore.org

DCMI Schemas (XML and RDF)

http://dublincore.org/schemas/

ANSI/NISO Z39.85-2001: The Dublin Core Metadata Element Set

http://www.niso.org/standards/resources/Z39-85.pdf

2.17. IEEE-LOM

top level categories

General

Life Cycle

Meta-Metadata

Technical

Educational

Rights

Relation

Annotation

Classification

2.18. IEEE-LOM evaluation

Built on an explicit data model

Specifies which aspects of a learning object should be described

International community contributes to standard

Education and learning sector in Europe is particularly invested in using the standard; is required in certain circumstances

Applicable to a broad array of learning objects

2.19. Uses of IEEE-LOM

Describe and share information about learning objects individually or as a group

Export as LOM in XML or RDF

Most descriptive elements mapped to Dublin Core

2.20. METS

Metadata Encoding and Transmission Standard (METS)a digital library standard for encoding descriptive, administrative, and structural metadata

2.21. METS structure

METS Header: Contains metadata describing the METS document itself

Descriptive Metadata: May point to descriptive metadata external to the METS document or contain internally embedded descriptive metadata, or both.

Administrative Metadata: Provides information regarding how the files were created and stored, intellectual property rights, etc.

File Section: Lists all files containing content which comprise the electronic versions of the digital object.

Structural Map: Outlines a hierarchical structure for the digital library object, and links the elements of that structure to content files and metadata that pertain to each element.

Structural Links: Records the existence of hyperlinks between nodes in the hierarchy outlined in the Structural Map.

Behavior: A behavior section can be used to associate executable behaviors with content in the METS object

2.22. METS metadata

METS

Header

Administrative

metadata

File

Inventory

Structure

map

Descriptive

metadata

Behavioral

metadata

optional

optional

optional

required

optional

optional

2.23. METS Descriptive Metadata

Metadata Reference: link to external descriptive metadata. type of link: included as an attribute

Metadata Wrapper: Included descriptive metadata, binary data or arbitrary XML using namespace metadata type: specified as an attribute.

2.24. METS Administrative Metadata

Technical Metadata: technical metadata regarding content files

IP Rights Metadata: rights metadata regarding content files or primary source material

Source Metadata: provenance information for content files.

Preservation Metadata: metadata to assist in preservation of digital content

2.25. METS File Inventory

File Group (fileGrp): hierarchically subdivids physical files (e.g. by type)

File : pointer to an external file (or includes file content internally)

2.26. METS Structural Map

provides a tree structure describing the original document

each division element is a node of the tree (identifies content files associated with that division by a METS Pointer or a File Pointer

XML extension "schemas"

descriptive metadata

Dublin Core, MARC, VRA...

administrative metadata

NISO image technical metadata

LC schemas for A/V technical metadata

Rights metadata

etc.

2.27. MODS

MODS : Metadata Object Description Schema

a schema for a bibliographic element set that may be used for different purposes, particularly for library applications. maintained by:

Network Development and MARC Standards Office

Library of Congress, USA

2.28. MODS

MARC21 compatible XML descriptive metadata standard

MARC21 derivative, simpler than MARC21 but compatible

MODS descriptive records can be contained inside METS record

richer than Dublin Core

Title Info

Name

Type of resource

Genre

Origin Info

Language

Physical description

Abstract

Table of contents

Target audience

Note

Subject

Classification

Related item

Identifier

Location

Access conditions

Part

Extension

Record Info

2.29. MODS evaluation

Element set is compatible with existing descriptions in large library databases

Element set is richer than Dublin Core but simpler than full MARC

Language tags are more user-friendly than MARC numeric tags

Hierarchy allows for rich description, especially of complex digital objects

Rich description that works well with hierarchical METS objects

2.30. MODS User Guidelines

http://www.loc.gov/standards/mods/v3/mods-userguide.html

[fragile]

2.31. element and attribute

portraits

Element

Attribute

Controlled vocabulary stipulated as the "authority"

Value (i.e. Controlled term)

Element

[fragile]

2.32. element and sub element

Element

no MPEG-3 standard today not(not to be confused with MP3 (MPEG-1 Audio Layer III)

MPEG-4: coding of audio-visual objects

MPEG-7: multimedia content description interface

MPEG-21: multimedia framework (iidentification, copyright, protection, etc.)

2.37. MPEG-7

content-based description for audiovisual information in multimedia environments

Content may include: still pictures, graphics, 3D models, audio, speech, video, and composition information about how these elements are combined (scenarios).

May cover facial expressions, personal characteristics, etc.

MPEG-7 Description tools allow to create descriptions of content that may include:

Information describing the creation and production processes of the content (director, title, short feature movie)

Information related to the usage of the content (copyright pointers, usage history, broadcast schedule)

Information of the storage features of the content (storage format, encoding)

Structural information on spatial, temporal or spatio-temporal components of the content (scene cuts, segmentation in regions, region motion tracking)

Information about low level features in the content (colors, textures, sound timbres, melody description)

Conceptual information of the reality captured by the content (objects and events, interactions among objects)

2.38. Application model

2.39. main elements of MPEG-7 standard

Descriptors (D): representations of Features, that define the syntax and the semantics of each feature representation,

Description Schemes (DS): specify the structure and semantics of the relationships between their components.(components can be both Descriptors and Description Schemes)

2.40. MPEG-7 DDL

DDL : Description Definition Language

defines the syntactic rules to express and combine Description Schemes and Descriptors

DDL is a schema language to represent the results of modeling audiovisual data, i.e. DSs and Ds.

DDL is an XML Schema application, with some specific extensions (such as array and matrix datatypes).DDL allows to define complexTypes and simpleTypes. The complexTypes specify the structural constraints while simpleTypes express datatype constraints.

2.41. MPEG-7 Visual

MPEG-7 Visual Description Tools:basic structures and Descriptors that cover the visual features: Color, Texture, Shape, Motion,Localization, Face recognition.Each category consists of elementary and sophisticated Descriptors

2.42. MPEG-7 Visual examples

Shape: the shape of an object may consist of either a single connected region or a set of disjoint regions, as well as some holes in the object

(Camera) motion: characterizes 3-D camera motion parameters. (can be automatically extracted or generated by capture devices

2.43. MPEG-7 Audio

MPEG-7 Audio Description Tools:

Audio Framework. The main hook into a description for all audio description schemes and descriptors

Spoken Content DS. A DS representing the output of Automatic Speech Recognition (ASR).

Timbre Description. A collection of descriptors describing the perceptual features of instrument sounds

Audio Independent Components. A DS containing an Independent Component Analysis (ICA) of audio

2.44. MPEG-7 and MPEG-21 community

http://www.multimedia-metadata.info/

3. 3 SEMANTIC CONTENT MANAGEMENT

3.1. Semantic content management/1

Semantic content management (RDF, RDF Schema, ontologies).

Semantic Web Layers.

Thesaurus.

Semantic web layers

3.2. RDF

is a W3C standard for describing Web resources

is a general method to decompose any type of knowledge into small pieces

method for expressing knowledge

the foundation of the Semantic Web

[fragile]

3.3. Why we need RDF?

Fekete PterFehr MriaNevenincsKft.300000

[fragile]

Fekete Pter

1304

Fehr Mria

NevenicsKft.

N3 (Notation3) language

primer - getting into the semantic web and rdf using n3.htm

[fragile]

3.4. URI

Uniform Resource Identifier

RDF:

qualified URI

(URI + optional detail description: #text)

[fragile]

3.5. RDF

A fact is expressed as a triple of the form

Subject, Predicate, Object

( a simple English sentence)

Subjects, predicates, and objects are names for entities, whether concrete or abstract, in the real world

.

Subject / Verb / Predicate / Property / Object

3.6. Subject, Predicate, Object

Subject, Predicate (Verb/Property) and an Object value forms a Statement

[fragile]

3.7. statement

.

RDF uses Web identifiers (URI: Web identifier, Uniform Resource Identifier)

Object can be a string: ,,34" .

Predicate expresses the other 2 elements relation:

. has . is of .

[fragile]

3.8. abbreviations

Semicolon (;)

>1 predicate to the same subject,

comma (,)

>1 object related to the same subject-predicate pair

, , ; ,,34" ; "blue" .

[fragile]

3.9. expressive strength

,,34"; "blue" . "3"; "green" . "5"; "green" .

[fragile]

[ "4"],[ "3"].

(Means: #pat has a #child which has #age of "4" and a #child which has an #age of "3" )

No identified object in this statement.

What you find in [ ] : refers to an existing object, but no URI for that.

More precisely: [ ] declares that there is something with that property, but you can't get right to refer to that.

If you need to do so:

[ "Pat"; "24"; "blue" ].[ "Al" ; "3"; "green" ].[ "Jo" ; "5"; "green" ].

[fragile]

"pat", "child", "age", e.t.c. are just characters, do not carry "meaning" to the computer!

until we do not state this (and is defined somewhere):

"Pat".

[fragile]

3.10. Common concepts

"title" (e.g. in web document, library catalogue) is just a concept. In case >1 resource (e.g. documents) will use that:

we need a a common understanding

we should use exactly the same "lexical item" to identify that concept

,,RDF basics for EIT Master Course".

( refers to the current document. In this example #title refers to a concept, what is defined by the document itself.)

3.11. Common concepts

Let's use Dublin Core (DC) definition!

http://purl.org/dc/elements/1.1/title "RDF basics for EIT Master Course".

make it shorter in N3:

@prefix dc: .

dc:title " RDF basics for EIT Master Course ".

[fragile]

Use reliable, widely accepted prefixes, e.g.:

@prefix rdf: .@prefix rdfs: .@prefix ont: .

[fragile]

Let

@prefix : .

Using this formalism the previous statement can be shorter:

:pat :child [ :age "4" ] , [ :age "3" ].

[fragile]

3.12. RDF vocabulary

is a defined set of predicates that can be used in an application.

dc:title = predicate

new vocabulary -> definition of classes and propeties

Class: what type is that something rdf:type

There is an abbreviation for that in N3: 'a'

Let's define a class of persons:

:Person a rdfs:Class.

We will use that in a document like this:

:Pat a :Person.

[fragile]

object can be in many classes (doesn't have to be any hierarchical relationship - but you can state this in RDF Schema, example. :Woman a rdfs:Class; rdfs:subClassOf :Person)

when a subject of any property must be in a class:domain (What can have this kind of property?)

when an object must be in a class range (What type of values can have)

:sister rdfs:domain :Person;rdfs:range :Woman.

Convention:

Class IDs begin with capitals

Properties lower case letters

http://tmit.bme.hu/index.html

mailto::[email protected]

http://vhol.org/DC/Creator





http://tmit.bme.hu/photo.html

http://vhol.org/schema/content

http://vhol.org/schema/together


3.13. Semantic Graph Data Model

RDF has an abstract syntax that reflects a simple graph-based data model.

Any statement in RDF is a collection of triples, each consisting of a subject, a predicate and an object

Triples together form RDF graphs, by a node and directed-arc diagram.

Nodes are subjects and objects; direction of the arc -> toward the object

A node may be a URI with optional fragment identifier, a literal, or blank. Properties are URI references

A URI reference or literal used as a node identifies what that node represents.

Simple logical statements are coded in the graph.

Directed arc ('P') from node 'A' to node 'B'

'P' predicate arc

"property of node 'A' is 'B'

Statement representation: (P,A,B) triple.

(where 'P' is a property, the predicate of the statement, 'A' is the subject, 'B' is the object.















OBJECT PREDICATE SUBJECT

"The creator of index.html is whois."









"The creator of index.html is whois and tome (works together). index.html contains photo.html"

3.14. RDF Schema

RDF is a data model that provides a way to express simple statements about resources, using named properties and values.

The RDF Vocabulary Description Language 1.0 (or RDF Schema or RDFS) is a language that can be used to define the vocabulary (i.e., the terms) to be used in an RDF graph.

The RDF Vocabulary Description Language 1.0 is used to indicate that we are describing specific kinds or classes of resources, and will use specific properties in describing those resources.

The RDF Vocabulary Description Language 1.0 is an ontology definition language (a simple one, compared with other languages such as OWL; we can only define taxonomies and do some basic inference about them).

The RDF Vocabulary Description Language is like a schema definition language in the relational or object-oriented data models (hence the alternative name RDF Schema - we will use this name and its shorthand RDFS mostly!).

[fragile]

3.15. RDF Schema

The RDF Schema concepts are themselves provided in the form of an RDF vocabulary; that is, as a specialized set of predefined RDF resources with their own special meanings.

The resources in the RDF Schema vocabulary have URIrefs with the prefix http://www.w3.org/2000/01/rdf-schema (associated with the QName prefix rdfs: ).

Vocabulary descriptions (schemas, ontologies) written in the RDF Schema language are legal RDF graphs. In other words, we use RDF to represent RDFS information.

3.16. Classes

A basic step in any kind of description process is identifying the various kinds of things to be described. RDF Schema refers to these "kinds of things" as classes.

A class in RDF Schema corresponds to the generic concept of a type or category, somewhat like the notion of a class in object-oriented programming languages such as Java or object-oriented data models.

[fragile]

3.17. Defining Classes

Suppose an organization example.org wants to use RDF Schema to provide information about motor vehicles, vans and trucks.

To define classes that represent these categories of vehicles, we write the following statements (triples):

ex:MotorVehicle rdf:type rdfs:Class .ex:Van rdf:type rdfs:Class .ex:Truck rdf:type rdfs:Class .

In RDFS, a class C is defined by a triple of the form

C rdf:type rdfs:Class .

using the predefined class rdfs:Class and the predefined property rdf:type.

[fragile]

3.18. Defining Instances

Now suppose example.org wants to define an individual car (e.g., the company car) and say that it is a motor vehicle.

This can be done with the following RDF statement:

exthings:companyCar rdf:type ex:MotorVehicle .

[fragile]

3.19. rdf:type

The predefined property rdf:type is used as a predicate in a statement

I rdf:type C

to declare that individual I is an instance of class C.

In statements of the form

C rdf:type rdfs:Class .

rdf:type

is used to declare that class C (viewed as an individual object) is an instance of the predefined class rdfs:Class.

[fragile]

3.20. Defining Classes (cont'd)

Defining a class explicitly is optional; if we write the triple

I rdf:type C

then C is inferred to be a class (an instance of rdfs:Class) in RDFS.

3.21. Notation

Class names will be written with an initial uppercase letter.

Property and instance names are written with an initial lowercase letter.

[fragile]

3.22. Defining Subclasses

Now suppose example.org wants to define that vans and trucks are specialized kinds of motor vehicle.

This can be done with the following RDF statements:

ex:Van rdfs:subClassOf ex:MotorVehicle .ex:Truck rdfs:subClassOf ex:MotorVehicle .

The predefined property rdfs:subclassOf is used as a predicate in a statement to declare that a class is a specialization of another more general class.

A class can be a specialization of multiple superclasses (e.g., the graph defined by the rdfs:subclassOf property is a directed graph not a tree).

[fragile]

3.23. Classes and Instances

The meaning of the predefined property rdfs:subClassOf in a statement of the form

C1 rdfs:subClassOf C2

is that any instance of class C1 is also aninstance of class C2.

Example: If we have the statements

ex:Van rdfs:subClassOf ex:MotorVehicle .exthings:myCar rdf:type ex:Van .

then RDFS allows us to infer the statement

exthings:myCar rdf:type ex:MotorVehicle .

[fragile]

3.24. Properties of rdfs:subClassOf

The rdfs:subClassOf property is reflexive and transitive.

Examples:

If we have a class ex:MotorVehicle then RDFS allows us to infer the statement

ex:MotorVehicle rdfs:subClassOf ex:MotorVehicle .

If we have the statements

ex:Van rdfs:subClassOf ex:MotorVehicle .ex:MiniVan rdfs:subClassOf ex:Van .

then RDFS allows us to infer the statement

ex:MiniVan rdfs:subClassOf ex:MotorVehicle .

[fragile]

3.25. RDF Schema Predefined Classes

The group of resources that are RDF Schema classes is itself a class called rdfs:Class. All classes are instances of this class.

In the literature, classes such as rdfs:Class that have other classes as instances are called meta-classes.

All things described by RDF are called resources, and are instances of the class rdfs:Resource.

rdfs:Resource is the class of everything. All other classes are subclasses of this class. For example, rdfs:Class is a subclass of rdfs:Resource.

[fragile]

3.26. Properties

In addition to defining the specific classes of things they want to describe, user communities also need to be able to define specific properties that characterize those classes of things (such as author to describe a book).

[fragile]

3.27. Defining Properties

A property can be defined by stating that it is an instance of the predefined class

rdf:Property.

Example:

ex:author rdf:type rdf:Property .

Then, property ex:author can be used as a predicate in an RDF triple such as the following:

ex:john ex:author ex:book123 .

[fragile]

3.28. Defining Properties

Defining a property explicitly is optional; if we write the RDF triple

S P O .

then P is inferred to be a property by RDFS.

[fragile]

Properties are resources too (this makes RDF and RDFS different than many other KR formalisms).

Therefore, properties can appear as subjects or objects of triples.

Example (provenance):

ex:author prov:definedBy ke:johnke:john prov:defined ex:author

We will see many more examples like the above.

In RDFS property definitions are independent of class definitions. In other words, a property definition can be made without any reference to a class.

Optionally, properties can be declared to apply to certain instances of classes by defining their domain and range.

[fragile]

3.29. Domain and Range

RDFS provides vocabulary for describing how properties and classes are intended to be used together in RDF data.

The rdfs:domain predicate can be used to indicate that a particular property applies to instances of a designated class (i.e., it defines the domain of the property).

The rdfs:range predicate is used to indicate that the values of a particular property are instances of a designated class (i.e., it defines the range of the property).

[fragile]

3.30. Datatypes for Ranges

The rdfs:range property can also be used to indicate that the value of a property is given by a typed literal.

Example:

ex:age rdf:type rdf:Property .ex:age rdfs:range xsd:integer .

Optionally, we can also assert that xsd:integer is a datatype as follows:

xsd:integer rdf:type rdfs:Datatype .

[fragile]

3.31. The Property rdfs:label

rdfs:label is an instance of rdf:Property that may be used to provide a human-readable version of a resource's name.

The rdfs:domain of rdfs:label is rdfs:Resource. The rdfs:range of rdfs:label is rdfs:Literal.

Multilingual labels are supported using the language tagging facility of RDF literals.

[fragile]

3.32. The Property rdfs:comment

rdfs:comment is an instance of rdf:Property that may be used to provide a human-readable description of a resource.

The rdfs:domain of rdfs:comment is rdfs:Resource. The rdfs:range of rdfs:comment is rdfs:Literal.

Multilingual documentation is supported through use of the language tagging facility of RDF literals.

[fragile]

3.33. The Property rdfs:seeAlso

rdfs:seeAlso is an instance of rdf:Property that is used to indicate a resource that might provide additional information about the subject resource.

A triple of the form S rdfs:seeAlso O states that the resource O may provide additional information about S. It may be possible to retrieve representations of O from the Web, but this is not required. When such representations may be retrieved, no constraints are placed on the format of those representations.

The rdfs:domain of rdfs:seeAlso is rdfs:Resource. The rdfs:range of rdfs:seeAlso is rdfs:Resource.

[fragile]

3.34. RDFS vs. Types (cont'd)

Benefits of the RDF approach: One can start with a property definition and then extend it to other uses that might not have been anticipated.

Shortcoming: In RDFS, it is not possible to say, for example, that if the property ex:hasParent is used to describe a resource of class ex:Human, then the range of the property is also a resource of class ex:Human, while if the property is used to describe a resource of class ex:Tiger, then the range of the property is also a resource of class ex:Tiger. This can be done in ontology languages that we will define later in the course.

3.35. Schema Languages

RDF Schema provides basic capabilities for describing RDF vocabularies, but additional capabilities are also possible, and can be useful.

These capabilities may be provided through further development of RDF Schema, or in other languages (for example, ontology languages such as OWL).

3.36. The Semantic Web "Layer Cake"

Semantic web layers - overview

3.37. URI, Unicode layer

URI and Unicode layer: ensures to identify objects using international character set IDs

unambiguous identification is essential to consistent statements

Typical URIsProtocol-dependent (http://, mailto:)Protocol/position independent (URN-s, PURL)"Identifiers and character set"

3.38. XML layer

To organize the resources (e.g. documents)

into structures

nothing is about meaning

"syntax"

3.39. RDF layer

"meaning layer":

triples,

all triples: subject, predicate, object

"data interchange"

3.40. RDF Scheme

Why we need scheme

in RDF no data types

Users should agree in given lexical units (used in documents)

RDF Schemas: what relations are accepted? to what resources are valid those relations?

RDFS provides basic vocabulary for RDF. RDFS: create hierarchies of classes and properties

"taxonomies"

3.41. Ontology layer

RDF schemas gives the "basics" (no inferences about concepts)"if A is true, than B is true""if C is true, than D is not true"

Ontology layer extends RDFS by adding more advanced constructs to describe semantics of RDF statements.Allows stating additional constraints (e.g. cardinality, restrictions of values, or characteristics of properties)Based on description logic. Brings reasoning power to the semantic web.

3.42. Controlled vocabularies

Vocabularies and other language constructions of native or artificial languages to support information management

Term list, free phrase list

Classification (categorization)

Relationship Schemes

Term list

Glossaries, Dictionaries, Authority Files

Classification

Subject Classification

Taxonomy

Relationship Schemes

Thesaurus

Semantic Network (e.g. WordNet)

Ontology

3.43. Thesaurus

A structured wordlist used to standardise terminology(more than a standard wordlist - helps with indexing and acts as a guide to a subject area)

Reasoning over thesaurus relationships:

automatic suggestion of terms to be considered for query

query reformulation ('like this' option)

semantic expansion of queries

3.44. Thesaurus standards

ISO 2788:1986 Documentation -Guidelines for the establishment and development of monolingual thesauri

ISO 5964:1985 Documentation -Guidelines for the establishment and development of multilingual thesauri

4 basic relationships in a thesaurus

Equivalence (preferred and non-preferred terms)

Hierarchy (narrower (more specific) / broader (more general) term)

Association (establishes non-hierarchical relationships, e.g. related term)

Scope notes (provide guidance and clarification)

DATABASE

M: Data stored electronically, available for search, distribution

F Technical document

X CD ROM

DATA STORAGE

M: A storage medium on which certain physical aspects describe data

A Celluloid film

Vinyl

Sound tape

Video disc

Magnetic storage

Optical storage

Sheet of paper

T Document

THESIS

M: A certification of a qualification or obtained title

F Certificate

H Diploma Thesis

L Bachelor's Thesis

DISSERTATION

M: A written work to acquire an academic title in a higher education program

H Doctoral Thesis

F Study

X Diploma Thesis

Thesis

3.45. Widely known thesauri

The Art and Architecture Thesaurus, Getty Institutehttp://www.getty.edu/research/conducting_research/vocabularies/aat/

Union List of Artist's Names, Getty Institutehttp://www.getty.edu/research/conducting_research/vocabularies/ulan/

British Museum Object Names Thesaurus http://www.mda.org.uk/bmobj/Objintro.htm

NASA Thesaurushttp://www.sti.nasa.gov/thesfrm1.htm

USA Library of Congresshttp://www.loc.gov/lexico/servlet/lexico/

4. 4 CMS types

4.1. CMS types

DAM: Digital Asset Management

DM: Document Management

LMS: Learning Management System

WCMS: Web CMS

KM: Knowledge Management

ECM: Enterprise Content Management

4.2. DAM: Digital Asset Management

Task of DAM is to handle, store, retrieve, annotate, distribute of large amounts of digital assets.

broad categories of DAM based on purpose of management handling

subtypes of DAM based on content types

4.3. Broad categories of DAM

Brand asset management systems focus on marketing- or sales-related content re-use within large organizations (e.g. product imagery, logos, etc.).

Library asset management systems handle with storage and retrieval of large amounts of infrequently changing media assets (e.g. text, video or photo archiving).

Production asset management systems focus on managing assets as they are being created for a digital media production (video game, music, animation, etc.).

4.4. Subtypes of DAM

Gereral DAM

Media Asset Management (MAM) (see also DMMS - Digital Media Management Systems), e.g. Convera, Virage, Cinebase. This contains high levels service to assist in creating, managing, distributing and using rich media contents / assets (audio, video).

4.5. DAM file types

Images

Audio

Video

Logos and line art

Animation like Flash

CAD

3-D

HTML

4.6. Application (areas) of DAM

Multimedia press Kits

Multimedia sales

Video on demand (VOD)

Rich Media libraries

Advertising and marketing

Tutorials (multimedia)

Film production

TV content production

4.7. DM: Document Management

The task of DM is to store and track of the (different versions of) electronic documents.

Document management systems commonly provide storage, versioning, metadata, security, as well as indexing and retrieval capabilities.

4.8. Components of DM

Metadata, Integration, Capture, Indexing, Storage, Retrieval, Distribution, Security, Workflow, Collaboration, Versioning, Searching, Publishing, Reproduction

4.9. Capture

Capture primarily involves accepting and processing images of paper documents from scanners.

Optical character recognition (OCR) software is often used in order to convert digital images into machine readable text.

4.10. Functionalities in DM

Integration with text processing tools (Word, Adobe).

Definition of content components within a document.

Assemble documents or document segments for reuse.

Modifying documents.

Storing of text effectively, often in a variety of different formats, including native formats and XML.

Provide text specific search features such as natural language search.

Apply metadata at varying levels within the document.

4.11. DM file types

Paper

File output from office automation tools (e.g. PPT)

PDF

Text output

Imaged documents

XML documents and fragments

Other text files, including HTML

Other images and multimedia files, typically as generic binaries

4.12. Application (areas) of DM

Contracts

Documentation/Manuals

Policy and Procedures

Forms

Research

Applications for regulatory product approval

Statements (program codes)

Articles

Reports

4.13. LMS: Learning Management System

is a software application for the

administration,

documentation,

tracking,

reporting and

delivery

of education courses or training programs.

4.14. Functionalities in LMS

centralize and automate administration

assemble and deliver learning contents

personalize content and enable knowledge reuse

deliver online training

support portability and standards

4.15. LMS standards

AICC

SCORM: Sharable Content Object Reference Model is a specification of the Advanced Distributed Learning (ADL) Initiative

4.16. Most important LMS softwares

Blackboard Learning System (leading provider), WebCT (acquired Blackboard in 2005)

Moodle (Open source)

4.17. Web CMS

is a software system that provides

website authoring,

collaboration,

administration tools

to create and manage website content with relative ease.

4.18. Functionalities in WCMS

Automated templates

Access control

Easily editable content

Web standards upgrades

Workflow management

Collaboration

Content syndication

Multilingual

Versioning

4.19. Most important WCMS softwares

WordPress originated as a blogging CMS, but later evolved into a fully CMS.

Joomla! can be used to easily create and edit webpages.

Drupal is more difficult to learn and understand than the above two CMSs, but is the most secure.

4.20. WCMS file types

Large variety in types

HTML,

XML,

PDF,

JPG, GIF, PNG, BMP,

WAV, MP3,

MPEG2, MP4

4.21. KM: Knowledge Management

is based on a range of practices used by an individual or organization to

identify,

create,

represent and

redistribute

knowledge.

4.22. Knowledge Management software

collects and contains information to then build knowledge that can be searched through specialised search tools including

concept building tools and or

visual search tools.

4.23. ECM: Enterprise Content Management

organizes and stores (enterprise's) documents, and other content, that relate to the organization's processes.

This deals with strategies, methods, and tools used throughout the lifecycle of the content.

4.24. Components of ECM

capture

manage

store

preserve

deliver

4.25. Capture component

Recognition technologies, like OCR, Optical mark recognition (OMR such as checkmarks or dots), Barcode recognition.

Aggregation combines documents from different applications.

Indexing (automatic or manual)

4.26. Manage component

Document management

Collaboration (or collaborative software, a.k.a. groupware)

Web content management (including web portals)

Records management

Workflow and business process management (BPM)

4.27. Store component

Repositories as storage locations,

Library Services as administration components for repositories, and

Storage technologies.

4.28. Preserve component

involves the long-term, safe storage and backup of static, unchanging information.

Long term storage media (Write once read many: WORM technology)

Long term preservation strategies with conversion and migration tools (migration of applications, emulation of older software)

4.29. Deliver component

Methods

On-premise as a traditional software application that companies implemented on their own networks.

Software as a service (SaaS) users access the application and their data online. It is also known as cloud computing, hosted, and on demand.

Hybrid

Transformation technologies (Personalization, Syndication)

Security technologies

Distribution

4.30. Output and distribution media

E-mail

Fax

Data transfer by EDI, XML or other formats

Mobile devices, like mobile phones, PDAs

Data media like CDs and DVDs

Digital TV and other multimedia services

4.31. Most important ECM systems

Autonomy

EMC Documentum

IBM/FileNet

Microsoft

Open Text/Hummingbird

Oracle/Stellent

4.32. Axes of Magic Quadrant

Vision

market understanding

marketing strategy

sales strategy

product strategy

business model

industry strategy

innovation

geographic strategy

Ability

product, service

overall viability

pricing

market responsiveness

marketing execution

customer experience

operations

4.33. CMS plan, design, integration

Plan of CMS functionalities

Analyzing the content lifecycle and designing content components

Roles: different authors

Integration

4.34. Plan of functionalities (1)

Document Capture: Includes import and transformation.

Authoring: Facilities for content creation.

Metadata Tagging: Describing the content.

Editing: Changing and updating the content.

Indexing: indexes of document and content items.

Searching: search capabilities both at document and content levels.

Viewing: content is viewed correctly in all devices.

Collaboration: allow users to collaborate.

Workflow: routing of content between individuals and processes.

Database Integration.

Security: Prevent unauthorized modification.


Version Control: Enables the tracking of changes to documents and content, providing an audit trail, and the possibility of rollback to previous versions.

Scheduling: Deciding when to display content.

Templating: Allows separation of content from its presentation.

Syndication: Allowing content to be displayed by others.

Personalisation: Displaying content differently dependent on the visitor.

Filtering: Ability to filter out content for publication or filter out specific users content retrieval.

Categorisation: Break the content of documents into components, categorise them and store the relationships.

Application Integration: Integration into existing infrastructure, suppliers, customers, partners (B2B) and third party product.


Storage: handling a wide variety of content.

Records Management: store and manage all records including physical records.

Content Aggregation: different pieces of content to put together .

Content Delivery: delivering content to users in the most efficient manner (caching).

Multiple Web Site Management (synchronisation, localisation, branding, and content delivery).

Platform Viewing Independence: Enables content to easily be reused and delivered to different channels (Internet, Mobile Devices and TV).

4.37. Analyzing the content lifecycle

Product content

The evolution of reuse in publications

The role of content strategy

Identifying content lifecycle

Identifying the players, processes, and issues

4.38. Roles

Content strategist

Content owners

External, internal authors

Business owners

Editors

Information architect

Publishing roles

4.39. CMS adopting, integration

CMS beta testing

CMS adopted as a solution

Formal plan developed to migrate content into the system

4.40. Content Migration

Schedule for the migration of current content into the CMS system

Migration occurred in small groups

Thousands of pages, images, and resource records were copied

4.41. Two approaches

Integration of CMSs - This approach enables an organisation to capitalise on the strengths of different providers rather than looking to one platform to manage all unstructured information.

Full ECM Suite (the acquisition of full ECM suites from a single vendor - This approach enables one platform to fulfill an organisation's enterprise wide needs.

5. 5 Information Retrieval (IR)

5.1. Text Databases and IR

Text databases (document databases)

Large collections of documents from various sources: news articles, research papers, books, digital libraries, e-mail messages, and Web pages, library database, etc.

Data stored is usually semi-structured

Traditional search techniques become inadequate for the increasingly vast amounts of text data

Information retrieval

A field developed in parallel with database systems

Information is organized into (a large number of) documents

Information retrieval problem: locating relevant documents based on user input, such as keywords or example documents

5.2. Information Retrieval (IR)

Typical IR systems

Online library catalogs

Online document management systems

Information retrieval vs. database systems

Some DB problems are not present in IR, e.g., update, transaction management, complex objects

Some IR problems are not addressed well in DBMS, e.g., unstructured documents, approximate search using keywords and relevance

5.3. IR vs DBMS

5.4. Basic Measures in the IR

Precision: the percentage of retrieved documents that are in fact relevant to the query (i.e., "correct" responses)

Recall: the percentage of documents that are relevant to the query and were, in fact, retrieved

5.5. Keyword-Based Retrieval

A document is represented by a string, which can be identified by a set of keywords

Queries may use expressions of keywords

E.g., car and repair shop, tea or coffee, DBMS but not Oracle

Queries and retrieval can consider synonyms, e.g., repair and maintenance

Major difficulties of the model

Synonymy: A keyword T does not appear anywhere in the document, even though the document is closely related to T, e.g., data mining.

Polysemy: The same keyword may mean different things in different contexts, e.g., mining.

5.6. Similarity-Based Retrieval in Text Databases

Finds similar documents based on a set of common keywords

Answer should be based on the degree of relevance based on the nearness of the keywords, relative frequency of the keywords, etc.

Stop list

Set of words that are deemed "irrelevant", even though they may appear frequently

E.g., a, the, of, for, with, etc.

Stop lists may be different at different corpus

Word stem

Several words are small syntactic variants of each other since they share a common word stem

E.g., drug, drugs, drugged

5.7. Retrieval : Ad hoc and Filtering

Ad hoc (Search): The documents in the collection remain relatively static while new queries are submitted to the system.

Filtering: The queries remain relatively static while new documents come into the system

5.8. architecture

user

Query matching

Learning component

Object base

(objects and their descriptions)

hits

query

feedback

5.9. IR System and Tasks Involved

INDEX

INFORMATION NEED

DOCUMENTS

User Interface

PERFORMANCE EVALUATION

QUERY

QUERY PROCESSING (PARSING and TERM PROCESSING)

LOGICAL VIEW OF THE INFORM. NEED

SELECT DATA FOR INDEXING

PARSING and TERM PROCESSING

5.10. Models for IR - Taxonomy

5.11. Formal Specification of the Task

Definition: An information retrieval model is a quadrupel where

is a set composed of logical views (or representations) for the documents in the set

is a set composed of logical views for the user information needs: queries.

is a framework for modeling document representations, queries, and their relationships.

is a ranking function which associates a real number with a query in and a document representation in . Such ranking defines an ordering among the documents with regard to the query .

5.12. Formal Specific. of the Task (Cont.)

Generally, we represent the query and documents through a set of terms where is the number of all unique index terms in the system.

We assume to be a weight for term in document with if is not in .

Document can be represented as an index term vector .

represents a function for which (i.e. given a document delivers the weight of term in ).

5.13. Boolean Retrieval Model - Queries

Based on set theory and Boolean algebra

Documents:

Index term vector with

Queries:

Terms combined with AND, OR, NOT

Boolean expression in disjunctive normal form (DNF)

A query is defined as a Boolean expression in DNF with being the conjunctive elements from .

or are the index term weight variables.

5.14. Boolean Retr. Model - Definition

We define the similarity sim of a document with query as

(A document is considered relevant if and irrelevant otherwise)

5.15. Boolean Retrieval Model

Advantages:

Precise, clean formalism

Offers great control and transparency,

Simplicity, easy math, easy implementation

Good for domains with ranking by other

means than relevance, i.e. chronological

Disadvantages:

Query might be hard to specify

Binary decision (relevant or not)

Often too many or too few results

5.16. Vector Model - Definition

Based on vector algebra

Formal Definition:

is defined as the weight associated with the pair and or

describes the number of all unique index terms

With this, we can define

Query vector

Document vector

5.17. Vector Model - Similarity

We should define similarity.

Using the inner product (arithmetical):

Using the cosinus of the angle between the 2 vectors

Weights: Often (or variants of it)

5.18. Vector Model

Advantages:

Fast and easy,

Finds similar documents (no binary decision),

Ranking based on similarity

Often better results than Boolean search

(because of the term weighting)

Disadvantages:

Terms are assumed to be independent

5.19. Weights

How are the weights obtained? Many variants.

One way: TF-IDF balance

TF: Term frequency

How well the term is related to the doc?

IDF: Inverse document frequency

How important is the term to distinguish documents?

Contradictory. How to balance?

5.20. TF-IDF ranking

TF: Term frequency

IDF: Inverse documentfrequency

Balance:

5.21. Classic Probabilistic Model (first)

Also called binary independence retrieval (BIR) model

Idea: Given a user query , and the ideal answer set of the relevant documents, the problem is to specify the properties for this set.

i.e. the probabilistic model tries to estimate the probability that the user will find the document relevant with ratio relevant to nonrelevant to )

5.22. Classic Probabilistic Model

An initial set of documents is retrieved somehow

User inspects these docs looking for the relevant ones (in truth, only top 10-20 need to be inspected)

IR system uses this information to refine description of ideal answer set

By repeting this process, it is expected that the description of the ideal answer set will improve


Definition

All index term weights are all binary i.e.,

Let be the set of documents know to be relevant to query

Let be the complement of

Let be the probability that the document is relevant to the query

Let be the probability that the document is nonelevant to query


The similarity of the document to the query isdefined as the ratio

Using Bayes' rule,

stands for the probability that a document randomly selected from the entire collection is relevant

stands for the probability of randomly selecting the document from the set of relevant documents


Assuming independence of index terms and given ,


stands for the probability that the index term is present in a document randomly selected from the set

stands for the probability that the index term is not present in a document randomly selected from the set


5.28. Estimation of Term Relevance

In the very beginning:

Next, the ranking can be improved as follows:

For small values for V

Let V be a subset of the documents initially retrieved

5.29. (Dis) advantages of Classic Probabilistic model (last)

Advantage:

Theoretical adequacy: ranks by probabilities

Disadvantages:

Need to guess the initial ranking

Binary weights, ignores frequencies

Independence assumption

5.30. Summary: taxonomy of IR models

Retrieval:

Ad hoc

Filtering

Classic Models

Browsing

Boolean

Vector

Probabilistic

Fuzzy

Extended Boolean

Set Theoretic

Algebraic

Generalized Vector

Lat. Semantic Index

Neural Networks

Inference Network

Belief Network

Probabilistic

5.31. Alternative Probabilistic Models

Inference Network Model

Belief Network Model

5.32. Bayesian Network

Let be a node in a Bayesian network and be the set of parent nodes of .

The influence of on can be specified by any set of functions that satisfy:

5.33. Belief Network Model (1/6)

The probability spaceThe set is the universe. To each subset is associated a vector such that .

Random variables

To each index term is associated a binary random variable.


Concept space

A document is represented as a concept composed of the terms used to index .

A user query is also represented as a concept composed of the terms used to index .

Both user query and document are modeled as subsets of index terms.

Probability distribution over


A query is modeled as a network node

This variable is set to 1 whenever completely covers the concept space

computes the degree of coverage of the space by

A document is modeled as a network node

This random variable is 1 to indicate that completely covers the concept space

computes the degree of coverage of the space by



Assumption

is adopted as the rank of the document with respect to the query .


Specify the conditional probabilities as follows:

Thus, the belief network model can be tuned to subsume the vector model.

5.39. Summary

Belief network model

Belief network model is based on set-theoretic view

Belief network model provides a separation between the document and the query

Belief network model is able to reproduce any ranking strategy generated by the inference network model

5.40. Alternative Set Theoretic models

Extended Boolean model

Combination of Boolean and Vector

In comparison with Boolean model, adds "distance from query"

some documents satisfy the query better than others

In comparison with Vector model, adds the distinction between AND and OR combinations

There is a parameter (degree of norm) allowing to adjust the behavior between Boolean-like and Vector-like

This can be even different within one query

Not investigated well. Why not investigate it?

5.41. Extended Boolean logic

considering the space composed of two terms kx and ky only.

ky

ky

kx

kx

5.42. Extended Boolean Model:

For query or is the point we try to avoid.Thus, we can use

to rank the documents

The bigger the better.

5.43. Extended Boolean Model:

For query and is the most desirable point.

We use

to rank the documents.

The bigger the better.

5.44. Extend the idea to m terms

5.45. Properties:

The norm as defined above enjoys a couple of interesting properties as follows. First, when it can be verified that

Second, when it can be verified that

5.46. Example:

For instance, consider the query . The similarity between a document and this query is then computed as

Any boolean can be expressed as a numeral formula.

6. 6 Information Retrieval on Web

6.1. Mining the World-Wide Web

The WWW is huge, widely distributed, global information service center for

Information services

Hyper-link information

Access and usage information

WWW provides rich sources for data mining

Challenges

Too huge for effective data warehousing and data mining

Too complex and heterogeneous: no standards and structure

Only a small portion of the information on the Web is truly relevant or useful

6.2. Web search engines

Index-based: search the Web, index Web pages, and build and store huge keyword-based indices

Help locate sets of Web pages containing certain keywords

Deficiencies

A topic may easily contain thousands of documents

Many documents that are highly relevant to a topic may not contain keywords defining them (polysemy)

6.3. Web Mining: A more challenging task

Searches for

Web access patterns

Web structures

Regularity and dynamics of Web contents

Problems

The "abundance" problem

Limited coverage of the Web: hidden Web sources, majority of data in DBMS

Limited query interface based on keyword-oriented search

Limited customization to individual users

6.4. Web Mining Taxonomy

Mining Mining Content Mining Mining Mining Pattern Tracking Usage Tracking


Web Structure

Mining

Web Content

Mining

Web Page Content Mining

Web Page Summarization

WebLog (Lakshmanan et.al.), WebOQL(Mendelzon et.al.) :

Web Structuring query languages;

Can identify information within given web pages

Ahoy! (Etzioni et.al.):Uses heuristics to distinguish personal home pages from other web pages

ShopBot (Etzioni et.al.): Looks for product prices within web pages

Search Result

Mining

Web Usage

Mining

General Access

Pattern Tracking

Customized

Usage Tracking


Web Usage

Mining

General Access

Pattern Tracking

Customized

Usage Tracking

Web Structure

Mining

Web Content

Mining

Web Page

Content Mining

Search Result Mining

Search Engine Result Summarization

Clustering Search Result (Leouski and Croft" Zamir and Etzioni,):

Categorizes documents using phrases in titles and snippets


Web Content

Mining

Web Page

Content Mining

Search Result

Mining

Web Usage

Mining

General Access

Pattern Tracking

Customized

Usage Tracking

Web Structure Mining

Using Links

PageRank (Brin et al.)

CLEVER (Chakrabarti et al.,)

Use interconnections between web pages to give weight to pages.

Using Generalization

MLDB , VWV

Uses a multi-level database representation of the Web. Counters (popularity) and link lists are used for capturing structure.


Web Structure

Mining

Web Content

Mining

Web Page

Content Mining

Search Result

Mining

Web Usage

Mining

General Access Pattern Tracking

Web Log Mining (Zaane, Xin and Han)

Uses KDD techniques to understand general access patterns and trends.

Can shed light on better structure and grouping of resource providers.

Customized

Usage Tracking


Web Usage

Mining

General Access

Pattern Tracking

Customized Usage Tracking

Adaptive Sites (Perkowitz and Etzioni)

Analyzes access patterns of each user at a time.

Web site restructures itself automatically by learning from user access patterns.

Web Structure

Mining

Web Content

Mining

Web Page

Content Mining

Search Result

Mining


Design of a Web Log Miner

Web log is filtered to generate a relational database

A data cube is generated form database

OLAP is used to drill-down and roll-up in the cube

OLAM is used for mining interesting knowledge

Data Cleaning

Data Cube Creation

OLAP

Data Mining

Web log

Database

Data Cube

Sliced and diced cube

Knowledge

6.11. Multilayered Web Information Base

: the Web itself

: the Web page descriptor layer

Contains descriptive information for pages on the Web

An abstraction of : substantially smaller but still rich enough to preserve most of the interesting, general information

Organized into dozens of semistructured classes

document, person, organization, ads, directory, sales, software, game, stocks, , , , etc.

and up: various Web directory services constructed on top of

provide multidimensional, application-specific services

6.12. Multiple Layered Web Architecture

Generalized Descriptions

More Generalized Descriptions


Layer-0: Primitive data

Layer-1: dozen database relations representing types of objects (metadata)

document, organization, person, software, game, map, image,

document (, authors, title, publication, , abstract, language, , , keywords, index, , , format, , , timestamp, , , )

person (, , , position, , phone, e-mail, , education, , publications, , timestamp, , )

image (, author, title, , , keywords, size, width, height, duration, format, , , , , , , timestamp, , )


Layer-2: simplification of layer-1

(, authors, title, publication, , abstract, language, , , , , format, , , )

(, , publications,affiliation, e-mail, , , )

Layer-3: generalization of layer-2

(, authors, title, publication, , abstract, language, , keywords, , form, , )

(affiliation, field, , count, , )

(, authors, affiliation, title, publication, , , keywords, , format, , )

(affiliation, , year, , count)

6.15. Benefits of Multi-Layer Meta-Web

Benefits:

Multi-dimensional Web info summary analysis

Approximate and intelligent query answering

Web high-level query answering (WebSQL, WebML)

Web content and structure mining

Observing the dynamics/evolution of the Web

Is it realistic to construct such a meta-Web?

Benefits even if it is partially constructed

Benefits may justify the cost of tool development, standardization and partial restructuring

6.16. Web Search Topics

Web Search

Architecture

Crawling

Search engines

Link analysis

6.17. Search on the Web

Corpus:The publicly accessible Web: static + dynamic

Goal: Retrieve high quality results relevant to the user's need

(not docs!)

Need

Informational - want to learn about something ()

Navigational - want to go to that page ()

Transactional - want to do something (web-mediated) ()

Access a service

Downloads

Shop

Grey areas

Find a good hub

Exploratory search "see what's there"

Bank information systems

National Airlines

Weather forecast

BUX data

Nikon CoolPix

Bank credits

6.18. Classic IR vs. Web Search

Classic IR: Generally a 2-step process

Select subset of relevant documents

Rank / sort by estimated relevance

Problems with web search:

Size of the WWW

Heterogeneity (length, quality, format, )

Not always self descriptive (e.g. search engine)

Can be manipulated (spam)

Ill-defined queries

Therefore: Pure content based ranking critical

6.19. General Web Search Engine Architecture

CLIENT

QUERY ENGINE

RANKING

CRAWL CONTROL

CRAWLER(S)

USAGE FEEDBACK

RESULTS

QUERIES

WWW

COLLECTION ANALYSIS MOD.

INDEXER MODULE

PAGE REPOSITORY

INDEXES

STRUCTURE

UTILITY

TEXT

6.20. The Indexer Module

Creates Two indexes :

Text (content) index : Uses "Traditional" indexing methods like Inverted Indexing

Structure(Links) index : Uses a directed graph of pages and links. Sometimes also creates an inverted graph

6.21. Storage Issues

Scalability and seamless load distribution

Dual access modes

Random access (used by the query engine for cached pages)

Streaming access (used by the Indexer and Collection Analysis)

Large bulk update - reclaim old space, avoid access/update conflicts

Obsolete pages - remove pages no longer on the web

6.22. Update Strategies

Updates are generated by the crawler

Several characteristics

Time in which the crawl occurs and the repository receives information

Whether the crawl's information replaces the entire database or modifies parts of it

6.23. Batch vs. Steady

Batch mode

Periodically executed

Allocated a certain amount of time

Steady mode

Run all the time

Always send results back to the repository

6.24. Partial vs. Complete Crawls

A batch mode crawler can

Do a complete crawl every run, and replace entire collection

Recrawl only a specific subset, and apply updates to the existing collection - partial crawl

The repository can implement

In place update

Quickly refresh pages

Shadowing, update as another stage

Avoid refresh-access conflicts

6.25. Generation Web Search Engines

Use only "on page" text data

1995-1997 (AltaVista, Excite, Lycos, etc.)

Extended Boolean model

Matches: Exact, prefix, phrase,

Operators: AND, OR, AND NOT, NEAR,

Fields: TITLE, URL, HOST,

AND is somewhat easier to implement, maybe preferable as default for short queries

Ranking

TF like factors: TF, explicit keywords, words in the title, explicit emphasis (headers), etc.

IDF factors: IDF, total word count in corpus, frequency in query log, frequency in language

6.26. The evolu