September 2003ERPANET - Marburg 2003View # 1 Data preservation Using CCSDS and ISO Standards Denis MINGUILLON [email protected]

September 2003 ERPANET - Marburg 2003 View # 1

Data preservationUsing CCSDS and ISO Standards

Denis [email protected]


The problem to solve

• Data is often expensive to produce– Bits must not be lost (preserved medium)– Bits must remain accessible (known syntax)– Bits must remain understandable (known

semantics)

• This presentation focuses on point 2 and 3


The syntactic aspect

• How to master the bit organisation– 2 possible solutions

• Define exchange formats– Better done by user communities (ex : FITS for astronomy)

– Not generic enough

• Define a standard way for describing any data– Was the CNES proposal to CCSDS

– Was entrusted to CNES (definition and tools) by CCSDS


History

• CNES proposed a description language– EAST for data syntax

– CCSDS recommendation since May 1997CCSDS 644.0-B-2 Issue 2 November 2000

– ISO standard since 2000ISO 15889-2000

CCSDS recommendations


EAST principles

• EAST is a standard language– Enhanced Ada SubseT

• inherits from a huge standardisation work

• enhanced with a few features missing for data description

• EAST has not to be mastered to be used• thanks to the tools

Data syntax description


EAST principles (2)

• Assumption : any kind* of digital data (available as an ordered set of bits) can be described as a tree.

• The minimal tree is a rake

*note : optimal for binary data, interesting for raw ASCII format, of no use for XML data

• To describe such a tree, EAST proposes :• composite types for the branches

• elementary types for the leaves



EAST data types

• Composite types are :• records• arrays• lists

• Elementary types are :• integers and reals• enumerated• characters



Optionality

• Some branches or leaves of the data tree may be optional.– EAST offers a way to indicate which data

entities are optional and under which condition they exist

– Example : • Housekeeping data exists if telemetry mode is set to

« housekeeping »



Variable size

• Some repetitions of data entities may have a variable number of elements

– EAST offers a means to indicate how many of those elements are present in the current data



EAST use conclusion

• All the features described in the previous slides have been sufficient to describe almost every data encountered since 1997

– Exceptions : • variable size not carried by the data

• variable size provided after the variable size field



History (2)

• CNES has also been in charge of writing the recommendation for data semantics description :– DEDSL (Data Entity Dictionary Specification Language)

– XML/DTD syntax

– CCSDS recommendation since January 2002CCSDS 647.1-B-1 Issue 1 January 2002

– ISO standard since 2003ISO 22643 : 2003

CCSDS recommendations


DEDSL principles

• The data semantics description can be kept as a set of attribute values.

• The recommendation defines standard attributes : name, definition, units….

• The recommendation gives a standard way to define user attributes to be added to the standard ones if needed

Data semantics description


DEDSL principles (2)

• A data description consists in the tree organisation (as seen on the sysntactic aspect) completed by all the fields (nodes or leaves) caracteristics given as attributes values.

• Example of the descripon of a leaf of such a tree description.


OASIS

• OASIS is the tool to describe data (syntax and semantics) in a friendly way (without any requested knowledge of the EAST or DEDSL syntax)

• A dedicated Man-Machine interface offers the requested features (those listed previously) to describe data

A tool to describe data


OASIS (2)

• Example : • a data tree as built and shown by OASIS



OASIS (3)

• Once the tree and each of its nodes and leaves described, the descriptions can be produced just by a click (EAST and DEDSL-XML files)

• Once saved, the EAST description can be used by the EAST tools to read and write the described data

• Once saved, the XML dictionary can be published as an html or word document



EAST tools• Application Programming Interface (API)

• GENERATOR : for data writing• INTERPRETER : for data reading

– Both available for C, C++, FORTRAN and Ada

• Standalone tools (based on the API) • data_checker : data vs description compliance• ascii_dump : display any data in ascii values• DUW : interactive data generator• other utilities…

Tools for EAST described data


EAST tools (2)

• EAST tools are available on :– PC (DOS, NT, Linux), – HP-UX, – SUN-Solaris, – Dec true64 (OSF1) , – VAX-VMS– other on request….

Tools for EAST described data


Interface Control Documents

• The DEDS-XML OASIS output can be transformed in an html or Word document

• A tool (named SCRIBE) is being developed for that purpose .

Documenting the interfaces


Synthesis

Data dictionaryDEDSL/XML

XML Data

EASTApis

Html or Word document

DCI generation

File descriptionsXML schema

ReadWrite

GenerateCheck

‘RAW’ Data

OASIS tool

File descriptionsEAST

ReadWrite

GenerateCheck

XML applications


Main advantages• Only one consistent reference for syntactic

description used for input/output and for semantic description (documentation for understanding)

• Reusable and reliable generic input/output software available

• User application easier to maintain due to less impact from changes of data formats

• Help in describing data and producing documentation

The project conclusions


EAST/DEDSL-XML in action

• Many projects at CNES– SPOT, SSALTO, SIPAD, HELIOS,

PHARAO…

• ESA projects – AMS, DEBAT

• NASA– Science data rehabilitation

Use in other projects


EAST/DEDSL-XML feed back

• PROs : • Easier sub-system integration

– description accuracy (even after long time!!!)

– early data simulation

– assisted data checking

• Easier ICD evolutions– full top-down process

• Helpful in heterogeneous platforms context– automatic conversions

• Helpful in multi-missions context

Projects feedback


EAST/DEDSL-XML feed back (2)

• CONs :• Performances to be again improved

– reading in time constrained context

– writing in any case

• User type sharing to be added– to facilitate evolutions

• Documentation production to be improved– project customisation

– add as much filters as possible to allow adaptation of the document to its target

Projects feedback


Current actions

• Documentation tool in progress• generic tool to transform XML-DED in Word or html documents

• 2nd prototype delivered

• Main lacks taken into account by the DEBAT project (ESA/ESRIN)

• Support to NASA• tests on data migration - good first results

• Support to XML schema production

On going activity


CNES policy for the EAST tools

• OASIS and EAST tools are freely available

• They can be downloaded from a CNES server

• Tools are maintained by a CNES team and a sub-contractor team (CS-SI)

• Policy : if CNES stops founding the maintenance the source code would be made available

Information


Links• For any support request :

• [email protected] (e-mail address)

• For information and documentation• east.cnes.fr (web server)

• To download the software :• logiciels.cnes.fr (web server)

• To get the standards• http://www.ccsds.org (web server)

– then under publications, recommendations and reports, information access and interchange systems.

Information