CMS Components Generic Infrastructure Bits

LCG/Blueprint RTAGLCG/Blueprint RTAG

August 5-6, 2002 Lassi A. Tuura, Northeastern Universityhttp://iguana.cern.ch

CMS ComponentsCMS ComponentsGeneric Infrastructure BitsGeneric Infrastructure Bits

Lassi A. TuuraLassi A. Tuura

Northeastern University, Boston

http://iguana.cern.ch/

2August 5-6, 2002 Lassi A. Tuura, Northeastern Universityhttp://iguana.cern.ch

Frameworks: StructureFrameworks: Structure

FederationFederationwizardswizards

Detector/EventDetector/EventDisplayDisplay

Data BrowserData Browser

Analysis jobAnalysis jobwizardswizards

Generic analysis Generic analysis ToolsTools

ORCAORCA

FAMOSFAMOS

ObjyObjytoolstools

GRIDGRID

OSCAROSCAR COBRACOBRADistributedDistributedData StoreData Store

& Computing& ComputingInfrastructureInfrastructure

CMSCMStoolstools

Consistent User Consistent User InterfaceInterface

Coherent basic tools Coherent basic tools and mechanismsand mechanisms

OverviewGraphicsClasslibPlug-in ManagerComponents






ODBMS GEANT 3 / 4 CLHEP PAW

ReplacementC++ Standard Library+ Extension Toolkits

Frameworks: LayeringFrameworks: Layering

CalibrationObjects Generic Generic

Application Application FrameworkFramework

Physics modulesPhysics modulesGrid-UploadableGrid-Uploadable

BasicBasicServicesServices

Adapters and ExtensionsAdapters and Extensions

ConfigurationObjects Event

Objects

(Grid-aware) Data-Products

SpecificSpecificFrameworksFrameworks

EventEventFilterFilter

ReconstructionReconstructionAlgorithmsAlgorithms

PhysicsPhysicsAnalysisAnalysis

DataDataMonitoringMonitoring







GEANT3 / 4 Qt Classlib Open

InventorC++ Standard Library+ Extension Toolkits

Generic ComponentsGeneric Components

CalibrationObjects Generic Generic

Application Application Framework Framework

Physics modulesPhysics modulesGrid-UploadableGrid-Uploadable

BasicBasicServicesServices

Adapters and ExtensionsAdapters and Extensions

ConfigurationObjects Event

Objects

(Grid-aware) Data-Products

SpecificSpecificFrameworksFrameworks

EventEventFilterFilter

ReconstructionReconstructionAlgorithmsAlgorithms

PhysicsPhysicsAnalysisAnalysis

DataDataMonitoringMonitoring







Quick IGUANA OverviewQuick IGUANA Overview A generic toolkit for user interfaces and visualisation

Builds on existing high-quality libraries (Qt, OpenInventor, Anaphe, …) Used to implement specific visualisation applications in other projects Used in CMS and D0 (and L3—pretty static) Portable: UNIX and Windows actively used

– Few limits to porting to new platforms

Main technical focus: provide a platform that makes it easy to integrate GUIs as a coherent whole, to provide application services and to visualise any application object Many categories / layers: GUI gadgets & support, application environment,

data visualisers, data representation methods, control panels, … Designed to integrate with and into other applications Apart from a tiny kernel, virtually everything else is in plug-ins

– Plug-ins and dynamic linking are 100% orthogonal








Quick IGUANA OverviewQuick IGUANA Overview

IntegrationIntegration

ActionAction

Visualise Results,Visualise Results,Modify Objects,Modify Objects,

Further InteractionFurther Interaction






Building BlocksBuilding Blocks Basic services layer

Portability + utilities Classlib GUI and 2D graphics Qt 3D graphics OpenInventor, OpenGL

Internal utility services Plug-in manager

– Generic base– Concrete implementation– (More details later on)

Internal state management and service components– (Components, abstract interfaces and concrete classes)







Graphics ToolkitsGraphics Toolkits GUI and 2D Graphics: Qt

Easy to use Very rich functionality, lots of community experience Very portable, uses platform GUI conventions: looks “natural” Excellent documentation Free for our purposes

3D Graphics: OpenInventor Same as above—portability from OpenGL; both commercial and free exist An incredibly rich toolkit!

Qt and OIV limited strictly to these parts of IGUANA Not using Qt or OIV classes (containers and many other utilities) outside

their respective “natural” graphics domains: other parts use other libraries Reasons explained when discussing our component approach







Debugging + RobustnessDebugging + Robustness Debug Assertions; Validation;

Stack tracing;Programmatic core dump

Log Rich debugging-oriented logging facility Signal POSIX; Standard C/C++; Win32 SEH;

Fatal error handling and recovery LowMemoryPool Graceful low-memory protection Error Exception chaining for increased error

abstraction as errors are propagated through

the application (much used in implementation)

SaveErrno






10


Major ToolsMajor Tools xtype_info

Extensible std::type_info replacement Minimal meta-class information

MultiMethod Multi-method dispatcher Works with arbitrary argument lists

– “Arbitrary”: Any polymorphic class can be used as “virtual” argument;any number of “virtual” and non-virtual arguments can be dispatched

– Macro sugar for up to 3 virtual arguments and 2 non-virtual args Optimised to be storage and performance efficient even if hundreds of

classes and methods exist Registry

Sort of like Windows registry but concept modified to actually work






11


System AbstractionSystem Abstraction SharedLibrary Shared library loading, unloading;

list of currently loaded libraries Time, TimeSpan s-resolution calendar time, conversions,

formatting, calculations TimeInfo High-resolution (ns) monotonic time,

system and process times HostInfo Host itself, DNS lookups ProcessInfo PID etc ResourceInfo Program + system resource info, limits SystemInfo System information: memory, os, … UserInfo Users: home dir, login, real name, …






12


Bits & NumericsBits & Numerics BitIterator Iterating over data as bits BitOps Collection of optimised bit magic operations BitPattern Construction of bit patterns BitTraits Bit type properties of integral types

IntBits Type selection by required number of bits:{exact, fast} x {signed, unsigned}

IntTraits Poor man's std::numeric_limits






13


MiscellaneousMiscellaneous Environment, ShellEnvironment Callback, Hook, HookList MapIterator PODVector Range Sequence SearchPath Unwind UUID Utility Macros






14


I/O: BasicsI/O: Basics IOChannel Abstraction of a non-seekable

I/O sources and targets Storage Abstraction of a seekable

(file-like) storage

IOBuffer, IOSize, Basic I/O insulation: systemIOOffset characteristics (e.g. LFS)

DirIterator Iterator for reading directorycontents like readdir()






15


I/O: Files, File SystemsI/O: Files, File Systems File, FileAcl, FileMapping

File operations File and anonymous memory mapping High-throughput I/O channel transfers (e.g. file to socket)

Filename Filename abstraction Existence, type, permissions checking Core support for file backup, versioning, autosaving, locking Filename expansion, substitution, abbreviation, completion, globbing File rename, move, copy Secure temporary file handling Current working directory Description of system file name conventions






16


I/O: IPCI/O: IPC IOChannelPair Pair of I/O channels

(e.g. pipe or socket pair) Socket All socket stuff (TCP, UDP, UNIX, …)

InetSocket, InetServerSocket, InetAddress LocalSocket, LocalServerSocket, LocalAddress SocketPair

NamedPipe File system FIFOs Pipe Anonymous FIFOs PipeCmd “popen()” IOSelector I/O switch a la “select” (implementations

available: select, poll, /dev/poll, real-timesignals, sigio; Win32 select simulation,overlapped I/O (completion ports?))






17


I/O: High-Level StreamsI/O: High-Level Streams Input/Output Streams

Base interface Buffered Filtering Linked Pushback (input only) C++ Standard (std::i/ostream) C Standard (FILE *) Storage (cf. I/O Basics) IOChannel (cf. I/O Basics)

Storage implementations Memory C++ Standard C Standard






18


Compression + ArchivesCompression + Archives Data checksums

Interface, Adler32, CRC32 (MD4, MD5?) Compressor and decompressor engines

Interface, BZIP, ZLIB Generic interactive engine

Input & output streams Compressing and decompressing base interface BZIP, ZLIB, GZIP, ZIP, TAR Checksum

Archives files and their members TAR, ZIP






19


ModulesModules What’s a module?

An (extension) component to a framework (often dynamically loaded) Framework knows only that a module extends a particular API (usually

abstract interfaces), not how and what it needs to do so Examples abound: Netscape plug-ins, Apache modules, Python and PERL

modules, Illustrator and PhotoShop extensions, … Already used in many ways in CMS projects: package builders, detector

units, hit/digi readers and writers, and many algorithms are essentially extensions of APIs provided by COBRA, and “subframeworks” in ORCA

Why modules? Eases management—framework doesn’t need to know lots of details about

the modules, but can simply load them and gain additional capabilities Helps to give symbolic names to capabilities and allow framework to reason

about them: “here’s all you will need to read data product X” Allows better services to external parties (Python, GRID, IGUANA, …)

without having to put that knowledge into many applications






20


The Plug-In ManagerThe Plug-In Manager Maintains a list of directories with plug-in definitions Plug-in = shared library with well-defined entry points

Self-describing through a query entry point But really orthogonal to dynamic loading

– Some or all plug-ins can be built into the program Automatically detects new, changed or removed plug-ins

For new or changed ones loads the plug-in, queries for its capabilities, and unloads the plug-in

Maintains a cache of plug-in properties Updates the cache if the directory is writable

When requesting information from the database, uses the cache instead of loading all the plug-ins all the time

A separate generic package IGUANA architecture defines a specific implementation






21







22


Abstract or Concrete?Abstract or Concrete? Abstract or concrete implementations?

We use both: a lot of abstract interfaces and decoupling at the bottom level,more concrete implementation usage on the top

If you need a specific service, you use it directly– Access through shared state object which records references to

“application state extensions”, but knows nothing about them– Clients ask specific services to look themselves up in the state– Services vary in “abstractness”: current selection state does not need to

know anything about GUI, object context menu is defined in terms of Qt– Only if the service has multiple alternative implementations the service

has an abstract interface; otherwise clients use a concrete implementation Services can and do have mini-plug-in architectures of their own Proxies and envelopes sometimes better than abstract interfaces

Catch-all IgExtension for those mini-plug-in systems but it’s not an IInterface– Concrete plug-in manager defines the interfaces specific to its mission

Trying to keep things simple and straightforward!






23


External ComponentsExternal Components Qt and OIV: use concrete implementations only

Not trying to encapsulate these in any way– … but restrict them to their “native domains”

Reason: build modularity firewalls in the system– No point in trying to abstract so rich toolkits– Major benefits in going for full integration– Changing package known to incur limited pain

We know exactly what we are using and from where We know we can get rid of the original package Plan: expend N months to swap with equally rich or better product






Documents

CMS Components Generic Infrastructure Bits