Embed Size (px)
Citation preview
ESGF Dashboard Working Team!
P. Nassisi, S. Fiore, G. Aloisio!Euro Mediterranean Center on Climate Change (CMCC)!
Marriott Hotel!Dec 8-11 2015, Monterey, CA !
Outline!
Working Team mandate, leads and members
Architecture in the large of the system
Work achieved over the past year – Intro
Coarse grain system -‐ Design view
Downloads staAsAcs
Downloads by IdenAty Provider
Downloads by User acAvity
Fine grain system -‐ Design view
Federated and cross-‐project staAsAcs
Project-‐specific staAsAcs
PrioriAzed development and roadmap for the next year
Working Team mandate, leads and members!
Working Team Acronym esgf-‐dwt (Dashboard Working Team) Mandate Design and implementaAon of a distributed and scalable system for (i) monitoring the Earth System Grid FederaAon and (ii) providing data usage staAsAcs in a comprehensive way and through a simple and intuiAve web interface. Leader Sandro Fiore (unAl 2014), Paola Nassisi (since 2015) Members Paola Nassisi, Alessandra Nuzzo, Maria Mirto, Sandro Fiore Modules esgf-‐dashboard and esgf-‐desktop
Architecture in the large of the system!
The main modules of the monitoring system are, for the back-‐end:
the Informa6on Provider It’s responsible for retrieving all the metrics and storing them in the esgcet catalog and binary files (for long term sta6s6cs).
the dashboard catalog A system database which stores all the informa6on about hosts, peer-‐groups, services, users, availability, deployment, etc.
the sensors Global and local metric sensors to retrieve and manage informa6on about node type, registered users, downloaded data, system metrics both a single site and federa6on level
For the front-‐end:
a web-‐based environment GUI: the ESGF Desktop modular web applica6on relying on a strong adop6on of Web 2.0 concepts and providing several views at different granularity levels
a set of configura6on files
Work achieved over the past year - Intro! AcAvity
Complete implementa6on of the first data usage sta6s6cs system (coarse grain), relying on the exis6ng access logging system. Status: Under tes6ng. Final bug fixing Delivery date: December 2015
Metrics Download stats
Data downloaded (GB/TB) Number of downloads Number of dis6nct files Number of dis6nct users Downloads by user Downloads by Iden6ty Provider
Clients staAsAcs Geographic map Country/con6nent distribu6on
Design and implementaAon of the new system (fine grain) Prototype Jan 2016 First release Feb 2016
Federation-Level Monitoring0 0 0
Metrics | ▿ … | ▿Peer Groups
IS-ENES nodeFederation Status
ChartGridMap
Temporal Aggregation Temporal Range
availablenode
not availablenode
Federation-Level Monitoring0 0 0
Metrics | ▿ … | ▿Peer Groups
Host Name Reg. UsersLat, Lon SW Version Node TypeIS-ENES node
ChartGridMap
Temporal Aggregation Temporal Range
Realtime Stats0 0 0
Metrics | ▿ Last k samples, intervalFilter by dates
Time
Mem
ory
(MB
)
11:00 AM : 10266
Data usage statistics
User
Visualizesdata download statistics
extension pointsNode/Peer group
MetricFilter
Selects a nodeSelects a peer
group
Number of downloads
Number of files
Number of users
Amount of downloaded data
<extends><extends>
<extends>
<extends>
<extends>
<extends>
Extension point:Node/Peer group
Extension point:Metric
Selects a type of aggregation
extension pointType of aggregation
<includes>
<extends>
Visualizes data node client geolocation
extension pointsNode/Peer groupVisualization type
Spatial aggregation
Selects a node
Selects a peer group
<extends>
Extension point:Node/Peer group
<extends>
Selects type of visualization
chart - grid - map
<extends>
Selects a countrySelects a continent
<extends><extends>
Extension point:Spatial aggregation
Extension point:Visualization type
Most downloaded datasets,
variables, models
<extends>
Selects filter
<extends>
Extension point:Filter
Selects type of aggregation
dayweek
month/yearyearhost
esgf_node_manager ESGCET
ESGF node manager filter
ID user_idp dura1on size 1mestamp remoteaddr …
… … … … … … …
… … … … … … …
… … … … … … …
access_logging table
ESGF dashboard Store sta6s6cs
ESGF DESKTOP
ESGF DATA NODE
Work achieved over the past year – Coarse grain system Architectural view!
Get sta6s6cs
Features: • Download sta6s6cs • Coarse grain stats • Clients sta6s6cs • Same view for all projects
Work achieved over the past year – Coarse grain system Download statistics (I)!
Work achieved over the past year – Coarse grain system Download statistics (II)!
Work achieved over the past year – Coarse grain system Download statistics (III)!
Work achieved over the past year – Coarse grain system Downloads by Identity Provider!
Work achieved over the past year - Coarse grain system Downloads by User (activity)!
Work achieved over the past year - Coarse grain system Client distribution!
Work achieved over the past year - Coarse grain system Client statistics!
esgf_dashboard ESGCET
ESGF node manager filter
ID url_path dura1on size 1mestamp success processed
… … … … … … …
… … … … … … …
… … … … … … …
DASHBOARD_QUEUE
ESGF dashboard ETL SOLR
getMetadata(url_path)
METADATA METADATA
ESGF DESKTOP
ESGF DATA NODE
Work achieved over the past year – Fine grain system Architectural view!
Mul1 Tier Database 1 Datawarehouse + A set of data marts
Features: • Extended set of sta6s6cs • Fine grain level • Project specific views • More scalable design access_logging table
no longer needed
Work achieved over the past year – Fine grain system Statistics database design at glance!
Work achieved over the past year – Fine grain system Federated and cross-project statistics!
Work achieved over the past year – Fine grain approachProject-specific statistics (CMIP5)!
Geoloca1on sta1s1cs
Top Ten experiments
Top Ten variables
Metrics Timeframe
Number of downloads grouped by experiments
Number of downloads over 1me grouped by experiments/
models
% of downloads grouped by experiments
Radar chart # of downloads by year grouped by experiments
Prioritized development and roadmap for the next year!
Status of the system and future roadmap for the fine grain system Database back-‐end design completed
Preliminary implementaAon available Single site & Federated Single and cross-‐project
Search lib in progress to extract project-‐specific informaAon e.g. obs4MIPS: cf_standard_name, processing level, realm, Ame frequency, variable, dateAme start, dateAme stop, etc.
New project specific views to be defined in the context of the ESGF-‐DWT Delivery date: February 2016 for CMIP5, CORDEX, obs4MIPs Ext. deps: link to the esgf-‐node-‐manager for gathering federaAon-‐level informaAon Stronger link with: node-‐manager, network and search working teams. REST APIs
Single node level – Jan 2016 FederaAon level (March 2016)
Extended set of views with geo-‐locaAon and federaAon-‐level staAsAcs by May 2016 New front-‐end presentaAon layer (Aug 2016 first release, Dec 2016 final release)
Thank you!
