ESA Climate Change Initiative Fire cci D3.1 System ...ESA Climate Change Initiative – Fire_cci D3.1 System Specification Document (SSD) Project Name ECV Fire Disturbance: Fire_cci

ESA Climate Change Initiative – Fire_cci

D3.1 System Specification Document (SSD)

Project Name ECV Fire Disturbance: Fire_cci Phase 2

Contract Nº 4000115006/15/I-NB

Issue Date 30/01/2018

Version 1.5

Authors Thomas Storm, Martin Böttcher, Grit Kirches

Document Ref. Fire_cci_D3.1_SSD_v1.5

Document type Public

To be cited as: T. Storm, M. Boettcher, G. Kirches (2018) ESA CCI ECV Fire Disturbance: System Specification Document, version 1.5.

Available at: http://www.esa-fire-cci.org/documents

http://www.esa-fire-cci.org/

Fire_cci System Specification Document

Ref.: Fire_cci_D3.1_SSD_v1.5

Issue 1.5 Date 30/01/2018

Page 2

Project Partners

Prime Contractor/

Scientific Lead & Project

Management

UAH – University of Alcala (Spain)

Earth Observation Team UAH – University of Alcala (Spain)

EHU – University of the Basque Country (Spain)

UL – University of Leicester (United Kingdom)

UCL – University College London (United Kingdom)

ISA – School of Agriculture, University of Lisbon (Portugal)

System Engineering BC – Brockmann Consult (Germany)

Climate Research Group MPIC – Max Planck Institute for Chemistry (Germany)

IRD - Research Institute for Development (France)

LSCE - Climate and Environmental Sciences Laboratory (France)

VUA - Vrije Universiteit Amsterdam (Netherlands)

Distribution

Affiliation Name Address Copies

ESA Stephen Plummer (ESA) [email protected] electronic copy

Project

Team

Emilio Chuvieco (UAH)

M. Lucrecia Pettinari (UAH)

Joshua Lizundia (UAH)

Gonzalo Otón (UAH)

Mihai Tanase (UAH)

Miguel Ángel Belenguer (UAH)

Aitor Bastarrika (EHU)

Ekhi Roteta (EHU)

Kevin Tansey (UL)

Marc Padilla Parellada (UL)

James Wheeler (UL)

Philip Lewis (UCL)

José Gómez Dans (UCL)

James Brennan (UCL)

Jose Miguel Pereira (ISA)

Duarte Oom (ISA)

Manuel Campagnolo (ISA)

Thomas Storm (BC)

Johannes Kaiser (MPIC)

Angelika Heil (MPIC)

Florent Mouillot (IRD)

M. Vanesa Moreno (IRD)

Philippe Ciais (LSCE)

Chao Yue (LSCE)

Pierre Laurent (LSCE)

Guido van der Werf (VUA)

Ioannis Bistinas (VUA)

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

mariavanesa.morenodominguez@cefe....

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

electronic copy



Issue 1.5 Date 30/01/2018

Page 3

Summary

This document is the version 1.5 of the System Specification Document for the Fire_cci

project. It introduces the structure, workflows and mode of operation of the processing

system used in Phase 2 of the Fire_cci project.

Affiliation/Function Name Date

Prepared BC

Thomas Storm

Martin Boettcher

Grit Kirches

26/01/2018

Reviewed UAH – Project Manager M. Lucrecia Pettinari 30/01/2018

Authorized UAH - Science Leader Emilio Chuvieco 30/01/2018

Accepted ESA - Technical Officer Stephen Plummer 30/01/2018

This document is not signed. It is provided as an electronic copy.

Document Status Sheet

Issue Date Details

1.0 24/02/2016 First document for Phase 2 of Fire_cci (formerly known as v.2.0)

1.1 16/03/2016 Addressing ESA comments according to

CCI_FIRE_EOPS_MM_16_0034.pdf

1.2 06/09/2016 Updated release of the document with minor corrections

1.3 25/10/2016 Addressing ESA comments according to

CCI_FIRE_EOPS_MM_16_0109.pdf

1.4 30/09/2017 Update for MODIS / SFD processing

1.5 30/01/2018 Addressing ESA comments according to CCI-EOPS-FIRE-MM-17-

0098.pdf

Document Change Record

Issue Date Request Location Details

1.1 16/03/2016 ESA

Version number

Section 1

Section 2.1

Section 2.2

Section 2.3

Section 3.1

Figure 3.1

Section 3.2

Section 3.4

Table 3.3

Section 5.1

Section 5.3

Annex

Change of previous version number from 2.0

to 1.0 to address this document being

independent from the one in Phase 1.

Minor changes in the text.

Re-ordering of sentences.


Addition of new reference documents.


Updated


Inclusion of text detailing the SoW

requirements.

Changes in the requirements.


Minor changes in the text. Inclusion of

additional information in the list of steps.

Addition of new acronyms.

1.2 06/09/2016 BC

Section 1

Section 2.1

Section 2.2

Section 2.3

Section 2.4

Section 3.1

Section 3.3

Updated.

Paragraph removed.

Minor changes in the text. Last paragraph

shortened.

Reference documents updated.


Figure 3.1 updated. Minor changes in the text.

Added information about Calvalus



Issue 1.5 Date 30/01/2018

Page 4

Issue Date Request Location Details

Section 3.4

Section 4

Section 4.2.3

Section 5.1

Section 5.1.3

MapReduce

Requirements text resumed. Reference to the

MERIS ATBD updated.

Removed sub-section “Docker”, as it is not

applicable anymore. The rest of the sub-

sections were re-numbered.

Figure 4.4 updated. Added information about

MapReduce and formatting tool

implementation, eliminated information about

Docker container.

Figure 5.1 updated.

Fully re-written according to system updates.

1.3 25/10/2016 ESA

Naming convention

All document

Section 3.1

Section 3.2

Section 5.1

Section 5.2

Reference to the year of the project added to

the name of the document.

The reference to the MERIS data as FSG was

changed to FRS.

Figure 3.1 updated. New entity in the text

added.

Figure 3.2 updated.

Small changes in the text.

Added information on versions of auxiliary

data.

1.4 30/09/2017 BC

Whole document Added system overview, technical methods

and workflows for SFD and MODIS

processing

1.5 30/01/2018 ESA

Sections 2.1, 3, 3.2,

3.3, 5.2, 5.2.3

Section 3.1

Section 4

Section 6

Small changes in the text

Figure 3.1 updated and text expanded

Section restructured, including previous sub-

section 3.4

New section added



Issue 1.5 Date 30/01/2018

Page 5

Table of Contents

1. Executive Summary ..................................................................................................... 7

2. Introduction ................................................................................................................. 7

2.1. Background ............................................................................................................. 7

2.2. Purpose and scope .................................................................................................. 7

2.3. Applicable and reference documents ...................................................................... 8

3. System overview ........................................................................................................... 9

3.1. Fire_cci system context .......................................................................................... 9

3.2. Main function and processing chain ..................................................................... 10

3.3. System requirements............................................................................................. 10

4. Fire_cci system architecture ..................................................................................... 12

4.1. High-level system decomposition ........................................................................ 12

4.2. Calvalus system .................................................................................................... 13

4.2.1. Technical methods and concepts used in Calvalus ........................................ 15

4.3. JASMIN system .................................................................................................... 20

4.3.1. Technical methods and concepts used in JASMIN ....................................... 20

5. Workflows .................................................................................................................. 21

5.1. MERIS global time series processing ................................................................... 21

5.1.1. MERIS Pre-processing .................................................................................. 22

5.1.2. MERIS Burned Area retrieval ....................................................................... 24

5.1.3. MERIS PSD-compliant formatting ............................................................... 26

5.2. MODIS workflow ................................................................................................. 28

5.2.1. Data acquisition ............................................................................................. 29

5.2.2. Data processing ............................................................................................. 30

5.2.3. Data repackaging and transfer ....................................................................... 30

5.2.4. PSD product production ................................................................................ 31

5.3. SFD workflow ...................................................................................................... 31

5.3.1. Data acquisition ............................................................................................. 32

5.3.2. Pre-processing ............................................................................................... 32

5.3.3. BA retrieval ................................................................................................... 32

5.3.4. Grid/pixel formatting ..................................................................................... 33

6. System outputs ........................................................................................................... 34

Annex: Acronyms and abbreviations .......................................................................... 35



Issue 1.5 Date 30/01/2018

Page 6

List of Tables

Table 3.1: System requirements in the SoW .................................................................. 11

Table 3.2: System requirements specified in the PSD and CCI data guidelines ............ 11

Table 3.3: System requirements specified in the ATBDs .............................................. 12

Table 3.4: System requirements specified in the DARD................................................ 12

Table 4.1: Calvalus infrastructure elements ................................................................... 16

Table 4.2: Software items for Fire_cci processing ......................................................... 19

Table 5.1: Pre-processing steps ...................................................................................... 23

Table 6.1: Output products ............................................................................................. 34

List of Figures

Figure 3.1: System context of the Fire_cci system with team, data providers, and other

entities ....................................................................................................................... 9

Figure 3.2: Requirements derivation logic with system requirements mainly derived

from PSD ................................................................................................................ 10

Figure 4.1: Decision matrix for MODIS processing. + indicates good, ++ indicates very

good, o indicates average, - indicates poor, -- indicates very poor. ....................... 13

Figure 4.2: Fire_cci system architecture layers with specific elements, Calvalus, and

Hadoop ................................................................................................................... 14

Figure 4.3: Archive-centric............................................................................................. 15

Figure 4.4: Data-local ..................................................................................................... 15

Figure 4.5: Calvalus architecture .................................................................................... 16

Figure 4.6: Software architecture ................................................................................... 18

Figure 5.1: High-level view of Fire_cci MERIS processing chain ................................ 22

Figure 5.2: MERIS Pre-processing ................................................................................. 24

Figure 5.3: MERIS BA retrieval .................................................................................... 25

Figure 5.4: Formatting .................................................................................................... 27

Figure 5.5: Sequence of MODIS processing actions ...................................................... 29

Figure 5.6: Sentinel-2 workflow ..................................................................................... 31

Figure 5.7: MERIS BA retrieval .................................................................................... 33



Issue 1.5 Date 30/01/2018

Page 7

1. Executive Summary

This document is the Fire_cci System Specification Document (SSD) for Phase 2. The

Fire_cci system is employed in order to generate the Fire_cci dataset, which will

comprise a number of global and regional variables (see below).

This document describes the Fire_cci system used in Phase 2, the workflows it defines

and uses, and the environment in which it exists.

In the current version, the SSD describes the processing system which has been used for

computing the MERIS global time series, the MODIS global time series, and the small

fires database (SFD) based on Sentinel-2 MSI data. Later versions will also describe the

processing of

- the PROBA-V dataset

- the Sentinel-3 OLCI/SLSTR global dataset

The general processing steps that are covered are:

- the pre-processing of MERIS FSG data and Sentinel-2 MSI data

- the BA retrieval of MERIS FSG, MODIS and Sentinel-2 data

- the formatting of the results of the BA retrieval to the PSD-compliant format.

2. Introduction

This document describes the system developed for the ESA Fire_cci project and its

elements used in the current phase.

2.1. Background

The ESA Climate Change Initiative (ESA CCI) stresses the importance of providing a

higher scientific visibility to data acquired by ESA sensors, especially in the context of

the IPCC reports. This implies to produce consistent time series of accurate Essential

Climate Variables (ECV) products that can be used by the climate (primarily) and other

scientists for their modelling efforts. Long-term observations and the international links

with other agencies currently generating ECV data are keys to this activity.

The fire disturbance ECV includes Burned Area (BA) as the primary variable. The

project includes algorithms for the pre-processing of MERIS, Sentinel-2 MSI, PROBA-

V, and Sentinel-3 OLCI/SLSTR data. This is done for several different improvement

reasons, e.g. improving geometric accuracy and removing atmospheric effects, BRDF

effects and potential confusions of other observations (clouds, cloud shadows, water,

snow, topographic shadows) with burned area. Algorithms for BA retrieval include

MODIS RNIR bands as well.

For all of the input datasets, the data must be acquired, algorithms and processing

workflows must be defined and implemented, and the data processing must be

performed.

2.2. Purpose and scope

Purpose of the Fire_cci system is the generation of the Burned Area product, the

exchange of the results with project partners and delivery of the final products to the

end users. There are various large satellite input datasets to be processed in a performant

way in the course of the project. This includes an extension of the temporal coverage of

the time series also in comparison with the Phase 1 output. The algorithms used for

processing will be continuously improved and adapted to new sensors with the need of



Issue 1.5 Date 30/01/2018

Page 8

several test and improvement cycles before full-scale processing. This shall also be

supported by the processing system.

This SSD exists in the context of other Fire_cci documents: The Fire_cci PSD [RD-1]

describes the content and format of the BA products to be generated, the DARD [RD-2]

describes the input datasets, and the ATBDs [RD-3], [RD-6], [RD-7] define the

algorithms that were used for BA processing. This SSD describes the design of the

system that generates these BA products, including the integration of the processors that

implement the algorithms defined in the ATBDs, the handling of the input datasets, and

the control of the production workflow for the Fire_cci processing chains. The Fire_cci

processing system is based on several layers of generic components of the Calvalus

processing system and a shared infrastructure. The main concepts of the elements used

by Fire_cci and the Fire_cci-specific configuration and extension are also in the scope

of this document.

This document currently covers the MERIS, MODIS, and Sentinel-2 workflows of

Fire_cci. It will be extended in the course of the project for the specifics of each sensor

that will be included.

2.3. Applicable and reference documents

The following documents are applicable to this document:

[AD-1] ESA Climate Change Initiative (CCI) Phase 2 Statement of Work,

prepared by ESA Climate Office, Reference CCI-PRGM-EOPS-SW-12-

0012, Issue 1.3, date of issue 24 March 2015

[AD-2] ESA Climate Change Initiative – CCI Project Guidelines. Ref. EOP-

DTEX-EOPS-SW-10-0002, Issue 1.0, date of issue 05 November 2010,

available at http://cci.esa.int/filedepot_download/40/4

[AD-3] Proposal for ECV Fire Disturbance, Phase 2 of the Climate Change

Initiative (GMECV), Issue 1.1, date of issue 24 July 2015

[AD-4] Data Standards Requirements for CCI Data Producers, CCI-PRGM-EOPS-

TN-13-0009, Issue 1.2, date of issue 09/03/2015

The following documents are further referenced in this document:

[RD-1] Fire_cci Product Specification Document (PSD), Chuvieco E., Pettinari M.L.,

Heil A., Storm T. (2017) Fire_cci_D1.2_PSD_v6.3, available at

http://www.esa-fire-cci.org/documents

[RD-2] Fire_cci Data Access Requirements Document (DARD), Pettinari M.L.,

Chuvieco E., Padilla M. Storm T. (2017) Fire_cci_D1.4_DARD_v2.5,

available at http://www.esa-fire-cci.org/documents

[RD-3] Fire_cci Algorithm Theoretical Basis Document –MERIS, Alonso-Canas I.,

Chuvieco E., Kirches G., Storm T. (2016) Fire_cci_D2.1.1_YR1_ATBD-

MERIS_v1.1, available at http://www.esa-fire-cci.org/documents

[RD-4] CEDA archive, services and JASMIN help docs, available at

http://help.ceda.ac.uk/

[RD-5] IBM Platform LSF documentation, available at https://www.ibm.com/

support/knowledgecenter/en/SSETD4/product_welcome_platform_lsf.html

http://cci.esa.int/filedepot_download/40/4




http://help.ceda.ac.uk/

https://www.ibm.com/%20support/knowledgecenter/en/SSETD4/product_welcome_platform_lsf.html

https://www.ibm.com/%20support/knowledgecenter/en/SSETD4/product_welcome_platform_lsf.html



Issue 1.5 Date 30/01/2018

Page 9

[RD-6] Fire_cci Algorithm Theoretical Basis Document (ATBD), Bastarrika A.,

Roteta E. (2018) Fire_cci D.2.1.2_ATBD-SFD_v1.0.pdf.

[RD-7] Fire_cci Algorithm Theoretical Basis Document – Lizundia J., Chuvieco E.,

Pettinari M.L. (2018) Fire_cci D2.1.3_ATBD-MODIS_v1.0.pdf.

3. System overview

This section provides an overview of the Fire_cci system with its system context, its

main function and processing chain, and its architecture. Also a set of system

requirements are derived from the requirements in [AD-1] as starting point of the

system design and later verification.

3.1. Fire_cci system context

The Fire_cci system’s two main counterparts are the Fire_cci team, and on the other

side data providers. There are a few more entities the Fire_cci system interacts with as

shown in Figure 3.1.

Figure 3.1: System context of the Fire_cci system with team, data providers, and other entities

The entities of the context are:

The Fire_cci team as provider of processors, high level requests to produce

certain results and feedback, and as first consumer of results, among them test

datasets and intermediate results for assessment. Note that for this version of the

system it is assumed that the Fire_cci team provides the validated products to

users (not the processing system). The Fire_cci CRG, as part of the Fire_cci

team, contributes to product quality control and assessment by intercomparisons.

Data providers of the required satellite input data, in particular ESA.

Other ECV projects as both providers of certain auxiliary datasets, among them

the CCI Land Cover product (LC_cci) for a background land classification map,

and also LC_cci as consumer of the BA product.

The CCI portal as a distinct receiver of the validated BA product.

Other data users, such as GCOS or Copernicus.

The common part of the processing infrastructure (i.e. the Calvalus system of

Brockmann Consult, and JASMIN).



Issue 1.5 Date 30/01/2018

Page 10

The interfaces towards these entities partially determine the Fire_cci system in addition

to its main function that is subject of the next section.

3.2. Main function and processing chain

The main function of the Fire_cci system is the repeated generation of the BA products

with different input datasets and algorithms which undergo phased updating.

Functions of data management as well as sharing help to cope with the large amount of

input data. Functions for processor integration, configuration control and versioning

help to support the continuous development of the Fire_cci project.

3.3. System requirements

System requirements are usually derived from use cases, user requirements, and other

inputs on the basis of a first decomposition of the system into elements. In the CCI

projects, the focus is mainly on the products to be generated, rather than on the system,

which is only the means to produce the product but not a deliverable itself. Therefore,

the main requirement for the system is to generate the product according to the ATBDs

and the PSD. The relationship among the different requirements is shown in Figure 3.2.

system requirements

community requirements

Figure 3.2: Requirements derivation logic with system requirements mainly derived from PSD

The Statement of Work (SoW) [AD-1] lists 16 requirements, a few of them within the

scope of the system development, in particular:

R-7 (algorithm improvements, re-processing capabilities, extension of the data

record)

R-13 (Sentinel data)

R-14 (system dimensioning and sustainability)

This leads to a first set of system requirements, as detailed in Table 3.1.

.



Issue 1.5 Date 30/01/2018

Page 11

Table 3.1: System requirements in the SoW

ID Title Requirement Source

010 Reprocessing

capability

The Fire_cci system shall be able to (re)process

complete missions of EO data.

SoW R-7

020 Processor

improvement

cycle

The Fire_cci system shall support the

improvement cycle of processors and provide

configuration control for processor versions

SoW R-7

030 Sentinel data

handling

The Fire_cci system shall ingest and process

Sentinel data

SoW R-13

040 Initial capacity

and resources

The Fire_cci system shall provide space for 150

TB inputs and 100 TB outputs at a time and

provide 200 cores for concurrent processing.

SoW R-14

050 System

scalability

The Fire_cci system shall be scalable by adding

new hardware, without change in the architecture.

SoW R-14

060 Additional

missions

The Fire_cci system shall be extendable for

additional missions by adding the corresponding

processors

SoW R-13

SoW R-14

The PSD [RD-1] identifies and specifies two products to be generated:

The “Pixel BA product”

The “Grid BA product”

The products shall further be compliant with the CCI data guidelines [AD-4] regarding

format (NetCDF-CF), naming, and metadata. This leads to another set of system

requirements, detailed in Table 3.2.

Table 3.2: System requirements specified in the PSD and CCI data guidelines


110 Pixel BA

product

The Fire_cci system shall generate the Pixel BA

product according to the specification in the PSD.

PSD

120 Grid BA

product

The Fire_cci system shall generate the Grid BA

product according to the specification in the PSD.

PSD

130 CCI data

guideline

compliance

The Fire_cci system shall generate NetCDF

products compliant in naming, metadata and

format to the CCI data producer recommendations

and to the CF convention.

CCI data

guidelines

The ATBDs [RD-3], [RD-6], [RD-7] identify algorithms to be integrated as processors

into the Fire_cci processing system:

Pre-processing processors

BA processors

This leads to another set of system requirements, detailed in Table 3.3.



Issue 1.5 Date 30/01/2018

Page 12

Table 3.3: System requirements specified in the ATBDs


210 Pre-processing

processors

The Fire_cci system shall integrate processors for

pre-processing to generate surface reflectance (SR)

products from Level 1 inputs according to the

ATBD.

[RD-3],

[RD-6]

220 BA processors The Fire_cci system shall integrate BA processors

to generate BA products from SR products

according to the ATBD.

[RD-3],

[RD-6],

[RD-7]

The DARD [RD-2] lists the input datasets required for Fire_cci processing (see Table

3.4). The system must be capable to handle them.

Table 3.4: System requirements specified in the DARD


310 MERIS FSG The Fire_cci system shall handle MERIS FSG data. DARD

320 MODIS SR The Fire_cci system shall ingest and handle MODIS

SR product data (MOD09Q1, MYD09Q1,

MOD09GQ, MYD09GQ)

DARD

330 Sentinel 3

OLCI

The Fire_cci system shall ingest and handle OLCI

Level 1 and Level 2 data.

DARD

340 Sentinel 3

SLSTR

The Fire_cci system shall ingest and handle SLSTR

Level 1 and Level 2 data.

DARD

350 Sentinel 2 MSI The Fire_cci system shall ingest and handle Sentinel-

2 MSI L1C data of Africa.

DARD

360 Landsat 8 The Fire_cci system shall be able to ingest and

handle Landsat 8 data as fallback for Sentinel 2 (in

case of missing coverage).

DARD

370 PROBA-V The Fire_cci system shall ingest and handle PROBA-

V S5-TOC data.

DARD

4. Fire_cci system architecture

This section describes the Fire_cci system architecture. It first provides an overview

over the high-level decomposition (4.1) and provides a justification why the

decomposition is needed. In the later sub-sections of this section, the components are

described in-depth: section 4.2 explains the Calvalus sub-system, and section 4.3

explains the JASMIN sub-system.

4.1. High-level system decomposition

In order to produce the BA dataset based on MERIS input data, the Calvalus system has

been used. According to the original plan, the MODIS-based BA dataset should have

been produced on Calvalus as well. However, during the project it was identified that

the system requirement 320 in Table 3.4 (“The Fire_cci system shall ingest and handle

MODIS SR product data (MOD09Q1, MYD09Q1, MOD09GQ, MYD09GQ)”) could

not be easily fully fulfilled by the Calvalus system, which has been used for producing

the MERIS global time series. Although the system would be able to handle that data, it

was not able to ingest the data due to limited bandwidth and the fact that NASA does



Issue 1.5 Date 30/01/2018

Page 13

not send the data on physical devices. As a consequence, some alternative high-

performance computer clusters have been evaluated for the production of the MODIS

global time series, with the decision matrix depicted in Figure 4.1 as outcome.

Figure 4.1: Decision matrix for MODIS processing. + indicates good, ++ indicates very good, o indicates average, - indicates poor, -- indicates very poor.

The considered high-performance cluster solutions are the cluster at IT4Innovations1,

the CEDA JASMIN2 system, Google Earth Engine

3, and Calvalus. Although Google

Earth Engine already had the data available, and boasts a very high performance for free

(as Fire_cci is a scientific project), the effort to convert the MODIS BA algorithm to a

compatible version was considered too high.

The only real drawback of the JASMIN system was that the storage space was limited,

which means that the processing needed to be done in cycles, which was acceptable.

IT4Innovations offered good download speed as well as sufficient storage and

performance, but the fact that team members have experience with JASMIN and know

that it is reliable and simple to use made it the best choice for the MODIS BA

processing. Thus, the overall Fire_cci system is composed from both the Calvalus

system and JASMIN.

The following sections address the architectures and technical concepts and methods

used for either sub-system.

4.2. Calvalus system

The Fire_cci system is to a large extent based on Calvalus and the underlying

infrastructure, with certain elements specifically configured and extended for Fire_cci

(Figure 4.2). In addition to the software stack with many functions of the Fire_cci

system provided by Calvalus, Hadoop or other frameworks there is a shared cluster

infrastructure that runs the corresponding services.

1 http://www.it4i.cz/?lang=en

2 http://www.ceda.ac.uk/services/analysis-environments/

3 https://earthengine.google.com/

http://www.it4i.cz/?lang=en

http://www.ceda.ac.uk/services/analysis-environments/

https://earthengine.google.com/



Issue 1.5 Date 30/01/2018

Page 14

productioncontrol

input dataauxiliary data

outputs

Linux(compiler)runtimes

HadoopSNAPGPF

BEAMGPF

other runtimes(Python, Java, Matlab,

GDAL, Docker)

Fire_cci processing systempreproc. and BA workflows

Fire_cci requests

BAprocessors

pre-processingprocessors

Fire_cci-specific elements

Calvalus

dataorganisation

processoradapters

productiontypes

portalcmd line

client

Frameworks layer

Tom

cat

Figure 4.2: Fire_cci system architecture layers with specific elements, Calvalus, and Hadoop

The main elements that are specific to Fire_cci are:

Access to the shared input data (MERIS FSG), space for auxiliary data and

Fire_cci project intermediates and outputs.

Processors or Fire_cci-specific processor configurations of existing processors

for pre-processing.

The BA processors and their wrappers for integration into Calvalus.

The Fire_cci processing system instance on a virtual machine for the control of

all Fire_cci workflows, with rules and request templates for pre-processing, BA

processing, and formatting and all the respective dependencies.

In addition to this there are several Fire_cci-specific configurations of shared elements,

in particular:

A “fire” queue in the Calvalus Hadoop scheduler for the adequate and fair

sharing of the cluster processing resources.

space on the shared Brockmann Consult’s (BC) FTP server

The main shared elements and functions used from Calvalus and underlying

frameworks are:

The Calvalus data organisation for earth observation (EO) data, auxiliary data,

and project-specific data, and the corresponding functions of the Hadoop

Distributed File System (HDFS) for data management.

The Calvalus production control tool pmonitor for the control of processing

chains and bulk production, in combination with the Calvalus production types

for massive-parallel processing, and Hadoop HDFS and YARN for fair

scheduling, load balancing, and robust failure handling for different layers of

production management.

The Calvalus processor adapters to wrap the Fire_cci processors, in particular

the adapter for Matlab runtime, but also SNAP for the processors for pre-

processing, and the corresponding automated deployment and runtime

provisioning for processor integration.



Issue 1.5 Date 30/01/2018

Page 15

The Calvalus framework for MapReduce, which empoys the Hadoop

MapReduce framework, and provides the basis for generic MapReduce

algorithms. This is used for the final formatting step.

How these elements and functions are used is described in more detail in Section 4.2.1.

4.2.1. Technical methods and concepts used in Calvalus

Calvalus is proprietary software developed by Brockmann Consult GmbH since 2010. It

employs the Apache Hadoop and adapts it to the processing of Earth Observation data.

It provides the possibility to perform full mission EO data processing, data aggregation,

validation, and value-adding with many frequently updated algorithms and data

processors on standard hardware scalable for the amount of data of Sentinel 1-2-3.

HDFS

HDFS, the Apache Hadoop distributed file system, is a distributed and highly scalable

file system. A typical cluster running HDFS has a single master plus multiple

datanodes. Each datanode stores blocks of data, and serves them over the network. This

file system is based upon the Google File System (GFS), introduced by Google

employees in 2003.

HDFS is designed to store large files, typically in the range of gigabytes to terabytes

across multiple machines. This data is replicated across multiple nodes in order to

ensure data availability. Data nodes communicate in order to rebalance data, to move

copies around, and to keep the replication of data.

The main advantage of HDFS is that it allows for data-local processing (see Figure

4.3¡Error! No se encuentra el origen de la referencia. and Figure 4.4). This means

that the scheduler distributes processing jobs to nodes with an awareness of the data

location. For example: if node A contains data X and node B contains data Y, the job

tracker schedules node B to perform processing tasks on Y, and node A scheduled to

perform processing tasks on X. This considerably reduces the amount of traffic that

goes over the network.

Figure 4.3: Archive-centric

Figure 4.4: Data-local

Software bundles

A software bundle is a Calvalus concept that allows system operators to deploy any

runnable software to the system. The processors used in Fire_cci, which are provided by



Issue 1.5 Date 30/01/2018

Page 16

project partners, are integrated into software bundles, and as such integrated into the

system.

Hardware Infrastructure

The infrastructure used for Fire_cci processing is a Calvalus/Hadoop cluster consisting

of computing hardware, storage, input/output elements, and network. This cluster is

shared with other projects that each provide parts of the hardware and get in turn a

corresponding share of the resources of the cluster. The current cluster has 98 nodes and

a storage capacity of about 1.6 PB.

Figure 4.5 shows the layout of the Calvalus/Hadoop cluster with master node,

computing and storage nodes, the feeder as input/output element, and an optional test

server for services and development. The computing and storage nodes are simple

computers with local disks. These disks together are the distributed storage of the

cluster managed by the Hadoop Distributed File System.

Figure 4.5: Calvalus architecture

The elements of this infrastructure are described in detail in Table 4.1.

Table 4.1: Calvalus infrastructure elements

Infrastructure

elements

Type Number, size

Computing

nodes

Rack-mounted standard computers, 1U,

1 quadcore CPU + hyperthreading, 16

GB main memory, 4 hard disks (11-16

TB), 1 Gbit network connection

(there are other nodes with 8 GB main

memory and one node with two 6-core

CPUs and 128 GB main memory in the

same cluster)

98 nodes in the cluster (overall

concurrency of ~550)

Storage As listed above the nodes have local

disks. They are altogether managed by

the Hadoop Distributed File System

HDFS.

Overall storage capacity of the

cluster is ~1.6 PB. The cluster is

used with a replication of 2 for

inputs and 1 for outputs which



Issue 1.5 Date 30/01/2018

Page 17

Infrastructure

elements

Type Number, size

In addition there are offline backups of

input datasets at Brockmann Consult

GmbH.

reduces this capacity

accordingly. Sharing of input

data among projects mitigates

the need for storage for all data.

I/O elements “feeder”: (empty) disk array for 24 hot-

pluggable hard disks, 4U, quadcore

CPU, 8 or 16 GB main memory, 10 Gbit

network connection.

FTP server (ftp://ftp.brockmann-

consult.de/), used for test and validation

data exchange with UAH.

2 feeders for concurrent reading

or writing of 48 disks.

The FTP server is shared with

other projects of BC. It is a

virtual machine (VM) on a

redundant VM server.

Network

infrastructure

There is a 10 Gbit backbone linking 4

switches and the 2 feeder nodes for data

I/O. Each switch provides 24 1-Gbit

ports that connect a set of computing

nodes.

The cluster is connected with 1 Gbit to

the BC internal network.

The FTP server is connected to the BC

internal network as well as to the

internet.

A 3-layers firewall protects the internal

production network from the internet.

The FTP server is in the outermost layer

accessible from the internet. The

Calvalus cluster is in the innermost

layer.

4 switches 10 Gbit/1Gbit

1 Gbit connection of cluster

nodes

10 Gbit connection between

switches and to feeders

Internet connection via the

backbone of the Geesthacht

Innovation and Technology

Centre (GITZ).

Operating

system

Ubuntu 14.04 LTS

This approach of Hadoop as middleware and with computing nodes that are at the same

time storage nodes has been selected for Fire_cci because the knowledge of the location

of the respective data allows for data-local processing with minimal use of the network,

which makes the approach suitable for massive parallel processing of the large data

volumes of pre-processing. The approach is scalable to several petabytes which is an

advantage regarding the considerable data volume in later stages of Fire_cci.

Software infrastructure

The software system for Fire_cci processing deployed on the infrastructure above is

shown in Figure 4.6. The different layers of the software system start from the operating

system up to the individual processors. Note that the MERIS pre-processing is done

using an adaptation of the LC_cci pre-processing algorithm, but creating daily

composites.



Issue 1.5 Date 30/01/2018

Page 18

Linux OS

Java Runtime

Hadoop HDFS + Job control

Calvalus production types, adapters, MapReduce

implementations

Fire_cci processing

system instance

Python

Interpreter

BEAM

MERIS BA

algorithm

AMORGOS

processor

LC pre-processing algorithm

nccopy

NetCDF

converter

Calvalus client

Sen2CorSentinel-2 BA

algorithm

Figure 4.6: Software architecture

The layers are:

Basic software (shown in grey) comprises the Linux operating system, the Java

runtime, and the Python interpreter.

Calvalus (shown in green) provides several layers, with a client layer that uses

Hadoop and a layer plugged into the Hadoop framework with production types

and adapters, both together implementing the Earth Observation functions not

available in bare Hadoop. This part also contains the formatting tool MapReduce

implementation.

Apache Hadoop with its distributed file system and its job scheduling functions

and cluster tools as well as ESA SNAP with its graph processing framework are

layers used by Calvalus for cluster processing. They are shown in blue.

Third-party data processors used within the Fire_cci processing are AMORGOS

and a NetCDF tool shown in pink. They are integrated as Unix executables into

Calvalus for parallel execution.

Specific elements developed or assembled for Fire_cci are the LC pre-processor,

the MERIS BA algorithm, Sen2Cor, the Sentinel-2 BA algorithm, and the

Fire_cci processing system instance. They are shown in red. They will get

updates to cover additional sensors, in particular PROBA-V and Sentinel-3

OLCI and SLSTR.

Like the hardware, the software is also for the most part shared with other projects. But

this needs more precise consideration. It is not always the same version of a software

item that is used by different projects. Therefore, the versioning approach has a key role

in the software deployment and runtime use for Fire_cci in the Calvalus environment:

Several versions of software items can be deployed in this environment at the

same time. The actual requests generated by the processing system instance

determine which combinations of versions are actually used. This is the case for



Issue 1.5 Date 30/01/2018

Page 19

processors (upmost layer) but also for BEAM/SNAP and Calvalus and the

processing system, optionally also for Java and Python. Only the operating

system and the Hadoop version being used is a single one at a time.

The software items are versioned and the versions are managed in version

control systems. For software items developed by Brockmann Consult the

version control system is Git. The software repositories are hosted on GitHub, a

cloud service. Some of the repositories are public, e.g. the one for SNAP which

is open source.

Other software items are version-controlled externally. Examples are Apache

Hadoop maintained as open source by Apache Foundation, the programming

language runtimes, or the AMORGOS processor provided by ESA and

maintained by ACRI-ST.

Table 4.2 lists the software items with their role in the Fire_cci processing subsystem

and its configuration control.

Table 4.2: Software items for Fire_cci processing

Software item Purpose, use Configuration control

Java runtime

environment

Basic software used for Hadoop, Calvalus,

ESA SNAP, some processors

Oracle SDK, Version 8

(1.8.0_40-b25), available

from

http://www.oracle.com/te

chnetwork/java/index.ht

ml2

Python interpreter Basic software used for pmonitor Version 2.7.6, available

from www.python.org

Apache Hadoop Cluster software with data management

HDFS, job scheduling, command line tools

and Web operating interface, several APIs:

client side for job submission and

monitoring, server side for plug-in of map

and reduce tasks.

Version 2.7.3, versions

maintained by Apache,

available from

hadoop.apache.org2

(open source)

Calvalus Earth Observation application layer on top of

Hadoop with HDFS archive directory

structure with archiving rules, client tool for

request submission, software deployment,

data ingestion processor adapters for various

programming languages (among them for

BEAM/SNAP operators processor

deployment as versioned software bundles).

User portal for on-demand processing.

Processing system instances for controlled

bulk production.

Version 2.7-SNAPSHOT,

version control on

GitHub in private

repository of Brockmann

Consult GmbH

ESA SNAP Framework for processor development and

execution, concepts for product, reader,

writer, operator, operator chaining, tile cache

and many more, basic software for LC pre-

processor, Level 3 aggregator, and overlap

determination.

Version 5.0, provided by

ESA, maintained and

further developed by

Brockmann Consult

GmbH, version control in

public repository of

Brockmann Consult

GmbH (open source).



Issue 1.5 Date 30/01/2018

Page 20

Software item Purpose, use Configuration control

Daily quicklook

aggregation

Generates quicklooks with global extent of

the input products of a day, uses L3

aggregation with sub-sampling, and final

JPEG formatting. Used for input screening

for quality checks.

Maintained as part of

Calvalus.

AMORGOS Improved geocoding of MERIS products

based on DEM, attitude and orbit files and

knowledge on acquisition features, adds

corrected Lat and Lon bands to MERIS

products.

Version 4.0p1, provided

by ESA, maintained by

ACRI-ST.

Fire Level 3

aggregator

(production type

“FireL3Productio

nType”)

Aggregation of surface directional

reflectance (SDR) values, reprojection,

application of a temporal cloud filter, tiling

and NetCDF-formatting of the result.


Calvalus.

Fire SR product

writer

This SNAP product writer implementation is

a modification to the LC_cci product writer,

which in turn extends the NetCDF 4 CF

writer to set band names and attributes,

global attributes, and the file name according

to the LC_cci product specification, on which

the MERIS BA retrieval is based. It is used in

the generic Calvalus product formatting step.


Calvalus.

Fire Formatting

Tool

Uses the burned area data in order to create

PSD-compliant, formatted final user data

sets.


Calvalus.

nccopy NetCDF

conversion tool

Used to convert SNAP-generated NetCDF 4

files into classic format as required by CCI

product convention

NetCDF library version

4.1.3, available for

Ubuntu via apt-get.

Quicklook

generator

Uses the generic L3 workflow to mosaic the

tiles into one quicklook JPEG image.


Calvalus.

fire-inst Processing system instance with workflow

control and processing progress and status.

Automated daily backup,

no version control,

instead re-configuration

for each production run.

fire-1.x Software bundle containing the BA processor

and the environment it uses.

Versioned; each update

gets a new version, while

old versions are kept and

can be re-used.

4.3. JASMIN system

JASMIN is described in-depth in [RD-4]. It employs the LSF software for scheduling

and running processing jobs, which is described in [RD-5]. From a Fire_cci perspective,

it is vital that the system is able to fulfil system requirement 320 (“The Fire_cci system

shall ingest and handle MODIS SR product data (MOD09Q1, MYD09Q1, MOD09GQ,

MYD09GQ)”), and allow to transfer the results.

4.3.1. Technical methods and concepts used in JASMIN

This section describes the environment on JASMIN from a client’s perspective only, as

this is the perspective we used in the Fire_cci project.



Issue 1.5 Date 30/01/2018

Page 21

Hardware environment

As described in [RD-6], the cluster consists of 218 processing hosts with 4764 CPUs

and > 50 TB physical RAM.

Job submission and processing environment

JASMIN offers to submit commands to the LOTUS cluster, which are scheduled and

executed by that engine. The commands which are submitted may be any executable

which is able to run within the prepared environment. Hence, in order to run a specific

task (such as the burned-area-retrieval for MODIS), it is necessary that the executable is

prepared in the user’s home directory, and it is ensured that all shared libraries are

available. If these prerequisites are fulfilled, the execution command may be submitted

to the LOTUS cluster.

5. Workflows

This section describes the processing workflows in detail, the data they use, the

techniques employed, and the outputs they create.

5.1. MERIS global time series processing

There is a sub-section for each processing step: Section 5.1.1 deals with the pre-

processing (which is basically the LC_cci pre-processing), Section 5.1.2 explains the

MERIS BA retrieval, and Section 5.1.3 describes the final step, the formatting.

Figure 5.1 shows a high-level view of the Fire_cci processing chain for MERIS.

The three main steps are pre-processing, BA retrieval, and BA formatting:

Starting from MERIS full-swath geo-corrected data (FSG), the pre-processing

generates a global surface reflectance (SR) product. Pre-processing includes

cloud screening, geometric and radiometric correction, aerosol retrieval and

atmospheric correction, and compositing into 10-degree-tiled daily composites.

It is explained more detailed in Section 5.1.1.

The time series of SR products serve as input to the MERIS BA retrieval which

takes a year +/- two months of data of a single tile as input and generates

monthly composites of BA detection dates. It is explained in more detail in

Section 5.1.2.

The tiles of the monthly composites are combined by a formatting step that

creates two output products. The first one is a monthly composite subdivided

into 6 different zones with temporal information on the start of burning and

other variables (see [RD-1]). It is called “pixel product”. The second output is a

set of bi-weekly global composites of a coarser resolution (0.25 degree) with

parameters on the coverage and characterisation of the burned area. It is called

“grid product”. Details are explained in the Product Specification Document

([RD-1]). These product outputs serve as the actual user data.



Issue 1.5 Date 30/01/2018

Page 22

Figure 5.1: High-level view of Fire_cci MERIS processing chain

5.1.1. MERIS Pre-processing

This section describes the workflows to generate the surface reflectance (SR) product,

which serves as input to the MERIS BA retrieval step.

The inputs for the Fire_cci MERIS pre-processing are MERIS FSG products, which

have been produced by applying the AMORGOS geo-correction software. The main

steps of the workflow in Fire_cci are the Level 2 processing, which reveals surface

directional reflectance (SDR) products (one product for each Level 1 input), and

aggregation, which reveals the daily SR composites.

Level 2 processing is the computationally most time-consuming and the most complex

step of pre-processing. The graph of operators chained for pre-processing of MERIS is

depicted in Figure 5.2.

The rectangular boxes starting from radiometric correction down to atmospheric

correction form the Level 2 processor. They are mainly implemented as (SNAP)



Issue 1.5 Date 30/01/2018

Page 23

operators chained in-memory using the SNAP Graph Processing Framework. Map

projection and compositing is implemented as concurrent aggregation utilizing the

Hadoop MapReduce framework, and controlled via Calvalus.

Table 5.1 explains the steps with their transformation and multiplicity. Multiplicity is

expressed in terms of Hadoop jobs and tasks. Jobs are the granularity that is used by the

processing system that controls production. It submits one job e.g. for a certain month to

be processed. Calvalus splits this job into tasks, e.g. one task for one input product. The

tasks are executed in a massive parallel fashion, each task in a process on some node of

the cluster.

Production is controlled by the Fire_cci processing system instance. A control script is

configured for both Level 2 processing and aggregation. It can be parameterised for the

time period (complete mission or only some years or months), and optionally the steps

to be performed. Production can be interrupted and resumed. Steps can be repeated if

required.

The MERIS pre-processing runs in a highly parallel fashion. Using the principle of data

locality, the Level-2 part of the processing – which is everything before the last step

“Compositing and Mosaicking” – is run on a cluster node that contains the input data.

This is run in parallel for sets of input data. The last step is also run in parallel for sets

of input data, but data locality cannot be guaranteed, because the input data for the

aggregation cannot be guaranteed to reside on a single node.

Errors are handled in a flexible way. If a processing step fails, it is re-tried on a different

node up to 3 times. This is done because the assumption is that the specific environment

on the node (processing load, memory usage) may be responsible for the failure, and by

performing the processing on a different node, the process may be successful. Only after

the fourth failed attempt, the processing step is considered a failure. If such failures

occur, the system engineers are informed on a specific processing overview webpage,

and exhaustive log files are generated and made available to the system engineers. In

any case, since the pre-processing progress is mature, and the input data has been

exhaustively quality controlled already in the LC_cci project, failures in this step are

expected to be very rare.

Table 5.1: Pre-processing steps

Step Transformation Multiplicity, cycles

Level 2

processing

SDR retrieval with radiometric

and atmospheric correction,

aerosol retrieval, pixel

identification

One task per Level 1 input product,

submitted as one job per month (~1200

files), ~120 jobs in total

Repetition of a single job in case of

failure, automated detection of

successfully transformed products, thus

only reprocessing of missing outputs

Level 3

processing

Spatial and temporal aggregation.

Called two times with different

parameters, first for the temporal

cloud filter (longer interval of ~1

month), second for SDR

aggregation and SR retrieval

(daily).

Two jobs per aggregation period, one for

the temporal cloud filter and one for the

daily composites, each job with as many

tasks as there are input products. ~520

jobs in total.

Repetition of a single job in case of

failure.



Issue 1.5 Date 30/01/2018

Page 24

Step Transformation Multiplicity, cycles

The outputs are provided as tiles

(global, 10 degrees) in the desired

Fire_cci NetCDF output format

including all required metadata.

Figure 5.2: MERIS Pre-processing

5.1.2. MERIS Burned Area retrieval

The Burned Area retrieval for MERIS is done using the MERIS BA algorithm

developed by UAH (see [RD-3]), which has been written in MATLAB. In order to run

MATLAB code on a different machine, there are two options: a) acquire a MATLAB

licence for the machine, b) compile it to an executable and run it using a special piece of

software, the MATLAB Runtime. As option (a) is not feasible on a distributed system

such as Calvalus (because for each of the 90 nodes, a licence would have to be

acquired), option (b) was chosen for the Fire_cci project.



Issue 1.5 Date 30/01/2018

Page 25

The algorithm itself runs for a single-tile time series of one year of input data produced

by the pre-processing, and also needs for each year the surrounding two months of the

previous and the following year. It creates a GeoTIFF-file for the input tile, one for each

month. Also, it creates exhaustive debugging information for each tile, which is used by

UAH for quality control. It is called for each of the LC_cci 10° global tiles.

As auxiliary data, it uses a binary map (burnable/non-burnable), which has been

produced by UAH, based on a reclassification of the LC_cci map, and hotspot

information derived from the MODIS Programme, provided as shapefiles, also provided

by UAH.

Figure 5.3 displays the actors in the MERIS BA retrieval.

Figure 5.3: MERIS BA retrieval

From the view of the Fire_cci processing system, the BA retrieval basically is one

hermetic step: the executable provided by UAH is called and produces the output. In

order to run this single step, however, the correct environment must be provided.

First of all, the correct version of MATLAB Runtime needs to be employed. In the case

of the MERIS BA retrieval algorithm v4.1, the MATLAB Runtime version is 2012b.

Also, a number of additional MATLAB modules need to be provided and added to the

LD_LIBRARY_PATH environment variable. The processor expects the input data as

well as the auxiliary data to reside in a specific folder structure.

In order to set up the environment, a Calvalus Production Type (Fire-MERIS-BA) and a

bash wrapper script (meris-ba-process.vm) have been created. The Calvalus Production

Type predominantly takes care of providing the input data in the expected structure and

collecting the output data after the processing has been done. The bash wrapper script

sets up the LD_LIBRARY_PATH, runs the MATLAB code, zips up the results and

marks them as output for the Production Type. If errors occur, the process fails, so it can

be re-done, and it is configured to automatically try four times before ultimately failing.

Similar to the pre-processing, the production is controlled by the Fire_cci processing

system instance. The same control script used for pre-processing can also be configured

for the MERIS BA processing. Hence, it can also be parameterised for the time period



Issue 1.5 Date 30/01/2018

Page 26

(complete mission or only some years or months), the production can be interrupted and

resumed, and steps can be repeated if required.

The MERIS BA processing runs in a less highly parallel fashion compared to the pre-

processing. This is due to the structure of the algorithm: it combines a set of input data –

as opposed to the pre-processing, where single input files are processed – and

aggregates them. For this reason data locality cannot be guaranteed. Still, the process is

run in parallel: the BA processing for a single tile, and for one year of input data, runs

on a single node, which implies that the maximum degree of parallelisation is the

number of tiles.

The error handling is done in the same way as error handling in the pre-processing (see

Section 5.1.1). If error message is memory-related, the process has failed due to the

processing environment on the specific node. The reaction is to consider the attempt a

failure, and re-try the same processing step on a different node.

Single attempts fail in ~2% of the attempts of performing BA retrieval. Due to the re-

tries of the single attempts and the error handling described above, though, the overall

failure rate is 0%.

5.1.3. MERIS PSD-compliant formatting

The PSD-compliant formatting of the BA data has been implemented in order to exploit

the massive parallelity of the Calvalus system. This implementation has been employed

to generate the user data compliant to the Phase-2 PSD. The formatting step works on

single months, and takes as input all global tiles for each month.

Grid processing

The grid processing basically consists of both a mapper and a reducer implementation.

The mapper implementation works in parallel on each available tile; it does the

aggregation of the burned area and computes the errors as well as the patch numbers,

the fractions of observed area, and the burned area per Land Cover class. Since there are

two grid products for each month, the mapper aggregates the data accordingly.

The reducer implementation retrieves the output of all the mappers, creates the two

result files (one for each half of the respective month), fills them with metadata, and

writes the mapper’s output into them. These result files are archived afterwards, ready

to be delivered to the users.



Issue 1.5 Date 30/01/2018

Page 27

As opposed to the approach described in

version 1.0 of this document, this

MapReduce approach is much closer to

the general concept of Calvalus,

allowing for a high degree of

parallelisation. So, formatting the whole

time series takes about half an hour, as

opposed to ~48 hours (for both grid and

pixel formatting) in the former

approach.

Error handling in the grid processing is

done using the Calvalus system’s error

handling protocol. Generally, no errors

are accepted, as the process is expected

to successfully use all available burned

area data. Consequently, the overall

failure rate of the process is 0%.

Pixel processing

Due to the higher resolution of the

resulting products, the pixel processing

has higher demands on memory

consumption. So, the process has been

split up twice: the first step computes

the output variables separately in a

MapReduce step, and the second step

puts them together. See Figure 5.4 for

an overview. The process runs for a

single month and one of the six target

areas defined in the PSD.

In the first step, a mapper

implementation takes each tile and

creates the respective output for one of

the three output variables day of year,

confidence level, and LandCover class.

So, it is run three times for each tile in

parallel. The pixel output is generally

equivalent to the input.

The PixelReducer runs once for each

output of the mapper. It creates actual

files in the file system (as opposed to

the PixelMapper, which keeps the data

in memory).

Finally, the PixelMergeMapper collects

these data, and creates the result product

and the respective metadata.

It is also true of the pixel processing

that this MapReduce approach is much

closer to the general concept of Figure 5.4: Formatting



Issue 1.5 Date 30/01/2018

Page 28

Calvalus than the approach described in version 2.0 of this document, and thus also

allows for a high degree of parallelisation. So, formatting the whole time series takes

about two hours, as opposed to ~48 hours (for both grid and pixel formatting) in the

former approach.

Error handling in the grid processing is done using the Calvalus system’s error handling

protocol. Generally, no errors are accepted, as the process is expected to successfully

use all available burned area data. Consequently, the overall failure rate of the process is

0%.

5.2. MODIS workflow

The MODIS workflow differs strongly from the MERIS workflow in two key aspects:

first, the basis for the processing is already pre-processed data, which means that no pre-

processing is done within the context of Fire_cci. Second, the input dataset has not been

available on the system at the start of the processing, meaning that it had to be acquired

first. Two datasets needed to be acquired to the system for processing: MOD09GQ

(daily surface reflectance data, global, 250m, 1200*1200 km tiles), and MOD09GA

(daily surface reflectance data, global, 1km and 500m SIN Grid).

A general issue was the limited space of the platform. As the whole input data amount

sums up to about 200 TB data, while only 84 TB are available on the platform, and ~10

TB of space had to be left for temporary files and results, the processing had to be done

in cycles. So, we have split the processing into continent datasets (Africa, Asia,

Australia + Europe, North America, South America), and used the following pattern:

1) process tiles of continent A

2) in parallel, download tiles of continent B

After 1) and 2) are done:

3) remove tiles of continent A

4) process tiles of continent B

5) in parallel, download tiles of continent C

… and so on, until all continents are processed.

Figure 5.5 shows the logical ordering of the workflow; the single steps are explained in

the upcoming subsections.



Issue 1.5 Date 30/01/2018

Page 29

Create symbolic links to MOD09GA

data

Download MOD09GQ data

Check completeness of MOD09GA data

Download missing MOD09GA data

Run MODIS BA processor

Run repackaging tool

Input data complete

BA data complete

Download BA data to Calvalus

Run PSD formatting

PSD data complete

Figure 5.5: Sequence of MODIS processing actions

5.2.1. Data acquisition

As stated before, there are two datasets which needed to be acquired to the system for

processing: MOD09GQ (daily surface reflectance data, global, 250m, 1200*1200 km

tiles), and MOD09GA (daily surface reflectance data, global, 1km and 500m SIN Grid).

In order to reliably download the MOD09GQ dataset, a python script has been set up,

which allows to start multiple (typically ~40) download jobs in parallel, tracks

successfully downloaded files, compares checksums of the downloaded files with the

checksums provided by the download site, and reports failing download attempts. Thus,

it is able to handle broken connections, incomplete downloads, and the regular and

irregular system downtimes. This script has been run on a dedicated transfer node of the

JASMIN system, which allowed for high download rates.

Large parts of the MOD09GA dataset had already been acquired by another working

group and could be re-used. Hence, in order to acquire the MOD09GA dataset, three

steps were necessary:



Issue 1.5 Date 30/01/2018

Page 30

1) Create symbolic links in the filesystem, so that the existing MOD09GA files are

visible to the BA retrieval process. In order to do that, a simple bash-script has

been created and run once.

2) Identify the missing MOD09GA files, as the archive turned out to have small

gaps, and was also missing 2016 completely. This could also be done with a

short bash-script, which checked the existence of input files against the expected

list of input files.

3) Download the missing MOD09GA files. In order to do that, a slightly adapted

version of the MOD09GQ download script has been used.

5.2.2. Data processing

The MODIS BA processor has been described in [RD-7].

In order to run the MODIS BA processor on JASMIN, the environment had to be

prepared accordingly. The MODIS BA processor has been written in Python2, and has a

set of dependencies which are not all fulfilled by the Python version already installed on

JASMIN, such as gdal, ogr, scipy.spatial, or cv2. Thus, another version of

Python2 has been prepared, which contains all the needed dependencies. Prior to the

processing execution, a dedicated activate-script has to be called, which sets up the

environment and ensures that the correct python version is being used.

The processing has been split up into MODIS tiles and into years, which leads to a large

number of processing tasks. In principle, the LOTUS system accepts the submission of

a large number of processing tasks at once, but in order to keep the Calvalus way of

monitoring, it was decided that the submission of jobs was done by a python script

using the same monitoring approach as the MERIS processing. This required the

implementation of three dedicated scripts, ordered bottom-up:

1) run.sh. This bash script is the wrapper around the MODIS BA processor. It sets

the environment (source mypython/bin/activate), runs the python

executable with the main function of the BA processor for the given tile and the

given year (mypython/bin/python2.7 modis-proc/MAIN.py $tile

$year), and returns the exit code of the BA processor to the start.sh script.

2) start.sh. This bash script does the actual submission of a processing job to the

LOTUS cluster by submitting the run.sh script with the necessary parameters

(bsub -o $logfile -W 24:00 -M 20971520 ./run.sh $tile

$year). Furthermore, it checks once every five minutes if the job is finished; if

it is finished, it checks if the job has finished successfully. If so, it exits with exit

code 0, signaling success, otherwise it archives the log file so that it may be

analysed, and exits with exit code 1, signaling failure.

3) modis.py. This script controls, starts and monitors the execution of the start.sh

script. According to its exit code, it marks processing attempts of tile and year as

successful or as failure.

5.2.3. Data repackaging and transfer

Since the PSD formatting is done using the Calvalus system instead of JASMIN, the

data needs to be downloaded to Calvalus. The data produced by the MODIS BA

processor contains a high number of image files, which are not used for PSD product

creation, but serve as temporary files or are used for further analysis. In order to

facilitate the download to Calvalus, it is repackaged; this means that the image files

relevant for PSD product creation (= burned area, uncertainty, and fraction of observed

area) are put together into an internally compressed NetCDF-file for each month and

tile. In order to achieve this repackaging, a Java tool has been written, based on the



Issue 1.5 Date 30/01/2018

Page 31

SNAP library. It accepts as input the result of the BA processor for a given tile, year,

and month, and writes as output the internally compressed NetCDF-file. Also, a bash

script has been developed, which acts as wrapper around that Java tool and ensures that

all results are considered, and that the NetCDF files are put into the download area.

On the Calvalus side, a script similar to the script described in Section 5.2.1 has been

put into place, which is responsible for controlling the download of the data.

5.2.4. PSD product production

As the creation of the PSD-compliant products is done on Calvalus, the same approach

is used for MODIS as it is for MERIS data, which is described in section 5.1.3.

5.3. SFD workflow

The workflow for producing the Small Fires Database (SFD) is depicted in Figure 5.6.

The single steps are discussed in the upcoming sections.

Data acquisition

BA retrieval

Pre-processing

BA merging

Sen2Cor results

BA results

Intermediate results

PSD product generation

PSD pixel results PSD grid results

Figure 5.6: Sentinel-2 workflow

Similar to the MERIS processing, the Sen2Cor production is controlled by the Fire_cci

processing system instance. A control script is configured for the pre-processing as well

as BA retrieval and aggregation. It can be parameterised for the time period (full time

series, or only some months), and optionally the steps to be performed; e.g. only pre-

processing may be done on demand. The production may be interrupted and resumed,

and steps may be repeated if required.



Issue 1.5 Date 30/01/2018

Page 32

5.3.1. Data acquisition

A dedicated ingestion service has been set up on Calvalus. It systematically fetches the

latest Sentinel-2 data over Africa from the Copernicus Services Data Hub and checks

the checksum of each single product, so consistency is ensured. On occasional failures,

the product is downloaded again. Furthermore, the ingestion service stores the product

into the right place of the file system, making them available for Calvalus processing.

5.3.2. Pre-processing

The ESA SNAP Sen2Cor processor4 has been identified as well-suited for the

requirements of the Sentinel-2 burned area retrieval algorithm. A dedicated software

bundle has been set up on Calvalus. This bundle contains:

- the Sen2Cor software, which has been put into a Miniconda5 package. This offers the

necessary dependencies without the need to change the Python version which is already

installed on Calvalus.

- the Sen2Cor configuration files.

- wrapper scripts that ensure that the process may be started by the Calvalus system.

5.3.3. BA retrieval

The Burned Area retrieval for Sentinel-2 is done using the Sentinel-2 BA algorithm

developed by EHU (see [RD-6]), which has originally been written in Python. In order

to facilitate running the processor on Calvalus, it has been re-written in Java. It uses

some geographical algorithms, which are not available by common Java libraries, so

these had to be implemented as well.

The algorithm itself runs for a single-tile of input data produced by the pre-processing.

First, it creates an intermediate NetCDF file containing the BA retrieval of the single

input tile. The latest four outputs for each tile are then merged afterwards, in order to

give consistent images.

Figure 5.3 displays the software actors in the Sentinel-2 BA retrieval: the core software

uses the Java Runtime libraries. Some generic libraries had to be written as well:

- the connected component library, which finds patches of equal pixels

- the ConvolveOperator library, which allows to run an image convolution with an

arbitrary kernel

- the DilateOperator library, which is used to buffer the active fires to 500m

- the HoleRemover library, which allows to remove cloudy areas of an arbitrary

maximum size

4 Available at http://step.esa.int/main/third-party-plugins-2/sen2cor

5 A reduced python environment, available at https://conda.io/miniconda.html

http://step.esa.int/main/third-party-plugins-2/sen2cor

https://conda.io/miniconda.html



Issue 1.5 Date 30/01/2018

Page 33

Sentinel BA retrieval

Java Runtime

Connected Component

Labeling libraryConvolveOperator

library

DilateOperator library

HoleRemover library

Figure 5.7: MERIS BA retrieval

In order to set up the environment, a Calvalus Production Type (Fire-S2-BA) se been

created. The Calvalus Production Type predominantly takes care of providing the input

data in the expected structure, and collecting the output data after the processing has

been done. If errors occur, the process fails, so it can be re-done, and it is configured to

automatically try four times before ultimately failing.

Similar to the pre-processing, the production is controlled by the Fire_cci processing

system instance. A control script similar to the script used for Sen2Cor pre-processing is

configured for the burned area retrieval. It can also be parameterised for the time period

(complete mission or only some months), the production can be interrupted and

resumed, and steps can be repeated if required.

The error handling is established by the control script which is described in section 5.3.

If the error message is memory-related, the process has failed due to the processing

environment on the specific node. The reaction is to consider the attempt a failure, and

re-try the same processing step on a different node.

Single attempts fail in ~5% of the attempts of performing BA retrieval. Due to the re-

tries of the single attempts and the error handling described above, though, the overall

failure rate is 0%.

5.3.4. Grid/pixel formatting

For the creation of the PSD-compliant products the same approach is used for Sentinel-

2 as it is for MERIS data, which is described in section 5.1.3.



Issue 1.5 Date 30/01/2018

Page 34

6. System outputs

The system, in the version described in this document, produces the datasets stated in

Table 6.1.

Table 6.1: Output products

MERIS MODIS Sentinel 2

Pixel

product

Grid

product

Pixel

product

Grid

product

Pixel

product

Grid

product

Geographic

extent

Continental

tiles divided

in 6 zones:

Africa, Asia,

Australia,

North+South

America,

Europe

Global

Continental

tiles divided

in 6 zones:

Africa, Asia,

Australia,

North+South

America,

Europe

Global

African area

corresponding

to Zone 5 of

the

continental

tiles, provided

in 2°x2° tiles.

Global,

but data

only

provided

over

African

landmass

Temporal

extent 2005-2011 2001-2016 2016



Issue 1.5 Date 30/01/2018

Page 35

Annex: Acronyms and abbreviations

1U, 4U The number of units a physical machine uses in its rack

AD Applicable Document

AMORGOS Accurate MERIS Ortho-Rectified Geo-location Operational

Software

API Application Programming Interface

ATBD Algorithm Theoretical Basis Document

BA Burned Area

BC Brockmann Consult

BEAM Basic ERS & Envisat (A) ATSR and Meris Toolbox

BRDF Bidirectional reflectance distribution function

CCI Climate Change Initiative

CEDA Centre for Environmental Data Analysis

CF Climate Forecast

CPU Central Processing Unit

DARD Data Access Requirements Document

DEM Digital Elevation Model

EHU University of the Basque Country

EO Earth Observation

ESA European Space Agency

ECV Essential Climate Variables

FSG Full Swath Geo-corrected

FTP File Transfer Protocol

GB Giga Byte

GDAL Geospatial Data Abstraction Library

GFS Google File System

GITZ Geesthacht Innovation and Technology Centre

HDF Hierarchical Data Format

HDFS Hadoop Distributed File System

I/O Input / Output

IPCC Intergovernmental Panel on Climate Change

JPEG (Joint Photographic Experts Group

LC Land Cover

LC_cci Land Cover CCI project

Lat Latitude

Lon Longitude

LTS Long Term Support

LSF Load Sharing Facility

MERIS Medium Resolution Imaging Spectrometer

MODIS Moderate Resolution Imaging Spectroradiometer

MSI MultiSpectral Instrument

NetCDF NETwork Common Data Format

OLCI Ocean and Land Colour Instrument

PB Peta Byte

PSD Product Specification Document

RNIR Red and Near Infrared

SDK Software Development Kit

SDR Surface Directional Reflectance

SFD Small Fire Database



Issue 1.5 Date 30/01/2018

Page 36

SIN Sinusoidal

SLSTR Sea and Land Surface Temperature Radiometer

SNAP Sentinel Application Platform

SoW Statement of Work

SR Surface Reflectance

SSD System Specification Document

TB Tera Byte

UAH University of Alcala

VM Virtual Machine

YARN Yet Another Resource Negotiator

Documents

ESA Climate Change Initiative Fire cci D3.1 System ...ESA Climate Change Initiative – Fire_cci D3.1 System Specification Document (SSD) Project Name ECV Fire Disturbance: Fire_cci