Clouds and the Earth’s Radiant Energy System NASA

How does one go from binary data to HDF files efficiently?

• Betty J. Lock• Betty.J.Lock@NASA.Gov

• (757-951-1903)

• September 29, 2010

3

The Performance issue

Compress AVG/ZAVG files

SYN 6780 data calls sfwdata

Capture and prepare SYN data

Process data by zone

0 500 1000 1500 2000 2500 3000

CPU Seconds

Processing just 1 zoneUsing: HDF4.2r4 on: ppc64

4

The ParametersNo. ofParameters

array definition

147 (360,180,8) Region,Zone,3-Hr data

3 (360,180) Region, Zone

1 (360,180,8,20) Region,Zone,3-Hr data, Surface levels

12 (360,180,8,5) Region,Zone,3-Hr data, Adj. Levels

18 (360,180,8,4) Region,zone,3-Hr data, cloud layers

5

The GOALZonal Binary – To Daily HDF

ZONE 1

ZONE n

ZONE 180

Day 1

Day n

Day 31

Binary Files HDF Files

This is done in 2 steps

6

Step 1Store in memory

Storing Daily informationZONE Binary File STORE in Memory

Region 1 (181 params)

Region n (181 params)

Region n+1 (181 params)

Region 360 (181 params)

Param 1 (360,180,8)

Param n (360,180,8)

Param n+1 (360,180,8)

Param 181 (360,180,8)

7

Step 2Write to Daily HDF

Compress sfscompress and write sfwdataStored in –memory1 day parameters

Day 1 HDF

Param 1

Param n

Param n+1

Param 181

Param 1

Param n

Param n+1

Param 181

For 31 days:Sfscompress takes .9 s.Sfwdata takes 2630 s.

8

Backup slides

9

File Sizes

Tile Type 1 ZONE 180 ZONES

AVG 12,643 KB 650,000 KB

SYN (daily) 3,470 KB 245,476 KB

ZAVG 1,060 KB 3,396 KB

Meta files 14 KB 75 KB

Call 1 ZONE 180 ZONES

6780 calls to sfwdata 2630 s 44 min 5980 s 100 min

Performance Details

Event Accumulated timeseconds

Read TSI & SYNI input 2

Setting variables 9 (sometimes)

Calc region average 4.5

Write SYN binary 1

Total time to process zone 18 (18*180= 3240s)

Read and Cache binary data 5 (5*180 = 900s)

Write 31 SYN HDF 2630 (43 minutes)

Compress AVG/ZAVG HDF 119

Write meta files 7

Based on 1 zone (54) processing

Calculating data & writing binary

Creating HDF2-3 hours for all zones

11

Write 31 SYN HDF

• Includes– Get dimension info– Get field ind– Get sds id– Sfscompress– sfwdata

12

From - To

From 180 binary zonal files 181 Parameters for each region

To HDF Daily Files 28 to 31 per Month

13

The Parameters• 3 data structures with an array of 360

regions, 180 zones• 147 data structures with an array of 360

regions, 180 zones, 8 3-hr data• 1 data structure with an array of 360

regions, 180 zones, 8 3-hr data, 20 surface levels

• 18 data structures with an array of 360 regions, 180 zones, 8 3-hr-data, 4 cloud layers.

• 12 data structures with an array of 360 regions, 180 zones, 8 3-hr-data, 5 layers.