DADDS DCS LRIT Prototype LRIT Latency Results Presented by Microcom Design, Inc. May 2014

DADDS DCS LRIT PrototypeLRIT Latency Results

Presented by

Microcom Design, Inc.

May 2014

Microcom Design, Inc. 2

Task Description

Authorized under a DCS Sustaining Engineering Contract. Develop a DADDS Process to create and disseminate

LRIT DCS Message files. Transfer files to LRIT system via SFTP. Develop text based and graphical tools to analyze

latency. Utilize Microcom LRIT Receiver to monitor data flow and

determine end-to-end latency.• DAMS-NT to DADDS to LRIT to Reception

Include configurable settings to determine file generation:

• Message Count

• Byte Count

• Time – timeout to minimize latency


DOMAST versus LRIT Comparison

DOMSAT streams DCS messages, while LRIT transfers files. For efficiency, multiple DCS messages must be collected into

files. DOMSAT stream “dedicated” to DCS, LRIT stream shares

multiple NOAA products; DCS, EMWIN, Imagery. DOMSAT uses Domestic Satellite, LRIT broadcast over

GOES. DOMSAT requires annual funding, LRIT has no user recurring

satellite usage costs. DOMSAT has limited coverage outside of CONUS, LRIT has

hemispherical coverage. DOSAT primarily used for other data, GOES/LRIT dedicated to

environmental usage. DOMSAT frequency in Ku band, while LRIT in L band.

DOMSAT highly susceptible to weather fading, LRIT far more robust.


LRIT Operational Summary 1

Downlink Frequency: 1691 MHz (L Band) Information Data Rate:128 kbps (16 kBps) Forward Error Correction: Convolution and Block

Inner Code: Rate ½ Viterbi, Constraint Length 7 Outer Code: Reed Solomon (255,223) – 223 information

bytes and 32 check bytes; can correct up to 16 bytes in error

Data Files are Packetized and Framed Frame: 1024 Bytes

• 4-byte FSS, followed by four interleaved 255 byte RS blocks. • ~ 880 bytes of information in each frame.

Packets: Variable up to 8 kilobytes• Each Packet terminated in CRC-16.• Large Files require multiple packets.• First Packet contains LRIT Headers.


LRIT Operational Summary 2

Files can be sent on one of 63 Virtual Channels (VC) Each Frame begins with a 6-byte sequence that defines the VC

and includes a 3-byte Frame Counter. Frame Sequence Counters are unique to VC.

Virtual Channels Represent a Priority The lower the VC number, the higher the priority. Priority scheme allows smaller, high priority files to be

interspersed with larger, low priority files. EMWIN uses VC 00; highest priority.

LRIT is a Continuous Transmission A 64TH Virtual Channel (VC 63) is used for “Fill”, i.e. when there

is no data to send. Actual transmission rate is 292.7 ksps (kbps)

128 = (292.7 / 2) * (223/255)


LRIT DCS Initial Analysis – Frame Utilization 1

LRIT Packet Distribution - 9/20/2012

0

20

40

60

80

100

20:00 20:05 20:10 20:15 20:20 20:25 20:30 20:35 20:40 20:45 20:50 20:55 21:00

Time, Local

Pe

rce

nt

Uti

liza

tio

n

DCS Imagery EMWIN Fill


LRIT Initial Analysis – Frame Utilization 2

Initial analysis showed that the LRIT stream was dominated by long periods of either DCS, Imagery, or Fill, but …

EMWIN maintained steady flow of data.

LRIT Packet Distribution - 9/20/2012

0

20

40

60

80

100

20:15 20:16 20:17 20:18 20:19 20:20

Time, Local

Per

cen

t U

tiliz

atio

n

DCS Imagery EMWIN Fill


LRIT Initial Analysis – Product Utilization

Initial Frame Analysis Showed: EMWIN: ~ 7.3 kbps ~5.7% DCS: ~15.3 kbps ~12.0% Imagery: ~41.2 kbps ~32.2% Fill: ~64.2 kbps ~50.1%

Initially: EMWIN used VC00 DCS used VC18 (High), VC38 (Medium), VC53 (Low) Imagery used VCs 13, 23, 33, 41, and 43

More Recently: EMWIN continues to use VC00 DCS was using 1, 48, and 53 Imagery shared VC53 with Low Priority DCS Files


LRIT DCS File Summary

High Priority Files: Contain one of two platforms (or both) that transmit on six

minute intervals at the top of the minute. Being used by NOS and NWS to facilitate their latency testing. Files sent in 1 Packet/Frame.

Medium Priority Files: Consist of approximately 500 platforms. All platforms appear to transmit on DCS channels 40-49. Files typically require 1 Packet sent over several Frames.

Low Priority Files: Everything not in Low or Medium priority files. Files are quite large (~200 kilobytes, ~1000 messages). Files typically sent about every 2 minutes.


LRIT Baseline Latency

Typical composite latency ~100 seconds with regular peaks over 300 seconds; some as high as 600 seconds.

Peaks believed to be primarily due to DCS Low Priority files sharing same VC as imagery files.


Prototype Target Goals

Create compatible DCS LRIT files from DADDS.

Determine if the LRIT system can handle smaller more frequent files.

Send all DCS messages as High priority. Accurately measure overall LRIT DCS

latency. Breakdown latency into constituent

components. See if DADDS can provide improvement

over existing dissemination path.


Initial Live Test Run

Test was run on April 16, 2014 from approximately 14:24:30 to 14:29:30 UTC.

All data sent as High Priority. DCS data collected and sourced from NSOF DADDS. Messages collected into files of approximately 8 kilobytes

each. Total of 58 files generated and transmitted. Approximately 2450 DCS messages were transmitted and

received. Test run was performed while WCDA DCS LRIT was also

flowing. No anomalies were detected and test ran as expected. After confirming data was flowing and no issues were

detected, performance data was collected from approximately 14:27:00 to 14:29:30 UTC.


1ST Test – Frame Utilization Graph

LRIT DADDS Test - Frame Utilization

0

10

20

30

40

50

60

70

80

90

100

14:2

7:56

14:2

8:01

14:2

8:07

14:2

8:12

14:2

8:18

14:2

8:24

14:2

8:29

14:2

8:35

14:2

8:40

14:2

8:46

14:2

8:51

14:2

8:57

14:2

9:03

14:2

9:08

14:2

9:14

14:2

9:19

14:2

9:25

14:2

9:31

14:2

9:36

14:2

9:42

Time

Fra

mes

Per

Sam

ple

(E

very

100

Fra

mes

)

DADDS DCS EMWIN Other (WCDA DCS & Imagery) Fill


1ST Test – Frame Utilization Notes

LRIT Frames counted for DADDS DCS, EMWIN, Fill, and Other. Other includes both Imagery and the DCS Data coming from WCDA.

A data sample set occurs every 100 LRIT Frames. NSOF DADDS DCS Data accounted for approximately 10% of the

Total Frames. Average: 9.8 Frames per 100 Minimum: 0 Maximum: 28

EMWIN accounted for approximately 6% of the Total Frames. Insertion of a second DCS stream did not create any major issues.

Only impact was a slight reduction in the transmission rate of the

“Other” frames.

This short duration test showed minimal Fill packets (~7%). Longer term testing has shown that the LRIT typically only runs at 65-

70% utilization (30-35% Fill).


1ST Test – Overall Results

Data points generated and accumulated every 100 messages. Computer generating graph had clock running 5-6 seconds fast.


1ST Test – Overall Notes

Latency calculated from the time the message was received to time it was ingested by test computer at Microcom.

Average DCS message latency for this test run calculated on every 100 messages. Note: After the test run was complete, the test

computer’s clock was checked and was found to be 5-6 seconds fast.

Accounting for computer’s clock error, average latency for the initial test run was actually on the order of 45-50 seconds.


1ST Test – LRIT File Latency Histogram

LRIT File Distribution Latency

0

2

4

6

8

10

12

14

15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40

File Generation to Received in LRIT Stream, Seconds

File

Co

un

ts


1ST Test – LRIT Latency Notes

Latency calculated in seconds from file generation time to reception by LRIT Receiver. Average: 25 seconds Minimum: 19 seconds Maximum: 35 seconds

This latency encompasses the transfer time from DADDS-to-LRIT, the LRIT processing time, and the transfer time from the LRIT processor to the LRIT uplink transmitter. Note: The times are approximately 3-5 seconds higher than actual. Microcom utilized the first DCS message in the file to generate the

filename timestamp. A better indication would be provided by using the time the last

message was added to the file.

Average file latency is predicted to be more on the order of 20 seconds or so.

Average Latency remained relatively consistent over initial run.


1ST Test – DADDS LRIT File Generation

DADDS LRIT File Generation Latency

0

5

10

15

20

25

11 12 13 14 15 16 17 18 19 20

First File Msg End Time To File Timestamp, Seconds

File

Co

un

ts


1ST Test – DADDS LRIT Latency Notes

Latency calculated in seconds from end time of first message in file to file timestamp: Average: 16.4 seconds Minimum: 13.5 seconds Maximum: 18.6 seconds

DADDS file latency on the order of 16 seconds. Higher than expected, but optimizing DADDS process was

not a primary goal of this task. Microcom believes DADDS latency can be easily reduced to

5-10 seconds. Latency was consistent over initial run.


Second Live Test Run

Test was run on April 23, 2014 from approximately 13:27:30 to 16:05:00 UTC.

All data sent as High Priority. DCS data collected and sourced from NSOF DADDS. Messages collected into files of approximately 8 kilobytes

each. A total of 2,035 files were generated, transmitted, and

received. Over 76,000 DCS messages were transmitted and received

(~29,000 messages per hour). Test run was performed without WCDA DCS LRIT data

flowing. No operational issues were detected; independent

confirmation of reception from NOS and NWS. Performance data was collected from almost the entire run

UTC.


2ND Test – Frame Utilization Graph

LRIT DADDS Test - Frame Utilization - Entire Test Run

0

10

20

30

40

50

60

70

80

90

100

13

:30

:39

13

:35

:17

13

:39

:56

13

:44

:35

13

:49

:14

13

:53

:52

13

:58

:31

14

:03

:10

14

:07

:49

14

:12

:27

14

:17

:06

14

:21

:45

14

:26

:23

14

:31

:02

14

:35

:41

14

:40

:20

14

:44

:58

14

:49

:37

14

:54

:16

14

:58

:55

15

:03

:33

15

:08

:12

15

:12

:51

15

:17

:30

15

:22

:08

15

:26

:47

15

:31

:26

15

:36

:05

15

:40

:44

15

:45

:22

15

:50

:01

15

:54

:40

15

:59

:18

16

:03

:57

Time

Fra

me

s P

er

Sa

mp

le (

Ev

ery

10

0 F

ram

es

)

DCS EMWIN Other (Imagery) Fill


2ND Test – Frame Utilization Notes

LRIT Frames counted for DCS, EMWIN, Fill, and Other; Other primarily includes Imagery.

A data sample set occurs every 100 LRIT Frames. DCS Data accounted for 10.24% of the Total Frames.

Average: 10.24 Frames per 100 Minimum: 0 Maximum: 45 Stnd Dev: 7.6

EMWIN accounted for 5.73% of the Total Frames. Other (Imagery) accounted for 57.14% of the Total Frames. Fill accounted for 26.89% of the Total Frames. EMWIN and DCS packets flowed at relatively consistent

rate, while Imagery & Fill dominated alternately.


2ND Test – Frame Utilization Hour 14

LRIT DADDS Test - Frame Utilization - Hour 14 UTC

0

10

20

30

40

50

60

70

80

90

100

14

:00

:00

14

:01

:52

14

:03

:43

14

:05

:35

14

:07

:26

14

:09

:18

14

:11

:09

14

:13

:01

14

:14

:52

14

:16

:44

14

:18

:35

14

:20

:27

14

:22

:18

14

:24

:10

14

:26

:01

14

:27

:53

14

:29

:44

14

:31

:36

14

:33

:27

14

:35

:19

14

:37

:10

14

:39

:02

14

:40

:53

14

:42

:45

14

:44

:36

14

:46

:28

14

:48

:19

14

:50

:11

14

:52

:02

14

:53

:54

14

:55

:45

14

:57

:37

14

:59

:28

Time

Fra

me

s P

er

Sa

mp

le (

Ev

ery

10

0 F

ram

es

)

DCS EMWIN Other (Imagery) Fill


2ND Test – Overall Results

Data points generated and accumulated every 100 messages.


2ND Test – Overall Notes

Latency calculated from the time the message was received to time it was ingested by test computer at Microcom.

Average end-to-end latency for the bulk of this run was on the order of 40 seconds.

From ~15:05 to ~15:25 the average latency gradually increased to a peak of over 65 seconds and then came back down to ~40 seconds.

Standard Deviation of Latency consistently remained under 5 seconds.


2ND Test – Existing Dissemination Comparison

Typical composite latency 60-80 seconds with peaks over 200 seconds. Improvement from initial baseline due to move to using VC 01 for all DCS

files.


2ND Test – LRIT File Latency Histogram

LRIT File Distribution Latency - Second Run

0

50

100

150

200

2501

011 1

21

31

41

51

61

71

81

92

02

12

22

32

42

52

62

72

82

93

03

13

23

33

43

53

63

73

83

94

04

14

24

34

44

5

File Generation to Received in LRIT Stream, Seconds

File

Co

un

ts


2ND Test – LRIT Latency Notes

Latency calculated in seconds from file generation time to reception by LRIT Receiver. Average: 22.9 seconds Minimum: 13 seconds Maximum: 42 seconds Stnd Dev: 4.2 seconds

Latency encompassed the transfer time from DADDS-to-LRIT, the LRIT processing time, and the transfer time from the LRIT processor to the LRIT uplink transmitter. Note: The times are approximate 3-5 seconds higher than actual. Microcom utilized the first DCS message in the file to generate the

filename timestamp. A better indication would be provided by using the time the last

message was added to the file. Average file latency is predicted to be more on the order of 20

seconds or so. Average Latency remained relatively consistent over entire run.


2ND Test – LRIT File Latency Over Run

LRIT File Latency Time Plot

0

5

10

15

20

25

30

35

40

45

13

:27

:52

13

:32

:47

13

:37

:17

13

:41

:39

13

:45

:59

13

:50

:56

13

:55

:19

13

:59

:47

14

:04

:00

14

:07

:37

14

:11

:15

14

:14

:47

14

:18

:23

14

:22

:11

14

:26

:11

14

:30

:43

14

:34

:53

14

:39

:19

14

:43

:39

14

:48

:00

14

:52

:50

14

:57

:22

15

:01

:53

15

:05

:48

15

:09

:31

15

:13

:12

15

:16

:39

15

:20

:12

15

:23

:50

15

:28

:01

15

:32

:13

15

:36

:37

15

:40

:53

15

:45

:07

15

:49

:53

15

:52

:39

15

:54

:54

15

:57

:12

15

:59

:18

16

:01

:37

16

:03

:36

Time, UTC

La

ten

cy

, Se

co

nd

s

Raw 10 Point Average


2ND Test – DADDS File Latency

DADDS LRIT File Generation Latency

0

5

10

15

20

25

30

35

40

45

13:2

7:27

13:3

3:57

13:4

0:38

13:4

7:12

13:5

4:13

14:0

1:10

14:0

6:48

14:1

2:04

14:1

7:22

14:2

3:00

14:2

9:32

14:3

6:07

14:4

2:34

14:4

9:22

14:5

6:15

15:0

2:53

15:0

8:29

15:1

3:59

15:1

9:17

15:2

4:43

15:3

1:11

15:3

7:44

15:4

4:03

15:5

0:56

15:5

4:14

15:5

7:31

16:0

1:01

16:0

3:58

Time, UTC

Lat

ency

, S

eco

nd

s


2ND Test – DADDS File Latency Notes Latency calculated from end timestamp of first DCS message in file to the

file generation time. Average: 18.9 seconds Minimum: 13 seconds Maximum: 42 seconds Stnd Dev: 6.4 seconds

Baseline average appears to be on the order of 16-17 seconds. Discounting three spiked up sections, latency remained relatively consistent

over entire run. Excessive rise in processing delay accounts for increased overall latency

shown in graph, i.e the point centered about 15:15 UTC where the overall latency increased to over 65 seconds.

Smaller spike at 14:10 in above graph also correlates to similar spike in overall graph.

At this time Microcom has no specific explanation for the increases in DADDS processing time. Most likely cause is server performance due to some other function that

required high CPU resources. Additional investigation will be required to determine exact cause. Should be readily addressable.


Target Goals Acheived

Testing was highly successful and achieved target goals: DADDS can flow DCS data to LRIT system. DADDS DCS data files were properly formatted:

• Message data was correctly received by Microcom’s LRIT Receiver, which also receives WCDA sourced DCS data.

• Independent confirmation provided by NOS (Mark Bushnell, Warren Krug) and NWS (Brian Jackson) during and post test run.

LRIT system has no issues in processing smaller more frequent files.

LRIT throughput not significantly affected by sending all DCS messages as High Priority.


Testing Conclusions

Testing provided solid baseline information on overall latency and distribution of latency in various systems. Overall typical latency is on the order of 35-45 seconds. Latency remains consistent over extended periods of

time. LRIT processing and distribution latency on the order of

20-25 seconds. Typical DADDS file generation latency is on the order of

15-18 seconds. The DADDS-to-LRIT prototype and subsequent analysis

by Microcom has clearly demonstrated capability to accurately measure and to pinpoint problem areas in the overall latency.


Possible Next Steps

The unusual and unexpected areas where latency above baseline will require additional investigation to determine the cause. However, Microcom is confident this can be mitigated.

Microcom is also confident that the DADDS file generation can be optimized and reduce the typical latency down closer to 10 seconds (or better).

Assuming an LRIT system latency of 20-25 seconds and a 5-10 second DADDS latency can be achieved, than it should be possible to achieve an average latency of 30 seconds.

Make suggested improvements and perform additional testing.

END OF PRESENTATION “THANK YOU” FOR YOUR ATTENTION

Documents

DADDS DCS LRIT Prototype LRIT Latency Results Presented by Microcom Design, Inc. May 2014