Uncertainty analysis for evaluating the accuracy of snow …...to measure a few millimeters of snow, but have issues with the representativeness of snow depth measurements, given the

HESSD12, 4157–4190, 2015

Uncertainty analysisfor evaluating theaccuracy of snow

depth measurements

J.-E. Lee et al.

Title Page

Abstract Introduction

Conclusions References

Tables Figures

J I

J I

Back Close

Full Screen / Esc

Printer-friendly Version

Interactive Discussion

Discussion

Paper

|D

iscussionP

aper|

Discussion

Paper

|D

iscussionP

aper|

Hydrol. Earth Syst. Sci. Discuss., 12, 4157–4190, 2015www.hydrol-earth-syst-sci-discuss.net/12/4157/2015/doi:10.5194/hessd-12-4157-2015© Author(s) 2015. CC Attribution 3.0 License.

This discussion paper is/has been under review for the journal Hydrology and Earth SystemSciences (HESS). Please refer to the corresponding final paper in HESS if available.

Uncertainty analysis for evaluating theaccuracy of snow depth measurements

J.-E. Lee1, G. W. Lee1, M. Earle2, and R. Nitu2

1Department of Astronomy and Atmospheric Sciences, Research and Training Team forFuture Creative Astrophysicists and Cosmologists, Kyungpook National University, 80Daehakro, Buk-gu, Daegu,702-701, Republic of Korea2Observing Systems and Engineering, Meteorological Service of Canada, EnvironmentCanada, 4905 Dufferin St., Toronto, Ontario, Canada

Received: 13 March 2015 – Accepted: 26 March 2015 – Published: 24 April 2015

Correspondence to: G. W. Lee ([email protected])

Published by Copernicus Publications on behalf of the European Geosciences Union.

4157

http://www.hydrol-earth-syst-sci-discuss.net

http://www.hydrol-earth-syst-sci-discuss.net/12/4157/2015/hessd-12-4157-2015-print.pdf

http://www.hydrol-earth-syst-sci-discuss.net/12/4157/2015/hessd-12-4157-2015-discussion.html

http://creativecommons.org/licenses/by/3.0/

HESSD12, 4157–4190, 2015


depth measurements

J.-E. Lee et al.

Title Page



Tables Figures

J I

J I

Back Close

Full Screen / Esc



Discussion

Paper

|D

iscussionP

aper|

Discussion

Paper

|D

iscussionP

aper|

Abstract

A methodology for quantifying the accuracy of snow depth measurement are demon-strated in this study by using the equation of error propagation for the same type sen-sors and by compariong autimatic measurement with manual observation. Snow depthwas measured at the Centre for Atmospheric Research Experiments (CARE) site of the5

Environment Canada (EC) during the 2013–2014 winter experiment. The snow depthmeasurement system at the CARE site was comprised of three bases. Three ultrasonicand one laser snow depth sensors and twelve snow stakes were placed on each base.Data from snow depth sensors are quality-controlled by range check and step test toeliminate erroneous data such as outliers and discontinuities.10

In comparison with manual observations, bias errors were calculated to show thespatial distribution of snow depth by considering snow depth measured from four snowstakes located on the easternmost side of the site as reference. The bias error of snowstakes on the west side of the site was largest. The uncertainty of all pairs of stakesand the average uncertainty for each base were 1.81 and 1.52 cm, respectively. The15

bias error and normalized bias removed root mean square error (NBRRMSE) for eachsnow depth sensor were calculated to quantify the systematic error and random errorin comparison of snow depth sensors with manual observations that share the samesnow depth target. The snow depth sensors on base 12A (11A) measured snow depthlarger (less) than manual observation up to 10.8 cm (5.21 cm), and the NBRRMSEs20

ranged from 5.10 to 16.5 %. Finally, the instrumental uncertainties of each snow depthsensor were calculated by comparing three sensors of the same type installed at thedifferent bases. The instrumental uncertainties ranged from 0.62 to 3.08 cm.

1 Introduction

Solid precipitation has a significant effect on human life, as it can lead to issues such25

as flight delays and slippery roads, harm to crops, and building collapses (Rasmussen

4158





HESSD12, 4157–4190, 2015


depth measurements

J.-E. Lee et al.

Title Page



Tables Figures

J I

J I

Back Close

Full Screen / Esc



Discussion

Paper

|D

iscussionP

aper|

Discussion

Paper

|D

iscussionP

aper|

et al., 2003). The impact of these issues can be mitigated with more accurate weatherforecasts and representative engineering standards, the development of which requireaccurate solid precipitation and snow on the ground measurements. In addition, precip-itation and snow on the ground measurements are important for various meteorologicaland hydrological applications, such as climate change, remote sensing calibration, and5

water supply forecasts (e.g. Michelson, 2004; Rasmussen et al., 2012; Theriault et al.,2012). However, accurate measurement of solid precipitation is difficult due to wind-induced loss and the spatial and temporal variability of snow shape, size, and density(Roebber et al., 2003; Nitu, 2013). The measurement of snow on the ground is alsoprone to numerous errors due to snow redistribution, blowing snow, and compaction10

(Ryal et al., 2008). Thus, there is a requirement for the accuracy of solid precipitationand snow on the ground measurements to be evaluated systematically. This study fo-cuses on the measurement of snow depth, which is the total vertical height of snow onthe ground within the observation period.

Graduated rulers or snow stakes are used to measure snow depth by trained hu-15

man observers; these manual measurements are considered to be the reference forsnow depth measurements. However, manual snow depth measurements have signif-icant limitations such as consistency, continuity, spatial and temporal resolution, andtime and manpower consumption (Ryan and Doesken, 2007). Meanwhile, snow depthsensors based on various operating principles have been developed as a result of the20

automation of meteorological observation systems (Nitu et al., 2012). Automatic snowdepth sensors can help to overcome the limitations of manual snow depth measure-ments, but they also have limitations (Ryal et al., 2008; Fischer, 2008; Haij, 2011). Ul-trasonic snow depth sensors are currently the most frequently used, due to their easeof use and low power consumption. Two aspects of the operation of these sensors25

need to be properly managed: first, the temperature dependency of ultrasonic pulses;and second, the risk of interference within the field of view, since ultrasonic pulses havethe shape of a cone, for example a cone of 22◦ for the Campbell Scientific SR50 sen-sor (Ryan and Doesken, 2008). Laser snow depth sensors have sufficient sensitivity

4159





HESSD12, 4157–4190, 2015


depth measurements

J.-E. Lee et al.

Title Page



Tables Figures

J I

J I

Back Close

Full Screen / Esc



Discussion

Paper

|D

iscussionP

aper|

Discussion

Paper

|D

iscussionP

aper|

to measure a few millimeters of snow, but have issues with the representativeness ofsnow depth measurements, given the small sample area (Haji, 2011).

The required resolution and uncertainty for snow on the ground measurementsare suggested by the World Meteorological Organization (WMO). The snow depthshould be reported with 0.1 cm resolution and the uncertainty should fall within 1 cm5

for ≤ 20 cm and 5 % for > 20 cm (WMO, 2008). Thus, the performance of snow depthsensors needs to be evaluated using field observations. Ryan and Doesken (2008)compared ultrasonic snow depth sensor with manual snow depth measurements atseven sites in the United States. The mean absolute error and root-mean square errornormalized by average snow depth ranged from 0.75–6.36 cm and 2–170 %, respec-10

tively. Random error also exists in snow depth measurements, but is not well quantified.Thus, uncertainty analysis of random errors should be performed to evaluate the accu-racy of snow depth measurements.

In 2010, the WMO initiated the Solid Precipitation InterComparison Experiment(SPICE). The main objective of SPICE is to define the (automatic) operational reference15

system and to evaluate the performance of various automatic snow depth and snowfallmeasurement instruments (WMO/CIMO, 2011, 2012a, 2012b, 2013). The Centre forAtmospheric Research Experiments (CARE) in Egbert, Ontario, Canada, is one of thelead SPICE sites and hosts one of the SPICE experiments for snow on ground. On thissite, several models of snow depth sensors (ultrasonic, laser) are tested against man-20

ual reference measurements, as part of an Environment Canada (EC) study project,as well as a contribution to SPICE.

The main purpose of this study is to document methodology for analyzing the un-certainty of snow depth measurements from automatic snow depth sensors. This pro-cedure is demonstrated using data collected at the CARE site during the 2013–201425

winter season. The quality control (QC) procedures applied to the data sets used aresimilar to those established for the first level of QC in SPICE, and constitute the base-line for any advanced QC that may be required. SPICE will investigate and report onmore advanced QC techniques, tailored to specific sensors. In Sect. 2, the methodol-

4160





HESSD12, 4157–4190, 2015


depth measurements

J.-E. Lee et al.

Title Page



Tables Figures

J I

J I

Back Close

Full Screen / Esc



Discussion

Paper

|D

iscussionP

aper|

Discussion

Paper

|D

iscussionP

aper|

ogy for the quantification of uncertainty in snow depth measurements is proposed. Theprocedures for snow depth measurements are described in Sect. 3. The manual andautomatic snow depth data are shown in Sect. 4, along with suggested QC proceduresfor automatic snow depth measurements. The uncertainties in manual and automaticsnow depth measurements are detailed in Sect. 5. Section 6 summarizes the results5

and provides conclusions.

2 Quantification of uncertainty

The uncertainty analysis is performed using two approaches: statistical measures andthe propagation of error. The standard quantities for measuring the accuracy are de-fined under statistical measures. In the propagation of error, the uncertainty of indi-10

vidual instruments is calculated from the difference of two measurements of the sametype. These approaches are explained in further detail below.

2.1 Statistical measures

Standard statistical measures are used to quantify the uncertainty of snow depth mea-surements. The Bias Error (BE), Mean Absolute Error (MAE), Root Mean Square Error15

(RMSE), and Bias Removed Root Mean Square Error (BRRMSE) are defined as fol-lows:

BE =1N

∑(y −x) (1)

MAE =1N

∑|y −x| (2)

RMSE =[

1N

∑(y −x)2

]0.5

(3)

4161





HESSD12, 4157–4190, 2015


depth measurements

J.-E. Lee et al.

Title Page



Tables Figures

J I

J I

Back Close

Full Screen / Esc



Discussion

Paper

|D

iscussionP

aper|

Discussion

Paper

|D

iscussionP

aper|

BRRMSE =[

1N

∑(y −x−BE)2

]0.5

(4)

where x and y are snow depths from pairs of manual measurements, manual andautomatic measurements sharing the same snow target, or two instruments of thesame type at different targets, and N is the number of data points for a given pair. TheNBE, NMAE, NRMSE, and NBRRMSE are the normalized forms in which BE, MAE,5

RMSE, and BRRMSE are divided by the average of x.In comparisons between manual observations, the BE is calculated to investigate

the spatial distribution of snow depth relative to the average snow depth measured bysnow stakes at each target. The average snow depth of four stakes on the same snowdepth target is considered as x in Eq. (1) for the calculation of BE in comparisons10

between manual observations and automatic snow depth sensors, which indicates thesystematic bias of measurements from individual snow depth sensors relative to thereference. The MAE (NMAE), RMSE (NRMSE) and BRRMSE (NBRRMSE) indicatethe random errors in snow depth measurements.

2.2 Error propagation15

The error propagation equation is used to quantify the uncertainty of manual snowdepth measurements and automatic snow depth sensors. When z is the differencebetween x1 and x2 (z = x1 −x2), the variance of z (σ2

z ) is expressed as follows:

σ2z = σ

2x1

(∂z∂x1

)2

+σ2x2

(∂z∂x2

)2

+2σ2x1x2

(5)

where x1 and x2 are the snow depths from pairs of two manual measurements or two20

instruments of the same type.The terms σ2

x1and σ2

x2represent the variance or error of x1 and x2 and the σ2

x1x2term

represents the covariance of x1 and x2. The random errors for two instruments of the

4162





HESSD12, 4157–4190, 2015


depth measurements

J.-E. Lee et al.

Title Page



Tables Figures

J I

J I

Back Close

Full Screen / Esc



Discussion

Paper

|D

iscussionP

aper|

Discussion

Paper

|D

iscussionP

aper|

same type, which have the same sampling volume and resolution, are nearly identical.Those for two manual measurements performed using the same procedure are alsoidentical. Thus, the σ2

x1and σ2

x2terms are assumed to be identical when two manual

measurements are compared and when two instruments of the same type are used.The covariance is set to be zero (σ2

x1x2= 0) by assuming the random errors from the5

two measurements are not correlated. Thus, σ2x1

or σ2x2

can be calculated by:

σ2x1= σ2

x2=σ2z

2(6)

Even though two manual measurements are performed by the same procedure, andthe two instruments are the same type, bias error can still exist in each case. Therefore,the variance of z in Eq. (6) can be also written as follows:10

σ2z =

1n

∑n

z2 −BE2 (7)

By combining Eqs. (6) and (7), the σ2x1

or σ2x2

terms can be expressed as follows:

σx1= σx2

=

√1n

∑nz

2 −BE2

2(8)

The uncertainties in manual observations are calculated using pairs of snow stakes.The average uncertainties for all snow stakes, each base, and each snow depth target15

are compared. The comparison among snow depth sensors of the same type is per-formed to quantify the instrumental uncertainty of each snow depth sensor using thesame procedure.

4163





HESSD12, 4157–4190, 2015


depth measurements

J.-E. Lee et al.

Title Page



Tables Figures

J I

J I

Back Close

Full Screen / Esc



Discussion

Paper

|D

iscussionP

aper|

Discussion

Paper

|D

iscussionP

aper|

3 Snow depth measurements

The CARE site (44◦14′ latitude, 79◦47′ longitude, 251 m elevation) has a humid conti-nental climate. The average temperature is −8.2 ◦C in January and the average totalsnowfall is 157 cm. The mean wind speed for the period from November to April is 3.5–4.0 ms−1 (WMO/CIMO, 2012a). The prevailing wind direction is west to east. The site5

has a slight slope east to west and is well-exposed.The layout of the snow depth measurement system at the CARE site during the

2013–2014 winter season is shown in Fig. 1. This system is configured around threebases, each with four snow depth sensors (three ultrasonic and one laser-based) point-ing at three snow depth targets developed by EC. Four snow stakes, used for the10

manual measurements, are posted at the corners of each snow depth target, in theshape of a square. A total of 36 manual observations are performed. Snow depthtargets are composed of grey plastic decking (Fig. 2a). The size of each target is130.18±0.64cm×129.54±0.64 cm. To help perceive even a few millimeters of snow,and to better represent the ground surface around the target, the surface of each target15

is painted grey. Holes in each target help in draining water, and the gap between thetarget and ground mimics the layer of short grasses on the ground, and acts like aninsulation layer.

The manual observations taken using wooden snow stakes (Fig. 2b) are consideredto be the reference observations for snow depth. Gradations with 0.5 cm resolution20

are marked on the snow stakes. The snow stakes are perpendicular to the surface ofthe ground and the snow targets. The trained human observer measures snow depthonce a day, starting from the southeast side of the site during non-precipitating peri-ods. The duration and resolution of manual observation are about 20 min and 0.5 cm,respectively.25

Three automatic snow depth sensors are installed on each base as in Fig. 2c. Thepole on each base has the three arms. One ultrasonic snow depth sensor is installedon each arm. The laser snow depth sensors, which are installed at the top of the pole,

4164





HESSD12, 4157–4190, 2015


depth measurements

J.-E. Lee et al.

Title Page



Tables Figures

J I

J I

Back Close

Full Screen / Esc



Discussion

Paper

|D

iscussionP

aper|

Discussion

Paper

|D

iscussionP

aper|

point to the middle snow depth target. Care has been taken during the installation ofthe laser snow depth sensor to avoid pointing at the holes of the snow depth target. Theoutput of all snow depth sensors under test is collected every 30 s. The snow stakesare located outside the field of view of each snow depth sensor.

In this experiment the following snow depth sensors are being tested, the FE-5

LIX SL300 (hereafter, FEL), SOMMER USH-8 (hereafter, SOM), CAMPBELL SR-50A(hereafter, SR50A), and JENOPTIK SHM 30 (hereafter, JEN). One each of these sen-sors is installed on each base. The FEL, SOM, and SR50A are ultrasonic snow depthsensors, and the JEN is a laser snow depth sensor. The general characteristics ofthe snow depth sensors are outlined in Table 1 (Sommer Mess-Systemtechnik, 2008;10

Jenoptik, 2009; Campbell Scientific Corp., 2011; Felix Technology Inc., 2014). Themeasurement range of FEL, SOM, SR50A, and JEN are 0.43–6.10, 0.00–8.00, 0.50–10.0, and 0.00–15.0 m, respectively. The resolution of SOM, SR50A, and JEN are 1.00,0.25, and 1.00 mm, respectively. The JEN requires 10–30 VDC (15–24 VDC) without(with) heating.15

4 Data

Snow depth data from the manual observations and automatic snow depth sensors(FEL, SOM, SR50A, and JEN) are used to demonstrate the proposed methodology forthe analysis of uncertainty in snow depth measurements. The data were collected atthe CARE site from 20 December 2013 to 26 March 2014.20

4.1 Manual data

Figure 3 shows the time series of snow depth at each snow stake from the manual ob-servations at each base (Fig. 3a–c), and the average snow depths from the four snowstakes at each target (Fig. 3d). The maximum snow depths recorded at bases 12A, 20,and 11A during the observation period were 40.0, 42.5, and 44.0 cm, respectively. The25

4165





HESSD12, 4157–4190, 2015


depth measurements

J.-E. Lee et al.

Title Page



Tables Figures

J I

J I

Back Close

Full Screen / Esc



Discussion

Paper

|D

iscussionP

aper|

Discussion

Paper

|D

iscussionP

aper|

average snow depth from the four snow stakes at each target is calculated to investi-gate the spatial distribution of snow depth and compare with the automatic sensors atthe same target. The variation in manual snow depth measurements between the fourcorners of a target is attributed to the uneven deposition of snow on the surface of thattarget (Fig. 4). The manual snow depth data are also used to analyze the uncertainty5

of manual snow depth measurements.

4.2 Automated data

The data collection for this experiment has been configured such that the JEN, FEL,and SR50A data are reported with one millimeter resolution, while the SOM data isreported with one centimeter resolution. The time series of raw (unfiltered) snow depth10

data from each sensor show apparent erroneous data, such as outliers and disconti-nuities (Fig. 5). Some of the data from FEL, SOM, and JEN fall outside the reasonablerange for a given site and observation period (Fig. 5a, b, and d). Snow depth sensordata over 1000 cm exceed the expected maximum value based on manual data for thisexperiment, and are considered to be outliers. Abrupt jumps or spikes (discontinuities)15

that are within the reasonable range of values are evident in the time series of snowdepth data from SR50A (Fig. 5c). These data are excluded from data analysis throughapplication of the QC procedures for snow depth sensor data (described in Sect. 4.3).These outliers and discontinuities could result from environment-, configuration-, orsensor-related causes. The investigation of these causes is outside the scope of this20

paper. The quality-controlled data are compared with manual observations on the samesnow depth target and among the snow depth sensors of same type. This comparisonwill enable the evaluation of uncertainty for each snow depth sensor.

4.3 QC of snow depth sensor data

The following QC procedures are applied to snow depth sensor data in this study: (1)25

range check, (2) step test; and (3) conversion of data from 30 s temporal resolution

4166





HESSD12, 4157–4190, 2015


depth measurements

J.-E. Lee et al.

Title Page



Tables Figures

J I

J I

Back Close

Full Screen / Esc



Discussion

Paper

|D

iscussionP

aper|

Discussion

Paper

|D

iscussionP

aper|

into 1 min resolution. These QC procedures are similar to those considered for the firstlevel QC of data in SPICE. The authors recognize that more advanced QC methods,taking into account the sensor specific manufacturer recommended procedures (e.g.,Campbell Scientific Corp. 2011), could further improve the data used for the analysis.This is outside the scope of this paper. SPICE will investigate and report on more5

advanced QC techniques.In the first QC stage, a range check is applied to remove outliers. In this study, values

of 0 and 60 cm are selected as the minimum and maximum limits, respectively. Themaximum snow depth threshold (60 cm) was selected based on the maximum depthmeasured by automatic sensors during the winter season. The second stage of the10

QC procedure is a step test, designed to remove discontinuities and retain only datashowing realistic changes with respect to time. If the difference between consecutivedata points exceeds a defined threshold, the data are flagged. Each subsequent datapoint is compared with the most recent preceding data point that was not flagged, soa single point or series of points exceeding the set threshold can be identified and15

flagged using the same procedure. In this study, the threshold value for the step testwas 2 cm per 30 s. If all data are flagged during a one hour interval, a new base lineis considered. The data during the two hour period before the new base line are alsoflagged, to draw analyst attention to scenarios in which sensor performance may havebeen impacted. The flagged data are excluded from data analysis. In the third QC20

stage, the snow depth data, which are sampled and transmitted twice per minute, areconverted into 1 min data by arithmetic averaging.

The snow depth data after applying the QC procedure are shown in Fig. 6. It isevident that the outliers and discontinuities observed in Fig. 5 have been removed.Several spikes still remain in Fig. 6a, underscoring the challenge of selecting general25

QC thresholds for data from different sensors. The data collected from FEL (base 20)after 21 February 2014 are removed by the QC procedure, since they exceed the rangecheck threshold. From Fig. 6, it is evident that the maximum snow depth measured by

4167





HESSD12, 4157–4190, 2015


depth measurements

J.-E. Lee et al.

Title Page



Tables Figures

J I

J I

Back Close

Full Screen / Esc



Discussion

Paper

|D

iscussionP

aper|

Discussion

Paper

|D

iscussionP

aper|

automatic sensors during the winter season was 58.9 cm, which is the basis for therange check threshold noted above.

It is acknowledged that the experimental configuration is such that any two identi-cal sensors do not measure the snow deposition on the same target. Owing to thevariability in snow distribution, there are differences in the depth of snow accumulated5

on different targets, which leads to differences in the data reported by sensors. Theseerrors are likely well represented by the variation of manual measurements at eachtarget. In addition, Fig. 7 shows that the snow depth sensor data show almost all zerosnow depth prior to the snow accumulation (ground clear), except for FEL (11A). Thus,the uncertainty determined in this study is most likely related to the sensor response10

to snow signal in various conditions, and not due to sensor malfunctions.

5 Results

5.1 Uncertainty of manual observations

The BEs and uncertainties of manual snow depth measurements are calculated toanalyze the spatial distribution of snow depth and uncertainty of manual snow depth15

measurements (Fig. 8 and Table 2). For calculation of BEs, the average snow depth ofsnow stakes 1 to 4 on base 12A is considered to be the reference for the purpose ofthis analysis. Figure 8a and Table 2 show that the BEs of base 12A (0.00, 0.86, and3.20 cm) are the smallest, which is to be expected, given the selection of the referencefor this analysis. Relative to the reference selected, the BEs of base 11A (6.46, 7.99,20

and 6.11 cm) are the largest. From these results, it was concluded that the snow depthis higher on base 11A (west side of the experiment area) than on base 12A (east sideon the experiment area). These results characterize the spatial distribution of snowdepth across the experiment area, as reported by the human observer. These resultsalso emphasize the necessity of several manual observations within the experimental25

site. The uncertainties (σdepth) for all pairs, on each base, and for each snow depth

4168





HESSD12, 4157–4190, 2015


depth measurements

J.-E. Lee et al.

Title Page



Tables Figures

J I

J I

Back Close

Full Screen / Esc



Discussion

Paper

|D

iscussionP

aper|

Discussion

Paper

|D

iscussionP

aper|

target are shown in Fig. 8b and Table 2. The total uncertainty for all pairs of stakes is1.81 cm, for this particular configuration and measurement resolution. The uncertaintyfor base 12A (1.55 cm) is the largest, while that for base 11A (1.5 cm) is the smallest.When comparing each snow depth target, the uncertainty for stakes 1 to 4 on base 20(1.58 cm) is the largest and that for stakes number 5 to 8 on base 12A (1.05 cm) is5

the smallest. The average uncertainties for each base (1.52 cm) are greater than thatfor each snow depth target (1.33 cm). The uncertainty gradually increases from target(1.33 cm) to base (1.52 cm) to all pairs of stakes on the site (1.81 cm). This is due tothe temporal variation of snow depth during manual observations, which was not takeninto account by the long-term BE removal. Thus, the uncertainty for manual snow depth10

measurements should be lower bound of the range of 1.33 to 1.81 cm.

5.2 Uncertainty of automatic observations

5.2.1 Comparison with manual observations

The snow depth measured by automatic snow depth sensors is compared with theaverage of the manual observations on the same snow depth target in Fig. 9. The au-15

tomatic snow depth observations at the halfway point of the manual observations (thatis, 10 min after starting the manual observations) are compared with the correspond-ing manual measurements, one data point per sensor, per day. The average snowdepth from manual observation on the same snow depth target is considered to bethe reference. The BE (NBE) ranged from −5.21 cm (−20.1 %) to 10.7 cm (57.8 %).20

The negative BE indicates underestimation of snow depth sensors relative to manualobservations.

Figure 10 and Table 3 show the BEs and NBRRMSEs of each snow depth sensor.The automatic snow depth sensors on base 12A (11A) tend to measure snow depthas much as 5.02–10.8 cm (0.46–5.21 cm) larger (less) than the manual observation25

(Fig. 10a). The NBRRMSEs of snow depth sensors on base 12A (16.5, 12.9, 12.2,and 8.90 %) are the largest and the those of snow depth sensors on base 20 (5.10,

4169





HESSD12, 4157–4190, 2015


depth measurements

J.-E. Lee et al.

Title Page



Tables Figures

J I

J I

Back Close

Full Screen / Esc



Discussion

Paper

|D

iscussionP

aper|

Discussion

Paper

|D

iscussionP

aper|

6.20, 7.00, and 5.70 %) are the smallest, based on the comparison among each base(Fig. 10b). The average NBRRMSEs of snow depth sensors of the same type arecalculated as follows: FEL= 10.1, JEN= 9.67, SR50A= 9.03, SOM= 8.63 %. Giventhe spatial variability in snow depth implied by the base-to-base variability in bias andrandom errors outlined above, the differences in random errors among the different5

sensor types are not considered to be significant.In general, the NBE (BE) ranges from −20.1 (−5.21 cm) to 57.8 % (10.7 cm) and the

random error ranges from 5.1 (1.00 cm) to 16.5 % (2.93 cm). Thus, the BE is more sig-nificant than the random error for the snow depth sensors used in this study. It is notcertain why the bias is so large; this question may require further thorough investiga-10

tion.

5.2.2 Comparison among automatic snow depth sensors

The snow depth measured by two snow depth sensors of the same type on differentbases is compared to quantify the instrumental uncertainty of individual snow depthsensors (Fig. 11). The data quality during a snow event could be poor for ultrasonic15

sensors, since it is a known limitation of these sensors that the sound waves are re-turned by the falling snow before reaching the target. This may have an impact on thecalculated uncertainty. A significant bias is shown in the comparison, and should beeliminated to quantify instrumental uncertainty. In addition, bimodal distributions areobserved for a few sensor pairs (SOM on base 20 vs. 12A and 11A vs. 12A; SR50A20

on base 20 vs. 12A, JEN on base 20 vs. 12A). The physical reasons are not known forthis peculiar characteristic.

The BEs and instrumental uncertainties of each snow depth sensor are shown inFig. 12. The snow depth sensors on base 12A are considered to be the reference forthe calculation of BE (diamonds in Fig. 12a), similar to the approach used for the as-25

sessment of manual observations. The circles in Fig. 12a represent the spatial distribu-tion of snow depth measured by the automatic sensors. To calculate these values, theBEs in Figs. 8a and 10a are added and the snow depths from sensors on base 12A are

4170





HESSD12, 4157–4190, 2015


depth measurements

J.-E. Lee et al.

Title Page



Tables Figures

J I

J I

Back Close

Full Screen / Esc



Discussion

Paper

|D

iscussionP

aper|

Discussion

Paper

|D

iscussionP

aper|

used as the reference. The BEs of snow depth sensors on base 20 and 11A are nega-tive. This could result from the spatial distribution of snow depth, and/or the systematicbias of snow depth sensors. The snow depths measured at bases 12A and 20 arelower than that measured at base 11A, based on the result from comparison of man-ual observations (Fig. 8a). Meanwhile, the snow depth sensors on bases 12A and 205

(11A) overestimate (underestimate) snow depth relative to the manual observations(Fig. 10a). Thus, the BEs of bases 20 and 11A are negative, as snow depths mea-sured by the snow depth sensors on base 12A are larger than those measured bysnow depth sensors on bases 20 and 11A.

When comparing each base, the instrumental uncertainties of each snow depth sen-10

sor on base 12A (2.08–3.08 cm) are the largest (Fig. 12b). The instrumental uncertaintyof FEL (11A) (0.62 cm) is the smallest in the comparison among each snow depth sen-sor type. The average instrumental uncertainties of snow depth sensors of the sametype are calculated as follows: SOM= 2.17, JEN= 1.86, SR50A= 1.84, FEL= 1.55 cm.The instrumental uncertainty of SOM is largest, and this could be due to the fact that15

the SOM reported the data in one centimeter resolution. These differences in instru-mental uncertainties of snow depth sensors are significant, given the variations in snowdepth across the site.

6 Summary and conclusion

This paper introduced a methodology for assessing the global uncertainty of instru-20

ments measuring snow depth using statistical measures and error propagation anal-ysis. The standard statistics such as BE (NBE), MAE (NMAE), RMSE (NRMSE), andBRRMSE (NBRRMSE) are calculated in statistical measures. The BEs of manual snowdepth measurements indicate the spatial distribution of snow depth on the CARE site.In addition, those computed in the comparison between manual and automatic snow25

depth measurements provide information about the systematic bias of each snow depthsensor. For error propagation analysis, the matrix is created from measurement pairs

4171





HESSD12, 4157–4190, 2015


depth measurements

J.-E. Lee et al.

Title Page



Tables Figures

J I

J I

Back Close

Full Screen / Esc



Discussion

Paper

|D

iscussionP

aper|

Discussion

Paper

|D

iscussionP

aper|

among three sensors of same type. The uncertainties of manual and automatic snowdepth measurements are calculated from the constructed matrix. This methodologyis demonstrated with its application on the snow depth data collected at the CAREsite from 20 December 2013 to 26 March 2014, to both manual and automatic sensormeasurements.5

The manual snow depth measurements are performed once a day, while the auto-matic measurements are output every 30 s. Over the course of the study period, thereported snow depth varied by location and method of measurement, likely being in-fluenced by the configuration of the experimental site. The values of maximum snowdepth recorded by manual observations are lower than those reported by automatic10

snow depth sensors. The snow depth is larger on base 11A (west) than 12A (east) atthe CARE site when we select the average snow depth of stakes 1–4 on base 12A asthe reference. The uncertainties of manual observations for all pairs, on each base, andfor each snow target were 1.81, 1.53, and 1.33 cm, respectively. The BEs of snow depthsensors on base 12A (11A) ranged from 5.02–10.0 cm (−5.31–0.46 cm) in comparison15

with manual observation sharing the same snow target. The average instrumental un-certainty was SOM= 2.17, JEN= 1.86, SR50A= 1.84, FEL= 1.55 cm.

As illustrated by the results of this study, the uncertainty of measurements can varyamong similar instruments collocated on the same site. The variability of results ob-tained through this study may indicate that other additional factors could influence the20

uncertainty of measurement of any sensor. The identification and treatment of the influ-ence of other factors could further improve the uncertainty of measurement, and shouldbe further investigated as part of SPICE. Two categories of factors are recognized toinfluence the uncertainty of measurements that would require further investigation. Thefirst is related to the site and sensor configuration, while the second is specific to a sen-25

sors ability to detect and measure snow on the ground.On the topic of site and sensor configuration, at the CARE site, although the simi-

lar instruments are located in close proximity, they do not measure snowfall and snowon the ground on the same sample area. The differences in the uncertainty of mea-

4172





HESSD12, 4157–4190, 2015


depth measurements

J.-E. Lee et al.

Title Page



Tables Figures

J I

J I

Back Close

Full Screen / Esc



Discussion

Paper

|D

iscussionP

aper|

Discussion

Paper

|D

iscussionP

aper|

surements for similar sensors would include the differences in the accumulation due totopography, wind influence, etc. Additionally, the accuracy of measurement of the initialdistance between the sensing element and the ground is critical, as is the ability tomaintain this distance throughout operations. This is best illustrated by the uncertaintyof measurement calculated for periods of time with no snow on the ground, before and5

after the snow season. The stability of the configuration would influence the ability toderive accurate snow depth measurements.

The second category of factors is related to how the sensor data is sampled andtreated. All snow depth sensors tested at the CARE site have their own internal dataprocessing algorithms and report data based on their own internal processing of mul-10

tiple raw samples. These sensors also output signal quality indicators, reflective ofthe interpretation of the returned raw signals, as processed by the internal sensor al-gorithms. Some manufacturers provide recommendations on the approaches for datafiltering.

The QC methodology reported in this paper has not taken into account the specific15

approaches recommended by manufacturers, focusing on testing generic approaches.Additional analysis of uncertainty of measurement and error propagation using moreadvanced data quality controlled will be included in the SPICE work.

The proposed methodology for assessing the uncertainty of measurement of auto-matic sensors is an effective tool to quantify and compare the ability to measure of20

various sensors, and SPICE will further investigate its use at various time scales andin different conditions, to develop recommendations on how to characterize the qualityof measurements of automatic measurements, in general.

Acknowledgements. The authors acknowledge the roles of Sorin Pinzariu and Jeffery Hooverof Environment Canada in the configuration of the snow depth measurement experiment at25

the Centre for Atmospheric Research Experiments (CARE) site and the provision of details ofthe measurement system, and the contributions of the site team and observers at CARE incollecting snow depth data and maintaining instruments. This work was funded by the KoreaMeteorological Administration Research and Development Program under Grant CATER 2013–2040.30

4173





HESSD12, 4157–4190, 2015


depth measurements

J.-E. Lee et al.

Title Page



Tables Figures

J I

J I

Back Close

Full Screen / Esc



Discussion

Paper

|D

iscussionP

aper|

Discussion

Paper

|D

iscussionP

aper|

References

Campbell Scientific Corp: SR50A Sonic Ranging Sensor Instruction manual, Canada, 63 pp.,2011.

Felix Technology Inc: Specifications of SL300 Snow Depth Sensor, available at http://www.snow-depth.com (last access: 25 January 2015), 2014.5

Jenoptik: SHM 30 Snow Depth Sensor Manual, Jena, Germany, 50 pp., 2009.Haji, M. J. de: Field test of the Jenoptik SHM30 laser snow depth sensor, KNMI Technical

Report No. 325, KNMI, De, Bilt, the Netherlands, 2011.Michelson, D. B.: Systematic correction of precipitation gauge observations using analyzed

meteorological variables, J. Hydrol., 290, 161–177, 2004.10

Nitu, R.: Cold as SPICE Determining the best way to measure snowfall, Meteorol. Tech. Int.,Meteorol. Tech. Int., 148–150, 2013.

Nitu, R., Rasmussen, R., Baker, B., Lanzinger, E., Joe, P., Yang, D., Smith, C., Roulet, Y. A.,Goodison, B., Liang, H., Sabatini, F., Kochendorfer, J., Wolff, M., Hendrikx, J., Vuerich, E.,Lanza, L., Aulamo, O., and Vuglinsky, V.: WMO intercomparison of instruments and methods15

for the measurement of solid precipitation and snow on the ground: organization of the formalexperiment, WMO, Brussels, Belgium, IOM No. 109, TECO-2012, 2012.

Rasmussen, R., Dixon, M., Vasiloff, S., Hage, F., Knight, S., Vivekanandan, J., and Xu, M.: Snownowcasting using a real-time correlation of radar reflectivity with snow gauge accumulation,J. Appl. Meteorol., 42, 20–36, 2003.20

Rasmussen, R., Baker, B., Kochendorfer, J., Meyers, T., Landolt, S., Fischer, A. P., Black, J.,Theiault, J. M., Kucera, P., Gochis, D., Smith, C., Nitu, R., Hall, M., Ikeda, K., and Gutmann,E.: How well are we measuring snow? The NOAA/FAA/NCAR winter precipitation test bed,B. Am. Meteorol. Soc., 93, 811–829, 2012.

Roebber, P. J., Bruening, S. L., Schultz, D. M., and Cortinas Jr., J. V.: Improving snowfall fore-25

casting by diagnosing snow density, Weather Forecast., 18, 264–287, 2003.Ryan, W. A. and Doesken, N. J.: Ultrasonic snow depth sensors for National Weather Ser-

vice Snow Measurements in the US: evaluation of Operational Readiness, 14th Symposiumon Meteorological Observation and Instrumentation, San Antonio, Texas, 17 January 2007Abstract number: 6.1, 2007.30

Ryan, W. A., Doesken, N. J., and Fassnacht, S. R.: Evaluation of ultrasonic snow depth sensorsfor US snow measurements, J. Atmos. Ocean. Tech., 25, 667–684, 2008.

4174





http://www.snow-depth.com



HESSD12, 4157–4190, 2015


depth measurements

J.-E. Lee et al.

Title Page



Tables Figures

J I

J I

Back Close

Full Screen / Esc



Discussion

Paper

|D

iscussionP

aper|

Discussion

Paper

|D

iscussionP

aper|

Sommer Mess-Systemtechnik: USH-8 Ultrasonic Snow Depth Sensor User Manual, Koblach,Austria, 30 pp., 2008.

Theriault, J., Rasmussen, R., Ikeda, K., and Landolt, S.: Dependence of snow gauge collectionefficiency on snowflake characteristics, J. Appl. Meteorol. Climatol., 51, 745–762, 2012.

WMO: WMO Guide to Meteorological Instruments and Methods of Observation, 7th edn., WMO5

No. 8, Geneva, 2008.WMO/CIMO: International Organizing Committee for the WMO Solid Precipitation Intercom-

parison Experiment Second Session, Final Report of the Second Session, Boulder, UnitedStates, WMO, Geneva, 74 pp., 2012a.

WMO/CIMO: International Organizing Committee for the WMO Solid Precipitation Intercompar-10

ison Experiment Third Session, Final Report of the Third Session, Brussels, Belgium, WMO,Geneva, 31 pp., 2012b.

WMO/CIMO: International Organizing Committee for the WMO Solid Precipitation Intercompar-ison Experiment Fourth Session, Final Report of the Fourth Session, Davos, Switzerland,WMO, Geneva, 54 pp., 2013.15

4175





HESSD12, 4157–4190, 2015


depth measurements

J.-E. Lee et al.

Title Page



Tables Figures

J I

J I

Back Close

Full Screen / Esc



Discussion

Paper

|D

iscussionP

aper|

Discussion

Paper

|D

iscussionP

aper|

Table 1. General characteristics of snow depth sensors. The L, ϕ, W, and H represent length,diameter, width, and height.

FEL SOM SR50A JEN

Range of measurement (m) 0.43–6.10 0.00–8.00 0.50–10.0 0.00-15.0Power requirements (VDC) 8–24 5–10 9–18 10–30Maximum current (mA) 80 200 250 –Operating temperature (◦C) −40–85 −35–60 −45–50 −40–50Dimensions (cm) L: 21.0 L: 35.0 L: 10.1 L: 30.3 W: 13.0

ϕ: 13.0 ϕ: 11.0 ϕ: 7.60 H: 23.4Weight (kg) 0.86 2.00 1.42 2.50Resolution (mm) – 1.00 0.25 1.00

4176





HESSD12, 4157–4190, 2015


depth measurements

J.-E. Lee et al.

Title Page



Tables Figures

J I

J I

Back Close

Full Screen / Esc



Discussion

Paper

|D

iscussionP

aper|

Discussion

Paper

|D

iscussionP

aper|

Table 2. BEs and uncertainties (σdepth) of manual snow depth measurements for all pairs, eachbase, and each snow depth target.

BE Uncertainty

All pairs – 1.81Base 12A – 1.55

20 – 1.5211A – 1.50

Snow depth target 12A 1–4 0.00 1.5612A 5–8 0.91 1.0512A 9–12 3.31 1.1520 1–4 1.96 1.5820 5–8 1.97 1.3420 9–12 3.80 1.2811A 1–4 6.62 1.4711A 5–8 8.18 1.4011A 9–12 6.28 1.18

4177





HESSD12, 4157–4190, 2015


depth measurements

J.-E. Lee et al.

Title Page



Tables Figures

J I

J I

Back Close

Full Screen / Esc



Discussion

Paper

|D

iscussionP

aper|

Discussion

Paper

|D

iscussionP

aper|

Table 3. BEs and NBRRMSEs of each snow depth sensor in comparison between manualobservation and snow depth sensors.

Base Snow depth sensor BE NBRRMSE

12A FEL 9.72 16.5SR50A 10.8 12.9JEN 9.52 12.2SOM 5.02 8.90

20 SR50A 2.30 5.10SOM 2.25 6.20JEN 2.34 7.00FEL 3.07 5.70

11A SOM −4.87 10.8FEL −4.08 8.20JEN −5.21 9.80SR50A −0.46 9.10

4178





HESSD12, 4157–4190, 2015


depth measurements

J.-E. Lee et al.

Title Page



Tables Figures

J I

J I

Back Close

Full Screen / Esc



Discussion

Paper

|D

iscussionP

aper|

Discussion

Paper

|D

iscussionP

aper|

Figure 1. Layout of snow depth measurement system at the CARE site during the 2013–2014winter season. The large circles (squares) represent bases (snow depth targets). The orangerectangles and small circles indicate snow stakes and snow depth sensors, respectively (cour-tesy: Environment Canada).

4179





HESSD12, 4157–4190, 2015


depth measurements

J.-E. Lee et al.

Title Page



Tables Figures

J I

J I

Back Close

Full Screen / Esc



Discussion

Paper

|D

iscussionP

aper|

Discussion

Paper

|D

iscussionP

aper|

Figure 2. Snow depth measurement system. (a) Grey plastic decking comprising snow depthtarget, (b) wooden snow stake, and (c) installation of snow depth sensors for base 11A (cour-tesy: Environment Canada).

4180





HESSD12, 4157–4190, 2015


depth measurements

J.-E. Lee et al.

Title Page



Tables Figures

J I

J I

Back Close

Full Screen / Esc



Discussion

Paper

|D

iscussionP

aper|

Discussion

Paper

|D

iscussionP

aper|

Figure 3. Time series of snow depth from snow stakes on base (a) 12A, (b) 20, and (c) 11Afrom manual observations over the period from 22 December 2013 to 26 March 2014. The linecolors indicate individual snow stakes. (d) Average snow depth of four snow stakes on samesnow depth target. The line color indicates each base. The solid, dotted, and dashed linesrepresent average snow depth of stake numbers 1–4, 5–8 and 9–12 on the same snow depthtarget.

4181





HESSD12, 4157–4190, 2015


depth measurements

J.-E. Lee et al.

Title Page



Tables Figures

J I

J I

Back Close

Full Screen / Esc



Discussion

Paper

|D

iscussionP

aper|

Discussion

Paper

|D

iscussionP

aper|

Figure 4. Photograph of the uneven snow deposition on the surface of snow depth targets.

4182





HESSD12, 4157–4190, 2015


depth measurements

J.-E. Lee et al.

Title Page



Tables Figures

J I

J I

Back Close

Full Screen / Esc



Discussion

Paper

|D

iscussionP

aper|

Discussion

Paper

|D

iscussionP

aper|

Figure 5. Time series of snow depth from (a) FEL, (b) SOM, (c) SR50A, and (d) JEN forthe period from 20 December 2013 to 26 March 2014. The black color indicates sensors onbase 12A. The blue color indicates sensors on base 20. Finally, the red color indicates sensorson base 11A.

4183





HESSD12, 4157–4190, 2015


depth measurements

J.-E. Lee et al.

Title Page



Tables Figures

J I

J I

Back Close

Full Screen / Esc



Discussion

Paper

|D

iscussionP

aper|

Discussion

Paper

|D

iscussionP

aper|

Figure 6. Same as in Fig. 5 except for data after applying the QC procedure described inSect. 4.3.

4184





HESSD12, 4157–4190, 2015


depth measurements

J.-E. Lee et al.

Title Page



Tables Figures

J I

J I

Back Close

Full Screen / Esc



Discussion

Paper

|D

iscussionP

aper|

Discussion

Paper

|D

iscussionP

aper|

Figure 7. Same as in Fig. 6 except for the events prior to snow accumulation. Snow was mostlynot seen on the ground on 08 November 2013 and from 12 November 2013 to 15 November2013.

4185





HESSD12, 4157–4190, 2015


depth measurements

J.-E. Lee et al.

Title Page



Tables Figures

J I

J I

Back Close

Full Screen / Esc



Discussion

Paper

|D

iscussionP

aper|

Discussion

Paper

|D

iscussionP

aper|

Figure 8. (a) BEs and (b) uncertainties of manual snow depth measurements. The BEs arecalculated for each snow depth target. The orange bar represents the σdepth for all pairs. The2nd–4th (5th–13th) columns indicate the σdepth for each base (snow depth target). The color ofbars indicates the same base; the blue, red, and green bars represent σdepth for bases 12A, 20,and 11A.

4186





HESSD12, 4157–4190, 2015


depth measurements

J.-E. Lee et al.

Title Page



Tables Figures

J I

J I

Back Close

Full Screen / Esc



Discussion

Paper

|D

iscussionP

aper|

Discussion

Paper

|D

iscussionP

aper|

Figure 9. Scatter plots of snow depth measured by automatic snow depth sensors (y axis) andaverage snow depth of manual observation (x axis) on bases 12A (left), 20 (middle), and 11A(right): FELIX (first column), SOMMER (second column), SR50A (third column), and JENOPTIK(fourth column). The values in brackets indicate the NBE, NMAE, NRMSE, and NBRRMSE.

4187





HESSD12, 4157–4190, 2015


depth measurements

J.-E. Lee et al.

Title Page



Tables Figures

J I

J I

Back Close

Full Screen / Esc



Discussion

Paper

|D

iscussionP

aper|

Discussion

Paper

|D

iscussionP

aper|

Figure 10. (a) BEs and (b) NBRRMSEs of each snow depth sensor. The blue, purple, red, andgreen diamonds represent FEL, SR50A, JEN, and SOM.

4188





HESSD12, 4157–4190, 2015


depth measurements

J.-E. Lee et al.

Title Page



Tables Figures

J I

J I

Back Close

Full Screen / Esc



Discussion

Paper

|D

iscussionP

aper|

Discussion

Paper

|D

iscussionP

aper|

Figure 11. Scatter plots of snow depth measured by two snow depth sensors of the same typeon bases 20 vs. 12A (left), 11A vs. 12A (middle), and 11A vs. 20 (right): FELIX (first column),SOMMER (second column), SR50A (third column), JENOPTIK (fourth column). The values inbrackets indicate the NBE, NMAE, NRMSE, and NBRRMSE.

4189





HESSD12, 4157–4190, 2015


depth measurements

J.-E. Lee et al.

Title Page



Tables Figures

J I

J I

Back Close

Full Screen / Esc



Discussion

Paper

|D

iscussionP

aper|

Discussion

Paper

|D

iscussionP

aper|

Figure 12. (a) BEs of each snow depth sensor. The diamonds (circles) are calculated by con-sidering the snow depth sensors on base 12A as reference (the BEs in Figs. 8a and 10a areadded and snow depth from snow depth sensor on base 12A are then used as reference). (b)σdepth of each snow depth sensor. The blue, purple, red, and green diamonds indicate FEL,SR50A, JEN, and SOM.

4190





Documents

Uncertainty analysis for evaluating the accuracy of snow …...to measure a few millimeters of snow, but have issues with the representativeness of snow depth measurements, given the