2 1

Processing Static Data

Charles “Chuck” Ghilani, Ph.D.

Surveying Engineering

Penn State University

Class Etiquette

• Turn off all cell phones– Or set them to vibrate

• Ask questions at any point during the class.– Simply speak up so that all can hear your

question

2 2

Course Description

This workshop will present

1. Methods used to check data before anadjustment

2. Statistical tests that can be used to help identifyerrors in GNSS data

3. How least squares post-adjustment statistics canbe used to check data for errors

4. Why processing the data yourself may providebetter solutions

Course Outline

• Pre-adjustment methods– Preparing the data

– Checking your reference frame

– Downloading a precise ephemeris

– Setting appropriate QA/QC values

– Geometric checks of data

• Post-adjustment methods– Analyzing the post-adjustment statistics

– Checking the residuals

2 3

Field Crew Preparations

• Always create a site log sheet– Minimum information on sheet

• Station occupied

• Session Start and End times

• PDOP at beginning and end of session

• Receiver – S/N

– Height (vertical or slant)

• Approximate station position

• Obstruction diagram and site sketch

2 4

Why Collect Information?

• Check data on download– Possible errors include

• Incorrect height or type (Slant instead of vertical)

• Obstruction/multipathing at site– PDOP does not match planning software value

– Visibility diagram shows obstruction to critical satellite

• Session’s time does not match second receiver – Started/stopped at different times

Collect Necessary Data

• Download precise ephemeris from NGS web site (http://www.ngs.noaa.gov)– Data & Imagery

• Orbit Data

• Orbit Data (on left bar)

2 5

Select Orbit

• Ultra-rapid– within 6 hr

– igu

• Rapid– within 13 hr

– igr

• Precise– about 2 weeks

– igs

• Note file name

Select for Data

Orbit File Naming Convention

• igrnnnnx.aaa, where– igr = International GPS Service rapid ephemeris

– nnnn = GPS week number, e.g.1775, 1776, ...

– x = day of the week where• Sunday = 0, Monday = 1,..., Saturday = 6

– aaa = file type where• sp3 is the precise ephemeris

• erp is the Earth rotation parameters

• Example– Wednesday, January 20, 2014 is igr17763.sp3

2 6

IGS Orbit Links

Precise ephemerides

ultra-rapid/rapid ephemerides

GNSS week can be found athttp://www.rvdi.com/freebies/gpscalendar.html

2 7

Precise Ephemerides

Sunday

Monday

.Z means it is a zipped file

CORS Download• To create at least one connection to a CORS

station, one session must be long enough to support the distance to the nearest CORS– E.G., 50 mi to nearest CORS

• Short “rapid/fast” static methods only viable for baselines < 20 km

– Session length should be 20 m + 2 m/km• Distance in km is 80.5 km

• Length of session should be 20 m + 161 m = 3h 01m!

– Quality of connection will be lower with less time• Or baseline processing of connection may not be possible – can’t

fix ambiguities

2 8

Getting CORS Data

Select Link

Getting CORS Data

• Select Data Products on left– Note: first screen

skipped

• Select User Friendly CORS (UFCORS)

2 9

Getting CORS Data• Enter date and time of survey

– Be sure to check appropriate time zone

– Only full hours downloadable

– Session started at 10:23 EDT and lasted 3h01m

• End time would be 13h24m

– Need to start at 10:00 EDT and obtain 4h of data• Can only get 15s epoch rate with this length of data

• Sufficient even if data collect at 5s

Getting CORS Data

• Select Site ID

• Select Sampling Rate– 15s for session longer

than 2 hr

• Select additional files– Coordinate file

– Meteorological file

– NGS data sheet

– Precise ephemeris

– Submit Request• Should get data in minutes

2 10

What You Will Get

Create Your Job• Create Job

– Set QA/QC parameters based on longest baseline

– For a horizontal survey• Assume manufacturer states accuracy of 3 mm + 0.5

ppm for horizontal static survey

• Assume setup error of ±1.5 mm

• Connection to CORS was 80.5 km

• 99% (2.576) estimated error on line is

• . 2.576 2 1.5 3 0.5 80.5 104mm

= ±.34 ft!

2 11

Problem with QA/QC Numbers

• 0.34 ft is acceptable to find blunders in 80.5 km line but what if remainder of lines are under 1 km?

• Try parts-per-million (ppm) – For example

10480.5

1.3

– But a 0.5 km line with same specifications would yield

• 2.576 2 1.5 3.500,000 9.5mm

• A ppm of 19!

• Need to analyze baselines separately after processing

Create Your Job

• Set Job units

• Set datum/reference frames– Dependent on source of control

• CORS/HARN use IGS08/NAD83 (2011) (epoch 2010)

• DoD uses WGS84 (G1673)

• Legacy control uses original NAD83– Note that you will have to localize survey to obtain

these coordinates!

– Load GEOID model to get elevations

2 12

Geoid Model

• Select GEOID from Data & Imagery– Only needs to be

performed once. After that simply load it into software.

– Select appropriate geoid

Geoid Model

• Select GEOIDXX Downloads

• Select appropriate site location

2 13

Geoid Model• Select appropriate

grid and file type– PA

• Type typically is le (little endian) for PC software

Load Data

• Load control coordinates first

• Load session data– Typically in a proprietary format, which contains

observations and broadcast ephemeris

– CORS RINEX files are• Navigation (broadcast ephemeris) files *.YYN

• Observation files *.YYO

• You must load the broadcast ephemeris to use the precise ephemeris

• Don’t forget to load precise ephemeris!

2 14

Check Occupations

• Get your site log sheets and look for1. Misidentified stations

• Field crews may mislabel stations because they are asked by the controller whether they are sure they want to overwrite the coordinates on the controller. E.g., Wil1, Wil1_2, ...

• Try entering coordinates in GOOGLE Earth to check possible setup problems/obstructions

2. Incorrect antenna type/heights• E.g. Controller in units of feet but antenna on 2 m

rod. Enter 2 with ft units resulting in height of 0.6096 m!

Check Occupations

• Incorrect antenna selection– Sometimes true on CORS sites

– Field crews sometimes not sure of their exact antenna type

– Sometimes obsolete antenna calibrations are maintained in your manufacturer’s file!

2 15

Example of Field Errors

• Note wrong antenna. Should be GR-3.

• Height entered in field as 2 ft but software in meters

Fix Errors

• Software has method to change entry errors

2 16

Check Session Lengths• Be sure that sessions

overlapa) Sessions start and stop

times different but session lengths sufficient to obtain valid baseline vectors

b) Connection to CORS station started too late to use with first session of data.

(a) Good

(b) Problem?

Too Short?

Sessions not started togetherPoor overlap

Check Session Lengths• Disable short sessions

2 17

Check Session Lengths• Disable short sessions

– Software often provides automatic removal using minimum session length

Set Processing Properties

• Methods vary by manufacturer but at a minimum set– Elevation mask

– Maximum length of vector to process• Removes connections between different crews in same project

– Shortest length of session to process• Removes sessions that are too short to provide good solutions

2 18

Check Baseline Processing Solutions

• Process baseline vectors– Check for float solutions

– Check for high variances

Trivial Baselines• Number of nontrivial baselines

n = r – 1 where n = # of nontrivial baselines (blue and red)

r = number of receivers

• Trivial baseline (dashed line)

S1

A B

C

S2 S3

S4

2 19

Disable Trivial Baselines

• Trivial baselines inflate the accuracies of the positions

• Only r − 1 nontrivial baselines per session

• Not a problem if only 2 receivers are used

• Disable one of the three baselines– Hint: Never close a figure in a single session

Trivial Baseline Vectors Disabled

Only one baseline processed during this session

2 20

Preadjustment Analysis• Three basic analyses

– Check fixed baselines

– Check repeat baselines

– Check loop misclosures• Can be used to isolate a baseline with a blunder

before an adjustment

• Checks repeatability of equipment and software

• Checks field procedures

• Checks control coordinates

Parts per Million• Typically computed as

1,000,000

• Isolate errors/blunders in data before adjusting network

• Parts-per-million (ppm) should match anticipated accuracy of network

• Isolate "bad" baselines by checking loop misclosures using all baselines– Support is provided for this in GNSS software typically

• PPM’s do NOT provide the accuracy of the work.

2 21

Checking Fixed Baselines

1. List the vector components of the baseline

2. Compute the vector components from the control coordinates

3. Compute the misclosures [absolute value of the difference between the observed (1) and computed (2) components]

4. The ppm is computed asmisclosure

∆ ∆ ∆1,000,000

Table 1 Baseline vector observations

Line ∆X ∆Y ∆Z

AC 11644.2232 3601.2165 3399.2550

AE −5321.7164 3634.0754 3173.6652

BC 3960.5442 −6681.2467 −7279.0148

BD −11167.6076 −394.5204 −907.9593

DC 15128.1647 −6286.7054 −6371.0583

DE −1837.7459 −6253.8534 −6596.6697

FA −1116.4523 −4596.1610 −4355.9062

FC 10527.7852 −994.9377 −956.6246

FE −6438.1364 −962.0694 −1182.2305

FD −4600.3787 5291.7785 5414.4311

FB 6567.2311 5686.2926 6322.3917

BF −6567.2310 −5686.3033 −6322.3807

AF 1116.4577 4596.1553 4355.9141

AB 7683.6883 10282.4550 10678.3008

2 22

Checking Fixed Baselines• Using the data from Table

– Baseline AB connects 2 control stations• A: (402.3509, − 4652995.3011, 4349760.7775)

• B: (8086.0318, −4642712.8474, 4360439.0833)

• AB: (7683.6883, 10282.4550, 10678.3008) (observed)

– Computed vector AB from coordinates• ∆X = 8086.0318 − 402.3509 = 7683.6809

• ∆Y = −4,642,712.8474 + 4,652,995.3011 = 10,282.4537

• ∆Z = 4,360,436.0833 − 4,349,760.7775 = 10,678.3058

– Misclosures (differences):• dX = |7683.6883 − 7683.6809| = 0.0074 m

• dY = |10282.4550 − 10282.4537| = 0.0013 m

• dZ = |10678.3008 − 10678.3058| = 0.0050 m

Checking Fixed Baselines

• Compute length of baseline– Can use either baseline vector components

7683.6808 10282.4537 10678.305816697.130

– Compute PPM values for each component∆X: 0.0074/16697.126 = 0.44 ppm

∆Y: 0.0013/16697.126 = 0.08 ppm

∆Z: 0.0050/16697.126 = 0.30 ppm

– Check against FGCS standards or manufacturer specifications

2 23

Specifications Check• Assume the following

– Manufacturer specifications are (5 mm + 1 ppm)

– Setup uncertainty is 1 mm (≈0.003 ft)

– Actual misclosure is

Misclosure 7.4 1.3 5 9.0mm– Specifications yield

2 1 5 16,697,130 17.5mm

– At 99%: 2.576(17.5) = ±34.2 mm (9 < 45.1)

Check Repeat Baselines

• Compute differences in each component (JASH – WIL1 B)∆X = −34.871 + 34.869 = 0.002∆Y = 57.621 − 57.621 = 0.000∆Z = 0.751 − 0.751 = 0.000

• Use either length of line • Compute ppm's for each baseline component

∆X: 0.002/67.361 29.7 ppm∆Y: 0.000/67.361 = 0 ppm∆Z: 0.000/67.361 = 0 ppm

• Again check against standards or specifications

2 24

Specifications Check

• Total misclosureMisclosure Δ Δ ∆ 2mm

• Specifications (assume setup error = 1 mm)

2 1 3110

67,361 3.3mm

• At 99%– σ = 2.576(3.3) = ±8.5 mm (2<8.5)

Checking Loop Closures• Check the loop closures

– Typically use triangles

• The ∆Ns, ∆Es, and ∆Hts should all sum to zero (similar to a differential leveling loop.– Sum up individual baseline components that form

a loop

– Compute the overall loop misclosure as a slant distance

ppm = lc/length * 1,000,000

2 25

Checking Loop Closures

• Loop WIL1−WIL1B − JASHBaseline dN (m) dE (m) dHt (m) Length (m)

WIL1-WIL1 B −81.692 −44.940 −17.155 94.802

WIL1 B-JASH 34.869 −57.621 −0.751 67.354

JASH-WIL1 46.817 102.561 17.894 114.152

SUM

• Sum misclosures in each component• Sum baseline lengths• Compute overall misclosure

• Misclosure: 0.006 0.012 0.0134 m

−0.006 0.000 −0.012 276.309

Check Loop Misclosures

• Misclosure was 0.0134

• ppm = .

.1,000,000 48.6

• Against specifications– Assume setup error = 1 mm

– Assume manufacturer specs of 3 mm + 0.5 ppm

– 99% error =

2.576 6 3 3 3 .

, ,276,309 23.2mm

– 13.4<23.2 so misclosure is acceptable!

• 23 mm is an acceptable QA/QC number

2 26

Loop Misclosures

• Software can do this– You can view

• All loops selected

• Only failed loops

– Use failed loops to identify possible problems such as• Height of antenna

• Incorrect antenna selected

• Be sure QA/QC or ppm set correctly in job configuration to match length of lines

Set Job Configuration

• Quality control checks need to match length of lines– Should be set to 99% error level, which means

a multiplier of 2.576

– Intended to catch “large” errors before adjustment

2 27

Prepare to Adjust Data

• Set control– Must have a minimum of

• 1 horizontal point (base station) fixed

• 1 vertical point fixed

• Use OPUS/CORS to obtain coordinates for 1 station– Caution: You really need 3 vertical control to fix horizontal

plane in space for large project

Job Configuration for Adjustment

• Select confidence level for tau criterion check and χ2 check– Typically 99% used for

blunder detection

• Select by method of rejection criterion– Quality Control (QC)

2 28

Job Configuration for Adjustment

• Select confidence level for tau criterion check and χ2 check– Typically 99% used for

blunder detection

• Select by method of rejection criterion– Quality Control (QC)

– Tau criterion a statistical method

Adjust Baselines

• Look at “rejected baselines”

• Check “Goodness of Fit” test– Some software states pass or fail

– Some show range and reference variances• UWE should be in bounds of UWE to pass goodness

of fit test – Neither did here!

2 29

99% χ2 confidence interval

Results from a GNSS Adjustment

Sample Report

Standard Deviation forHorizontal Component

Standard Deviation forVertical Component

Adjustment type: Plane + Height, Minimal constraint Confidence level: 99 % Number of adjusted points: 9 Number of plane control points: 1 Number of used GPS vectors: 31 A posteriori plane UWE: 1.182194 , Bounds: ( 0.7369072, 1.272843 ) Number of height control points: 1 A posteriori height UWE: 1.304971 , Bounds: ( 0.6345145, 1.385954 )

Goodness of Fit Test

• Typically before an adjustment observations are weighted as

1

• After the adjustment the reference variance should be equal to 1

2 30

Goodness of Fit Test

• However in a GNSS adjustment the weights come from the baseline reduction adjustment as

Σ• Baseline reduction adjustment is

unweighted

• Thus W from this adjustment is not always good– E.G. Does not include correlation in baselines

It Can Fail If

• Computed as∑

redundancies– It could be that the residuals () are too large

• Analyze the individual residuals and look for "large" values– It could be that the residuals are too small! But not since the

adjustment's variance is greater than 1.

– It could be that the weights are incorrect

– It could be that the test failed!• It is wrong #% of the time. At 99%, incorrect result 1% of time

• Consider it a warning flag in need of further investigation. We have been warned.

2 31

Normal Distribution Curve• Plotting residuals from a population of data

• Normal distribution curve defined by

2 221( )

2f x e

f(x)

x

of data

εε

Properties of Distribution• Laws of Probability

– Positive and negative errors occur with equal probability and frequency

– Small errors occur more frequently than large errors• Large errors seldom occur

2 32

Useful Probabilities

Symbol Multiplier Percentage

E50 0.675 σ 50

E90 1.645 σ 90

E95 1.960 σ 95

E99 2.576 σ 99

E99.7 2.965 σ 99.7

• Note: 99% of data in a normally distributed data set should be within ±2.576σ of the mean

• Use these values to define large error

Properties of Random Errors

• Random errors are1. Generally small in magnitude

– Large random errors seldom occur

2. Follow the laws of probability– Are as likely to be negative as positive in sign

3. Impossible to avoid

• Use these principles to help identify blunders

2 33

Analyze Adjustment Residuals

Partial list of NEU Residuals

Name dN (m) dE (m)dHt(m)

Dist (m)Res n (m)

Res e (m)

Res u (m)

JAGER−SHUPPSKI −49.497 77.779 4.042 92.290 0.004 0.002 0.012

JAGER−SHUPPSKI −49.498 77.777 4.039 92.290 0.003 0.001 0.009

JAGER−WIL1 −46.617 128.576 20.975 138.379 −0.004 0.000 −0.009

JAGER−WIL1 −46.616 128.577 20.982 138.381 −0.003 0.001 −0.002

JASH−WIL1 46.817 102.561 17.894 114.163 −0.001 −0.006 −0.010

• Residual = valueComp – valueObs

• Positive residual means that observed too low

Check Jager-Shuppski

• Length = 92.290 m

• Assuming setup errors of 1.5 mm

• Manufacturer specification for vertical of 5 mm + 0.5 ppm

• 2.576 2 1.5 5.92,290

• E99 = ±14 mm

• So 12 mm < 14 mm

• Baselines are less than manufacturer’s specs

2 34

Review All Residuals

• Set up spreadsheet to check residuals of all observations– Setup error = ±1.5 mm

– Const error: Hor = 3 mm; Ver = 5 mm

– ppm: 0.5

Microsoft Excel Worksheet

Tau Criterion

• Used in automatic blunder detection

2 35

Statistical Methods for Isolating Blunders

• Create the covariance matrix of the residuals

• Compute standard deviations for the residuals– If we know Sv, then we can predict range of

acceptable v’s

• Use tau criterion (τ test) to statistically isolate blunders

Theory: Residual’s Covariance Matrix

• LetAX + C = L + V

– Where • C is a vector of constants and A,X, L and V as

described previously

• RewritingV = AX − B; where B = L − C

2 36

Theory: Residual’s Covariance Matrix

• The least squares solution is= (ATWA)−1ATWB; where B = L − C

• Let ε represent a vector of true errors for the observations

• Thus

where is the true values for X

Theory: Residual’s Covariance Matrix

• SubstitutingWA

• Expanding

• Since (ATWA)−1 ATWA = I

• Factoring WεWA

2 37

Theory: Residual’s Covariance Matrix

Qxx

• Again WA

• We define as

– Then V = −QυυWε

• Recognizing that Qll = AQxxAT

– Then Qυυ = W−1 − Qll

• Finally, Sυυ = Qυυ, – covariance matrix for residuals

Tau Statistic Theory

• Standardized residual is computed as

• A computed parameter divided by its standard deviation is a τ statistic

where qii is the diagonal of the i’th diagonal of the cofactor matrix for the residuals, Qυυ

2 38

Tau Statistic

• Alan J. Pope (from NGS) suggested the tau statistic be used where τ is computed as

• With the rejection criteria being |τi| >

• The test are generally performed at α = 0.001, or 99.9% probability.

Tau Statistic

• Works very well for small blunders.

• Standardized residual is compared against a statistical quantity.– Observation may have a blunder if standardized

residual > rejection level

• Software either automatically removes observations with blunders or highlights them

2 39

Manual Steps

1. Locate all measurements that qualify for rejection.

2. Reject the single observation with the largest standardized residual.

3. Repeat adjustment and steps 1 & 2 until all observations the qualify for rejection are rejected.

4. Reinsert observations that were rejected into adjustment in a one-at-a-time fashion to see if they are still rejected.

Automatic Blunder Removal

• Software displace how many equation used versus the number rejected.

2 40

Automatic Blunder Detection

However, auto blunder detection sensitive in incorrect weighting model

Manual Method

• Software highlights observations that failed test

2 41

So Why Process Your Own Data?

• Professionals should know the product they deliver to a client

• Using OPUS provides a nonhomogeneous solution when more than one station is solved

• Self processing combines a single homogeneous solution

So Why Process Your Own Data?

• Allows you to determine what went wrong and to correct actions that caused mistake– Will hopefully result in better field and office

procedures in the future

• You can combine several different sources of data– Combined conventional observations with

GNSS observations

2 42

So Why Process Your Own Data?

• Use OPUS or the CORS network to connect to the national spatial reference system.– Create reproducible coordinates for future

– Don’t use OPUS to create coordinates for all stations in your survey

• Professionals should know the product they deliver to a client

QuestionsTrue - False

1. Site log sheets allow the user to identify entry errors in survey controller.

2. A precise ephemeris is available within 6 hours of data collection.

3. A large residual is one that is greater than 10 cm.

4. The tau criterion is used in automatic blunder detection

5. Professionals should know the product they deliver to their client

T F

T F

T F

T F

T F