Applying a Decade of RCS Calibration Activity Experience to the IR/Visible/UV Measurement Community Dr. Randy J. Jost Assistant Professor of Electrical

Applying a Decade of RCS Calibration Activity Experience to the IR/Visible/UV Measurement

Community

Dr. Randy J. JostAssistant Professor of Electrical & Computer Engineering

Utah State University(435) 797-0789

[email protected]

Roger W. DavisConsultant, DynCorp/CSC

(301) [email protected]

mailto:[email protected]

mailto:[email protected]

Overview• Introduction

• Defining Certification

• Key Differences Between RCS Community and EO/IR Community

• Creating a Range Book

• Report of Measurement / Uncertainty Analysis

• Moving Forward as a United Community

• How Big a Tent ?

• Home – conference

• Recommendations

• Summary

• References

Introduction

There is general agreement that the EO/IR has need of a mechanism to improve it ability to:

• Compare data between sensors, systems and programs

• Understand the quality of measurements and errors

• Gain consensus in a very diverse community

Key question is “How do we get there from here?”

The purpose of this presentation is to raise awareness of some of the key issues, engender discussion, and suggest a starting point for addressing the concerns of the community members.

Certification - Why?• When you are up to your rear in alligators, it’s hard to

remember your objective was to drain the swamp (moat) or ….• Certification helps you to achieve the desired end result by

understanding and documenting the “measurement alligators” a.k.a error sources.

Non Sequitur by Wiley,Published in the Washington Post, 15 Jan 1997

Certification - What?

• Objective: Uniformly improve signature measurements across the EO/IR spectrum by all “measurement facilities” for all sponsors

• Method: Establish ISO-9000 Quality System within the EO/IR measurement community

• How: Standardize and document measurement processes & procedures

Certification - What?• A Quality System:

– Within a Formal Structure, establish:

• Defined processes & responsibilities

• A means for evaluating effectiveness

• A means for making improvements

– Based on a Cultural Mindset that includes:

• All activities involved in generating the product* or service must be controlled to some extent, ideally in a systematic manner

• Every person & action affecting the quality of the product or service is a part of the Quality System

* Product = EO/IR Measurement Data Set(s)


• Preparation

– Educating, Assessing, Planning, Scheduling

• Documentation

– Define policies, objectives

– Develop/document procedures & work instructions

• Implementation

– Operate as documented, internal audits, corrective & preventive actions, management reviews

• External Assessment

– External evaluation & registration are optional

• Surveillance & Improvement

– On-going internal audits, continuous improvements

Phases of a ISO-9000 Quality System


• Key ISO Definitions:• ISO-XX “Registered”: A company or sub-unit attests that it

has met the relevant criteria for ISO-XX as determined by an internal “self inspection” review. (Sometimes called “self” certification)

• ISO-XX “Certified”: A company or sub-unit who is ISO-XX “Registered” is reviewed by a third party of peers (from other companies or other divisions) and that third party attests that the company does indeed meet the relevant criteria for ISO-XX

• ISO-XX “Accredited” : Company is reviewed by an independent organization/firm/company who specializes in ISO-XX compliance. (I.E. reviewers are full-time specialists). Company is “accredited” if this firm attests they meet the relevant criteria for ISO-XX.

Lowest

Medium

Highest

ISO XX Compliance

Level


• Key ISO Definitions:• ISO-XX “Registered”: A company or sub-unit attests that it

has met the relevant criteria for ISO-XX as determined by an internal “self inspection” review. (Sometimes called “self” certification)

• ISO-XX “Certified”: A company or sub-unit who is ISO-XX “Registered” is reviewed by a third party of peers (from other companies or other divisions) and that third party attests that the company does indeed meet the relevant criteria for ISO-XX

• ISO-XX “Accredited” : Company is reviewed by an independent organization/firm/company who specializes in ISO-XX compliance. (I.E. reviewers are full-time specialists). Company is “accredited” if this firm attests they meet the relevant criteria for ISO-XX.

Lowest

Medium

Highest

ISO XX Compliance

Level


• Registered : Incurs 80+% of sunk cost and effort

• Certification: Most of the Technical Benefit comes from the third party peer review

– Different perspective offered by reviewers

– Reviewers have a wealth of experience to share (add value)

– If you can’t explain a process properly to a reviewer, chances are the process needs clarification (feedback).

– If reviewer(s) rate a practice “unsatisfactory” they are required to provide written comments so range becomes satisfactory! (Goal: Pass, not fail)

– Certification Effort protects proprietary / competitive interests

– ANSI-Z-540 review process is confidential…only the final certification compliance report is available to customers

– “Corrective actions” need not be shared once implemented (Embarrassment factor is understood and respected)

Why Certified instead of Registered or Accredited?


The term “CERTIFICATION,” as used in the DoD demonstration program and RCS community (in the sense of a facility being “certified”), is defined as having successfully completed a detailed review, of an RCS Range Book1 by a peer review committee using published evaluation criteria2.

1 Written in accordance with the quality documentation standard:

“ RCC/SMSG Report 804-01” 2 Evaluation Criteria are in the appendix of RCC Report 804-01

For more information on the history and details of the DoD RCS Range Certification Demonstration program, see the paper “Lessons Learned in Applying ANSI/NCSL Z-540 to the RCS Community and Transferring Those Lessons to the IR Measurement Community,” by Davis/Jost on the CalCon CD provided with the registration material.

Certification - What?• Detailed review and compliance assessment that a

range has met the relevant quality assurance documentation standard for ANSI Z-540

• Certification is:– A Review committee’s positive assessment of compliance

with the ANSI-Z-540 standard

– An endorsement of the range’s measurement processes

– “Deliverable” Documentation Package is the Range Book

• Certification is NOT:– A rating system

– A review committee endorsement of day-to-day quantitative capabilities (accuracy, uncertainty, etc)

Key Differences Between RCS and EO/IR Communities

• The IR community is a much more segmented group than the RCS community, and thus faces additional challenges and advantages.

• Government is not the only driver, but still a big player.

• Because of the large commercial component, community consensus is even more important.


• Challenges– More types of measurements => more types of

calibration approaches.• reflection, emission, transmission and absorption

processes.

• Incoherent, partially coherent, coherent radiation

– Spectrum of interest is much more “diverse” than that of the RCS community.

– Environment has a much bigger impact on measurement and resultant uncertainty.


• Range book sections will have to be tailored for many more different types of measurements:• Active vs. Passive• Spectral vs. temporal• Imaging vs. non-imaging• Visual vs. IR vs. UV vs. Laser• Single band / Broadband vs. Multi-band

vs. Hyper-spectral• Component vs. subsystem vs. system

Building a Range book for the EO/IR Community

1 Introduction and Endorsement2 References3 Glossary4 Organization and Management5 Quality System, Audit, and Review6 Personnel7 Accommodation and Environment8 Equipment and Reference Materials9 Measurement Traceability and Calibration10 Calibration Methods

11 Handling of Calibration Items12 Records13 Certificates and Reports14 Subcontracting of Calibration15 Outside Support Services and Suppliers16 Complaints17 Inter-range Comparison Programs18 Data Processing Procedures19 Range-Specific Uncertainty Analysis20 Ongoing Research, Planned Improvements

RCS community has developed the following structure, following the ANSI Z540 “outline”:




Priority One – Direct Impact on Data QualitySections 9, 10, 11, 17, 18, 19





Priority Two – Enabling Data QualitySections 5, 8, 13, 20





Priority Two – Enabling Data QualitySections 5, 8, 13, 20

Priority Three – Supporting DocumentationSections 1, 2, 3, 4, 6, 7, 12, 14, 15, 16


1 Introduction and Endorsement2 References3 Glossary - Terminology4 Organization and Management5 Quality System, Audit, and Review6 Personnel7 Accommodation and Environment8 Equipment and Reference Materials9 Measurement Traceability and Calibration10 Calibration Methods


Priority One – Direct Impact on Data QualitySections 3, 9, 10, 11, 17, 18, 19, 20

Priority Two – Enabling Data QualitySections 5, 8, 13

Priority Three – Supporting DocumentationSections 1, 2, 4, 6, 7, 12, 14, 15, 16

• Appendix 2 is a copy of NIST TN1297, 1994 Edition.

• This should be the starting point for developing range book content.

• Supports both Section 3 and Uncertainty Analysis

NIST Handbook 152

Recommended Practice; Symbols, Terms, Units and Uncertainty Analysis for Radiometric Sensor Calibration

Report of Measurement /Uncertainty Analysis

• Next to capturing the details of the measurement process (and the activities that affect the data quality), this is probably the key portion of the range book.

• If the community does nothing else but determine an accepted, common format for an Uncertainty Analysis or Report of Measurement, it will be a huge step forward.

• Effort put forth to accomplish this will provide immediate payoff.

• Will provide a mechanism to make quantifiable comparisons between data sets or measurements from different programs, sensors, etc.

Report of Measurement Contents

• Identify target being calibrated

• Identify parameter(s) being determined [Major activity of this Workshop]

• Briefly describe and reference the measurement procedures

• Specify parameter values AND uncertainty

• Tabulate sources of error AND uncertainty for each source

• At least reference how each uncertainty value was obtained

• At least state how individual source uncertainties were combined to obtain overall uncertainty

• Clearly state the type of measurement the report of measurement is reporting on, and the frequency range/band the measurements are good for

Moving Forward as a Community

Who are the Players?

Government – DoD, NASA, NOAA, NIST, etc.

Industry – Government Contractors, Private Companies

Academia – Universities, Research Labs

Who becomes the Prime Mover?Government entityCommunity consensus


Where to talk about this effort?

SPIE/ASPRS/ISPRS – Lost in the crowd?

SPIE



• Military Sensing Symposia

- Caters to DoD

- Membership

- Classification issues



• Calcon

- Favors remote sensing community, but open to other users

- Covers the breadth of optical, IR and UV calibration issues

Join us at Utah State University September 13-16, 2004,

For the 13th Annual CalCon

Annual Conference on Characterization & Radiometric Calibration for Remote Sensing:

Addresses characterization, calibration, and radiometric issues within the IR, Visible and UV spectrums.

Session Topics Include:

Concepts and Applications of Measurement Uncertainty

Solar, Lunar and Stellar Radiometric Measurements

Pre-launch to On-orbit Calibration Transfer: Approaches and On-orbit Monitoring Techniques

Developing National Calibration/Certification Standards for EO/IR Systems

Recommendations

• Develop a consensus on where the “community” wants to go

• Find a forum for like-minded players to use as a focal point for developing the measurement improvement program

• Develop a common terminology

• Develop a taxonomy of measurement “categories”

• Develop Uncertainty Analysis / Report of Measurement template

• Conduct education process with customers and end users

Summary

Range certification is worth doing.

It IS a lot of work, and does require resources, but … experience has taught that benefits will outweigh the costs.

Community: needs to decide what it wants to do, to what extent and with whom.

First task is to start speaking the same language and partitioning the problem into manageable “chunks.”

Summary

Measurement Facility: Starting point is documenting (in detail) what you do and how you do it.

Range book is the best way to store your process documents and supporting documentation and make them available to your customers (internal and external).

Report of Measurement (RoM) with uncertainty analysis is a key document.

Tailor contents of Range book, RoM, to your needs while maintaining consistent structure of documentation across ranges.

References

ISO 17025 Home Page: http://www.fasor.com/iso25/

NIST Tech Note 1297 – Guidelines for Evaluating and Expressing the Uncertainty of NIST Measurement Results. Online version at: http://physics.nist.gov/Pubs/guidelines/contents.html or a PDF version is at http://physics.nist.gov/Documentation/tn1297.pdf

Uncertainty of Measurements Home Page: http://physics.nist.gov/cuu/Uncertainty/index.html

Signature Measurements Home Page: http://www.signaturemeasurements.org/

“Air Force Research Laboratory Advanced Compact Range RCS Uncertainty Analysis of a General Target,” B.M. Welsh, W.D. Muller, and B.M. Kent, APS Magazine, June 2003, pp. 195-200.

Antenna Measurement Techniques Association (AMTA) Annual Symposium Proceedings for past several years.

http://www.fasor.com/iso25/

http://physics.nist.gov/Pubs/guidelines/contents.html

http://physics.nist.gov/Documentation/tn1297.pdf

http://physics.nist.gov/cuu/Uncertainty/index.html

http://www.signaturemeasurements.org/

Concluding Thought

"If you can't describe what you

are doing as a process, you

don't know what you're doing."

- W. Edwards Deming

“If you can’t document your process,

no one else will believe you

know what you are doing.”

- R. Jost

Backup Slides

For more information

Source: Wyatt, “Electro-optical System Design For Information Processing, McGraw-Hill, 1991.

What drives the error / uncertainty?


“Easy” to characterize

Accuracy – “High”

Uncertainty – “Low”



“Hard” to characterize

Accuracy – “Low”

Uncertainty – “High”


• ANSI version of ISO 25, now superceded by ISO 17025.

• Starting point for the RCS community

ANSI/NCSL Z540-1-1994

Handbook for the Interpretation and Application of ANSI/NCSL Z540-1-1994

• Contains interpretive information for implementing Z540-1

• Useful, but community needs to write its own version tailored to its needs.

ANSI/NCSL Z540-2-1997

• This is the ANSI version of “The Guide to the Expression of Uncertainty in Measurement.

• Very good reference for developing an Uncertainty Analysis or Report of Measurement.

NIST Technical Note 1297, 1994 Edition

Meant to capture NIST’s policy for including uncertainty values in NIST measurements and reports.

Why Facility Tailoring is the Keyor

Consequences of Not Using Common Senseor

The Dangers of Extrapolation

In the space of one hundred and seventy-six years, the Lower Mississippi has shortened itself two hundred and forty-two miles. That is an average of a trifle over one mile and a third per year. Therefore, any calm person, who is not blind or idiotic, can see that in the Old Oolitic Silurian Period, just a million years ago next November, the Lower Mississippi River was upward of one million three hundred thousand miles long, and stuck out over the Gulf of Mexico like a fishing rod. And by the same token any person can see that seven hundred and forty-two years from now the Lower Mississippi will be only a mile and three-quarters long, and Cairo and New Orleans will have joined their streets together, and be plodding comfortably along under a single mayor and a mutual board of aldermen. There is something fascinating about science. One gets such wholesale returns of conjecture out of such a trifling investment of fact.

Mark Twain, Life on the Mississippi (1883).

For More InformationFor a copy of an example RCS range book, contact:

Chris ClarkMission Research Corporation3975 Research BoulevardDayton, OH 45430-2108(937) 253-8379Email: [email protected]

For on-line resource material:

Signature Measurements Home Page:





Documents

Applying a Decade of RCS Calibration Activity Experience to the IR/Visible/UV Measurement Community Dr. Randy J. Jost Assistant Professor of Electrical