6
BSPAWP 060709 © Copyright BSPA Laboratory and Purdue University 2009. All rights reserved. Elliott & Kukula| Page 1 of 6 Abstract—Existing definitions for biometric testing and evaluation do not fully explain errors in a biometric system. This paper provides a definitional framework for the Human Biometric-Sensor Interaction (HBSI) model. This paper proposes six new definitions based around two classifications of presentations, erroneous and correct. The new terms are: defective interaction (DI), concealed interaction (CI), false interaction (FI), failure to detect (FTD), failure to extract (FTX), and successfully acquired samples (SAS). As with all definitions, the new terms require a modification to the general biometric model. I. INTRODUCTION BSERVATIONS made during biometric scenario and operational evaluations involving several thousand individuals across multiple modalities have led the authors to identify and categorize interactions between either the human and sensor, or the samples and biometric system itself, which have not been previously defined or thoroughly analyzed. As the subfield of Human-Biometric Sensor Interaction matures and peers within the biometric community and related fields of ergonomics and usability provide input, an etymology for the field can now be presented. In doing so, the authors re-examine current biometric testing and evaluation metrics and definitions and propose additional metrics that align with the HBSI model. The origin of the Human Biometric Sensor Interaction (HBSI) sub-field within biometrics resulted from participation in the U.S. national biometric standards committee INCITS M1, as well as the international committee ISO/IEC JTC1 SC37. Participation on these committees, especially during the development of the framework testing and evaluation standards [1-4], has enabled the authors to understand the breadth of the current standards and their related measurements, as well as isolate areas during biometric performance evaluations that have been traditionally out of scope. The areas outside the traditional metrics include the human-sensor interaction and how this interaction impacts the biometric samples in the system itself. HBSI evaluation concentrates on the minutiae of each interaction to fully understand how users interact with BSPAWP 060709 posted June 7, 2009 on www.bspalabs.org/publications . S. J. Elliott, Ph.D. is Director of the Biometric Standards, Performance, & Assurance (BSPA) Laboratory and Associate Professor in the Department of Industrial Technology at Purdue University, West Lafayette, IN 47907 USA (phone: 765-494-1088; fax: 765-496-2700; e-mail: elliott@ purdue.edu). E. P. Kukula, Ph.D. is a Visiting Assistant Professor & Senior Biometric Researcher in the BSPA Laboratory in the Department of Industrial Technology at Purdue University, West Lafayette, IN 47907 USA (e-mail: kukula@ purdue.edu). a biometric system and how the biometric system responds to this human-sensor interaction. II. THE HUMAN-BIOMETRIC SENSOR INTERACTION (HBSI) The HBSI model is conceptualized in Fig. 1 and derived from separate areas of research, namely ergonomics [5], usability [6], and biometrics [7]. Please see [8-13] for a complete discussion regarding the HBSI research area. The purpose of the model is to demonstrate how metrics from biometrics (sample quality and system performance), ergonomics (physical and cognitive), and usability (efficiency, effectiveness, and satisfaction) overlap and can be used to evaluate overall functionality and performance of a biometric system. Including metrics from different disciplines enables evaluators and system designers to better understand what affects biometric system performance. Core research questions the HBSI addresses are: How do users interact with biometric devices? What errors do users make? What are the most common errors or issues that users face? Why do users continually make these interaction errors and how do we prevent or avoid them from happening? What level of training and experience is necessary to successfully use biometric devices? Fig. 1. The HBSI conceptual model [8, 11, 12]. A Definitional Framework for the Human-Biometric Sensor Interaction Model Stephen J. Elliott, Ph.D. and Eric P. Kukula, Ph.D., Member, IEEE O

(2009) A Definitional Framework for the Human-Biometric Sensor Interaction Model

Embed Size (px)

DESCRIPTION

Existing definitions for biometric testing and evaluation do not fully explain errors in a biometric system. This paper provides a definitional framework for the Human Biometric-Sensor Interaction (HBSI) model. This paper proposes six new definitions based around two classifications of presentations, erroneous and correct. The new terms are: defective interaction (DI), concealed interaction (CI), false interaction (FI), failure to detect (FTD), failure to extract (FTX), and successfully acquired samples (SAS). As with all definitions, the new terms require a modification to the general biometric model.

Citation preview

Page 1: (2009) A Definitional Framework for the Human-Biometric Sensor Interaction Model

BSPAWP 060709

© Copyright BSPA Laboratory and Purdue University 2009. All rights reserved. Elliott & Kukula| Page 1 of 6

Abstract—Existing definitions for biometric testing and evaluation do not fully explain errors in a biometric system. This paper provides a definitional framework for the Human Biometric-Sensor Interaction (HBSI) model. This paper proposes six new definitions based around two classifications of presentations, erroneous and correct. The new terms are: defective interaction (DI), concealed interaction (CI), false interaction (FI), failure to detect (FTD), failure to extract (FTX), and successfully acquired samples (SAS). As with all definitions, the new terms require a modification to the general biometric model.

I. INTRODUCTION

BSERVATIONS made during biometric scenario and operational evaluations involving several thousand

individuals across multiple modalities have led the authors to identify and categorize interactions between either the human and sensor, or the samples and biometric system itself, which have not been previously defined or thoroughly analyzed. As the subfield of Human-Biometric Sensor Interaction matures and peers within the biometric community and related fields of ergonomics and usability provide input, an etymology for the field can now be presented. In doing so, the authors re-examine current biometric testing and evaluation metrics and definitions and propose additional metrics that align with the HBSI model.

The origin of the Human Biometric Sensor Interaction (HBSI) sub-field within biometrics resulted from participation in the U.S. national biometric standards committee INCITS M1, as well as the international committee ISO/IEC JTC1 SC37. Participation on these committees, especially during the development of the framework testing and evaluation standards [1-4], has enabled the authors to understand the breadth of the current standards and their related measurements, as well as isolate areas during biometric performance evaluations that have been traditionally out of scope. The areas outside the traditional metrics include the human-sensor interaction and how this interaction impacts the biometric samples in the system itself. HBSI evaluation concentrates on the minutiae of each interaction to fully understand how users interact with

BSPAWP 060709 posted June 7, 2009 on www.bspalabs.org/publications.

S. J. Elliott, Ph.D. is Director of the Biometric Standards, Performance, & Assurance (BSPA) Laboratory and Associate Professor in the Department of Industrial Technology at Purdue University, West Lafayette, IN 47907 USA (phone: 765-494-1088; fax: 765-496-2700; e-mail: elliott@ purdue.edu).

E. P. Kukula, Ph.D. is a Visiting Assistant Professor & Senior Biometric Researcher in the BSPA Laboratory in the Department of Industrial Technology at Purdue University, West Lafayette, IN 47907 USA (e-mail: kukula@ purdue.edu).

a biometric system and how the biometric system responds to this human-sensor interaction.

II. THE HUMAN-BIOMETRIC SENSOR INTERACTION (HBSI)

The HBSI model is conceptualized in Fig. 1 and derived from separate areas of research, namely ergonomics [5], usability [6], and biometrics [7]. Please see [8-13] for a complete discussion regarding the HBSI research area. The purpose of the model is to demonstrate how metrics from biometrics (sample quality and system performance), ergonomics (physical and cognitive), and usability (efficiency, effectiveness, and satisfaction) overlap and can be used to evaluate overall functionality and performance of a biometric system. Including metrics from different disciplines enables evaluators and system designers to better understand what affects biometric system performance. Core research questions the HBSI addresses are:

How do users interact with biometric devices? What errors do users make? What are the most common errors or issues that users

face? Why do users continually make these interaction

errors and how do we prevent or avoid them from happening?

What level of training and experience is necessary to successfully use biometric devices?

Fig. 1. The HBSI conceptual model [8, 11, 12].

A Definitional Framework for the Human-Biometric Sensor Interaction Model

Stephen J. Elliott, Ph.D. and Eric P. Kukula, Ph.D., Member, IEEE

O

Page 2: (2009) A Definitional Framework for the Human-Biometric Sensor Interaction Model

BSPAWP 060709

© Copyright BSPA Laboratory and Purdue University 2009. All rights reserved. Elliott & Kukula| Page 2 of 6

III. BIOMETRIC TESTING AND EVALUATION METRICS

Traditional biometric performance testing and evaluation metrics include: failure to enroll (FTE), failure to acquire (FTA), the false match rate (FMR), the false non-match rate (FNMR), the false accept rate (FAR), and the false reject rate (FRR). These metrics are used for the different types of performance evaluations, including technology, scenario, and operational evaluations. A discussion of these metrics and definitions leads to continued dissection of their meaning. Of main concern to the HBSI is FTA, which has been the biometric community’s de facto “usability” metric. FTA is defined as the “proportion of verification or identification attempts for which the system fails to capture or locate an image or signal of sufficient quality” [2]. Through the authors’ experience in testing and evaluation, interactions with a biometric system are traditionally viewed as a binary system; presentations either become successfully acquired samples or result in an acquisition failure (FTA). This binary result has impeded system designers the ability to understand how users are interacting with a biometric system and how the biometric system responds to this human-sensor interaction as all acquisition errors have been treated the same.

As the authors dissected this definition of FTA, it became apparent that factors and interactions existed, but did not fit within the existing definition or purpose of FTA as noted above. The HBSI framework for biometric interactions examines each recorded human-sensor interaction as an event. Each event is classified as either an erroneous presentation or correct presentation. Readers may question the difference between FTA (traditional performance and evaluation metrics), traditional usability metrics, and the metrics contained in the HBSI framework for biometric interactions. Recall that failure to acquire is a system reported metric for which the biometric sub-system “fails to capture or locate an image or signal of sufficient quality” [2]. The HBSI framework for biometric interactions is concerned with each interaction with the biometric device, regardless of result. Summarizing the difference, FTA is concerned with biometric performance, whereas metrics in the HBSI framework evaluate biometric systems from both the system and user perspective. Including both perspectives differentiates the HBSI framework from traditional usability evaluations, as HBSI is not only concerned with improving user efficiency, effectiveness, and satisfaction, but also is concerned with the impact the interactions have on the biometric system. More in line with traditional usability testing, [14] discusses an alternate usability taxonomy for biometrics that only considers the user perspective. Work is currently underway with these researchers to map the HBSI framework and usability taxonomy.

IV. THE HBSI FRAMEWORK METRICS

The purpose of the HBSI framework is to understand what common correct and incorrect movements or behaviors are occurring with biometric devices. Correct presentations are interactions with the sensor that can be classified as a: failure to detect (FTD), failure to extract (FTX), or successfully

acquired samples (SAS). An erroneous presentation is an interaction with the sensor that should be classified in three ways: defective interaction (DI), false interaction (FI), or concealed interaction (CI). The sum of these six classifications is equal to all the human-biometric sensor interactions with the system being evaluated. The following subsections discuss the six classifications included in the HBSI framework that is shown in Fig. 2.

A. Erroneous Presentation

The first subsection of measurements within the HBSI framework that is discussed involves erroneous presentations. These presentations are further classified into three metrics: defective interactions, concealed interactions, and false interactions. 1) Defective Interaction (DI)

A defective interaction (DI) occurs when a bad presentation is made to the biometric sensor and is not detected by the system. A DI results from an individual placing biometric features incorrectly on or to the sensor or in the case of biometric camera devices (face, iris, etc…) not looking in the appropriate area or direction. An example of a DI is a user looking away from the camera when approaching an access control gate controlled by iris recognition. The biometric system was “correct” in not detecting the presence of the subject because the individual did not present appropriate biometric characteristics to the sensor.

DI’s are important to measure because they provide quantifiable data for system design and throughput time, for example. Additionally, DI’s provide crucial quantifiable data to facilitate better training materials or policy guides for system implementation.

2) Concealed Interaction (CI) The next classification of erroneous presentations are called concealed interactions (CI). CI’s occur when an erroneous presentation is made to the sensor that is detected by the biometric system, but is not handled or classified correctly as an “error” by the biometric system. Therefore CIs are accepted as successfully acquired samples even though it was from an erroneous presentation. In other words, concealed interactions are those attempts where the user presents biometric characteristic(s) to the sensor, but used the wrong biometric characteristic, yet the sensor recorded the interaction as a successfully acquired sample. The CI rate is defined as the proportion of presentations that contain incorrect biometric characteristics acquired by the sensor that are classified as acceptable by the biometric system. In order to better explain where the anomaly occurred CI’s are segmented into two categories: user concealed interactions and system concealed interactions. User CI: User concealed interactions result from presentations where the user is at fault for the erroneous presentation that is recorded as a successfully acquired sample. An example of user CI with fingerprint recognition would be a user that is instructed to use one finger, but chooses, for whatever reason, to use a different one. Even

Page 3: (2009) A Definitional Framework for the Human-Biometric Sensor Interaction Model

BSPAWP 060709

© Copyright BSPA Laboratory and Purdue University 2009. All rights reserved. Elliott & Kukula| Page 3 of 6

Fig. 2. HBSI framework for biometric interactions.

though the interaction with the incorrect finger over the sensor was performed correctly, the system accepted the presentation, even though the incorrect finger was used. System CI: The second category of CI is from the system perspective. System CI’s are the result of an erroneous presentation that contains unrecognizable features which are subsequently recorded as a successfully acquired sample. An example of this in a fingerprint system would be where a user inadvertently interacts with the sensor with their fingertip or interphalangeal joint when the expected or desired characteristics were the volar pads of the fingers. However, the system records this inadvertent interaction as a successfully acquired sample. Another example of a system CI involving face or iris recognition would be where an image of a shadow, reflection, unrecognizable features, etc… are captured by the system and stored as successfully acquired samples.

The CI rate regardless of whether it is user or system concealed might have implications for measuring habituation. It should be noted that the commonly used term “habituation” might have to be revisited as we learn more about how people learn to use such devices. For example, with dynamic signature verification, individuals are probably habituated to the act of signing, but not the interaction with a particular signature pad. Another such example would be where users are used to interacting with a biometric sensor one way, but not another; for example in fingerprint recognition: swipe versus slap. The authors will endeavor to define habituation, training, and learning with metrics and experimental data in subsequent articles.

3) False Interaction (FI) The last classification that is possible within the erroneous presentation category is false interaction (FI). A FI occurs when a user presents their biometric features to the biometric system, which are detected by the system and is correctly classified by the system as erroneous due to a fault or errors that originated from an incorrect action, behavior, or movement executed by the user. The FI rate is defined as the proportion of interactions with the sensor that the biometric system detects and correctly classifies as erroneous.

FI’s are concerned with presentations in which a user does not interact properly with the sensor, and the biometric system correctly recognizes the attempt as a problematic or erroneous presentation. The user presents biometric characteristic(s) to a biometric sensor where the system both detects and acknowledges the error with a message and/or feedback to the user to retry.

The false interaction (FI) rate is proposed to evaluate the effectiveness of sensor design and training materials. Additionally, one would expect a difference in the FI rates of effectively trained or ineffective or non-trained individuals, which has been examined in [15].

B. Correct Presentation

The second category of measurements within the HBSI framework involves correct presentations. These are further divided into three subcategories: failure to detect (FTD), failure to extract (FTX), and successfully acquired samples (SAS).

Page 4: (2009) A Definitional Framework for the Human-Biometric Sensor Interaction Model

© Copyright BSPA Laboratory and Purdue University 2009. All rights reserved. Elliott & Kukula| Page 4 of 6

1) Failure to Detect (FTD) The first classification involving a correct presentation to the sensor but not detected by the biometric system is called a failure to detect (FTD). The definition of FTD is the proportion of presentations to the sensor that are observed by test personnel but are not detected by the biometric system. Failure to detect errors can be separated as system errors or external factor (environmental) errors. System FTD: A system FTD occurs when a user presents their biometric characteristic(s) properly (or appears correct to the data collection / analysis personnel) to the sensor, but the system does not detect that a presentation was made and the system remains in the same state as before the user interaction took place. The biometric sub-system does not detect the correct interaction of the user. For example, with iris recognition, a user stands in the appropriate position for the camera to collect iris characteristics, but the system fails to respond and/or does not detect the presence of the subject or iris with the sensor, i.e. the biometric system remained in the same state as before the interaction. External Factor FTD: An external factor FTD results from an extraneous factor impacting the ability of the biometric system to recognize a user’s correct presentation to the sensor. For example with face recognition, iris recognition, or hand geometry if a light source is in the field of view of the device, the biometric system will not be able to detect the presence of biometric characteristic(s). Due to factors beyond the user, and the biometric system, the presentation cannot be detected. The failure to detect rate provides data to system designers revealing the user interactions the system did not detect, regardless of the cause. The FTD rate exposes user interactions that have typically not been collected during performance evaluations. Understanding these issues will enable system designers to further improve devices and algorithms and reduce user frustration of those who believe they are interacting with the sensor correctly, yet the sensor does not detect their features as being present. 2) Failure to Extract (FTX) After a correct presentation is made to the sensor, and it is detected by the sensor, and acquisition occurs, the system attempts to create biometric features from the collected sample. In the general biometric model, this occurs in the signal processing module. A failure to extract (FTX) is concerned with samples from the data collection module that are unable to be processed completely. This may occur for a number of reasons, such as segmentation, feature extraction, or quality control. The authors debated on whether to segment these errors into individual components described in the general biometric model. However, as these components may, or may not be present in a biometric system, it would be hard to differentiate these errors into failure to segment, failure to feature extract, or failure to determine quality, and therefore a grouping metric of failure to extract was determined. The biometric system would provide this metric

in the log. Formally, the definition of failure to extract is the proportion of samples that are unable to process or extract biometric features. FTX is a system error. An example of this error would be when the biometric system acquires a fingerprint sample from the data collection module, but is unable to process the sample into biometric features, thus returns an error. 3) Successfully Acquired Samples (SAS) As the name implies, a successfully acquired sample (SAS) occurs if a correct presentation is detected by the system and if biometric features are able to be created from the sample. SAS result from presentations where biometric features are able to be processed from the captured sample, which are then passed to the biometric matching system.

V. REVISIONS TO THE BIOMETRIC MODEL

The HBSI framework that has been discussed until now was dependent upon one thing – the involvement of the human during data collection to thoroughly understand the human-biometric sensor interaction. To align the general biometric model (Fig. 3) with the proposed framework, the authors propose revising the signal processing module where template creation exists for two reasons.

Additionally, the HBSI framework can easily be applied to scenario and operational performance evaluations, but does not fully align with technology evaluations. Biometric technology evaluations typically use data that was collected at an earlier point in time to test biometric algorithms. The authors have noticed during some of these evaluations that the data used originally was not meant for a biometric system. Thus, the general biometric model should reflect the use of non-biometrically collected (NBC) data. The following sections will discuss these two revisions.

Fig. 3. General Biometric Model [2].

A. Template Creation and Enrollment

In the current general biometric model (Fig. 3), the creation of an enrollment template is undertaken in the signal processing module (Fig. 4A). In order to re-align the model with the system errors proposed in this paper, namely the failure to extract (FTX), the enrollment template creation is moved outside of the signal processing module (Fig. 4B). The movement of template creation outside signal processing is

Page 5: (2009) A Definitional Framework for the Human-Biometric Sensor Interaction Model

© Copyright BSPA Laboratory and Purdue University 2009. All rights reserved. Elliott & Kukula| Page 5 of 6

proposed to differentiate the Failure to Enroll (FTE) metric from the Failure to Extract (FTX) metric, which as mentioned earlier is a grouping metric for failures due to quality, feature extraction, and/or segmentation. In addition to differentiating FTE and FTX, this movement is also proposed to better handle non-biometrically captured (NBC) data.

Fig. 4. General biometric model (A) Current signal processing module and (B) Proposed signal processing module with Template Creation relocation.

B. Non-Biometrically Captured (NBC) Data

There are some cases where the signal processing system may accept samples offline, such as in technology testing, which may use data that was not collected by a biometric system. This data is vastly different from data collected with a biometric device and should be labeled accordingly. The authors propose classifying data of this kind non-biometrically captured data, or NBC. An example of this would be face recognition. In this example, photographs were taken by an operator of a camera that was not intended at the time for face recognition. The camera does not have any software that specifically looks at the image as a face. Therefore the operator acquires a photograph, which may not be able to be extracted by the face recognition system. Another example is inked ten-print cards for use with fingerprint recognition. At the time the data was collected, the intent was not to be scanned and used with a biometric system.

With respect to the biometric model, the hashed line in Fig. 5 stemming from the NBC data storage module indicates the process flow. The hashed line divides post signal processing once biometric features have been extracted for either enrollment or matching. Again, the proposition of the NBC data storage area by the authors is due to the fact that some performance metrics associated with online data collection might not be available. In the current general biometric model it is assumed that the process is done online, and this NBC route allows for offline processing, which is in line with technology testing.

Fig. 5. General biometric model with additions of template create and Non Biometrically Captured Data (NBC).

VI. CONCLUSION

The general biometric model and the various definitions wrapped into it have provided the basis for biometric testing and reporting standards. As the biometric community examines and discusses these definitions and continues to test and evaluate systems, additions to the general biometric model and testing and reporting standards have to be considered. In our previous work, we have explained the development of the concept of the human biometric sensor interaction, and this paper provides a series of terms and definitions for that model.

VII. FUTURE WORK

The authors have a series of future works planned in this area and have already applied the error framework to fingerprint recognition. Additional modalities will be evaluated with this framework, such as hand geometry, iris recognition and dynamic signature verification. Other terms that are in use within the biometric community will also be examined within the context of the HBSI model; habituation is one example of this.

ACKNOWLEDGMENT

The authors would like to thank members of INCITS M1.5 for their input during a presentation of this HBSI framework [12] in Morgantown, WV April 16, 2009. The authors would especially like to recognize Brad Wing, Patrick Grother, and Rick Lazarick for their feedback and contribution.

Page 6: (2009) A Definitional Framework for the Human-Biometric Sensor Interaction Model

© Copyright BSPA Laboratory and Purdue University 2009. All rights reserved. Elliott & Kukula| Page 6 of 6

REFERENCES [1] International Organization for Standardization, "ISO/IEC JTC1/SC37

Standing Document 2 - Text of WD Standing Document 2 Version 11 (SD2) Harmonized Biometric Vocabulary," ISO/IEC, Geneva SC37N3068, 2009.

[2] International Standards Organization, "ISO/IEC 19795-1: Information technology - Biometric performance testing and reporting - Part 1: Principles and framework," ISO/IEC, Geneva ISO/IEC 19795-1(E), April 1, 2006 2006.

[3] International Standards Organization, "ISO/IEC 19795-2: Information technology - Biometric performance testing and reporting - Part 2: Testing methodologies for technology and scenario evaluation," ISO/IEC, Geneva ISO/IEC 19795-2:2007(E), February 1, 2007.

[4] International Standards Organization, "ISO/IEC TR 19795-3: Information technology - Biometric Performance Testing and Reporting – Part 3: Modality-Specific Testing," ISO/IEC, Geneva December 15 2007.

[5] F. Tayyari and J. Smith, Occupational Ergonomics: Principles and Applications. Norwell: Kluwer Academic Publishers, 2003.

[6] International Organization for Standardization, "ISO 9241: Ergonomic requirements for office work with visual display terminals (VDTs) - Part 11: Guidance on usability," ISO, Geneva 1998.

[7] International Standards Organization, "Information technology - Biometric performance testing and reporting - Part 1: Principles and framework," ISO/IEC, Geneva ISO/IEC 19795-1(E), April 1, 2006 2006.

[8] S. Elliott, E. Kukula, and S. Modi, "Issues Involving the Human Biometric Sensor Interface," in Image Pattern Recognition: Synthesis and Analysis in Biometrics, vol. 67, Series in Machine Perception and Artificial Intelligence, S. Yanushkevich, P. Wang, M. Gavrilova, and S. Srihari, Eds. Singapore: World Scientific, 2007, pp. 339-363.

[9] S. Elliott, E. Kukula, and N. Sickler, "The challenges of environment and the human biometric device interaction on biometric system performance," presented at International Workshop on Biometric Technologies - Special forum on Modeling and Simulation in Biometric Technology, Calgary, Alberta, Canada, 2004.

[10] E. Kukula, "Understanding the Impact of the Human-Biometric Sensor Interaction and System Design on Biometric Image Quality," presented at NIST Biometric Quality Workshop II, Gaithersburg, MD, 2007.

[11] E. Kukula, "Design and Evaluation of the Human-Biometric Sensor Interaction Method," in Industrial Technology, vol. Ph.D. West Lafayette: Purdue University, 2008, pp. 510.

[12] E. Kukula, "Framework for Human, System, and Administrative Errors in Biometric Systems," INCITS, Washington, DC m1/09-0208, April 15 2009.

[13] E. Kukula, S. Elliott, and V. Duffy, "The Effects of Human Interaction on Biometric System Performance " presented at 12th International Conference on Human-Computer Interaction and 1st International Conference on Digital-Human Modeling, Beijing, China, 2007.

[14] R. Micheals, B. Stanton, M. Theofanos, and S. Orandi, "A Taxonomy of Definitions for Usability Studies in Biometrics," NIST, Gaithersburg NISTIR 7378, November 2006.

[15] E. P. Kukula and R. W. Proctor, "Human-Biometric Sensor Interaction: Impact of Training on Biometric System and User Performance," presented at 13th International Conference on Human-Computer Interaction, San Diego, 2009.