Adapting ISO 20462 Softcopy Quality Ruler Method for on-line Image Quality Studies

Adapting ISO 20462 Softcopy Quality Ruler Method for On-line

Image Quality Studies

Peter D. Burns, Jonathan B. Phillips1 and Don Williams2

Burns Digital Imaging, 1NVIDIA Corp. and 2Image Science Associates

IS&T and SPIE Electronic Imaging Symposium, Feb. 2013

Reference: P. D. Burns, J. B. Phillips and D Williams, Adapting ISO 20462 Softcopy Quality Ruler Method for On-line Image Quality Studies, Proc. SPIE Vol. 8653, 86530E-1, 2013

IS&T and SPIE Electronic Imaging 2013 2

Outline

1. ISO 20462 Image quality ruler method

2. Extended to softcopy image quality

3. Adaptation for Internet-based image quality evaluations• Viewing method• Image selection and preparation• Design for judgments

4. Verification study• Presentation of softcopy ruler (images)• Previous texture-loss experiment

5. Conclusions and recommendations• Comparison with ISO softcopy ruler results


Introduction

ISO 20462 Image quality ruler method• Set of calibrated reference hardcopy images• Primary Standard Quality Scale, SQS1, in overall JND (just noticeable

difference) units• Ruler is a set of prints based on a single scene, varying from ideal

to lower image quality

Extended to softcopy image quality• Digital Reference stimuli, updated ISO 20462• Calibrated display and viewing conditions, verified with four labs.• Secondary Standard Quality Scale, SQS2

Internet-based image quality evaluations• Our objective: Develop and test method that uses commercial

online presentation • efficient methods for gathering evaluation data (e.g. crowdsourcing)

that can be related to the ISO method• when commonly used (uncontrolled) viewing conditions, hardware,

software and lighting are needed


ISO softcopy ruler method

Ruler image is adjusted until it matches test image on right• GUI software used slider

Updated by slider

movement

Previous work: ISO softcopy ruler application for texture blur

Measurement of image quality degradation due to loss of texture, caused by noise-cleaning image processing

• Ruler varied in image sharpness• test scenes varied in texture loss

Memorial Art Gallery, Flowers, Girl and Mountain.Scenes used in their original form (1088 x 816, or 1253 x 834 pixels),


Previous work: ISO softcopy ruler application for texture blur

Noise-cleaned test images compared to ruler images varying in sharpness to obtain SQS2 JNDs for each test image

Ruler, Standard Quality Scale 30 Ruler, Standard Quality Scale 15

noise-cleaned sample treatment texture loss level 6


Adaptation of ISO Method for On-line Evaluations

1. Presentation• Internet-browser based• Commercial online presentation, Survey Monkey

– Likely maximum display size and resolution• Presentation of images without cropping or scaling

– Image file size limitations• Static presentation of ruler, rather than dynamic slider-

controlled– Button selection with anchors (two sets)

2. Image selection and preparation• Preserve image microstructure

– Crop selected ruler and test images• Use previous texture-loss study and images

– Ruler based on range of image sharpness– Texture loss due to noise-cleaning image processing

Presentation and Image Selection, Preparation



1. Presentation• Button selection with two anchor reference images• Each test image was rated twice, compared with two sets of

references [0 (low), 15] and [15, 30 (high)] SQS2 values


2. Image selection and preparation• Selection of scenes based on JND sensitivity found in previous work• Texture loss due to noise-cleaning image processing


(1088 x 816, or 1253 x 834 pixels),



2. Image selection and preparation• Commercial service: Survey Monkey• URL vs. upload• Display field aimed at display of 1600 x 900 pixel

Original scenes in previous work (1088 x 816, or 1253 x 834 pixels)

Cropped scenes in current experiment (525 x 330 pixels per scene)


Study: Image Quality Due to Noise-Cleaning

1. Same scenes and treatments that were used in previous verification study

2. Initial subjective results with 21 overall quality buttons

• Observers with and without involvement in the imaging industry. (N=21)

3. Second study with 9 overall quality buttons for the observers to select

• Similar observer population, though not the same observers (N=11)

• Experienced observers in image quality evaluation (N=20)

Rate the quality of the test image based on the position between the left and right reference images


ISO Softcopy Results for the Four Scenes• From ISO method study rsults• SQS2 JNDs for the 4 scenes and noise-cleaning treatments levels for

calibrated reference points from previous study. • Error bars are standard errors, N=17


Results 1: Adapted Online Method

Inferred SQS2 JNDs for the 4 scenes and treatments levels for 21 buttons of overall quality

• 21 observers



Inferred SQS2 JNDs for the 4 scenes and treatments levels for 9

buttons of overall quality • 11 observers



Comparison of our inferred SQS2 JND results with those of the reference experiment (previous work) with calibrated SQS2 JNDs. Averaged over observers and scenes.


Summary

1. Softcopy ruler method has been adapted for unconstrained on-line image quality studies• Static ruler-anchor display vs. slider-ruler selection• Button selection of image quality rating• Cropped scenes

2. Results show similar scene-differences in IQ sensitivity3. 9- and 21-button results were similar, implying a quantizing of ruler

scale response, as intended4. Pooled results data for all four scenes were consistent with results

from the original softcopy ruler study5. Uncontrolled, on-line results showed more sensitivity to the overall

image quality degradation (rather than less, as anticipated)• Possibly due to cropping most salient image areas, or

uncontrolled viewing distance6. Conclusion: Internet-based image quality evaluations can be related

to ISO 20462 standard if common reference images are used.

SUBJECTIVE IMAGE QUALITY EVALUATION(Not presented at EI 2013)

Image quality ruler

ISO 20462 Part 3 – The Quality Ruler Method

“Photography—Psychophysical experimental methods to estimate image quality—Part 3: Quality ruler method”

Standardization of anchored pair method based on JND units

Substitute length units with an image set having a series of sharpness levels

Quality ruler approach allows for repeatable and definable results

Compare test image to images on ruler and pick best match on rulerHigh quality (sharp) Low quality (blurry)

ISO 20462 Part 3 – The Quality Ruler Method

32 24 16 8 0

Standard Quality Scale (SQS)

Standard Reference Stimuli (SRS) or Digital Reference Stimuli (DRS)

Not Worth Keeping

PoorFairGoodVery Good

Excellent

1 JND = 75:25 split in paired comparison; 2 JNDs = advertisable differences; 6 JNDs = interval of image quality categories

Softcopy Quality Ruler

Softcopy quality ruler method based on ISO 20462 Part 3

• 30" Apple Cinema Display monitor• Controlled viewing distance with

chin/head rest*– 25", 34", or 43"

• Simultaneous viewing of ruler and test images

• Quality ruler reference stimuli span 30 just noticeable difference values in sharpness

* e.g., University of Houston College of Optometry (UHCO) HeadSpot

Softcopy Ruler Method Development

Raw image

Linear sRGBimage

Camera MTFResponse

Demosaic

WB + CCM

Monitor displaysSingle bright pixel

PSF image captured and analyzed

Monitor MTFResponse

System MTF = Camera MTF * Monitor MTF * Downsampling MTF ( 1MP)

21 Quality Rulers

• 21 scene choices• 30 ruler images per scene• SQS from 2 to 31 in increments of approximately 1

JND• Matlab code for running quality ruler

Soon to be officially located at:http://www.imaing.org/ist/resources/standards/standards.cfm

Secondary location:

http://www.aptina.com/ImArch/

http://www.imaing.org/ist/resources/standards/standards.cfm

http://www.imaing.org/ist/resources/standards/standards.cfm




Softcopy Quality Ruler Screen

In situ

Screen Capture

Softcopy Quality Ruler Method Workflow

Perform psychovisual

studies

Analyze data and report

results

Obtain test images

Specifications on obtaining test images at any time of the year and around

the globe in a timely

manner

Specifications on subject selection,

vision screening, number of

subjects, etc.

Obtain ruler

images

Set up viewing

environment

Obtain software

tools

Fixed sets of digital anchor images, i.e.- 21 scenes- 30 images per scene

Specifications on viewing

distance and ambient lighting.

Create ICC profile for the

monitor.

Software applications to display

images and collect viewer

responses.

Tools for analyzing the results and report template

One-time setupOne-time setup Running the tests Running the tests

Technology

Adapting ISO 20462 Softcopy Quality Ruler Method for on-line Image Quality Studies