© Devi Parikh 2008

Localization and Segmentation of

2D High Capacity Color Barcodes

Gavin Jancke

Microsoft Research, Redmond

Devi Parikh

Carnegie Mellon University

© Devi Parikh 2008

Motivation

UPC Barcode

QR Code Datamatrix

© Devi Parikh 2008

HCCB

Microsoft’s High Capacity Color Barcode

© Devi Parikh 2008

Application Uniquely identifying commercial audiovisual

works such as motion pictures, video games, broadcasts, digital video recordings and other media

© Devi Parikh 2008

GoalLocate and Segment the barcode from consumer

images

© Devi Parikh 2008

Overview

Design specifications of Microsoft’s HCCB

Approach

Localization

Segmentation

Progressive Strategy

Results

Conclusions

© Devi Parikh 2008

Microsoft’s HCCB

4 or 8 colors

Triangles

String of colors

palette

© Devi Parikh 2008

Microsoft’s HCCB

© Devi Parikh 2008

Microsoft’s HCCB

© Devi Parikh 2008

Microsoft’s HCCB

© Devi Parikh 2008

Microsoft’s HCCB

R rows

S symbols per rowS = (r+1)*R

Aspect ratio: r

© Devi Parikh 2008

Approach

Thresholding

Orientation prediction

Corner localization

Row localization

Symbol localization

Color assignments

Barcode localization

Barcode segmentation

point inside the barcode is known

© Devi Parikh 2008

Localization: Thresholding

Identify thick white band and row separators

Normalization

Adaptive

© Devi Parikh 2008

Localization: Orientation

orientation orientation

dist

ance

-90 900sum

mat

ion

© Devi Parikh 2008

Localization: Corners Rough estimates

whiteness mask non-texture mask combined mask

© Devi Parikh 2008

Localization: Corners Gradient based refinement

© Devi Parikh 2008

Localization: Corners Line based refinement

© Devi Parikh 2008

Segmentation: Rows

Summation

Flip?

© Devi Parikh 2008

Segmentation: Symbols

S E

Local search

Number of symbols per row

q(S,E) = q(samples|S,E)

© Devi Parikh 2008

Segmentation: Colors

Palette

© Devi Parikh 2008

Segmentation results given accurate localization Satisfactory

Corner localization Unsatisfactory

No one strategy works well on all images However (1) Errors of different strategies are

complementary (2) Results are verifiable with decoder in the

loop!

Observations

© Devi Parikh 2008

Progressive strategy Hence – progressive strategy!

Similar to ensemble of weak classifiers Or hypothesize-and-test

Multiple strategies: Rough + gradient + line, or rough + line, or

rough + gradient, or rough alone Different values of threshold during rough

corner detection Total 12

Order of strategies

© Devi Parikh 2008

Results

Dataset of 500 images

Performance metric: % barcodes successfully decoded

Decoder model: Barcode successfully decoded if 80% of symbols are correctly identified

© Devi Parikh 2008

Results

Allows for explicit trade-off between accuracy and computational time

© Devi Parikh 2008

Results

© Devi Parikh 2008

Results

© Devi Parikh 2008

Results

© Devi Parikh 2008

Results

© Devi Parikh 2008

Results

© Devi Parikh 2008

Results

© Devi Parikh 2008

Results

© Devi Parikh 2008

Results

© Devi Parikh 2008

Results

© Devi Parikh 2008

Results

© Devi Parikh 2008

Conclusions 2D High Capacity Color Barcode (HCCB)

Successful localization and segmentation of HCCB from consumer images

Varying densities, aspect ratios, lighting, color balance, image quality, etc.

Simple computer vision and image processing techniques

Progressive strategy

© Devi Parikh 2008

Acknowledgements

Microsoft Research Larry Zitnick Andy Wilson Zhengyou Zhang

Carnegie Mellon University Advisor: Tsuhan Chen

© Devi Parikh 2008

Thank you!