Pengfei Xu Hongbo Fu Hong Kong Univ. City Univ. of Hong Kong …kucg.korea.ac.kr/new/seminar/2013/ppt/ppt-2013-01-16.pdf · 2002-01-17 · Pengfei Xu Hongbo Fu Hong Kong Univ. City

YOUNGBIN KIM

2013/01/16

Korea University

Computer Graphics Lab.

Pengfei Xu Hongbo Fu Hong Kong Univ. City Univ. of Hong Kong

Oscar Kin-Chung Au Chiew-Lan TaiCity Univ. of Hong Kong Hong Kong Univ.

SIGGRAPH Asia 2012

Korea UniversityComputer Graphics Lab.

YOUNGBIN KIM | 2013/01/16 | # 2KUCG |

Abstract

• Lazy Selection

Scribble-based tool • Quick selection of one or more desired shape elements

• By roughly stroking through the elements ‗ Automatically refines the selection

‗ Reveals the user’s intention

Introduction



Introduction

• Manually selecting shape elements or objects

One of the most frequently performed tasks

Click-based selection • One of the most commonly used tools

‗ Mainly due to its simplicity

‗ However, since it requires every click to be placed exactly

• Even a slight improvement in its efficiency would save significant user time as a whole



Introduction

• New element selection tool based on a scribble-based user interface

Scribble-based UI to provide the advantages1. More effectively recognize the user’s intent

‗ Through shape matching between the scribble and its covered elements

2. Support the selection of multiple elements in a single action ‗ Allow the user to quickly select a desired group of elements

3. More tolerant to errors due to either imprecise input means or casual selection‗ Selection decision is made not based on only the input location

information alone



Introduction

• Automatically picking the desired element(s)

From the set of elements (partially/fully) covered by a user-specified scribble• Not an easy task

‗ No obvious decision strategy for revealing the user’s intent from the scribble



Introduction

• Present a two-phase approach

1. Establish a set of selection candidates, one of which the user might intend to select

2. Rank each selection candidate based on its location and shape with respect to the scribble

• Paper presents the main application of our tool

Smart selection of shape elements in images • Represented as 2D closed polygons

Related Work



Related Work

• Tools for Ungrouped Elements Selection

• Perceptual Grouping of Elements

• Scribble-based User Interfaces

• Sketch-based Retrieval

• Shape Writing on Stylus Keyboard



Related Work

• Tools for Ungrouped Elements Selection

Click-based selection + lasso selection• Inconvenient to select the elements of interest that are

interlaced with other elements

Various research prototypes have been proposed• Sloppy Selection [Lank and Saund 2005]

• DynaSpot [Chapuis et al. 2009]

• Harpoon Selection [Leitner and Haller 2011]‗ These techniques select a target solely based on whether the target

is touched or enclosed by a selection gesture with tolerance



Related Work

• Perceptual Grouping of Elements

Magic Wand in Adobe Illustrator• Selecting all similar objects in the entire scene

‗ Lack local or direct selection control

Several techniques have been proposed• Select a group from possibly overlapping groups

‗ Including repeated clicking cycles, pose matching and path tracing

Our work offers a more general and powerful selection tool• Rendering pose matching and path tracing as special cases

of ours



Related Work

• Scribble-based User Interfaces

Extensively used for interactively cutting out regions of interest• From bitmap images, videos or 3D models

Adopted for various interactive applications • Appearance editing, repeated element extraction, and

colorization of images or cartoons‗ Require the scribbles to be precisely within the regions of interest

• AppProp [An and Pellacini 2008], LazyBrush [Sykora et al. 2009]

‗ Support imprecise scribble placement

∙ Different nature of the problems



Related Work

• Sketch-based Retrieval

Our work is also related to content-based retrieval of images or 3D shapes • Using a hand-drawn sketch as a query

‗ Our scribbles are typically drawn across shape elements instead of along their boundaries, thus posing a different matching problem



Related Work

• Shape Writing on Stylus Keyboard

Our solution bears some resemblance to several methods • Literature of shorthand writing on stylus keyboard

‗ ZHAI, S., AND KRISTENSSON, P. –O. 2003. Shorthand writing on stylus keyboard. In CHI

‗ KRISTENSSON, P. –O., AND ZHAI, S. 2004. Shark2: a large vocabulary shorthand writing system for pen-based computers. In UIST

• Our problem imposes its own unique set of challenges‗ Arbitrary shape of elements

‗ No given set of selection candidates

Algorithm



Algorithm

• Develop a two-phase approach

1. Establishes a set of selection candidates by examining the underlying patterns (Section 3.1)

2. Picks the best selection candidate(s) by ranking individual candidates with respect to the scribble (Section 3.2)



Selection Candidates

• Attributes for Grouping

Various types of attributes can be used to group elements and thus reveal the underlying patterns• Magic Wand tool in Adobe Illustrator

‗ Selects objects with the same or similar attributes based on stroke weight, stroke/fill color, opacity, or blending mode

• Nan et al. [2011] ‗ Explored the integration of five principal Gestalt rules

∙ Similarity, proximity, continuity, closure, and regularity

Law of Proximity Law of Similarity Law of Closure





Focus on modeling the interactions between the three principles• Kubovy and van den Berg [2008], Luna and Montoro [2011]

• Proximity

• Similarity in shape

• Common region in visual perception‗ Common region principle applies to elements within another

element

∙ i.e., enclosed by the boundary of another element





Our experimentation on additive models

‗ First term : PCA-based similarity measure

∙ 𝜆 𝑖𝑘

: k-th eigenvalue obtained by applying PCA to element i

∙ 𝑐𝑖 : Shape compactness of element i, defined as the ratio of its squared perimeter to its area

‗ Second term : Strengths of proximity

∙ 𝑔𝑖𝑗 : Spatial distance between the two centers of elements

∙ 𝛾: Weight to control the contribution of the proximity principle(=0.3)

‗ Third term : Strengths of common region

∙ 𝑏𝑖𝑗 : 0.7 if elements are inside a common region, 1 otherwise




• Grouping

By a single scribble, a user may target • A single element

• A group of elements that have one or several characteristics in common

• Multiple groups of elements

• All the elements covered by the scribble‗ This naturally demands a hierarchical representation of the scribble-

covered elements

To this end, we take a greedy bottom-up grouping approach




• Grouping

Find the minimum grouping error 𝑓𝑖𝑗• Iteratively group the pairs of elements

‗ Using a greedy bottom-up approach

‗ Until all the input elements form a single group



Ranking

• Based on the following key observations or rules

Rule 1• For elements of small to moderate sizes, the user prefers

to draw a scribble which is close to the elements and whose shape is similar to the overall layout of the elements



Ranking

Rule 2• For elements of rather large size, the user may specify the

scribble more freely as long as the scribble is within or strokes through the elements



Ranking

Rule 3• Since the user’s effort should be respected as much as

possible, the more the scribble covers a candidate, the more likely the candidate is desired



Ranking

Rule 4• When the user draws a scribble very fast, the location of

the scribble might be less accurate. In contrast, the shape of the scribble is largely insensitive to the scribbling speed

Rule 5• As the user usually pays more attention to the beginning

and ending points of the scribble than the rest of the scribble, the elements close to the two endpoints of the scribble are more likely to be picked



Ranking

• Ranking algorithm

Starts with uniform point sampling• Regions of the elements in a candidate 𝑐 ∈ 𝐶

‗ 𝐶 denote a set of selection candidates

‗ 𝑃𝑐 denote the resulting sets of points for the candidate

Equidistant point sampling of the scribble curve‗ 𝑃𝑠 denote the resulting sets of points for the scribble



Ranking

• Ranking algorithm Next, compute the closest distance from every point in

𝑃𝑠 to 𝑃𝑐• Denoted as 𝑇𝑠→𝑐 = {𝑖𝑛𝑓𝑝∈𝑃𝑐

𝑑(𝑝, 𝑞)|𝑞 ∈ 𝑃𝑠}

‗ 𝑑(𝑝, 𝑞) is the Euclidean distance between a pair of points

Also measure the closest distance from every point in 𝑃𝑐to 𝑃𝑠• Denotes as 𝑇𝑐→𝑠 = {𝑖𝑛𝑓𝑝∈𝑃𝑠

𝑑(𝑝, 𝑞)|𝑞 ∈ 𝑃𝑐}



Ranking

• Ranking algorithm

Let 𝜇∗ and 𝜎∗ be the mean and the standard deviation of 𝑇∗, respectively• 𝜇𝑐→𝑠 and 𝜇𝑠→𝑐 measure the degree of matching between

the scribble and a candidate in terms of location

• 𝜎𝑐→𝑠 and 𝜎𝑠→𝑐 measure the degree of shape matching

Matching error function 𝑟(𝑐)• 𝑟 𝑐 = 𝛽 𝜇𝑠→𝑐 + 𝜇𝑐→𝑠 + (1 − 𝛽)( 𝜎𝑠→𝑐+ 𝜎𝑠→𝑐)

‗ 𝛽 controls the relative contribution of the location and shape information

‗ 𝜇∗ and 𝜎∗ are the normalized versions of 𝜇∗ and 𝜎∗



Ranking

• Enforcing Rule 4

Dynamic weighting scheme

• 𝛽 = min{1,𝑒1−𝑣

2}

‗ 𝑣 is the estimated scribbling speed, normalized by the pre-computed average scribbling speed

• Enforcing Rule 5

Simply decrease the matching errors• By a scaling factor less than 1

‗ For the candidates that cover the beginning and ending points of the scribble



Ranking

• Click Selection

Matching function is reduced• 𝑟 𝑐 = 𝜇𝑠→𝑐 + 𝜇𝑐→𝑠



Feedback and Output Interface

• Adopt a linear menu

For the organization of the top-ranked candidates

Extension to 3D Elements Selection



Extension to 3D Elements Selection

• Easy selection of 3D elements Preprocessing stage

• Manually pre-segment 3D models into segments as shape elements

Runtime• Covered by a user-drawn 2D scribble are grouped on the

fly again ‗ Via a bottom-up clustering process

‗ Uses a PCA-based similarity measure alone as the grouping strength measure

• To match the scribble and a candidate1. Pre-sample 3D points on the surface of each candidate element

2. Project them to 2D for matching with the sampled points from the scribble curve

Results and Discussion




• Tested algorithm on a wide range of images

With various selection targets using the mouse or touch input• Click selection, scribble selection, and paint selection




• Impact of the brush size on the selection results

Our tool permits scribbling with different brush sizes • Size of the brush can be increased/decreased for

rougher/finer selections




• Largely insensitive to the exact scribble locations

• Geometrically similar but semantically different elements nearby may still confuse our algorithm




• Compare our tool to the LazyBrush tool and a straightforward method

LazyBrush tool• Sykora et al., Eurographics 2009

• State-of-the-art technique ‗ pixel-level cartoon coloring or segmentation

∙ Preference of short boundaries

∙ Unawareness of underlying patterns

Rule-of-majority method• Assigns individual elements to the scribbles that cover

them the most




• Limitations

In some cases, might not produce satisfactory results• Desired candidate might not be included in the list

‗ At most 10 candidates in our current implementation

∙ Pick the most satisfactory one from the list and refine it with more selection

∙ Deselection scribbles or simply undo and reissue a new scribble

‗ Desired elements might fail to form a candidate in the grouping process

∙ Interesting to explore alternative grouping strategies like [Nan et al. 2011]




• Limitations

In some cases, might not produce satisfactory results• Less effective for rough hand-drawn images

‗ Instead, such problems are the focus of pixel-level segmentation methods like LazyBrush



User Study

• Comparison with the standard click and lasso selection tools

Apparatus• IBM Thinkpad X220t with 12.5-inch Multitouch HD

‗ Size of the painting brush : roughly 1.5cm

Participants• 17 university students

‗ Had extensive experience in shape element selection

‗ Had various experience with multitouch interaction

Task• 17 (participants) x 2 (techniques) x 2 (window sizes) x 39 (target

selections) = 2652 trials

Performance Measures• Completion time of individual trials

• Number of operations used for selection, deselection, undo, and image translation



User Study

Results



User Study

Results• Superimposed visualization of the scribbles by all the

participants

Conclusion and Future Work




• Presented for the first time a scribble tool

For smart selection of shape elements

Our technique has the following advantages • Applicable to the selection of single or multiple elements

of various sizes

• Supporting click, scribble and painting selection unified in a single tool

• Tolerant to imprecise input means like touch input

• Working consistently well for the selection of 2D and 3D elements

• Selection with our tool is significantly faster than with the standard click and lasso selection tools




• Devised a bottom-up grouping approach

With the simple encodings of three grouping attributes• Similarity, proximity and common background

This approach can be improved by incorporating the scribble information• Coverage ratio of elements by the scribble, relative

location of elements to the scribble

Different grouping algorithms can be devised• Supports a larger set of grouping attributes will be one of

our future works

Documents

Pengfei Xu Hongbo Fu Hong Kong Univ. City Univ. of Hong Kong …kucg.korea.ac.kr/new/seminar/2013/ppt/ppt-2013-01-16.pdf · 2002-01-17 · Pengfei Xu Hongbo Fu Hong Kong Univ. City