Extraction of Shoe-Print Patterns from Impression Evidence usingConditional Random Fields

Veshnu Ramakrishnan and Sargur SrihariCenter of Excellence for Document Analysis and Recognition (CEDAR)

Department of Computer Science and EngineeringUniversity at Buffalo, The State University of New York, USA

Amherst, New York 14228{vr42,srihari}@cedar.buffalo.edu

Abstract

Impression evidence in the form of shoe-prints arecommonly found in crime scenes. A critical step inautomatic shoe-print identification is extraction of theshoe-print pattern. It involves isolating the shoe-printforeground (impressions made by the shoe) from the re-maining elements (background and noise). The problemis formulated as one of labeling the regions of a shoe-print image as foreground/background. It is formulatedas a machine learning task which is approached us-ing a probabilistic model , i.e., Conditional RandomFields (CRFs). Since the model exploits the inherentlong range dependencies that exist in the shoe-print itis more robust than other approaches, e.g., neural net-works and adaptive thresholding of grey-scale imagesinto binary. This was demonstrated using a data set of45 shoeprint image pairs representing latent and knownshoe-print images.

1. Introduction

Shoe-prints are among the most commonly found ev-idence at crime scenes. Approximately 30% of crimescenes have usable shoeprints[1]. In some jurisdictions,the majority of crimes are committed by repeat offend-ers and it is common for burglars to commit a numberof offenses in the same day. As it would be unusualfor an offender to discard footwear between crimes[3], timely identification and matching of shoe-prints al-lows different crime scenes to be linked. Since manualidentification is laborious there exists a real need for au-tomated methods.

Prints found at crime scenes, referred to here as la-tent prints (Figure 1(a)), can be used to narrow-down

(a) Example of noisy impres-sion evidence as a gray-scaleimage (latent print).

(b) Example of shoe-printpattern created as a chemicalprint (known print).

Figure 1. Example of latent and known prints.

the search space. This is done by elimination of thetype of shoe, by matching it against a set of known shoe-prints (captured impressions of many different types ofshoes on a chemical surface).

A critical step in automatic shoe-print identificationis shoe-print extraction – which is the task of extractingthe foreground (shoe-print) from the background (sur-face on which he print is made). This is because theprint region has to be identified in order to compare itwith the known print. The matching of these extractedshoe-prints to known prints largely depends on the qual-ity of the shoe-print extracted from the latent print.

An example of a known print is shown in Figure1(b). A known print is obtained in a controlled environ-ment by using a chemical foot stamp pad. The impres-sion thus formed are scanned. The shoe-print extractionproblem can be formulated as a image labeling problem.Different regions(defined later in Section 5) of a latentimage are labeled as foreground (shoeprint) or back-

ground. The matching of these extracted shoeprints tothe known prints largely depends on the quality of theextracted shoeprint from latent print. The labeling prob-lem is naturally formulated as a machine learning taskand in this paper we present an approach using Con-ditional Random Fields(CRFs). Similar approach wasused for an analogous problem in handwriting labeling[6].The model exploits the inherent long range depen-dencies that exist in the latentprint and hence is morerobust than approaches using neural networks and otherbinarization algorithms. Once the shoeprint has beenextracted, the matching process is a task that is com-mon to other forensic fields such as fingerprints wherea set of features are extracted and compared. Our focuson this paper is only the shoeprint extraction phase andit suffices to say that the matching problem can be ac-complished by using ideas from other forensic domains.

The rest of the paper is organized as follows. Sec-tion 2 describes the dataset and its acquisition. Section3 described the CRF model followed by parameter es-timation in section 4. Features for the CRF model aredescribed in Section 5 followed by experimental resultsand conclusion in Section 6 and section 7 respectively.

2. Dataset

Two types of shoe-print mages were acquired from45 different shoes: a latent print and a known (chemi-cal) print. These were obtained from the right shoes of45 different people as described next.

Latent prints. Bodziac [2] describes the process ofrecovery of latent prints. People are made to step ona powder and then on a carpet so that they leave theirshoeprint on the carpet. Then the picture of the print istaken with a forensic scale near to the print. The currentresolution of the image is calculated using the scale inthe image and then it is scaled to 100 dpi.

Known prints. Known prints are chemical prints ob-tained by a person stomping on a chemical pad and thenon a chemical paper, which would leave clear print on apaper. All Chemical prints are scanned into images at aresolution of 100 dpi.

3 Conditional Random Field Model de-scription

The probabilistic CRF model is given below.

P (y|x, θ) =eψ(y,x;θ)∑y′ eψ(y′,x;θ)

(1)

where yi ∈ {Shoeprint, Background} and x :Observed image and θ : CRF model parameters. It

is assumed that a Image is segmented into 3X3 non-overlapping patches. The patch size is chosen to besmall enough for high resolution and big enough to ex-tract enough features. Then

ψ(y, x; θ) =

m∑j=1

A(j, yj ,x; θs) +∑

(j,k)∈E

I(j, k, yj , yk,x; θt)

(2)

The first term in equation 2 is called the stateterm(sometimes called Association potential as men-tioned in [4]) and it associates the characteristics of thatpatch with its corresponding label. θs are called thestate parameters for the CRF model. Analogous to it,the second term, captures the neighbor/contextual de-pendencies by associating pair wise interaction of theneighboring labels and the observed data(sometimes re-ferred to as the interaction potential). θt are called thetransition parameters of the CRF model. E is a setof edges that identify the neighbors of a patch. Weuse 24 neighborhood model. θ comprises of the stateparameters,θs and the transition parameters,θt.The association potential can be modeled as

A(j, yj ,x; θs) =∑i

(fi · θij)

where fi is the ith state feature extracted for that patchand θli is the state parameter. The state features that areused are defined later in section 5 in table 1. The statefeatures, fl are transformed by the tanh function togive the feature vector h. The transformed state featurevector can be thought analogous to the output at thehidden layer of a neural network. The state parametersθs are a union of the two sets of parameters θs1 andθs2 .

The interaction potential I(·) is generally an innerproduct between the transition parameters θt and thetransition features ft. The interaction potential is de-fined as follows:

I(j, k, yj , yk,x; θt) =∑l

(f l(j, k, yj , yk,x) · θtl )

4 Parameter estimation

There are numerous ways to estimate the parame-ters of the CRF model [7]. In order to avoid the com-putation of the partition function the parameters arelearnt by maximizing the pseudo-likelihood of the im-ages, which is an approximation of the maximum likeli-hood value. The maximum pseudo-likelihood parame-ters are estimated using conjugate gradient descent with

line search. The pseudo-likelihood estimate of the pa-rameters, θ are given by equation 3

ˆθML ≈ arg maxθ

M∏i=1

P (yi|yNi,x, θ) (3)

where P (yi|yNi,x, θ) (Probability of the label yi for a

particular patch i given the labels of its neighbors, yNi),

is given below.

P (yi|yNi,x, θ) =

eψ(yi,x;θ)∑a e

ψ(yi=a,x;θ)(4)

where ψ(yi, x; θ) is defined in equation 2.Note that the equation 3 has an additional yNi inthe conditioning set. This makes the factorizationinto products feasible as the set of neighbors for thepatch from the minimal Markov blanket. It is alsoimportant to note that the resulting product only givesa pseudo-likelihood and not the true likelihood. Theestimation of parameters which maximize the truelikelihood may be very expensive and intractable forthe problem at hand.

From equation 3 and 4, the log pseudo-likelihood ofthe data is

L(θ) =M∑i=1

(ψ(yi = a, x; θ)− log

∑a

eψ(yi=a,x;θ)

)

Taking derivatives with respect to θ we get

∂L(θ)∂θ

=M∑i=1

∂ψ(yi, x; θ)∂θ

−∑a

P (yi = a|yNi, x, θ)·

∂ψ(yi = a, x; θ)∂θ

(5)

The derivatives with respect to the state and transitionparameters are described below. The derivative withrespect to parameters θs2 corresponding to the trans-formed features hi(j, yj ,x) is given by

∂L(θ)∂θiu

=M∑j=1

fi(yj = u)−

∑a

P (yj = a|yNi, x, θ)fi(a = u)

(6)

Here, fi(yj = u) = fi if the label of patch j is uotherwise fi(fj = u) = 0

Similarly, the derivative with respect to the transitionparameters, θt is given by

∂L(θ)∂θtlcd

=M∑j=1

∑j,k∈E

fl(yj = c, yk = d)−

∑a

∑j,k∈Ecd

P (yj = a|yNi , x, θ)fl(a = c, yk = d)

(7)

5 Features

Features of a shoe-print might vary according to thecrime scene. It could be powder on a carpet, mud on atable etc. So generalization of the texture of shoe-printsis difficult. So we resort to the user to provide the tex-ture samples of the foreground and background from theimage. The sample size is fixed to be 15X15 which isbig enough to extract information and small enough tocover the print region. There could be one or more sam-ples of foreground and background. The feature vectorof these samples are normalized image histograms. Thetwo state features are the cosine similarity between thepatch and the foreground sample feature vectors and thecosine similarity between the patch and the backgroundsample feature vectors. Given normalized image his-togram vectors of two patches the cosine similarity isgiven by

CS(P1, P2) =P1 ∗ P2

|P1||P2|(8)

The other two state features are entropy and standarddeviation (Table 1). Given the probability distributionof gray levels in the patch the entropy and standard de-viation is given by

E(P ) = −n∑i

p(xi) ∗ log(p(xi)) (9)

STD(P ) =

√√√√ n∑i

(xi − µ)2 (10)

The transition feature is the cosine similarity betweenthe current patch and the surrounding 24 patches.

Table 2. Segmentation Results.Segmentation Meth. Precision Recall F1

Otsu 40.97 89.64 56.24Neural Network 58.01 80.97 59.53CRF 52.12 90.43 66.12

Table 1. Description of the 4 state features usedState Feature DescriptionEntropy Entropy of the patchStandard Deviation Standard deviation of the patchForeground Cosine Similarity Cosine Similarity between the patch and the foreground sampleBackground Cosine Similarity Cosine Similarity between the patch and the background sample

(a) LatentImage

(b) Otsu (c) NeuralNetwork

(d) CRF Seg-mentation

Figure 2. Segmentation Results

6 Experiments and Results

The shoe-print is converted from rgb jpeg format tograyscale image, with a resolution of 100 dpi, beforeprocessing. For the purpose of baseline comparison,segmentation was done using Otsu [5] and neural net-work methods in addition to conditional random fields(Figure 2).

Otsu thresholding is based on a threshold which min-imizes weighted within class variance. The neural net-work had two layers with four input neurons, three mid-dle layer neurons and one sigmoidal output. Both theneural network and the conditional random field modelsuse the same feature set other than the transition feature.

Performance is measured in terms of precision, re-call and F-Measure. Precision is defined as the percent-age of the extracted pixels which are correctly labelledas shoe-print and Recall is the percentage of the shoe-print successfully extracted. F-Measure is the equallyweighted harmonic mean of precision and recall. Preci-sion, recall and F-measure are given in Table 2.

Performance of Otsu thresholding is poor if eitherthe contrast between the foreground and the backgroundis less or the background is inhomogeneous. The neu-ral network performs a little better than by exploitingthe texture samples that the user provided. CRF tendsto outperform both by exploiting the dependency be-tween the current patch and its neighborhood, i. e., ifa patch belongs to foreground but is ambigous, the evi-dence given by its neighborhood patches helps in decid-ing its polarity.

7 Summary and Conclusion

The paper has described a probabilistic approach tothe task of obtaining shoe-print patterns from latentprints for the purpose of matching against known prints.The CRF method of of extraction performed better thantwo other base-line methods.

8 Acknowledgements

This work was funded by Grant Number 2005-DD-BX-K012 awarded by the National Institute of Justice,Office of Justice Programs, U. S. Department of Justic.Points of view are those of the authors and do not nec-essarily represent the official position or policies of theU. S. Department of Justice.

References

[1] G. Alexandre. Computerised classification of theshoeprints of burglars soles. Forensic Science Interna-tional, 82:59–65, 1996.

[2] W. Bodziac. Footwear Impression Evidence: Detection,Recovery ond Examination. CRC Press, 2000.

[3] A. Bouridane, A. Alexander, M. Nibouche, andD. Crookes. Application of fractals to the detection andclassification of shoeprints. Proc. International Confer-ence on Image Processing, 1:474 – 477, 2000.

[4] S. Kumar and M. Hebert. Discriminative fields for mod-eling spatial dependencies in natural images. Advancesin Neural information processing systems(NIPS-2003),2003.

[5] N. Otsu. A threshold selection method from gray levelhistogram. IEEE Transaction on Systems, Man and Cy-bernetics, 9:62–66, 1979.

[6] S. Shetty, H. Srinivasan, M. Beal, and S. Srihari. Seg-mentation and labeling of documents using conditionalrandom fields. Document Recognition and Retrieval XIV,SPIE Vol 6500:65000U1–9, 2007.

[7] H. Wallach. Efficient training of conditional randomfields. Proc. 6th Annual CLUK Research Colloquium,2002.