A Portable Structured Light System for the Rectification of Printed Text

Andrew ZieglerUniversity of California, San Diego

aziegler@cs.ucsd.edu

Abstract

The goal of this project is to develop a portable struc-tured light scanner which can capture the 3D structure ofarbitrarily warped printed text and then rectify it. The re-sulting rectified text can then be sent to a standard OCRengine for recognition and layout analysis. The rectifiedimage of the text can also be used as a reference image fornon-rigid surface detection. Pairing non-rigid surface de-tection with the OCR results provides an infrastructure foraugmented reality applications. The system is wearable andconsists of a pico projector, a webcam, and a portable com-puter. This project is part of an on going effort to implementan assistive reading device for the visually impaired.

1. IntroductionCurrently there exist several portable devices to assist vi-

sually impaired individuals in reading printed text; notablythe Intel Reader and the KNFB Reader. These devices workwell when the text being captured is printed on a surface thatis approximately planar. However, neither system correctsfor the distortion of curved text, let alone arbitrarily warpedtext, which is prevalent (i.e. books, magazines, newspapers,prescription drug bottles, etc.). Since OCR engines assumethat text is planar, these devices cannot handle distorted text.

The proposed device is designed to correct for the dis-tortion of text printed on paper. Using structured light tech-niques and finite element models, distorted text is virtuallyflattened. The resulting image is post-processed to furtherreduce distortion. Yielding a rectified image of the originaltext.

The rectified image serves two purposes. First, it recov-ers an approximation of the text in its planar state. Usingthe rectified image as input to an OCR engine results inbetter recognition rates. Second, there are real-time non-rigid surface detection methods that assume an initial pla-nar state [4]. Using OCR and non-rigid surface detectiontogether has the potential to provide a base for a variety ofaugmented reality applications. The focus of this device isto provide the visually impaired with easier and more natu-

ral access to printed text.

2. Rectification2.1. Finite Element Modeling

Finite element modeling was chosen as the mechanismfor rectification out of many available methods. It has sev-eral characteristics which make it suited to this application.It allows for a simple, yet arbitrarily accurate representationof paper. Using a discrete model allows for a compromisebetween speed and fidelity to the data. The particular modelchosen is robust to the noise that will inevitably be presentin the range data.

2.2. Applicable Surfaces

Paper can be naturally modeled as an applicable surface,which are the surfaces that are isometric with the plane.The analytical definition of an applicable surface is cum-bersome, which spurred research using applicable meshesto model paper [5].

Here the modeling method described in [5] is summa-rized. Using structured light techniques the 3D structure ofan arbitrarily warped reading material can be recovered asa point cloud. An initially planar mesh is fit to this pointcloud. The mesh nodes are moved along the plane normaltowards the nearest data point. To become more robust tonoise the mesh nodes can be limited from moving to faraway. During the warping process the distance between themesh nodes changes. Thus the mesh is not generally appli-cable to the plane after warping and it must be relaxed backto that state. The mesh can naturally be modeled as a sys-tem of springs with initial lengths corresponding to those oforiginal planar mesh. Minimizing the elastic energy in thesystem restores the mesh to an applicable state.

2.3. Texture Mapping

The relaxed mesh has the same edge lengths as the orig-inal planar mesh, but it now approximates the data. Thereis a one to one mapping between these two meshes. This isthe isometry with the plane obtained by using an applicablemodel. To rectify the text the final mesh is backprojected

Figure 1. Top:Text taped onto a book to follow its curvature.Bottom: The rectified image using an applicable surface model.

to the image plane providing a texture map [5]. Applyingthe texture map to the planar mesh warps it into a rectifiedstate. Figure 1 shows the result of this process applied totext distorted by the curvature of a book.

2.4. Cylindrical Objects

There are many times in daily life that one encountersprinted text on cylindrical objects. The method described in[5] works well for many situations. However, in the extremecase of a cylindrical object the method does not perform aswell as it could.

This is a result of the initialization process and the re-laxation of the mesh. The method starts with a planar meshand only allows the mesh nodes to move along the planenormal. For most situations this is appropriate since paper,even when warped, is normally near planar. However, whenan object is cylindrical this method performs poorly both interms of convergence and accuracy.

The problem with using a planar mesh is the fact thatall the mesh triangles have the same normal. The mesh tri-angles near the edges of a cylindrical object will have nor-

mal vectors nearly perpendicular to their ideal value. Af-ter warping the nodes along the normal, the nodes near theedges will be far apart. This results in an initial model thathas more elastic energy near the edges.

When relaxing the mesh two things occur that result ina poor model. First, the surface area of the initial mesh isnot large enough to cover the visible portion of the cylinder.This is because the bounding box of the object’s projectionon the plane is used to determine the extent of the mesh.During relaxation there is dramatic shrinking causing themodel to lose information. Second, the nodes on the edgesare only connected on one side to the system of springs,making them prone to curl and warp. The additional elas-tic energy associated with these nodes only exacerbates thisdistortion of the model. Which affects the final image, as isapparent in figure 2.b near the periphery.

This poor performance suggests fitting a cylindricalmesh to the data instead of a planar mesh. The same pro-cess as before can be used to rectify the text. In the commoncase cylindrical objects are smooth and the initial mesh suf-ficiently approximates the data. Thus the warping and relax-ation steps can be skipped. This gives a tremendous speedup over the plane fitting method, since relaxation is slowto converge in this case. It also results in a more accuratemodel and therefore better rectification results. In figure 2.cthe result of the cylindrical fit is shown with considerablymore of the text rectified and less distortion near the edges.

There are some occasions when an object is only roughlycylindrical and is possibly rough or bumpy. The initial fitwill not be accurate and it is necessary to warp the mesh asbefore. The nodes should move towards the data along thenormal of the cylinder at the point. This can be made evensimpler by using polar coordinates and simply changing theradius of the mesh node. This warped mesh can then berelaxed in the same manner as before.

2.5. Post Processing

The rectified images from the previous step need to befurther processed to achieve accurate OCR results. OCRengines make an additional assumption that the text base-lines are straight. After rectification there is some resid-ual distortion to the text baselines which must be corrected.Software from [1] is used to further rectify the text. Thesoftware corrects the text baseline distortion and filters theimage. Observe the results in figure 2.e-f, the text baselinesare approximately straight. To accomplish this backgroundnormalization and binarization are first performed on theimage. Afterwards the lines going through the center ofthe text are found using linear least squares. The contoursof the recovered lines is then used to rectify the text lines.This method used without the initial rectification can fail asis the case with figure 2.d. However, when this software iscoupled with the applicable surface model, OCR recogni-

tion rates increase significantly.For the rectified image of the book in figure 1 the soft-

ware from [1] was not able to find the baselines after bina-rization. This could be caused in part by the slightly rotatedorientation, but the poor resolution is more likely the cause.

2.6. Preliminary Results

The images from figure 2.d-f were fed into the TesseractOCR engine, the results are shown in figure 3. Both theplane and cylinder methods produce improved OCR results.The cylinder model gives better results, but that is expectedfor this case.

It is apparent that the text that is recognized is highlyvisible in the processed images. More precisely, text that ismore faint and feathered is not correctly recognized. Twospecific observations are that bold-face fonts and text that isnearer to the center are recognized more consistently. Dur-ing background normalization and binarization some pixelinformation near the edges of the text is lost. The bold-facefonts retain enough of their pixels after the process that theyare more readily recognized. This suggests that the resolu-tion of the camera is one limiting factor to OCR recognitionrates. It is also not surprising that the text near the centeris recognized more often, since it was nearly rectified in theoriginal image. This is equivalent to saying that triangleswith normals facing the image plane are less distorted bybackprojection.

These observations suggest that recognition rates couldbe improved by using a higher resolution camera and mul-tiple views. The higher resolution camera will provide ahigher density of pixels and thus more text should be rec-ognizable after post-processing. Using multiple views willallow for the selection of the best texture map for a partic-ular triangle, resulting in more successful rectification. Inaddition, using multiple views allows for the constructionof a complete model of the reading material given a limitedfield of view.

3. Non-Rigid Surface Detection

3.1. Problem Statement

An initial view of a non-rigid surface and a referencemesh with equal edges is given. As well as a test imageand its correspondences to the reference image. The goalis to find a transformation that warps the initial mesh tofit the possibly deformed surface in the test image or de-termine that the surface is not present. When finding sucha transformation it is important to fit the correspondences,but smoothness is important to enforce to avoid over-fitting.This can be modeled as a energy minimization problem asin [4]. An example of non-rigid surface detection is shownin figure 4.

3.2. User Interaction

In the proposed system non-rigid surface detection willprovide the mechanism for interaction. The rectified imageof the text can be used as the reference view. A regular meshcan be generated to go along with this as the initial input tothe detection algorithm. At this point it is important to notethat OCR engines return not only recognized text, but alsopositional information and document hierarchy. The refer-ence mesh can be augmented with this meta-data. In thisway a specific mesh triangle will have information aboutthe text contained within it.

The non-rigid surface detection method proposed in [4]is robust to occlusions. An example of this can be seen infigure 5. In the presence of an occlusion the occluded fea-ture points do not contribute to the calculation of the trans-form. If a triangle is completely occluded than it is solelyinfluenced by the smoothness constraint. This causes theoccluded triangles to stay near their neighbors and remainsmooth.

To add interactivity to this system, pointing gesturesseem like a natural starting place. If a pointing gesture isdetected, non-rigid surface detection can be used to createa local coordinate system. The location of the pointing ges-ture in this coordinate system give access to the meta-dataregistered to the mesh. This provides a mechanism to createa more natural interaction with text. One possible applica-tion is pointing to text and having it read aloud by a speechsynthesizer. Which is the first intended purpose for this sys-tem.

4. Software Resources

Code from several sources was used to develop this sys-tem. Currently the structured light system being used wasprovided with [3]. It implements a system based on binarycoded patterns which is not conducive to an interactive ap-plication. This will eventually need to be replaced by a real-time technique possibly one of the methods discussed in [6].I used Matlab, C++, OpenCV, OpenGL, Tesseract for OCR,the camera calibration toolbox from Caltech, and an imageprocessing library called Leptonica [1]. I also used somefiles from the Matlab exchange to perform basic steps suchas finding the bounding box [2].

5. ConclusionWith more work this system has the potential to give vi-

sually impaired people easy access to printed text. In thefuture a device like this could help people read prescrip-tion drug bottles, nutrition facts on canned food, menus atrestaurants, books, and possibly even wrinkly newspapers.This system also has the potential for interactive use andextension to augmented reality applications involving ges-tures with printed text. This could be used to more naturally

interact with the text printed on paper and enhance the ex-perience of sighted people as well.

References[1] D. Bloomberg. Dewarping text pages.

http://www.leptonica.com/dewarping.html, August 2010.[2] J. Diener. 2d minimal bounding box.[3] D. Lanman and G. Taubin. Build your own 3d scanner: 3d

photograhy for beginners. In SIGGRAPH ’09: ACM SIG-GRAPH 2009 courses, pages 1–87, New York, NY, USA,2009. ACM.

[4] J. Pilet, V. Lepetit, and P. Fua. Fast non-rigid surface detec-tion, registration and realistic augmentation. Int. J. Comput.Vision, 76:109–122, February 2008.

[5] M. Pilu. Undoing paper curl distortion using applicable sur-faces. In Computer Vision and Pattern Recognition, 2001.CVPR 2001. Proceedings of the 2001 IEEE Computer SocietyConference on, volume 1, pages I–67 – I–72 vol.1, 2001.

[6] S. Zhang. Recent progresses on real-time 3d shape measure-ment using digital fringe projection techniques. Optics andLasers in Engineering, 48(2):149 – 158, 2010. Fringe Projec-tion Techniques.

a d

b e

c fFigure 2. (a) A cylinder wrapped with text. (b) The result of de-warping with the applicable surface model. (c) The result of de-warping using a cylindrical model. (d-f) The result of binarizationand background normalization. As well as dewarping of the textbaselines for (e-f)

’;_fC* >g mum ssR d-Facc Slnnlwl SMM.is *{<pq x\zx0U. smmw xvmmi H ((11231 55:1>q xvznuu, g.,=/ ..˜ VH., u,,.,T$5Z‘ (91??‘mmuw ,7/Zu,_ _ ˜ OU, ..+=/i\pQR m Jl (1/12.*13Pq XYZ’St . .... ..""m;;;;\ ;q?aqs;D {01awww msi N L.\QL/. U-+3Dpqk ..mn /! (012.751\wR Y MNq xvzAOU-,,m’vxv XY;/wu ss l’{*’*˜.,y}.XYZA0u. |?-+=/B (01234 saw) .

55: UCI! lvstinag M:1i99JSO B0|dF3<?c S}:u1!!S!.1.\|_,g_..‘ QR mnupq X\‘Z.:\t‘)(}. !,-+=; ,1mu xyriiiii U (Ul23| 56?$, - zwpq XYZXOU. !?+=/.-,7 stuvu xyzib __(Ql234 S."2!f rmwpq xrznov. !?-+;/{ . mvu.17;z66a ji {M234 56789._’OPQR mnopq XYZAO0.\’W\? stuvw xyz@_ii__B [012J- QI! muopq X YZA OU. ! ?+ ayZ; r e::sw x_v2iifiii B (01234 @6789]*.01*QIt mzuvq XYZKUU. z;.’\’’k swvw gyygjtiib xs (0133.0 mnopq XYZAOU. !?+=/ ’ Is.rv xyzitiii B (01234 56789) < >_ he;...----- .... --.

$1 UC‘}? Ivsting - Mssiflfl JS{ { Bold-Face SI;aggry˜d Sm.-x1.t.C{ .’ QR rmmpq XYZ.\i)[|. !‘.-;j{ r :mw xyriibii li (01231 56789}F2 mopq XYZROU. !?-+=/ ",*6; ncuvv Xyziiiii __(Ql234 567[F QR vrmvpq X}Zl()U. !?-+=/rz . f.vw .ryzii56 jf (UIQ.?} 56789)L\’0PQR mnopq XYZXOO. !?{\"Wr stuvw _00 (01234{PQI! mnopq xvmov. !?+.:,’E; gmvw x_sziic‘ir‘i B (0!23_-{ 9)) (1t()IQ}l xmovq XYZAUU. !?+|{’.’\\‘n STUVW )<YZ_}ii)l) ((112331{Q mnopq XYZAOU. !?+=/ " Fl]Em- xw xyzili B (01234 56789) <>!I 1

Figure 3. From top to bottom: the ground truth with the visibleportion delimited,OCR results for the original image,OCR resultsfor the applicable surface model, OCR results for the cylindricalmodel.

Figure 4. Top: the reference image and mesh. Bottom: the testimage with the appropriately warped mesh shown. Notice howcorresponding triangles contain the same text.

Figure 5. Top: the reference image and mesh. Bottom: the testimage with the appropriately warped mesh shown. Even in thepresence of occlusions corresponding triangles contain the sametext.