ParameterizedHardwareDesignonReconﬁgurableComputers ...mqhuang/papers/2010_image_ijrc.pdfimage processing algorithms are adopted as a case study to demonstrate how hardware designs

Hindawi Publishing CorporationInternational Journal of Reconfigurable ComputingVolume 2010, Article ID 454506, 11 pagesdoi:10.1155/2010/454506

Research Article

Parameterized Hardware Design on Reconfigurable Computers:An Image Processing Case Study

Miaoqing Huang,1 Olivier Serres,2 Tarek El-Ghazawi,2 and Gregory Newby3

1 Department of Computer Science and Computer Engineering, University of Arkansas, Fayetteville, AR 72701, USA2 Department of Electrical and Computer Engineering, The George Washington University, Washington, DC 20052, USA3 Arctic Region Supercomputing Center, University of Alaska Fairbanks, Fairbanks, AK 99775, USA

Correspondence should be addressed to Miaoqing Huang, [email protected]

Received 1 July 2009; Accepted 10 February 2010

Academic Editor: Elıas Todorovich

Copyright © 2010 Miaoqing Huang et al. This is an open access article distributed under the Creative Commons AttributionLicense, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properlycited.

Reconfigurable Computers (RCs) with hardware (FPGA) co-processors can achieve significant performance improvementcompared with traditional microprocessor (μP)-based computers for many scientific applications. The potential amount ofspeedup depends on the intrinsic parallelism of the target application as well as the characteristics of the target platform. Inthis work, we use image processing applications as a case study to demonstrate how hardware designs are parameterized by theco-processor architecture, particularly the data I/O, i.e., the local memory of the FPGA device and the interconnect betweenthe FPGA and the μP. The local memory has to be used by applications that access data randomly. A typical case belongingto this category is image registration. On the other hand, an application such as edge detection can directly read data throughthe interconnect in a sequential fashion. Two different algorithms of image registration, the exhaustive search algorithm and theDiscrete Wavelet Transform (DWT)-based search algorithm, are implemented on hardware, i.e., Xilinx Vertex-IIPro 50 on theCray XD1 reconfigurable computer. The performance improvements of hardware implementations are 10× and 2×, respectively.Regarding the category of applications that directly access the interconnect, the hardware implementation of Canny edge detectioncan achieve 544× speedup.

1. Introduction

Reconfigurable Computers (RCs) are traditional computersextended with co-processors based on reconfigurable hard-ware like FPGAs. Representative RC systems include SGIRC100 [1], SRC-6 [2], and Cray XD1 [3]. These enhancedsystems are capable of providing significant performanceimprovement for scientific and engineering applications [4].The performance of a hardware design on an FPGA devicedepends on both the intrinsic parallelism of the design as wellas the characteristics of the FPGA co-processor architecture,which consists of the FPGA device itself and the surroundingdata interface. Due to the limited size of the internal BlockRAM memory, it is not applicable to store large amountsof data inside the FPGA device. Therefore external, however,local SRAM modules are generally connected to the hardwareco-processor for data storage, such as the example shown in

Figure 1. Furthermore, an FPGA co-processor can directlyaccess the host memory through the interconnect, whichgenerally provides a sustained bandwidth up to several GB/s.The available number and data width of local memory banksand the interconnect channels play important roles in thehardware implementation on an FPGA device and decide theparallelism a design can achieve in many cases. In this work,image processing algorithms are adopted as a case study todemonstrate how hardware designs can be parameterized bythe data I/O of the FPGA co-processor in order to achieve thebest performance.

Image processing applications are capable of gaining asignificant performance improvement with hardware design[5, 6]. Two categories of image processing applications areselected to represent hardware designs that present differentdata I/O requirements. The first category of applications isimage registration, which requires the use of local memory

2 International Journal of Reconfigurable Computing

Local memorybank n− 1

Local memorybank 1

Local memorybank 0

Vendor-specific service logic

User logic (IP)

FPGA device

InterconnectMicro-processor

DMA

Hostmemory

...

Figure 1: General architecture of a reconfigurable computer.

for data access. The second category of applications isedge detection, which directly reads from or writes to theinterconnect in a streaming fashion. Image registration is avery important image processing task. It is used to align ormatch pictures taken in different conditions (at a differenttime, angle, or from different sensors). A vast majority ofautomatic image processing systems require the use of imageregistration processes. Common image registration applica-tions include target recognition (identification of a smalltarget inside a much bigger image), satellite image alignmentin order to detect changes such as land usage or forest fires[7], matching stereo images to recover depth information,and superposing medical images taken at different momentsfor diagnosis [8, 9]. As an example of the applications in thesecond category, edge detectors encompass image processingalgorithms that identify the positions of edges in an image.Edges are discontinuities in terms of intensity or orientationor both and generally represent meaningful characteristicsof the image (boundaries of objects, e.g.). Commonly, edgedetectors are used to filter relevant information in an image.Thus, they greatly reduce the amount of processing neededfor interpreting the information contents of an image. One ofthe most important edge detection algorithms is the Cannyedge detection [10]. The Canny edge detection operatorwas developed by Canny in 1986 and uses a multistagealgorithm to detect a wide range of edges in images. Itremains until now, as a state-of-the-art edge detector usedin many applications.

The implementation of image registration and edgedetection on reconfigurable computers has been previouslyreported in [11, 12], respectively. In this paper, we not onlypresent the detail of hardware design itself, but also exploitthe role of data I/O of the FPGA co-processor in the designprocess. More precisely, we demonstrate how the designof image processing applications is parameterized by localmemory architecture and DMA interface on reconfigurablecomputers. Furthermore, the hardware processing time ofboth applications are formalized in terms of clock cycles.The remaining text is organized as follows. In Section 2,we discuss the related work based on literature survey. InSection 3, two related image registration algorithms, theexhaustive search algorithm and the DWT-based search algo-rithm, and their hardware implementations are discussed.Section 4 focuses on the design of Canny edge detection inhardware. The implementation of all applications on theCray XD1 reconfigurable computer is presented in Section 5.Finally, Section 6 concludes this work.

2. Related Work

Since image registration is a computation-demanding pro-cess in general, hardware (e.g., FPGA device) is leveraged toimprove the processing performance. In [8], Dandekar andShekhar introduced an FPGA-based architecture to acceler-ate mutual information-(MI)-based deformable registrationduring computed tomography-(CT)-guided interventions.Their reported implementation was able to reduce theexecution time of MI-based deformable registration fromhours to a few minutes. Puranik and Gharpure presenteda multilayer feedforward neural network (MFNN) imple-mentation in Xilinx XL4085 for template search in standardsequential scan and detect (SSDA) image registration [13].In [14], Liu et al. proposed a PC-FPGA geological imageprocessing system in which the FPGA was used to imple-ment Fast Fourier Transform-(FFT)-based automatic imageregistration. In [15], El-Araby et al. prototyped an automaticimage registration methodology for remote sensing usinga reconfigurable computer. However, these previous workonly emphasized the image registration algorithm itself. TheFPGA data I/O as a factor in the design was not discussed indetail.

Low-level image processing operators, such as digitalfilters, edge detectors and digital transforms are goodcandidates for hardware implementation. In [16], a genericarchitectural model for implementing image processingalgorithms of real-time applications was proposed and eval-uated. In [17], a Canny edge detection application writtenin Handel-C and implemented in the FPGA device wasdiscussed. The proposed architecture is capable of producingone edge-pixel every clock cycle. The work in [18] illustratedhow to use design patterns in the mapping process toovercome image processing constraints on FPGAs. However,most of the previous works, for example, [16–18], onlyfocused on the algorithms alone and did not consider theplatform characteristics as a factor in the design.

3. Image Registration

In this section, we discuss how to implement image registra-tion algorithms in hardware to exploit the processing paral-lelism, which is bounded by the local memory architecture.

3.1. Background. Image registration can be defined as amapping between two images, the reference image R and thetest image T , both spatially and with respect to the intensity

International Journal of Reconfigurable Computing 3

•(x1, y1)

•(x2, y2)

X

X′

First image

Second image

OO′ty

tx

θ Y

Y ′

Figure 2: Rigid-body transformation (scale = 1).

[19]. If these images are defined as two 2D arrays of a givensize denoted by I1 and I2 where I1(x, y) and I2(x, y) eachmap to their respective intensity values, then the mappingbetween images can be expressed as

I2(x, y

) = g(I1(f(x, y

))), (1)

where f is a 2D spatial-coordinate transformation and g is a1D intensity or radiometric transformation. More precisely,f is a transformation that maps two spatial coordinates, xand y, to new spatial coordinates x′ and y′:

(x′, y′

) = f(x, y

). (2)

g is used to compensate gray value differences caused bydifferent illuminations or sensor conditions.

According to [19], image registration can be viewed asthe combination of four components:

(1) a feature space, that is, the set of characteristics usedto perform the matching and which are extractedfrom the reference and test images;

(2) a search space, that is, the class of potential transfor-mations that establish the correspondence betweenthe reference and test images;

(3) a search strategy, which is used to decide how tochoose the next transformation from the searchspace;

(4) a similarity metric, which evaluates the matchbetween the reference image and the transformed testimage for a given transformation chosen in the searchspace.

The fundamental characteristic of any image registrationtechnique is the type of spatial transformation or mappingused to properly overlay two images. The most commontransformations are rigid-body, affine, projective, perspec-tive, and global polynomial. Rigid-body transformation iscomposed of a combination of a rotation (θ), a translation(tx, ty), and a scale change (s). An example is shown inFigure 2. It typically has four parameters, tx, ty , s, θ, which

map a point (x1, y1) of the first image to a point (x2, y2) ofthe second image as follows:

⎛

⎝x2

y2

⎞

⎠ =⎛

⎝tx

ty

⎞

⎠ + s

⎛

⎝cos θ − sin θ

sin θ cos θ

⎞

⎠

⎛

⎝x1

y1

⎞

⎠

or p2 = t + sRp1,

(3)

where p1 and p2 are the coordinate vectors of the two images;t is the translation vector; s is a scalar scale factor; R is therotation matrix. Since the rotation matrix R is orthogonal,the angles and lengths in the original image are preservedafter the registration. Because of the scalar scale factor s,rigid-body transformation allows changes in length relativeto the original image, but they are the same in both x and yaxes. (Please note both (x1, y1) and (x2, y2) are coordinates inthe same Cartesian coordinate system with the origin O.)

Computing the correlation coefficient is the basic statis-tical approach to registration and is often used for templatematching or pattern recognition. A correlation coefficientis a similarity measure or match metric, that is, it gives ameasure of the degree of similarity between a template (thereference image) and an image (the transformed test image).The correlation coefficient between the reference image Rand the image T′, which is the test image after rigid-bodytransformation, is given as

∑x,y

(R(x, y

)− μR)(T′(x, y

)− μT′)

√∑x,y

(R(x, y)− μR

)2∑x,y

(T′(x, y)− μT′

)2, (4)

where μR and μT′ are mean of the image R and T′. If theimage R matches T′, the correlation coefficient will have itspeak with the corresponding transformation. Therefore, bycomputing correlation coefficients over all possible transfor-mations, it is possible to find the transformation that yieldsthe peak value of the correlation coefficient.

In this work, rigid-body transformation is selected forthe registration between two images and the correlationcoefficient is used to measure the similarity. Further, weassume that both the reference image and the test image are8-bit grayscale and share the same size.

Given a search space, (ΔΘ,ΔX ,ΔY) theoretically all tuplesof (θ,tx,ty) are to be tested to find the tuple that generatesthe maximum correlation coefficient between the referenceimage and the transformed test image (In this work, thescale factor s is fixed at 1.). Figure 3 shows the two stepsto test each tuple. The first step is to apply a rigid-bodytransformation on the test image T to get T′. The second stepis to calculate the correlation coefficient between T′ and thereference image R. As shown in Figure 3(b), only the pixelsof both images within the shaded region are used during thecalculation. Since the test image T is rotated and translatedto obtain T′, some parts of T′ are beyond the shaded region.In other words, some portions of the shaded region (shownas crossed regions in the left Cartesian coordinate system ofFigure 3(b)) do not belong to T′. For the pixels belonging tothese crossed regions, their values are treated as zeros in thecalculation.


X

Y

O

T′

T

(a)

X

Y

O

T′

X

Y

O

R

(b)

Figure 3: Two steps in image registration: (a) rigid-body transformation on the test image T , (b) calculate correlation coefficient betweenthe transformed test image T ′ and the reference image R.

#0 #1 #2 #3 #P + 1mem mem mem mem mem

R T T′0 T′1 T′P−1

Figure 4: Local memory data storage layout in the exhaustive searchalgorithm for image registration.

In the remaining part of this section, two differentapproaches based on rigid-body transformation are dis-cussed in details. The first approach literally tests the wholesearch space to find the best tuple of (θ,tx,ty). The secondapproach applies DWT on both the reference image andthe test image to reduce the search resolutions in order toimprove the search efficiency.

3.2. Exhaustive Search Algorithm. As its name implies, theexhaustive search algorithm tests all possible tuples of(θ,tx,ty) with a fixed search resolution, δθ , δx, and δy , oneach dimension, respectively, in order to find the tuplethat produces the highest correlation coefficient betweenthe transformed test image and the reference image. If thisalgorithm is implemented on a scalar microprocessor, thesetuples have to be tested in sequence. However, if the samealgorithm is implemented in hardware, multiple tuples canbe tested in parallel to improve the performance.

Since the size of an image is normally bigger than theamount of available Block RAM inside an FPGA device, thelocal external memory is used to store images. Assumingthere are P + 2 individual local memory banks connecteddirectly to the FPGA device, and one bank keeps the referenceimage R, one bank keeps the test image T . The otherremaining P banks are used to store P transformed testimages T′s using different tuples of (θ,tx,ty), as shown inFigure 4. If we further assume that each memory bank hasits own independent read and write ports, P transformationsof the test image can be carried out concurrently. Thecalculation of correlation coefficients between the image Rand P different T′s can be performed in parallel as well.

Given the coordinate of one pixel in the original image,(3) is used to calculate the coordinate of the correspondingpixel in the transformed image. Then, the intensity of thepixel (x1, y1) in T can be written into T′ at the coordinateof (x2, y2). If we assume that there are S pixels in the originalimage and the hardware implementation is fully pipelined,the transformation step would take approximately S clockcycles. Furthermore, some extra clock cycles are neededto initialize the intensities of all pixels within the shadedregion to zero due to two reasons. First, there are severalregions within which the pixels do not belong to T′, asshown in Figure 3(b). Second, there may exist artifacts whosecoordinates are within both the shaded region and T′, but arenot calculated due to discretization, as shown in Figure 5. Ifthe intensities of these pixels are left randomly, it may affectthe accuracy of the correlation coefficient. Therefore, it isnecessary to initialize the intensities of all pixels in the shadedregion to zero in the first place. Since the data width of thememory bank’s access ports is multiple-byte, multiple pixelscan be initialized in one clock cycle. If we assume that thedata width is D-byte, then the initialization process wouldtake roughly S/D clock cycles. Overall, the transformationstep of P tuples would take ((D + 1)/D)S clock cycles. Themean intensity μT′ of each T′ can be calculated during thetransformation step, hence it takes no extra time. The meanintensity μR can be precalculated by the microprocessorand forwarded to the FPGA device later since it remainsunchanged during the whole image registration process.

Although the calculation of the correlation coefficient as(4) between R and T′s is more complicated than the trans-formation step, it takes the same time as the initializationprocess since D pixels can be read and processed in the sameclock cycle. Altogether, these three steps, including initializa-tion, transformation and correlation coefficient calculation,would take ((D + 2)/D)S clock cycles for testing P tuples of(θ,tx,ty). If the entire search space consists of ΔΘ∗ΔX ∗ΔYtuples, the whole registration process would take

ΔΘ∗ ΔX ∗ ΔY ∗ (D + 2)PD

S (5)

clock cycles. Apparently, the image registration time inhardware can be significantly reduced by increasing thenumber of local memory banks. Widening the data width of


(a) (b)

Figure 5: Artifacts due to discretization in rigid-body transformation ((a) the original image; (b) the transformed image).

Rows Columns

Original orprevious low-pass results, Ik

High pass:decimate

columns by 2

Low pass:decimate

columns by 2Lk+1

Hk+1

Low pass:decimaterows by 2

Low pass:decimaterows by 2

High pass:decimaterows by 2

High pass:decimaterows by 2

LLk+1

LHk+1

HHk+1

HLk+1

Ik+1

Figure 6: DWT decomposition of an image.

Decomposition

Search

I0 I1 Ik−1 Ik

I0 I1 Ik−1 Ik

Image size

Search space

· · ·

· · ·

Figure 7: Decrease the search space and increase the searchresolution in the search process.

access port of local memory can improve the performance aswell; however, it can also hit the upper bound very quicklydue to the fact that

limD→∞

D + 2D

= 1. (6)

3.3. DWT-Based Search Algorithm. Although the exhaustivesearch algorithm is quite straightforward, it is computation-demanding as well. In [20], a DWT-based image registrationapproach was proposed. As shown in Figure 6, both thetest image and the reference image go through severallevels of Discrete Wavelet Transform before applying imageregistration. After each level of DWT, the image size isshrunk to 1/4 of the previous level. In the meantime theimage resolution is reduced to half. For example, if k levelsof DWT are applied on both the test image T0 and thereference image R0, two series of images, T0, T1, . . . ,Tk, andR0, R1, . . . ,Rk, are obtained. The registration process startsfrom the exhaustive search between Tk and Rk among thesearch space of (ΔΘ,ΔX ,ΔY) with the search resolution,2k ∗ δθ , 2k ∗ δx, and 2k ∗ δy , on each dimension. Theregistration result between Tk and Rk, (θk,txk ,tyk ) becomesthe center of the search space of the registration betweenTk−1 and Rk−1. In other words, the registration betweenTk−1 and Rk−1 is among the search space of (θk ± 2k ∗δθ , txk±2k∗δx, tyk±2k∗δy). However, the search resolutionis increased to 2k−1 ∗ δθ , 2k−1 ∗ δx, and 2k−1 ∗ δy , oneach dimension. In general, when the registration processtraces back one level, the search scope is reduced to half on


Table 1: Search strategy summary for rotation.

Decomposition Level Search Space Search Resolution Result

k ΔΘ 2k ∗ δθ θkk − 1 [θk − 2k ∗ δθ ; θk + 2k ∗ δθ] 2k−1 ∗ δθ θk−1

k − 2 [θk−1 − 2k−1 ∗ δθ ; θk−1 + 2k−1 ∗ δθ] 2k−2 ∗ δθ θk−2

· · · · · · · · · · · ·2 [θ3 − 23 ∗ δθ ; θ3 + 23 ∗ δθ] 22 ∗ δθ θ2

1 [θ2 − 22 ∗ δθ ; θ2 + 22 ∗ δθ] 2∗ δθ θ1

0 [θ1 − 2∗ δθ ; θ1 + 2∗ δθ] δθ θ∗0∗

The final output.

mem #0 mem #1

R0 T0

R1 T1

R2 T2

......

Figure 8: Store the original and decomposed images in the samememory bank in the DWT-based image registration.

each dimension, and the search resolution is increased twotimes on each dimension, respectively. This search strategyis illustrated in Figure 7. Table 1 details the search space andsearch resolution at each step in which rotation is taken as anexample.

Different DWT decomposition processes have to becarried out in a sequence. Similarly the search processes atdifferent levels need to be performed one after the other. Dueto these two reasons, the original image and the decomposedimages can be stored in the same memory bank, as shown inFigure 8 in which Rk or Tk denote the decomposed image atthe level k.

If we use the same assumptions as in Section 3.2, that is,both original images consist of S pixels, the data width of thelocal memory isD-byte, and there are P+2 independent localmemory banks, then the DWT decomposition step alone willtake

S

D

(

1 +14

+ · · · +(

14

)k−1)

= 4S3D

(

1−(

14

)k)

(7)

clock cycles.The search between the decomposed reference image

and the test image at each level can use the same methoddescribed in Section 3.2. By observing Table 1, it is found thatthe search space at each level, except the level k, is 125 tuples

of (θ,tx,ty). Therefore, the search from level 0 to level k − 1will take

125S(D + 2)PD

(

1 +14

+ · · · +(

14

)k−1)

= 500S(D + 2)3PD

(

1−(

14

)k)(8)

clock cycles. The search at the level k itself takes

ΔΘ∗ ΔX ∗ ΔY ∗ (D + 2)32kδθδxδyPD

S (9)

clock cycles.

4. Edge Detection

Edge detection aims at identifying pixels in a digital image atwhich the image brightness changes sharply, that is, havingdiscontinuities. Most edge detection algorithms involvethe convolution process between the image and a kernel.Convolution provides a way of “multiplying together” twoarrays of numbers, generally of different sizes, but of the samedimensionality, to produce a third array of numbers of thesame dimensionality. This can be used in image processingto implement operators whose output pixel values are simplelinear combinations of certain input pixel values. In an imageprocessing context, one of the input arrays is normally just agrayscale image. The second array is usually much smaller,and is also two dimensional (although it may be just asingle coefficient). The second array is always known as thekernel, as shown in Figure 9. If the image has M rows andN columns, and the kernel has m rows and n columns, thenthe size of the output image will consist of M − m + 1 rowsand N − n + 1 columns. Mathematically we can write theconvolution between the image I and the kernel K as

O(x1, y1

) =m−1∑

x2=0

n−1∑

y2=0

I(x1 + x2, y1 + y2

)× K(x2, y2), (10)

where x1 runs from 0 to M −m and y1 runs from 0 to N − n.From Figure 9 and (10), we can find that the calculation

of different pixels in the output image is independent to eachother. Therefore, the intensities of multiple output pixels canbe computed in parallel in a hardware design. Since the input


I0,0

I1,0

I2,0

I3,0

I4,0

I5,0

I6,0

I7,0

I8,0

I0,1

I1,1

I2,1

I3,1

I4,1

I5,1

I6,1

I7,1

I8,1

I0,2

I1,2

I2,2

I3,2

I4,2

I5,2

I6,2

I7,2

I8,2

I0,3

I1,3

I2,3

I3,3

I4,3

I5,3

I6,3

I7,3

I8,3

I0,4

I1,4

I2,4

I3,4

I4,4

I5,4

I6,4

I7,4

I8,4

I0,5

I1,5

I2,5

I3,5

I4,5

I5,5

I6,5

I7,5

I8,5

I0,6

I1,6

I2,6

I3,6

I4,6

I5,6

I6,6

I7,6

I8,6

I0,7

I1,7

I2,7

I3,7

I4,7

I5,7

I6,7

I7,7

I8,7

I0,8

I1,8

I2,8

I3,8

I4,8

I5,8

I6,8

I7,8

I8,8 Input image

(a)

K0,0

K1,0

K2,0

K0,1

K1,1

K2,1

K0,2

K1,2

K2,2

Kernel

(b)

Figure 9: An example small image (a) and kernel (b) to illustrate convolution.

Memorymodules

Proprietaryinterconnect

WritingDMAengine

ReadingDMAengine

Writing buffer

Reading buffer

Alg

orit

hm

logi

c


FPGA device

Figure 10: Streaming data transfer mode on reconfigurable computers.

image reading and the output image writing are both in asequential mode, the data storage in local memory can beavoided. Instead, the user logic can access the interconnectdirectly for fetching the source image and storing the resultimage. In order to optimize the hardware performance, theinterconnect and the user logic are chained into a pipeline,and the data run through the pipeline as a stream. We callthis architecture Streaming Data Transfer Mode, shown inFigure 10. Two DMA engines work in parallel to retrieveraw data blocks from and return result data blocks to themain memory. Under ideal circumstances, the reading DMAengine receives one raw data block from the input channeland the writing DMA engine puts one result data block tothe output channel every clock cycle.

Being one stage of the overall pipeline, the design ofthe algorithm logic is parameterized by the characteristicsof other components in the pipeline, that is, the data widthof the interconnect between the FPGA device and the μP.Because the data width of the interconnect fabric is multiple-byte wide in general, several pixels are fed into the algorithmlogic in the same clock cycle. To maximize the throughput ofthe overall architecture, the algorithm logic has to be capable

of performing the operations of multiple pixels concurrentlyand taking new data input every clock cycle. In the followingdiscussion, we assume that (1) the image is in 8-bit grayscale,(2) the image size is M × N and the kernel size is m ×n, and (3) the data width of the interconnect is D-byte.Furthermore, pixels in the original image are delivered intothe algorithm logic in a stream, starting with the pixel atthe top-left corner, ending with the pixel at the bottom-rightcorner.

The diagram of the algorithm logic is shown in Figure 11.The architecture consists of four components, one LineBuffer, one Data Window, an array of PEs, and the DataConcatenating Block.

The quantity of PEs is D, that is, the data width ofthe interconnect in byte. Every PE is fully pipelined and iscapable of taking a new input, that is, one block of m × npixels, every clock cycle. The output of one PE is the intensityof one pixel in the result image.

The Data Window is a two-dimension register array ofm × (n + D − 1) in charge of providing image blocks to thedownstream PEs. Analogously, the Data Window scans theoriginal image from left to right and from top to bottom


Inputimage

Outputimage

Line buffer

Data window

PE1 PE2 PED

Data concatenating

D pixels read from original image

D+n−2

Datawindow

Linebuffer

0

m−1

1

m−2

0 1 2 3 4 D+1DD−1 N−3N−2N−1· · ·· · ·

· · ·

· · ·

· · ·

· · ·· · ·

· · ·

· · ·

· · ·

· · ·

· · ·

· · ·· · ·

...

...

0

Figure 11: Diagram of the algorithm logic in edge detection.

p1,7

p2,7

p3,7

p4,7

p5,7

p1,8

p2,8

p3,8

p4,8

p5,8

p1,9

p2,9

p3,9

p4,9

p5,9

p1,10

p2,10

p3,10

p4,10

p5,10

p1,11

p2,11

p3,11

p4,11

p5,11

p1,12

p2,12

p3,12

p4,12

p5,12

p1,13

p2,13

p3,13

p4,13

p5,13

p1,14

p2,14

p3,14

p4,14

p5,14

p1,15

p2,15

p3,15

p4,15

p5,15

p1,16

p2,16

p3,16

p4,16

p5,16

p1,17

p2,17

p3,17

p4,17

p5,17

p1,18

p2,18

p3,18

p4,18

p5,18

p1,19

p2,19

p3,19

p4,19

p5,19

p1,20

p2,20

p3,20

p4,20

p5,20

p1,21

p2,21

p3,21

p4,21

p5,21

p1,22

p2,22

p3,22

p4,22

p5,22

p1,23

p2,23

p3,23

p4,23

p5,23

p1,24

p2,24

p3,24

p4,24

p5,24

p1,25

p2,25

p3,25

p4,25

p5,25

p1,26

p2,26

p3,26

p4,26

p5,26

Figure 12: The content of Data Window in two consecutive steps (assume the kernel size is 3× 3 and D is 8).

Localmemorybank 2

Localmemorybank 3

Localmemorybank 0

Localmemorybank 1


User logic

To host(1.6 GB/s)

From host(1.6 GB/s)

FPGA device (XC2VP50)

64

64

64

64

64

64

64

64

64 64

Figure 13: Cray XD1 FPGA local memory architecture.

with a horizontal stride of D pixels and a vertical strideof 1 pixel. Figure 12 demonstrates the scanning from left toright and shows the contents in two consecutive steps of thedata window. Once the data window receives a valid input,that is, one block ofm×D pixels, it does aD-byte left shift forall rows with the input tailing the right most n− 1 columns.

The Line Buffer is a two-dimension register array of(m−1)×N . The original image is transferred into the FPGAas a stream. However, PEs request image blocks that spreaddifferent rows. The purpose of the line buffer is to keep allpixels in registers until one m × D block forms. One m × Dblock, shown in gray in Figure 11, comprises the new arrivedD pixels and another (m − 1) × D pixels that reside at thehead of every row of the line buffer. Every time the linebuffer receivesD pixels of the original image, it performs twoactions simultaneously. The first action is to deliver the newformed image block to the data window. The second actionis to do a D-byte left shift in a zigzag form, in which two

neighbor rows are linked together by connecting their tailand head.

The design of the Data Concatenating Block is straight-forward because what it does is to concatenate the outputs ofthe upstream PEs together. Once it receives valid output fromthe PEs, that is, D consecutive pixels in the output image, itsends them into the output channel.

In general, the four components in Figure 11 form apipeline chain and each of them is fully pipelined as well.This architecture is able to accept D pixels of the originalimage every clock cycle and output D pixels of the resultimage every clock cycle at the same time. In case of amultiple-stage algorithm, such as the four-stage Canny edgedetector, various stages can be chained together and eachstage consists of these components with different parametersand functionalities. Under ideal scenario, it would take (M×N)/D clock cycles to perform an edge detection operationon an input image. However, the real performance is upper-bounded by the sustained bandwidth of the interconnect ingeneral.

5. Implementation and Results

These two image registration algorithms and a Canny edgedetection algorithm have been implemented on the CrayXD1 reconfigurable computer with Xilinx XC2VP50 FPGAdevices. On the Cray XD1 platform, each FPGA deviceis connected to four local SRAM modules, 4 MB each, asshown in Figure 13. Every local memory module has separatereading and writing ports connected to the FPGA device,and is able to accept reading or writing transactions every


(a) (b)

Figure 14: The original image and the image after applying Canny edge detection.

Table 2: Performance improvement of image registration and Canny edge detection on Cray XD1.

Algorithm name Computing time (s) Speedup Resource utilization

Opteron 2.4G Cray XD1 Slices Built-in multipliers

Image registration∗Exhaustive search 157.347 16.193 9.72 10.766 (45%) 42 (18%)

DWT-based search† 1.298 0.829 1.57 20.205 (85%) 108 (55%)

End-to-end throughput (MB/s)

Canny edge detection 2.20 1.196 543.6 15.015 (63%) 200 (86%)∗The sizes of reference and test image are both 1024× 1024. The search spaces of ΔΘ, ΔX , and ΔY are all from −8 to +8.†Three levels of DWT are performed before the search, which is based on LL coefficients only.

clock cycle. Furthermore, the interconnect has separate inputand output channels, both of them are 64-bit wide. Themaximum operating clock rate for the user logic is 200 MHz,which is achieved in all our hardware implementations.

For the exhaustive search approach, the reference imageand the test image are stored in two separate local memorymodules. Every time, two possible tuples of (θ,tx,ty) aretested, that is, two different rigid-body transformations areperformed on the test image simultaneously and producetwo transformed test images, T′ and T′′, which are storedin two additional local memory modules separately. Thecalculation of the correlation coefficients between these twoimages and the reference image is carried concurrently aswell. All the hardware modules are fully pipelined in ourdesign in order to realize the full potential of a hardwareimplementation. Compared with the performance of thesoftware implementation running on a single micropro-cessor, an AMD Opteron 2.4 GHz, the performance ofthe hardware implementation on a single FPGA deviceis approximately 10× better, as shown in Table 2. Theperformance improvement of hardware implementationcompared with software implementation is mainly due tothe fully pipelined hardware design. For example, duringthe correlation coefficient calculation step, it only takes oneclock cycle for hardware to calculate 8 pixels. On the otherhand, the software implementation has to calculate one pixeleach time. The measured time on the Cray XD1 is the end-to-end time including data transfer time between the μP

and the FPGA and the hardware processing time on theFPGA. In the 16.193 s, data transfer time is merely 6 ms.Therefore, the performance of hardware implementation inthis case is almost upper-bounded by the available numberof local memory banks since only 45% of the FPGA slicesare used. As shown in (5), the hardware processing time isreciprocal to the number of memory banks that are usedto store transformed test images. Therefore the speedup ofhardware implementation compared with software versionis linearly proportional to the number of local memorybanks.

For the DWT-based approach, three levels of DWT areapplied on both the test image and the reference imagebefore the registration process starts. All the original imageand decomposed images of different levels are stored inlocal memory modules, as depicted in Figure 8. Since thewhole process consists of two steps, the DWT decompositionand the search, the hardware utilization almost doubles inthis case. More built-in multipliers are used in the DWTdecomposition as well. Compared with the software imple-mentation, the hardware is barely 2× faster for the DWT-based search approach. The comparatively low speedupachieved by hardware in this case is due to the fact that theamount of computation is significantly reduced because ofthe DWT decomposition.

For both cases, the hardware implementation is mainlycharacterized by the local memory architecture, andthe performance can be improved if more processing


concurrency is allowed, given that more independent localmemory modules are available.

For the category of applications that use the interconnectfor data access, a Canny edge detection algorithm is imple-mented following the architecture in Figure 11. The Cannyedge detector comprises four stages in which each stage takesthe output from the preceding stage and feeds its output tothe following stage as follows:

Input Image −→ Gaussian Smoothing

−→ Compass Edge Detecting

−→ Non-maximum Suppression

−→ Edge Tracingand Thresholding.

(11)

Because the interconnect is 8-byte wide, 8 edge detectionoperators are implemented and execute in parallel. Figure 14shows the original image and the corresponding output afterapplying the Canny edge detection algorithm. In Cannyedge detection algorithm, the processing of each pixel inthe output image involves neighbor pixels in the inputimage. Furthermore, this processing consists of 4 stages andtakes hundreds of clock cycles as latency. Due to the fullypipelined design in hardware implementation, the FPGAdevice is capable of computing 8 pixels in the outputimage every clock cycle no matter how complicated thecomputation of each pixel is. On the other hand, the pixelsin the output image are computed one by one in thesoftware implementation. Further, it would take thousandsof cpu cycles to compute one pixel. Therefore hardwareimplementation of the Canny edge detection algorithmachieves 544× speedup compared with the correspondingsoftware implementation. Higher speedup can be achievedif multiple images can be processed simultaneously givenseveral interconnect channels are connected to the sameFPGA co-processor and there are enough hardware resourcesto implement multiple instances of edge detection operators.

6. Conclusions

In this paper, we demonstrate how the parallelism of ahardware design on reconfigurable computers is param-eterized by the co-processor architecture, particularly thenumber and the data width of local memory banks and theinterconnect. Image registration algorithms based on rigid-body transformation are adopted as a case study to representapplications that use local memory. Two related; however,different algorithms, the exhaustive search algorithm and theDWT-based search algorithm, are described in detail. Forthe exhaustive search algorithm, the performance is linearlyproportional to the available number of local memorybanks. On the other hand, the DWT-based search algorithmimproves the efficiency by applying DWT decompositionon both the reference and test images before the search inorder to reduce the search scope. Compared with softwareimplementations, hardware implementations of exhaustivesearch and DWT-based search achieve 10× and 2× speedup,respectively. For the category of applications that directly

access the host interconnect, edge detection is selected asa case study. A streaming data transfer mode in which theuser logic and the interconnect are chained into a pipelineis proposed. A user logic hardware architecture whoseparallelism is decided by the data width of the interconnectis discussed in detail. A Canny edge detection applicationfollowing the proposed architecture is capable of achieving544× speedup compared with the corresponding softwaredesign.

As a future work, we will extend our work to the Tile64 platform [21]. Parameters such as the external memorybandwidth and the intertile bandwidth will be taken intoaccount to implement image processing algorithms. Weexpect to take advantages from Tile 64’s capability of creatingmulticore pipelines internally.

Acknowledgment

This work was supported in part by Arctic Region Supercom-puting Center (ARSC) at the University of Alaska Fairbanks.

References

[1] Reconfigurable Application-Specific Computing User’s Guide(007-4718-007), Silicon Graphics, 2008.

[2] SRC Carte C Programming Environment v2.2 Guide (SRC-007-18), SRC Computers, 2006.

[3] Cray XD1 FPGA Development (S-6400-14), Cray, 2006.[4] T. El-Ghazawi, E. El-Araby, M. Huang, K. Gaj, V. Kindratenko,

and D. Buell, “The promise of high-performance reconfig-urable computing,” Computer, vol. 41, no. 2, pp. 69–76, 2008.

[5] C. Torres-Huitzil and M. Arias-Estrada, “FPGA-based config-urable systolic architecture for window-based image process-ing,” Eurasip Journal on Applied Signal Processing, vol. 2005,no. 7, pp. 1024–1034, 2005.

[6] J. Cho, S. Mirzaei, J. Oberg, and R. Kastner, “FPGA-Basedface detection system haar classifiers,” in Proceedings of the 7thACM SIGDA International Symposium on Field-ProgrammableGate Arrays (FPGA ’09), pp. 103–111, Monterey, Calif, USA,February 2009.

[7] J. Le Moigne, W. J. Campbell, and R. F. Cromp, “An automatedparallel image registration technique based on the correlationof wavelet features,” IEEE Transactions on Geoscience andRemote Sensing, vol. 40, no. 8, pp. 1849–1864, 2002.

[8] O. Dandekar and R. Shekhar, “FPGA-accelerated deformableimage registration for improved target-delineation duringCT-guided interventions,” IEEE Transactions on BiomedicalCircuits and Systems, vol. 1, no. 2, pp. 116–127, 2007.

[9] B. Zitova and J. Flusser, “Image registration methods: asurvey,” Image and Vision Computing, vol. 21, no. 11, pp. 977–1000, 2003.

[10] J. Canny, “A computational approach to edge detection,” IEEETransactions on Pattern Analysis and Machine Intelligence, vol.8, no. 6, pp. 679–698, 1986.

[11] M. Huang, O. Serres, T. El-Ghazawi, and G. Newby,“Implementing image registration algorithms on reconfig-urable computer,” in Proceedings of the 10th Military andAerospace Programmable Logic Devices Conference (MAPLD’08), September 2008.

[12] M. Huang, O. Serres, S. Lopez-Buedo, T. El-Ghazawi, andG. Newby, “An image processing architecture to exploit I/O


bandwidth on reconfigurable computers,” in Proceedings of the4th Southern Conference on Programmable Logic (SPL ’08), pp.257–260, San Carlos de Bariloche, Argentina, March 2008.

[13] M. S. Puranik and D. C. Gharpure, “FPGA implementationof MFNN for image registration,” in Proceedings of the IEEEInternational Conference on Field-Programmable Technology(ICFPT ’02), pp. 364–367, December 2002.

[14] S. Liu, N. Li, and W. Wang, “Efficient codesign for geologyimage processing,” in Proceedings of the 49th Midwest Sympo-sium on Circuits and Systems (MWSCAS ’06), vol. 1, pp. 272–275, San Juan, Puerto Rico, USA, August 2006.

[15] E. El-Araby, M. Taher, T. El-Ghazawi, and J. L. Moigne, “Auto-matic image registration for remote sensing on reconfigurablecomputers,” in Proceedings of the 9th Military and AerospaceProgrammable Logic Devices Conference (MAPLD ’06), 2006.

[16] M. A. de Barros and M. Akil, “Low level image pro-cessing operators on FPGA: implementation examples andperformance evaluation,” in Proceedings of the 12th IAPRIntarnational Conference on Pattern Recognition, vol. 3, pp.262–267, October 1994.

[17] V. Muthukumar and D. V. Rao, “Image processing algorithmson reconfigurable architecture using HandelC,” in Proceedingsof the EUROMICRO Systems on Digital System Design (DSD’04), pp. 218–226, Rennes, France, August-September 2004.

[18] K. T. Gribbon, D. G. Bailey, and C. T. Johnston, “Usingdesign patterns to overcome image processing constraints onFPGAs,” in Proceedings of the 3rd IEEE International Workshopon Electronic Design, Test and Applications (DELTA ’06), pp.47–53, Kuala Lumpur, Malaysia, January 2006.

[19] L. G. Brown, “Survey of image registration techniques,” ACMComputing Surveys, vol. 24, no. 4, pp. 325–376, 1992.

[20] T. A. El-Ghazawi, P. Chalermwat, and J. L. Moigne, “Wavelet-based image registration on parallel computers,” in Proceed-ings of the ACM/IEEE Supercomputing Conference (SC ’97), pp.20–28, November 1997.

[21] http://www.tilera.com/.

Documents

ParameterizedHardwareDesignonReconﬁgurableComputers ...mqhuang/papers/2010_image_ijrc.pdfimage processing algorithms are adopted as a case study to demonstrate how hardware designs