Volume 50, number 4, 1995 29

George Vosse~rnan i

;4 e

Applications of tree search methods photogrammetry

Digital photogrammetry has emerged from the combination of analytical photogrammetry and digital image processing and interpretation techniques. Methods like resampling, contrast enhancement and image matching already found their way to many applications in photogrammetry. Other methods, like tree search, are less well known. This paper explores the possibilities for using tree search methods to solve photogrammetric problems. It is shown that tree search methods can be useful for line following and knowledge-based image analysis. Application of tree search methods to matching problems is also possible. This application, however, requires fairly good image segmentations, which usually cannot be obtained for aerial imagery.

1. Introduction

Digital photogrammetry has emerged from the combination of analytical photogrammetry and digital image processing and interpretation tech- niques. The developments in digital photogramme- try that have taken place up to now are mainly based on two classes of image processing tech- niques: image transformations (both geometric and radiometric) and matching.

Transformations of digital images using resam- piing and contrast enhancement techniques are used for the calculation of digital orthophoto's, rectified imagery and mosaics (Wiesel, 1975). The ease with which these products can now be made and their large range of applications within ge- ographical information systems have made them very popular.

Much attention also has been paid to image matching methods• With various ~pes of area- based and feature-based matching methods several photogrammetric tasks like interior and relative orientation, point transfer and DEM generation can now be done by computer (Schenk et al., 1991; Ackermann and Krzystek, 1991; Hellwich et al., 1994; Tsingas, 1994)•

Beside image transformations and matching many other classes of methods are used in corn-

1Faculty of Geodetic Engineering, Delft University of Tech- nology, Thijsseweg 11, 2629 JA Delft, Netherlands.

puter vision research that could be useful for solv- ing photogrammetric tasks• One such class is the class of tree search methods. This kind of methods is still hardly used for photogrammetric tasks. The purpose of this paper is to discuss several examples of potential applications of tree search methods in digital photogrammetry.

Tree search methods solve problems that can be represented in search trees. In the next section we therefore start with describing a few photogram- metric tasks and show how the solution spaces of the related problems can be represented in search trees. In Sect. 3, the basic concepts of tree search methods are reviewed• Attention is also paid to heuristic search strategies that help to minimize the average search times. In Sect. 4 we will then re- turn to the photogrammetric problems and discuss the suitability of the presented tree search methods and strategies for solving those problems.

2. Search trees of photogrammetric problems

2.1. Object location and recognition

Object location in digital images is a well- known computer vision research topic. In the so-called bin-picking problem, the position of a specified object is to be determined in an im- age with several objects. The common approach is to describe both the parsed object model and the segmented image in terms of features and

ISPRS Journal of Photogrammetry and Remote Sensing, 50(4): 29-37 0924-2716/95/$09.50 © 1995 Elsevier Science B.V. All rights reserved.

30 1SPRS Journal of Photogrammetry and Remote Sensing

their relationships (so-called relational descrip- tions (Shapiro and Haralick, 1981)) and to find the corresponding features by a comparison of the attribute values of the features and their relation- ships.

Two relational descriptions can be matched by tree search methods. This is called relational matching (Boyer and Kak, 1988; Vosselman, 1992). In the search tree all combinations of possibly corresponding features are represented.

Let the features of the two relational descrip- tions be denoted by Pi E P and qj ~ Q. In case of object location, pi and qj could be features of the object model and the segmented image, respec- tively. The features Pi of set P are called units, the features qj of set Q are called labels. The search tree should contain all possible mappings from P

Pl q2 ~- q4 "~ q7 / \ / \ P2 q~ °'4 q2 % ql

/ \ R , P8 qs % ql %

t / t \ P4 q7 qz qs qs Figure 1. Search tree representing mappings between the fea- tures of two relational descriptions.

to Q. At each level of the search tree in Fig. 1, a label qj is sought for a unit Pi. After combining, e.g., unit Pt with label q4, a label for unit P2 is

a

ii

Figure 2. (a) Segmented aerial image. (b) Model of topographical control point. (c) Result of object location.

b

Volume 50, number 4, 1995 31

Figure 2 (continued). (c) Result of object location.

sought at the next level of the tree. Thus each path from the root node to a leaf node represents a dif- ferent mapping P --, Q. A tree search algorithm should select the path of the mapping with the best similarity between the attributes of the correspond- ing features and their relations. After completion of the tree search the location of the object in the image can be inferred from the corresponding features.

Unlike object location, object recognition usu- ally involves models of multiple objects. Often it is unknown how many instances of which objects are contained in the image. Object recognition is clearly more complex than object location. The method of matching the image with one of the object models is, however, the same.

Comparison of images and models for object location has found several applications in pho-

togrammetry. In Schickler (1992) and Haala and Vosselman (1992) it is used to locate topographical control points (Fig. 2). Especially in close-range photogrammetry it is also used to locate signalized points. Not all applications, however, use reiational matching to solve the object location problem. Other methods like clustering and neural networks are also used.

2. 2. Stereo image matching

Finding corresponding points in overlapping images is another matching problem that is of in- terest to both photogrammetrists and computer scientists. In photogrammetry, area- and feature- based matching methods have been successfu!l_y used for matching digitized aerial photographs. Both methods rely on similarity in grey value pat-

32 ISPRS Journal of Photogrammetty and Remote Sensing

terns measured by squared differences or corre- lation coefficients. These similarity measures are good indicators for the correspondence if the im- ages are approximately parallel and good approxi- mate values for the object shape are known. These requirements are fulfilled for aerial stereopairs that are matched hierarchically.

For convergent stereopairs, that are often en- countered in close-range photogrammetry, corre- sponding points may not have similar grey value distributions. The common area and feature-based methods will therefore fail. The matching prob- lem can, however, be solved if the matching al- gorithm operates at a more symbolical level at which the features are more invariant under per- spective transformations. For this purpose both images have to be segmented and described by features and their (topological) relationships. Like the object location problem, the image matching problem can now be considered as a problem to determine corresponding features in two relational descriptions and be solved by a tree search method. Therefore, the search tree of the image matching problem has the same structure as the tree in Fig. 1.

2.3. Edge and line following

Of al~ components of photogrammetric data acquisition the actual mapping phase is most time consuming. Although many research efforts take place to automate this image interpretation by developing segmentation algorithms, automated mapping is still far out of reach. Mapping in digital imagery can, however, be supported by image pro- cessing. One example is tracking of lines with high contrast, starting at a point that is indicated by a human operator.

Ideally, lines in digital images are sequences of pixels with high gradients. The candidate line pixels are selected by thresholding the gradient strength. For each pixel, the direction of the edge can also be calculated. Figure 3a shows the edge direction of pixels with a high gradient in a small subimage. Two edge pixels can be connected to a line if they are adjacent and have approximately the same edge direction. This results in a graph of possible connections (Fig. 3b) (el. BaUard and Brown, 1982). The task is to select the line with the highest sum of the gradient strengths, i.e., the line with the best contrast.

0 1 2 3 4

°iiiiiiiiii!iiliiiiiiiiiiiii ,: 1 1 " , ' i .... i . . . . . . l . . . i .J . i . . . . i

' i 1 ! t t i !1! • . . . . • . . . . . . . . . . . . . . i . . . . •

0 1 2 3 4 Oi i 2

3! ° °

4: • . . . . .

(a} (b)

/ \ (2,=)

,=1, / . t , , /.t,,,,.,, ( s ~ (s,s} {s~} (s,sl

/ I X I\ IX IX IX (4,o) (4,~ (43.) (4,2) (a,a) (a,4) (4A) (4,2) (a,s) (3A) (4,4)

(c) Figure 3. (a) Local edge directions. (b) Possible pixel connec- tions. (c) Tree of possible paths.

All possible paths in the graph can be repre- sented in a tree. In the graph of Fig. 3b the line is to start in pixel (0,2). This results in the tree of Fig. 3c. Note that some of the subtrees in this figure are identical, because some pixels can be reached along multiple paths of the graph. The tree explicitly lists all sequences of pixels that can be connected to a line. The task of a tree search algorithm is to select the line with the highest contrast.

A line-following algorithm may then proceed by constructing a tree of all possible paths of a certain length (e.g., 5 pixels like in the example), selecting the best path and continuing with a new tree search at the end of the selected path. This process iterates until no more pixels with high gradients are found.

2.4. Geometric reasoning

Reasoning problems form another class of problems that can be tackled with tree search

Volume 50, number 4, 1995 33

(a)

I1 / l \

/ ! \ 13 + .- D,

I 14 +

i / \ / \ / \

• ~ D, + m

(b)

Figure 4. (a) Polyhedron with convex (+), concave ( - ) and occluding (~) edges. (b) Line drawing interpretations in a search tree.

methods. A well-known example is the Waltz' la- belling of polyhedra (Waltz, 1975), which aims at the interpretation of a line drawing of a three- dimensional object. Each line segment has three possible interpretations (Fig. 4a): (1) it may be a line. where two faces meet at a convex angle; (2) a line where two faces meet at a concave angle; (3) or an edge of a face which occludes the objects behind it. At each junction of lines only a few com- binations of interpretations are physically possible. This problem can be solved by a tree search algo- rithm. For this purpose we use a tree like the one in Fig. 4b. At each level of the tree, one of the lines is labelled as convex (+) , concave ( - ) or occluding (~.). Before assigning a label to a line at a certain level of the tree, the search algorithm should check whether this label is consistent with the labellings at the higher levels of the tree.

Related interpretation tasks in photogramme- try can be found in Schickler (1989), Braun (1992, 1994). Schickler tries to derive three-dimensional models of topographical control points (houses) by interpreting two-dimensional orthogonal sketches. Braun describes the inference of house shapes

from perspective views. Both reasoning problems have solution spaces that can be represented in search trees.

3. Tree search

Several search trees have been introduced above. In each tree one of the paths from the root node to a leaf node represents the optimal solution. The task of the tree search method is to systematically scan the search tree for this best solution. The common tree search methods which can be found in most textbooks on artificial intelli- gence (Barr and Feigenbaum, 1981; Nilsson, 1982; Pearl, 1984), differ in the order in which they scan the branches of the search tree. Since search trees tend to be very large, heuristic search strategies are used in addition to the standard search methods in order to reduce the average search times.

3.1. Search methods

In most search trees, cost or merit values are associated with the edges of the tree, e.g., gradient strengths or similarity values. These costs or merits can be used to guide the tree search algorithm to the best solution. Tree search methods that take advantage of a cost or merit function to optimize the search are called informed search methods.

Some trees, however, do not have costs that are associated with the edges of the tree. Thl: is the case in many geometric reasoning probiem~. These problems have to be solved by so-called blind search methods.

Both blind and informed methods start at the root node of the tree and try to find a path to a leaf node that represents the best solution to the problem. At each node, several tests have to be performed, e.g., to check whether a label has been used before, to check whether a label is con- sistent with prior labellings, or to check whether attributes of the unit and the label are compati- ble. If those tests are completed successfully, the successor nodes of the next lower level are gener- ated, i.e., the node is expanded. The order in which the nodes of a tree are expanded is different from search method to search method.

3.1.1. Blind methods Blind search metr, ods simply scan the search

tree Jn a systematic order. There are two basic con- cepts: depth-first search and breadth-first search

34 ISPRS Journal of Photogrammetry and Remote Sensing

(Barr and Feigenbaum, 1981; Nilsson, 1982). As al- ready indicated in its name, the depth-first search always first tries to move down to a node at the next lower level of the tree. Only when it encoun- ters a node which does not pass the tests, the search moves back to the previous level and is continued by moving down along other branches. The breadth-first search, on the contrary, only pro- gresses to the next lower level of the tree after all nodes at the current level have been explored.

3.1.2. Informed methods Blind search methods often take a long time to

find a solution. Whenever it is possible, it is useful to provide the search method with information by which it can select the path that is most likely to lead to a solution. This information is given in the form of costs or merits that are associated with the edges of the tree. The best solution is defined as the path from the root node to a leaf node which has the lowest sum of costs (or highest sum of merit values) that are associated with the edges of that path.

Three well-known informed methods are hill climbing, best first search and the A* algorithm. They differ in the amount of information that is used to select the node that should be expanded first (Barr and Feigenbaum, 1981; Nilsson, 1982; Pearl, 1984). The A* algorithm uses most infor- mation. It not only takes into account the costs of all edges that have been explored, but also uses an estimate of the costs that will be found along the edges that have not yet been explored. For each node that has been generated but not yet expanded it thus makes an estimate of the costs of the best path through that node. The A* algorithm then first expands the node for which the estimated costs of the path through that node are the lowest.

If the estimate for the costs of unexplored edges never is higher than the actual costs, then it is guaranteed that the path to the first leaf node that is found by the A* algorithm represents the best solution. In that case the search method and the estimate of the costs are called admissible.

3.2. Search strategies

The common problems that are represented in search trees have combinatorial or ~xponential complexity. This implies that tlae time needed to complete the search will grow very fast with the

number of levels in the tree. In order to reduce the search efforts several strategies have been devel- oped that can be applied to most of the presented search problems. The four strategies that will be discussed here are unit ordering, forward checking, relaxation of the admissibility, and beam search. The first two strategies do not affect the admissi- bility of the search method. However, if one of the latter two strategies is used, admissible methods like the best-first search and the A* algorithm do not necessarily find the best solution.

3. 2.1. Unit ordering Three of the four search problems in Sect. 2

have been described as problems of finding the correct correspondence between units and labels. One unit is labelled at each level of the search tree. The order in which the units are labelled will not affect the final result. We can therefore change this order such that the number of nodes in the tree is minimized. Since the search effort is proportional to the number of nodes, the search effort will also be minimized.

3.2.2. Forward checking Because of relationships between the units and

between the labels, it may occur that there are no valid labels for a unit at a low level of the tree due to a wrong labelling of a unit at a high level. This mistake remains unnoticed until the search has proceeded to that lower level. In the mean time many nodes in the subtree below the node with the wrong label have been examined. The forward-checking strategy (Haralick and El- liot, 1980) prevents this by first checking whether there is at least one valid label left for each of the units that have not yet been labelled before continuing the search at a lower level.

3.2.3. Relaxation of admissibility In order to ensure the admissibility of the A*

algorithm, the estimate of the costs of the best path in a subtree below a generated node (the so-called future costs) always has to be less or equal to the actual future costs. For many search problems it is difficult to define an estimator of the future costs that is both admissible and realistic.

There is, however, a variation to the admissi- bility criterion that is very useful. It says that if it is ensured that the future costs never are overesti- mated by more than a factor ¢, then, in the worst

Volume 50, number 4, 1995 35

case, the A* algorithm will select a path with costs that are only a factor e higher than the costs of the best path. This strategy is of great importance for search problems with large search trees which contain many near-optimal solutions that can be accepted.

3. 2. 4. Beam search Despite the above strategies, maw search prob-

lems are still too large to be solved efficiently. For the best first method and the A* algorithm, the memory requirements for the stack of the unex- panded nodes often grow beyond the available system resources. Beam search is a strategy that reduces the required stack space by exclusively focusing the search on promising nodes. In its sim- plest form the beam search strategy only keeps the best few child-nodes whenever a node is expanded. Only these generated nodes are kept. This strategy is clearly inadmissible in a sense that the path of the best solution may be pruned.

4. Tree search applied to photogrammetric problems

We will now revisit the photogrammetric prob- lems and discuss how they can be solved by the described tree search methods and strategies.

4.1. Object location and recognition

Consistent labelling problems like matching re- lational descriptions of images and object models often suffer from a very high computational com- plexity. These problems can only be solved in an acceptable time if the number of features is rel- atively small or if the relationships between the Frimitives effectively constrain the number of valid combinations of unit-label pairs.

The similarity between the attributes of the fea- tures and the relations has to be used to guide the search. Preferably the A* algorithm is used, since this method makes the best use of the information provided and can therefore prune many branches that will be explored by other methods.

The use of unit-ordering and forward-checking strategies certainly pays off, because they do not affect the admissibility of the search methed and have a low computational complexity. Whenever fairly good, but non-optimal solutions are accept- able, the use of a realistic but sometimes inad- missible (pessimistic) estimator of the future costs

should be preferred as it can reduce both searci~ time and memory requirements considerably (Vos- selman, 1992).

The search time also heavily depends on the number of segmentation errors. Due to these er- rors, there is no one-to-one mapping between the features of the image and the object model. Also, the topological relationships will not agree. These differences make it more difficult to find the correct correspondences. Unfortunately, segmen- tations of aerial photographs often contain many errors. In those cases the tree search may need a long time to locate topographical control points.

4.Z Stereo image matching

The remarks on the optimal search methods and strategies for solving object location problems also hold for stereo matching problems. The search trees of these problems are very large. They can only be solved when good constraints are available in the relationships between the features.

For several reasons image matching is often a little harder to solve than object location. In the first place, the object model that is compared with the image does not contain noise, whereas in the case of matching stereo images both image descriptions are perturbed by segmentation errors. Furthermore, the constraints that can be inferred from a known geometrical transformation between the datasets are less tight in the case of matching stereo images than in the case of object location.

4.3. Edge and line following

If the line-following algorithm only examines fairly short paths (e.g., 5 pixels), the search tree will remain relatively small and can be scanned completely with a simple search technique. For the examples shown in Fig. 5 the breadth-first search method was combined with a beam search strategy.

Although merits (gradients) were associated with the edges of the search tree, they were not used to optimize the search direction, i.e., a blind search method was used. The beam search strategy used the gradient sums of the paths in order to prune the search tree. Especially for grey value edges with sharp contrast, the beam search strat- egy hardly imposes any risks. The optimal path is almost certainly among the selected paths, it would have been possible to use the gradient sums

36 ISPRS Journal of Photogrammetry and Remote Sensing

', .

I! 'r-M"u s ,. .

0o, %@

c g'

% Q

O ¢ ®

O b

Figure 5. (a) Edge pixels selected by the lower threshold of the applied hysteresis thresholding. The contrast in this test image was 100 grey values. The image was perturbed by white Gaussian noise with a standard deviation of 20 grey values. (b) Lines found by the line.following algorithm. Short lines have been eliminated.

in an informed search method like the A* algo- rithm, but for small search problems this would be unnecessarily complicated.

The use of strategies other than beam search is not possible. Unit ordering is not allowed in this problem because only neighbouring pixels can be connected and node ordering would imply a change in the pixel sequences. Forward checking can only be used for labelling problems, and the relaxation of admissibility only applies to the A* algorithm.

4. 4. Geometric reasoning

In many geometric reasoning processes, like the Waltz' labelling of line drawings, one can only determine whether a certain decision or label was correct or not. There is no value that indicates to what extent a decision is correct. This implies

that the search spaces of those problems can only be scanned by blind methods. Therefore, strategies like unit ordering and forward checking are the only tools that can help to limit the search time. For the rest, the success of the reasoning process will heavily depend on the constraints that can be inferred from the knowledge about the geometry of the objects to be interpreted.

5. Conclusions

In this paper several photogrammetric prob- lems were converted into tree search problems. The usage of the common tree search methods and strategies to solve these problems was discussed. Although there certainly are other possible appli- cations of search trees in photogrammetry (e.g., in flight planning or planning of close-range record- ings), it is clear that, generally, the search trees are

Volume 50, number 4, 1995 37

very large. Therefore efficient search strategies are necessary to reduce the search times.

Segmentations of aerial images often contain many errors. Using relational matching to solve object location or image matching problems is cumbersome, because these errors significantly in- crease the required search effort. Applications in close-range photogrammetry may be more success- ful if the imagery is easier to segment.

Tree search can be applied to line following, but it is just one of many methods that can be used for that purpose. Reasoning processes, however, show properties (like comparing alternatives and searching by error and trial) for which tree search methods seem to be very suitable.

References

Ackermann, E and Krzystek, E, 1991. MATCH-T: automatic Mensuration of Digital Elevation Models. Paper pre- sented at the 3rd Technical Seminar of the Sociedad Espanola de Cartografia Fotogrametria y Teledeteccion, April 12, 1991.

Ballard, D.H. and Brown, C.M., 1982. Computer Vision. Prentice-Hall, Englewood Cliffs, N.J.

Barr, A. and Feigenbaum, E., 1981. The Handbook of Artifi- cial Intelligence. Kaufmann Inc., Los Altos, Calif.

Boyer, K.L. and Kak, A.C., 1988. Structural stereopsis for 3-D vision. IEEE Trans. Pattern Anal. Machine Intell., 10(2): 144-166.

Braun, C., 1992. Interpreting single images of polyhedra. Int. Arch. Photogramm. Remote Sensing, 29, part B'~: 514-521.

Braun, C., 1994. Interpretation and coirection of single line drawings for the reconstruction of objects in space. Int. Arch. Photogramm. Remote Sensing, 30, part 3/1: 85-90.

Haala, N. and Vosselman, G., 1992. Recognition of road a~ld river patterns by re|ational matching. Int. Arch. Pho- togramm. Remote Sensing, 29, pzrt B3: 969-975.

Haralick, R.M. and Elliot, G.L., 1980. Increasing tree search efficiency for constraint satisfaction problems. Artif. In- tell., 14: 263-313.

Hellwich, O., Heipke, C., Tang, L., Ebner, H. and Mayr, W., 1994. Experiences wi~h automatic relative orientation. Int. Arch. Photogramm. Remote Sensing, 30, part 3/1: 370- 378.

Nilsson, N.J., 1982. Principles of Artificial Intelligence. Tioga Publishing Company, Palo Alto, Calif.

Pearl, J., 1984. Heuristics: Intelligent Search Strategies for Computer Problem Solving. Addison-Wesley, Reading, Mass.

Schenk, T, Li, J.-C. and Toth, C., 1991. Towards an au- tonomous system for orienting digital stereopairs. Pho- togramm. Eng. Remote Sensing, 57: 1057-1064.

Schickler, W., 1989. Interpretation yon Hausskizzen. MSc The- sis, Uaiversity of Stuttgart, 1989.

Schickler, W., 1992. Feature matching for outer orientation of single images using 3-D wireframe controlpoints. Int. Arch. Photogramm. Remote Sensing, 29, part B3: 591- 598.

Shapiro, L.G. and Haralick, R.M., 1981. Structural descrip- tions and inexact matching. IEEE Trans. Pattern Anal. Machine Intell., 3: 504-5i9.

Tsingas, V., 1994. A Graph-theoretical approach for multiple feature matching and its application on digital point trans- fer. Int. Arch. Photogramm. Remote Sensing, 30, part 3/1: 865-871.

Vosselman, G., 1992. Relational Matching. Lecture Notes in Computer Scienec, Voi. 628. Springer Verlag, Heidelberg.

Waltz, D.L., 1975. Understanding drawings of scenes with shadows. In: P.H. Winston (Editor), The Psychology of Comput,~r Vision. McGraw-Hill, New York, pp. 19-91.

Wiesei, J., 1975. Digital image processing for ortho-photo generation. Photogrammetria 40: 69-76.