A novel approach for image sweeping functions using

approximating scheme

Pierre-Emmanuel Leni, Yohan D. Fougerolle, and Frederic Truchetet

<pierre-emmanuel.leni, yohan.fougerolle, frederic.truchetet>@u-bourgogne.fr

Universite de Bourgogne, Laboratoire LE2I, UMR CNRS 5158,

12 rue de la fonderie, 71200 Le Creusot, France

ABSTRACT

Using Kolmogorov’s superposition theorem, complex N -dimensional signal can be expressed by simpler 1Dfunctions. Precisely, Kolmogorov has demonstrated that any multivariate function can be decomposed intosums and compositions of monovariate functions, that are called inner and external functions. We present oneof the most recent method of monovariate functions construction. The algorithm proposed by Igelnik in [1]approximates the monovariate functions. Different layers are constructed and superposed. A layer is constitutedby a couple of internal and external functions, that realizes an approximation of the multivariate function witha given precision, which corresponds to a representation of the multidimensional function through a tilage. Eachlayer contains new pieces of information, which improves the whole network accuracy. A weight is associated toeach layer. The network is trained, i.e., the monovariate functions and the weights associated to each layer areoptimized to ensure the convergence of the network to the decomposed multivariate function.

Sprecher has demonstrated in [2] that using internal monovariate functions, scanning functions can be con-structed; i.e., a space filling curve connects every couple of the multidimensional space and uniquely matchescorresponding values into [0, 1]. Igelnik’s construction produces a space filling curve per network layer: a uniquepath through the tiles of a layer.

The contributions of this paper are the presentation and the analysis of an approximating scheme for images.Starting from an image traditionally represented as rows and columns of pixels, we extract two kinds of mono-variate functions, one representing the scanning path through the image, and the second one being the core forthe image reconstruction using Kolmogorov Superposition Theorem. We have applied the algorithm on imagesand presents the decomposition results as composition of monovariate functions. We also present compressionresults, taking advantage of the continuity of monovariate functions.

Keywords: Kolmogorov superposition theorem, multidimensional function decomposition, neural network, sig-nal processing, image analysis

1. INTRODUCTION

In 1900, Hilbert has conjectured 23 mathematical problems. Amongst them, the 13rd states that high orderequations cannot be solved by sums and compositions of bivariate functions. 57 years later, Kolmogorov demon-strated his superposition theorem (KST) and proved the existence of monovariate functions, such that everymultivariate functions can be expressed as sums and compositions of monovariate functions. Unfortunately,Kolmogorov did not propose any construction method of these monovariate functions.

The superposition theorem, reformulated and simplified by Sprecher in [3], can be written as:

Theorem 1.1 (Kolmogorov superposition theorem). Every continuous function defined on the identityhypercube ([0, 1]d noted Id) f : Id −→ R can be written as sums and compositions of continuous monovariatefunctions:

f(x1, ..., xd) =∑2d

n=0 gn

(

ξ(x1 + na, ..., xd + na))

ξ(x1 + na, ..., xd + na) =∑d

i=1 λiψ(xi + an),

(1)

with ψ continuous function, λi and a constants. ψ is called inner function and g(ξ) external function. Theinner function ψ associates every component xi from the real vector (x1, ..., xd) of Id to a value in [0, 1]. The

function ξ associates each vector (x1, ..., xd) ∈ Id to a number yn from the interval [0, 1]. These numbers yn arethe arguments of functions gn, that are summed to obtain the function f .

There are two different parts in this decomposition: components xi, i ∈ J1, dK of each dimension are combinedinto a real number by a hash function (the inner function ξ), that is associated to corresponding value of ffor these coordinates by the external function g. Hecht-Nielsen has shown in [4] that the KST-constitutingmonovariate functions can be organized has a one hidden layer neural network.

Figure 1. Representation of the KST as a one hidden layer neural network, from [5].

Most recent contributions related to KST broach the problem of the monovariate function construction.Sprecher has proposed an algorithm for exact monovariate function reconstruction in [6] and [5], that introducesfundamental notions (such as tilage). Igelnik has presented in [1] an approximating construction that offersflexibility and modification perspectives over the monovariate function construction.

The structure of the paper is as follows: we present Igelnik’s algorithm in section 2. In section 3, we present theresults of Igelnik’s algorithm applied to the images, detailing monovariate functions, and compression possibilities.In the last section, we conclude and consider several research perspectives.

Our contributions are a synthetic explanation of Igelnik’s algorithm and its application to gray level images.Precisely, we present the monovariate decomposition of a gray level images, that can be split into two parts: ascanning function of the image; and monovariate, image-dependent functions.

2. IGELNIK’S ALGORITHM

Igelnik’s algorithm provides flexibility at the expense of reconstruction error: the monovariate functions areapproximated. The original equation 1.1 has been adapted to:

f(x1, ..., xd) ≃

N∑

n=1

angn

( d∑

i=1

λiψni(xi)

)

(2)

For a given layer n, d inner functions ψni are randomly generated: one per dimension (index i) and per layer(index n), independently from function f . The convex combination of these internal functions ψni with real valuesλi is the argument of external function gn, choosing real numbers λi (one per dimension) linearly independent,

strictly positive and such that∑d

i=1 λi 6 1. Finally, external functions gn are constructed. To conclude layerconstruction, the functions ψ and g are sampled with M points, that are interpolated by cubic splines. Eachlayer is weighted by a coefficients an and summed to approximate the multivariate function f .

The tilage is constituted with hypercubes Cn obtained by cartesian product of the intervals In(j), defined asfollows:

Definition 2.1.

∀n ∈ J1, NK, j > −1, In(j) = [(n− 1)δ + (N + 1)jδ, (n− 1)δ + (N + 1)jδ +Nδ],

where δ is the distance between two intervals I of length Nδ, such that the function f oscillation is smaller than1N

on each hypercube C. Values of j are defined such that the previously generated intervals In(j) intersect theinterval [0, 1]. Figure 2 illustrates such a construction of intervals I.

(a) (b) (c)

Figure 2. (a) Intervals I1(0) and I1(1) for N = 4. (b) Disjoint tilage generated by cartesian product of intervals In. (c)Sideview of superposition of tilage layer.

2.1 Inner functions construction

Each function ψni is defined as follows: generate a set of j distinct numbers ynij , between ∆ and 1−∆, 0 < ∆ < 1,such that the oscillations of the interpolating cubic spline of ψ values on the interval δ is lower than ∆. j isgiven by definition 2.1. The real numbers ynij are sorted, i.e.: ynij < ynij+1. The image of the interval In(j)by function ψ is ynij . This discontinuous inner function ψ is sampled by M points, that are interpolated by acubic spline. We obtain two sets of points: points located on plateaus over intervals In(j), and points M ′ locatedbetween two intervals In(j) and In(j+1), that are randomly placed. Points M ′ are optimized during the neuralnetwork construction, using a stochastic approach. Figure 3 represents final function ψ on the interval [0, 1].

Once functions ψni are constructed, the function ξn(x) =∑d

i=1 λiψni(x) can be evaluated. On hypercubes

Cnij1,...,jd, the function ξ has constant values pnj1,...,jd

=∑d

i=1 λiyniji. Every random number yniji

generatedverifies that the generated values pniji

are all different, ∀i ∈ J1, dK, ∀n ∈ J1, NK, ∀j ∈ N, j > −1.

Figure 3. Example of function ψ sampled by 500 points that are interpolated by a cubic spline.

2.2 External function construction

The function gn is defined as follows:� For every real number t = pn,j1,...,jd, function gn(t) is equal to the N th of values of the function f at the

center of the hypercube Cnij1,...,jd, noted bn,j1,...,jd

, i.e.: gn(pn,j1,...,jd) = 1

Nbn,j1,...,jd

.� The definition interval of function gn is extended to all t ∈ [0, 1]. Consider A(tA, gn(tA)) and D(tD, gn(tD))two adjacent points, where tA and tD are two levels pn,j1,...,jd

. Two points B et C are placed in A and D

neighborhood, respectively. PointsB and C are connected with a line defined with a slope r = gn(tC)−gn(tB)tC−tB

.PointsA(tA, gn(tA)) andB(tB , gn(tB)) are connected with a nine degree spline s, such that: s(tA) = gn(tA),s(tB) = gn(tB), s′(tB) = r, and s(2)(tB) = s(3)(tB) = s(4)(tB) = 0. Points C and D are connected with asimilar nine degree spline. The connection condition at points A and D of both nine degree splines givesthe remaining conditions. Figure 4(a) illustrates this construction, whereas (b) gives a complete overviewof the function gn for a layer.

Remark 1. Points A and D (values of function f at the centers of the hypercubes) are not regularly spaced onthe interval [0, 1], since their abscissas are given by function ξ, and depend on random values ynij ∈ [0, 1]. Theplacement of points B and C in the circles centered in A and D must preserve the order of points: A,B,C,D,i.e. the radius of these circles must be smaller than half of the length between the two points A and D.

(a) (b)

Figure 4. (a) From [1], plot of gn. Points A and D are obtained with function ξ and function f . (b) Example of functiongn for a complete layer of lena decomposition.

See figure 5 for an overview of the monovariate functions organization.

Figure 5. Overview of a 5 tilage layers network.

2.3 Neural network stochastic construction

Igelnik defines parameters during construction that are optimized using a stochastic method (ensemble approach):the weights an associated to each layer, and the placement of the sampling points M ′ of inner functions ψ thatare located between two consecutive intervals.To evaluate the network convergence, three sets of points are constituted: a training set DT , a generalization setDG, and a validation set DV .N layers are successively built. To add a new layer, K candidate layers are generated with the same plateausynij , which gives K new candidate neural networks. The difference between two candidate layers is the set ofsampling points located between two intervals In(j) and In(j+ 1), that are randomly chosen. We keep the layerfrom the network with the smallest mean squared error that is evaluated using the generalization set DG. Theweights an are obtained by minimizing the difference between the approximation given by the neural networkand the image of function f for the points of the training set DT . The algorithm is iterated until N layers areconstructed. The validation error of the final neural network is determined using validation set DV .

To determine coefficients an, the difference between f and its approximation f must be minimized:

‖Qnan − t‖ , noting t =

f(x1,1, ..., xd,1)...

f(x1,P , ..., xd,P )

, (3)

with Qn a matrix of column vectors qk, k ∈ J0, nK that corresponds to the approximation (f) of the kth layer forpoints set

(

(x1,1, ..., xd,1), ..., (x1,P , ..., xd,P ))

of DT :

Qn = [q0, q1, ..., qn], with ∀k ∈ [0, ..., n], qk =

fk(x1,1, ...xd,1)...

fk(x1,P , ...xd,P )

.

An evaluation of the solution Q−1n t = an is proposed by Igelnik in [7]. The coefficient al of the column vector

(a0, ..., an)T is the weight associated to layer l, l ∈ J0, nK.

3. IMAGE APPROXIMATION

The external function has a noisy shape, which is related to the global sweeping scheme of the image: Sprecherand al. have demonstrated in [2] that using internal functions, space-filling curves can be defined. Function ξ

associates a unique real value to every position of the multidimensional space [0, 1]d. Sorting these real valuesdefines a unique path through the tiles of a layer: the space filling curve.We can generate space filling curves for Igelnik internal functions as well. We obtain a different curve for eachlayer (a new function ξ is defined for each layer). Moreover, each function ξ has constant values over tilage blocks,which introduces indetermination in space filling curves. Figure 6 presents an example of internal function ξ andthe associated space filling.

(a) (b)

Figure 6. (a) Internal function ξ, and associated space filling curve (b).

Figure 7 presents three scanning functions, using different constants λ1, λ2: (a) is obtained using almost equalweights for every dimension (λ1 ≃ 0.5 and λ2 = 0.4); figure 7(b) is obtained with λ1 ≃ 0.95 and λ2 = 0.04,and figure 7(c) is obtained with λ1 ≃ 0.95 and λ2 = 0.00004, which tends to the classical line-per-line imagesweeping. (d)(e)(f) are the associated external functions: λi not only affect the space filling curves, but also theexternal functions spatial repartition.

(a) (b) (c)

(d) (e) (f)

Figure 7. Space filling curves: (a) with λ1 ≃ 0.95 and λ2 = 0.04, and (b) with λ1 ≃ 0.95 and λ2 = 0.00004. (c)(d)Associated external functions.

Constants λi allow control on the global sweeping scheme, but the control on space filling curves for specificimage areas remains an open problem. The construction of inner functions ψ is independent of the image: theplateaus are randomly generated. Developing adapted space filling curves implies that ynij have to be imagedependent.

In Igelnik’s algorithm, tiles size can be adjusted. Figure 8 presents gray level image reconstructions, usingdifferent tilage densities, and the corresponding scanning functions. One can remark that the path through pixelscovered by the same tile is irrelevant, since they have the same image through function ξ.

(a) (b)

(c) (d)

Figure 8. One pixel per tile: (a) space filling curve, and reconstruction (c). 25 pixels per tile: (b) space filling curve, andreconstruction (d).

For lossy compression purposes, we present in table 1 and figure 9 the reconstruction error (PSNR(dB)) as afunction of the number of pixels utilized to define external functions.

Pixels utilized in external functions PSNR(dB)100% 48.985% 27.770% 26.655% 25.640% 24.425% 22.615% 21.3

Table 1. PSNR of image reconstruction.

Figure 9. PSNR of image reconstruction as a function of the number of pixels utilized to define external functions.

Another compression is achieved taking advantage of monovariate functions continuity: the reconstructionimage can be of any number of pixels. Figure 10 illustrates the reconstruction of lena in several resolutions,using the same original network containing 100% of 100x100 pixels lena. The reconstruction PSNR is 17.04dBfor the 200x200 pixels lena, and 15.37dB for the 400x400 pixels lena.

(a) (b) (c)

Figure 10. 100x100 pixels(a), 200x200 pixels(b), 400x400 pixels(c) reconstructions with network computed with a 100x100pixels lena.

4. CONCLUSION AND PERSPECTIVES

We have dealt with the problem of multivariate function decompositions into monovariate functions. Thereare two types of monovariate functions: internal functions for every dimension, that are linearly combined todefine a hash-function. This hash-function references every position of the multidimensional space as a realvalue. These real values are the argument of external functions. To approximate monovariate functions, wehave presented Igelnik’s algorithm, and applied it to gray level image decomposition. We have presented severalcompression capabilities of such a decomposition: using different tile sizes, the number of pixels from the originalimage contained in external functions can be adjusted. Using monovariate functions continuity, images of higherresolution than the original image can be reconstructed.

Internal functions also define space filling curves, i.e. image sweeping schemes. We have shown that theglobal sweeping scheme can be globally controlled, and presented different image scanning. The question of thelocal control of space filling curves remains open. This issue is related to the information repartition betweeninternal and external functions: every information is contained in external functions, since internal functions arerandomly constructed. How this repartition can be controlled? Can we describe an image using only externalor internal functions? The image representation as monovariate continuous functions could offer new imageprocessing methods: monovariate functions can be processed using 1D signal processing methods, and then, theimage can be reconstructed with processed monovariate functions.

REFERENCES

[1] Igelnik, B. and Parikh, N., “Kolmogorov’s spline network,” IEEE transactions on neural networks 14(4),725–733 (2003).

[2] Sprecher, D. A. and Draghici, S., “Space-filling curves and Kolmogorov superposition-based neural net-works,” Neural Networks 15(1), 57–67 (2002).

[3] Sprecher, D. A., “An improvement in the superposition theorem of Kolmogorov,” Journal of MathematicalAnalysis and Applications 38, 208–213 (1972).

[4] Hecht-Nielsen, R., “Kolmogorov’s mapping neural network existence theorem,” Proceedings of the IEEEInternational Conference on Neural Networks III, New York , 11–13 (1987).

[5] Sprecher, D. A., “A numerical implementation of Kolmogorov’s superpositions ii,” Neural Networks 10(3),447–457 (1997).

[6] Sprecher, D. A., “A numerical implementation of Kolmogorov’s superpositions,” Neural Networks 9(5),765–772 (1996).

[7] Igelnik, B., Pao, Y.-H., LeClair, S. R., and Shen, C. Y., “The ensemble approach to neural-network learningand generalization,” IEEE Transactions on Neural Networks 10, 19–30 (1999).

[8] Brattka, V., “Du 13-ieme probleme de Hilbert a la theorie des reseaux de neurones : aspects constructifs dutheoreme de superposition de Kolmogorov,” L’heritage de Kolmogorov en mathematiques. Editions Belin,Paris. , 241–268 (2004).

[9] Braun, J. and Griebel, M., “On a constructive proof of Kolmogorov’s superposition theorem,” Constructiveapproximation (2007).

[10] Igelnik, B., Tabib-Azar, M., and LeClair, S. R., “A net with complex weights,” IEEE Transactions onNeural Networks 12, 236–249 (2001).

[11] Koppen, M., “On the training of a Kolmogorov Network,” Lecture Notes in Computer Science, SpringerBerlin 2415, 140 (2002).

[12] Lagunas, M. A., Perez-Neira, A., Najar, M., and Pages, A., “The Kolmogorov Signal Processor,” LectureNotes in Computer Science, Springer Berlin 686, 494–512 (1993).

[13] Moon, B., “An explicit solution for the cubic spline interpolation for functions of a single variable,” AppliedMathematics and Computation 117, 251–255 (2001).