Ugo Montanari On the optimal approximation of descrete functions with low- dimentional tables

Ugo Montanari

On the optimal approximation of descrete functions with low-

dimentional tables.

Overview

Introduction Approximation with a sum of low-dimensional

functions Optomal approximation with a given interaction

graph Optimal approximation with a fixed amount of

memory

Introduction

Problem of storing large high-dimentional arrays is often critical (dynamic programming optimization techniques, belief propagation etc.)

Montanari proposes methos of optimal approximation (in the least square sence) of the given function with a sum of lower-dimentional functions.

Advantages

The decoding process is very simple (a fixed number of summations)

The compression ratio often is high, mean error can be small, if the interaction between separated variables is limited

The approximation function has a form of a sum of terms and is therefore suitable for the dynamic programming optimization

Approximation with a sum of low-dimentional functions

F – function of n discrete variables (with same domains)

Lettice

Approximation with a sum of low-dimentional functions

Average projection of the function:

Proper function:

A function g(Xi) such that its average projections on all

the subsets of Xi are identically zero will be called a

proper function of Xi.

Theorem 2.1

The set Si of all the proper functions of X

i is a vector space and

is called the proper space of Xi.

Theorem:

The proper space Si of all the elements X

i of lattice L are mutually

orthogonal.

Proof:

Characteristic function B

B:L -> 0,1 Monotonocity constraint:

The meaning of the characteristic function B is to specify the form of an approximate sum of terms for function F

Example of characteristic function

We want to approximate function F(x1,x

2,x

3) with

a sum of the form F = f1(x

1,x

2)+f

2(x

2,x

3)

Function B:

Characteristic space

SB:

Problem A

Algorithm A that solves Problem A

Step 1. Compute the average projections of F on all elements Xi

of lattice L.

Step 2. Let

Step 3. Execute next step for all r = 1,...,n

Step 4. For all elements Xi of L having cardinality r, let

where the summation is extended to all Xj of L smaller then X

i

Compute function:

Theorem 2.2

Theorem 2.2 proves validity of Algorithm A

Proof of theorem 2.2

(a) For every we have:

(b) and (c). We assume inductively the thesis is true for function k

j(X

j) and spaces S

j with cardinality X

j smaller then r, and prove

for r.

Proof of theorem 2.2 cont'd

Prove that :

If , then

From written as

Proof of theorem 2.2 cont'd

is proved to be solution to Problem A

Optimal approximation with a given interaction graph

Sum of terms:

Interaction graph: Alternative sum of terms:

Problem B

Given function F and interaction graph G find the sum such that G is the interaction graph of and the error |F- | is minimal

Note: interaction graph does not define uniquely the form of approximating function, so Problem B is not trivially reducibleto Problem A

Theorem 3.1

Theorem proves that the form of the optimal approximating sum depends only on given interaction graph G, and not upon the actual values of F.

Theorem:

Characteristic function B of an optimal approximating sum is computable as follows. We have B(X

i) = 1 iff the

set of vertices Wi corresponding to the set of variables

Xi defines a complete subgraph of G.

Proof of theorem 3.1

Example

Problem B reduces to:

- Finding all complete subgraphs of graph G

- Solving problem A

Optimal approximation with a fixed amount of memory

Problem C:

Given function F find a sum whose terms can be stored as tables in no more than M cells of memory and such that the error |F- | is minimal

2 ways of storing a sum

1) The sum

reqires 2N^2 cells

2) Store 6 functions from the table, such that

if any of arguments of f1 : f

5 is zero, then the value of function is

zero and it's not stored. Total storage space:

2 ways of storing a sum

In general, first methos requires

cells, where summation is extended to all maximal sets. Second method requires

Cells, but the summation extends to all sets and is optimal, because it is exactly equal to

the number of dimentions of vector space

2 ways of storing a sum

Error

By definition,

Thus

Translation of Problem C into integer programming problem (0,1) resticted

Problem D:

Determine the integer variables yi (i=1,...,m) (0,1)

restricted such that

with the constraints

Correnpondence between Problem C and Problem D

In Problem D both the objective function and the constraints are linear. Therefore linear interger programming methods apply.