HARMONIC NETWORKS: DEEP TRANSLATION AND ROTATION EQUIVARIANCE

Daniel E. Worrall, Stephan J. Garbin, Daniyar Turmukhambetov, and Gabriel J. Brostow

OUR GROUP

OUTLINE

• Brief intro: Invariance and Equivariance

• What has been done for rotations?

• Harmonic Networks and the Fourier Shift Theorem

• Appendix: Further theory and future works

INVARIANCE

• Many tasks invariant to input transformations

Doggie DoggieDoggie

INVARIANCE

e.g. translation

Mapping independent of Tg, for all Tg

f(I) = f(Tg[I])Notational aside: (in our case)

Tg[I](x) = I(Mgx+ bg)

Tg[I](x) = I(x+ bg)

INVARIANCE WARNING: DIFFERENT NOTATION TO PAPER

INVARIANCE: EXAMPLES

Var[p] = Var[Tgp]Variance: translations

Fourier spectrum modulus: translations����Z 1

�1I(x)e�◆!x dx

���� =����Z 1

�1I(x� b)e�◆!x dx

����

Pooling: smooth invertible mappings (typ. diffeomorphisms)Z

x2⌦I(x) dx =

Z

x2⌦I(Tgx) dx max

x2⌦I(x) = max

x2⌦I(Tgx)

spectrum[M] = spectrum[QMQ>] QQ> = Q>Q = I

Eigenspectrum: orthogonal transformations

EQUIVARIANCE“The man swallowed the chicken”

versus “The chicken swallowed the man”

EQUIVARIANCE: GENERAL DEFINITION

Sg[f(I)] = f(Tg[I])Mapping preserves algebraic structure of transformation

Sg = Id

Invariance is special case of equivariance

Different representations of same transformation

EQUIVARIANCE: EXAMPLESFourier spectrum phase: translations

e

�◆!b

Z 1

�1I(x)e�◆!x dx =

Z 1

�1I(x� b)e�◆!x dx

Mean: translationsaEp[x] + b = Ep[ax+ b]

[f ⇤ g](x� b) = [Tb[f ] ⇤ g](x)Convolution (and correlation)

|◆!|nh(!) = d@n

x

h(!)

Fourier transform: differentiation (and pretty much any other linear transformation)

[w ? Tbf ](x) =X

y2X

w(y � x)Tbf(y)

=X

y2X

w(y � x)f(y � b)

=X

y02X

w(y0 + b� x)f(y0)

= [w ? f ](x� b)

TRANSLATION EQUIVARIANCE OF Z-CORRELATION

Boundaries imply local equivariance

Source: https://github.com/vdumoulin/conv_arithmetic

LACK OF ROTATION EQUIVARIANCE:

ROTATION EQUIVARIANCE METHODS

Rotate in network Rotate input

Warped Convolutions (Henriques et al., 2017) Polar Transformer Nets (Esteves et al., 2017)

Exploiting Cyclic Symmetry (Dieleman et al., 2016)

Rotate filters Rotate activations Log-polar trans.Brute-forceRotation-Invariant Neoperceptron (Fasel et al., 2006) TI-Pooling (Laptev et al., 2016)

Roto-Translation Scattering(Oyallon et al., 2014) Group Equivariant CNNs(Cohen et al., 2016) Oriented Response Networks (Zhou et al., 2017) Steerable CNNs (Cohen et al., 2017)

Store full response Store max/mean responseRoto-Translation Invariant Scattering(Sifre et al., 2013) Rotation Equivariant Vector Fields (Marcos et al., 2017)

ALL DISCRETE ROTATIONS

"Do you want the wrong answer to the right question or the right answer to the wrong

question? I think you want the former.”

DAVID BLEI

PART II: HARMONIC NETWORKS

THE FOURIER SHIFT THEOREM

Interestingly this generalises beyond 1D shifts to any transformation if considering groups

Equivariant Invariant

m 2 Z

Constraint on angular component of weights

Z ⇡

�⇡I(�� ✓)e�◆m� d� =

Z ⇡�✓

�⇡�✓I(�0)e�◆m(�0+✓) d�0

= e�◆m✓

Z ⇡

�⇡I(�0)e�◆m�0

d�0

CIRCULAR HARMONICS

Constrained

Example with Gaussian radial component

Live in orthogonal spaces, so don’t interactNote: the sign

flip in the angular

component

Free

Wm(r;�) = R(r)e◆(m�+�)

CIRCULAR HARMONIC WEIGHTS: IN PRACTICAL TERMS

Wm ? I = (WRem ? IRe �W Im

m ? IIm) + ◆(WRem ? IIm +W Im

m ? IRe)

Real Imaginary

CIRCULAR HARMONIC WEIGHTS

K K

CIRCULAR HARMONICS

X-correlation w. circular harmonics returns orientation field

Wm2 ? g([Wm1 ? T✓I])

CIRCULAR HARMONIC ALGEBRASo far we have a mapping

But what about: 1) nonlinearities? 2) subsequent layers?

W? : RM⇥N⇥K ! CM⇥N⇥1

NONLINEARITIESTo maintain linear equivariance apply nonlinearities to invariant magnitude only g(|F |e◆\F ) = g(|F |)e◆\F

�(|F |+ c)ReLU(|F |+ c) LeakyReLU(|F |+ c)

CIRCULAR HARMONIC ALGEBRA

?

CIRCULAR HARMONIC ALGEBRA

Sum orders along chained cross-correlations

Wm2 ? g([Wm1 ? T✓I]) = Y e◆(m1+m2)✓

CIRCULAR HARMONIC ALGEBRA

?

???

?

??

?

?

CIRCULAR HARMONIC ALGEBRAAt a layer, we sum over incoming feature maps, but need to pay special attention to rotation order

CIRCULAR HARMONIC ALGEBRAAt a layer, we sum over incoming feature maps, but need to pay special attention to rotation order

CIRCULAR HARMONIC ALGEBRA

The Equivariance Condition

X

p2Pi

mp = i

Path through i

Rotation order of filterConstant at

feature map i

My conjecture: For Lie Groups, streams identified with basis vectors of Lie Algebra, and harmonic filters act as raising/lowering operators, jumping between basis vectors

CIRCULAR HARMONIC ALGEBRA

CIRCULAR HARMONIC ALGEBRA

SOME EXPERIMENTS

20

20

20

20

20

20

CNN

8

8

16

16

35

35

H-Net

m=0

m=110

8

8

H-DSN

m=0 m=1

16

16

32

32

64

64

64

64

1

1

1

1

1

1

DSN

1

1

1

1

1

1

32

32

64

64

128

128

256

256

256

256

Brute force

Rotation + Scale

ORN-8 [30] 1.37 ~1MTI-Pooling [13] 1.21 ~1MPTN-CNN-B++ 0.89 ~254k

Recent results

SOME EXPERIMENTS

CNNCNN + data aug.H Net (ours)

0.880.900.920.940.960.981.00

2000 4000 6000 8000 10000 12000Training set sizeN

orm

aliz

ed te

st a

ccur

acy

SOME EXPERIMENTS

20

20

20

20

20

20

CNN

8

8

16

16

35

35

H-Net

m=0

m=110

8

8

H-DSN

m=0 m=1

16

16

32

32

64

64

64

64

1

1

1

1

1

1

DSN

1

1

1

1

1

1

32

32

64

64

128

128

256

256

256

256

Dynamic Steerable ResNet 0.732 0.751Recent results

SOME EXPERIMENTS

SOME EXPERIMENTS

SOME EXPERIMENTS

SOME EXPERIMENTS

LIMITATIONS

• Filters too basic

• No stability guarantee for noise/high freq. deformations

• Do we really need higher order streams?

• What about other kinds of transformation?

Working on all of these

REFERENCESRotate filters- Deep Roto-Translation Scattering for Object Classification (Oyallon et al., 2014) - Group Equivariant Convolutional Networks (Cohen et al., 2016) - Rotation Equivariant Vector Field Network (Marcos et al., 2017) - Oriented Response Networks (Zhou et al., 2017) - Steerable CNNs (Cohen et al., 2017) - Roto-Translation Invariant Scattering (Sifre et al., 2013)

Rotate activations

Rotate input- Rotation-Invariant Neoperceptron (Fasel et al., 2006) - TI-POOLING: Transformation-invariant Pooling for Feature Learning in Convolutional

Neural Networks (Laptev et al., 2016) - Warped Convolutions: Efficient Invariance to Spatial Transformations (Henriques et al.,

2017) - Polar Transformer Networks (Esteves et al., 2017)

- Exploiting Cyclic Symmetry in Convolutional Neural Networks (Dieleman et al., 2016)

PART III: THEORY

THEORY: THERE IS A LOT

Group theory

Lie groups

Steerable function spaces

Scale-space theory

Gaussian scale-space

Harmonic Analysis

H-NetsRepresentation theory

THE FOURIER SHIFT THEOREM ON LIE GROUPS

Theory is very involved, so this is just a brief outline

GROUP THEORYA group G=(G, ●) is a set G and a binary

composition operator ●, with

1. Closure

2. Associativity

3. Identity element

4. Unique inverses

(f • g) • h = f • (g • h)

g • h 2 G

g • e = e • g = g

g • g�1 = g�1 • g = e

GROUP THEORY: EXAMPLE

y · z 2 C \ {0}(x · y) · z = x · (y · z)

x · 1 = 1 · x = x

x

�1 =1

|x|e�◆\x

1. Closure 2. Associativity 3. Identity element 4. Unique inverses

(C \ {0}, ·)

2D ROTATIONS FORM A GROUPThese are some different ways to describe rotations

SO(2)

([0, 2⇡),+mod 2⇡)

(C, ·)C = {z 2 C : |z|}C = {z 2 C : |z| = 1}

�{A 2 M2⇥2 : A>A = AA> = I, |A| = 1}, ·

�

In fact this group has a manifold structure: it is a Lie Group (Don’t worry if none of this makes sense—I only understand it every 2nd day)

There are an infinite number of representations to describe essentially the same thing

PATCH IS FUNCTION ON SO(2)

FOURIER TRANSFORMS ON LIE GROUPS

PreviouslyZ

f(�)e�◆m� d� e�◆m�e�◆m✓ = e�◆m(�+✓)

m(✓) m(�) = m(✓ • �)Homomorphism property

NowZ

f(�) m(�) d�

(We lose commutativity)Z

GT✓[f ](�) m(�) d� =

Z

Gf(✓�1�) m(�) d�

=

Z

✓�1Gf(�) m(✓�) d(✓�)

= m(✓)

Z

Gf(�) m(�) d�

Need left-invariant Haar measure

InvariantEquivariant

Extension of Fourier Shift Theorem

INVARIANTS ON LIE GROUPSCan show that the irreducible representations are unitary so

kHk2 = hH|Hi = trace(H>H)

Invariant on group

In such a way we can proceed as before to define a harmonic network over a matrix Lie group

kFm{T✓f}k2 = k m(✓)Fm{f}k2

= h m(✓)Fm{f}| m(✓)Fm{f}i

= trace( ˜Fm{f}> ˜ m(✓)

> m(✓)| {z }

I

Fm{f})

= kFm{f}k2

kFm{T✓f}k2 = k m(✓)Fm{f}k2

= h m(✓)Fm{f}| m(✓)Fm{f}i

= trace( ˜Fm{f}> ˜ m(✓)

> m(✓)| {z }

I

Fm{f})

= kFm{f}k2

kFm{T✓f}k2 = k m(✓)Fm{f}k2

= h m(✓)Fm{f}| m(✓)Fm{f}i

= trace( ˜Fm{f}> ˜ m(✓)

> m(✓)| {z }

I

Fm{f})

= kFm{f}k2

kFm{T✓f}k2 = k m(✓)Fm{f}k2

= h m(✓)Fm{f}| m(✓)Fm{f}i

= trace( ˜Fm{f}> ˜ m(✓)

> m(✓)| {z }

I

Fm{f})

= kFm{f}k2

Fm{f} :=

Z

Gf(�) m(�) d�

WORK IN PROGRESSOpen questions:• How to jump between streams?

• Transformations such as scalings are non-compact, can we approximate them well on an interval?

• Not all transformations form groups. Harmonics are eigenfunctions of the Laplace-Beltrami operator.

• Could we define equivariance on general manifolds or even graphs? • What about eigenfunctions of other operators? c.f. Sturm-Liouville theory

• There are other equivariant transforms such as the Gaussian convolution (affine equivariant). This has a semi-group structure, how does this fit in?

THANK YOU