Understanding RBM

Wang Yuantao

08/22/2009

2

Outline

• RBM Model– For Netflix Prize Problem

• RBM Algorithm– Implementation– Technical detail

• Contribution– Model– Result

3

Taxonomy

• By Determinacy – Deterministic: SVD/kNN– Stochastic: RBM

• By Optimisation– Empirical: kNN– Optimal

• Gradient descent: SVD• Maximum likelihood: RBM

4

Model Structure

By Hinton

5

Model Assumption

• 1

• 2

6

Optimal Object

Maximize:

7

Training

Sampling

8

Training Phases

1. Init v0 by real rating

2. Sample h0 by v0

3. Reconstruct v1 by h0

4. Re-sample h1 by v1

5. Update W

6. Compute Eh

9

Prediction

10

Technical detail

• Sparseness• Sample

– Gibbs 1-step sampling

• Update– Batch v.s. online(per-case)– Training bias

• Learning rate– Weight Decay– Momentum– Anneal

11

Temporal RBM

F=16, lrate=0.002

Original RBM 0.9392

0.9385

0.9381

0.9373

12

Contribution

0.8694

0.8688

RBM

13

Thanks

• Contact–王元涛– ytwang@freewheel.tv– 13521106828

• Q&A