Objectives

Approach

Experiments

References

NYU Multimedia and Visual Computing Lab,

Fan Zhu, Jin Xie and Yi Fang

Heat Diffusion Long-Short Term Memory Learning for 3D Shape Analysis

We aim to develop a 3D shape representation by utilizing the heatflows on 3D surfaces and the corresponding temporal dynamicsof the heat flows within the diffusion period.We employ LSTM to capture the temporal dynamics of heatflows and extract joint information between different time-stepsthat are either consecutive or with a large interval.We incorporate a 3-layer fully-connected neural network withLSTM, and extend the proposed approach to a cross-domainscenario.

The heat kernel , is introduced to measure the amount ofheat that has been transformed from point to point on the 3Dshape surface at time , and it can be used to compute the HKS[1]of each vertex on the 3D shape surface at time :( , ) = ( , )

• HD-LSTM learns the heat kernel probability distribution overmultiple diffusion time-steps, so as to capture the temporaldynamics.

• HD-LSTM minimizes the reconstruction between discriminativevectors and the hidden unit outputs through time:

Discriminative random vectors are generated from ground-truthshape labels, and are used to guide HD-LSTM toward learningdiscriminative 3D shape representations.

We generalize HD-LSTM to learning multi-view data by connectingHD-LSTM to another neural network through discriminativevectors. Learning the 3-layer neural network can be achieved byminimizing the reconstruction error between at the target layer:

, , , || ||

, || , ( )|| + || ||McGill dataset (shape query)

SHREC14 dataset (sketch query)

[1] Sun, J., Ovsjanikov, M., Guibas, L.: A concise and provably informative multiscale signature based on heat diffusion, in Computer graphics forum. Vol. 28, pp 1383-1392, 2009[2] Xie, J., Fang, Y., Zhu, F., Wong, E.: Deepshape: Deep learned shape descriptor for 3D shape matching and retrieval, in Computer Vision and Patter Recognition, 2015

Source code: https://github.com/blacksmithfan/HD_LSTM

NYU Tandon School of Engineering, New York, USNew York University Abu Dhabi, UAE