Choreographer-mathematician collaborationdeveloping machine segmentation techniques for motion capture

analysis of dance

Associate Professor Kim VincsDr Wai Kuan Yip

Dr Vicky MakMs Kim Barbour

Motion capture & dance analysis

Motion capture offers precision and true 3D tracking of dance movement

Recent development over the last 10 years

Relatively few motion capture labs with dance focus

A new gold standard for quantitative dance analysis?

Challenges in motion capture dance analysis

Analysis needs to be culturally and artistically relevant (deLahunta)

Pattern recognition in dance crosses multiple frames of reference both between and within genres (?ref)

Approach to capture protocol, marker set design and feature extraction need to be appropriate to the needs of end-users, i.e. dance artists (Norman)

‘Capturing Dance’ project at Deakin Motion.Lab

Aims to isolate features of motion capture data that are artistically useful to choreographers

Features may be different for different dance genres and different dance artists

Will work with artists across 3 dance genres; contemporary dance, ballet and Australian indigenous dance over 3 years

Project team

Associate Professor Kim VincsDr Vick Mak

Associate Professor Richard SmithDr Wai Kuan YipMs Kim BarbourMr Daniel SkovliMr Peter Driver

Ms Lisa BolteMs Carlee Mellow

Ms Phoebe RobinsonMs Mee Young Yuk

This paper

Analysis of a single, complex contemporary dance phrase

20 repeats of the phrase by a single dancer

64 marker-set adapted to capture spine-arm-foot relationships important to dance

Approaches to segmentation into dance-meaningful chunks for analysis

Necessity of choreographer-mathematician collaboration in developing appropriate analysis techniques

Pattern recognition process

Segmentation

Manual segmentation (human/expert)

Unsupervised machine segmentation based on kinematic assumptions

Supervised machine learning

Unsupervised machine learning

Challenges in segmenting dance

Dance phrases tend to overlap

Different body segments need different levels of smoothing

Both large and small movements may be artistically significant

Different combinations of body segments may initiate new movements

Dance artists themselves don’t agree on how phrases are segmented (deLahunta 2005)

Some previous approaches to segmenting dance movement

Inter-limb correlation (Nakata 2007)

Laban Movement Analysis, (Bouchard & Badler 2007)

Minimum velocity (Hachimura & Nakamura 2001)

Triple minima; force, kinetic energy & momentum (Kahol, Tripathi & Panchanathan 2004)

Preliminary data – manual segmentation of a ‘tendu’

Expert segmentation problematic as depends on weighting conflicting factors

Surprising variability within samples (5 samples, elite ballet dancer)

Highlights difference between dancers’ conceptual map of the movement and the detailed ‘motion capture view’

Cortex file – tendu segment

Comparison of segmentation methods

Our approach

Primary question is what segmentation is artistically meaningful?

We used a collaborative, practice-based approach to develop segmentation ‘schema’ for the phrase

Choreographer, 2 dancers and 2 mathematicians developed definition of segments based on kinematics

Aimed to create an automated system based on the artistic schema

Problems identified

Dancers used multiple conceptual frameworks, e.g. velocities, heights, correlations of body parts

Dancers’ framework is ‘procedural’ – segment definitions only make sense in relation to the preceding movement

Segmentation is phrase-specific and cannot be generalized

Our schema

Segmentation steps

Conversion from positional information to hierarchical (parent-child) translation and rotation format to make analysis scale & position invariant

Smoothing to remove noise – Butterworth 6Hz, moving average 20-50 window, Gaussian smoothing

Segmentation steps

Polygonal approximation to estimate gradients of consecutive points to find local minima

3 possible scenarios;Negative gradient positive gradient Zero-gradient positive gradient

Positive gradient zero gradient

Examples of polygonal approximation

Important to select the right threshold to estimate local minimaAbove: Maximum tolerable threshold 5*SD (just nice)Above Right: 15*SD (too loose)Right: 30*SD (way too loose)

Local minimum segmentationRoot location

Right elbow location

Absolute rotational velocity

Absolute rotational velocity

Local minimum segmentation

Local minimum segmentationExample of compliant sample (#11)

Example of non-compliant sample (#2)

Local minimum segmentationExample of compliant sample (#17)

Example of non-compliant sample (#12)

Accuracy

Comparison of methods

Insert cortex file of Carlee Bitter

Summary

Method works very well for movements that rely on body-shape change, but breaks down for large loco-motor and turning movements

Large variation in ‘dancer compliance’

Variation or ‘dancer compliance’ does not correlate with dancer/choreographer ratings – variation in inherent in the style

Conclusions

Dancer-mathematician collaboration useful for identifying recognition problems that need to be solved

Dance recognition process needs to be able to deal with a high level of intra-performer variability

Different types of movement may require different approaches to segmentation

Further techniques for optimizing machine segmentation (eliminating false segmentation points) needed to supplement dance-generated phrase descriptions

ReferencesBouchard, D & Badler, N 2007, 'Semantic Segmentation of Motion Capture Using Laban Movement Analysis', in C. Pelachaud et al. (ed.), Intelligent Virtual Agents, Springer-Verlag, Berlin Heidelberg, vol. 4722, pp. 37-44.

DeLahunta, S & Barnard, P 2005, 'What's in a Phrase?', in J Birringer & J Fenger (eds), Tanz im Kopf: Jarbuch 15 der Gesellschaft für Tanzforschung, LIT Verlag, Münster.

Hachimura, K & Nakamura, M 2001, 'Method of generating coded description of human body motion from motion-captured data', Robot and Human Interactive Communication, 2001. Proceedings. 10th IEEE International Workshop on, pp. 122-7.

Kahol, K, Tripathi, P & Panchanathan, S 2004, 'Automated gesture segmentation from dance sequences', Proceedings, Sixth IEEE International Conference on Automatic Face and Gesture Recognition, 17-19 May pp. 883-8.

Nakata 2007, 'Temporal Segmentation and Recognition of Body Motion Data Based on Inter-Limb Correlation Analysis', Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2007), Oct. 29 2007-Nov. 2 2007 pp. 1383 - 8

Norman, SJ 2006, 'Generic Versus Idiosycratic Expression in Live Performance Using Digital Tools', Performance Research, vol. 11, no. 4, pp. 23-9.