Upload
gus
View
52
Download
0
Tags:
Embed Size (px)
DESCRIPTION
BACS Review Meeting. FCT-UC Jorge Dias 17 th – 19 th March 2008 Collège de France, Paris. FCT-UC Key Role within BACS for M13 – M24. Contributions in WP2: Bayesian models for sensor fusion (in collaboration with College de France). WP5.4: - PowerPoint PPT Presentation
Citation preview
BACSReview Meeting
FCT-UC
Jorge Dias
17th – 19th March 2008Collège de France, Paris
2
FCT-UC Key Role within BACS for M13 – M24
• Contributions in WP2: Bayesian models for sensor fusion (in collaboration with
College de France).
WP5.4:• Computational Laban Movement Analysis (LMA) using the
Bayesian framework for Task 5.4.2 (Vision-based detection and reconstruction of human actions ( MPS))
• Task 5.4.3: Multi-modal sensor integration including ego-motion
WP6: D6.2: Robotic Implementation of Gaze Control and Image Stabilization
WP4 WP7 WP8
WP5 WP6
WP1 WP2 WP3
Dis
se
min
ati
on
an
d
Co
mm
un
ity
In
teg
rati
on
Bayesian approach to complex systems
Study of Artificial Cognitive Systems
Study of Living Systems
Real World Applications
Neural implementation of Bayesian computation
Bayesian model of systems level behaviors
Infe
ren
ce
an
d L
ear
nin
g i
n
co
mp
lex
Bay
es
ian
sy
ste
ms
To AllFrom
All
3
FCT-UC Overview: Effort & Infrastructure
• List of people involved in BACS (changes M1-12 to M13-24) Paid by BACS
• Joerg Rett PhD Student 1,32• Filipe Ferreira PhD Student 6,6• José Prado PhD Student 7,79• Amilcar Pedrosa Technical Support 3,04• Hugo Faria Techinal Support 2,66• Luis Santos PhD Student 1,1• Alberto Neves PhD Student 4,95• Cátia Pinho PhD Student 4,4• Hadi Aliakbarpour PhD Student 2,64• Total 34,5
Own Resources• Jorge Dias Professor 2,4• Jorge Lobo Professor 2,4• J Filipe Ferreira PhD student 2,4• Luis Mirizola PhD student 2,4• Diego Faria PhD student 2,4• Total 12
• Infrastructure involved in BACS (M13 – M24) IMPEP Nicole platform for gesture recognition
5
WP 5, Task 5.4.2 Goal
Computational Laban Movement Analysis (LMA) using the Bayesian framework
Problem:
The research field of computational Human Movement Analysis is lacking a general underlying modeling language.
How to map the features into symbols?
How to model human behavior?
Solution:
A semantic descriptor allowing to recognize a sequence of symbols taken from an alphabet consisting of motion-entities.
Benefit:
Establish a set of labels for observable human behavior.
The possibility to build large databases with labeled training data.
6
WP 5, Task 5.4.2 Goal
Computational Laban Movement Analysis (LMA) using the Bayesian framework
Laban Movement Analysis:
Model for human behaviour
Bayesian Model:
Probabilistic model to analyse
human interaction
7
WP 5, Task 5.4.3 Goal
Multi-modal sensor integration including ego-motion
Biological Perception,Bayesian Model
Artificial Perception,Bayesian Model
Artificial Observer
SensorReadings
Artificial Perception
Human/BiologicalObserver
Perception Psychophysical Study Model Analysis
Artificial & Biological
Model Output Comparison
Ego-MotionIllusions, Conflicts & Ambiguities
Model Re-
evaluation
Model Re-
evaluation
Model
SynthesisM
odel
Synthesis
Sensation
3D Scene
& Moving Objectsw/ Static Objects
Biological Perception,Bayesian Model
Artificial Perception,Bayesian Model
Artificial Observer
SensorReadings
Artificial Perception
Human/BiologicalObserver
Perception Psychophysical Study Model Analysis
Artificial & Biological
Model Output Comparison
Ego-MotionIllusions, Conflicts & Ambiguities
Model Re-
evaluation
Model Re-
evaluation
Model Re-
evaluation
Model Re-
evaluation
Model
SynthesisM
odel
SynthesisM
odel
SynthesisM
odel
Synthesis
Sensation
3D Scene
& Moving Objectsw/ Static Objects
3D Scene
& Moving Objectsw/ Static Objects
Biomimetic Artificial Multimodal Perception Systems
How does the observer perceive:
his own motion (egomotion)
the 3D structure of all objects in the scene
the 3D trajectory and velocity of moving objects (independent motion)?
A moving observer observes a non-static 3D scene, possibly containing several moving objects:
8
WP 5, Task 5.4.3 Goal
Multi-modal sensor integration including ego-motion• We mainly expect to contribute in developing novel
perceptual computational models which:
1.are based on the fusion of perceptual modalities of vision, audition, haptic and inertial sensing;
2.mimic as closely as possible biological multimodal perceptual fusion processes;
3.perform perceptual fusion within a Bayesian framework.
• and in the process to: implement unimodal perceptual modules for Bayesian cue
integration within each modality. implement and assemble modules with several existing
state-of-the-art computational models of visual, auditory, haptic and vestibular perception.
9
WP 6, (D6.2) Goal
• Gaze control and image stabilization: rely on fusing inertial and visual sensing
modalities• human and biological system also combine the two
sensing modalities for the same goal.
contribution of psychophysical studies• Bayesian models have been successfully used to
explain psychophysical experimental findings
a robotic implementation using Bayesian inference.
10
WP 5, Task 5.4.2 Achievements
Computational Laban Movement Analysis (LMA) using the Bayesian framework
TrackingLow Level Feature
Computation
3-D Points LLF
Bayesian Inference
LMA labels
Bayesian Inference
Classified behavior
Processes for online classification
11
WP 5, Task 5.4.2 Results – Examples and Demos
02
46
810
1234E
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
E.Ti
1
2
1 3 5 7 9
E.Sp
1
2
1 3 5 7 9
Laban descriptors
Behavior hypothesis
Effort.Space
Effort.Time
K
0
1
2
3
4
1 3 5 7 9
Low-level features
Curvature
Acc
0
1
2
3
4
1 3 5 7 9
Speed Gain
Tracked positions
1520
25-10-5
05
1015
20
-25
-20
-15
-10
-5
0
5
10
Move-ment
Space1 2 3 4 5 6 7 8 9 10
U U U UR L 0 0 R R 0
12
WP 5, Task 5.4.2 Future Plans
• D5.17FCT-UC/Probayes: Publication on ‘Computational Laban Movement Analysis
based on Vision and 3-D Position Estimation’ T31
• D5.18FCT-UC/Probayes: Publication on ‘Bayesian Model for Computational Laban
Movement Analysis’ T33
• D5.20FCT-UC/Probayes: Publication on ‘Computational Laban Movement Analysis
using Multi-Camera Systems’ T36
13
WP 5, Task 5.4.3 Achievements
• Experimental Setup: The Integrated Multimodal Perception Experimental Platform (IMPEP) current version operational and new one under construction
• Bayesian volumetric map (BVM), egocentric log-spherical, for multimodal perception of 3D structure and motion
An egocentric, log-spherical spatial memory map has been devised as a framework for multimodal sensor fusion, named the Bayesian Volumetric Map (BVM)
This map stores the independent probabilistic states of occupancy OC and velocity VC for each cell C in a volumetric grid with log-spherical configuration
14
WP 5, Task 5.4.3 Results – Examples and Demos
• The Integrated Multimodal Perception Experimental Platform (IMPEP) .
• Artificial multimodal active perception system with gaze control capabilities for image stabilization and perceptual attention with:
a stereovision setup; a binaural setup; a motorised head platform, with inertial sensors
emulating the vestibular system.
current version new version under construction(PoP FP6-IST-2004-027268)
15
WP 5, Task 5.4.3 Results – Examples and Demos
Hardware
Perceptual System
Multimodal Perception Module/Bayesian Volumetric Maps ( BVM) Gaze Control
Attention andTracking
BayesianGaze Control for
ImageStabilisation
Sensory Signals (time t) Motor Commands (time t+1)
FeatureMaps
Sensors and Interfaces Active Head Motors
Bayesian InertialModule
EstimationP(O
CV
C|Z C)
PredictionP(O
CV
C| C)
P(Z|OC
VC
C)Observation
Vision Module
MotionPerception Unit
BayesianVision Sensor
Model
StereovisionUnit
Auditory Module
MonauralCochlear (AIM)
Processor
BinauralProcessor
BayesianAudition
Sensor Model
[ , ]motor
Sensors and InterfacesSensors and Interfaces Active Head MotorsActive Head Motors
IMPEP/BVM Framework OverviewIMPEP/BVM Framework Overview
16
WP 5, Task 5.4.3 Results – Examples and Demos
Using the BVM Framework for Entropy-Based Active Exploration Using the BVM Framework for Entropy-Based Active Exploration
C,
CGaze
Computation
ÑH(c)
Gaze Control
Motor Commands
Perceptual Scene
Bayesian Multimodal Perception
EstimationP(O
CV
C|Z C)
PredictionP(O
CV
C| C)
P(Z|OC
VC
C)
Observation
Z1...Z
S
H(c)
17
WP 5, Task 5.4.3 Future Plans
• D5.15FCT-UC: Integrated multimodal perception experimental platform demo
V1.0 T27
• D5.16FCT-UC/INRIA/CNRS-Gren./Probayes: Publications on Bayesian models of multimodal perception of
3D structure and motion T30
• D5.21FCT-UC : Integrated Multimodal Perception Integrated Platform demo
v2.0 T36
• D5.25FCT-UC/ CNRS-LPPA: Publication on ‘Bayesian visuo-inertial gaze control’ T42
18
WP 6, (D6.2) Achievements
• Robotic implementation of gaze control and image stabilization
• Simple probabilistic optical flow algorithm: population code-type data structure storing two-dimensional pdfs on the image
velocity space Du, Dv as an output. Primarily based on Zelek's adaptation of the block matching (correlation)
algorithm.• Bayesian program for processing of inertial data:
Bayesian model of the human vestibular system [Laurens and Droulez(2006)] adapted to the use of inertial sensors estimate the current angular position and angular velocity mimicking human vestibular perception
Probabilistic Block Matching Optical Flow
19
WP 6, (D6.2) Results – Examples and Demos
• Text
We can see a strong correlation between the pan and yaw, and tilt and roll signals, since the pan&tilt is compensating the observed motion at a low sample rate.
The small remaining optical flow observed shows that the controller is working to some extent, although it is not fully reliable since abrupt motions are not observable.
Observed yaw and roll, pan and tilt motor control, and remaining observed optical flow.
20
WP 6, (D6.2) Future Plans
Our work will focus on the multimodal sensor integration within WP5, and future work will address:
•Implement the image stabilization alrorithm with the new robotic system.
•Adding the magnetic data to our Bayesian implementation, providing a more robust attitude estimation.
•Focus on gaze control and attention models
•Contact partners within BACS (CNRS-LPPA): Following previous contacts, confront partners with our models and
implementation and discuss possible parallel trials of robotic and psychophysical experiments
Going beyond the initial implementation reported in D6.2, and propose joint work and subsequent publication D5.22 T24-T42 with FCT-UC/ CNRS-LPPA:
D5.25: Publication on ‘Bayesian visuo-inertial gaze control’ T-42
21
WP 5Summary 1
• Major Achievements during M13-M24 Computational Laban Movement Analysis based on Vision
and 3-D Position Estimation Experimental Setup: The Integrated Multimodal Perception
Experimental Platform (IMPEP) current version operational and new one under construction
Bayesian volumetric map (BVM), egocentric log-spherical, for multimodal perception of 3D structure and motion
• List of Deliverables D5.7 (MPS) Report on computational human pose recovery in
clutter M18 D5.8 (FCT-UC) State of the art on (Artificial) 3D Structure and
Motion Multimodal Perception M15 D6.2 (FCT-UC) Robotic Implementation of Gaze Control and
Image Stabilization M18
22
WP 5Summary 1
• Conference Rett, J., Dias, J.: Human-robot interface with anticipatory characteristics based on Laban
Movement Analysis and Bayesian models. In: Proceedings of the IEEE 10th International Conference on Rehabilitation Robotics (ICORR). (2007)
Rett, J., Dias, J.: Human Robot Interaction based on Bayesian Analysis of Human Movements. In: Proceedings of EPIA 07, Lecture Notes in AI, Springer Verlag, Berlin. (2007)
Luiz G. B. Mirisola, Jorge Lobo, and Jorge Dias. 3D map registration using vision/laser and inertial sensing. In European Conference on Mobile Robots (ECMR2007), Freiburg, Germany, Sep. 2007.
J. Dias, Carlos Simplicio, Diego R. Faria - 3D Photo-realistic talking head for human-robot interaction - Proceedings of the 3rd Internacional Conference on Advanced Research in Virtual and Rapid Prototyping, Leiria, Portugal, 24 to 29 September, 2007
F. Ferreira, V. Santos and Jorge Dias, Robust Place Recognition Within Multi-sensor View Sequences Using Bernoulli Mixture Models, The 6th IFAC Symposium on Intelligent Autonomous Vehicles, IAV 2007, Toulouse, France
Rett, J. and Dias, J. “Computational Laban Movement Analysis using probability calculus.” In the Proceedings of Workshop on Robotics and Mathematics, RoboMat 2007.
Rett, J. and Dias, J.:”Bayesian models for Laban Movement Analysis used in Human Machine Interaction.” Proceedings of ICRA 2007 Workshop on "Concept Learning for Embodied Agents“.
Ferreira, F. , Davim, L. , Rocha, R., Santos, V. and Dias, J.:”Using Local Features To Classify Objects Having Printable Codes”. Proceedings of the International Conference 5th Workshop on European Scientific and Industrial Collaboration on promoting Advanced Technologies in Manufacturing, WESIC
• Journal Jorge Lobo and Jorge Dias, "Relative Pose Calibration Between Visual and Inertial Sensors",
International Journal of Robotics Research, Special Issue 2nd Workshop on Integration of Vision and Inertial Sensors, vol.26, n.6, June 2007, pages 561-575.
Peter Corke, Jorge Lobo and Jorge Dias, "An introduction to inertial and visual sensing", International Journal of Robotics Research, Special Issue 2nd Workshop on Integration of Vision and Inertial Sensors, vol.26, n.6, June 2007, pages 519-535.
• Thesis Jorge Lobo, "Integraton of Vision and Inertial Sensing", PhD Thesis, Supervisor: Jorge Dias,
defended July 2007.
23
WP 5Summary 2
• Major collaborations within BACS RDG2: collaboration between FCT-UC, IDIAP, ETH Zürich and MPS Tübingen was
initiated, with the purpose of developing tools for audiovisual VR world production, so as to create stimuli both for human perception studies (WP6.4) and for simulations related to this Task, so as to test and demo the artificial multimodal perception system. This ongoing effort has already produced a co-authored State-of-the-Art Report: “State of the Art on 3D Audiovisual APIs/SDKs for Stimulus Generation and Presentation”.
RDG3, a collaboration effort between the FCT-UC, INRIA Rhône-Alpes, Probayes and CNRS-Grenoble concerning the subject “Bayesian Models for Multimodal Perception of 3D Structure and Motion” involving a student exchange from M20 to M22 was undertaken, having been successfully completed and for which several deliverables are being finalised and expected to be ready by M24, namely:
• A final Technical Report, describing the work done during the exchange period and delineating future collaboration between these partners.
• 2 joint-publications, to be submitted for ICVS 2008 and CogSys 2008 …
• Major collaborations outside BACS FCT-UC collaborates with the Perception on Purpose project (PoP FP6-IST-2004-
027268, of which FCT-UC is also a partner) sharing common physical resources and know-how, namely in the construction of the new robotic vision head
• Summary of future plans Computational Laban Movement Analysis using Multi-Camera Systems Bayesian models of multimodal perception of 3D structure and motion Going beyond the initial implementation reported in D6.2, and propose joint work and
subsequent publication D5.22 T24-T42 with FCT-UC/ CNRS-LPPA: