Upload
natalia-diaz-rodriguez
View
175
Download
1
Embed Size (px)
Citation preview
CAN SUPERVISED AND UNSUPERVISED LEARNING MARRY HAPPILY?
USE CASES ON HUMAN ACTIVITY RECOGNITION
Natalia Díaz RodríguezVisiting scholar
University of California, Santa [email protected]
4th August 2015, Mobilize Center, Stanford University
OUTLINE
•ABOUT ME & MY RESEARCH•RECENT PROJECTS•SUPERVISED AND UNSUPERVISED MACHINE LEARNING•Use Cases on Activity Recognition
ABOUT ME: What excites me in AI?
• Semantic computing• Interpretable, intuitive, human-readable knowledge representation
• Unsupervised and Deep Learning (e.g. Computer Vision) • Automatic and powerful statistical learning
• Can semantics and statistics blend together?• Cognitive neuroscience
• Can the brain’s learning mechanism inspire machine learning?• Quantified Self
• Actionable, personalized well-being
RECENT ACTIVITIES
•Google Anita Borg Scholar •/SYS/TUR Global•Women in Math and CS
•India collab.NIT Meghalaya:Wearables activity recognition
RECENT ACTIVITIES: India collab: IIT KharagpurVirtual tutor: India folklore dance:
Kinect touch free interaction
RECENT ACTIVITIES: SICN 2015 Fellow Summer Institute of Cognitive Neurosciences
http://sicn.cmb.ucdavis.edu/ Program http://sicn.cmb.ucdavis.edu/SI_2015_Week_1_Schedule_1_1_15.pdf http://sicn.cmb.ucdavis.edu/SI_2015_Week_2_Schedule_1_2_15.pdf
RECENT ACTIVITIES:Smart Dosing (with Nursing Science Dept., Finland)
Medication tray filling and dispensing in hospital wards
RECENT ACTIVITIES: First Person Vision for Activity RecognitionUnsupervised annotation (collab. with A. Betancourt, Netherlands)• Recording public Autographer dataset
Betancourt, A., Morerio, P., Barakova, E. I., Marcenaro, L., Rauterberg, M., & Regazzoni, C. S. (2015). A Dynamic Approach and a New Dataset for Hand-Detection in First Person Vision. In International Conference on Computer Analysis of Images and Patterns. Malta.
Currently: Visiting Scholar
HYBRID POSSIBILISTIC AND PROBABILISTIC SEMANTIC MODELLING OF UNCERTAINTY FOR SCALABLE HUMAN ACTIVITY RECOGNITION
•Prof. Lise Getoor•Probabilistic Soft Logic (PSL)
RESEARCH AGENDA
• Supervised and unsupervised learning USE CASES:1. Remote rehabilitation with Kinect2. Human activity recognition in Ambient Intelligence3. Semantic lifestyle profiling with wearables4. Conciliating probabilistic and possibilistic Activity
Recognition with PSL• Mentally stimulating future challenges
Classical data-driven Machine Learning
Knowledge-based Machine Learning
•Event calculus
•Situation calculus
•Rule-based systems
•Fuzzy logic
Knowledge-based Machine Learning
WHY Semantic Technologies & Ontologies?
•Semantic Web: well-defined meaning
•Ontology:
• In Philosophy: study of entities and their relations
• In Artificial Intelligence: “Explicit specification of a
conceptualization” [Gruber, 93]
• Web Ontology Language (OWL)
Background: Ubiquitous computing and Ambient Intelligence
•Smart Space (SS)
•Context-awareness: •Infrastructure and architectures•End-user programming frameworks for AmI
16
Background: Ambient Assisted Living
•Usage of technology to provide assistance to peoplewho needs it in their daily activities, in the less obstrusive way
•Aim: support older/disadvantaged people, independent living, safety
•Includes: methods, systems, products and services
17
Case study 1: A Kinect ontology for physicalexercise annotation and recognition
•Active Healthy Ageing project (EIT Digital)
with Philips Personal Health Labs (PHL)
•Sensor data aggregation platform
18
19
Kinect for remote rehabilitation DEMOS
•Sit-to-stand testhttps://www.youtube.com/watch?v=g8HOtFTk80c• Remote monitoring of post-surgery rehabilitation exercises on shoulder, knee, hip
https://www.youtube.com/watch?v=XL4JexDNs-Q
Ontology features
Skeleton tracking (bone joint rotations + bone orientations)
21
Exercises & Workouts Ontology
22
Kinect ontology: examples of use
23
• Example 1: Defining basic movement (Stand, BendDown, TwistRight, MoveObject, etc.)
• Example 2: Provide workout feedback (# series in time, quality comparison with medical guidelines)
Kinect ontology: examples of use
24
• Example 3: Historic analysis can monitor posture quality in time. E.g. having back less straight than 1 year ago-> notify to correct/prevent on time.
• Example 4: Notifications for office workers sitting too long/ not properly
Case Study 2: PhD (Cum laude, Finland-Spain): Semantic and fuzzy modelling for human behaviour recognition in Smart Spaces, a case study on Ambient Assisted Living
Ros et. Al. 2011
26
SPAIN: 15m of elders in 2049 (1/3 of the population) (INE)FINLAND population 65+ years: 18.14% [1]
• [1] http://www.finnbay.com/media/news/government-prepares-to-set-out-new-requirements-for-senior-caretakers/
PHD OBJECTIVES
27
•Understand Smart Spaces•Human Activity Modelling and Recognition
•Program Smart Spaces
PHD OBJECTIVES
•Understand Smart Spaces•Human Activity Modelling and Recognition
•Program Smart Spaces
28
Activity Recognition in Smart Spaces 29
[Image: http://www.businesskorea.co.kr/sites/default/files/field/image/smart%20home.jpg + The noun project]
Human Activity Recognition
30
Handling uncertainty, vagueness and imprecision
• Broken/ missing sensors
• Incomplete data, vagueness
• Different ways of performing activities• Different object usage, duration, etc.
• Behaviour change
31
Tools
32
33
[CONON Context Ontology]
Methods: Ontologies
34
JULIOANA MARIA
NATALIA
Has Brother
Has Mother
Methods: Ontologies
35
JULIOANA MARIA
NATALIA
Has Brother
Has UncleHas Mother
Methods: Ontologies
Methods: Fuzzy LogicWHY fuzzy (description) logics and fuzzy ontologies?
• Real life is not black & white• Classical (Crisp) Logic: True/False
• Fuzzy Logic: [0, 1]
• e.g. blond, tall
• For automatic reasoning about uncertain, vague or imprecise knowledge
• For natural language expressions
[Bobillo 2008 fuzzyDL: An Expressive Fuzzy Description Logic Reasoner: http://gaia.isti.cnr.it/straccia/software/fuzzyDL/intro.html] 36
37
[Image: http://www.harmonizedsystems.co.uk/]
Example: Take medication
A fuzzy ontology for activity modelling and recognition
38
Classes, Individuals, Data Properties and Object Properties
SUBJECT PREDICATE OBJECT
User performs activity Taking medicine =
(0.3 User performs sub-activity reach Cup or Medicine Box)
(0.3 User performs sub-activity move Cup or Medicine Box)
(0.1 User performs sub-activity place Cup or Medicine Box)
(0.1 User performs sub-activity open Medicine Box)
(0.1 User performs sub-activity eat Medicine Box)
(0.1 User performs sub-activity drink Cup)
39
SurveyingActivityRecognitiontechniques
40
SurveyingActivityRecognitiontechniques
Surveyingontologies for activitymodelling
41
AR Ontologies ranking: domain coverage evaluation
42
OOPS! (OntOlogy Pitfall Scanner!) evaluation
43
Case study 1: A fuzzy ontology for AR in the office/workenvironment
44
45
2-phased algorithm:
1. Sub-activities (data-driven phase)
2. High-level activities (knowledge-based phase)
Validation: CAD-120 dataset:
•10 sub-activities, 10 activities, 10 objects, 4 users46
Hybrid activity recognition with fuzzy ontologies
Cornell Activity Dataset [Koppula et al. 2013]
47
Hybrid data-driven andknowledge-basedactivity recognition
48
a) Data-driven sub-activity recognition phase
49
b) Knowledge-driven sub-activity recognition phase
50
Ontological
definitions
Ontological definitions: object interaction
51
Ontological definitions: object affordances
52
10 semanticrules
53
SUB-ACTIVITY prediction accuracy
54
55
ACTIVITY prediction accuracy
Activity recognition - comparison with state-of-the-art
56
Activity recognitiontimes (ms)
57
PHD OBJECTIVES
•Understand Smart Spaces•Human Activity Modelling and Recognition
•Program Smart Spaces
58
Deployment: Programming Smart Spaces
59
Ros et. Al. 2011
A visual language to configure the Smart Space behaviour
•TARGET USER: a) Developerb) Non-technical background
•AIM: •Rapid & easy programming of applications•Improve interoperability and usability
60
Programming environments for novice programmers
61
[Scratch] [IFTTT]
PROPOSAL: SS visual language mapping to OWL 2
62
PROPOSAL: Smart Space visual programming
63
PhD main contributions
1. A set of ontologies to model human behaviour and tackleuncertainty and vagueness inherent to real life
2. An architecture that integrates Semantic Web and Fuzzy Logicfor interpretable activity recognition
3. A hybrid knowledge-based and data-driven algorithm for real-time, robust activity recognition (84.1% prec.)
4. Design & development of a toolbox for non-expert users and rapid programming of Smart Spaces
[4 Journals -3 on Q1-, 9 conf. Papers. Google Anita Borg, Nokia and HLF scholar. University entrepreneurship award]
64
Startup technology transfer:
65
USE CASE 3: Semantic lifestyle profiling with wearables
Can we recognize lifestyle patterns automatically?1. Provide meaning to large heterogeneous data
-Interpretable, actionable insights2. Knowledge-based methods & uncertainty handling
-Behavior vs Profile recognition
Day routines and lifestyles:• Work/shop-aholics, Gym addicts• Pet/ Partner/ Kids • Retired/ Worker/ On holiday
USE CASE 4: Activity recognition with Probabilistic Soft Logic
•Can manual work be automated?•Can we improve…
•Model & rule learning •Accuracy•Scalability •Genericity
Probabilistic Soft Logic (PSL) :
Expresses collective inference problems mapping logical rules to convex functions(defining a hinge-loss Markov Random field).• FOL Predicate: relationship, property or role• Atom: (continuous) random variables• Rule: dependencies or constraints• Set: aggregatesPSL Program = Rules + Input DB
[PSL (open-source): psl.umiacs.umd.edu]
PSL advantages against fuzzy OWL
•Statistical relational learning •Adds probabilistic component to possibilistic one•Captures cyclic dependencies•Rule weight & latent variable learning
•Scalable (convex optimization) learning•Most probable explanation (MPE) inference
Current work @LINQS Lab
•Can we seamlessly blend…• Knowledge-based and data-driven mechanisms• Supervised and unsupervised learning
for a general activity recognition framework?
•Can TIMED streams be handled naturally?•Cost-sensitive, Online model learning and evolution
•Can models balance•Flexibility and reproducibility?•Accuracy VS deviation/anomaly detection?
General activity recognition framework
AIM: Automate heuristics while maintaining rich semantic granularity:
• Context-awareness • Object interaction, cardinality, recursion, rule subsumption
• Unordered/ordered (sub)sequences of sub-activity-object pairs • Min/ max pattern sequential repetition in Δt
Activity rule examples (PSL)
performsSubActivity(Object, ObjPosition, Time)
m.add rule:
performsMove(MedicineBox, P, T1) &
PerformsDrink(WaterGlass, P, Tn)) >>
PerformsTakingMedicine(Tn),
weight : 10;
Future Challenges in Activity Recognition
•Multiple human sensing•Parallel/interleaved activities
•Automatic ontology learning and evolution•Reduce manual work
•(FOL/DL) Logics support for temporal constraints73
• Unsupervised activity modelling• First camera vision AR
• Automatic dataset annotation
• Wearables sparsity and uncertainty• Scaling and real-timeness
74
Future Challenges in Activity Recognition
TAKE HOME MESSAGE
• Supervised ML: Statistical and probabilistic approaches
• Unsupervised ML: Interpretable knowledge-based techniques
Both are needed!
Domain expert knowledge and common sense knowledge: Classical ML & Deep Learning: unable to exploit it!
THANK YOU!
Brainstorming/collaboration ideas, tips, pointers are welcome!
Natalia Díaz Rodríguez [email protected]
https://research.it.abo.fi/personnel/ndiaz
COST Action on Architectures, Algorithms and Platforms for Enhanced Living
Environments: aapele.eu and Finnish Foundation for Technology Promotion
Kinect for remote rehabilitation DEMOS
•Sit-to-stand testhttps://www.youtube.com/watch?v=g8HOtFTk80c• Remote monitoring of post-surgery rehabilitation exercises on shoulder, knee, hip
https://www.youtube.com/watch?v=XL4JexDNs-Q
Comparison with existing state-of-the-art (sub-activity and activity recognition modules)
78
Human Activity Recognition
A crucial but challenging task in Ambient Intelligence and AAL. Requires:
Context-awareness and heterogeneous data sources
Training data: examples
Common-sense knowledge /domain experts
Adaptation of behavioursAlzheimer, Parkinson
79
ACTIVITY prediction accuracy (ideal situation)
(100% accurate sub-activity prediction) 80
SUB-ACTIVITY prediction: accuracy results
81
82
Fuzzy KB and rules in fuzzyDL
Smart Space Architecture: Smart-M3
83
84
Equivalent SPARQL Query
85
Each rule is converted into a SPARQL query, which can be transformed into a Smart-M3 subscription.
Crisp to fuzzy OWL query mapping to improvesemantics and usability
86
Handling uncertainty reasoning whenprogramming Smart Spaces
• Fuzzy reasoners: expressivity VS
computational requirements and
platform versatility:
• Best compromise: fuzzyDL
87
Activity recognition Algorithm
89
Ontology classes, data & object properties
90