Embed Size (px)
Citation preview
Introduction
Augmentative and Alternative Communication
• Assist people with Intellectual Disabilities (ID)• Allow them to communicate or use the Internet• Increase life quality by reducing social isolation• Picture-based communication systems are
a form of AAC technology
Introduction
Picture-based Communication
Three types:• Universal pictographs• Emoticons• Pictographs for people with disabilities
Beta pictographsSclera pictographs
hond (dog) ik (I)
fruit-persen (squeeze-fruit)
gebraden-kip (roasted-chicken)
hond (dog) ik (I)
hond-eten-geven(dog-food-feed)
afwas-proper-terugbrengen
(dishes-clean-return)
Introduction
Pictograph Sets
Introduction
Problem Definition
• The internet and social media websites play an important role in our lives– However, they are not accessible for people with ID– Reading and writing difficulties, no access to pictographs– There is a risk of e-exclusion
• Solution: Text2Picto and Picto2Text
• Enabling internet access for people with ID• Specific applications for these users
– Chat and e-mail client with the Picto engines– Easy-to-Read News – Text-to-Speech– Games
• www.wai-not.be
The Users
WAI-NOT
• European project: Beginning of March 2014 – End of March 2017• Goal: Improve the living conditions of people with Intellectual
Disabilities– Natural Language Processing tools can bring independency to them– Ex. Social media websites, email
• Accessibility Layer based on three key technologies:– Tech 1: Text and content simplifier (Simplext): English & Spanish– Tech 2: Text-to-speech functionalities: Dutch, English & Spanish– Tech 3: Text-to-Pictograph and Pictograph-to-Text translation: Dutch,
English & Spanish
The Users
Able to Include
• Our partners (Belgium, UK, Spain) tested the tools in three “user scenarios”:– Labour integration: email client, word processor, pdf reader– Mobility: guidance systems for people with ID– Leisure within information society: Facebook, Twitter, WhatsApp,…
Text2Picto
Introduction
Text
Mijn kat eet vis.My cat eats fish.
Mi gato come pez.
Sequence of pictographs
Sclera
Beta
I eat straberries.(Spelling error made on purpose)
I eat straberries .I eat strawberries .
pronoun verb noun punctuationI (to) eat strawberry .
06832033 01168468 04696316 n/a
ik.png voeding.png aardbei.png n/a
TokenizationSpelling correction
Part-of-speech taggingLemmatization
Find synset IDs
Find pictographs
• “Language-independent” design of the Text2Picto tool:– Same architecture for Dutch, English, and Spanish– The system can easily be extended toward other languages
• Users with ID and their coaches gave us feedback on the original Text2Picto tool: difficulties, needs, wishes,…
• We are now improving the Text2Picto tool (case study: Dutch)– Syntactic simplification for pictograph translation – Automated spelling correction for users with ID
Picto2Text
Introduction
Text
Mijn kat eet vis.My cat eats fish.
Mi gato come pez.
Sequence of pictographs
Sclera
Beta
mijn.png kat.png eten.png vis.png my…
catkittencats
the catsthe kitten
a cat…
eatate
consumeconsuming
eatingeats
…
fishguppy
the fish…
• Challenge: a pictograph corresponds to multiple words in a given language, withmultiple inflected forms (number, tense,…)
• Again: “Language-independent” design of the Picto2Text tool:– Same architecture for Dutch, English, and Spanish– The system can easily be extended toward other languages
Picto2Text
Composing pictograph messages
• The static hierarchy of pictographs– A three-level category system– Based on topic counts, frequency of use,…
• The dynamic predictor of pictographs– Can be compared to Google’s autocomplete function for text– Uses n-gram information and semantic associations
