and the ArtificialThe Human

Tilburg Center for Cognition and Communication

e-Humanities Research

Colofon

This brochure is meant for special-interest groups about research at Tilburg Center for Cognition and Communication, Tilburg University, the Netherlands.

Publisher Tilburg University, School of HumanitiesEditor in chiefJaap van den Herik Editors Leon Heuts, Annemeike Tan Contributing editors Mieke Fiers, Leon HeutsPhoto editors Beelenkamp Ontwerpers Tilburg, Annemeike TanPhotography Johner Royalty Free/Hollandse Hoogte (cover), Jean Claude Moschetti/Hollandse Hoogte (p.6), Harold Miesen (p.11), Paul Vogt (p.12), Richard Jones/Hollandse Hoogte (p.18), Seunghan Son/Hollandse Hoogte (p.25)Translation Taalcentrum VU Amsterdam, Hans VerhulstLay out and graphic design Beelenkamp Ontwerpers TilburgPrinter Drukkerij Groels, Tilburg

© 2012 Tilburg University, School of Humanities

Contact

TiCC Tilburg University, School of HumanitiesPO BOX 901535000 LE TilburgPhone +31 (0)13 466 8118E-mail ticc@tilburguniversity.eduWebsite: www.tilburguniversity.edu/ticc

3

e-HumanitiesWill we ever be able to interact with computers in a totally famil-iar and natural way? Could we communicate with them in exactly the same way as we do with other people? That is what TiCC’s research into creative and intelligent systems is working towards. Under the slogan, “Make the computer adapt to us, not the other way round.”

As such, our work is exploring both intelligent systems and human intelligence. To find out how computers can communi-

cate with us in a natural fashion, we need to know more about how we do that with each other. How do we use language? And what about non-verbal means of communication, such as hand gestures?

TiCC brings together a wide spectrum of academic research disciplines: computer sci-ence, artificial intelligence, psychology, neurosciences, linguistics and sociology. In our work on AI, for example, we are trying to train computers to generate their own rules or algorithms from large quantities of data. A computer learns by measuring and checking, by recognizing patterns and by applying intelligent combinations and recombinations. Amongst the resources being developed by TiCC with the help of machine learning are intelligent databases. Another line of research is developing algorithms that can be used to tell the difference between a genuine Van Gogh, say, and a forgery. And there are more questions we hope to answer. For example, can a computer identify potentially dangerous situations by analysing the movements of a group of people? Or differentiate a tumour from a healthy tissue?

Then there is our research into human communication. Even though this is something we are all intimately familiar with, it is difficult to formalize in terms of hard and fast rules. We use deictic adverbs like “here” and “there” effortlessly in conversation, and when talking we often gesticulate without even thinking about it. But when exactly? And how? Our work is identifying conversational situations that are enhancing our knowledge of human communication, with a view to enabling computers to reproduce them in a naturalistic manner. By, for example, recognizing human emotions. Through this research we are entering a new phase in the development of artificial intelligence, with intuitive computers that can “sense” situations and so respond appropriately to unexpected circumstances. Eventually, we will be able to communicate with computers on an equal footing.

By understanding the computer better, we are learning to understand people better. That is why I call our field “e-humanities”.

Jaap van den Herik, professor Computer Science and director of TiCC

Pref

ace

PrefaceJaap van den Herike-Humanites

GAMESInterview: Fighting fires

with serious gamesResearch portrait:

Serious game, psychological tool

ART AND VISIONInterview: Fake Van Goghs, suspect shipping and real tumours

HUMAN COMPUTER INTERACTIONInterview: The computer must adapt to us, not us to itStudent projects:User interface design for a better life

SOCIAL MEDIAIs a Facebook friend a real friend? Research portrait:The social signals in human voice

3

8

24

CULTURE AND LANGUAGEInterview: Language development in children in Mozambique and the NetherlandsResearch portrait: History and computers

12

TEXT AND COMPUTERSResearch portrait: From books to tweets: 500 million words in one corpus

11

20

NON VERBAL COMMUNICATIONInterview: A sensitive avatar Research portrait: Hand gestures and computers

Language, Communi-cation and Cognitionled by Emiel Krahmer and

Marc Swerts

Creative Computingled by Eric Postma and

Pieter Spronck

4

16

Computer games are now more than just entertainment. People in dangerous professions, like the fire and ambulance services, are using them to train in virtual environments. But how do you make them as realistic as possible?Deep in concentration, the soldier scans the hostile landscape, alert to any suspicious movement or potential danger. The turret of his tank obscures part of his view, and his radio is crackling in his ear. He is not in Iraq or Afghanistan, though, but in a virtual world in the Netherlands. There, troops are trained by a sensemaking system in “managing the unexpected”. Serious gaming is one of the research topics being addressed by Professor of Computer Science Jaap van den Herik and researcher Pieter Spronck. In this field, Tilburg University works closely with the Netherlands Defence Academy (NLDA).

“When we talk about the intelligent use of computer systems,” says Spronck, “we can divide that into three distinct forms.” First there is the computer as a repository of knowledge. A tool, in other words. “You look something up – on your iPad, for example. Then you, the user, interpret what you find.” The second form brings the computer into the human world. This covers everything from “intelligent” garden furniture (“Where do I position myself so that my owner is sitting in the sun?”) to chess computers, a subject Jaap van den Herik has studied

to express them realistically.” That degree of realism is also a “Holy Grail” for entertainment games, since the enjoyment of immersing yourself in an entirely different world is spoilt every time the computer “shines through”.

“There are games and training environments that look perfect,” says Spronck. “Photorealistic, you might say. But you can still see immediately that they’re not real. If a robot here in the room walks into a wall, we laugh. If a virtual person in a game does it, we say ‘how stupid’.” So the computer world not only has to look real, the people in it

4 5

Gam

es Gam

es

Interview Jaap van den Herik and Pieter Spronck

in depth and made a major contribution to as a researcher. Finally, you can take people into the computer’s world so that they experience a total virtual environment.

There is already a flourishing industry in intelligent computer worlds and entertainment games, like World of Warcraft and Second Life. But gaming is now also rapidly gaining ground in the professional domain. Firefighters, for instance, can practise how to enter burning buildings in situations which would be too dangerous for real-life training. Police officers and other front-line officials can use simulations to learn how to deal with difficult members of the public. Managers can improve their specific skills and ambulance crews can brush up their patient communication abilities.

For the training to be effective, it is vital that the “games” – in the form of simulations – be as realistic as possible. That is still not always the case, says Spronck. “In the training game for ambulance crews, for example, actors are often employed to play out the emotions involved. A computer is simply unable

Lifelike games help in high-risk jobs

Fighting fires with serious gaming

6 7 Research portrait

Giel van Lankveld

Gam

es Gam

es

also have to behave realistically. A good deal of the research being conducted by Van den Herik and Spronck is homing in on this factor.

For a computer to imitate a person convincingly, you have to teach it what typical human behaviour is. And that is difficult to pin down. “People and computers are so different,” explains Van den Herik. “They reason in such different ways. Chess computers may be able to beat even the world champion, but when you really think about it you realize that they’re not actually playing the same game. If you feed a computer with examples of ‘well-mannered’ drinking, say, then it analysed what constitutes ‘well-mannered drinking’. At the moment, that learning process is still quite laborious.” But Spronck predicts that it will become steadily easier for the machine. “We have to reach the point where a computer is able simply to observe its environment and pick up from that any mannerisms it is capable of learning.”

“Recognizing behaviour and non-verbal communication is becoming increasingly important,” he continues. “If we start using robots to help us move house, it’s not such a problem if they behave like computers. But in the not too distant future we’re also going to need them in the healthcare sector. And for that we’ll need a model of a human being.” Those computers will have to learn to recognize emotions and to respond accordingly.

Van den Herik’s research group is interested in every possible form of human behaviour. For instance, two PhD students from the National Police Services Agency (KLPD) are working on the fight against terrorism. Nick Koeman is studying the characteristics of attacks like the one targeting the Dutch royal family on Queen’s Day 2009, whilst Peter Berenschot is looking into factors whereby you can identify somebody as a terrorist. “The result of his research will be a realistic training module,” says Van den Herik. “And together,” adds Spronck, “the two studies should enable a computer to identify any weak points in security systems at an early stage.”

Serious game, psychological tool How you respond in a particular situation. Your way of thinking. What motivates you... All are related to your personality. And not just in “real life”. Research by PhD student Giel van Lankveld has revealed that a person’s game play in the virtual world is also influenced by their personality, in a variety of ways.

Van Lankveld spent several years studying Computer Science, earned a degree in Psychology and now combines the two fields in his work at TiCC. “I originally began my research to make computer games better suit their players,” he explains. To do that, he drew upon ideas from psychology. For example, he applied an incongruence model to games. This predicts that new users of software like Word or visitors to a website form a picture of what they can do with it. If the experience turns out to be more complicated than expected, they become frustrated. More experienced users, on the other hand, can cope better with the software but eventually start to find it boring. “Applied to games,” says Van Lankveld, “you can conclude from this that they have to be flexible in order to retain their appeal and to interest everyone.”

Another psychological concept he has applied to the world of games is personality theory. Again, the original aim was to adapt games to their users. But he has now shown that personality plays a key part in determining player behaviour, whether they are on something like World of Warcraft or taking part in one of the serious training games used by, say, the police. This raises another interesting question: can a serious game help in classifying an individual’s personality? Psychologists conduct a personality test to do this, but that has a number of drawbacks. In particular, the test currently in use is not suitable for children. Van Lankveld is now investigating whether a game might deliver comparable results, or perhaps even better ones. In this way he hopes to develop a useful tool for psychologists.

“Managers can improve their specific skills and ambulance crews can brush up their patient communication abilities.”

Newstweet

Robots will be able to do everything better than people in the future #TiCC

Want to know more?TED Talk:

Peter Singer on military robots and the

future of war

http://www.ted.com/talks/pw_singer_on_robots_of_war.html

8 9

Art and V

ision Art

and

Vis

ionThe endless possibilities of image recognition

Fake Van Goghs, suspect shipping and real tumours

Got a fake Van Gogh in your attic? Don’t get excited, because art historians have recently added a new weapon to their armoury of tools for determining the authenticity – or otherwise – of the artist’s work. It is a computer system which analyses putative Van Goghs by comparing their visual structure with that of his known paintings.

The man behind the system is Eric Postma, Professor of Artificial Intelligence at TiCC. The Van Gogh project is just one of many he is working on in the field of visual perception. We humans can tell the difference between, say, a spoon and a book at a glance. But the underlying brain processes involved in drawing that distinction are actually

A technology that recognises painters from their brush-strokes, spots ships acting suspiciously and identifies pos-sible tumours at an early stage – image recognition does it all.

Interview Eric Postma and Jeroen Janssens

An analysis of the brush-strokes

in Van Gogh’s “The Sower”

very complex. Postma and his colleagues want to teach computers to do much the same thing with visual images. A technology that could be used to help track down criminals, for example, but also one with the potential to improve search engines like Google Images.

Postma is developing image-recognition algorithms, from which a new image can be identified or classified based upon a collection of known examples. “The algorithm translates the image into a series of numbers,” he explains, “in such a way that visual representations which resemble one another produce similar numerical sequences.” The scientist does need to give the computer a helping hand, though. “We know, for instance, that people tend to concentrate upon the transitions from light to dark. That’s something we can describe mathematically and tell the computer about.”

Inspired by a visit to an exhibition of Impressionist paintings, Postma and his colleagues decide to test one of their algorithms on those works. That was the beginning of a highly successful project. They ‘fed’ a computer with digital reproductions of paintings, together with the names of the artists. From that input the system was able to automatically identify particular characteristics associated with each painter. “Van Gogh used a very particular brushstroke,” explains Postma. “His paintings from certain periods are composed of a huge number of very short strokes.” Given a new, unidentified work to analyse, the computer would try to attribute it based upon the characteristics it knew. And it was successful in 95 percent of the cases. The group’s follow-up project homed in on works by Van Gogh.

Converting the numerical sequences into a graph produces a scatter plot in which the genuine Van Goghs are grouped closely together. If a new painting falls outside that “cloud”, the art historian needs to take a really good look at it. “This is not a process that replaces people,” Postma stresses, “but a tool they can use.”

Jeroen Janssens, a PhD student working at TiCC under the supervision of Eric Postma and Jaap van den Herik, is using the same principle to tackle a very different issue. For the defence technology firm Thales Netherlands, he is designing a computer system to assist the coastguard. “At any given time,” he explains, “there are 600 or so ships off the coast of Rotterdam. The coastguard keeps an eye on them all on big screens in its control rooms.” Janssens’ system picks out those which are behaving in an unusual fashion, and so require extra monitoring. “That could be a fishing boat which is stationary in an area where fishing is prohibited, a

10 11

Art and V

isionTe

xt a

nd C

ompu

terstanker entering the port too fast or a vessel sailing in the wrong direction – even out

at sea there are shipping lanes you have to follow.”

“The computer has to learn what constitutes strange behaviour on the part of ships,” adds Postma. “But what does that mean? Is it standing still? Coming too close to another vessel? And how close is that?” Because it has proven difficult to define hard and fast rules for unusual behaviour, Janssens has been exploiting the computer’s ability to recognize patterns in the data it is supplied with. From a huge number of examples, he has abstracted “normal behaviour”. Not so much by defining a norm as by identifying deviations from it.

The techniques being developed by Janssens, Postma and their colleagues are abstract. “It doesn’t matter to the computer whether it’s dealing with ships or paintings or apples,” Janssens explains. But the theoretical systems are always linked and applied to real-life domains, be they art or maritime security. “The two sides of the process go hand in hand. By testing it constantly in practice, you know whether your algorithm is working and what direction you need to take it in.”

It is a tough challenge, but the possibilities seem endless. “We’ve recently teamed up with Marius Nap, a pathologist at the Atrium Medical Centre in Heerlen and research affiliate of TiCC,” Postma continues. “He works with cross-sections of cancerous tissue. It should be possible for a computer program to identify the anomalies in them.”

At the other end, there are several Internet forums. In a campaign in which an iPad could be won, his team recently managed to collect more than 50 thousand text messages. Once the texts are in the real work begins. The widely varying files are standardized and checked, texts are divided into sections, words and punctuation marks are separated, all the meta-data of the text are recorded. And every word is accompanied by as much information as possible, specifying things like ‘name’, form, plural or conjugations.The labor-intensive, meticulous job is now almost finished. Reynaert: “Researchers often build corpora for a specific research project. This is not the case here; the SoNaR-corpus will simply be there, available for everyone.”

Dictionaries, translation software, talking computers – for the development and improvement of these, language research is required. This research in turn requires the availability of corpora. Martin Reynaert is compiling one such corpus for the STEVIN-project of the Dutch Language Union (‘Nederlandse Taalunie’). SoNaR is going to be a collection of written texts in present-day Dutch and Flemish, comprising a total number of at least 500 million words.That is an awful lot – the average paperback contains some 80 thousand words –, but the really tricky part is to balance the texts with respect to the various origins. “Two years ago we got the opportunity to download a Flemish Internet forum, which by itself already contained almost 500 million words”, Reynaert explains. But SoNaR is supposed to record a cross-section of present-day written Dutch and Flemish. Some forty different types of text need to be represented in it – ‘everything from books to tweets’ – one third of which are to be in Flemish; the other two-thirds in Dutch.

Collecting the text is a daunting task in itself, because permission has to be acquired for almost every text. Reynaert noticed that publishers are particularly reluctant to cooperate. And e-mails turn out to be too private.

Research portrait

Martin Reynaert

From books to tweets: 500 million words in one corpus

“The Dutch reference corpus SoNaR will simply be there, available for erveryone”

Newstweet

The better computers become at detecting counterfeits, the better counterfeiters will get at making them. #TiCC

In 20 years, computers and computerized analysis methods will be a standard part of the art historian’s toolbox. #TiCC

Want to know more?

Article: Martin Reynaert et al., ‘Balancing SoNaR: IPR versus

Processing Issues in a 500-Million-Word Written Dutch

Reference Corpus’

http://ilk.uvt.nl/downloads/pub/papers/Reynaert.LREC2010.

BalancingSoNaR.549.Paper.pdf

This photograph illustrates the use of a filter that is sensitive to the visual contours of brush-strokes. The higher the intensity, the more visible the brush-strokes.

Postma is also looking forward to his next project. That centres on non-verbal communication, a topic which is already engrossing several researchers at TiCC. “Right now,” he says, “they still have to record manually how often, say, a particular television newsreader raises her eyebrows on air. But we can develop an algorithm to recognize that.” That knowledge could then be used to teach computers to identify and reproduce facial expressions. “In the future our computer screens will show a virtual face which can communicate with the user both verbally and non-verbally.”

12 13

Culture and Language C

ultu

re a

nd L

angu

age

Language development in children in Mozambique

and the Netherlands

to know what it means. You start with a hypothesis, but in order to check if it is correct you need additional information. You get this information through interaction.”In his CASA MILA research project, Paul Vogt investigates language development and the role interaction plays in this process. This not merely concerns oral feedback, but also gestures like pointing and nodding, and showing things. With the use of a camera, Vogt records everyday life in families in Mozambique and the Netherlands. All conversations and gestures – a nod of the head, pointing at something with a hand – everything is meticulously analyzed. What kind of interaction is there between a child learning a language and the people around him? And how does that relate to development of language in children?

Vogt’s involvement with language development is not a recent thing. In the past, he has tried to gain more insight into how children learn or develop language by using computer models. “Most researchers teach a computer program a language by giving it huge numbers of example sentences as input. But when people learn a language they are in constant interaction.” He points to a piece of chocolate on the table and he mimics a conversation: ‘cookie’?’ ‘No, that’s not a cookie, that’s a piece of chocolate.’ Because of the vital importance of interaction, Vogt built a simulation model that includes interaction. The model mimics a family scene involving ‘children’ who as yet know very few words, and a number of people who speak the language. In this model, he creates situations in which interaction takes place. His models did indeed show that interaction influences the speed of learning. But in order to simulate reality, he first had to know more precisely what exactly this reality looks like.

How do children learn a language? Interaction – verbal and non-verbal

feedback – plays a crucial role in this.

Because Vogt could not find any usable answers in the literature he decided to collect suitable material himself and to apply this to his simulation model: the CASA MILA project. Research organization NWO supports Vogt’s project with a VIDI grant. Among other things, he used the money to appoint PhD student Doug Mastin on the project.

At the beginning of 2010, they went to Mozambique together. “The Netherlands as a modern society is quite special; it is a bit odd to do general research solely on the basis of the Dutch situation”,

What does ‘gavagai’ mean? Good question. The word ‘gavagai’ was coined by mathematician and philosopher W.V.O. Quine. He imagined an anthropologist investigating a tribe whose language he does not know. At a certain moment a rabbit scurries by and one of the natives says ‘gavagai’. In the first instance, the anthropologist will think that the word means ‘rabbit’, but, as researcher Paul Vogt points out, it may just as well mean ‘look, food’, ‘let’s go hunting’ or ‘creature with long ears.’ “In principle, the range of possible meanings is endless, so you need hints

Paul Vogt and Doug Mastin

currently analyzing at several levels. Every hour of recorded data takes about a week to analyze. Eventually, this yields data with various types of interactions, and with information on how often these occur and how they relate to an (estimated) vocabulary. Subsequently, Vogt will combine these data with his computer model. If the model is correct, then identical types of interactions should yield identical types of development in the child’s vocabulary as are observed in reality. There will not be a perfect match straight away. Vogt will have to adapt the model. By mimicking reality in a computer, Vogt hopes to arrive at a better understanding of this reality, because a correct computer simulation will make the process of language development transparent. In a way, it affords us a peek into (the workings of) the brain. “At the moment, we only know what it is like on the outside: we know that information goes into it and that this yields language development, but we do not know what goes on inside the brain.”Vogt’s research touches on various types of research conducted within TICC, such as the research on gestures, or on machine learning. For Vogt it is no more than logical that he should combine various disciplines in his research. “It is only through multidisciplinary thinking that something as complex as the development of language can be properly understood. You need to familiarize yourself with a range of disciplines.” In other words, it’s got TICC written all over it, this CASA MILA project.Mastin’s research component, on the other hand, is in no way connected to computers. “I certainly feel out of place sometimes in the department”, he admits. “However, with more and more research becoming interrelated and applied in computer science, it is very important to keep a research group like TiCC varied in regard to research techniques and applications. I am happy to represent cognitive science within the research center.”

14

Culture and Language

With whom did Ferdinand Domela Nieuwenhuis associate? Who were his friends? Who did he argue with? PhD student Matje van de Camp is currently researching the relationships maintained by the great politician Domela Nieuwenhuis (1846-1919), one of the founders of the socialist movement in the Netherlands. Her social network analysis could thus shed new light on the early years of Dutch socialism. As she explains, “If two people start out as friends but later on they are enemies, you can safely assume that something has happened between them. You can then look for what that was.”

Van de Camp has been able to draw upon the huge historical archive held by the International Institute of Social History (IISG) in Amsterdam. “It contains a lot of material about Domela Nieuwenhuis,” she says. “He appears frequently in biographies of other people. We have manually classified a portion of those texts to produce the input for the system.” From that material, the computer can then be trained to retrieve similar information from the remaining texts and classify it in order to produce an overall picture of Domela Nieuwenhuis’ relationships. But there are plenty of hurdles. In many cases, for example, the relationships are described by third parties and thus are rather subjective.

Van de Camp’s research forms part of HiTiME – short for Historical Timeline Mining and Extraction – a joint project by TiCC and the IISG to make historical data accessible and comprehensible.

For Van de Camp herself, the collaboration this involves has been inspir-ing. “Once every two or three months,” she explains, “we all sit down together: people from the IISG and us researchers from TiCC. They let us know what information they are interested in finding, which makes us look at our research from a different angle. And when we show them just what the technology can do, that inspires them anew.”

Research portrait

Matje van de Camp

“It’s not easy charting Domela Nieuwenhuis’ network”

Poster: Paul Vogt and Doug Mastin, ‘Analyzing

multimodal gesture usage and infants’

vocabulary development in natural environments’

http://ilk.uvt.nl/~paul/publications/

posterVogtMastin.pdf

Article: Paul Vogt and Elena Lieven, ‘Verifying

Theories of Language Acquisition Using

Computer Models of Language Evolution’

http://ilk.uvt.nl/~paul/publications/

abVogtLieven2010.pdf

Web-page: The CASA MILA home pagehttp://ilk.uvt.nl/~paul/one/

CASA-MILA.html

Newstweet

Thanks to user-friendly online resources, non-experts will be able to find their way through an enormous mountain of documents much more easily in the future. #TiCC

Vogt says, explaining his choice to not only observe Dutch families. In Mozambique, they gathered data on film in the countryside as well as in the towns. In the villages in particular, there are many people involved in the raising of children; there are often aunts, brothers and sisters around that all play a role in the process. This is what Mastin’s research is focused on: the correlations between time spent in interactions with different types of communication partners and an infant’s vocabulary development.The two researchers have now collected 42 hours of video material in Mozambique, which they are

15

Cul

ture

and

Lan

guag

e

Want to know more?

A sensitive

avatar

To convey a story convincingly, you need to make sure that your body language matches your message. So wrote the great Roman rhetorician Quintilian in his Institutio Oratoria. Good politicians, actors and other ‘performers’ understand

the need to master that art: they know that good communication is not just about the spoken word, but also about non-verbal expression. But what exactly

is the secret of these masters? “Despite the fact that we’re so familiar with body language,” says Marc Swerts, “we still lack a good deal of explicit knowledge on

how it functions in human interactions.” As Professor of Discourse studies, Swerts has been conducting research on non-verbal communication. During a working

visit to animation studio Pixar, he once saw at first hand how weeks of work went into creating a single expression on the face of cowboy Woody from the film Toy

Story. “That’s how difficult it is. And artistic creation is very different from scientific research. One thing we want to do at TiCC is to create a dataset or corpus from

which a computer can both generate various forms of non-verbal communication and recognize them in people. What we are actually doing is coding non-verbal

signals as instructions a computer can carry out. There are many potential applications. For example, we could create a synthetic face through which the

machine can communicate in a ‘natural’ fashion. But it’s laborious work. Just take the eyebrows. People use them all the time to add expression to their spoken words,

If we are to communicate with computers in a natural way, it is vital

that they be able to recognize and repro-duce non-verbal signals. In the near

future we will be able to talk to a “sensitive” avatar that knows what

a raised eyebrow means.

Interview Marc Swerts and Martijn Goudbeek

16 17

Nonverbal C

omm

unication Non

verb

al C

omm

unic

atio

n

non-verbal communication is far more subtle than that, and goes beyond emotions alone. When we’re holding a conversation, for instance, how do we know so well when it’s the right moment for us to speak? We do it instinctively, but it’s really difficult to understand why. There are many variables involved: intonation, eye contact...

“And even with emotions, things are much subtler,” adds Martijn Goudbeek, a lecturer who is carrying out research into their production and perception. “It’s not difficult to recognize the basic

emotions: happiness, anger, fear and disgust. But what about a secondary one, like relief? Or tenderness? Yet they are actually much more common. So if we’re going to compile a good corpus, we’ll have to focus upon those subtle feelings.”

An affectionate computer, is that the goal? “Call it an empathetic computer,” says Swerts. “What we want is that, instead of us having to learn the computer’s language, it adapts itself to us.” Goudbeek provides an example: “Computer-generated faces or avatars. They’re a much more pleasant means of communication. In fact, they exist already. There was ‘Clippy’, for instance, the Windows paperclip, although that was really primitive.” “But,” continues Swerts, “imagine an avatar in an educational program for children – one which not only recognizes that the user has difficulty with sums, but can also offer them encouragement in a naturalistic way.” Goudbeek: “An empathetic intelligent system could be useful in other areas, too. Like crowd control, for the early identification of aggression within a group. Or in product evaluations. At the moment manufacturers use questionnaires to

but it’s extremely difficult to teach a computer to replicate exactly when and how they do that. For example, raising the eyebrows too often makes the face look neurotic.”

As Swerts succinctly puts it, “Non-verbal expression is not what you say but how you say it. That can be visually – through hand gestures, a frown, posture and so on. But it can also be audible: style of speech, intonation, variations in tempo and loudness… These signals are vital to good communication. Imagine that someone tells you his girlfriend is pregnant. It matters quite a lot how he puts the message across. Is he furious? Happy? Surprised? If you fail to pick up on that, it can leave you in an embarrassing situation even though the meaning of the words is perfectly clear. Communications between people and computers are no different. Take a call to a travel helpline that uses speech-recognition technology. It can save you a lot of frustration if the computer ‘recognizes’ from your intonation that you are annoyed and responds accordingly.”

“It is already relatively easy to identify clear-cut emotions like anger or joy,” continues Swerts. “But everyday

The computer of tomorrow will recognize emotions and non-verbal expression

18 19

measure consumer satisfaction, but they are often quite unsophisticated and respondents are inclined to provide answers they think are socially acceptable. A computer with emotion-recognition capability, on the other hand, could ‘read’ the subject’s face and speech directly.”

Although Quintilian wrote his advice on the subject two millennia ago, scientific research into non-verbal expression is still in its infancy. “Only in the past couple of years,” says Swerts, “have new digital techniques made it possible to manipulate images of facial expressions or recordings of speech with very great precision, and also made it fairly easy to present the results to a large group of assessors.” A slight difference in the position of an eyebrow, higher or lower, in two otherwise identical portraits can have a huge effect upon how the subject’s expression is perceived. “What we notice time and again,” says Goudbeek, “is how subtle and versatile non-verbal communication is. Take adaptation, for example – the habit of adjusting our behaviour according to who we are with – which is something we have been doing a lot of research on recently. We find that there are personal and cultural factors involved.” Swerts: “A person can laugh at a funeral. It takes a refined form of recognition to interpret that properly. We are not just learning a lot about artificial intelligence, but also about people themselves.”

Research portrait

Lisette Mol

Hand gestures and computers“When people talk, their hands are usually all over the place, depicting many things. Are these hand gestures part of their communication? And what do gestures tell us about the way people think? These are central questions in my research.”

“When communicating through gesture, it is of course necessary to see one another. In day-to-day communication, this is often the case. Yet what happens when you cannot see the person you are talking to, whereas they can see you? Or vice versa? Our research shows that people adapt their gesturing to what their interlocutor sees. This shows that some gestures are meant to be seen!”

“When communicating, people adapt their behavior to one another in many ways. Their gesturing too starts looking more and more alike. Yet what exactly are interlocutors aligning here? Is it their movement, or their way of thinking? We tested this by showing people gestures that almost, but not quite, fitted a particular situation: pointing out a route on a map. In response, they produced gestures that were not exactly like the ones they had seen, but entirely like one would point on a map. This shows that people did not so much copy the movements of their interlocutor, but rather the meaning of these movements. Therefore, we now know that gestures aid mutual understanding.”

“This knowledge of the use and significance of gestures is useful in developing ‘natural’ forms of distance communication, such as videoconferencing.”

“Our hands show that we really do understand the message”

You can tell whether a person is dominant or not, not only from how loudly they speak, but also from their tone of voice. #TiCC

Some emotions can be recognized better in the voice than in the face, and vice versa. This often depends on the evolutionary role of the emotion. #TiCC

Juslin, P. N., & Scherer, K. R. (2005). Vocal expression of affect. In J. A. Harrigan, R. Rosenthal,

& K. R. Scherer (Eds.), The new handbook of methods in nonverbal behavior research (pp.

65-135). New York: Oxford University Presshttp://www.affective-sciences.org/publication/684

Expressive Synthetic Speech: a collection of examples of synthetic affective speech

conveying an emotion or natural expression

http://emosamples.syntheticspeech.de/

Newstweet

Nonverbal C

omm

unication Non

verb

al C

omm

unic

atio

n

Want to know more?

20 21

Social Media So

cial

Med

ia

Not so long ago, parents worried if their kids spent too much time hanging around on street corners. “Now they’re concerned about the amount of time they spend

on Facebook or other social network sites,” says Marjolijn Antheunis, who is investigating the effect of social media on social psychological well-being and the

formation of friendships. “And even I don’t know the answer yet. We are on the threshold of a massive social change. A lot of research is needed before we’ll be

able to properly assess the impact of the network society. At this stage, though, the scientific studies seem to indicate that parents have no need to worry.”

At first sight, social scientist Antheunis looks rather odd amongst all the cognition experts at TiCC. But times like ours require a multidisciplinary approach, says

Professor of Communication and Cognition Fons Maes. “Today’s social upheavals are occurring hand in hand with a digital revolution. New technical possibilities in

communication are changing social relationships, whilst at the same time those relationships are creating a demand for new technologies.” Antheunis cites an

example: “The way social media are changing the meaning of the term ‘friendship’. We studied whether the number of friends young people have on the web bears

any relation to their actual popularity. That turns out not to be the case. But what does that actually mean? Theories about friendship all pertain to the ‘real’ world,

and there the number of friends you have is significant. It’s clear that different rules apply online. We are now in the last phase of a long-term study asking how social

network sites influence young people’s sense of well-being, measured over two years. That has not been studied before.”

For that research, Antheunis is focusing on early adolescents aged 11-14, since that is a period when friendship is essential. It is also when their parents have moral

concerns; they are worried about their children’s welfare. But there is something else, too. This generation – now extending to those in their early twenties – is the first not

to know what it is like to live without a digital network. The social upheaval is most apparent in this group, with technology and internet use developing accordingly. “For example,” Maes explains, “you can see how social media are supplanting e-mail and the blog. What was innovative yesterday is obsolete today. Yet there is one constant:

digital communications are becoming steadily more infused with ‘cues’. There is an ongoing shift from asynchronous to synchronous conversation, as described in

media synchronicity theory (MST). In other words, more simultaneous interaction – and with a number of advantages over face-to-face communication:

Social media have become an integral part of our lives, but we still know almost nothing about their impact on friendships and “real-life” social contacts. Long-

term research at TiCC is set to change that.

Interview Fons Maes and Marjolijn Antheunis

How social media are changing the world

Is a Facebook friend a real friend?

Com

mun

icat

ing

via

com

pute

r: n

onve

rbal

info

rmat

ion

is v

isib

le.

it is not confined to one place, and there are opportunities to gather more information during the conversation.“Social network sites are evolving into a ‘café plus’,” continues Antheunis. If you fall into conversation with someone in a café, you receive extra information. For example, the group of friends they are with says a lot about them. Social media go even further. They provide the dynamics of a ‘real’ conversation plus references to personal websites or holiday photos. And there is plenty more to come.” Maes suggests an example. “The eyecatcher, a device that makes it possible to look into one another’s eyes when chatting with a webcam. Recent research by our department has revealed that digital communication now very much resembles its face-to-face equivalent.”

“And that’s interesting for me as a social scientist,” says Antheunis. “What are the implications of this kind of more naturalistic communication? It is known, for instance, that people who meet online tend to like one another more than when they meet offline. But will that still apply when they can make eye contact?” “Or,” asks Maes, “if computers can provide additional information such as details of the other person’s mood? From the research carried out at TiCC, that’s not unthinkable.”

The new social relationships are also going to change marketing. Here lies another unexplored area of research: how does marketing work on social media? That social aspect, after all, makes them particularly well-suited to marketing activities. As Antheunis explains, “Brands like Apple and Coca-Cola know that it’s beneficial to have consumers attach intangible values to your product, like ‘friendship’ or ‘coolness’. That creates what‘s known as brand loyalty or brand engagement. In partnership with [Dutch networking website] Hyves and others, we have investigated whether social media can play a part in that. For example, how important is the relationship with the sender of the message? Our reserach has shown that a strong tie positively affects the value attached to a product or a campaign. Another timely question that we are currently investigating is: do consumers mind if companies make use of private information on their personal pages? Say you rave about a band online – do you then want to be presented with ads for its new CD?”

“Increasingly,” concludes Maes, “research into communication mediated by digital networks is focusing on the subtleties involved. The research at TiCC lays bare things that are very common in everyday life. Subtle emotions, gestures, eye contact… Therein lie the secrets of communication.”

22 23

Social Media So

cial

Med

ia

Research portrait

Marie Postma

The social signals in human voice“When a company in a crisis situation selects a spokesperson to inform the general public, what should the person sound like? Our study has shown that a low-pitched female voice and a high-pitched male voice are the most appropriate. A low-pitched male voice comes across as too dominant – and is considered unfriendly.” “My research focuses primarily on auditory prosody, that is, audible non-verbal cues such as intonation, timbre, tempo and volume. The use of these vocal properties has a number of social functions. For example, people adapt their voice to each other to show that they belong together or that they want to cooperate with each other. Vocal behavior also enables us to deduce the social hierarchy of participants in a conversation and how the speakers are feeling.

Another line of my investigation involves individual differences in the perception of speech. There is a significant individual variation in how people perceive complex sounds such as speech or music. Due to discrepancies in our neuro-cognitive architecture, we may respond differently to the same musical piece and give preference to different musical instruments. Both the left and the right hemisphere process auditory signals, but the way in which they are involved varies from one person to another. A greater right-hemisphere dominance in pitch processing leads us to focus more on timbre, whereas a left-hemisphere dominance results in attention to basic melody and rhythm. Studies have shown that listeners of the second type are better able to recognize other people’s emotions.”

The number of friends on social network sites is no indication of popularity. #TiCC

Newstweet

People get to like each other more quickly online than offline. #TiCC

Want to know more? Antheunis, M.L., (2010). Getting Acquainted

through Social Networking Sites: Testing a Model of Online Uncertainty Reduction

and Social Attraction. Computers in Human Behavior, 26, 100-109.

http://www.sciencedirect.com/science search for author: Antheunis

24 25

Hum

an Com

puter Interaction Hum

an C

ompu

ter

Inte

ract

ion

Able to understand the finer points of language, and easy to use too

The computer must adapt to us, not us to it

What do we want from a computer? Emiel Krahmer can sum it up: “A machine that adapts to us, not one that we have to adapt to. The ‘Holy Grail’ is a computer

that understands our language, rather than us having to learn computer speak.” Suleman Shahid has even higher expectations. “We want a computer to be valuable

not just in enabling interpersonal communication, but valuable to society in its own right. For example, intelligent computer systems could help the elderly and

disabled to live better lives or support all of us in leading a more sustainable existence.”

Krahmer and Shahid are researchers working on human-computer interaction, a discipline dedicated to creating a friendly, helpful machine that communicates with

us in the most “natural” way. A computer with human traits, in other words, and one capable of performing all kinds of useful tasks for us. One possible solution to that challenge lies in the further development of avatars, computerized characters

with whom you can communicate in the same way as with a real person. According to Krahmer, they represent a promising line of investigation from a technical point of view. “Computing power has increased so enormously in the past few years that

it is now becoming theoretically possible for machines to understand and imitate the finer points of human communication. The problem is this: what exactly are

those points? And how do we explain them to a computer?”

In fact, two separate approaches are required. The first is for us to actually understand all the secrets and subtleties of human communication. That

is Krahmer’s specialization. “It’s called psycholinguistics,” he explains. “A combination of psychology and communication science. The research focuses

upon how people produce and comprehend language, both verbal and non-verbal.”

Meanwhile, Shahid’s work is homing in on the computer itself. At its heart is the creation of user-friendly hardware and software. For example, how can an

application remind a forgetful elderly person to switch out a light, say, or to eat

Computers that are easy to use, help improve our quality of life and can

understand the subtleties of human communications – that is the way we

need to go.

properly and healthy? “In the near future,” he says. “this so-called user interface design is going to become an increasingly common part of our everyday lives. But to do that it has to satisfy a number of preconditions. The system must be fast and easy to use, and not too prominent. It should sense the user and their needs, as it were.”

And therein lies the problem, continues Krahmer. “Take automatic language generation, for example. In other words, technologies that translate information into speech. They certainly make the computer user-friendlier, as in the

case of a device that gives you spoken directions as you walk through a city. But such a system is very difficult to develop. It requires a lot of research, especially if you want it to respond naturally to what the user sees and says. Let me give you an example. Every day we describe things almost without thinking about them. If someone asks us the way, for instance, we might say, ‘Turn right at the building on the corner’. But how do we get the computer to reproduce that kind of instruction in a natural fashion? One way, you would think, is to teach it to give an exact description. One with no superfluous information, but just the facts needed to distinguish that one building from all the others in the vicinity. In practice, though, people tend

Interview Emiel Krahmer and Suleman Shahid

to provide far more detail than is strictly necessary. We don’t actually say ‘the building on the corner’, but ‘the old

house on the corner, with the big windows’. In other words, we overspecify. And that’s something it’s difficult to teach a

computer to replicate.

“Another issue, as highlighted in research we completed only very recently, is paraphrasing. That’s the human ability to

express exactly the same thing in quite different ways. Compare these two sentences: ‘Turn right at the church and walk down

to the bridge.’ And: ‘When you get to St Mary’s, take a right and then carry straight on until you reach the river.’ Both give exactly

the same directions, but using totally different terminology. We’ve just released a dataset containing more than two million words, to help automatically identify and generate paraphrases. This is the only dataset of its kind, and should help to produce

more varied and more human descriptions of routes.”

“Research shows that people are very apt to ascribe computers with all kinds of human qualities,” continues Shahid. “A machine may not respond in the same way as a person, but often we rather expect it to. That’s why research into a so-called ‘user-centred’ intelligent system is so important. Moreover, I’m convinced that this work is going to help make the world a better place. It’s partly

for this reason that it fits so well into the activities of a school of Humanities: it’s showing how computers can play their part in putting human values into effect. We see a good example of this

in the applications students have to develop for final assessment of the undergraduate course in User Interface Design. These must either support a sustainable lifestyle, take over simple care

tasks, offer instructive entertainment or provide personal assistance. Results have included an app to measure domestic energy consumption, an electronic diary for autistic children and a set of games to make kids more environmentally aware. They’re all small things, but they show how

technology can contribute to a better world.”

“And even if the trend towards making computers more human were to stall tomorrow,” adds Krahmer, “then it would still all have been worthwhile. After all, building them is perhaps the best

way of understanding people better.”

Why do I have to learn how a computer works? Why can’t it learn how I work? #TiCC

NewstweetProject Bridging the gap between

psycholinguistics and computational linguistics: the case of referring expressions.

http://bridging.uvt.nl/

26 27

Hum

an C

ompu

ter

Inte

ract

ion H

uman C

omputer Interaction

Want to know more?

The Bachelor’s course in User Interface Design (UID), taught by Suleman Shahid, gives students an introduction to the underlying theory of interface design, and provides them with the skills to put the theory into practice. During this course, students develop apps designed to make everyday life easier and more enjoyable. Here are two examples.

The Augenda The Augenda is a digital diary for young people with autism, which was developed by five students. The prototype can help these young people to structure their lives by giving them checklists and appropriate reminders. It also uses emoticons because previous research has shown that those with autism respond much more easily to visual stimuli. Parents and carers can also be given feedback. The students were invited to present their project at the renowned Assets conference in Orlando.

User Interface Design for a better life

The Green Life Guide The Green Life Guide, which was also developed by five students, is a smartphone application which helps people to have a more sustainable lifestyle. The application has four green functions: location-based services, cooking, news, and transport. Users receive immediate feedback on the number of trees that have been saved by making use of more sustainable services.