Kinaesthetic Thinking

7/30/2019 Kinaesthetic Thinking

http://slidepdf.com/reader/full/kinaesthetic-thinking 1/20

To appear in: Computers in Human Behavior , Vol 13, No. 4, pp. 443 - 463, 1997, Elsevier 1997

Kinaesthetic Thinking:

The Tacit Dimension of Interaction Design

Dag Svanæs

Department of Computer and Information Science

University of Trondheim, NTNU

7055 Dragvoll, Norway

e-mail: [email protected]

home page

Abstract

Designers of computer-based material are currently forced by the available design tools to express interactivity with concepts derived from the logical-

mathematical paradigm of computer science. For designers without a training as programmers this represents a barrier. The three psychological experiments

presented indicate that it is possible to express interactive behaviour in a more direct fashion by letting the designers compose software from interaction

elements with built-in behaviour. The resulting «kinaesthetic thinking» of the software designers shows similarities with visual and musical thinking. To be

able to support this style of design it might be necessary to rebuild from scratch parts of today's software using simple interactive building blocks. As an

illustration, a design tool is presented, based on a pixel-level agent architecture.

INTRODUCTION

On most popular personal computer platforms a wide variety of multi-media tools are currently available. These are easy to use, require little or no skill in

programming, and range from editors for pixel graphics and animation, to tools for integrating the different media resources. Most of the tools provide

excellent support for graphics, sound and video. The problems arise when designers want to be creative concerning interactivity. If a designer wants to create

interactive solutions that were not imagined by the tool makers, he or she has to make use of a scripting language like HyperTalk or even leave the tools alltogether and do programming in a programming language like C++ or Java.

Kinaesthetic Thinking http://www.idi.ntnu.no/~dags/papers/chb97.html

1 di 20 22/02/2012 10.37









Kinaesthetic Thinking http://www idi ntnu no/~dags/papers/chb97 html



«mouse button release.» All examples were deterministic. The subjects were offered an additional «repeat» button that enabled them to bring the FSAs back

to their initial state whenever they liked.

Figure 2. The FSAs as they were presented to the subjects.

Figure 2 shows a snapshot from the screen with a two-square FSA. The FSAs can be described formally as an initial state and a set of production rules. The

complexity of the 38 FSAs range from one-state FSAs with no transitions to a 8-state FSA with 24 transitions. The FSAs were presented in an increasing

order of complexity, starting with all the FSAs with only one square, then all the two square FSAs, and last all the three-square FSAs.

In all three experiments the user worked with a Macintosh IIci personal computer running SMALLTALK/V.

Procedure

The subjects were instructed to freely describe the examples as they explored them. They were explicitly told that the experiment was set up to learn about

how people experience interacting with computers; not to learn about how good that individual subject was at reaching some answer.

Apple's Usability Testing Guideline (Tognazzini, 1992) was followed rigorously when setting up the experiment. This includes instructing the subjects on how

to think-aloud, explaining to them beforehand the purpose of all the equipment in the room, and telling them that they had the right to stop the experiment at

any time without loosing their pay if they for any reason felt uncomfortable.

The 38 examples were presented to the subject one at a time, and the subject was given control of the progress of the experiment. As seen in Figure 2, a

«next» button was put on the screen for this purpose. Care was taken not to induce in the subject any preconceptions about the nature of the examples they

were to explore. The experimenter referred to the examples by pointing to «that there» on the screen. The experimenter sat by the side of the subject

throughout the session, and was available for technical assistance.

The sessions were videotaped. The camera was pointed at the screen, and the subjects were aware that they themselves were never filmed. An external

directional microphone was used to increase audio quality.


6 di 20 22/02/2012 10.37

Kinaesthetic Thinking http://www idi ntnu no/~dags/papers/chb97 html



In addition all interactions were logged with time stamps that made it possible to reproduce the interaction in detail. The resulting four hours of taped material

was then transcribed together with the non-verbal interactions.

Results

All subjects got engaged in the task very quickly. On some occasions they asked the experimenter whether they were doing the right thing, but they never lost

concentration. With a total of 3 exception, all FSAs were explored and described by all subjects. The subjects all controlled the progress of the experiment in

a very natural way. It seemed as if at a certain point in the exploration of a FSA; it was completely «understood» , and the subject was ready for the next one.

All subjects developed a good skill at exploring the FSAs. In some cases complex FSAs were fully «grasped» in a few seconds. They often had a much harder

time describing them. One subject put it: «It is easy to do, but hard to explain.»

The subjects often described a FSA as being identical to some previously tested FSA. The FSAs were also described as modification of previous FSAs as in:

«It is like that other one, but when …» What was being compared were the actual FSAs as they were experienced in interaction, and not some decomposed

understanding of the interaction. A possible term for the FSAs as perceived reality could be Interaction Gestalts.

Lakoff & Johnson's (1980) theory of metaphor was used as an inspiration for doing an analysis of the implicit metaphors used by the subjects in describing

interactive behaviour. To show how this analysis is done, an example is provided.

Figure 3. The state transition diagram for an FSA.

Figure 3 shows the state transition diagram of a three-square FSA. The squares are numbered from the left. The FSA starts out in the colour combination

white-black-white. It has four states. The colour of the rightmost square can be changed by clicking on it. This square has «toggle behaviour.» When the

rightmost square is black, it is possible to swap colour between the leftmost and the middle square by clicking on the one being white.

One subject (female) described this behaviour as:

« ...OK, you have to begin with the one to the right to be able to do anything ... then you can move that one [square 1] ... and turn the one to the right on andoff ...»

Kinaesthetic Thinking http://www.idi.ntnu.no/ dags/papers/chb97.html

7 di 20 22/02/2012 10.37




We see here that she sees the FSA as consisting of a switch in position 3 which can be turned on and off, and a white square that can be moved between

position 1 and 2 whenever the switch is «on.» Her mental model of the FSA is illustrated in figure 4:

Figure 4. A graphical representation of a mental model.

A total of ten implicit metaphors were identified in the material. The first four of these were:

The FSA consists of squares that change colour. This is the implicit metaphor in a «mathematical» description of the FSA. For example: «The white

square gets black.»

The squares are white areas that can be coloured with black paint. Examples: «It gets coloured,» or «the colour is removed.»

A square is a switch that can be off or on. Example: «It goes off when I click on it.»

The FSA or some part of it is locked or dead . Example: «I cannot change it; it is locked.»

The last six metaphors all describe movements in some space. The object moving is either the user as in: «then I move to the left,» or the FSA or some part of

it as in: «the white square moves to the right.» The three spaces are:

Cartesian 2D space. This is the most obvious space and the one projected onto the behaviour of squares 1 and 2 in the above example.

State space. In a lot of cases the behaviour was described as either the user or the FSA moving in the state space of the FSA.Linear time. In some cases the behaviour was described in terms of linear time.

g p g p p

8 di 20 22/02/2012 10.37




Table 1 shows the six combinations with example quotes:

Table 1. The six metaphors involving spatialisations.

These ten metaphors were often used in combination as exemplified by the example.

EXPERIMENT II: TESTING A DESIGN TOOL BASED ON EXPERIMENT I

g p g p p

9 di 20 22/02/2012 10.37




Figure 5. The Interaction Gestalt editor.

To test the practical usefulness of the idea of Interaction Gestalts, an experiment was set up. The editor shown in Figure 5 is based on the idea of Interaction

Gestalts and was created for the experiment. The user here works directly with interactive single-square FSAs constructing complex FSAs from a set of four

basic building blocks with built-in behaviour. Each FSA is treated as a separate entity that can be cut, copied and pasted. A set of unary and binary operations

have been defined on the FSAs. The editor can be seen as implementing an algebra of interactive single-square FSAs.

The tool was tested individually on three high school students (ages 16-17) with no background in programming. They first ran through the examples of

experiment I to learn about the design domain, and were then asked to reproduce some of the FSAs using the tool.

10 di 20 22/02/2012 10.37




All subjects were able to reproduce all problems given. They worked directly with interactive user-interface elements in this non-verbal manner, constructing

complex new widgets from the set of elementary FSAs.

EXPERIMENT III: DESIGNING A NEW TOOL FROM SCRATCH

There is a strong tradition in Scandinavia for involving end-users directly in the design of software through techniques of participatory design. In the third

experiment the design process is used as a gateway to the naive theories of the target domain. The approach consisted of letting a group of end-users

construct a tool for building FSAs through techniques of participatory design with iterative prototyping.

Subjects and procedure

The subjects were 5 high school students (ages 16-17) with no programming background. As in experiment II they first individually went through the same

FSAs as in experiment I to get acquainted with the problem domain.

They were then as a group presented with the design problem and asked to come up with as many design ideas as possible. This brainstorming lead to an

initial design that was prototyped by a programmer and tested in the following session. Modifications and extensions were incorporated into the next

prototype, and this iteration was repeated five times until the group was satisfied with their result.

The subjects were not exposed to any tools or design ideas before or during the design process. They consequently had to use their own conceptions about

the domain as basis for their design.

Results

The group designed an advanced tool incorporating ideas similar to those leading to experiment II. The final version of their editor can be seen in figure 6.

11 di 20 22/02/2012 10.37




Figure 6, The result of the design process

It allows for the construction of two-square widgets. The design group identified a set of 10 elementary one-square widgets (FSAs) which they listed on the

left. They further defined operations and modifiers on these interactive building blocks. When asked to reconstruct some given widgets with their tool, they

reasoned directly with their elementary interactive widgets.

Figure 7. State-Transition Diagram for the widget

This widget shown in figure 7 consists of two squares, the left initially white, and the right initially black. When you press the mouse button in any of the

squares, the colours change. This creates an illusion of spatial movement.

12 di 20 22/02/2012 10.37




Their «language game» evolved in parallel with the evolution of the prototype in what could be called a language-artefact cycle.

This widget was described:

«I think it is a five [widget #5] and a seven [widget #7]."

"No, it is a seven and an eigth with both-react [a modifier]."

Figure 8. "A seven and an eigth with both-react"

Figure 8 illustrates graphically how the widget was conceptualised by the subjects. They saw it as consisting of two single-square widgets with toggle

behaviour, the left initially white, and the right initially black. These single-square widgets were available as building blocks in their tool (widgets #7 and #8

respectively). They further added their property "both react" which makes both widgets sensitive to the same input. This gives a two-square widget with the

required behaviour.

DISCUSSION

From a computer-science perspective the editors of experiment II and III represent hybrid solutions that introduce an unnecessary level of abstraction on top

of the underlying internal state-transition formalisms. From a user centred (phenomenological) perspective it seems that simple widgets with built-in

behaviour (perceived as Interaction Gestalts) are natural atomic objects for this problem domain. The state-transition formalisms normally used for describingGUI behaviour rest on an abstraction of linear time with stable states. When designing interactive behaviour directly and non-verbally with Interaction

Gestalts, there seem to be no need to involve abstractions of time.

If we contrast these results with the dominant programming paradigm in computer science (Object-Oriented Programming, or OOP), we find some striking

differences concerning the conception of time and concerning the nature of objects.

Time

13 di 20 22/02/2012 10.37










Avoid forcing the designer to work in a sense modality different from the sense modality of the product. For interaction design, this means providing

tools that enable the designer to express the interactive aspects of the product directly through interaction.

Avoid representations of behaviour involving abstractions of discrete time and states.

Provide building blocks with inherent behaviour, and implement algebras that allow for the construction of complex behaviour from these simple

interaction elements.

Avoid a "run" mode different from the "build" mode.

APPLICATION: PAINTING WITH INTERACTIVE PIXELS

In an attempt to try out the ideas presented here, a very first prototype was made of a design tool that enables interaction designers to construct graphical

user interfaces by "painting" with pixels with inherent behaviour. I built this prototype to show the basic functionality of a possible new class of design tools.

Informal user tests have been done which encourage me to continue developing the prototype along the lines outlined here. The basic idea is to allow the

designer to construct interactive behaviour directly at the pixel level with "tools" resembling tools in pixel-based paint programs like MacPaint. The prototype

has strong similarities with AgentSheets (Reppening 1993). In AgentSheets, the agents are represented as icons on a grid. Here, I have taken this approach to

an extreme by letting each pixel be an interactive agent communicating with its neighbouring agents (i.e. pixels).

The functionality of the prototype is best shown through an example. Figure 9 shows an example of an interactive drawing which was constructed with the

prototype. It illustrates an on/off switch which controls a light bulb. The switch has toggle behaviour.

Figure 9. An interactive drawing.

To construct this interactive drawing in black & white , four additional "colours" were needed as illustrated in figure 10:

White toggle. This "colour" is initially white, and goes black when clicked on. It returns to white the next time you click on it.

Black toggle. This "colour" is the reverse of the previous one. It is initially black, and goes white when clicked on. It returns to black the next time you

click on it.

White and black connectors. These "colours" are used to connect interactive pixels without changing their own colour.

17 di 20 22/02/2012 10.37




Figure 10. An interactive drawing in the making.

A runtime system takes care of making adjacent interactive pixels share input. This makes it possible to construct large areas, like the switch in the above

example, which behave as interactive objects. Interactive objects can similarly be split into separate interactive objects simply by taking them apart on the

canvas.

CONCLUSIONS AND FUTURE WORK

The present study shows that users with no formal training in programming are able to reason about the behavioural aspects of GUIs without breaking the

interactions down into discrete events. I have introduced the term Kinaesthetic Thinking to signify the modus operandi involved when interactive behaviour

is designed directly in the kinaesthetic sense modality, i.e. without making use of textual or visual representations of behaviour. This is the "visual thinking" of

interaction design.

Kinaesthetic Thinking is poorly supported by the current user-interface design tools. Great benefits should be expected concerning both productivity,creativity, and job satisfaction from building design tools that better support this artistic way of working with interactive behaviour.

It is to me an open question how the psychology of Kinaesthetic Thinking should be studied further. How should the experiments be set up, and how should

the results be interpreted? Will it be possible to identify simple gestalt principles in the kinaesthetic domain similar to those in the visual domain? If so, will

these principles be as applicable to interaction design as the gestalt principles of the visual domain and the insights about visual thinking have been to

graphical design?

I plan to develop the presented prototype into a working design tool together with a representative user group. A first step will be to develop a version of the prototype that enables kids to add interactivity to their computer drawings. I plan to develop this tool in Java and make it publicly available on World Wide

18 di 20 22/02/2012 10.37




Web to gain practical experience with tools supporting Kinaesthetic Thinking.

To extend this approach from the well defined domain of the experiments to the domain of today's complex software, we are faced with a trade-off between

visual elegance and flexibility. To be able to give the end-users back the control of their tools, we might have to rebuild today's software systems from scratch

using simple interactive building blocks that fit together in simple ways. By doing this we will probably lose some of the elegance of the current software, but

we will gain a new kind of flexibility. The situation is very similar to the difference between a toy built from LEGO bricks and other toys. Most ordinary toyshave only one use, and when the kids are tired of them they are thrown away. A LEGO toy can be taken apart, recycled, modified, and extended for ever.

The drawback is of course that a toy made of LEGO bricks looks much less like what it is supposed to mimic.

It is currently an open question how the market would react to user interfaces and application that have different look and feel than today's software, but

which can be easily opened up, studied, understood, modified, and re-used by most ordinary users.

ACKNOWLEDGEMENTS

Thanks to J. Sphorer, S. Houde, A. Cypher and numerous others at Apple Computer who gave me the opportunity to discuss these ideas during my sabbatical

in their Advanced Technology Group. Also special thanks to W. Verplank, F. Nake, A. Repenning and A. Mørch for fruitful feedback.

References

Andersen, P. B. & Madsen K. H.(1988) Design and professional languages. In P. B. Andersen & T.Bratteteig (Eds.), Computers and Language at Work ( pp.

157-196.), Oslo: SYDPOL, University of Oslo.

Andersen, P.B, Holmqvist, B., & Jensen, J.F. (1993) The computer as medium. Cambridge: Cambridge University Press

Arnheim, R. (1969)Visual Thinking. Berkeley, CA: University of California Press.

Cypher, A. (1993). See What I Do, Boston: M.I.T. Press.

Dreyfus, H. L., (1991) Being-in-the-world: A Commentary on Heidegger's Being and Time, Division I. Cambridge, MA: MIT Press.

Itten, J. (1973). The art of color: the subjective experience and objective rationale of color ,. New York: Van Nostrand Reinhold.

Johnson, M.(1987) The body in the mind . Chicago: Univ.. of Chicago Press.

19 di 20 22/02/2012 10.37




Kandinsky, W. (1994). Complete writings on art . New York: Da Capo Press.

Laban, R. (1988) Modern Educational Dance. Plymouth: Northcote House.

Lakoff, G. , Johnson, M.(1980). Metaphors we live by. Chicago: University of Chicago Press.

Lakoff, G. (1987).Woman, Fire, and Dangerous Things. Chicago: Univeristy of Chicago Press, .

Martin, F.G. (1995) A Toolkit for Learning: Technology of the MIT LEGO Robot Design Competition. MIT.

Merleau-Ponty, M. (1962). Phenomenology of Perception. Colin Smith (transaltor). New York: HUmanities Press.

Newell, A., Simon, H.A. (1972.) Human Problem Solving. Cliffs, N.J.: Prentice-Hall, Englewood

Norman, D. (1993)Things that make us smart. Reading, Ma: Addison-Wesley, .

Papert, S. (1992) The Children's Machine. New York: Basic Books.

Polyani, M. (1966) The Tacit Dimension. London : Routledge & Kaufman Paul.

Repenning, A. (1993) Agentsheets: A Tool for Building Domain-Oriented Dynamic, Visual Environments. Colorado: Ph.D. thesis, Dept. of Computer

Science, Univerisity of Colorado, .

Schön, D. (1983) The Reflective Practitioner. How professionals think in action. London: Basic Books.

Tognazzini, B. (1992). TOG on Interface, pp. 79-89, Reading MA: Addison-Wesley.

Turkle, S. (1984). The second self: Computers and the human spirit. New York: Simon & Schuster.

White, B.Y. & Horwitz, P. (1987). Thinker tools: Enabling children to understand physical laws. (Report No. 6470). Cambridge, MA: BBN Laboratories.

20 di 20 22/02/2012 10.37

Documents

Kinaesthetic Thinking