What active users and designers contribute in the design process

What active users and designers contribute

in the design process

E. Olsson*

Department of Information Technology, Uppsala University, Box 337, 751 05 Uppsala, Sweden

Received 8 October 2003; revised 20 January 2004; accepted 27 January 2004

Abstract

With the hope of creating usable systems, we declare repeatedly that users should be involved in

the design and development of computer systems, without questioning the reasons and motives

behind this declaration. What, in fact, can users contribute to design and how can we best include

their contributions in the development process in order to produce usable computer systems?

This paper presents a study of the hands-on work of one group of designers and one group of user

representatives (in this case marine captains) on a given design task. The groups met on separate

occasions. The aim of the study was to present a qualitative analysis of the potential contributions to

design by user representatives compared with interaction designers.

The results are discussed in terms of methods and techniques that sanction the use of a particular

domain-specific vocabulary, giving advantages to those who have a good command of that

vocabulary. In addition, the study discusses how users’ narratives may reveal qualitative domain

knowledge that could function as the glue that keeps users and designers together in the design process.

q 2004 Elsevier B.V. All rights reserved.

Keywords: User involvement; User participation; User-centered design; Participatory design; Design decisions;

Domain knowledge

1. Introduction

1.1. Active user participation

Users active participation in the design of work systems has been the focus of

research and discussion for several decades now, see for example, Kujala’s (2003)

0953-5438/$ - see front matter q 2004 Elsevier B.V. All rights reserved.

doi:10.1016/j.intcom.2004.01.001

Interacting with Computers 16 (2004) 377–401

www.elsevier.com/locate/intcom

* Tel.: þ46-18-471-28-81; fax: þ46-18-471-78-11.

E-mail address: [email protected] (E. Olsson).

http://www.elsevier.com/locate/intcom

discussion on the benefits and drawbacks of the various existing approaches. Although

the benefits of user-centered systems design (UCSD) and participatory approaches are

acknowledged (Muller et al., 1997; Schuler and Namioka, 1993), the application of

such methods by practitioners is limited (Gould et al., 1991, Gulliksen et al., 2003). A

survey among usability experts attending CHI and UPA conferences revealed that

although ranked far below participatory design (PD) in efficiency, heuristic evaluation

was the most common method used (Rosenbaum et al., 2000). Similar findings have

been confirmed in studies that are more recent, where the most popular usability

activities were distant from active user participation; they included usability testing,

prototyping and heuristic evaluation (Gunter et al., 2001; Vredenburg et al., 2002). The

projects studied by Clement and Besselaar (1993) in their retrospective study had a

short-term focus and critical resources were withdrawn upon completion of the projects.

These conditions influenced the lack of self-sustained processes of local participation to

continue within work settings, but Clement and Besselaar (1993) concluded that the

main reasons were related to organizational inertia and resistance. A number of other

reasons might explain the low acceptance and sporadic use of PD in real life, a few of

these are discussed next.

A common criticism against PD is its imprecise definition of the concept of

participation. Muller et al. (1997) point out that techniques described as

participatory—where the users merely constitute a source of information and therefore

may be exploited, objectified or manipulated—may turn the whole process of

participation into an illusion. They refer to cases where the authors claim that they

have performed PD activities and yet in fact, no users were involved. Active

user participation is one of the key principles in user-centred systems design

(Gulliksen et al., 2003).

A well-defined user population is clearly an advantage in achieving user involvement

(Grudin, 1991), but it might be difficult to define the users. Carmel et al. (1993, p. 40) put

forth that “an unambiguous definition of user is impossible” and claim that the main

difference between the many different participatory approaches is the degree to which

users are allowed to participate in the project. Consequently, different approaches,

dependent on specific settings and conditions, offer many varying definitions.

Furthermore, it is both difficult and requires great effort to succeed in involving users

representing all potential users of a specific system. If, for instance, the top management,

middle-level managers, and workers are all potential users of the same system, they

probably need to have access to different parts of the system and different views of the

information in the system. If their respective requirements must be fulfilled by the same

system, the task of involvement can be perceived as cumbersome and demanding from the

developer’s perspective.

1.2. Users as informants or co-designers

Involving users in analysis and evaluation is relatively uncontroversial but there is a

variation in exactly how much one wants to involve the user in the design work, and the

way such intentions are expressed (Fig. 1). On the one hand, users are considered as

informants (Constantine and Lockwood, 2002; Scaife et al., 1997) who can supply facts

E. Olsson / Interacting with Computers 16 (2004) 377–401378

about work procedures, but who—in most cases—have no HCI or design knowledge and

therefore in general, should have little to say about particular design issues (Webb, 1996).

On the other hand, there are user representatives that participate for years in user groups as

sources of information for design projects. These users run the risk of becoming

professional domain experts, developing a new career detached from actual work

procedures.

Rather than increasing user involvement in the development process, there is a shift

from full involvement to users being considered informants or even subjects (as in objects,

subjected to observational procedures). Many commercial development processes, e.g.

RUP, do not encourage user-centered design as defined by Gulliksen et al. (2001, 2003).

This definition has been adopted in this paper. In fact, usage-centered design (Constantine

and Lockwood, 1999) prescribes selective user involvement in contrast to the substantial

user involvement recommended in user-centered design. Users continue to serve merely as

information sources answering specific questions that may arise; they still have little or no

influence on the design. Constantine and Lockwood (2002) even suggest excluding or

minimizing user involvement. They maintain the view that “user studies may easily

confuse what users want with what they truly need”.

Cooper (1999) makes a case for the use of Personas, instead of letting programmers

decide what users need. In his opinion, the ‘user’ expression should be eradicated, and

instead Personas should serve as a safeguard against developers considering themselves

as users of the system. Although this perspective might be useful in preventing

developers from maintaining false ideas about the users, it does not safeguard the

interests of the users.

When a process of change is initiated, it is often difficult to envision what the future

work situation will be since people are occupied with the current work situation and its

inherent routines. Greenbaum and Kyng (1991) argue that PD in many ways can be

transformed into a trap for the future users that participate in the development process.

It is not possible for them to deliver ‘upfront’ the appropriate demands on the design,

since they are not aware of all possibilities, and furthermore, new possibilities can

initiate the development of current work practices (Vicente, 1999). Beyer and

Holtzblatt (1998) even suggest that reorganizing work is equally important as creating

new computer systems.

Fig. 1. Degrees of user involvement. The figure shows how users can be given the role of anything on a scale from

active empowered partners, to active agents without power, and finally to passive subjects to be observed. The

order of the labels picked from concurrent literature is not conclusive, merely one way of categorizing different

ways of focusing on users in systems development.

E. Olsson / Interacting with Computers 16 (2004) 377–401 379

Design, as it is discussed in this paper, does not merely refer to design of the user

interface. The design process involving users is also a way of eliciting knowledge about

the domain and the work practices, and thereby provides essential requirements on the

future support system. By identifying and discussing work practices in the design process,

the process will also include job redesign.

It is not uncommon that the participating users end up in a ‘hostage’ situation. When the

system is ready and the result is unsatisfactory, the users that have been involved in the

development process are assumed to have had a major responsibility for the outcome.

Symon (1998) gives a good picture of designer-user interaction in in-house development

where the developers orchestrated the design decisions, but nevertheless claimed that the

user representatives made all the decisions. One reason for such misconceptions may be

that many design decisions are implicit; they just emerge as a result of somebody doing

something. Explicit design decisions on the other hand, are often made in, e.g. modeling

sessions with users (Boivie et al., 2004).

1.3. The background to this study

Previous HCI research prescribes that real users and their needs must be in focus early

on and continuously in development projects that aim to produce usable systems (Bjerknes

et al., 1987; Gould et al., 1997; Norman and Draper, 1986; Gulliksen et al., 2003). Such a

perspective was adopted in the TRAIN project (Kecklund et al., 2003) that has inspired the

study reported here. Knowledge about the tasks involved in train driving was gathered

through extensive field research. Next, a group of train drivers was assembled in order to

test hypotheses derived from the fieldwork, and at the same time put user-centered design

work into practice in the development of a new user interface. At five meetings of one or

two days, this group collaborated with researchers on the interface design. The origins of

the present study were the researchers’ reflections on the work carried out together with

the train drivers. The below example illustrates how the interaction designer thought and

worked as compared to the train drivers.

The group had agreed on the location and rough appearance of information about

stations and timetables on the display. The designer spent hours on moving information

items two pixels back and forth in order to make, e.g. station signatures fall out nicely

one after the other in an aligned column, intertwined with information from the

timetable and critical safety information, as the train traveled forward. Different fonts

and sizes were evaluated as well as surrounding frames of different size, thickness, and

color, in combination with opaque or semi-transparent backgrounds. These efforts were

made to find a suitable symmetry in the design at the same time that was efficient and

pleasant for the viewer. The basis for these concerns might possibly be derived from

cognitive psychology, in terms of the perception and interpretation of information, and

from HCI, in terms of readability, concurrence, effectiveness, and satisfaction with the

display.

The train drivers group, on the other hand, discussed for hours how the control system

worked and the unacceptable result presented on their displays. They often enhanced their

explanations with vivid descriptions of particular places where the infrastructure revealed

this—often odd—behavior.


While the drivers group got enthusiastically involved in lengthy discussions about their

work, the researchers—in their interaction designer role—tried to focus on design

problems and to make decisions within the framework of the given time schedule. From

that point of view, the need for evidence of progress was obvious and ever-present.

Although the available time was limited, the participating drivers were given more time to

develop their thoughts on their work than is common in development projects, and they

were pleased with the task they accomplished and would have been more than willing to

continue their work.

The idea of investigating what users and developers, respectively, contribute to design

developed from an urge to delve deeper into the complexity of user participation and to be

able to continue recommending user participation not only as a matter of routine.

The following assumptions had evolved as a result of cumulative observations and

active experiences and they now developed into the framework for the qualitative study

reported below.

† User representatives focus on domain issues, and current work practices.

† Interaction designers fall back on their HCI knowledge and focus on typical HCI issues.

† Interaction designers spend more time on the appearance and presentation, in

comparison with user representatives who concentrate on functionality and

effectiveness.

2. Method

The aim of the study was to present a qualitative analysis of the potential contributions

to design by user representatives compared with interaction designers. Most development

projects involve a number of different parties. In an attempt to simplify matters somewhat,

the present study concentrates on users and designers—the latter often having the role of

system developer as well, thus ‘designer’ is used in the text although developer might have

been equally appropriate in some places.

Two methods were used to gather qualitative data, video recordings from group

discussions and individual questionnaires.

2.1. Participants

Two independent groups were gathered to carry out a design task on separate

occasions. Three designers with between 8 and 16 years of experience in interaction

design formed one group, the designers group. The user representatives group consisted

of four captains, teaching at Kalmar Maritime Academy, with between 10 and 35 years

of maritime experience. The user representatives had no prior knowledge on or

experience with design work.

2.2. The design task

The design task was to integrate and improve two existing display designs. The new

display would be installed as the main display used by captains in the daily work on


the ship bridge of a high-speed craft. The groups were instructed to present, by the end of

the day, a rough draft of their suggested solution and a shortlist of prioritized information

elements in their design solution understandable to a maritime professional.

The given displays were:

† the ‘conning’ display, which usually contains status information about the ship, and at

times, also presents information about forces exerted from propulsion and external

forces that affect the ship

† the electronic chart display (ECDIS), with radar and target information overlay.

The integrated display should show the present situation, with potential hazards related

to collision and grounding clearly observable, as well as give an idea of how the present

situation might develop over the next few minutes, given, for example, the current speed,

heading, rate of turn and the surrounding traffic situation. The groups had complete

freedom to add or remove information elements as they saw fit.

The members of the groups were also instructed to respect each other’s opinions, to let

everyone speak freely, and to restrain from an immediate dismissal of each other’s ideas

and suggestions while they worked in the group. Finally, the groups worked for

approximately 4 h, with a 1-h break for lunch.

The entire group work session was recorded on videotape, and although an observer

was present from time to time (exchanging videotapes), it was assumed that the groups

would work on their own and accomplish the given task in the manner they wished, using

the supplied reference material, whiteboard, OH-film, pens and paper, Post-it notes, etc.

2.3. Procedure

The meetings started with a 45-min oral presentation of the two given displays

described in Section 2.2 and brief explanations of the rationales behind the design.

Some of the comments obtained on a preceding evaluation performed by three captains

were also presented (Olsson et al., 2002). Each participant was provided with display

designs in the form of color printouts, together with definitions of the most common

elements and concepts in the displays. The groups had access to reference material as

design guidelines and specifications from the International Maritime Organization, a

technical report of the design work performed on the displays (Barcheus, 2002) plus the

results from the evaluation referred to above. The organization of the study allowed

proper documentation of both the work process and the results. The work material

provided and the settings were comparable, but the participants had full freedom to

develop the work in their own way, using their own experiences in their efforts to

improve a given design.

The introduction, including questions from the participants, was followed by half an

hour of individual reflections on how to deal with the task. In the mean time, each

participant wrote down answers to the following questionnaire:

1. What would be your first measure?

2. How do you prioritize the measures you intend to take?


3. What knowledge do you think you can contribute with from the horizon of your specific

competence?

4. What knowledge/competence do you think you lack, in terms of further work with a

design solution?

5. How would you acquire that knowledge, or alternatively, which kind of competence

would you seek advice from?

2.4. Analysis

Data were collected from the questionnaire and the videotaped sessions. The videotapes

were reviewed and the discussions transcribed. Notes were also taken on the behavior and

methods used by the subjects.

The author wanted to perform a relatively unbiased exploration of the topics

discussed by the two groups. Thus, a grounded theory (Glaser and Strauss, 1967)

approach was chosen for the analysis of the transcripts from the sessions, since it aims

to develop a theory from data rather than to gather data in order to test a theory or

hypothesis.

The transcripts from the two sessions were organized in separate tables, where

statements were carefully clustered according to topics, determined by the contents of the

statements, in line with grounded theory coding (Glaser, 1994). In this paper, the term

statement is used to describe a cluster of sentences composed of complete thoughts

articulated by one or more group members including support or objections given by the

others.

Each cluster was given an initial label consisting of a primary category and, when

appropriate, it was labeled with a sub-category as well. Then the tables were sorted

according to category and sub-categories. The texts belonging to a certain category, now

easily compared, were then further refined, rearranged and split up if necessary. Finally,

printouts of the texts, following the original timeline, were analyzed to further reveal how

the groups went about the task.

Finally, the labeled statements were counted to acquire a quantification of the

qualitative differences obtained in the two groups.

3. Results

3.1. Questionnaire

1. First measure. Two of the user representatives started to specify the information items

whose presentation they wanted to improve. One wanted to take the raw data, such as

course from gyro, speed, etc. and experiment with a presentation of these on uniform

instruments (similar to what has been done in cars) and only use the conning display for

the presentation of calculated and integrated data of high priority, such as deviation and

angles. The other two decided to start with more general tasks, such as determining

required information in different conditions, in order to reduce excessive information

from the display.


In the designer group, there was a general agreement on the need for increased

knowledge of the target environment and the task to be performed, and one of them

mentioned putting together a team where such knowledge was represented.

2. Priority. The user representatives split into two groups, two of them went straight for

details needed to improve such as the GPS, cross-track error, and deviation angle

presentation. The other two wanted to start on a higher level and work with

specification of information items needed on the new display. They also wanted to find

out present and future technical possibilities for the design solution.

The designers prioritized the improvement of their own domain knowledge through

field studies and interviews, contextual analysis, etc. and by bringing in people with

domain knowledge.

3. Competence. The user representatives judged their largest contribution to be their wide

experience of maneuvering and navigating different types of ships in different

conditions, using different systems and displays, sometimes in collaboration with

crewmembers having unknown training and qualifications. Two of them mentioned in

particular their experiences with overloaded displays, as a useful competence for the

design task.

The designers judged HCI knowledge as their main contribution, including

experiences in user-centered methods, in analysis, in explaining and simplifying

technical solutions, and finally in their ability to translate the result into design. One of

them reflected on their role, considering it as a catalyst between users and system

developers.

4. Lack of knowledge. The user representatives pointed out knowledge deficiencies in

quite specific areas such as water jet propulsion, current technical systems, and the

possibilities for integration of information from different technical systems. They also

mentioned lack of knowledge about human behavior, information processing and

decision-making, and finally a lack of knowledge about how to design displays that

communicate with users in the best possible manner.

For the designers, the general lack of domain knowledge was identified as an obstacle.

Moreover, two of them brought up a lack of knowledge about technical constraints, and

one designer wanted to know in particular what the economic constraints would be.

5. Remedies. The user representatives wanted to improve their own competence by

getting practice in simulators and on board high-speed craft, by interviewing

present users, by taking courses, by getting information about the tools currently

available and those that would be available in the future, and finally by involving

behavioral scientists and graphic designers.

According to themselves, the remedy for the designers’ lack of domain knowledge

would be to bring in people into the design team with such knowledge. Users, as well as

designers with technological experience, would help the designers to develop their domain

knowledge in the target environment.

3.2. Analysis of the group discussions

In the user representatives group, the clustering of statements resulted in 120

statements divided among 40 categories (Fig. 2). In the designers group, the clustering


Fig. 2. Statements in the captains group. The analysis resulted in 120 statements made by the captains divided

among 40 categories.


of statements resulted in 318 statements made during the day, divided among 31

categories (Fig. 3). This does not imply that the conversation was less energetic in the

user representatives group; rather it indicates that they took some time on each topic.

The difference in how the two groups spent their time was obvious, particularly in terms

of the designers’ focus on issues related to the design procedure (Fig. 4), and the user

representatives’ strictly work task related discussions.

3.2.1. The user representatives group

Although the discussion was active, the user representatives in the study remained

seated at the table for most of the time. The graphic representation of particular

information items was discussed from time to time. However, they deliberately left such

tasks to graphic designers, who were considered specialists in conveying a message using

graphic representations.

I would imagine that there are people who have this knowledge, how you communicate

a picture to the user, and avoid flickering tenth of figures.

On a few occasions, one of them went up to the whiteboard to explain a feature, or

visualize a suggestion with a sketch on the whiteboard. For this particular task, they

obviously did not have an urgent need for sketches; they shared a deep knowledge of the

work and this probably made non-representational discussions possible.

They questioned the design solution given in the assignment and remarked that they

would have wanted to start their work from scratch to get rid of inappropriate information

items. For instance, the compass rose and ship silhouette were realistically reproduced on

the screen; probably in an attempt to achieve an ecological interface design (Vicente and

Rasmussen, 1992). However, the user representatives judged them as ineffective and

consequently discarded them:

Why should it look like this, the compass rose and stuff like that?

They were specific about raw data and wanted to discriminate information transmitted

directly from a particular source and information derived from other sources. They also

discussed specific information items they wanted to remove completely from the display.

The user representatives often backed up their conclusions and suggestions with a

description of a situation they had experienced, a system they had been exposed to as well

as descriptions of information displays from different suppliers, and by relating to

incidents and accidents.

You can send the complete radar image to the ECDIS, that’s what we are looking at

here… XXX for instance have that on all their ships, you get tired after half an hour

because it’s too much.

Of course! This is the XXX-accident last Christmas, they used 2 17 or port 17 as mid

ship, it (the autopilot) works around 17, they are in a storm, with a speed of 20 knots, if

they had reduced speed they couldn’t have maintained the course at all…


Fig. 3. Statements in the designers group. The analysis resulted in 318 statements made by the designers divided

among 31 categories.


Fig. 4. References made to issues related to the design procedure. The distribution of the user representatives seven references during the day is displayed on the upper

diagram and the distribution of the designers’ 93 references is displayed on the lower one.

E.

Olsso

n/

Intera

cting

with

Co

mp

uters

16

(20

04

)3

77

–4

01

38

8

They discussed work-related issues, such as what they wanted to see, what kind of

information would be relevant and in what particular situations it would be relevant.

Continuous references were made concerning how you navigate and maneuver a ship,

what you hear and see, what you feel, and what you typically do not feel on modern ships

using water jet propulsion. They referred to experiences gathered on different kinds and

sizes of ships, as well as experiences from different locations where either traffic or

weather posed particular demands on the navigation and maneuvering. The group judged

their knowledge on high-speed craft as unsatisfactory. One of them had more experience

than the others had so he was able to explain details about water jet propulsion.

There were some minor differences of opinion on particular information items, and

from time to time throughout the day, they returned to the issue of being able to

personalize information displays with simple actions.

I can imagine an instrument where I can choose the contents, the Charlie-button, we are

all different, and every piece of information that is added steals your attention.

It is obvious from the narratives that work on the bridge varies very much with the

situation, depending on weather, surrounding vessels, the experiences of individuals of the

crew, and on their work practices. The particular information items asked for by the user

representatives in order to maintain situation awareness are partly dependent on these

work practices.

Maneuvering a high-speed craft is like driving a car on a soaped floor, you have

enormous engine power, and weather vanes to help you and you don’t see a thing.1 It is

all about seeing trends and large easy-to-read instruments, not a lot of damned small

figures. I don’t think that you should integrate engine power, wind and ship movements

in the same display…

The other interesting thing then is the GPS (Global Positioning System); you want to

know that it works because you moor on the GPS. The Stena ferry in the Irish see go

round with only two meters to each side.

The radar is the truth, if you overlay the chart with radar and it matches then it is

the truth.

In fog your eyes wander to the chart at mooring, you don’t watch the radar primarily, it

has to be right.

Finally, on three occasions, the group briefly discussed their own design task. They also

referred to design procedure issues on seven occasions, as for instance their own sketches

(Table 1).

1 The bridge is placed on top of large ferries, which means that at arrival/departure, the water in the

vicinity is obscured by the ferry itself, thus the captain cannot easily monitor distance between the ferry and

the key.


3.2.2. The designers group

The designers immediately took possession of the room. After the initial 30 min of

individual work, they never sat down again for more than a few seconds. They moved a

table closer to the whiteboard. They started to talk about the procedure to get the job

done, quickly agreeing on the required steps to go through, such as stating user

definitions, user roles, design criteria and information points.

They discussed how to organize their work, their sketches, how to use overhead film in

layers as a way to visualize layers in the design, how to use whiteboard and note pads, how

to document their work and so on. They went back and forth between the whiteboard, the

table in front of it, and the table where they placed reference material further back in the

room. The tasks of taking notes, drawing displays and finding out basic information from

the reference material were evenly distributed among the group members. Within a few

minutes, they had written down prioritized information on Post-it notes, given priorities to

and sorted them visually on the table in front of the whiteboard, based on the information

given in the reference material as well as on their own attempts to draw conclusions.

According to the design criteria this is supplementary information. The green

(Post-it) notes also show design criteria.

Other notes describing salient design features and multiple display views eventually

accompanied the notes. On several occasions during the day, the designers returned to and

reflected on the procedure to check that they carried out and finished the steps involved in

the design task, as they perceived it.

They tried to make a series of decisions, frequently assuring that their work proceeded

with statements such as, ‘now we have decided’ and ‘let’s assume’. These decisions and

conclusions were properly documented.

We have to make a few assumptions here. There are a lot of dimensions of information

presentation that we have to consider, the data presentation, analogue, digital or a

picture. You must be able to scan the information. The position, where to present data,

the coding and so on. We won’t be able to take all design decisions into account. I start

a listing here.

Table 1

Analysis of the statements related to design procedure issues in the user representatives group

Statements related to design procedure in user representatives group Number

Sketches 3

Design decisions 1

Graphic design 1

Design examples 1

Contextual analysis 1

Total 7

Sketches were discussed on three occasions. They did not mention contextual analysis explicitly, but one

statement was related to such an analysis.


The designers based most of their conclusions on general HCI knowledge. They

commented on information density, simultaneous presentation of relevant information,

on how the user should be in control, how displays should support decisions. Further,

they discussed to what degree automation should be implemented, its negative

consequences, on trust in automation, disturbing alarms, and how they are shut off, on

how people are different and the consequences of this fact, in terms of personal

settings.

From time to time, the designers remarked on their deficient domain knowledge.

Lacking the domain knowledge, they instead referred to experiences from other industries,

such as the air and car industries and pleasure boat experiences.

Many questions were related to which activities different members of a crew do on the

bridge and how different tasks are performed.

You have almost connected the situation to the role, a captain or a marine pilot

can certainly perform the mooring, then you need particular information. Is it a

matrix—a role in a particular situation or is it just the role: mooring?

They originated in uncertainties regarding different roles and responsibilities, and

seemed to be the result of an urge to make sure that only the required information was

presented on displays tailored to specific users, and that irrelevant information was

removed.

The largest cluster of statements in the designers group concerned design procedure

issues (29%), they made 93 such references (Fig. 4). These statements were thus separated

into sub-categories to allow for further analysis. Many statements concerned the

designers’ own work procedures (21%), but frequently they also made explicit references

to design decisions (17%) and design criteria (16%) as well as contemporary software

development procedural steps, such as contextual analysis and conceptual design

(Table 2):

Table 2

A detailed analysis of the statements related to design procedure issues in the designer group

Statements related to design procedure in designer group Number

Own work procedures 20

Design decisions 16

Design criteria 15

User participation 6

Roles 5

User categories; scenarios 4

Design examples; information elements; contextual analysis; prioritization 3

Design suggestion; conceptual analysis; conceptual design 2

User test; design without users; documenting; prototyping; sketches 1

Total 93

Most of them concerned their own procedures while working on the task, followed by references to decisions

made, indicating significant progress in carrying out the work assignment.


But isn’t this a hypothesis that we could work pretty much with, overlay in different

levels. Now I am pretty much ahead of the design. This is a design decision.

What’s the difference between… Exactly! This was dependent on the different systems.

Now we have to, we could say that this is a design decision, not to separate the symbols

depending on the system, a vessel is a vessel.

The design decisions they arrived at concerned visualization of information, how to use

colors and symbols, principles for the design solutions such as optimization of different

display views for different situations, and assumptions resulting from a lack of domain

knowledge. Sometimes they backtracked and revised decisions made earlier without

hesitation.

Even when domain related issues were discussed, they related to design procedure

issues, for instance, in the midst of a discussion about the chart, different scales and zoom

functionality, one of them stated that:

The possibility to switch between different scales may constitute a design criterion.

On a few occasions, the designers’ negative experiences from work with users

emerged. There were remarks such as:

…then they suddenly change their minds.

…then they want…

When they finally get a system like this, they might not like it at all.

Finally, they commented on the difficulties involved in agreeing on requirements,

without being able to evaluate them, for instance, by creating a prototype.

3.2.3. Technical issues

The designers discussed what the technical prerequisites might be and explicitly

declared several times that this was an important criterion that should be agreed on. They

decided to assume that a 4200 high-resolution screen would be reasonable, given the

enormous cost involved in building a craft like this.

Augmented reality was a familiar concept to the designer group and they discussed

whether this would be a competitive alternative to a conventional display. The user

representatives reasoned in similar ways when they sought solutions for the presentation of

information in stressful situations, such as when the captain absolutely needs to watch the

quay at mooring—although the user representatives related to how information previously

has been projected on the windscreen in fighter jets, instead of referencing the augmented

reality technique as such:

…as they do in aircraft, project on the windscreen, for instance wind direction, a big

arrow that comes from starboard.


3.2.4. References to appearance

The designers made 32 references to the appearance of their suggested design solution.

The references fell within two categories; 15 direct references to nice or ugly, e.g.:

How nice can you get it?

This will look pretty good.

That would look ugly.

There were 17 references concerning a messy and confusing display, e.g.:

The display may become messy if this information is added.

How can we make the display cleaner?

The user representatives only referred to appearance twice, once when talking about an

existing system as messy and once in general terms when stating that too much

information in a view would make it look cluttered.

3.2.5. Task solution

The given task was to integrate two displays into one. On several occasions,

both groups discussed views—for instance, displaying different views of the

information on the display that would be adapted to different situations or different

users. In the user representatives group, it was the most frequently discussed category

(18%) and in the designer group, it was the second most frequently discussed category

(19%).

The user representatives criticized the given designs and questioned the feasibility of an

integrated display.

A solution like this (the provided design) cause tunnel vision, I can confirm that.

They haven’t asked the expertise that is knowledgeable in this area (visualization)…

…then we have this (the provided design), I feel that they have just taken (…) and

stripped off (…), but it is not so damned well thought out.

This design (the provided design) is not satisfactory as it is now.

They decided that it would be an inadequate solution, and finally established that they

needed three different views (ocean, coast, and harbor) depending on the different


requirements in different situations. A few examples related to the three views are given

below:

There is no stress in the open sea, the time to decision is long, you need anti-

collision…You don’t need to see the instrument, it’s better to have it dark and cozy, it’s

better to collect rate of turn via hearing… (ocean mode)

In coastal mode you see the route, it is a red line, but you may have to deviate from the

route, sometimes you need to know if you can turn to starboard or port, and still have a

safe passage.

…movements is the interesting thing here, how you can visualize it. If you had ONE

point of rotation, it had been easy, but we know that you need some kind of docking

display where you see the bow movement, the stern movement and the rotation (named

harbor mode later).

The user representatives referred to what they thought was necessary, based on their

own experiences—for example, in the following scenario: at sea, at night, you want to sit

quietly without being disturbed by lit up screens that destroy your night vision, you only

want to see potential hazards, therefore, a display with only absolutely relevant

information is desired. Correspondingly, when you are approaching the quay, other

information elements will be relevant.

The information required and prioritized in, e.g. the coastal view was heading, speed

over ground, graphical representation of rudders (or corresponding nozzles on water jets)

in the bottom of the display, depth-information if deviating from ECDIS, and finally cross-

track error (a measure of deviation from the route). Depending on different preferences

that had emerged in the group, they suggested the possibility of having individual settings,

for instance, for information related to wind, bow-thrusters, the movement for bow, etc.

Having considered the statements given by captains at the previous review of the design

available in the reference material, the designers arrived at a similar solution of using

different views in different situations:

It could be one display for each situation.

…a common display for particular information items, but different views.

It should be an integrated display, but we could find that solution inadequate.

In terms of different views, the designers discussed the possibilities as well as the

potential negative consequences of automated functionality—specifically, if views should

be switched automatically (e.g. zooming in the chart when approaching the harbor)

depending on the speed or current location of the vessel, for example.

Both groups discussed whether information elements not belonging in the center of the

chart should be placed along the borders. They also examined the possibility to integrate


the information in the chart display—perhaps in the form of overlays, or to reduce the

chart to allow for panels on both sides. Further, against the current trend of squeezing in as

much information as possible in each display, they also wanted to remove information

items that were not truly necessary.

Finally, both groups also discussed the possibility of using the windshield as

an alternative display for navigation information, as has been previously suggested

(Olsson et al., 2002).

4. Discussion

In this study, a group of interaction designers and a group of user representatives

worked on a solution to a design problem on separate occasions, and with methods of their

own choice. The intentions were to reveal what the two groups contribute to the design

process, respectively. In particular, the user representatives group was exposed to an

unusual approach, as they were allowed to initiate design work without interference from

usability professionals—contrary to recommended procedures in design work with user

involvement (Buur and Bødker, 2000). Their reflections on what they would require in real

life in order to do a high-quality job were expected, but they also thought a great deal about

innovative design solutions. The users’ role in the design process and their contributions to

design as indicated by empirical data are discussed in Section 4.1.

Unsurprisingly the designers focused on design procedure issues and progress, but the

degree to which they reflected on the design procedures and dominated the room is worth

noting. Negative consequences for user involvement related to such practices will be

discussed in Section 4.2.

The groups were not expected to produce innovative and complete design solutions,

given that they were unprepared and only had a day to spend on the task, and the quality of

the resulting solutions has not been evaluated here.

4.1. Actively involved users

Reservations have been expressed against listening too carefully to users’

descriptions of their current work tasks for fear of perpetuating current work procedures.

Gentner and Grudin (1990) argued that tools of the past have shaped the users’ tasks and

a blind adaptation of an interface to an existing task would lock users into obsolete

behavior. This might be a legitimate fear in environments where newly introduced

computer systems make parts of current work routines superfluous. The opposite

situation—where well-established work procedures are ignored and new systems make it

completely impossible to get the job done—has been thoroughly documented (Kuhn,

1996; Eason, 1997).

However, work practices are developed and refined over time, and they have a value of

their own that must be understood before a radical change can be implemented. Suchman’s

(1993) work on air traffic controllers further demonstrates how work in practice may differ

from formal procedures (which sometimes are the basis for design) and how groups work

together to develop a common understanding of problems.


The train drivers participating in the study referenced in Section 1.3 were given

plenty of time to discuss work in their own vocabulary and the designers tried to

avoid being authoritative. Even so, the drivers were more or less expected to deliver

answers to the questions that the designers had. The train drivers were free to take on

the expertise role; still they assumed a lower profile, probably acknowledging that the

researchers were more knowledgeable on for instance design, decision-making, and

automation. The opportunity to start in the way the user representatives did here

allowed them to assume the role of being an expert rather than having to adapt to

being informants inquired about certain aspects of their work. Sachs (1995) found that

extensive discussions, data collection, and reflection were required to get workers to

let go of organizational thinking about their work and instead integrate tacit elements

of work, e.g. taken for granted knowledge that we are unconscious of (Polanyi, 1958),

into their analyses and design efforts. In comparison, the user representatives in this

study managed to get on without an extensive introduction, perhaps depending on the

lack of demands on formal methods. They were also able to speak in their own

domain-specific vocabulary with their colleagues, a situation that gave them time to

develop their own thinking without irrelevant interruptions that distracted their line of

thinking.

The users in this study represent operators of dynamic control systems. They have

lots of opportunities to consider and evaluate their work and their support systems

during periods when their workload is low, and no immediate actions are required

(e.g. going for hours on autopilot, only surveying potential hazards from surrounding

traffic). Crews are also renowned for their flexibility; they often shift from one ship to

another. These circumstances give crewmembers extensive experience in support

system features and exposure to what works, and what does not. Narrations about

incidents, accidents and work experiences, which was a vital part of the user

representatives’ discussions, are also a naturally occurring fundamental part of the job.

This circumstance deserves to be more acknowledged in design work. Erickson (1996)

believes that the stories people tell about what they do and how they do it contain

information essential to designing good interfaces. According to Muller (Rohn et al.,

2002, p. 893) they can provide “another kind of hybrid or third space in which the

domains of end-users and of software professionals can enter into a productive and

insightful dialogue”.

People use a specific vocabulary to talk about their work. These concepts are not

arbitrarily chosen, rather, they arise from the need to handle and talk about work in

terms of fundamental concepts. Users and software developers need to share the kind

of common work concepts that are dependent on thorough domain knowledge in

order to produce efficient systems (Gulliksen and Sandblad, 1995). Acquiring a

domain-specific vocabulary is the key to a better understanding of what takes place

at work. The stories that the user representatives are telling here are ‘real’ in the

sense that there are no formal principles to them and they are not artificially

constructed to primarily reveal requirements on design. According to Erickson (1996,

p. 35) they “reveal a user’s eye view of the landscape and provide an extremely

efficient way for getting people—both users and designers—involved and talking with

one another”.


Finally, captains may be considered conservative from an outsider’s point of view, but

they often have creative ideas. For instance, a user representative in this study remarked

that:

I want to stick to the frontline; these displays (the given designs) are damned old

fashioned. We continuously talk radar, GPS, and technical systems, but we haven’t

mentioned sound impressions, we feel how the ship moves. We exclude an extensive

amount of information that never will be considered in this display.

The study shows that given the time and opportunity to express their opinions, these

users are more than happy to discuss problems with support systems and innovative

solutions to them. The habit of using narratives to describe how work is performed might

disclose tacit knowledge, which is difficult to deduct from structured work task modeling

sessions, but may be revealed when people actually do things or are reminded of things.

Even if separating users and designers cannot be recommended as future practice, the

outcome of an initial group discussion among domain experts would probably be valuable

as an introduction for systems developers as well.

As the above discussion indicates, strategies based on focused methods and notations,

such as, modeling use cases together with users, may actually prevent the users from telling

their stories. However, such anecdotes may reveal important details and create a shared

understanding, which could function as the glue that keeps users and designers together.

4.2. Must users become computer experts?

In HCI research, users are often brought together with designers using rather rigorous

methods to solve design problems. The negative consequences arising from the clash

between different competencies thus brought together have been reported frequently

(Cooper, 1999). Although this study involved people confident in their professional roles,

it revealed a difference in behavior between the groups. The designers took possession of

the room. They had a well-established pattern and structure for their work:

OK, we are going to build a system. What do we need to know?

What are the user categories?

What are the constraints?

Let’s get going!

This will be good, this will look nice.

The user representatives were more cautious; they remained seated and used

whiteboards and other material sparingly. Their discussion focused on how work

is presently performed and what is really required to perform work proficiently.


The difference in behavior between the groups could not easily be explained by everyday

work habits or the settings that the groups were exposed to; both groups performed their

task in familiar settings and knew the other members of their group.

Cooper (1999) deems interaction design in teams with representatives from different

disciplines as ineffective and judges that the user is often the most poorly equipped to

articulate her concern in such a team. The discussions are largely controlled by developers

and designers; they join the meetings armed with questions, modeling techniques, design

suggestions and pre-conceptions about what the system should look like (Symon, 1998;

Boivie et al., 2004), and they know what systems development is about. They exercise

power—they have the right to question work contents and current procedures, and in the

end, to explain what is proper and what is possible to implement in the software.

Furthermore, they have a common domain-specific vocabulary and this occupational

vocabulary enables them to communicate with each other in a brief and efficient manner.

The users who are not familiar with the jargon easily become bystanders, ‘probably

disoriented and discouraged about the potential for successful progress of the project’

(Bahn, 1995, p. 78). Although they were working on their own, the designers in this study

displayed many of these patterns of behavior, which taken together may hinder effective

user-designer communication.

User who accept these conditions and become extensively involved in a development

project may change their initial focus on being a user representative and defending that

perspective. Often, at least to some extent, they become steeped in the system developers’

way of thinking and in fact, change their vocabulary; a new vocabulary evolves

(Boivie et al., 2004). This may well be an instinctive survival strategy—either you pick up

the ruling developer vocabulary and stay in the project or you become marginalized and

leave. Besides, designers’ habit of discussing solutions in terms of interface components

available in the current development tool is very catching, and users who pick up the habit

of discussing design in terms of interface components may become limited in their

thoughts. Instead of achieving real change in work practices, the design may become

restricted to what is feasible given the particular development tool in use.

5. Conclusions

In conclusion, the designers in this study focused on design procedure issues to a

great degree. They used their methods and professional vocabulary to ensure progress

and results. The user representatives on the contrary had a narrative way of talking

about their work; they also spent more time on each discussed issue than the

designers did. If these competencies were brought together without further

introduction, there is a risk that the designers—with their strong reliance on a

sequential chain of steps towards a final goal—would have set the pace, and guided

the outcome of the design meeting without including proper consideration of domain

and use specific circumstances.

One way of giving the users a head start would be to encourage them to reflect on their

present and future work procedures and practices prior to modeling sessions, before they

become entangled in systems development and the vocabulary of designers. Such an


opportunity would make them better prepared to convey the ‘the big picture’ as well as

the meaning of details that matter in their work domain. Further, a future concern must be

to avoid that user representatives change their vocabulary and professional identity and

become IT workers when they get involved in systems development projects—and instead

to encourage them to maintain and develop their original professional identities.

Continuing use of and reflection upon the design narrative is an approach that could allow

them to develop their practices.

Designers and users must collaborate in the design process. Encouragingly enough, the

two groups in the study expressed the need for other competencies, both in the initial

questionnaire and later on in the work with the design task. The designers identified the

need for domain knowledge and mentioned user participation in a number of ways,

through, for instance, field studies, interviews, and user testing. Likewise, it was discussed

in the user representatives group how a ‘graphic designer’ probably would have solved

visualization issues since they know how to design a display that communicates with the

viewer. Similar references were made in the designer group, where they decided to leave

particular presentations of information elements as they were, since a graphic designer

would solve the problem more gracefully.

However, designers must realize that regardless of the kind of user involvement one

chooses, it does not mean that users will be able to provide upfront, straight answers or

requirements on the new system. Users know what they want to achieve at present and,

given time, they may have great ideas and visions about other tasks that maybe could be

performed in a new system. They can easily express such knowledge and visions in their

own domain vocabulary. Nevertheless, users who deliver requirements on demand are

rare, and consequently you cannot hope to capture requirements by involving users.

Designers are the ones who must transform task and domain knowledge, perhaps delivered

in narrative form, into requirements.

Acknowledgements

I would like to express my gratitude to the captains from Kalmar Maritime Academy

and the designers from Redina-Enea AB who took part in the study, and finally thanks to

my colleagues at the department who graciously took time to share their experiences and

insights, in particular Inger Boivie, Jan Gulliksen, Anders Jansson, Stefan Blomkvist and

Bengt Sandblad.

The research reported here was conducted within the SASAM project, which has been

funded by the Swedish Agency for Innovative Systems (VINNOVA), the Swedish

Maritime Administration, and the Swedish Mercantile Marine Foundation.

References

Barcheus, F., 2002. Integrated Instruments and Augmented Reality on High-Speed Craft. Master Thesis,

Department of Industrial Economics and Management, Royal Institute of Technology, Sweden, ISSN

1403-7777.


Bahn, D.L., 1995. System Designer-User Interaction: An occupational subcultures perspective. In: Proceeding of

SIGCPR conference on Supporting teams, groups, and learning inside and outside the IS function reinventing

IS, Nashville, TN, USA, pp. 72–80.

Beyer, H., Holtzblatt, K., 1998. Contextual Design. Defining Customer-centered Systems. Morgan Kaufmann,

San Francisco, CA.

Bjerknes, G., Ehn, P., Kyng, M. (Eds.), 1987. Computers and Democracy—A Scandinavian Challenge, Avebury,

Aldershot.

Boivie, I., Aborg, C., Persson, J., Lofberg, M., 2004. Why usability gets lost or usability in in-house software

development. Interacting with Computers 15 (4), 473–639.

Buur, J., Bødker, S., 2000. From usability lab to ‘design collaboratorium’: reframing usability practise. In:

Boyarski, D., Kellogg, W.A. (Eds.), Proceedings of the Conference on Designing Interactive Systems, ACM

Press, New York, pp. 297–307.

Carmel, E., Whitaker, R.D., George, J.F., 1993. PD and joint application design: a transatlantic comparison.

Communications of the ACM 36 (4), 40–48.

Clement, A., van den Besselaar, P., 1993. A retrospective look at PD projects. Communications of the ACM 36

(6), 29–37.

Constantine, L.L., Lockwood, L.A.D., 1999. Software for Use: A Practical Guide to the Models and Methods of

Usage-centered Design. Addison-Wesley, Reading, MA.

Constantine, L.L., Lockwood, L.A.D., 2002. Usage-Centered Engineering for Web Applications. IEEE Software,

pp. 42–50.

Cooper, A., 1999. The Inmates Are Running the Asylum. Macmillan, Indianapolis.

Eason, K., 1997. Understanding the organisational ramifications of implementing information technology

systems. In: Helander, M., Landauer, T.K., Prabhu, P. (Eds.), Handbook of Human–Computer Interaction,

Elsevier, Amsterdam, pp. 1475–1494, Second revision (Chapter 61).

Erickson, T., 1996. Design as storytelling. Interactions 3 (4), 30–35.

Gentner, D.R., Grudin, J., 1990. Why good engineers (sometimes) create bad interfaces, In: Proceedings of ACM

CHI’90 Conference on Human Factors in Computing Systems, pp. 277–282.

Glaser, B.G., 1994. More Grounded Theory Methodology: A Reader. Sociology Press, California.

Glaser, B.G., Strauss, A.L., 1967. The Discovery of Grounded Theory: Strategies for Qualitative Research.

Aldine, Chicago.

Gould, J.D., Boies, S.J., Lewis, C., 1991. Making usable, useful, productivity-enhancing computer applications.

Communications of the ACM 34 (1), 75–85.

Gould, J.D., Boies, S.J., Ukelson, J., 1997. How to design usable systems. In: Helander, M., Landauer, T.K.,

Prabhu, P. (Eds.), Handbook of Human–Computer Interaction, Elsevier, Amsterdam, pp. 231–254, Second

revision.

Greenbaum, J., Kyng, M., 1991. Design at Work: Cooperative Design of Computer Systems. Lawrence Erlbaum,

Hillsdale, New Jersey.

Grudin, J., 1991. Interactive systems: bridging the gaps between developers and users. IEEE Computer 24 (4),

59–69.

Gulliksen, J., Sandblad, B., 1995. Domain-specific design of user interfaces. International Journal of Human–

Computer Interaction 7 (2), 35–151.

Gulliksen, J., Goransson, B., Lif, M., 2001. A user-centred approach to object-oriented user interface design. In:

Van Harmelen, M., (Ed.), Object Modeling and User Interface Design: Designing Interactive Systems,

Addison-Wesley, Reading, MA, pp. 283–312.

Gulliksen, J., Goransson, B., Boivie, I., Blomkvist, S., Persson, J., Cajander, A., 2003. Key principles for user-

centred systems design. Behaviour and Information Technology 22 (6), 397–410.

Gunter, R., Janis, J., Butler, S., 2001. The UCD Decision Matrix: How, When, and Where to Sell User-Centered

Design into the Development Cycle, http://www.ovostudios.com/upa2001/.

Kecklund, L., Olsson, E., Jansson, J., Kecklund, G., Ingre, M., 2003. The TRAIN Project: Effects of Organizational

Factors, Automatic Train Control, Work Hours, and Environment: Suggestions for Safety-Enhancing Measures.

In: Proceedings of the Human Factors and Ergonomic Society, 47th Annual Meeting, Denver, Colorado.

Kuhn, S., 1996. Design for people at work. In: Winograd, T., (Ed.), Bringing Design to Software, ACM press/

Addison-Wesley, New York, pp. 273–289.


http://www.ovostudios.com/upa2001/

Kujala, S., 2003. User involvement: a review of the benefits and challenges. Behaviour and Information

Technology 22 (1), 1–16.

Muller, M.J., Haslwanter, J.H., Dayton, T., 1997. Participatory practices in the software lifecycle. In: Helander,

M., Landauer, T.K., Prabhu, P. (Eds.), Handbook of Human–Computer Interaction, Elsevier, Amsterdam.

Norman, D.A., Draper, S.W. (Eds.), 1986. User Centered System Design: New Perspectives on Human–

Computer Interaction, Erlbaum, Hillsdale.

Olsson, E., Seipel, S., Jansson, A., 2002. The Windscreen as a display for navigational information, In:

Proceedings of Human Factors in Ship Design and Operation II, RINA, London, Oct 2–3.

Polanyi, M., 1958. Personal Knowledge: Towards a Post-Critical Philosophy. Routledge & K. Paul, Chicago.

Rohn, J.A., Spool, J., Ektare, M., Koyani, S., Muller, M., Redish, J., 2002. Usability in practice: alternatives to

formative evaluations—evolution and revolution, In: Proceedings of CHI ‘02 Extended Abstracts on Human

Factors in Computing Systems, Usability in Practice Session, Minneapolis, Minnesota, USA, pp. 891–897.

Rosenbaum, S., Rohn, J.A., Humburg, J., 2000. A toolkit for strategic usability: results from workshops, panels,

and surveys, In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, The

Hague, The Netherlands, pp. 337–344.

Sachs, P., 1995. Transforming work: collaboration, learning and design. Communications of the ACM 38 (9),

36–44.

Scaife, M., Rogers, Y., Aldrich, F., Davies, M., 1997. Designing for or designing with? Informant design for

interactive learning environments, In: Proceedings of ACM CHI’97 Conference on Human Factors in

Computing Systems, pp. 343–350.

Schuler, D., Namioka, A., 1993. Participatory Design: Principles and Practices. Lawrence Erlbaum, Hillsdale, NJ.

Suchman, L., 1993. Technologies of accountability: of lizards and aeroplanes. In: Button, G., (Ed.), Technology

in Working Order: Studies of Work, Interaction, and Technology, Routledge, London, UK.

Symon, G., 1998. The work of IT system developers in context: an organizational case study. Human–Computer

Interaction 13 (1), 37–71.

Vicente, K.J., 1999. Cognitive Work Analysis. Lawrence Erlbaum, UK.

Vicente, K.J., Rasmussen, J., 1992. Ecological interface design: theoretical foundations. IEEE Transactions on

Systems, Man and Cybernetics. SMC-22, 589–606.

Vredenburg, K., Mao, J.-Y., Smith, P.W., Carey, T., 2002. Design methods: a survey of user-centered design

practice. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: Changing Our

World, Changing Ourselves 4 (1).

Webb, B.R., 1996. The role of users in interactive systems design: when computers are theatre, do we want the

audience to write the script? Behaviour and Information Technology 15 (2), 76–83.


Documents

What active users and designers contribute in the design process