Sequencing the elements in references

Applied Ergonomics 1981, 12.1, 7-12

Sequencing the elements in references James Hartley Department of Psychology, University of Keele

Brian Shackel has argued that the system of referencing used in Applied Ergonomics has certain advantages that were not considered in an earlier article in this journal. Evidence is presented which is contrary to Prof Shackel's views and which, in fact, supports the original conclusions.

In a previous article (Hartley eta/, 1979) my colleagues and I presented the results from a study of the preferences of experienced British and American journal readers for various ways of presenting references at the end of a journal article. It appeared that preferences were clearly affected by the spatial arrangement of the elements within and between each reference list, and that typographic cueing took a secondary role. It was also apparent that there were differences between the preferences of the British and the American readers. The British, for example, preferred to see the date of publication after the name of the author(s), whereas the Americans preferred the date of publication after the title of a journal, or following the name of the publisher of a book. We concluded, therefore, that a compromise solution was probably best where the main elements would be separated from each other, each starting on a new line, and the date would appear in both positions.

The following reference is given as an example in each of three styles: (i) British, (ii) American, (iii) compromise.

(i) Broadbent, D.E. (1977) Language and ergonomics. Applied Ergonomics, 8, 15-I 8.

(ii) Broadbent, D.E. Language and ergonomics. Applied Ergonomics, 1977, 8, 15-18.

(iii) Broadbent, D.E. (1977) Language and ergonomics. Applied Ergonomics, 1977, 8, 15-18.

Our results indicated that separating the elements in this way was much preferred to the typical 'run-on' method of presenting references but that the British readers were not too keen on the compromise style (preferring the

We would encourage readers to consider the layout of references in Applied Ergonomics and the findings of this paper. It would be of value, both to Dr Hartley and the Editors, if readers' views could be sent to Dr Hartley at Keele University. The style of presentation in the journal was carefully considered and discussed in the paper by E.C. Poulton, T.R. Warren and J. Bond, 'Ergonomics in Journal Design', (Applied Ergonomics 1970, 1, pp 207 - 209). If the results of this paper are supported by the views of readers, the style of referencing in Applied Ergonomics may be modified. (Editors).

British one). The Americans, however, were more neutral towards the compromise style, liking the fact that it retained the journal date/volume sequence, and that the author/date sequence matched that given in the text.

The compromise style is of course different from the style used in Applied Ergonomics (which we did not include in our original experiment). Here this reference would be set thus: Broadbent, D.E. 1977 Applied Ergonomics, 8, 15-18. Language and

ergonomics.

We concluded our article by presenting our references in the compromise style so that readers of Applied Ergonomics could compare references set in the compromise style with those set in the Applied Ergonomics style for themselves. It can be seen that for journal articles there are three main points of difference:

(i) the names are printed in bold type in Applied Ergonomics,"

(ii) the date is separated from the author(s); (iii) the title of the journal comes before the title of the

article.

In a recent communication Brian Shackel was concerned to point out that our procedure (of asking for preferences for different designs in familiar sets of references) was to a certain extent false. Shackel suggested that readers of articles in 'real-life' are likely to have some familiarity with the content and with the references given in the article. In these circumstances readers will know some of the authors and some of the references from their previous reading. Shackel maintained: "'I consider that in quite a number of cases they will distinguish between references in terms of whether it is 'Bloggs 1978 or Bloggs 1979' and also in terms of whether it is 'the Bloggs 1978 paper in Ergonomics or the one in Applied Ergonomics'".

In summarising his letter Shackel thus explained the decisions behind the design of the reference s~'stem used in Applied Ergonomics. He wrote: "Thus, I am really proposing the hypothesis that a majority of readers who refer actively to the references do so to decide whether they know the reference given, or not, and to decide whether to follow up that reference (because of its ease of access or potential value); further, that this progressive differentiation is usually made by a knowledgeable reader, for whom the

0003-6870/81/01 0007-06 $02.00 O IPC Business Press Applied Ergonomics March 1981 7

successive items of relevant data are author, date, journal or book title, and article title; and finally, that the layout of these data should enable the reader's eyes to follow that sequence with the mininmm number of fixations (thus the position of author and date in the Applied Ergonomics reference layout requires only one eye fixation)".

Shackel indicated that the design of this reference system was based upon reasoned opinion rather than on scientific evidence, and that he would be interested in judging any evidence on this issue.

This paper thus presents some evidence for his - and the readers' - consideration. The paper is divided into three parts. In Enquiry One, experienced journal readers expressed their preferences for various layouts of references. In Enquiry Two the preferences of less experienced readers were considered; and in Enquiry Three the practices used in other journals were assessed.

Enquiry one

To test the validity of Shackel's views five different ways of presenting a page of references (based on a page from Applied Ergonomics) were typed out. These are illustrated as Figs. (a) - (e) respectively below. A brief summary of their characteristics is as follows:

(a) References typed in the style of Applied Ergonomics (but without the bold type for the name);

Allan, M.D. 1957 OccupationalPsychology, 31, 113-119. Training in

perceptual skills.

Annett, J. 1961 The role of knowledge of results in learning: A survey.

NAVTRADEVCEN Technical Report No 342-3, US Naval Training Device Center, New York.

Annett, J. 1969 Feedback and human behaviour. Harmondsworth:

Penguin.

Annett, J., and Duncan, K.D. 1967 Occupational Psychology, 41, 211-221. Task analysis

and training design.

Chancy, F.B., and Teel, T.S. 1967 Journal of Applied Psychology, 51, 311-315.

Improving inspector performance through training and visual aids.

Drury, C.G., and Fox, J.G. (eds) 1975 Human reliability in quality control London: Taylor

and Francis.

Embrey, D.E. 1976 Signal detection theory in the analysis and

optimisation of industrial inspection tasks. Unpublished PhD thesis, University of Aston in Birmingham.

Gibson, E.G. 1953 Psychological Bulletin, 50, 401-431. Improvement

in perceptual judgements as a function of controlled practice or training.

Mackie, R.R., and Harabedian, A. 1964 A study of simulation requirements for sonar operator

trainees. US Naval Training Device Center, Technical Report No 1320-1.

Sinclair, M.A. 1979 Applied Ergonomics, 10, 17- 25. The use of

performance measures on individual examiners in inspection schemes.

Figure (a)

(b) References typed in the same style, but with each main element in the list starting on a new line;

Allan, M.D. 1957 OccupationaIPsychology, 31, 113 119.

Training in perceptual skills.

Annett, J. 1961 The role of knowledge of results in learning: A survey.

NATRADEVCEN Technical Report No 342-3, US Naval Training Device Center, New York.

Annett, J. 1969 Feedback and human behaviour

Harmondsworth: Penguin

Annett, J., and Duncan, K.D. 1967 Occupational Psychology, 41, 211-221.

Task analysis and training design.

Chaney, F.B., and Teel, T.S. 1967 Journal of Applied Psychology, 51, 311-315.

Improving inspector performance through training and visual aids.

Drury, C.G., and Fox, J.G. (eds) 1975 Human reliability in quality control.

London: Taylor and Francis.



Gibson, E.G. 1953 Psychological Bulletin, 5 O, 401-431.

Improvement in perceptual judgements as a function of controlled practice or training.


trainees. Technical Report No 1320-1, US Naval Training Device Center.

Sinclair, M.A. 1979 Applied Ergonomics, 10, 17-25.

The use of performance measures on individual examiners in inspection schemes.

Figure (b)

(c) References typed in the same style, but with the title of the author's work preceeding the place where it can be found;

Allan, M.D. 1957 Training in perceptual skills.

Occupational Psychology, 31, 113-119.

Annett, J. 1961 The role of knowledge of results in learning: A Survey

NAVTRADEVCEN Technical Report No 342-3, US Naval Training Device Center, New York.

8 Applied Ergonomics March 1981

Annett, J. 1969 Feedback and human behaviour.

Harmondsworth: Penguin

Annett, J., and Duncan, K.D. 1967 Task analysis and training design.

Occupational Psychology, 41, 211-221.

Chaney, F.B., and Teel, T.S. 1967 Improving inspector performance through training

and visual aids. Journal of Applied Psychology, 51, 311-315.

Drury, C.G., and Fox, J.G. (eds) 1975 Human reliability in quality control.

London: Taylor and Francis.



Gibson, E.G. 1953 Improvement in perceptual judgements as a function

of controlled practice or training. Psychological Bulletin, 50, 401-431.


trainees. Technical Report No 1320-1, US Naval Training Device Center.

Sinclair, M.A. 1979 The use of performance measures on individual

examiners in inspection schemes. Applied Ergonomics, 10, 17-25.

Figure (c)

(d) References typed in the British style advocated by Hartley et al (1979);

Allan, M.D. (1957) Training in perceptual skills. Occupational Psychology, 31, 113-119.

Annett, J. (1961) The role of knowledge of results in learning: A survey. NAVTRADEVCEN Technical Report No 342-3, US Naval Training Device Center, New York.

Annett, J. (1969) Feedback and Human Behaviour Harmondsworth: Penguin.

Annett, J., and Duncan, K.D. (1967) Task analysis and training design. Occupational Psychology, 41, 211-221.

Chaney, F.B., and Teel, T.S. (1967) Improving inspector performance through training and visual aids. Journal of Applied Psychology, 51, 311-315.

Drury, C.G., and Fox, J.G. (eds) (1975) Human Reliability in Quality Control. London: Taylor and Francis.

Embrey, D.E. (1976) Signal detection theory in the analysis and optimisation of industrial inspection tasks. Unpublished PhD thesis, University of Aston in Birmingham.

Gibson, E.G. (1953) Improvement in perceptual judgements as a function of controlled practice or training. Psychological Bulletin, 50, 401-431.

Mackie, R.R., and Harabedian, A. (1964) A study of simulation requirements for sonar operator trainees. Technical Report No 1320-1, US Naval Training Device Center.

Sinclair, M.A. (1979) The use of performance measures on individual examiners in inspection schemes. Applied Ergonomics, 10, 17-25.

Figure (d)

(e) References typed in the compromise style in the layout recommended by Hartley et al.

Allan, M.D. (1957) Training in perceptual skills. Occupational Psychology, 1957, 31, 113-119.

Annett, J. (1961) The role of knowledge of results in learning: A survey. NAVTRADEVCEN Technical Report No 342-3. US Naval Training Device Center, New York, 1961

Annett, J. (1969) Feedback and Human Behaviour Harmondsworth: Penguin, 1969.

Annett, J., and Duncan, K.D. (1967) Task analysis and training design. Occupational Psychology, 1967, 41, 211-221.

Chaney, F.B., and Teel, T.S. (1967) Improving inspector performance through training and visual aids. Journal of Applied Psychology, 1967, 51, 311-315.

Drury, C.G., and Fox, J.G. (eds) (1975) Human Reliability In Quality Control. London: Taylor and Francis, 1975

Embrey, D.E. (1976) Signal detection theory in the analysis and optimisation of industrial inspection tasks. Unpublished PhD thesis, University of Aston in Birmingham, 1976.

Gibson, E.G. (1953) Improvement in perceptual judgements as a function of controlled practice or training. Psychological Bulletin, 1953, 50, 401-431

Mackie, R.R., and Harabedian, A. (1964) A study of simulation requirements for sonar operator trainees. Technical Report No 1320-1, US Naval Training Device Center, 1964.

Applied Ergonomics March 1981 9

Sinclair, M.A. (1979) The use of performance measures on individual examiners in inspection schemes, Applied Ergonomics, 1979, 10, 17-25.

Figure (e)

Comparison between the figures indicates that not every feature that could be varied systematically has been varied, but the aim was to concentrate on the major differences and to keep the consequent judgement task simple.

The procedure was to ask experienced journal readers (all lecturers and/or research workers in the psychology department at Keele) which of these five sets of references they preferred, and to put them in rank order. More specifically the instructions were as follows:

'Suppose you are reading a journal article. Which of these layouts for the references would you prefer to have at the end of the paper? There are five of them. Can you examine the full page of references for each of them please, and then put them in rank order?'

When this was done, each participant was asked to explain why he or she had placed each layout in the rank provided, and these reasons were recorded.

Table 1: The preferences of individual judges for the five styles, the medians and the ranges

mJ

Style

Judge Sex a b c d e

1 M 4 5 3 1 2

2 M 5 4 3 2 1

3 M 4 3 2 1 5

4 M 4 3 2 1 5

5 M 5 4 3 1 2

6 M 5 4 3 2 1

7 M 5 4 3 2 1

8 M 5 3 1 2 4

9 M 3 2 1 5 4

10 M 5 4 3 1 2

11 M 5 4 3 1 2

12 M 5 4 2 1 3

13 M 4 5 3 1 2

14 M 5 4 3 1 2

15 F 5 4 3 t 2

16 F 5 4 3 1 2

17 F 5 1 2 4 3

18 F 5 4 3 1 2

19 F 5 4 3 2 1

20 F 3 2 1 4 5

Median rank 5 4 3 1 2

Range 3 - 5 2--5 1 - 5 1 - 5 1 - 5

Table 2: A selection of comments made by judges on the various layouts (NB some judges gave more than one reply for each layout)

Layout a

Layout b

Layout c

Layout d

Layout e

Don't like the run-on method 10 Don't like the sequence 9 Don't like the position of the date 2 An economical way of presenting references 1

Don't like the sequence 11 Like new elements starting on separate lines 5 Don't like the position of the date 3 Like the position of the date 3 Like the sequence of elements 3

Don't like the position of the date 7 Like to have the article t i t le before the

journal reference 7 Like the position of the date 3 Like consistency in the order of the elements

that is not present in layouts a and b 1 Like new elements starting on separate lines 1

Like the date after the author's names 6 Like it because it's what I'm used to 5 Like new elements starting on separate

lines 3 Like to have the article t i t le before the

journal reference 3

Don't like the dates repeated (confusing, irritating, wasteful) 7

It's useful to have the dates twice for they serve dif ferent purposes 5

Like the date after the author's name 3 Like to have the article t i t le before

the journal reference 2 Like new elements starting on separate

lines 1

Results Twenty judges acted as participants in this study, and

the results are shown in Table 1. This table provides the ranking of each layout by each judge (1 = most preferred: 5 = least preferred) and the overall medians and ranges. It is clear that in order of preference Layout (d) (the British style) is rankest highest and Layout (a) (from Applied Ergonomics) lowest.

Statistical tests (Wilcoxon Matched Pairs Sign Tests: Siegel, 1956) indicated that Style (d)was significantly preferred to Styles (a), (b) and (c) (p < '001 in each case), but it was not significantly preferred to Style (e). Style (e) was significantly preferred to Style (a) (p < "01), Style (b) (p < "01) but was not significantly preferred to Style (c). Style (c) was significantly preferred to Styles (a) and (b) (p < "001 in each case) and Style (b) was significantly preferred to Style (a) (p < "001). These results indicate that the British style was significantly preferred to all of the others except the compromise style. The compromise style was less preferred than the British style (but not significantly so) but it was more preferred than Styles (a) and (b) and (although not significantly) Style (c) - the variant of Applied Ergonomics with elements in the sequence suggested by Hartley et al. In brief then, these results reflect those found by Hartley et al and give no support for Shackel's views.


Table 2 shows the frequency with which various reasons were given for the choices made. The comments encapsulated in this table suggest that the preferences of the judges were largely governed by the position of the date, the placing of the main elements on separate lines, and by their sequencing (article titles were preferred before journal references).

There were, however, some supporters for Shackel's views. In terms of the position of the date one said, "The relationship between the names and the dates is perceptually evident in the way one codes it in reading the paper". Another said, "The date in the margin allows one to look down quickly and pick out more recent articles". In terms of the sequencing of elements, Shackel's position found few supporters, although one did say, "The things which are to the fore are the things you most need in searching out an article".

It is clear from Table 2, however, that most judges did not respond in this way. They preferred the date after the author(s): "It matches what you read in the text". "I suppose it's what we're used to." They liked the titles of articles before where they could be located: "It's logically correct". "It's consistent throughout." And they liked each main element to be placed on a new line: "It helps you if you are looking for different things".

Layout (e) produced some conflicting comments. Some judges appreciated the extra date: "It's useful to have the dates twice as they serve different purposes", but others were more hostile: the double date was "thoroughly confusing", "irritating" or even considered "offensive in this time of economic restraint".

Conclusions The results of this first enquiry show two things

unequivocally:

(i) There is no clear support for the British system, Layout (d). The compromise system Layout (e) emerges as second best. These findings are in accord with the earlier results presented by Hartley et al.

(ii) There is little support, if any, for Shackers hypothesis. It helps to space out Shackel's original system (Layout (b) is preferred to Layout (a)), but his way of sequencing the elements is not generally appreciated.

These results, of course, have been obtained by asking for subjective preferences. Shackel maintains, and I would agree, that it might be better if one could ask readers how they were using particular references, and what they preferred, as they were reading an article specific to their own interests. Nonetheless, despite my agreement, I would suggest that the procedures used in our original and this present enquiry are not entirely invalid. It seems to me that experienced readers do have ideas about what they normally look for in the design of references, and that they can apply this knowledge to sample reference pages. If this is true then one might predict that the preferences of less experienced readers would be different from those of more experienced ones. This notion leads to Enquiry Two.

Enqui ry two

Eighteen second-year students attending a practical class in psychology were each given a copy of the materials used in Enquiry One and asked to place them in rank order

Table 3: The preference of psychology students for the five styles, the medians and the ranges

Style

Judge Sex a b c d e

1 M 5 3 4 1 2

2 M 4 3 5 1 2

3 M 3 2 1 5 4

4 M 5 4 2 1 3

5 M 5 4 3 2 1

6 M 5 4 3 2 1

7 M 3 1 2 4 5

8 M 4 5 1 2 3

9 F 5 1 2 3 4

10 F 5 4 3 2 1

11 F 1 5 4 3 2

12 F 5 4 1 2 3

13 F 5 4 3 2 1

14 F 4 1 2 3 5

15 F 5 1 2 3 4

16 F 2 1 3 5 4

17 F 5 4 1 3 2

18 F 4 5 2 1 3

Median rank 5 4 2 2 3

Range 1.-5 1 - 5 1 - 5 1 - 5 1 - 5

(following the same instructions as given above). This choice of participants was deliberate: these students had been studying psychology for four terms and they had written essays and practical reports, but judging from these, their knowledge of reference systems seemed quite deficient. The results obtained from this enquiry are shown in Table 3.

It can be seen that layout (a) was ranked significantly below layout (b) (Sign test p < .02) but that layout (b) was not ranked significantly differently from layouts (c), (d) and (e). Indeed, there were no clear preferences for these last three designs.

These findings thus support the argument put forward above: the less experienced judges judged differently from the more experienced ones. However, even the less experienced judges did not like layout (a). This was mainly because the students preferred the main elements of a reference to be placed on separate lines. There was some suggestion that the students preferred the main elements to be sequenced in the way that the experts preferred (layout (b) was less liked than layouts (c), (d) and (e)) but these differences were not significant. The students appeared to have no preference for where the date should be positioned,

In nay view these results indicate that there is some value in subjective preferences as a measure in research of this kind. The fact that the preferences of experienced judges are different from those of inexperienced ones indicates that the measure has some sensitivity. It would be useful, nonetheless, to supplement such preference measures with additional ones - such as search times, or costs of production.

Applied Ergonomics March 1981 11

Such additional measures, of course, may lead to different outcomes. Spacing out the text may lead to faster retrieval times for specific items (or slower, if more page turn-over is involved), but, if more paper is used, spacing out the text may be more expensive. However, if, as I shall argue below, the compromise solution to the layout of references meets the requirements of different readers in different ways without any increases in cost, and if readers prefer it to the layout currently used in Applied Ergonomics, then there seems to be some advantage in using it.

Enquiry three

A third way of reflecting on Shackel's hypothesis is to examine existing practices in other journals. Do other journals present references in the style used in Applied Ergonomics? Do other journals separate out the date in the way advocated by Shackel, and do they present journal references before the article titles? If editors and printers share some tacit knowledge about journal readers how is this reflected in existing practice?

One way of answering questions such as these is to look at current journals. I therefore examined five journals taken at random from each of the separate disciplines that make up the Science and the Social Science Faculties of the University of Keele. I excluded from my sample those journals which did not provide article titles. (This is common in Chemistry and Physics, and also occurs to some extent in Biochemistry, Biology and Geology.) No medical or engineering journals were examined as Keele does not have a medical or engineering school.

As a result of this procedure I examined the reference systems of 95 journals (40 in the Social Sciences and 55 in the Sciences). The observation that I made was that not one of these journals provided the journal reference before the author(s)' article title. Perhaps Shackel can cite a specific example or two, but the sequencing of elements that he recommends was not reflected in the results of my semi-random search.

Interestingly enough, however, there was some support in actual practice for Shackel's views about the positioning of the date. My search revealed three journals that positioned the date in the manner similar to that used in Applied Ergonomics. These were the American Sociological Review, the Canadian Journal of Sociology and Human Organisation: The Journal of Social and Applied Anthropology. None of these three journals, in my view, however, had quite such an elegant solution as that provided in Applied Ergonomics. There was insufficient line space between the entries in the American Sociological Review, and the layout of the other two journals was spoilt by the fact that the date was indented (on the new line) but the subsequent text was not: this procedure reduced the saliency of the date.

We may conclude then from these observations that although there is a little support in existing practice for positioning the date in the manner used by Applied Ergonomics, there is no support for sequencing the elements of references in the way used by this journal.

Concluding comments

Shackel's hypothesis has raised some interesting questions, most of which, in my view, have been answered in the negative. For many readers these issues may seem relatively

trivial, but behind this niggling over details there are some important issues. I will briefly colnment on three of them here

First of all there is the question of methodology. When are subjective preferences a useful tool? Should they be supplemented by other measures? Preferences seem to provide useful information when all the judges (or a clear majority of them) agree. The investigator can then decide whether or not the judges are mistaken (and if so why) or whether or not this information is valid for his purpose (the stance taken in this paper). When there is no clear agreement between the judges this information will still be informative, but it may be less helpful for decision making and additional criteria may be required.

Next there is the problem of using space to clarify text. This issue has" not been discussed in this paper, but in the original one we were concerned to show how the spatial organisation of the elements in references had an important effect on preferences, and that spatial considerations were more important than typographic ones. Judges preferred, for example, the title of a journal to start on a new line rather than be separated by the use of italic on the same line. These findings are reflected in this present paper: the run-on layout, Layout (a), was the least preferred in Enquiries One and Two.

Finally, there is the intriguing question of how best to present references in journals. Shackel made his critique from the standpoint of an experienced reader. But journals have a duty to non-experienced readers and to readers from other disciplines, too. So the choice for the designer is either (i) to design a layout that will help the novice (either in experience or in the discipline) and not hinder the expert, or (ii) to assume that the novice will have to learn to use the reference system in the expert's way (as is implied in Chemistry and Physics journals, for example).

The compromise solution Layout (e) is an attempt at solution (i). It helps the novice, the expert and the reader from different disciplines:

(i) by separating out the main elements; (ii) by placing these elements in a sequence that experts

prefer; (iii) by presenting journal titles in full (and not

abbreviating them, as was the practice in earlier issues of Applied Ergonomics, and is still the case in many science journals); and

(iv) by positioning the date of publication m two positions so that the reference style is familiar to both British and American readers.

Despite Shackel's arguments, the data provided in this paper are clearly in line with the data presented earlier by Hartley et al. British judges prefer layout (d), which has all these features except the last one. However, because not all readers are British the concluding references to this article are set in layout (e).

References

Hartley, J., Trueman, M. & Burnhill, P. (1979) The role of spatial and typographic cues in the layout of journal references. Applied Ergonomics, 1979, 10, 165-169.

Siegel, S. (1956) Non-Parametric Statistics for the Behavioral Sciences. New York: McGraw Hill, 1956.


Documents

Sequencing the elements in references