From little slips to big disasters: an error quest
DOI:10.1093/acprof:oso/9780199228768.003.0018
Abstract and Keywords
This chapter presents a personal narrative which explores experiences in an error study. It explains that the study started with a quite narrow and strictly cognitive focus and later included an adequate theory of human skill that accounted for coherent action sequences that deviate from current intentions. In the mid-1980s, the distinction was made between unintended errors and intentional deviations from standard procedures, and by the late 1980s and 1990s, the study focused on organizational accidents and the so-called Swiss cheese model was developed.
Keywords: error study, human error, cognition, human skill, coherent action, unintended errors, Swiss cheese model, organizational accidents
A bizarre beginning
One afternoon in the early 1970s I was making tea in our kitchen when the cat arrived clamouring to be fed. I opened a tin of cat food and flicked a large spoonful into the teapot. I laughed it off, blamed the cat, and washed out the teapot. But on reflection, I saw that this slip had interesting properties. Both making tea and feeding the cat were routine activities performed in a familiar environment. My attention, hitherto occupied with unrelated matters, had been captured by the cat's pleading just at the moment when I was about to spoon tea-leaves into the pot; the act of dolloping cat food into some receptorlike object had migrated into the tea-making sequence. Even the spooning actions were appropriate for the substance: sticky cat food requires a flick to separate it from the spoon; dry tea-leaves do not. Local object-related control mechanisms were at work.
Although bizarre, it was evident that there was nothing random or potentially inexplicable about this action-not-as-planned. Both behavioural sequences were largely automatic. Both competed for my limited attention and effectors simultaneously, and an element from the higher priority cat-feeding task had intruded into the tea-making sequence at a time when spoon use was appropriate, and the teapot, like the cat's dish, afforded containment.
Little did I realize at the time that these and related issues would occupy me for the next thirty-odd years. This largely naturalistic mode of enquiry suited me well as I had little talent for or interest in laboratory experiments, particularly in cognitive psychology.
The scope of the chapter
When my error quest began, its focus was quite narrow and strictly cognitive in flavour. Any adequate theory of human skill must account for coherent (p.234) action sequences that deviate from current intentions, particularly when they take relatively few recurrent forms. Everyday absent-minded actions occupied the early years, the data being collected by either diaries or self-report questionnaires.
In the mid-1980s, largely due to analyses of the Chernobyl disaster (Reason 1987), it was clear that a distinction needed to be made between unintended errors and intentional deviations from standard procedures.
In the late 1970s and 1980s, a number of nasty disaster scenarios—Tenerife, Three Mile Island, Challenger, King's Cross, Zeebrugge, Clapham Junction, and the like—called for an even broader disciplinary spectrum. We need to distinguish between slips and mistakes (rule-based and knowledge-based), and between active failures (errors and violations) and latent factors—poor working conditions, fallible managerial decisions, cultural deficiencies, and regulatory and economic influences.
In the late 1980s and 1990s, my goal was to find a conceptual framework that could be applied to all such organizational accidents, regardless of domain: the so-called ‘Swiss cheese’ model. This went through a number of versions, but common to all of them was the central importance of systemic defences, taking many different forms, that prevented local hazards from coming into damaging contact with people, assets, and the environment.
Slips and lapses: the diary studies
Between the mid-1970s and early 1980s, we carried out a number of diary studies of everyday absent-minded slips of action (Reason and Mycielska, 1982) to clarify what goes absent in absent-mindedness, and to establish what circumstances are most likely to provoke unintended departures of action from intention. We also hoped to develop a classification of what appeared to be a limited number of recurrent error types.
As expected, absent-minded slips were most likely to occur in highly familiar and relatively unchanging surroundings—kitchens, bathrooms, bedrooms, offices, and the like—and while carrying out well practised tasks that diarists rated as being recently and frequently performed, and largely automatic in their execution.
Another factor was the inappropriate deployment of the limited attentional resources at some critical choice-point in the action sequence. For the most part this involved attentional capture by external distraction or internal preoccupation. But there were occasions when too much attention was directed at some largely automatic action sequence. This usually involved a ‘Where am I?’ query following an interruption. Two wrong answers could ensue: either that (p.235) the person was not as far along as they actually were—resulting in a repetition—or that they were further along—resulting in an omission.
Approximately 40% of all absent-minded slips, by far the largest single category of error, involved strong habit intrusions. Here actions are diverted from their intended route by the lure of some other well-trodden pathway. More specifically, they take the form of intact, well-organized sequences that are judged as recognizably belonging to some activity other than that currently intended. This other activity was consistently rated as being recently and frequently engaged in, and as sharing similar locations, movements, and objects with the intended actions.
Two further classes of action slip could be identified from our error corpus: place-losing errors, mostly involving omissions and repetitions, and interference errors, involving the blending of inappropriate routines, or the transposition of items within an action sequence.
Place-losing errors arise under a variety of circumstances. First, when a person makes a conscious check on the progress of a largely automatic sequence of actions, and comes up with the wrong answer—what we termed the ‘nosy supervisor’ syndrome. Second, when a habitual task is interrupted by the need to carry out another routine that was not planned. (For example: ‘I walked to my bookcase to find the dictionary. In taking it off the shelf other books fell on to the floor. I put them back then returned to my desk without the dictionary.’) Third, when a series of tasks is planned to run sequentially, and the individual moves to the next task before the current one is complete— the ‘premature exit’. And then there is the forgetting of previous actions when the person has no recollection of successfully completed routines. (For example: ‘While showering I could not remember whether I had washed my hair or not. If I had, then the evidence had been washed away.’)
Interference errors, on the other hand, result from ‘cross-talk’ between two currently active tasks (blends and spoonerisms), or between elements of the same task (reversals or spoonerisms). A typical blend is when elements from the previous task carry over into the next. (For example: ‘I had just finished talking on the phone when my secretary ushered in some visitors. I got up from behind the desk and walked to greet them with my hand outstretched saying “Smith speaking”.’) A reversal is when the actions are correct, but the objects for which they were intended get transposed.
Slips and lapses: questionnaire studies
In the 1980s, research groups in Oxford and Manchester used self-report questionnaires to examine individual differences in proneness to absent-minded (p.236) action slips and memory lapses (Broadbent et al. 1982; Reason 1989, 1993; Reason and Lucas 1984). In general, these questionnaires described a wide range of minor slips and lapses, and asked people to indicate along some ordinal scale how often a particular kind of error cropped up in their daily lives.
Although the specific forms of these questionnaires and the types of people to whom they were given differed from one research group to another, the results from these various studies showed a surprising degree of agreement— surprising, that is, for this kind of psychological research. The main findings are summarized below.
- ◆ There is strong evidence to indicate that people do, in fact, differ widely in their proneness to absent-minded errors. Proof that this is not simply a question of how they would like to present themselves is their spouses' agreement with their self-assessments. Moreover, responses suggested that a characteristic liability to error is a fairly enduring feature of the individual, at least over a period of 16 months or more.
- ◆ Liability to minor cognitive failures spans a wide range of mental activities and does not appear to be specific to any one domain such as memory, action control, and so forth. Thus, individuals who acknowledge that they experience more than their fair share of memory lapses also report making a relatively large number of errors resulting from failures of attention and recognition, and conversely. It would seem, therefore, that susceptibility is determined by the characteristic way in which some universal mental control process—a limited attentional resource perhaps—is deployed. This appears to operate relatively independently of the particular parts of the cognitive system in which it could show itself.
Attention control in a nutshell
Conscious concerns, whether internally or externally generated, consume the major part of the limited attentional resource during waking hours. In addition to continual moment-to-moment variations in the total amount of this resource that is available, the quantity drained off by these conscious concerns differs according to their nature and intensity. All mental and physical activities, no matter how automatic they may appear, make some demands on attention. The more habitual the activity, and the more invariant the environment in which it occurs, the smaller is this demand. But it is always present in some degree.
One difficult to dispute feature of absent-minded errors, as the term itself suggests, is that most slips and lapses occur when a large part of this resource (p.237) has been ‘captured’ by something other than the task in hand. Now, if our highly routine actions depart from intention because the limited resource is being employed elsewhere, the obvious conclusion is that, on those particular occasions, a greater degree of attentional involvement is necessary to ensure the desired outcome. I am not suggesting that a fixed amount of attentional resources is required throughout. But there are occasions, particularly at critical choice-points at which a familiar sequence branches into a variety of well-trodden paths, when a larger attentional investment is necessary.
As schemata (knowledge structures in long-term memory) appear capable of being activated independently of current intentions—by needs, emotions, context, associations with other schemata, and the frequency and recency of past use—some part of the attentional resource is always being deployed to restrain those activated schemata not required for our current plans. The more highly activated these unwanted schemata are, the more of the attentional resource will be consumed in inhibiting them. Execution of any activity requires the correct sequencing of several necessary schemata, and, as each of these transitions between schemata might potentially lead in many directions, some attentional supervision is needed to keep them on the right track. To do this it is necessary not only to select, but also to suppress, those preprogrammed action sequences that seek to usurp control at transitions.
If schemata were merely passive entities, like computer programs, which acted only on orders from above, this problem would not arise. But schemata behave in an energetic and highly competitive fashion to try to grab a piece of the action. Our evidence suggests that when the attention resource is largely claimed by something other than the immediate activity, it is this suppressive function that is most likely to fail. In short, this appears to be what goes absent in absent-mindedness.
Mistakes
If we define error as the failure of planned actions to achieve the actor's desired outcome (Norman 1981; Reason 1977), then there are two ways in which this can occur. Either the plan may be adequate, but the actions do not go as planned—these are the slips, lapses, trips, and fumbles that have been discussed above. Or the actions conform to the plan but the plan is inadequate to achieve its goal. Here, the failures reside at a higher level—with the mental processes involved in assessing the available information, planning, formulating intentions, and judging the likely consequences of the planned actions. These are termed mistakes and have been further subdivided into two categories according to the performance levels at which they occur (see Rasmussen 1983): rule-based mistakes and knowledge-based mistakes.
(p.238) Mistakes generally occur when we have to stop and think; that is, when we are faced with a problem or the need to make a decision or formulate a changed plan of action. Expertise in any field is based in large part on the possession of stored production rules of the kind if X then do Y, or if X has the features A, B, C, then it is a Y. Human beings are furious pattern-matchers. When confronted with an unplanned-for situation we are strongly disposed to identify a familiar pattern and, where necessary, apply a problem-solving rule that is part of our stock of expertise. But these pattern-matching and rule-applying processes can be in error. Rule-based mistakes take three basic forms.
- ◆ We can misapply a normally good rule because we fail to spot the contraindications. (Example: A general practitioner fails to identify that a child with a fever in a flu epidemic has meningitis.)
- ◆ We can apply a bad rule. (Example: The technician involved in rewiring a signal box just prior to the Clapham rail disaster had acquired the habit of bending back the old wires rather than removing them.)
- ◆ We can fail to apply a good rule. Standard operating procedures (SOPs) usually embody good rules. Failing to comply with SOPs can be both an error and a violation. We will discuss violations shortly.
The under-specification hypothesis
Is there a general rule that could be applied to all categories of error? Here is my attempt at one:
This may lead to error but it is none the less a highly adaptive process. For example, when asked who said,‘The lamps are going out all over Europe; and we shall not see them lit again in our lifetime’ most people (more than 90% in British audiences) respond with Winston Churchill. It was actually said by Sir Edward Grey, the Foreign Secretary, in 1914. The Churchill answer was wrong but it followed a sensible pathway. The quotation suggests that an English-speaking statesperson said it on the verge of something cataclysmic—the (p.239)Second World War. Who was the gabbiest statesman of that time? Winston Churchill. If we can predict an error with a high degree of confidence, we begin to understand the processes that give rise to it.When the mental processes necessary to ensure correct performance are under-specified— by any of a number of means: inattention, forgetting, incomplete knowledge, noisy sensory data, and the like—the cognitive system tends to ‘default’ to a response that is frequent, familiar, and hitherto useful in that particular context.
Widening the scope
Until the mid to late 1980s, my primary concern had been the cognitive mechanisms of error and the light that they could throw on the largely covert processes by which we control our routine everyday thoughts and actions. To this end, I sought to provoke predictable errors in captive lecture audiences. But the real world of hazardous industries was making itself increasingly felt, even in the ivory tower; most particularly this involved aviation, nuclear power generation, road transport, mining, oil and gas exploration, chemical process plants, railways and maritime operations. Gradually my focus shifted away from absent-minded people in familiar surroundings to the unsafe acts of those on the front line of complex, well-defended systems, as well as their teams, workplaces, and the organizational processes that shaped their behaviour. The detailed reports of catastrophic events proved to be a rich source of material regarding real-life mistakes and violations.
Violations
Violations—deliberate but usually non-malevolent deviations from safety rules and procedures—fall into three categories: routine, optimizing, and necessary violations. In each case, the decision not to abide by SOPs is shaped by individual, social, and organizational factors, though the balance of these influences varies from one type of violation to another.
- ◆ Routine violations typically involve corner-cutting at the skill-based level of performance (see Rasmussen 1983)—that is, taking the path of least effort between two task-related points. These shortcuts readily become a part of the person's behavioural repertoire, particularly when the work environment is one that rarely sanctions violations or rewards compliance. Routine violations are also promoted by ‘clumsy’ procedures that direct action along what seems to be a longer-than-necessary pathway.
- ◆ Optimizing violations, or violating for the thrill of it, reflect the fact that human actions serve a variety of motivational goals, and that some of these are quite unrelated to the functional aspects of the task. Thus, a driver's functional goal is to get from A to B but, in the process, he or she (usually he) can optimize the joy of speed or indulge aggressive instincts. Optimizing tendencies are characteristic of certain demographic groups, particularly young males.
- (p.240) ◆ Whereas routine and optimizing violations are clearly linked to the attainment of personal goals—that is, least effort and thrills—necessary violations have their origins in particular work situations. Here, non-compliance is seen as essential in order to get the job done. Necessary violations are commonly provoked by systemic deficiencies with regard to the site, tools, and equipment. They can also provide an easier way of working. The combined effect of these two factors often leads to these violations becoming routine rather than exceptional.
Organizational accidents and Swiss cheese
What distinguishes organizational accidents from other types of bad event is the presence within the system of diverse and redundant barriers, controls, and safeguards—defences-in-depth. Some of these defences are engineered: alarms, containments, physical barriers, automatic shutdowns, and the like. Others rely on people: pilots, ship's watch-keepers, surgeons, train drivers, and control room operators. Yet others depend on procedures, regulations, safety management systems, and administrative controls. Their collective function is to protect potential victims and assets from the operational hazards. Most of the time they do this very effectively, but there are always gaps and weaknesses.
In an ideal world, each defensive layer would be intact. But in reality they are more like slices of Swiss cheese, having many holes—though, unlike the cheese, these holes are continually opening and shutting, and shifting their location. The presence of holes in any one layer does not normally cause a bad outcome. Usually, this can happen only when the holes in many ‘slices’ line up momentarily to permit a trajectory of accident opportunity, bringing hazards into harmful contact with victims, assets, or the environment.
The holes in the defences arise for two reasons: active failures and latent conditions. Most organizational accidents involve a combination of these two factors.
- ◆ Active failures are the unsafe acts committed by people at the ‘sharp end’, the front-liners at the human-system interface. They take a variety of forms: slips, lapses, mistakes, and procedural violations. Active failures have an immediate but usually short-lived impact on the integrity of the defences. Followers of the person approach often look no further for the causes of an accident once they have identified these proximal unsafe acts. But, as mentioned above, nearly all such acts have causal contributions that extend back in time and up through the levels of the organization.
- ◆ Latent conditions and latent failures are the ubiquitous ‘resident pathogens’ within the organization. All systems have them regardless of whether they (p.241) have been involved in an accident or not. They arise from decisions made by designers, builders, procedure writers, maintainers, and senior management. These may have been mistaken, but they need not be. All such strategic decisions have the potential for introducing pathogens into the system. Latent conditions have two kinds of adverse effect: they can translate into error-provoking conditions within the local workplace (for example: understaffing, time pressure, inadequate tools and equipment, poor training, inexperience, and the like), and they can create long-lasting holes or weaknesses in the defences (untrustworthy alarms and indicators, unworkable procedures, weakened barriers, design and construction deficiencies, etc.). As the term suggests, latent conditions can lie dormant within the system for many years before they interact with local triggers and active failures to create an accident opportunity. This is when the holes in the cheese line up to create a pathway (often of very brief duration), allowing the hazards to cause harm and loss. Unlike active failures, whose specific forms are hard to foresee, latent conditions can be identified and remedied before an adverse event occurs. Understanding this leads to proactive rather than reactive safety management.
Current concerns
From the mid-1990s until the present, my main concern has been the problem of patient safety in medical care. This is a huge problem and exists everywhere. Around 10% of hospital patients are harmed, and sometimes killed, by medical and institutional errors. Healthcare professionals are not unusually fallible, but the conditions under which they work are highly error provoking: a huge diversity in tasks and equipment, little sharing of the lessons learned from adverse events, great uncertainty and limited knowledge, vulnerable patients, a one-to-one (or few-to-one) delivery of care, and a culture that equates error with incompetence.
A number of high-level reports (Donaldson 2000; Kohn et al. 2000) have strongly endorsed a systemic, as opposed to a person-oriented, approach to medical error. This is good, but it does little for the front-line professionals in the short term. They cannot easily change the system. Rather, they need ‘mental skills’ that will help them identify and step back from error-prone situations. We are developing tools to deliver this ‘error wisdom’ at the present time.
End piece
For me, there have been three main conclusions to emerge from this error quest. First, the same situations keep on producing the same errors in different (p.242) people—the problem lies mainly with error-provoking situations rather than error-prone people. Second, fallibility is part of the human condition. It is here to stay. We may tweak it, but we cannot change it. We can, however, change the conditions under which people work to make them less provocative and more forgiving. Third, error per se is very banal, even boring: it is like breathing or dying. What is truly interesting, however, is not the ‘human-as-hazard’ aspect, but the notion of the ‘human-as-hero’, that is, the adaptations, adjustments, compensations, and recoveries that front-line people make on a daily basis to keep imperfect systems working and mostly safe. These will occupy me for the next thirty years.
No comments:
Post a Comment