Do checklists work?

Tagged Checklist, Complications; Comments 2 ; Posted 11 years ago

In 1934 the US Army evaluated 3 different aircrafts for testing, including the Boeing model 299.  The Boeing plane seemed to be superior to any of the other two aircraft. At formal testing the aircraft made a normal taxi and take-off.  It began a smooth climb but then suddenly stalled.  The aircraft toppled over and fell, bursting into flames upon impact.  From five people on board two died of their injuries.

An investigation found “pilot error” as the cause.  The pilots had forgotten to release a lock prior to take-off.  Once the plane was in the air the engineers apparently realised the mistake and tried to reach the lock handle but it was too late. The public opinion at the time was Boeing 299 was “too much a plane for one man to fly”.

Some pilots sat down afterwards to analyse the accident.  They were sure that the model was not too much of a plane to fly.  It was simply too complex for any one’s memory.  They developed four checklists: one for take-off, flight, prior to landing and after landing.  With those check lists in place 12 aircraft managed to fly 1.8 million miles without a serious accident.  The idea of pilot checklists was born and it is now unthinkable to fly any aircraft without checklists.

There are numerous similarities and dissimilarities between the aviation industry and medicine. Would aviation be as (un)safe as medicine, no one would fly.

Surgery is a rather complex undertaking and sometimes we only notice complexity when things don’t go well or fall apart. We notice it when people involved in a surgical procedure are unfamiliar with it or if staff doesn’t exactly know what to do and how to assist with the procedure. Every surgeon knows that it is virtually impossible to verbalise all tasks that need to be done. Let alone to verbalise things that must not be touched, etc.

I work at my best if I don’t actually need to ask my scrub nurse for a particular instrument. If my surgery goes really smooth, my scrub nurse would be so much on my side and with me at the operation that she would exactly know what we are doing at every step; she would recognise difficulties and problems without me saying a word. Me and my team are one.

While not all complications are the results of errors, checklists can prevent errors and therefore reduce complication rates. In 2009 the WHO checklist project, driven by general surgeon Dr Atul Gawande was published. It contained 22 items that are supposed to be checked before induction of anaesthesia, before skin incision and before the patient leaves the operating room. Dr Gawande’s NEJM paper introduction of checklists decreased the inpatient mortality by half and reduced inpatient complication rates by one third. The WHO checklist was implemented in virtually all hospitals in Australia based on recommendations by professional standards organisations.

The science behind the WHO checklist has been criticised for mainly two reasons. First and foremost, the study was not randomised. There was no other reason than lack of funding to not to randomise and we should consider that randomised trials remain the golden standard to prove the efficacy of an intervention. Secondly, from the eight hospitals that tested the checklist, four noticed a benefit from the introduction of the checklist but the other four did not benefit. No attempt was made to explain that.

A couple of weeks ago I came across a review that attracted my attention: “Surgical checklists: a systematic review of impacts and implementation”. There are three or four major surgical checklists and the most commonly and widely used is the WHO checklist. The group of authors reviewed 23 studies on the WHO checklist and extracted information on outcomes (avoidance of complications) as well as facilitators and barriers of implementation.

• A UK hospital tested for 485 orthopaedic operations in 2009.  Outcomes were compared pre vs. post training (historical comparison group). Correct checklist use was 97% and it took two minutes to run the checklist. One in five surgeons thought that running the checklist caused an unnecessary time delay.  Early complications reduced marginally from 8.5% before checklist training and 7.6% afterwards.  Mortality remained unchanged.  Lower respiratory tract infections were 2.1% before checklist training and slightly higher after.  Surgical site infections were slightly lower after checklist training and the unplanned return to the operating room was completely unchanged. 
• Two hospitals in Liberia participated in a before - after evaluation of the WHO checklist.  Patients were followed for 30 days. The authors concluded that the introduction of the checklist allowed the hospital to purchase vital equipment, such as pulse oximeters. However, the checklists did little to change the entrenched hierarchy and relationship dynamics. Nevertheless, the introduction of the checklist was associated with decreased surgical site infections by 80%, decreased surgical complications rate by 65% but the effect was limited to only one of two hospitals.  There was no improvement in surgical outcomes otherwise.
• Finally, an Iranian study that was rolled out nationally, suggested that the introduction of checklists reduced surgical complications from 22.9% to 10.0% after checklist implementation.

Twenty-three studies onthe WHO checklist were analysed. None of the studies were randomised. Three studies included a historic control group and reported health outcomes. From these three studies, two groups have shown dramatic improvements; one from Liberia, one from Iran; both originate from low-resource counties. One study from the UK has shown (almost) no improvement.

All other studies either did not include any comparison group or they did not report health outcomes. The science of checklists …. sparse.

My personal experience with hospital-provided checklists is mixed. I coudl not see myself running a theatre without a checklist. That would be pure blind flight. However, some adminsitrators don't mnake it easy for us doctors to accept and embrace them. For example, the insist in some palces to run the exact checklist three times in the operating room before the operation gets underway. That’s not safe, that’s insecure. To re-introduce each other in the context of “everybody knows everybody by name” again is awkward. I spent a week at a New York hospital and the throughput of nurses for only one case was about five to ten times bigger than with any of my long cases. While introducing and re-introducing each other in this context (with many people attending a case) makes sense, it leaves a feeling of ????? in my theatre. Context is critical, as Dr Gawande points ourt repeatedly. 

I run my own checklist in addition to the hospital mandatory checklist very comfortably and I would not give up running them for a lot of money. Also, I am not alone doing so. Some 43% of Australian O&G specialists use their own preoperative checklist in addition to any mandatory hospital checklists, according to a survey conducted in February this year amongst A&NZ O&G specialists and trainees.

If we consider checklists as just another therapeutic intervention we would probably assume:

1. Very likely, checklists do work;
2. The magnitude of its effect is unknown and highly likely depends on its environment;

To subscribe and receive regular blog posts through email - leave your email address here.

Feel free to pass this blog on to colleagues.