The paper by Spiegelhalter is a valuable contribution to the literature on presenting and displaying performance related outcome measures.1 It provides further methodological guidance on identifying service providers whose performance falls outside control limits using funnel plot methodology. When reporting on performance it is important to have procedures in place which should be followed when outliers are identified. These issues have been considered by the Paediatric Intensive Care Audit Network (PICANet) who use the funnel plot methodology for reporting risk adjusted mortality from all paediatric intensive care units (PICU) in England and Wales. Before producing these funnel plots for the latest national report, we issued a policy statement drawn up in consultation with both our Clinical Advisory and Steering Groups.2 In summary, the PICANet policy (published in full at http://www.picanet.org.uk) recognises that a PICU whose risk adjusted mortality lies outside the control limits will be identified as having returned data that are markedly different from other PICUs. It is important to note that this is not sufficient evidence to suggest that it has either markedly higher or lower mortality than other PICUs, but merely that the data it has returned are different from those of other PICUs. To resolve why these data are different, PICANet will work with the units to provide a satisfactory explanation using the following plan.
- Review the data to investigate whether there are data driven reasons for a PICU lying outside the control limits (it is known that risk adjustment tools can be unreliable when a PICU has a particularly high proportion of patients at either end of the bounds of the tool).
- Review the quality of data supplied by the PICU. The quality of the data is the PICU's responsibility. PICANet will provide feedback from PICU data validation visits and central validation procedures. PICUs will be expected to check the quality of individual data items.
- Plot the data quality indicators over time to identify whether the anomaly can be traced to a certain data collection period.
- Plot the mortality ratio over time to identify whether the anomaly can be traced to a certain data collection period.
- Plot the observed mortality over time to identify whether the anomaly can be traced to a certain data collection period.
- Plot the expected mortality over time to identify whether the anomaly can be traced to a certain data collection period.
- Investigate the primary diagnoses for admissions to the PICU. If the PICU has a very different diagnostic case mix than other PICUs, this may suggest that further refinements to the risk adjustment method are required.
- Produce a brief summary report of the above for the lead clinician and Chief Executive at the PICU concerned together with an invitation to meet in person to review the data with the PICANet team.
We believe that having such a policy in place, clearly outlining our interpretation and proposed actions before publication of such funnel plots, is vital to the chances of such information being accepted by staff at the participating units and thus more likely to result in positive actions being taken.