Hospital Standardised Mortality Ratio

The Health Quality & Safety Commission (HQSC) is publishing, probably on an annual basis, a new quality measure called the Hospital Standardised Mortality Ratio or HSMR. The first release of the report was on 24 September 2015, and covers how the HSMR nationally and for each district health board has changed since 2007 (the baseline year) to 2013.

The ratio compares actual deaths with deaths predicted by the HSMR model.

Many other countries, such as the UK, have been publishing this form of information for a number of years.

Waikato DHB results 2013 (published 2015)

This document shows just the national results and the Waikato DHB results [PDF, 585 KB]

Refer here for the full results of all district health boards.(external link)

The purpose of HSMR

The simplest way to think of the HSMR is as an earlier indicator of areas that may warrant closer investigation. When the HSMRs are significantly greater than 1, it is a signal to carry out a more detailed investigation, to check if there are issues with health care quality. This investigation should be done in the context of reviewing other quality indicators for the hospital. Some of these include patient experience, safety, workforce, clinical and operational effectiveness, and leadership and governance.

The HSMR has been under development for a considerable period by the Ministry of Health, working closely with HQSC. The Ministry has for some time been liaising with individual DHBs at chief executive level concerning earlier drafts of the HSMR and receiving feedback.

The data shows that New Zealand in line with many countries has a declining HSMR - however, as we would expect, there are variations between DHBs. The measure however is not intended to be used to compare DHBs with other DHBs.

A high or low HSMR (that is, a HSMR which is significantly higher or lower than 1) could be due to a range of causes.  It may reflect:

  • Differences in the organisation of end of life care in an area, with hospice services provided differently
  • Differences in the underlying risk of dying which the model cannot adjust for, such as the relative sickness of each patient with a similar condition
  • Differences in practice in recording data about patients and classifying them
  • Differences in quality of care provided
  • Random chance. 

From the measure alone we cannot tell which of these causes, or which combination of these causes, is responsible for the high or low HSMR.  For this reason a HSMR should never be used by itself as an absolute assessment of quality. HSMR should be looked at alongside other quality indicators such as patient experience and selected safety measures covering healthcare associated infections, falls in hospital, safe surgery and medication safety (the Quality and Safety Markers).  

Key points to note

  • Hospital Standardised Mortality Ratios (HSMRs) are one among many tools for quality improvement in hospitals.
  • The measure compares the observed number of deaths in a hospital with the number predicted adjusting for a range of known risk factors.
  • Every District Health Board (DHB) and hospital is different in the patients it sees – so HSMRs take factors like age, gender and clinical conditions into account.  Every DHB will also be different in how its services are organised.
  • It is important HSMRs take these differences into account because they can affect the results, but have nothing to do with actual quality of care.  However, these adjustments can never be made perfect and some differences may be due to other factors.
  • HSMRs will best be used as an indicator of where further investigation may be beneficial.
  • HSMRs should never be used to rank hospitals against each other, to draw simplistic conclusions about quality of care, or be interpreted as ‘avoidable deaths’.
  • New Zealand overall has a declining HSMR over time.  This trend is also observed in comparable countries such as the United Kingdom and Canada.
  • It is important HSMRs as used in New Zealand continue to be fit for purpose.  That is why the results will continue to be reviewed by the Ministry of Health, the Health Quality & Safety Commission, and the DHBs as part of the wider quality improvement package.

 

Frequently asked questions 

What are Hospital Standardised Mortality Ratios (HSMRs)?
HSMRs are a different way of looking at how one part of the health system is working over time. It is a useful addition to the growing range of quality-of-care indicators which are publically available. HSMRs are a way of comparing how many patients died within 30 days of admission to hospital, with how many we would have predicted to die given the age, gender, diagnosis and so forth of the patients entering the hospital. This comparison allows us to take account of factors which affect the risk of someone dying which may have nothing to do with the hospital in which they were treated. HSMRs will be published annually.

How are HSMRs used?
HSMR is widely used internationally, but on its own, it is not an indicator of the quality of care provided. It is best used to track changes over time. A sudden increase in HSMR may serve as a ‘flag’ highlighting a trend to investigate. A secondary use is to see if a particular hospital has a ratio significantly higher or lower than we would have predicted. Again, this may act as a prompt for further enquiries.

What does the number in the HSMR mean?
HSMR is calculated by dividing the number of observed deaths by the number of predicted deaths. So a HSMR of 1 means, that, the number of observed deaths is the same as the number of predicted deaths. When the HSMRs are significantly greater than 1, it is a signal to carry out a more detailed investigation, to check if there are issues with health care quality. This investigation should be done in the context of reviewing other quality indicators for the hospital. Some of these include patient experience, safety, workforce, clinical and operational effectiveness, and leadership and governance.

What causes a high or low HSMR?
HSMR takes many complex factors and produces a single number. However it is important to understand that this number in itself is not a guide to quality of care.

A high or low HSMR (that is, a HSMR which is significantly higher or lower than 1) could be due to a range of causes. It may reflect:
• Differences in the organisation of end of life care in an area, with hospice services provided differently
• Differences in the underlying risk of dying which the model cannot adjust for, such as the relative sickness of each patient with a similar condition
• Differences in practice in recording data about patients and classifying them
• Differences in quality of care provided
• Random chance.

From the measure alone we cannot tell which of these causes, or which combination of these causes, is responsible for the high or low HSMR. For this reason a HSMR should never be used by itself as an absolute assessment of quality.

What do you mean by “significantly higher or lower”?
Ratios go up or down a little every time they are measured. There are well-established statistical methods that identify which changes or differences are due to chance and which reflect something real. The word “significant” is used to describe changes or differences which we are confident are not caused by chance alone. Fuller details of how we work this out are provided in “The Ministry of Health and the Health Quality & Safety Commission Hospital Standardised Mortality Ratio Model – Methodology Paper (July 2015)”

What should HSMRs not be used for?
HSMRs should never be interpreted as a judgement of quality. They should also not be used to rank different hospitals. Most hospitals will not be significantly different to each other (that is, we suspect that any differences are likely to be caused by chance alone) and if we re-measured a month later the order would be likely to be completely different purely for reasons of chance. It is also not appropriate to interpret a high HSMR result (that is, the HSMR is significantly greater than 1) as evidence of ‘avoidable deaths’. The only way to tell whether a particular death was avoidable is through close review of the case notes and other locally held information. Even with this very intensive approach, it is not always possible to tell whether a death was avoidable. HSMRs simply do not do this.

What other quality dimensions need to be considered alongside HSMR?
HSMR should be looked at alongside other quality indicators such as patient experience and selected safety measures covering healthcare associated infections, falls in hospital, safe surgery and medication safety (the Quality and Safety Markers). These indicators are published regularly on the HQSC’s website. For further information on these indicators refer to the following web page: http://www.hqsc.govt.nz/our-programmes/health-quality-evaluation/(external link)

What questions might HSMRs prompt?
HSMRs will always require further investigation in order to be understood. The questions that HSMRs should prompt include:

• What is the overall ratio made up of?
• What is the variance between different parts of the hospital – are ratios high for some conditions or patient groups and low for others?
• Are there any issues about data or coding of patient conditions that we should take account of?
• Has there been any change recently, and has that change been focused in specific parts of the hospital?
• Are there any issues of quality that we are aware of from other sources? How do HSMRs coincide with these?

How will DHBs use HSMRs to improve their care?
HSMRs work best as a prompt for questions such as those above. Where these throw up quality issues, quality improvement activities that draw on evidence-based practice, increased consumer involvement and good improvement approaches are shown to drive improvement of services.

What is a predicted death? How is this calculated?
Predicted deaths are a term used by statisticians to refer to the likelihood of dying based upon age, gender, diagnosis and so forth. This measure is used internationally. It is applied after a patient has been discharged or died using the data collected about them as part of their hospital admission. This model produces a risk score for each patient of between 0 and 1, where 0 indicates no risk of dying and 1 indicates certain to die (in practice a score of 1 will almost never be applied). The sum of each of these risk scores inside a hospital for a given period of time forms the “predicted deaths” total, against which the actual number of deaths is compared. For example, a serious car crash victim will be likely to have a predicted death score of closer to 1, as their chance of survival is not high. This helps prevent hospitals which treat a large amount of very seriously ill patients having their HSMR higher as a result.

Why does predicted death number fluctuate each year?
The predicted deaths will depend upon both the number of patients admitted and the mix of risk factors. As both these change every year, the predicted deaths number for each DHB will also change every year.

Do these figures include palliative care?
Palliative care is care intended to treat symptoms rather than cure at the end of someone’s life. It is often synonymous with the care given in hospices. However, organisation of this sort of care varies. In some areas palliative care is provided by public hospitals, in others by hospices separate from the hospital. As palliative care is associated with very high mortality ratios, this variance in organisation can make a huge difference to HSMR. Various methods are available to address this. The method used in this model includes palliative care patients and all deaths within 30 days of a hospital admission regardless of where the death took place. This means palliative care patients who are admitted to hospital are accounted for in the same way in the model regardless of whether they are being cared for in a hospital or subsequently outside of a hospital. The model does not take into account of palliative care patients admitted directly to hospices, which may vary from DHB to DHB.

Can DHBs be compared using HSMRs? Why/why not?
Comparison between individual DHBs is unhelpful and misleading. Most differences will be statistically insignificant (that is, the difference is likely to be by chance alone, and if you repeated the measurement a month later you would be likely to get a different result, even though nothing about the two hospitals’ relative quality of care has changed in the meantime). What can be useful is to compare a hospital with the national average and within the hospital over time. A significant difference should be the trigger to carry out a more detailed investigation, to check if there are issues with health care quality. However, quality issues are just one of a number of potential causes of a significant difference, so no judgement about quality should be made until more detailed local analysis has been undertaken.

What does this data tell me about my DHB or hospital?
The data shows two things. How the HSMR for the DHB or hospital has changed since 2007 (the baseline year), and how the 2013 HSMR compared with the range that we expected it to be in. The HSMR does not on its own tell you how ‘good’ a hospital is.

What work have the Ministry and the Commission been doing on HSMRs?
As a result of international events, such as the case of Stafford Hospital in England (which resulted in the Mid Staffordshire NHS Foundation Trust Public Inquiry), HSMRs have been instituted in many countries as a way of early detection of potential problems. The Ministry and the Commission have reviewed a wide range of international variants of the HSMR measure and settled on the National Health Service Scotland as an appropriate methodology to replicate.

Why did the Ministry choose the Scottish model?
Similarities between health and data collection systems in the two countries make replication of the Scottish methodology more viable. The Scottish HSMRs also include post-discharge deaths (deaths outside hospital and within 30 days).

Can they be broken down by how people are dying?
No. HSMRs do not provide detail on the cause of death for individual events. The model uses primary diagnosis as one of the risk factors.

What are outlier results?
Outlier results refer to hospitals or DHBs which are significantly higher or lower than the national average – that is, the result is very unlikely to be just a reflection of random variation. Ratios go up or down a little every time they are measured. It is important not to see every small change as evidence of improvement or deterioration when in fact it may just be a reflection of random variation (sometimes known as “common cause variation”). There are well-established statistical methods that identify which changes are meaningful. In this document we use one of these – statistical process control – to differentiate sustained, meaningful changes (sometimes known as “special cause variation”).