QUESTION: I need surgery for a relatively minor condition and I’ve been speaking with my family doctor about which hospital to choose for my care. I’ve been researching them online and have come across a large number of ‘report cards’ and other information that seems to rate hospitals and doctors like hotels. How do I know if these ratings are reliable for making the best choices for my care?
ANSWER: In theory, scorecards of hospitals and doctors should be extremely helpful for patients. You could find out who is the best surgeon for an operation, or what hospital is the safest place to have surgery performed.
But, unfortunately, the existing rating systems don’t provide a lot of useful information to guide these decisions – at least for Canadians.
Patients in the United States appear to have more resources to consider. Major media outlets routinely produce glossy supplements touting the top docs and hospitals in America although some experts have taken issue with these rankings.
In Canada, fewer attempts have been made to rate our health-care system. One such project was spearheaded by CBC’s The Fifth Estate. The end result, “Rate My Hospital”, met with some criticism from the medical community.
I won’t comment about the merits of the CBC’s project. But I can share with you my experiences as a former Health Editor at The Globe and Mail. When I was at the paper some years ago, the senior editorial managers wanted to rate the safety of hospitals across Ontario. They planned to team up with a private think tank to do the analysis. The number of hospital deaths would be used as a key measure of overall safety.
I had misgivings about the project. I pointed out that it’s not appropriate to compare a hospital that treats very sick patients with one that performs relatively minor procedures. A hospital with the sickest patients is going to have more deaths – that’s to be expected. I was assured that those factors would be taken into account when the think thank crunched the numbers.
However, the analysis concluded that a small rural hospital was the safest medical centre in the province. I don’t doubt that the winning hospital does a good job patching up patients with broken bones and minor abrasions. But it wouldn’t be your first choice if you needed a heart operation, brain surgery or cancer treatments. The findings were essentially meaningless for all practical purposes.
In the end, The Globe backed away from the project, deciding not to publish the think-tank’s results.
The debate about how best to compare doctors and hospitals goes as far back as the 19th century when Florence Nightingale, the “Lady of the Lamp,” helped save countless wounded soldiers during the Crimean War by implementing hand-washing and other hygiene practices in the field hospital where she worked.
After returning to England, Nightingale who was a nurse, statistician and social reformer, felt that many hospital patients were dying needlessly because of inappropriate care. She teamed up with another statistician to produce tables comparing death rates at different hospitals in England. Some critics insisted that it wasn’t fair to compare rosy-cheeked patients at rural hospitals with sicker patients being treated in London’s grimy inner city. They had a valid point.
Since then, health policy experts have come to realize that any comparison has to take into account that patients are not the same in terms of their risk of dying. But that’s easier said than done.
One commonly used approach is known as the hospital standardized mortality ratio or HSMR. It compares the observed number of deaths in a given hospital with the expected number of deaths based on factors like the age and diagnoses of the patients.
In other words, some patients are going to die, regardless of the care they receive. What you want to know is whether the care makes a difference in the outcomes. For instance, some hospital patients may catch infections that lead to their deaths when they would normally have survived.
Using the HSMR is supposed to put hospitals on a level playing field in order to make fair comparisons.
But even this approach can generate misleading results, says Dr. Kaveh Shojania, who is the director of the Centre for Quality Improvement and Patient Safety at the University of Toronto.
The severity of a patient’s condition may not be accurately reflected in the statistics that are used for these studies, says Dr. Shojania, who is also a physician of internal medicine at Sunnybrook Health Sciences Centre.
Dr. Shojania gives the example of a 40-year-old male patient who is admitted to hospital with severe pneumonia. In some ways, the middle-aged male appears less sick than an 80-year-old who has multiple health problems. “Yet the 40-year-old could have really, really bad pneumonia and may be at a higher risk of death than an 80-year-old with mild pneumonia,” says Dr. Shojania.
So the validity of the findings largely depends on the data on which the study is based. In some cases, certain hospitals may do a better job of recording the actual seriousness of a patient’s condition. In other situations, the data just do not reflect illness severity.
Dr. Shojania also notes that some hospitals may appear to be less “safe” than other medical centres simply because they are actively trying to identify problems that need fixing.
“If you have a good infection-control program, you are going to look very carefully for any case that might be C. difficile (a common cause of infectious diarrhea) – and you’re going to find more cases,” he explains. “There might be other hospitals that have high rates of C. difficile but the cases are not identified. They are going to look like they are better hospitals.”
Dr. Shojania believes that it’s important for hospitals to keep track of mortality figures, hand-washing efforts and readmission rates from complications following surgery.
Even when these measures are recorded accurately, they may still not serve as a meaningful guide for patients who want to know where they should seek care.
“If you’re having a knee replacement, you’re not expecting that you might die from the procedure,” says Dr. Shojania. The hospital mortality rate may not seem very relevant.
But there are other questions you want answered, such as: how long will you have to wait for an appointment with your preferred surgeon; what will the recovery involve; and when will you be able to resume normal activities after the operation?
In Canada, we don’t tend to collect that type of information and make it readily available to the public.
The National Health Service in England, however, has been gathering data on the effectiveness of care delivered to patients as perceived by the patients themselves. It’s called Patient Reported Outcome Measures or PROMs.
To get input from patients, hospitals in England have been inviting patients to complete questionnaires before and after certain types of surgeries, including groin hernia operations, hip replacements, knee replacements and varicose vein operations.
“In some ways, the U.K. strategies have been admirable,” says Dr. Shojania. “It’s a laudable initiative.”
Canada should also consider collecting similar data, says Dr. Chaim Bell, an Associate Professor of Medicine and Health Policy Management & Evaluation at the University of Toronto. Patients might find the information extremely informative when they are making decisions about their care, he says. As well, it could point to new ways of improving the delivery of health care.
“It is not easy to measure patient-related outcomes, but they are important,” says Dr. Bell, who is also a physician and scientist at Mount Sinai Hospital in Toronto.
With this background in mind about the limitations of rating systems, let’s now consider what’s available online.
The rating systems fall into two general categories:
- One type relies on the opinions of patients who remain anonymous. They voluntarily log onto websites, such as RateMDs.com, to express their views about their own physicians.
- The other type is based on information collected through the health-care system. The data may be derived from a review of patients’ medical charts. Or, it could come from an analysis of provincial records of doctor billings. The Canadian Institute of Health Information, for instance, uses billing information to create the “Your Health System” website which provides comparisons of indicators such as hospital deaths and readmission rates.
The first thing you need to know about the anonymous comments is that they don’t include a representative sample of patients.
“It is a very specific population of people who post online,” explains Dr. Jessica Liu, an internal-medicine physician and a quality-improvement researcher at Toronto General Hospital.
People have to feel very strongly to make the effort of adding their opinions to a website. The patients who do so “are often very satisfied or very dissatisfied with their physician, so in that way, online reviews can be polarized,” says Dr. Liu.
Furthermore, some points of view will be absent because not all patients are computer savvy. A doctor could do a great a job of looking after seniors, but few of these patients may make their views known online.
Despite these limitations, the websites are a “reflection of the patient experience and give us insights into understanding what patients feel is good-quality care,” says Dr. Liu.
Of course, with any opinion-based rating system, you need to take the views expressed with a healthy dose of salt. It’s just like a review for a restaurant or a movie. Ask yourself if you would feel the same way. Does the complaint or praise seem authentic, legitimate and resonate with you?
In some respects, rating systems built upon health-care data have the appearance of being more scientific – and therefore more reliable. But data can be misleading, too, identifying minor differences that are not necessarily meaningful. “Only the top 2.5 per cent and the bottom 2.5 per cent are statistically different from everyone else,” says Dr. Shojania. “In reality, those in the middle are indistinguishable from one another.”
He also points out that care may not be the same across all departments of a hospital. So, when you are looking at any rating system, you should consider how the information relates to the treatment you need. “A particular hospital may be the best place for transplants, but if you get admitted for a problem in another specialty, it might not be so great,” he says.
Ratings, based on numbers alone, can’t capture all the intangible experiences that are important to patients. “Sometime it’s about the environment,” says Dr. Shojania. One department might have received a large donation for a brand-new wing or a major renovation. The upgraded facilities could make patients feel better about their treatment.
If your main goal is to be operated on by a “top” surgeon, keep in mind that what happens while you’re under the anesthetic is only part of your hospital journey. Other staff members, such as nurses, usually provide much of the post-surgical care. And if you’re at a teaching hospital, you will be treated by physicians-in-training – residents, fellows and medical students. In fact, your surgeon may not actually do the procedure. Instead, all or part of the operation may be carried out by one of the residents or fellows working under the supervision of your surgeon.
Overall, most patients can expect to have an “average” experience – and that’s not necessarily a bad thing provided the health system is delivering generally good and timely care. “The reality is that only a small percentage of hospitals or doctors will be significantly different from the rest,” say Dr. Shojania. “A small number will be at the extremes. They are the outliers. They will be clearly worse or clearly better. But most will be just average.”
Nonetheless, our fixation with ratings of all kinds will likely mean there will be more scorecards for Canadian hospitals and doctors. That could be a positive development if we eventually get access to information that’s truly meaningful to patients.
But a scorecard – even a good one – is not a substitute for an actual conversation with your prospective surgeon.
Dr. David Urbach, a surgeon at Toronto General Hospital, says friends, family and hospital VIPs often ask him for his opinion of other doctors. “The bottom line is that no one really knows where someone stands in a ranking.”
He says patients should try to make their own assessment of a doctor. “Meet with the surgeon,” he advises. “If you feel the surgeon is listening to you, has answered your questions and if what is being recommended sounds sensible, then I would go with that surgeon because the chances are that you will be fine.”
If you have doubts or reservations, then it is certainly worthwhile getting another opinion, he adds. Your family doctor can usually arrange a referral to another surgeon.
“What you really want is someone who is going to be available for you – and is recommending a course of treatment that make sense to you,” says Dr. Urbach.
