Major Howard Colivn: We can show you charts and statistics like they mean something, but you're going back to your home tonight, we're going to be in our patrol cars and them boys still gonna be out there on the corners. Deep in the game. This is the world we got, people, and it's about time all of us had the good sense to at least admit that much.
Man at meeting: So, what's the answer?
Major Howard Colvin: I'm not sure. But whatever it is, it can't be a lie.
--"The Wire"
Whether it's doctors or schoolteachers or policemen or baseball players, everyone wants to see "good" performers rewarded and retained and bad players discarded.
My mother, who taught high school for 30 years used to say, "The only important job for a principal is to know who his good teachers are, and to keep them, and to know who his bad teachers are, and to get rid of them."
The problem is, when judging human performance in complex service arenas, how do you define "good" and how do you measure it? We would all like some objective criteria in a subjective world. Objective criteria mean that we do not have to trust the evaluators quite so much. After all, the "teacher's pet" can curry favor and be promoted for all the wrong reasons.
Police departments have famously embraced "statistics" to judge the efficacy of their police officers, with increasingly notorious results. "The Wire" explored the folly of the misuse of data driven policing in Baltimore long before the recent riots there exposed the rotten core of that police culture.
After many episodes demonstrating the perversions and distortions of good policing, Major Howard "Bunny" Colvin says, simply, "The problem is, statistics done killed this job."
Atul Gawande, writing in the New Yorker investigated what made the number one rated Cystic Fibrosis Clinic in America so very good. He was asking the question: How do you get good performance out of an organization? So he went to the clinic and followed its director around. The director, a physician, steps into a room with a patient, scowling at the results of her pulmonary function tests, which have shown a steep decline. "What happened here?" he asks her. Turns out, at at 23 she has her first boyfriend, has moved into his apartment and she does not want to do her thrice daily "pulmonary toilet" there, which involves trying to york up sputum and involves many revolting sounds she knows would turn him off. The director explodes, "So you stopped doing your part here! What I'd like to ask you is what you are doing to make this the best CF clinic in America? You have betrayed yourself and this clinic and all we've done for you."
Of course, the patient dissolves in tears and it's clear she'll never set foot again in that clinic. But her departure means the clinic will continue have excellent statistics, the pulmonary function tests of its patients will not be polluted by her poor results and if her lifespan is shortened, at least those diminished years will not reflect poorly on the clinic.
Gawande never comments on the obvious problem here. He is too in thrall with the "success" the clinic has.
In Moneyball, Billy Bean explores the weakness of the subjectivity of the traditional baseball scouts, who look at players and judge them and their potential. The problem with their subjective approach is they tend to pick players who "look good in jeans" but they miss the players whose bodies may look awkward but who make teams win. Employing statistics, "metrics" allows Bean to find players who were missed because they looked homely or too small but who got on base a lot and scored runs and his team thrived.
The rub here is that there is a bottom line "score" to follow to judge the success of an organization. In teaching, medicine, policing defining the "bottom line" is often more difficult.
"The Wire" has mordantly funny scenes based on the perversion of the use of statistics. At the Homicide office they keep a large whiteboard where murder victims are listed and there is a column for the police officer investigating and a column for solved and a column for unsolved. When a row of abandoned townhouses is discovered to have scores of bodies lined up inside, the higher up try to prevent more boarded up townhouses from being opened, for fear more unsolved murders will pile up on their whiteboard. The solved murder rate drops from 55% to 30% almost overnight. The police brass are apoplectic. They are not concerned about the gang who is murdering; they are concerned about the statistics and their own jobs.
Yesterday, I got a complaint forwarded to me from the administrators from a primary care doctor. He had referred me a patient whose glycohemoglobin test was pretty dismal. This test reflects the blood sugar control for the prior three months. A low number, 6%, indicates good control a high number indicates poor control. After two years of working with this patient, her number had gone from 13.5% to 13.8%.
She missed most of her appointments. Over 2 years she kept only 4 of 8 appointments and each time I discovered she had not initiated the insulin I had recommended. Finally, at her last visit, I asked again why she was not making the changes I suggested, not testing her blood sugars at home, not keeping appointments.
"Well," she said, "I don't check the sugars because they are always awful. I don't keep the appointments because I don't want to think about my diabetes. And I don't take the insulin because I think I'm already on too much medicine."
"Sounds like you're not really ready to deal with this problem," I said.
She nodded. We had gone over all this, all her options, tried to make accommodations to her regimen to pare it down to the minimal effort but she was simply unwilling to face her disease. We explored the "why" of this but she simply smiled and said, "That's just where I am."
So, I wrote her primary care physician saying, when she is ready to be part of her own care, send her back, but we needed spots in the Endocrine Clinic for patients who might benefit.
The PCP promptly called the Medical Director, who called the administrators. "You've left the PCP dangling. Now he's got a patient with a glycohemoglobin of 13.8%. That's going to ruin his statistics. That means he'll get penalized on "quality" and his salary will drop."
Nowhere in any of this was the patient mentioned.
I was instructed to send the patient to nutritional counseling and to the Wellness Program, which consisted of three scheduled phone calls with a wellness nurse.
"Do you really think the nutritionist or the telephone nurse can convince this lady to take her insulin?"
"Of course not, but it's the protocol. This is all about process."
"How about asking the patient if she wants to see a psychiatrist?" I asked. Such referrals, in our system, have to come from the PCP, not the consultant. "Or maybe, another endocrinologist?"
"That's not the protocol."
This is not unique to our medical system. Teachers' evaluations may come down to how well their students do on standardized tests. Never mind, good teachers may be faced with the results of poverty, generations of uneducated parents who work three jobs and who do not read themselves and who do not read to their children, who are not home for their children, or parents who spend the day in a heroin stupor on the couch in front of the TV and all that produces the child in the classroom who does not do well on a standardized test and it's the teacher's fault.
In another scene in "The Wire" a young cop presents statistics at a community meeting showing the drop in crime in a neighborhood. A woman stands up and waves dismissively at the charts and says she remembers a time when their neighborhood had a policeman on the beat, a White guy, but a good guy nonetheless. He knew her name and would sit down for a moment and ask about each of her children. He knew everyone in the neighborhood and they knew him. And when a kid got shot, or a girl got raped, and the neighbors knew who did it, they told this cop. But now the only cops they see flash by in police cars with the windows up. Nobody knows these cops and certainly would not talk to them or give them any information. How does all that show up on your charts?
These are all questions about management and intelligence at the top. When you use the wrong metrics, you get a perversion of the ultimate aim. Is our public education system working? Well, look at graduation rates. What happens? Schools game the system and graduate kids who cannot read, have never learned to show up on time, if at all and so the high school diploma from an inner city school is meaningless to any employer or college.
I'd like to know whether these problems have been solved anywhere else in the world any better than we have solved them.
Do other countries look at their police, their educational systems, their health care systems any more intelligently than we do?
No comments:
Post a Comment