Listening to David Simon on Bill Moyer's Journal, one thing stood out among all the many truths Simon spoke: The use of numbers, statistics, has been used in corrupt and damaging ways which have undermined the very purposes of the institutions we created to protect ourselves.
This is made particularly clear in The Wire as it pertains to the way crime statistics have perverted and nearly destroyed police departments and this is true for public schools, which have been similarly damaged by the unintelligent and corrosive use of statistical measures, judging teachers by the tests scores of their students, and it was true during the Viet Nam war where "body counts" were used as evidence we were winning that war, when in fact we were losing it. Robert MacNamara was the epitome of the man who made a career creating an image of himself as the smartest boy in the room, the man who could prove with mathematical "precision" how we could and would win that war.
This is made particularly clear in The Wire as it pertains to the way crime statistics have perverted and nearly destroyed police departments and this is true for public schools, which have been similarly damaged by the unintelligent and corrosive use of statistical measures, judging teachers by the tests scores of their students, and it was true during the Viet Nam war where "body counts" were used as evidence we were winning that war, when in fact we were losing it. Robert MacNamara was the epitome of the man who made a career creating an image of himself as the smartest boy in the room, the man who could prove with mathematical "precision" how we could and would win that war.
One area which would seem immune from the criticism concerning the importance of statistics would be medicine, which is, after all a science, we would like to believe. Another is finance and economics, which are, after all, in their very essence, numbers games.
But the problem is, numbers are really just dumb tools and they have to do with measuring things. But in order to understand the way things work, you really have to have a qualitative insight, and the quantitative tools cannot by themselves provide that.
The numbers are supposed to protect us from our emotional side, from the subjective, from biases. They are just the facts ma'm.
Consider two instances of the uses of numbers which have been found to be very persuasive by those who wanted to believe, in a qualitative way, what these numbers were supposed to mean. Both instances arise from New Yorker articles by Atul Gawande.
Gawande, it must be understood, is a very nice man. He is well meaning, bright, earnest and when he is telling a story about an individual patient, he can be quite moving and effective. That is, he is telling us something which has quality to it. But when he gets out of the operating room and tries to tell us about a whole, complex system based on numbers when he really doesn't understand how they were collected, he unwittingly misleads.
And the President reads Gawande.
When Gawande strays into territory beyond his immediate experience, beyond the particular experience he has had with a particular patient, he begins to worship the false god of quantitative thinking, of misinterpreted numbers.
The first instance is his article about a cystic fibrosis clinic which has far better numbers for survival than any other CF clinic in the country. I forget the exact numbers, but if the average life expectancy at the average cystic fibrosis clinic is 34 years, then this clinic was getting 45 years, numbers like that. A really impressive difference.
Gawande, it must be understood, is a very nice man. He is well meaning, bright, earnest and when he is telling a story about an individual patient, he can be quite moving and effective. That is, he is telling us something which has quality to it. But when he gets out of the operating room and tries to tell us about a whole, complex system based on numbers when he really doesn't understand how they were collected, he unwittingly misleads.
And the President reads Gawande.
When Gawande strays into territory beyond his immediate experience, beyond the particular experience he has had with a particular patient, he begins to worship the false god of quantitative thinking, of misinterpreted numbers.
The first instance is his article about a cystic fibrosis clinic which has far better numbers for survival than any other CF clinic in the country. I forget the exact numbers, but if the average life expectancy at the average cystic fibrosis clinic is 34 years, then this clinic was getting 45 years, numbers like that. A really impressive difference.
To his credit, Gawande did not simply read the numbers, but he hopped a plane and visited the clinic to find out what this clinic could teach us about how to improve the quality of care for CF patients. The numbers, after all, clearly told us this is this is by far the best, the highest quality care for cystic fibrosis in the country. The numbers were undeniable, and could only mean one thing. (Beware numbers which can only mean one thing. Remember, numbers are dumb.)
The patients at this clinic lived ten years longer than anywhere else; they got beyond death in young adulthood to middle age. Following the head of the clinic, Gawande saw this doctor examine the record on a patient who had been doing very well on her lung function tests but had over the prior three months deteriorated sharply. Questioning the patient relentlessly, it turned out she had acquired her very first boy friend, and she had moved in with him, and not wanting to gross him out with the "pulmonary toilet" she had previously performed on herself several times a day, she had done less and her lung function tests revealed the difference in her effort.
The patients at this clinic lived ten years longer than anywhere else; they got beyond death in young adulthood to middle age. Following the head of the clinic, Gawande saw this doctor examine the record on a patient who had been doing very well on her lung function tests but had over the prior three months deteriorated sharply. Questioning the patient relentlessly, it turned out she had acquired her very first boy friend, and she had moved in with him, and not wanting to gross him out with the "pulmonary toilet" she had previously performed on herself several times a day, she had done less and her lung function tests revealed the difference in her effort.
The doctor was outraged. What have you done to make this clinic the best CF clinic in the world? You have let us down. You've let yourself down. Sure the coughing and wretching and hacking up secretions from your lungs may disgust your boyfriend, but we're talking about your lung function here. This set of testing makes our clinic look bad. We look like a failure. The patient broke down in sobs and we got no more information from Gawande about whether or not she ever returned to clinic.
We really didn't need it. It was pretty obvious that patient was either going to comply with her pulmonary toilet regimen or she would not have the nerve to ever show her face again at that clinic.
So who benefited? If the patient slinked out of the clinic, never to return again (a distinct possibility, given her humiliation) she'd take her stinking numbers with her and rid the clinic of the stench, but who was served by humiliating her and throwing her off the train?
At least in that case, the numbers reflected what was actually happening, i.e. a loss of lung function which resulted from a loss of effort. But the loss of lung function might be a price that woman was willing to pay, even if it meant she would die at age 40 instead of 45. Is it better to die a virgin at 45 with great numbers or to have lived and loved and die 5 years earlier?
I could think of numbers which didn't even tell you what you thought they did: I signed death certificates for years while covering for other doctors who signed out their patients over the weekend by telling me Mrs. Smith might die over the weekend and I should sign the death certificate. With what diagnosis? Oh, I don't know. She's got a lot lot of things racing to be the cause of death, uterine cancer, heart failure, diabetes. Pick one and sign your name. So I picked one out, more or less at random and that diagnosis became her cause of death and got entered into the stream of information emanating from death certificates. And then you read about "leading cause of death" or how many people die from heart failure and you know they are based on garbage in/ garbage out systems of collecting those numbers.
The more famous New Yorker article was the piece about the Texas communities, where one town had a hospital which billed the highest rates of Medicare cost in the country while a neighboring town was at the low end of the scale. Gawande tells us that by "every measurement of quality of care" the high cost hospital was no better than the low cost hospital. And he flies to Texas, has dinner with doctors from the high cost place and they "admit" they have probably been bilking the system, doing procedures which fattened their own wallets but did not much improve the health of the patients. And Gawande accepted their confessional as an admission of guilt by five doctors on behalf of the hundreds of doctors who were not at the dinner.
Of course, this plays to the longstanding suspicion about doctors doing unnecessary things to enrich themselves.
But the sad fact is, there are no good "measurements of quality" in medicine, or if there are any, they are very limited and you certainly could not judge differences between the two hospitals without looking much more carefully at individual patients. You'd have to do chart review and ward rounds at both places for months to get a sense for whether there was any real qualitative difference. That is, you'd have to be scrounging around the wards, and not just looking at numbers.
That's what Howard "Bunny" Colvin is talking about when he says statistics ruined policing in Baltimore. Used to be the cop sat on the stoop with the ladies in the neighborhood, and when there was a shooting, the cop had people who considered him a friend and told him who and what and wherefore. That quality of the cop on the beat could not be measured by arrest statistics.
That is the difference between teaching kids probablity theory by tricking them into thinking they are not learning, when you show them the probabilities which govern the dice game of craps. How do you capture the creativity of a the classroom teacher to sizes up the psychology of his students and finds a way to sneak some learning into them? In The Wire, the teacher has the choice of teaching his students what they want to learn or teaching them nothing in a doomed effort to teach to the test. The test results generate statistics. Real learning in the classroom does not.
That is the difference between teaching kids probablity theory by tricking them into thinking they are not learning, when you show them the probabilities which govern the dice game of craps. How do you capture the creativity of a the classroom teacher to sizes up the psychology of his students and finds a way to sneak some learning into them? In The Wire, the teacher has the choice of teaching his students what they want to learn or teaching them nothing in a doomed effort to teach to the test. The test results generate statistics. Real learning in the classroom does not.
Juking the stats means lieutenants make majors and majors make colonels and it's great for career advancement and for running for re election but it's terrible for school children, teachers, doctors, clinics and for most important institutions in this country.
Our addiction to the manipulation of dumb numbers has corroded our society from within. It's why we are no longer the best in health care; it's a small part of the reason our schools are failing to do as well as they might otherwise.
We have gotten way stupid in our attempt to look smarter by quoting numbers. Until you understand how numbers are collected, you are as dumb as the numbers themselves. And I use that word "dumb" advisedly, in the sense of dumb as in unable to speak. Numbers do not speak. Someone has to speak for them.
We have gotten way stupid in our attempt to look smarter by quoting numbers. Until you understand how numbers are collected, you are as dumb as the numbers themselves. And I use that word "dumb" advisedly, in the sense of dumb as in unable to speak. Numbers do not speak. Someone has to speak for them.
No comments:
Post a Comment