Wednesday, July 6, 2011

Meaningful Use of Meaningless data - Lesson from the Teachers in Atlanta.

People are shocked by the story that teachers who are supposed to be setting good examples for school kids were involved in cheating.  The teachers were erasing incorrect answers on students standardized tests and entering the correct answers.
From CBS evening news:
 " It was to show phony progress at often troubled schools, what the report calls "the pressure to meet targets in the data-driven environment." Educator Diane Ravitch, author of "The Death and Life of the Great American School System: How Testing and Choice Are Undermining Education," blames it on a federal law that links funding with test performance.
"We have a terrible federal law called No Child Left Behind that says that all schools have to have 100 percent of their students proficient in reading in math by the year 2014 or their schools will be shut down," Ravitch said.
From CNN
"I think the overall conclusion was that testing and results and targets being reached became more important than actual learning for children," (Gov. Nathan) Deal said. 
So is anyone surprised that this would happen?  When a single number becomes more important that the process, in high stakes situations, people will do what it takes to improve that target number rather than improve the process that the number is supposed to measure.  

Why is this an important lesson for Medicine?
1.  We have for long struggled with the USMLE scores as being a single number that is used to measure the quality of a medical student.  This is what is used as one of the most important criterion for selection to residency training programs.  This is in spite of there being no good data that scoring high on the USMLE equates to potential to be a good doctor.  Recently there was a an article in the leading journal on medical education highlighting this problem.  The USMLE has 3 steps.  Students who score well on Step 1 will postpone taking step 2 till they have finished interviewing for residency programs - just in case they get a lower score on step 2 and jeopardize their chances!  Many medical schools give students dedicated time to prepare for the USMLE.  So students who have been getting a curriculum that medical educators believe is appropriate to make them good doctors for 2 years take a couple weeks off to read for a test!  USMLE is a test of medical knowledge and it also tests whether a student can apply this to the clinical environment.  But it cannot measure a lot of other important competencies like communication skills, professionalism, ability to be lifelong learners etc.

2.  The recent mandate for Meaningful Use (HITECH Act) of EHR data has the risk of falling into the same trap.  While well-intentioned, it has several measures of very important clinical processes.  If done right, these processes have the potential of improving healthcare.  The problem is that these measures are linked to incentive payments from the Center for Medicare and Medicaid Services which make them high stake for doctors working on razor thin margins.  Doctors who fail to use EHRs meaningfully with their Medicare patients are at risk of penalties. Let us take an example:
One of the 25 measures of meaningful use is to show that doctors are doing reconciliation of medications (checking what patients are taking and where there are discrepancies, updating the medical record or educating the patient appropriately).  One popular EHR system has a system for putting a check mark against each medication as you confirm it or to delete it if the patient is not taking it.  Doctors and nurses go over this exercise with the patients and it can be extremely time consuming.  One would assume that after you have gone through the entire list of medications, you should have complied with the meaningful use requirement.  Unfortunately, after doing all this, to record in the database that you reviewed the medications, you have to click a button that marks that you reviewed the medications.  As doctors get pressured to see more patients in less time, guess what is going to happen?  Yes, they may be tempted to click that button whether or not they went through the process of reconciling the medications.  This is an example of how poorly designed software will lead to physicians targeting a number (% of office visits where reconciliation occurred) rather than do what is a critical part of a visit - making sure the patient is taking the right medication.  
The teachers in Atlanta has taught us a valuable lesson.  Even people who are expected to set examples of high moral values, when forced to meet requirements that are measured by numbers without actually getting the support they need to do their job right, will take shortcuts to make the numbers look better.  In today's data-driven society, this is a valuable lesson.


  1. This is definitely a hot topic. I can fondly recall a few sessions on this in an "economics of education" class at CWRU taught by Eric Bettinger, who is now at Stanford. Search some of his work and you'll find some thought provoking studies and interesting insights into what is wrong with American education today...

  2. Our society is driven to measure and evaluate everything we do, even when things we are trying to measure are intangible. This is a direct result of the application of scientific principles as defined since the industrial revolution. The underlying assumption is that if we keep taking things apart, breaking things down, analyzing them to their 'core', that somehow this will yield 'the core truth' of things. The problem is that this approach misses the forest sometimes.
    It is not surprising then that we think that we can somehow measure success in teaching by breaking down components and taking one or two things out of these that we can measure and use them to evaluate the whole.
    In medicine, studies are designed to measure as many variables as possible, including the intangible ones like lifestyles.
    Not to say that it is not a good idea to look closely at the trees - as long as we don't loose sight of the forest.

  3. @Brad, thanks for suggesting Dr. bettinger's name.

    @Pranav good points. I think it is possible that the person who first comes up with a variable to measure an intangible process is aware of it's limitations but as time passes, others will adopt this measure without that awareness. It is some people's job to look at dashboards of such data and decisions start getting made based on combinations of such measures. For some it might seem better to be decisive rather than appear to be paralyzed when deliberating merits of the measures.

  4. Thanks to @Anish for pointing out to me the similarity b/w the Alanta story and the case described in Freakonomics (Chicago school system).

  5. Neil, the problem is that when non-scientists look at the data that includes intangibles, they do not appreciate the complexity, and decisions get made that are at times highly regrettable. The example that comes to mind is this: over the past 20-30 years, researchers have been looking at the concept of self esteem. Without understanding the somewhat intangible concept of self-esteem, our society has chosen to interpret it as feeling good about one's self, and the result is a generation of kids who have grown up thinking they are special, that they deserve something for just showing up (see a recent Atlantic Monthly article on this, link on my fb page). Essentially, we as a society have raised our children's self esteem by lowering our standards.

  6. This comment has been removed by a blog administrator.