About That New Lead Study….
Mother Jones
<!DOCTYPE html PUBLIC “-//W3C//DTD HTML 4.0 Transitional//EN” “http://www.w3.org/TR/REC-html40/loose.dtd”>
A new study was released recently about the effect of childhood lead poisoning on future academic performance. After reading it, I decided not to post about it, but since it’s getting some attention I should probably explain why. This will take a while, so be patient.
First things first: The basic idea here is uncontroversial. We’ve known for decades that childhood lead exposure reduces IQ, stunts academic development, and leads to lower test scores. But most of the original studies in this area were done a long time ago, when childhood lead levels were much higher than they are now. Blood lead levels are measured in micrograms per deciliter, and kids in the 70s and 80s frequently had levels as high as 20 or 30. Today that’s rare, so this paper focuses on something different: small changes in children who already had fairly low lead levels. For example, what would be the effect of a drop from 4 to 3?
To measure this, they rounded up records for nearly every third-grader in Rhode Island. These records included both blood lead levels in infancy and academic performance later in childhood, which is just what you need. The problem is that you can’t just compare those two things. It’s common knowledge that kids with high lead levels also tend to be poor, have less educated mothers, belong to minority groups, etc. Since all of these things are correlated with poor academic performance, you have to control for them somehow. It’s very difficult to do properly since you can never be entirely sure there isn’t something you haven’t overlooked.
So the authors looked at another variable unique to Rhode Island. Starting in 1997, Rhode Island required landlords to certify their rentals as lead-free. Kids who live in certified housing are likely to have lower lead levels, which means you can compare that to academic performance instead. Unfortunately, you run into the same problem: people who live in certified housing are unlikely to be a random subset. You have to control for different stuff, but you still have to run a lot of controls.
To address this, the authors used an instrumental variables approach. They constructed a remarkably complex variable that models “the probability that a child’s home was certified at the time of birth as a function of the number of certificates that had been issued in their census tract as of their year of birth, as well as family characteristics, and tract, year, and month of birth fixed effects.” After all that, though, they found only small effects:
The estimated effects of lead in these models are strongly statistically significant but relatively small: The column (4) estimates suggest that a one point increase in mean BLLs is estimated to reduce reading scores by .306, and math scores by .193.
So going from a lead level of 4 to 3 raises test scores by less than a third of a point on an 80-point scale. A 3-point reduction—which is fairly large these days—would raise test scores by about a point in reading and half a point in math.
But that’s not the end. There are two ways of measuring lead levels: venous (a standard blood draw) and finger pricks. Venous is more accurate, but finger pricks are more common. The venous measures show a stronger effect from lead exposure, so the authors constructed yet another instrumental variable to take this into account, and that produced a bigger estimate of lead on test scores: about half a point for reading and a third of a point for math.
But we’re not done yet. The authors then generate another instrumental variable, along with all the usual controls, and this produces an even bigger estimate: about one point for reading and 0.4 points for math. In both cases, however, the standard errors are quite large and the correlation coefficients are minuscule. In the case of math, the results are not statistically significant even at the 10 percent level.
This is the point at which I emphasize that I’m no expert in the design of studies like this. Controls are perfectly legit. Instrumental variables are perfectly legit—though you have to be careful not to get over-clever about them. Trying to correct for measurement problems is perfectly legit. And yet, when you put this all together it leaves a bad taste in my mouth. There are lots of controls. The main instrumental variable might be appropriate, but I couldn’t quite convince myself of that. It’s also a very complex instrument, which makes it hard to evaluate. The measurement stuff looks suspiciously like a post-hoc way of generating a bigger effect. It all feels very fragile. And even after all this, the statistical value of the results is weak.
I may be wrong about every aspect of this. It will take a real expert to go through the paper and make an informed judgment. In the meantime, though, I’d take it with a grain of salt. There’s no question that childhood lead exposure reduces academic performance, but for now I’d say I’m skeptical that the effect is as large at low levels as the authors suggest.
Link:
