How do we form our beliefs about the solutions to societies most complex problems? Do we trust data? Theory? Both? What does it mean to base policy on science and evidence?
According to Manski:
"Social scientists and policymakers alike seem driven to draw sharp conclusions, even when these can be generated only by imposing much stronger assumptions than can be defended. We need to develop a greater tolerance for ambiguity. We must face up to the fact that we cannot answer all of the questions that we ask."
I think Russ Roberts puts it well in his EconTalk Episode with Noah Smith:
"Can you think of a study that was so decisively performed in terms of the crossing of t's and dotting of i's that the identification and all the econometric challenges were met with such impressiveness that people on the other side of the debate had to throw up their hands and say, 'Well, I guess I was wrong. I've got to change my view.' Because I can't think of one. I can't think of one. And if that's true, then I would suggest that economics has some serious problems in claiming it's a science."
When it comes to evidence there are lots of challenges. For the most part, in economics and the social sciences it's often impossible to implement randomized controlled trials to identify treatment effects related to policy changes. For the most part we have to leverage observational data using quasi-experimental designs. The challenge for both approaches as Jim Manzi discusses in his book ''Uncontrolled: The Surprising Payoff of Trial-and-Error for Business, Politics, and Society' is the issue of 'high causal density.'
In an environment of high causal density "the number of causes of variation in outcomes is enormous, and each has significant potential effects compared with those of the potential cause of interest. We don't know enough to list each of them and hold them constant, but if we randomly assign patients to the test and control groups, then these hidden conditionals won't confound our estimate of treatment causality."
Unfortunately in the social sciences, causal pathways are extremely complex. There are always hidden conditionals we may not be able to measure or don't have sufficient knowledge to even consider. Given that hidden conditionals are always present, a well entrenched proponent of a given policy can always find a reason to explain why it has failed to prove itself out in the face of evidence.
But Jim does more than offer criticisms of theory and methods. He introduces the concept of 'Liberty as Means.' Embracing the concepts of evolutionary economics, he promotes a flexible system of government that sounds a lot like federalism. As he discusses, the mistake we often see from both the right and the left is enforcement of social norms at the national level vs. fostering numerous experiments at the local level.
While economic theory and applied econometrics are useful and powerful tools for policy analysis, these tools will not necessarily help provide clear cut always defensible evidence to improve public policy. These methods will never discover a 'Polio vaccine' for policy. It is in fact their shortcomings that provide the strongest argument for our constitutional republic and federalism that our founders envisioned.
Reference: Uncontrolled: The Surprising Payoff of Trial-and-Error for Business, Politics, and Society, by Jim Manzi https://www.amazon.com/Uncontrolled-Surprising-Trial---Error-Business/dp/046502324X/
See also: EconTalk: Manzi on Knowledge, Policy, and Uncontrolled
*this is cross posted at: https://ageconomist.blogspot.com/2019/01/economics-evidence-and-high-causal.html?m=0
