"...most... attempts to direct social change are doomed to failure, or actually create more problems than benefits."and
"...without directed change -- regardless of the odds against it -- we are doomed to careen through history from one crisis to another; and, since the crises are getting bigger due to the sheer power of humans (both individually and collectively), this is an unacceptable option."I firmly agree with both those statements. I don't believe they are mutually exclusive in any sort of way.
Howie and I spent tremendous effort researching different trading techniques (and of course Howie is still doing it). The vast majority of the experiments were failures. Nonetheless, Howie's trading is more profitable today because of those efforts. Granted, we had (and have) the luxury of trying new things without actually committing money ahead of time. However, even if we had to commit money before testing, if we kept the experiments small, it would have still been important to try them. In fact, we wouldn't have survived without doing the research (in my opinion, Howie's opinion may differ).
Most scientific experiments fail as well. That doesn't mean we shouldn't pursue science. Indeed, we often learn from the failures as well as the successes. The failures are rarely publicized. Everybody knows Edison invented the light bulb, but few know that he had to try thousands of filaments before he found one that worked for even 24 hours.
And the same goes for government and social change. Certainly many or most of the things we try will fail. But without incremental experimentation, I agree that we will careen from crisis to crisis. In fact, I'd go farther than Jim. If most attempts at social change are not failing, then we're not making enough attempts.
But (and you knew there was going to be a "but", didn't you?), I think the experiments (i.e., attempts at social change) need to be well designed.
They have to be designed with the possibility of failure (since most experiments will fail). This includes minimizing the cost of failure and having an escape hatch enabling the experiment to be undone if necessary. The easiest way to minimize cost is to not do the experiment at the national level until it's been shown to work well at local and state levels. This also provides an escape hatch. If an experiment turns out to be catastrophic for some locality, the rest of the country can always bail out those affected. If a policy fails catastrophically at the national level, well, then like communism and Russia, it may take a long, long time to recover.
The results of the experiments should be quantifiable. In other words, can we measure the impact of the experiments? For example, Clinton's efforts to get welfare recipients back to work seems to have helped. There are now significantly fewer people on the dole. Would have this happened anyway? Maybe or maybe not. Perhaps an even better experiment would have been to do it in half the states. The other half would be a control group. At least the welfare experiment didn't seem to hurt.
Lots of small experiments are better than a few large experiments. First, it provides more data. Second, it enables experimental design to take into account local factors and needs. For example, a policy that would work well in Berkeley may not work so well in San Diego, may work poorly in Coral Springs, and may be a total, complete, and irreversible disaster in Mississippi.
A recurring theme in the experimental design is to devolve money, responsibility, and authority for the experiments to as local a level as possible. At the national level the cost of failure is too high, it makes it difficult to quantify the results of the experiment (since there is nothing to compare it to), and it limits the number of experiments which reduces the data and what we can learn from those experiments.
The next question is who should direct these experiments? The president of the United States? Congress? Ahnold Schwarzenegger? Mayor Willie Brown? Some combination of the above?
I think this may be where the biggest disagreement lies amongst members of this blog. I think it is extremely unlikely that any politician (current or future) in any political office (current or future) can provide the most effective and efficient leadership to promote social change. Power corrupts, and giving our politicians the power to effect significant change will corrupt them more deeply. As described by Mancur Olson ("The Rise and Decline of Nations"), Jonathon Rauch ("Demosclerosis"), Charlotte Twight ("Dependent on DC"), and even David Corn ( "The Lies of George W. Bush, Mastering the Politics of Deception"), politicians inherently deceive us for their own advantage. In other words, the change they will effect will benefit them, their families, and their cronies far more than the population. This is due to the relationship between special interests and politicians, the ability of politicians to increase transaction costs that hinder the public from limiting the politicians' ability to further his or her personal agendas, and a huge number of other factors that are described in those books and numerous other places. As a result, I would like politicians to get out of the leadership business and instead be civil servants.
So, you may be wondering what's the alternative. We don't need politicians to be leaders because we have many other leaders in our society. Paul Krugman is a leader. So is Christopher Hitchens, Andrew Sullivan, Bill Gates, Victor Davis Hanson, Noam Chomsky, Michael Moore, and so forth. There are lots and lots of leaders who are not politicians. They're smart, they debate the issues, and we listen. Because of the Internet and blogs and email and online media we provide feedback for their debate. Eventually, for a given problem, we'll agree on a possible solution and on an approach to implementing that solution.
Then we should tell the politicians what to do. And they should implement it. I have a set of essays that I'm working on that will describe the details of how this will (or at least could) happen. Because of enormously better communications capabilities, we have a huge number of options available to us regarding democracy and governance that simply weren't possible even a few short years ago. Direct democracy in a society of 300 million people is now possible, and as I will argue in future essays, desirable.