Daniel Kaufmann had some fun compiling a list of the authoritative commentators that predicted that – unilke in Tunisia – in Egypt the disgruntled population would not take to the street, or anyway not in such a way to threaten the regime. Everybody seems to have fallen for it, from Foreign Policy to the BBC, from Time magazine to the Economist.

Forecasting was always tricky business, and is getting more so. In a society as complex as ours, even the best analyst are lousy at prediction. In an entirely different context, David Lane and others (yours truly included) are suggesting that in some cases prediction might be replaced by a system of early warning, that spots emergent social dynamics in its early stages, when correction is still possible. This would be done by combining and filtering large masses of data, many of which collected on the web. The idea — which might ring familiar to those who use the Internet as a social filtering device for information, is that the global conversation is an entity that exists at a level superior to ours, and as such might know things that none of us, mere participants, know.

To describe this hypothetical system, David likes to quote post-marketing surveillance on pharmaceuticals after the Thalidomide scandal. This drug used to be prescribed to pregnant women in the 1950s, and it could induce terrible deformities in newborn children, but only combining in exquisitely nonlinear ways with other agents: it was cleared for rollout because the lab tests did not allow to discover the problem. It was the doctors treating the mothers of deformed or sick children that discovered, in the ocean of statistical noise, the weak signal of taking Thalidomide during pregnancy. As a consequence of this story, pharmaceutical companies now work with physicians to spot correlations to weak to be spotted in the lab, but that might be revealed by processing the mass of data obtaiend by tapping all doctors.

It is a fascinating topic, at least for me. And — going back to Egypt — it leads to an unexpected conclusion: it suggests another way that Wikileaks might be a good thing. Laks feed the global conversation, and thereby increase the probability that bloggers, citizens and activists poolf their knowledge and discover emergent trends. It has been argued that Wikileaks is nefarious, because it might hinder the work of diplomacy: but without better analysis, diplomacy cannot do an acceptable job anyway.