New theories to predict freak events

By concentrating on the unusual rather than the average, a probability analysis has provided insights into finance and the weather.

Extreme Value Theory is now being used to predict the probability of extreme weather conditions, such as this tornado in Kansas last year.

The one word most often used to sum up 2011 is "extreme". Hardly anyone escaped the effects of last year's turmoil, with its political and financial upheavals. Even the elements joined in: the US usually gets three or four weather disasters annually that cost a billion dollars each; last year it was hit by 12.

Time was when we thought of such extremes as rarities, events seen perhaps once in a lifetime. Now the extreme looks set to become the routine.

But the shocking prevalence of freak events comes as less of a surprise to those familiar with a technique now being increasingly used to make sense of them: Extreme Value Theory (EVT).

Put simply, EVT is a branch of probability that gives insight into the seemingly unknowable: the chances of events so extreme they may never have been seen.

When mathematicians began developing its foundations in the 1920s, their starting point was the idea that extreme events of any kind - the highest daytime temperature, say, or the worst market crash - follow a so-called probability distribution, which shows how the chances of such an event vary with its size.

Most of us are familiar with at least one distribution: the so-called "bell-curve", more properly called the normal distribution.

It is familiar because it is ubiquitous. From the outcome of tossing coins to the heights of children in a classroom, the hump-shaped normal distribution reflects both the most likely outcome - shown by the position of its peak - and the chances of getting more unusual outcomes.

Just how likely these are depends on how the bell curve spreads out to either side of the peak, a feature captured by a number dubbed "sigma". Once the value of one sigma is known, the normal distribution spits out the corresponding probabilities.

Any quantity following a normal distribution has a 32 per cent chance of lying more than one sigma away from the central peak, a 5 per cent chance of getting one more than two sigmas away, and only a 1 in 2 million chance of an outcome over five sigmas away.

Small wonder, then, that David Viniar, chief financial officer of Goldman Sachs, admitted his perplexity in August 2007 after announcing that the loss of 27 per cent of the value of one of the firm's flagship funds represented a 25-sigma event. Such an event would not normally be expected to occur even once during the entire history of the universe.

What that means, of course, is that the losses probably weren't following a normal distribution. This was a possibility hinted at more than a decade earlier by the then-chairman of the US Federal Reserve, Alan Greenspan.

In November 1995, after the two rogue trader meltdowns, Mr Greenspan hinted there may be merit in adopting "the statistical distribution of extreme events".

Few heeded the call: three years later the financial sector witnessed the US$4.6bn collapse of Long Term Capital Management, a hedge fund that fell foul of extreme events.

But now such ostrich-like attitudes are no longer acceptable - due in no small measure to the success of the risk management guru Nassim Nicholas Taleb's best-seller The Black Swan - his term for extreme events. And the events of recent years have revealed the perils of expecting the normal distribution to be reliable when gauging the chances of extreme events.

The core of the problem is revealed by EVT. In essence the graceful bell curve is just too graceful to handle extreme events, which can be shown to follow a different distribution. This is what EVT provides, and it shows that the bell curve's "tails" radically underestimate the chances of extreme events. To get a better estimate, EVT requires historical data on the past extremes - for example, the highest temperatures in Abu Dhabi for the past 50 years. These can then be used to fit a distribution to what is known - and extrapolated to gauge the size of what could still await us. The recent calamities have triggered a surge of work applying EVT methods to a host of issues, especially in finance and climate.

For example, Marco Rocco of the Bank of Italy, among others, has shown how EVT can boost the reliability of so-called Value at Risk calculations by financial institutions, defined as the biggest loss that could occur over a fixed time-period (typically 10 trading days) with a probability of, say, 1 in 100. If the calculations assume normal distributions, they are all too likely to leave investment banks feeling too confident about the future.

Meanwhile, the Bank of Canada analysts Toni Gravelle and Fuchun Li have put EVT at the heart of their method for gauging the contribution of individual banks to the overall health of the financial system.

Earlier this month, researchers in Switzerland used EVT to make sense of the lethal heatwave that struck Europe in 2003, leaving more than 40,000 dead.

They found that such an event was indeed extraordinary, happening just once in 2,000 years. But by factoring in the effects of global warming, they showed that by 2050 the chances of a repeat of the calamity may rise six-fold.

Not everyone is convinced of the power of EVT; ironically, sceptics include the author of The Black Swan. But anyone looking for a demonstration of confidence in the theory should visit the Netherlands. In February 1953, a huge storm surge broke through the nation's sea dykes, killing 1,800 people and destroying tens of thousands of homes. Within weeks, the Dutch government set up an expert committee, instructing it to design new sea defences that could be relied on to beat the worst nature could throw at them for the next 10,000 years.

After analysing centuries of data, engineers used EVT to calculate that a system of five-metre dykes would suffice. Today, 16 million people are living under the protection offered by this extraordinary mathematical technique.

Robert Matthews is a visiting reader in science at Aston University, Birmingham, England