The science of algorithms

Al Khwarizmi was one of the greatest minds of the Islamic Golden Age, keeping alive mathematical traditions dating back to the Greeks and adding many of his insights. Ironically, he is best known today as the person who gave his name to one of the hottest ideas in 21st-century business: algorithms.

An illustration of Abu Ja'far Muhammad Ibn Musa Al Khwarizmi, one of the greatest minds of the Islamic Golden Age. Fred Matamoros for The National

As the author of a celebrated business text, Abu Ja’far Muhammad Ibn Musa Al Khwarizmi enjoys an impressive reputation. He is credited with making a host of powerful techniques available to non-specialists in a clear yet precise style.

But you won’t find this Baghdad-based writer’s magnum opus in bookshops. That is because The Compendious Book on Calculation by Completion and Balancing has been out of circulation for a while now, having been written about 1,200 years ago.

Al Khwarizmi was one of the greatest minds of the Islamic Golden Age. He kept alive mathematical traditions dating back to the Greeks, and added many of his insights.

Ironically, he is best known today as the person who gave his name to one of the hottest ideas in 21st-century business: algorithms.

These mathematical recipes take their name from the Latinised version of Al Khwarizmi’s name, which, when typed into Google, pops up in a host of headlines.

In the last few days alone, attributions have been made to algorithms for everything from weeding out spoof Twitter postings to seeking out ethnically diverse job applicants. Finnish software engineer Atanas Boev even claims to have found an algorithm for the perfect spouse.

But the term crops up most often in coverage of a major controversy shaking the financial world. It centres on the use of algorithms to make fortunes from stock markets.

This month marks the fourth anniversary of the so-called Flash Crash, when the Dow Jones Industrial Average plunged almost 1,000 points – about 9 per cent – in a matter of minutes.

According to the United States Securities and Exchange Commission (SEC), the principal cause was computer algorithms reacting to a big order placed in an already jittery market.

The appearance of US$4 billion worth of contracts on an exchange set off a feeding frenzy among so-called high-frequency trading (HFT) algorithms, which mindlessly tried to buy and sell the contracts, causing the market to go haywire.

While no permanent damage was done to share prices, the same cannot be said of confidence in the stability of financial markets.

CNBC markets commentator Jim Cramer this month called attention to the strange behaviour of certain stocks, which he blamed on the action of algorithms.

Cramer’s suspicions were roused by the stock price of Yelp, an online business reviews site. The company has never turned a profit, but recently announced quarterly results that narrowly beat analysts’ expectations.

That would normally cause the stock price to improve briefly before falling back – often prompting similar stocks to fall as well.

This time, however, the market took a different view of Yelp’s prospects, and its price did not experience the initial jump.

Then something weird happened: the stock price of similar companies started to rise. Cramer thought this was the result of trading algorithms, which had been designed to expect an initial jump in Yelp’s share price, to be followed by a decline that dragged similar stocks down with it.

When this failed to happen, the algorithms became confused – and started driving up the price of the other stocks.

Cramer’s suspicions might not be well-founded, but he was certainly right about one thing: trading algorithms can do things that baffle even their human creators.

This would not be such an issue were it not for the astonishing speed at which the algorithms operate. Carrying their instructions as electronic pulses, they can affect markets in millionths of a second.

Such speed is believed to have exacerbated the Flash Crash in May 2010. Technical glitches on the New York Stock Exchange led to incorrect time-stamps on the price of stocks. In the resulting confusion, human traders pulled out, but high-frequency algorithms swooped in, trading with inaccurate prices and intensifying the market turmoil.

Such high-frequency trading algorithms are under intense scrutiny following the publication of Flash Boys by the financial writer Michael Lewis. He claims traders are using algorithms that act so fast that they can “front run” orders from others, buying shares ahead of the order, jacking up the price and ensuring a profit – to the detriment of other investors, including pension funds.

While some commentators have dismissed Lewis’s claims as scaremongering, the SEC is taking them seriously. It has already begun creating software systems to detect market manipulation.

But the regulators are also turning their attention to another aspect of algorithmic trading, which is almost spooky in its implications.

In their endless quest for profit, finance firms began trading “off exchange” in the 1990s, where they could execute buy-and-sell orders without affecting share prices on the stock market.

Known as “dark pools”, these private all-electronic marketplaces soon began to spawn their own special trading algorithms, triggering a Darwinian struggle for the survival of the fittest.

By the early 2000s, the algorithms were no longer just mindless lists of instructions such as “If the price of commodity X rises, then buy company Z”. The fitter, smarter algorithms had been equipped with artificial intelligence code, which allowed them to spot the actions of less sophisticated algorithms and learn to outwit them.

Their success in squeezing profit out of trades has led to about 10 per cent of all stock trades in the US occurring in these dark pools now.

That means trillions of dollars worth of stocks are being traded by these incredibly smart algorithms.

Regulators are understandably concerned that some of these algorithms are more than just smart. They may also be capable of actions that constitute illegal behaviour, such as insider trading.

But there is another concern. Not even the designers of the algorithms can predict what their progenies will do in every circumstance.

The methods used to create the algorithms produce code which is often beyond analysis. Worse, as the algorithms evolve they can acquire traits never intended by their creators.

Four years on from the Flash Crash, it is far from clear that we have really got to grips with the digital beasts named after a long-dead Islamic genius.

Robert Matthews is a visiting reader in science at Aston University, Birmingham