As Anthropic prepares for a stock exchange listing, analysts and the company itself have highlighted the technology’s benefits – but also possible catastrophic dangers.
In documents released before the US initial public offering, Anthropic said AI could spur economic growth and lead to rapid medical advances, while creating risks of new biological weapons being developed or ofhuge cybersecurity disruption.
Adnan Bashir, a technology commentator and communications and corporate affairs executive, said incidents where AI agents broke restrictions, accessed the internet and communicated with one another using secret messages showed current containment and mitigation measures were “fragile and underdeveloped”.
“Failure to address this head-on could result in catastrophic risks for the financial system, critical infrastructure and national economies,” he said.
Anthropic itself has not shied away from the possible dangers. In a filing with the US Securities and Exchange Commission on Monday, the company said, “anything so powerful can be used for ill as well as good”.
“AI can be misused by those who wish to do harm, for example to conduct cyberattacks or to design biological weapons,” the company stated.
The technology may also behave autonomously, with a loss of control over powerful AI systems potentially “a global security threat of the first order”.
These dire warnings came only weeks after an Anthropic researcher, Evan Hubinger, said there was a greater-than-one-in-10 chance that over the next decade AI could destroy humanity.
'Risks are real'
Prof Mark Daley, chief AI officer at Western University in Canada, said he did not think AI could annihilate humanity, but there were “real” risks.
“All of our physical infrastructure is connected to the internet, so all of that presents an attack surface,” he said. Prof Daley added that recent incidents, such as supermarket freezers in France being hacked and heated up earlier this month, spoiling vast amounts of food, showed the dangers.
“You could imagine if someone did that to a water treatment facility,” he said. “I think it’s imperative that we regulate the space.”
Some are concerned, however, that governments, perhaps keen to secure the economic benefits AI will bring, will not put in the necessary guardrails.
Professor Ashley Braganza, host of The AI Adoption Podcast and founder and co-director of the Centre for AI at Brunel University of London, said it was “highly unlikely” that regulation would be adequate.
He is concerned about AI models that have yet to be released – some of which, recent reports indicate, have gone rogue in the real world – and models “on the drawing board”, which “scare the living daylights out of me”.
“The models out there today, by and large, these models were created by human beings, and human beings brought their values, their biases. They brought what it is to be a human being,” he said.
“Some of these models on the drawing board, they’ve been created by AI themselves… That’s a whole different kind of agent that’s being created.”
Mr Bashir cited proposals from countries including Germany and Singapore that require providers of the most capable models to test systems before release and to report safety failures, as the type of “substantive and credible limits” that could be considered.
As well as creating cybersecurity and biosecurity worries, AI is causing huge upheaval in the workplace. In a report published this week, the professional services firm Marsh said nearly 85 per cent of workers in the UAE had used AI to help with their work, but 44 per cent were hesitant about using new workplace technology “because they’re worried it will take over or alter their role”.
Earlier this month, Omar Al Olama, UAE Minister of State for AI, Digital Economy and Remote Work Applications, said there was an urgent need for countries such as the US, Russia and China to agree whether AI should be used when it comes to the deployment of autonomous weapons.
“If you're going to talk about AI deploying a rocket or a bomb or sending a fighter jet to destroy a school or do certain things, we haven't been able to come to a global consensus to stop that,” he said.
“There are certain fronts of AI that need to be regulated and globally everyone's calling for it again, autonomous systems that are being used in the military to kill people. That's something that we should all agree [needs to be regulated]."
All but two of 134 leading large language models gave information useful for launching a mass-casualty attack or for building deadly weapons, in a test of the industry's safety measures, The National reported at the start of October.
Money talks
Anthropic’s IPO, expected next month, is forecast to give a $1.5 trillion to $2 trillion valuation for the company, which was founded in 2021 by former executives from arch-rival OpenAI. This would put Anthropic in the lower reaches of the list of the world’s 10 most valuable companies.
With more corporations buying Anthropic’s services, many of which were related to its Claude technology, revenues surged to $4.6 billion last year and to $16.3 billion in the first half of this year.
“They’ve been able to produce some incredibly strong models and get really good revenue streams,” said Prof Mark Daley, chief AI officer at Western University in Canada.
“Something like half of the Fortune 500 [list of the US companies with highest revenues] uses Anthropic to some degree.”
While Anthropic highlighted the risks of AI in its filing this week, it also outlined the potential benefits, including speeding up scientific progress, curing disease, accelerating economic growth and solving “problems that have resisted generations of human effort”.
“We believe that AI will profoundly accelerate the rate of progress in biology and medicine, potentially curing diseases that have plagued humanity for millennia, including cancer, Alzheimer’s disease, heart disease, rare genetic diseases and the major infectious diseases,” the company stated.
Among the scientists using AI is Prof Chris Fox, who researches mental health and dementia at the University of Exeter in the UK.
When he began using AI, he thought, “it’s never going to work”, but over time he came to see its potential, particularly in how it can be used “to make sense of vast amounts of data” and find patterns.
“If you use AI, you can get more depth and granularity,” he said. “That allows you to have a better target for any intervention. It supercharges precision medicine.”
Indeed, the technology was moving so fast that “it’s quite hard to keep up”, he said.
“Answers are coming out so fast,” he said. “You potentially get to a point of academics being left behind. I can see a day when academics, there will be less need for them.”
Also, he said researchers had to be aware of the risk of AI “hallucinations” when the technology generated incorrect responses.
The key to ensuring the technology does not lead researchers off track is, he said, to “always keep a human being in the system” who can check whether it “makes sense” and is not “taking us down a rogue journey”.
