Is Anthropic's new chatbot Claude 3 a worthy rival to OpenAI's ChatGPT?

It's 'most intelligent' model Opus outperforms OpenAI's best large language models

Anthropic's new Claude 3 family – Haiku, Sonnet and Opus – claim to offer a 'new standard for intelligence'. Getty Images

Anthropic, the US artificial intelligence start-up backed by Amazon and Google, has introduced its more recent chatbot directly aimed at challenging generative AI leader OpenAI.

The chatbot, Claude 3, offers a “new standard for intelligence” and, according to the San Francisco company, outperforms Google's Gemini and OpenAI's ChatGPT.

Claude 3 “sets new industry benchmarks across a wide range of cognitive tasks … [its] models are better at following complex, multi-step instructions”, Anthropic said.

“They are particularly adept at adhering to brand voice and response guidelines, and developing customer-facing experiences our users can trust.”

What does Claude 3 offer?

Claude 3 comprises a family of three large language models (LLMs), the underlying algorithm that uses deep learning and analyses significant amounts of data to generate content.

The LLMs include Haiku, Sonnet and Opus, each offering “increasingly powerful performance, allowing users to select the optimal balance of intelligence, speed and cost for their specific application”.

Anthropic has named Claude 3's LLMs after artistic works – a haiku is a three-line poem, a sonnet has 14 lines, while an opus is a composition – each offering increased capabilities relative to their definitions.

Haiku can summarise “thousands” of documents into structured data, Sonnet helps in conversation and translating language, while Opus, which “achieves near-human comprehension capabilities”, can act as an economic analyst, the company said.

One example Anthropic gave for Opus is looking up US gross domestic product trends and listing them in a table.

How does it compare to GPT-4 and Gemini?

According to Anthropic's benchmarking statistics – comparisons to LLMs currently available commercially that have released evals (an industry term for evaluating LLMs) – Claude 3 outperforms both Gemini and ChatGPT.

For example, in primary school maths, Opus has 95 per cent, compared to 92 per cent of OpenAI's GPT-4 and Gemini 1.0 Pro's 94.4 per cent.

In reasoning over text, those figures are 83.1 per cent, 80.9 per cent and 82.4 per cent, while for common knowledge, they are 95.4 per cent, 95.3 per cent and 87.8 per cent.

Anthropic Claude 3 benchmarks

Perhaps the most telling statistic is that Claude 3 can summarise up to 150,000 words – compared to ChatGPT's 3,000.

While some results have Claude 3 winning by a hairline, the consensus is that Opus outperforms ChatGPT and Gemini in every metric.

On the lower end, Sonnet and Haiku also largely outperform GPT-3.5 and Gemini 1.0 Pro.

Compared to its predecessors, Sonnet is two times faster than Claude 2 and Claude 2.1. Opus can deliver speeds similar to Claude 2 and 2.1, but “with much higher levels of intelligence”.

Where is Claude 3 available and in what languages?

Sonnet and Opus are now available in 159 countries, including the UAE. Haiku will be available soon, the company said.

Claude 3 is in English and can converse in other languages including Spanish, Japanese and French. Anthropic did not provide a full list of languages.

How much does Claude 3 cost?

Haiku is the most affordable option at $0.25 per million tokens (MTok) for input and $1.25 for output, while Sonnet is at $3 and $15. Opus is, unsurprisingly, the most expensive at $15 and $75.

For comparison, OpenAI’s GPT-4 Turbo comes in at $10 for input and $30 for output, and also with a smaller context window of 128,000 MTok.

Tokens in AI are the basic unit to calculate the length of text and can include punctuation marks and spaces. Token counts vary from one language to another.

Opus and Sonnet are being pitched for businesses, “designed to empower enterprises to automate tasks, generate revenue through user-facing applications, conduct complex financial forecasts, and expedite research and development across various sectors”, according to Claude 3's model card.