Amazon has agreed to invest up to $4 billion in generative AI firm Anthropic, which has tried to carve out a niche in the market for artificial intelligence services by focusing on so-called constitutional AI, a system designed to reduce potential misinformation and harm caused by the technology.

Amazon will initially invest $1.25 billion in the company, with the option to increase that to a total of $4 billion in the future. In February, Amazon Web Services rival Google invested $300 million in the company.

Founded by former senior members of Microsoft-backed OpenAI in 2021, Anthropic is the company behind Claude, an AI assistant that the company trained on constitutional AI, in contrast to other models.

Constitutional AI is a system that uses a “set of principles to make judgments about outputs,” which helps Claude to “avoid toxic or discriminatory outputs” such as helping a human engage in illegal or unethical activities, according to a blog posted by Anthropic. The company said this has enabled it to broadly create an AI system that is "helpful, honest, and harmless.”

Anthropic has made some headway in the competitive market for AI services, recently scoring deals with search engine DickDuckGo and with videoconferencing company Zoom.

Anthropic released its latest Claude 2 model in July 2023, claiming it could score 76.5% on the multiple choice section of the US Bar law exam, up from 73.0% with Claude 1.3. Additionally, Anthropic said that Claude 2 is harder to prompt to produce offensive or dangerous output.

“The agreement is part of a broader collaboration to develop the most reliable and high-performing foundation models in the industry,” Anthropic said in a statement announcing the news. “AWS will become Anthropic’s primary cloud provider for mission critical workloads, providing our team with access to leading compute infrastructure in the form of AWS Trainium and Inferentia chips, which will be used in addition to existing solutions for model training and deployment.”

Amazon developers and engineers will be able to build on top of Anthropic’s models via Amazon Bedrock, the company said, adding that this will enable organizations to incorporate generative AI capabilities into their work and existing applications.

Amazon Bedrock, is a foundation model API service that allows small companies who lack the necessary people power to develop their own LLMs to access pre-trained models, including those built by AI21 Labs, Anthropic, and Stability AI.

Earlier this month, Amazon announced it would be launching a new generative AI tool that creates copy listings for users selling items on the company’s e-commerce platform. The new generative AI tool is powered by a large language model (LLM) that Amazon has been developing internally and contained “a couple of hundred million endpoints.”