Random Image Display on Page Reload

OpenAI Wants AI to Help Humans Train AI

Jun 27, 2024 1:05 PM

OpenAI Wants AI to Help Humans Train AI

Having humans rate a language model’s outputs produced clever chatbots. OpenAI says adding AI to the loop could help make them even smarter and more reliable.

3D rendering of multicolored glowing speech bubbles in front of a blue background

Illustration: Andriy Onufriyenko/Getty Images

One of the key ingredients that made ChatGPT a ripsnorting success was an army of human trainers who gave the artificial intelligence model behind the bot guidance on what constitutes good and bad outputs. OpenAI now says that adding even more AI into the mix—to help assist human trainers—could help make AI helpers smarter and more reliable.

In developing ChatGPT, OpenAI pioneered the use of reinforcement learning with human feedback, or RLHF. This technique uses input from human testers to fine-tune an AI model so that its output is judged to be more coherent, less objectionable, and more accurate. The ratings the trainers give feed into an algorithm that drives the model’s behavior. The technique has proven crucial both to making chatbots more reliable and useful and preventing them from misbehaving.

“RLHF does work very well, but it has some key limitations,” says Nat McAleese, a researcher at OpenAI involved with the new work. For one thing, human feedback can be inconsistent. For another it can be difficult for even skilled humans to rate extremely complex outputs, such as sophisticated software code. The process can also optimize a model to produce output that seems convincing rather than actually being accurate.

OpenAI developed a new model by fine-tuning its most powerful offering, GPT-4, to assist human trainers tasked with assessing code. The company found that the new model, dubbed CriticGPT, could catch bugs that humans missed, and that human judges found its critiques of code to be better 63 percent of the time. OpenAI will look at extending the approach to areas beyond code in the future.

“We’re starting work to integrate this technique into our RLHF chat stack,” McAleese says. He notes that the approach is imperfect, since CriticGPT can also make mistakes by hallucinating, but he adds that the technique could help make OpenAI’s models as well as tools like ChatGPT more accurate by reducing errors in human training. He adds that it might also prove crucial in helping AI models become much smarter, because it may allow humans to help train an AI that exceeds their own abilities. “And as models continue to get better and better, we suspect that people will need more help,” McAleese says.

The new technique is one of many now being developed to improve large language models and squeeze more abilities out of them. It is also part of an effort to ensure that AI behaves in acceptable ways even as it becomes more capable.

Earlier this month, Anthropic, a rival to OpenAI founded by ex-OpenAI employees, announced a more capable version of its own chatbot, called Claude, thanks to improvements in the model’s training regimen and the data it is fed. Anthropic and OpenAI have both also recently touted new ways of inspecting AI models to understand how they arrive at their output in order to better prevent unwanted behavior such as deception.

The new technique might help OpenAI train increasingly powerful AI models while ensuring their output is more trustworthy and aligned with human values, especially if the company successfully deploys it in more areas than code. OpenAI has said that it is training its next major AI model, and the company is evidently keen to show that it is serious about ensuring that it behaves. This follows the dissolvement of a prominent team dedicated to assessing the long-term risks posed by AI. The team was co-led by Ilya Sutskever, a cofounder of the company and former board member who briefly pushed CEO Sam Altman out of the company before recanting and helping him regain control. Several members of that team have since criticized the company for moving riskily as it rushes to develop and commercialize powerful AI algorithms.

Dylan Hadfield-Menell, a professor at MIT who researches ways to align AI, says the idea of having AI models help train more powerful ones has been kicking around for a while. “This is a pretty natural development,” he says.

Hadfield-Menell notes that the researchers who originally developed techniques used for RLHF discussed related ideas several years ago. He says it remains to be seen how generally applicable and powerful it is. “It might lead to big jumps in individual capabilities, and it might be a stepping stone towards sort of more effective feedback in the long run,” he says.

Will Knight is a senior writer for WIRED, covering artificial intelligence. He writes the Fast Forward newsletter that explores how advances in AI and other emerging technology are set to change our lives—sign up here. He was previously a senior editor at MIT Technology Review, where he wrote about fundamental… Read more
Senior Writer

Read More

How Game Theory Can Make AI More Reliable

Researchers are drawing on ideas from game theory to improve large language models and make them more correct, efficient, and consistent.
Steve Nadis

AI Is Coming for Big Tech Jobs—but Not in the Way You Think

Companies aren’t replacing workers with AI yet. But they are sacrificing thousands of jobs in the race to further innovation in the technology.
Amanda Hoover

Airbnb’s Olympics Push Could Help It Win Over Paris

Paris officials have placed tough new restrictions on Airbnb rentals in recent years. The company is using the Olympics to try and win over locals and broaden its footprint in the iconic city.
Amanda Hoover

The $50 Billion Musk Referendum

Tesla shareholders will decide whether to back Elon Musk’s leadership—and unprecedented 11-figure bonus—in a pivotal moment for the carmaker.
Morgan Meaker

LinkedIn’s AI Career Coaches Will See You Now

LinkedIn’s new generative AI features include chatbots based on popular career coaches and tools to rewrite résumés and cover letters.
Amanda Hoover

The EU Is Taking on Big Tech. It May Be Outmatched

From the Digital Services Act to the AI Act, in five years Europe has created a lot of rules for the digital world. Implementing them, however, isn’t always easy.
Luca Zorloni

Danish Media Threatens to Sue OpenAI

Danish media outlets want a group deal with OpenAI amid fears that smaller countries will become marginalized as chatbots become prolific. If they don’t get one, they are threatening to sue.
Morgan Meaker

As Google Targets AI Search Ads, It Could Learn a Lot From Bing

Microsoft and Google are bringing ads to their AI search experiences. But users don’t always find it helpful.
Paresh Dave

*****
Credit belongs to : www.wired.com

Check Also

Break the monotony with the Olive Green Redmi Note 13 Pro 5G

It is always exciting to see fresh takes on smartphone design, a welcome change from …