Topline
A senior researcher who leads Anthropic’s alignment efforts said Tuesday that many in the company believed AI could wipe out humanity and warned that the company is not on track to solve the issue of aligning AI’s goals with humanity’s, despite its efforts, after another researcher quit the company, accusing it of not acting responsibly.
Anthropic’s alignment lead warned that recursive self improving AI could pose a serious threat to humans and its an issue his company has not yet solved.
Copyright 2024 The Associated Press. All rights reserved
Key Facts
Evan Hubinger, the Alignment Science Lead at Anthropic, wrote on X that he and his colleagues do “earnestly believe AI could kill all humans,” and he pegged his own estimate at more than 10% in the next decade.
Hubinger said the company was “trying its best,” but it does not yet have a plan to solve the issue of “alignment for superintelligence” and is not “clearly on track” to do so.
Hubinger’s post responded to an X thread by another Anthropic researcher, Jacob Coxon, who announced he is resigning from the company over AI safety concerns.
Coxon, who said he has worked on pretraining research at OpenAI and Anthropic, warned that the rival companies were not acting responsibly by “racing straight to self-improving superintelligence and gambling with our lives.”
The departing researcher noted that people building AI believe it could “kill us all by the end of the decade,” and this was not a “marketing stunt.”
crucial quote
In a post following up his dire warning, Hubinger cited Anthropic’s latest risk report and said the threat posed by present models is low and added: “What I am worried about is superintelligence arising from recursive self-improvement, as we have said is happening faster than we thought.”
What Do We Know About The Anthropic Researcher’s Resignation?
The Wall Street Journal first reported Coxon’s departure from Anthropic over concerns about the push to build self-improving AI systems that could threaten humanity. Coxon told the Journal that he believes the world is on track for a “lot of the most aggressive of these scenarios where by the end of next year things could be out of control already.” In his X posts, Coxon compared working at OpenAI and Anthropic, noting that staffers at the ChatGPT-maker have not “deeply internalized the civilizational stakes.” He said the stakes were “well-understood” at Anthropic, but the company was “locked in a race to get there first” as they believe “no one else will act responsibly, so they must do it themselves, despite the risk.”
Leave a comment