June 2024 Newsletter

MIRI updates MIRI Communications Manager Gretta Duleba explains MIRI’s current communications strategy. We hope to clearly communicate to policymakers and the general public why there’s an urgent need to shut down frontier AI development, and make the case for installing an “off-switch”. This will not be easy, and there is a lot of work to... Read more » The post June 2024 Newsletter appeared first on Machine Intelligence Research Institute.

Jan 26, 2025 - 14:49

MIRI updates

MIRI Communications Manager Gretta Duleba explains MIRI’s current communications strategy. We hope to clearly communicate to policymakers and the general public why there’s an urgent need to shut down frontier AI development, and make the case for installing an “off-switch”. This will not be easy, and there is a lot of work to be done. Some projects we’re currently exploring include a new website, a book, and an online reference resource.
Rob Bensinger argues, contra Leopold Aschenbrenner, that the US government should not race to develop artificial superintelligence. “If anyone builds it, everyone dies.” Instead, Rob outlines a proposal for the US to spearhead an international alliance to halt progress toward the technology.
At the end of June, the Agent Foundations team, including Scott Garrabrant and others, will be parting ways with MIRI to continue their work as independent researchers. The team was originally set up and “sponsored” by Nate Soares and Eliezer Yudkowsky. However, as AI capabilities have progressed rapidly in recent years, Nate and Eliezer have become increasingly pessimistic about this type of work yielding significant results within the relevant timeframes. Consequently, they have shifted their focus to other priorities.

Senior MIRI leadership explored various alternatives, including reorienting the Agent Foundations team’s focus and transitioning them to an independent group under MIRI fiscal sponsorship with restricted funding, similar to AI Impacts. Ultimately, however, we decided that parting ways made the most sense.

The Agent Foundations team has produced some stellar work over the years, and made a true attempt to tackle one of the most crucial challenges humanity faces today. We are deeply grateful for their many years of service and collaboration at MIRI, and we wish them the very best in their future endeavors.
The Technical Governance Team responded to NIST’s request for comments on draft documents related to the AI Risk Management Framework. The team also sent comments in response to the “Framework for MItigating AI Risks” put forward by U.S. Senators Mitt Romney (R-UT), Jack Reed (D-RI), Jerry Moran (R-KS), and Angus King (I-ME).
Brittany Ferrero has joined MIRI’s operations team. Previously, she worked on projects such as the Embassy Network and Open Lunar Foundation. We’re excited to have her help to execute on our mission.

News and links

AI alignment researcher Paul Christiano was appointed as head of AI safety at the US AI Safety Institute. Last fall, Christiano published some of his thoughts about AI regulation as well as responsible scaling policies.
The Superalignment team at OpenAI has been disbanded following the departure of its co-leaders Ilya Sutskever and Jan Leike. The team was launched last year to try to solve the AI alignment problem in four years. However, Leike says that the team struggled to get the compute it needed and that “safety culture and processes have taken a backseat to shiny products” at OpenAI. This seems extremely concerning from the perspective of evaluating OpenAI’s seriousness when it comes to safety and robustness work, particularly given that a similar OpenAI exodus occurred in 2020 in the wake of concerns about OpenAI’s commitment to solving the alignment problem.
Vox’s Kelsey Piper reports that employees who left OpenAI were subject to an extremely restrictive NDA indefinitely preventing them from criticizing the company (or admitting that they were under an NDA), under threat of losing their vested equity in the company. OpenAI executives have since contacted former employees to say that they will not enforce the NDAs. Rob Bensinger comments on these developments here, strongly criticizing OpenAI for this policy.
Korea and the UK co-hosted the AI Seoul Summit, a virtual mini-summit following up on the first AI Safety Summit (which took place in the UK last November). At the Seoul summit, 16 AI companies committed to create and publish safety frameworks, including “thresholds at which severe risks posed by a model or system, unless adequately mitigated, would be deemed intolerable.”
California State Senator Scott Wiener’s SB 1047 passed in the California State Senate and is now being considered in the California State Assembly. The bill requires pre-deployment testing and post-deployment monitoring for models trained with 10^26 FLOP and $100M.