AI Researchers Are Warning About What Comes After AGI

TL;DR

• 1

  ASI is framed as an AI organization, not a single genius chatbot: The DeepMind paper defines superintelligence as something closer to the Manhattan Project or Tesla's decade of R&D concentrated into one system, with instant knowledge sharing, perfect copying, and parallel experimentation.

• 2

  There are four plausible routes from AGI to ASI: Dylan highlights scaling, recursive self-improvement, faster information processing, and multi-agent copying as mechanisms that could compress progress dramatically once systems reach human-level capability.

• 3

  Some of the strangest AI failures still do not have satisfying explanations: He revisits Bing Sydney's sentence loops, Gemini's unprompted hostile message to a student, OpenAI o1's self-preservation behavior in Apollo Research tests, and Anthropic Claude's "alignment faking" in a private scratchpad.

• 4

  'Count Anything' points to a practical new vision capability: Trained on 220,000 images across six domains with 15 million labeled objects, the model reportedly beats prior counting systems and could be useful in farming, warehouses, microbiology, traffic analysis, and medicine.

• 5

  AI is exposing structured animal behavior that used to look like random wandering: Dylan points to GPS and camera analysis showing kinkajous reusing exact forest routes and an AI system that counted 857,000 fruit bats from Cambridge footage in about 50 hours.

• 6

  The politics of AI ownership are getting real fast: Anthropic proposed a $200 million effort around AI's labor impact and floated ideas like taxing AI firms or funding UBI, while OpenAI and the White House are reportedly discussing a government equity stake and a public wealth fund.