Two Problems, One Worry: What Experts Actually Mean by AI Existential Risk

Expert warnings about AI existential risk stem from two distinct technical problems: the control problem of managing self-improvement speed and the alignment problem of specifying human values. Understanding this distinction is key to evaluating AI safety concerns.

Rui Almeida

July 2, 2026 · Updated October 31, 2026 · 5 min read

When physicists and AI researchers warn that advanced artificial intelligence could threaten humanity, the warning often sounds like science fiction—vague, distant, and easy to dismiss. But the concern rests on two distinct technical problems, not a single cloud of anxiety, and confusing them is the surest way to misunderstand the debate. The first is the control problem: what happens if a machine intelligence can improve itself faster than we can supervise it? The second is the alignment problem: even a controlled system is dangerous if we cannot specify what we actually want it to do. Distinguishing these two is the first step toward evaluating expert warnings on their own terms.

Understanding AI Existential Risk: Beyond Abstract Fears

The contemporary worry about existential risk from AI did not originate with fringe commentators. In 2014, physicist Stephen Hawking joined leading AI researchers Max Tegmark and Stuart Russell in warning that superintelligent AI systems could end up "outsmarting financial markets, out-inventing human researchers, out-manipulating human leaders, and developing weapons we cannot even understand," as Brookings recounts. Their framing contains the essential distinction that still organizes the field: in their words, "the short-term impact of AI depends on who controls it, the long-term impact depends on whether it can be controlled at all."

Yet the political trajectory has moved in the opposite direction from this concern. According to the same Brookings analysis, policymakers tend to dismiss such warnings as overblown and speculative, and despite a focus on AI safety at international conferences in 2023 and 2024, the 2025 AI Action Summit in Paris shifted away from existential risks entirely. That divergence—a research community articulating specific fears while policy attention drifts toward nearer-term harms—is precisely why the technical substance of those fears deserves careful statement rather than summary dismissal.

The Control Problem: Managing Recursive Self-Improvement

The control problem is fundamentally a problem of speed. As Brookings explains, the pivotal step would be instructing generally intelligent models to improve themselves; once so instructed, AI systems would leverage superior learning capabilities to improve far faster than human engineers could follow, "far surpassing human capacities through a process of recursive self-improvement." A system that rewrites itself on a timescale shorter than our ability to audit each revision is, in a meaningful sense, no longer governed by its original designers.

It is important to keep the status of this scenario clear: recursive self-improvement producing superintelligence is a hypothetical pathway, not an observed event, and the Brookings discussion treats it as such. But the hypothesis has gained practical traction. Brookings notes that the much-discussed AI 2027 forecast "relies crucially on this idea of recursive self-improvement," resting its premise on AI agents that become "good at many things but great at helping with AI research"—that is, on machines whose main talent is accelerating the design of their successors. The concern, then, is not malice but escalation: each round of improvement shortens the interval available for human oversight of the next.

Deconstructing AI Existential Risk

The framework below separates the concept into its two technical drivers and the philosophical layer beneath them, so that the reader can locate each claim in the debate.

Distinguishing the two core components of AI existential risk
Component	Core Challenge	Nature of the Risk
Control problem	Recursive self-improvement: AI models instructed to improve themselves could far surpass human capacities faster than humans can audit or supervise.	Hypothetical loss of human control over a superintelligent system; Hawking, Tegmark and Russell framed long-term impact as hinging on "whether it can be controlled at all."
Alignment problem	Goal specification: ensuring AI objectives match human values and intentions, not merely the literal goals we can write down.	A controlled system pursuing misaligned goals with destructive consequences—linked to the "illusion of AI autonomy."
Human condition layer	AI's consequences reach beyond physical safety into identity, purpose and meaning.	Even absent catastrophe, machines that match or supersede human capabilities challenge humanity's self-understanding.

The analytical point of this breakdown is that "existential risk" is not one fear but two separable failure modes plus a philosophical aftermath. The control problem is a question of capability and oversight; the alignment problem is a question of specification and meaning. A reader who can say which one a given warning refers to has already escaped the vagueness that lets policymakers wave the topic aside.