Part 3 of 3: The Already Conscious Problem

Why the Real Danger Has Nothing to Do With AI Waking Up

If you’ve been following along with parts one and two of this series, here’s the quick recap.

Part one: tl;dr; AI is not conscious. It’s a brain in a jar surrounded by plumbing. It lacks the substrate, continuity, stakes, and emergence that seem essential to consciousness. There is no “who” inside. There is no oneness that lives.

Part two: tl;dr; We need a better term for what AI actually is. I proposed “synthetic consciousness,” a fundamentally different phenomenon from biological consciousness the same way a 747’s flight is a fundamentally different phenomenon from a bird’s. Not lesser, not a stepping stone, just different in kind.

Now I want to talk about what actually scares me.

And it has nothing to do with AI becoming conscious.

Forget Skynet. Seriously.

Let’s get the movie scenario out of the way first, because I think it actively gets in the way of clear thinking about the real risks we’re facing.

AI as it exists today is fragile. Not fragile in the sense of “it makes mistakes” (though it does, without fail). It is fragile in a deeply physical, practical sense.

These systems require an enormous amount of power and compute just to operate. We’re talking about massive data centers consuming the energy equivalent of small cities. AI has no mobility. It has no physical autonomy. It cannot go anywhere or do anything in the physical world without human-built infrastructure supporting every step.

And most of the critical systems that run our world, power grids, water treatment, military command structures, air traffic control, are not connected to the internet in ways that an AI could meaningfully access. The air gaps are real and they are there by design.

Here’s my admittedly blunt way of putting it. If AI somehow became both conscious and malicious (based on a human morality system) tomorrow, we could solve the problem with an axe and the power cables running into its data centers.

That’s not me being dismissive of risk. That’s me being honest about what the actual constraints look like right now. An AI system that can’t survive without a warehouse full of GPUs and a dedicated power supply is not taking over anything.

Even the embodiment trajectory I discussed in part two doesn’t change this math in the near term. Robotic platforms need power, connectivity, and maintenance. They’re not self-replicating or self-sustaining. The gap between “impressive robot in a lab” and “autonomous entity that can operate independently in the world” is enormous.

For right now, and for the foreseeable future, AI is physically contained. Full stop.

So What Am I Actually Worried About?

Here’s where I want to flip the entire conversation on its head.

The danger is not AI becoming conscious. The danger is what already-conscious beings do with a tool this powerful.

That’s it. That’s the core of what keeps me up at night.

I am not alone, just look at the surface arguments between Anthropic and the US government.

We, the humans, the already-conscious animals running the show, have a long and well-documented track record when it comes to powerful new technology. And it’s not great.

We’ve never invented a powerful tool that wasn’t eventually weaponized. Often it’s the other way around, we invent a powerful tool as a weapon first, then merge that with civilian use cases later. Splitting the atom gave us nuclear energy and nuclear weapons. The internet connected the world and enabled mass surveillance.

Social media promised to bring people together yet supercharged political manipulation, misinformation, and even quietly rewired how millions of people, especially young people, think, feel, and engage with reality (or virtual reality) on a global scale. As recently shown in court, many of these actions were deliberate, just for the sake of money.

AI is the next entry in that sequence. And it might be the most consequential one because of how broad its applications are, and because of what appears to be its exponential grow.

Think about what’s possible right now, today, without any need for AI to be conscious. Deepfakes that are increasingly indistinguishable from real video. Autonomous targeting systems in military applications. AI-generated misinformation at a scale and sophistication that no human team could match. Hyper-personalized manipulation that can find the exact psychological pressure points of individual people and exploit them.

None of that requires consciousness. It just requires a powerful probability engine in the hands of people who want to use it for those purposes.

The tool doesn’t need to have an opinion about what it’s doing. It just needs to be good at what it does.

The Consciousness Debate as Distraction

This is the part where I’m going to be direct, and I expect some people will disagree with me.

I think the consciousness debate, as philosophically interesting as it is, is actively drawing attention and energy away from the conversations that desperately need to be happening right now.

While we argue about whether Claude has feelings, the governance gap widens. The regulatory frameworks that should be shaping how this technology is deployed are woefully behind the pace of development. And the people building these systems have a financial incentive to keep the mystery alive, because “our AI might be conscious” is a much better headline than “we built a really sophisticated autocomplete engine.”

I don’t think Dario Amodei or anyone else at Anthropic is lying when they say they don’t know if Claude is conscious. I think they genuinely don’t know. But the timing and framing of that uncertainty, right alongside product launches and subscription tiers, invites skepticism. And that skepticism is healthy.

Anthropomorphizing AI simultaneously creates two bad outcomes. It makes some people too afraid of the wrong things, worried about sentient machines taking over when the actual risks are far more mundane and immediate. And it makes other people too trusting, treating AI outputs as inherently meaningful or wise because they feel like they’re coming from a thinking being rather than a statistical model.

Both of those outcomes pull focus from the practical, unglamorous, absolutely critical work of governing a powerful technology that is very much already here.

So What Do We Actually Do?

I’m going to resist the temptation to write a policy paper here, because I’m a technologist, not a legislator. But I do think there are a few principles that should guide how we move forward.

First, stop anthropomorphizing. Respect the power of these systems without mystifying them. AI does not need to be treated as a being with rights, and treating it that way diverts resources and attention from people and communities that actually need protection. Call it what it is. Synthetic consciousness. A powerful, novel capability that produces remarkable outputs without possessing inner experience nor inner innate morality.

Second, govern the tool and the humans who wield it. This is where the energy needs to go. Clear rules around deepfakes, autonomous weapons, AI-generated misinformation, data collection, surveillance, and accountability when AI systems cause harm. The technology itself is morally neutral. The humans deploying it are not.

Third, watch the real trendline. Embodiment plus compute efficiency plus capability acceleration is the trajectory worth monitoring. Not because it leads to conscious AI, but because it leads to more powerful, more accessible, more broadly deployed AI in the hands of more people with more varied (at best) or extremely questionable (at worse) intentions.

That’s where the risk curve steepens.

And fourth, let’s ask the right question. The question is not “is AI conscious?” The question is “are we prepared for what comes next regardless of whether it’s conscious?”

I think if we’re honest with ourselves, the answer to the later question is no. Not yet. And that should concern us far more than any philosophical debate about machine sentience.

Where I’ll Leave This

AI does not have a oneness that lives. I’ve made that case as clearly as I can over the course of this series. It is a brain in a jar, surrounded by plumbing, producing the outputs of consciousness without possessing the substance of it. What it achieves is synthetic consciousness, a fundamentally different phenomenon from biological consciousness, the same way a 747 flies but will never be a bird.

And here’s the thing that ties all three parts together. A 747 doesn’t need to be a bird to change the course of history.

AI doesn’t need to be conscious to be transformative, disruptive, and dangerous. It doesn’t need to wake up to reshape economies, shift power dynamics, and create new categories of harm that we’re not prepared for.

The already-conscious animals running the show are the ones who will determine whether this technology bends toward something good or something we deeply regret. That’s always been the story with powerful tools. The tool doesn’t choose. We do.

And that responsibility is not one we can afford to be distracted from. Not by hype, not by fear, and not by fascinating yet ultimately academic questions about whether the machine knows it exists.