Curious Companion: Will AI Kill Us All?

Is AI actually going to kill us all?There’s been a ton of talk about this topic in the media lately, and so I’m dedicating an episode to sharing my thoughts. Spoiler alert, no, I don’t think that AI is going to kill us, but that doesn’t mean that this technology has no risk. I walk through the Jacob Coxon resignation thread and the fact that Anthropic’s own alignment team admits there’s genuine risk, then get into why the doom talk feels more irresponsible than credible. I don’t think regulation is coming to save us anytime soon, but to me, the answer isn’t fear, it’s curiosity.

LISTEN TO THE EPISODE

I Have Thoughts

Today we’re gonna keep it nice and light and chat about whether or not AI is going to kill us all within the next ten years.

Perhaps you’re like, what in the actual fuck are you talking about Maestro?! And if that’s the case, awesome. I’m glad to see that you’ve been marked safe from the media shitstorm. If you do know what I’m talking about and have seen all the chatter on the interwebs (or perhaps you read about it in last week’s Curious Companion), this is me making good on my promise to devote an entire episode to sharing my thoughts on the matter.

Right off the bat let me directly answer the question: No, no part of me believes that AI will kill us all.

I’m not sure if any of you believe that it will (not here to shame you if you’re in that camp), but what I do know is that all the media noise last week had folks stirred up, and I had some people reach out to me to ask for my thoughts.

I don’t claim to be an official AI expert by any means, but I have spent (and continue to spend) a ton of time learning about it from all angles (financial, political, not just the tech side), and I absolutely have thoughts.

Some Background

Again, to state plainly, I do not think AI is going to kill us all. But before I go into why I think this, allow me to provide a little background on that statement for those of you who have no idea wtf I’m talking about:

On September 8th, former Anthropic employee Jacob Coxon resigned and posted a 7-part thread on X that shared why he resigned along with his concerns about AI. In one part he wrote: “The people building AI earnestly believe that it could kill us all by the end of the decade.”

That thread has over 170 million views at the time I’m writing this.

Things got even more interesting when Evan Hubinger, head of alignment stress testing at Anthropic (translation: he runs the team that tries to break Anthropic’s safety measures on purpose), tweeted: “Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.”

He followed that up by writing: “To be clear, as we say in our latest Risk Report I think the risk from present models is low. What I am worried about is superintelligence arising from recursive self-improvement, as we have said is happening faster than we thought.”

Ah, I feel so much better.

New Kids on the Block: Superintelligence and RSI

Let’s take a moment to step back from the dramatic predictions and go over some terms that I think you should know, as they are the new kids on the block and you’ll likely continue to see them used in AI discussions: superintelligence and recursive self-improvement (RSI).

Superintelligence — AI that substantially exceeds human ability across essentially all cognitive tasks, not just faster or bigger, but smarter than the best humans at everything, including the task of improving AI itself.
Recursive self-improvement (RSI) — an AI improving its own intelligence or capabilities, which lets it improve itself again, a feedback loop where each iteration makes the next one faster or better, with no human in that loop.

I wrote about both these terms last episode, and how researchers have moved the AI-accomplishment goalposts from AGI (artificial general intelligence: AI that can understand, learn, and perform any intellectual task a human can, across different domains, not just the narrow things it was trained for) to superintelligence with RSI.

Ultimately, it’s not so much that I don’t think superintelligence is possible, it’s that these concepts continue to be talked about in such a nebulous manner and that makes it difficult to take the foreboding claims seriously.

A Handy Prediction

I will not deny that AI as we know it is an absolutely incredible technology that has advanced rapidly since the release of ChatGPT (GPT-3.5) in 2022. However, like I discussed in the previous episode, these advances have continued to focus on agentic capabilities, with improvements most noticeable on benchmark testing and for folks using it for things like coding, research, and long agentic workflows.

For something supposedly headed towards superintelligence, aka it should exceed human ability across essentially all cognitive tasks, you’d expect all users to notice an improvement in capability. I’m not sure of the last time you had AI try to write something for you, but that use case seems pretty far off from super-intelligence.

But circling back to its coding capabilities, I’ve also spent the past few episodes speaking about the fact that I’m most concerned about AI’s hacking capabilities, and that concern still stands. I read a great article by Jake Handy (I’d link it, but it’s behind a paywall as part of his Substack) that spoke to his prediction about our future with AI. He wrote:

“What I do think is that it’s very likely something big will break within five years. There are clear gaps and vulnerabilities in these lab-based systems that are starting to show. Inevitably, this will lead to some kind of expensive crisis. Financial plumbing, a clearing or settlement system, a major cloud dependency, a payments network, a utility’s billing or scheduling layer. One or more of these things can, and likely will, get wrecked by an AI system soon, and hopefully it’s a continuation of the wake up call Coxon has kickstarted.”

This is a prediction I can absolutely get behind.

Give Us Receipts, Not a Tweet

I’m not going to say that the AI killing us all talk is propaganda, but I will say that it’s irresponsible. What do you mean you think there is a >10% chance that AI kills us within the next decade, and all that you’re doing is posing on X about it? Give us some depth, give us some more words, give us some transparency and actually explain the danger. Bring the receipts! Right now all we have are tweets.

Where this leaves me is what I’ve said before: I absolutely believe in the capability of AI, I respect the fact that it could be dangerous and needs regulations, but I also find it hard to fully believe the words of the whistleblowers and the folks in charge because they continue to be vague.Realistically, there are so many things that if left unchecked and given full access to one’s imagination, could be positioned as possibly capable of killing humans within 10 years.

I’m never here to be a conspiracy theorist, but like I mentioned earlier, I consume a lot of content surrounding all facets of AI and I’d be remiss if I didn’t mention the fact that stories that tout how powerful AI is (even when those stories speak to human extinction) have the potential to be net positive for AI companies who are concerned about conveying how valuable this technology is and why it should continue to receive funding.

Yes, this is incredible technology, yes it absolutely needs regulations and should be taken seriously, and also, yes, it makes perfect sense to question the motives of those in power.

If They Wanted To, They Would

I recently listened to a podcast episode that had Ed Zintron, a well-known AI antagonist pundit, on as guest. During the episode he made an excellent point regarding regulation and basically said that if they wanted to, they would. Not wrong. My addition here is that Mamdani has shown us that if you really want to, it is in fact possible to get shit done. We don’t have AI regulations because our government doesn’t want to regulate this technology, it wants to profit from this technology.

Trump has gone so far as to say that data centers are the oil of the next 20-25 years. It’s all about money. He’s not trying to regulate shit, because regulation means decreased revenue. If you haven’t seen it already, prepare yourself for the China propaganda and messaging that we have to beat them in the AI race, whatever that means.

As for the heads of AI (ex: Dario Amodei, Sam Altman, Elon), following Coxongate, they each made statements encouraging slowing down the pace of AI research and advancements, yet only Dario Amodei, CEO of Anthropic, has set forth a plan. He wrote a long blog post about it, but honestly to me it feels like lip service and wishful thinking. If you’ve got some time, give it a read and be the judge yourself.

In the meantime, allow me to share a little snippet with you that will perhaps convey why I feel how I do. In one part of the paper Dario writes: “I believe all frontier labs should partner with government to formalize the idea of permanent embedded evaluators to better prevent and document internal alignment incidents like those that have occurred in the last few months, and to implement regulation focused on keeping capabilities in balance with safety.”

Trump is out here likening data centers to oil and saying we’ve got to beat China in the AI race. I wouldn’t hold my breath about any kind of significant government partnership focused on regulation.

It’s Not a Vacuum

Hopefully I’m not coming off at all doom and gloom, because I really don’t feel doom or gloom about AI. I consider myself an AI realist, and that’s how I continue to approach things.

One thing that I continue to return to is the fact that change doesn’t occur in a vacuum.

Yes, AI will be making advances over the next decade, but so will humans in general. Things like YouTube and Reddit (and any open-source content/platform) continue to give me hope and remain a beacon of what I believe to be the core of the human spirit, with so many folks uploading tutorials and typing out complex multi-step solutions to obscure problems…just to be helpful. These same people will be showing up for the next decade. The good girl/guy/they/them hackers (they do exist) will also be around, making advances.

Change doesn’t happen in a vacuum, and we’d be well served to remember that we won’t be forced go stay the exact same as AI gets to improve for the next decade.

How I Used AI This Week

Each episode I share a quick example of how I used AI that week.

This past week I Swapped out the thermostat on my Jeep Wrangler, and when I took the new housing out of the box, it made a rattling sound. I consulted my French assistant Claude to see if that noise was normal, and he confidently said no and that the housing was probably broken.

Spoiler: it was not broken, and the noise was in fact normal. I wound up driving to the dealership to buy another thermostat housing, and one of the mechanics there taught me that the noise was from a small bearing that helps regulate the pressure.

All that to say, as we know, AI can absolutely be wrong, and it will do so confidently. Feel free to use this analogy to speak to AI’s proximity to achieving superintelligence, but as it relates to a more immediate application, remember when using AI, always check your work.

Da Wrap-up

No, AI is not going to kill us all. The doom talk is loud, but it’s also vague, and perhaps even more than that, we gotta remember that change doesn’t occur in a vacuum. I absolutely respect the fact that AI is extremely capable in many domains (some potentially dangerous), and I’ll be the first to advocate for regulation. But no, I do not think that AI is going to kill us all, and as per always my preferred response to the noise is…curiosity.

As always, endlessly grateful for you and your curiosity.

Catch you next Thursday.

Maestro out.