I can't see a path where this would ever be true. AI is too valuable of a thing for everyone to link hands and cooperate.

Additionally, this seems like an incomplete answer. Unless the moratorium lasts for nearly 75 years, the only way it'd help is by giving time to agree on some other "real" fix that will endure. But in that case, the moratorium will just be a footnote in history.

The world coordinated to get enough time to save humanity and the future, when a lot of people thought it was impossible.

Whatever the mean to secure AI we found after this, it will not be a footnote in history, it is epic.

One path is that the most powerful AIs are never given intrinsic motivations of their own.

If a person uses AI as a tool to engineer a super virus, who actually killed humanity? The gun, or the person holding the gun?

In this "the person holding the AI did it" viewpoint, we're just as likely to see AI used to save humanity, e.g. is used to help develop a vaccine.

To develop this a little further, what if we handed every org of more than 100 people the ability to launch a nuclear warhead. That kind of power in the hands of so many people is obviously folly, because someone is going to decide to launch.

AI is of course the warhead, but AI is more flexible. Every org is going to consider the risk, and many of them will set their AIs to work on building a nuclear defense shield (or equivalent).

Offense is usually easier than defense. I think humanity will be in for a rough time. But AI will get used for both roles, and it may ultimately be a human who's "to blame" if AI shatters civilization.

“An AI singularity or intelligence explosion never happened.”

Removed a typo, this sentence uses to be

"As AI became safer as it got more powerful without much human effort outside of some RLHFing"

This question got me thinking about optimal formats for this, so I'm trying a weird one where all trades are cancelled after a week but then it re-resolves in 2060.

/Joshua/if-humanity-avoids-ai-doom-until-20

“AutoGPT6 escaped, started doing something, was caught and vivisected. This was enough of a warning shot to create AI CERN, which miraculously succeeded."

I am sufficiently pessimistic about humanity's ability to coordinate that I think most surviving worlds in 2100 are ones in which we are just lucky and it turns out that the relevant technology is much harder to invent than we think it is. Specifically, we might be lucky and:

A) The next AI breakthrough on the order of Transformers simply never arrives. LLMs keep getting better, but no amount of additional training data makes them a superintelligence.

or

B) It turns out there are no superweapons. Nanotechnology just doesn't work how we currently expect it to, engineering super-viruses turns out to be impossible, etc. Without any easy way to kill us all instantly, AI decides to work with us instead.

I am sure someone could phrase these better than me, but they are what I'm hoping for. I still think we should be desperately trying to coordinate moratorium treaties and develop human intelligence augmentation etc, but I doubt we pull those off.

@Joshua I’d like to second (B) especially. I work in nanoscience and I’m shocked by how seriously people take Eric Drexler’s ideas (I hesitate to say pseudoscience, but they’re certainly not very rigorous). I just don’t think it’s plausible that even a superintelligence could figure out how to engineer self-replicating nanobots and the like.

Previous questions like this to mine for answers:

The new answer format you're using is better, of course.

@Joshua Thanks. I totally missed this despite doing some searching. Maybe I'll close the market if the overlap is too large.

@Jono3h Okay, that being a "pick one" question makes it pretty unappealing. My market stays live!

@Jono3h Don't close it! An unlinked multichoice is much better than those old linked parimutual markets.

@Joshua Oh this also exists which is a duplicate of EY's market but with the current linked format that allows shorting:

So I would expect it to perhaps have more accurate percentages than EY's original, even though it has fewer traders. Probably also worth including the description?

My reasoning is that in general, large groups of people mostly make big changes in response to disasters.

Most of my probability mass is on things like energy scarcity, climate issues, etc. that just make AI research unfeasible.

Also significant is a failed takeover, causing everyone to understand the risk more viscerally. But that's hard to estimate. It's hard to imagine an AI causing significant enough damage without also just winning.

If it turns out to be too hard to make creative agents, then we survive for free. I wouldn't count on it but possibly it's true.

reposting because its a good question, please submit!

this is a great question, but is too far beyond my planning horizon to expect useful resolution. no bet.

@L 2030 or 2040 feel more bettable

I can make a copy of this for 2040, though I expect similar results (and these questions cost me 30% of my entire capital to create)
Gimme bucks or make one and I'll link it.