> In 1993 Vernor Vinge drew on computer science and his fellow science-fiction writers to argue that ordinary human history was drawing to a close. We would surely create superhuman intelligence sometime within the next three decades, leading to a “Singularity”, in which AI would start feeding on itself.

Yes it was Vernor, but he said something much more interesting: that as the speed of innovation itself sped up (the derivative of acceleration) the curve could bend up until it became essentially vertical, literally a singularity in the curve. And then things on the other side of that singularity would be incomprehensible to those of us on our side of it. This is reflected in Peace and Fire upon the deep and other of his novels going back before the essay.

You can see in this idea is itself rooted in ideas from Alvin Toffler in the 70s (Future Shock) and Ray Lafferty in the 60s (e.g. Slow Tuesday Night).

So AI machines were just part of the enabling phenomena -- the most important, and yes the center of his '93 essay. But the core of the metaphor was broader than that.

I'm a little disappointed that The Economist, of all publications, didn't get ths quite right, but in their defense, it was a bit tangental to the point of the essay.

When I read the Coming Technological Singularity back in the mid-90s it resonated with me and for a while I was a singularitarian- basically, dedicated to learning enough technology, and doing enough projects that I could help contribute to that singularity. Nowadays I think that's not the best way to spend my time, but it was interesting to meet Larry Page and see that he had concluded something familiar (for those not aware, Larry founded Google to provide a consistent revenue stream to carry out ML research to enable the singularity, and would be quite happy if robots replaced humans).

[ edit: I reread "The Coming Technogical Singularity". There's an entire section at the bottom that pretty much covers the past 5 years of generative models as a form of intelligence augmentation, he was very prescient. ]

Fire Upon the Deep is the first of 2 (or 3?). I highly recommend that. Older readers will love the Usenet jokes.

It seems impossible that, given current economic trajectories, there won't be software archaeology in the future.

Our abusive relationship with modern commercialism has disintegrated the value of art, folklore and tools, both in the eyes of consumers and producers, and we no longer as a society seek to preserve the greatest works of today's most cutting-edge mediums. It's quite a sad state of affairs, which will only be mourned with increasing intensity as time marches on and curious researchers try to piece together the early days of the internet and personal computing.

In the more common case, it's simply archiving -- we still have hardware it can run on (e.g. x86).

The GPGPU and ML stuff is likely going to age poorer, although at least we had the sense to funnel most high level code through standardized APIs and libraries.

It’s not hard to imagine a program of ever increasing complexity, highly redundant, error-prone, yet also stochastically, statically, increasingly effective.

The primary pushback from this would be our pathetically tiny short term memory. Blow that limitation up and complexity that seems incomprehensible to us becomes perfectly reasonable.

Situation that have 500 viewpoints and 10,000 variously interacting facts to consider won't be any more comprehensible for a mere human, short term memory limiting us or not.

Other things are quite advanced - but computing is rather "dumb". What is there is driven by a great weight of inertia of massive software (all written by regular humans) that itself is too large to fully audit and understand.

The story it tells is very much one about humans and societies.

We do this through categorization. For example, integrated circuits can be broken down into simplistic AND, OR, XOR, NOR, NAND, X OR, and NOt gates. Yet tying those together creates usable components such as Wifi radios and manufacturing automation.

Over simplification are concepts that lack understanding of complex systems. These are exploited by politicians. For example, deregulation will help drive the economy. Yet regulations have been proven to assist the economy long term by preventing the next variation of thalidomide. The panacea drug of the 1950s and 1960s that killed their users or caused massive birth defects.

Making a competing standard and then having to deal with interop between standards is not simpler or better unless there's an actual benefit to be had.

Was the sarcasm lost on me?

So behind all the top-level interfaces was layer under layer of support. Some of that software had been designed for wildly different situations. Every so often, the inconsistencies caused fatal accidents. Despite the romance of spaceflight, the most common accidents were simply caused by ancient, misused programs finally getting their revenge.

“We should rewrite it all,” said Pham.

“It’s been done,” said Sura, not looking up. She was preparing to go off-Watch, and had spent the last four days trying to root a problem out of the coldsleep automation.

“It’s been tried,” corrected Bret, just back from the freezers. “But even the top levels of fleet system code are enormous. You and a thousand of your friends would have to work for a century or so to reproduce it.” Trinli grinned evilly. “And guess what—even if you did, by the time you finished, you’d have your own set of inconsistencies. And you still wouldn’t be consistent with all the applications that might be needed now and then.”

Sura gave up on her debugging for the moment. “The word for all this is ‘mature programming environment.’ Basically, when hardware performance has been pushed to its final limit Basically, when hardware performance has been pushed to its final limit, and programmers have had several centuries to code, you reach a point where there is far more significant code than can be rationalized. The best you can do is understand the overall layering, and know how to search for the oddball tool that may come in handy—take the situation I have here.” She waved at the dependency chart she had been working on. “We are low on working fluid for the coffins. Like a million other things, there was none for sale on dear old Canberra. Well, the obvious thing is to move the coffins near the aft hull, and cool by direct radiation. We don’t have the proper equipment to support this—so lately, I’ve been doing my share of archeology. It seems that five hundred years ago, a similar thing happened after an in-system war at Torma. They hacked together a temperature maintenance package that is precisely what we need.”

“Almost precisely.” Bret was grinning again. “With some minor revisions.”

“Yes, which I’ve almost completed.”

The essence of many many projects and bane and also the source of so much work for all of us (software devs)

The hero saves the day by hacking old routines lying in the depth of the ship systems.

I'm not sure why there would be cognitive dissonance- sure, my tools may be primitive, but I can also grab my chisel and plane and see that it's similar in form to chisel and plane from 2000 years ago (they look pretty much the same, but these days they're made of stronger stuff). I can easily imagine a Real Programmer 2000 years from now looking back and thinking that python, or even a vacuum tube, is merely a simplified version of their quantum matter assembler.

Even as IT professionals this is true. How many developers these days can debug a problem in their JavaScript runtime or Rust developers track down a bug in their CPU? There’s so much abstracted away from us that few people can fully grasp the entire stack their code executes. So those outside of tech don’t even stand a chance understanding computers.

And that’s just focusing on IT. I also depend on medical equipment operated by doctors but have no way of completely understanding that equipment nor procedure myself. I drive a car that I couldn’t repair. Watch TV that I didn’t produce beamed to me via satellites that I didnt built nor fire into space. Eat food that I didn’t grow.

We are already well past the point of understanding the technology behind the stuff that we depend upon daily.

I originally started from writing Python scripts and running Web servers, after 10 years I decided that pure software is too boring, and now I came to know the basics of high-speed digital signal propagation in circuit boards.

So I agree with you conceptually. But practically speaking, it also requires an eternal life, an external brain storage, or possibly both. At least in my experience, I find that my time and attention doesn't allow me to investigate everything I'd like to.

"No one on the planet knows how to build a computer mouse."

https://briandbuckley.com/2012/09/12/who-knows-how-to-make-a...

Counterpoint:

If a man has done it, a man can do it. Also, It wouldn't be research if you already knew what you were doing.

Just because a particular assembly has it's tasks divided up between a myriad of people does not mean it is impossible to unify those tasks into a single person. In point of fact, the continued existence of mice can be directly attributed that someone has the network of knowledge/knowers of pieces of the problem already nailed down.

Yes, the level of detail the real world flings at us on a regular basis is surprisingly deep, but hardly ineffable.

I don't understand your argument, in particular, the linked article already refuted this point by saying the following, but you didn't provide a counterargument to that:

> But let’s imagine an extraordinarily talented man who started on the factory floor, worked his way through an engineering degree, moved up through the ranks to design the very thing he was building before, and knows the roles of everyone on his team so well that he could do all their jobs himself. Surely this brilliant person knows how to make a mouse. Or does he? He may understand circuits – but does he know every detail of how to build a diode from raw materials? He may understand plastics – but could he single-handedly synthesize a plastic from its constituent chemicals? Does he understand how to mine silicon out of the ground? Nobody in the world – not one single human being anywhere – knows how to make a mouse. It’s orders of magnitude too complex for a solitary mind.

If we've done it before, we can do it again. The key is navigable access to the right information which is sadly dependent on A) willingness to document, and B) structuring of the set of data for navigable retrieval.

Both were problems we've got solved. Not in the Internet of course. Not anymore, but in libraries.

Also, I reject where the goalposts of the accomplishments of the person in question are stated to arbitrarily end. Nothing keeps one from diving into these secondary areas or specialties. Only perhaps the obstacle of having to be profittable while doing it. And that's what I reject. I do not hold the prevailing wisdom that knowing the Riddle of Mice is intractable to a singular human being. I hold it is intractable to a member of a social system wherein profitable engagement of every member as guided by some subpopulation might tend to make it seem intractable by any member of the governed group. That's a far cry from true ineffability however.

But nobody has done it before. Even the very first computer mouse used off-the-shelf mechanical and electrical components, each of those involved at least one type of technology that took a scientist or engineer's entire lifetime to develop.

> The key is navigable access to the right information which is sadly dependent on A) willingness to document, and B) structuring of the set of data for navigable retrieval.

Now you mention the importance of documentation, it reminds me of Vannevar Bush's Memex and Ted Nelson's Project Xanadu. So it seems that there's a mutual misunderstanding of the actual topic in this debate. We understood the debate as:

* The all-knowing engineer: Whether it's possible for a single individual to learn and understand a technology entirely, down to its every aspect and detail.

Meanwhile, you're in fact debating about different problem, which is:

* The Engineering Library of Alexandria: Whether we can create sufficient documentation of all technical knowledge, the documentation is so complete about every aspect and detail that in principle, it would allow someone to open the "blackboxes" behind every technology for understanding or replication if they really want and need to. Whether or not it can be done in practice by a single physical person is unimportant, perhaps only one or a few blackboxes are opened at a time, not all simultaneously. The question is whether the preserved information is sufficient to allow that in theory. This is similar to the definition of falsifiability in science - impractical experiments still count.

If you're really arguing the second point rather than the first point, I would then say that I can finally understand some of your arguments. So much unproductive conversions can be avoided if you've expressed your points more clearly.

If anything, you’re massively underestimating the amount of diverse specialist knowledge required in modern society for even simple interactions.

People dedicate their entire lives to some of these fields that underpin the tech we use. It’s not something the average person can pick up in open university while working a full time job and bringing up kids. To suggest it’s just a motivational problem is absurd. And that’s without addressing that the level of physics or mathematics required for some specialties is beyond what some people can naturally accomplish (it would be stupid to assume everyone has the same natural aptitude to all disciplines).

Honestly, I don’t know how you can post “This is the most intellectually lazy take I've ever seen” in any seriousness then go on to say it’s just a problem of will power.

Cars, electronic devices, manufacturing tools/processes, lawn equipment, chemical processes, software, algorithms, common subassemblys, logistics, mechanics of materials, measuring systems, what excuse does one have to not develop some level of familiarity with these aspects of modern life in a day where information is literally at your fingertips?

I'll give you that the well is heavily poisoned, and that sometimes a return to dead tree mediums of info storage are required to boost signal; but there isn't really an excuse to remain ignorant other than willful ignorance. The answers are out there, you just need to have the will to hunt them down and wrest them from the world.

This could also be described as the Dunning-Kruger effect

https://en.m.wikipedia.org/wiki/Dunning–Kruger_effect

There is a massive difference between “learning the basics” and understanding it to a competent level that we can reproduce it.

I know the basics of how rockets work but that doesn’t mean I could build a rocket and successfully launch it. Yet your comment suggests I’m already qualified to single-handedly deploy the next Starlink.

That's just something you tell yourself to make yourself feel better about trusting the black boxes your life depends on. But it simply isn't true.

People devote their lives to specializing n these things. Yes, "with enough will" - and time, and money - you could pick one or two subjects and do the same. But that still leaves you as a dilettante at best when it comes to everything else.

Rand is garbage, and I'm insulted to end up being brought anywhere near that slop in your philosophical address space. The point I'm trying to make is that there is a helplessness taught by the optimizations we structure our societies around. A capitalist, consumerist society is going to focus around training the largest portion of it's population to act as specialized cogs that can be orchestrated by someone the next layer of abstraction up in pursuit of purely profitable/profitmaking engagements.

If you change the optimizations, you change the system. You change the people that compose that system, you change the very bounds of the human experience.

Just look at how much the world changed around the pandemic. The old order of the office & commute was shattered for many. Look at how deemphasis of throwing more fossil fuels at a problem changes the direction of innovation.

You say the Riddle of Mice is ineffable by a single person. I say you're full of it, and looking at the problem wrong. It's completely effable, you just can't imagine society having a place for such a person given your priors about how the world operates. And in that, you may be right!

That does not equal the Riddle of Mice being conquered by a single person can't happen. It's just unlikely, and dhould such an extraordinary individual exist, you'd likely see them as mad.

Actual AGI is nowhere near realism yet anyway.

I admit looking at the 100 years before 1993, it looks like innovation is constantly speeding up, but even then there's not going to be a moment that we suddenly have infinite knowledge. There's no such thing as infinite knowledge; it's still bound by physical limits. It still takes time and resources to actually do something with it.

And if you look at the past 30 years, it doesn't really look like innovation is speeding up at all. There is plenty of innovation, but is it happening at an ever faster pace? I don't see it. Not to mention that much of it is hype and fashion, and not really fundamentally new. Even AI progress is driven mostly by faster hardware and more data, and not really fundamentally new technologies.

And that's not even getting into the science crisis: lots of science is not really reproducible. And while LLMs are certainly an exciting new technology, it's not at all clear that they're really more than a glorified autocorrect.

So I'm extremely skeptical about those singularity ideas. It's an exciting SciFi idea, but I don't think it's true. And certainly not within the next 30 years.

Are we sure things like biology, or heck, even the universe as a whole and its parts, aren't "glorified x thing"? Can't we apply this argument to just about anything?

As for the prompt(s), to use such a limiting term, they could as well come from a self-reinforcing loop that starts when we're born and is influenced by external stimuli.

I could be wildly off base, but seeing many of the (often heated) arguments made about what AI is or isn't or should or could be, it makes me wonder.

It seems to me that on one extreme there are people easily anthropomorphising advanced computing and on the other extreme there are people trivializing it with sentences like "glorified x thing". This time around it's "glorified autocorrect" and its derivations. It's always something that glorifies another artificial thing, and I suspect that if and when we will have recreated the human brain, or heck, another human, it will still be a "glorified x thing".

As 0x0203 said, maybe it is to be ascribed to the religious substrate that takes offence at anything that arrogantly tries to resemble the living creatures made by God, or God himself.

It's another application of advanced technology appearing to be magic, but imagine transitioning into it in a matter of hours, then with that advanced tech transitioning further into damn-near godhood within minutes.

Then imagine what happens in the second after that.

It may be operating within the boundaries of physics, but they would be physical rules well beyond our understanding and may even be infinite by our own limited definition of physics.

That's the curve.

Hardware is a bottleneck. Better hardware can't be designed, manufactured and installed in minutes.

You can't just infinitely improve software sitting on the same hardware. At some point the software is as efficient as it can possibly be. And realistically a complex piece of software won't be close to optimal.

You people love dreaming up possibilities but you never stop to actually think about the limitations. You just assume everything will work out magically and satisfy your fantasy.

So singularity people think 2 fallacies:

1. That it is possible to write an exponential function that somehow scales to asymptotic

2. That it is possible to write software that is so energy efficient that its output somehow exceeds the power draw

1 can be redefined to say "effectively" vertical. Ok. Cool motive, not a singularity.

2 can be handled by pumping more and more power in to the machine and building new nuclear plants to do so. This is what Microsoft is doing today.

Yes you can. The Cloud.

I use LLMs regularly in my job - many times a day - and I suspect you haven't used them much if you think this.

They're relevant to the singularity discussion though, because they already give a taste of what superhuman intelligence could look like.

ChatGPT, for example, is objectively superhuman in many ways, despite its significant limitations. Once systems like this are more integrated with the outside world and able to learn more directly from feedback, we'll get an even bigger leap forward.

Dismissing this as "glorified autocorrect" is extremely far off base.

But so is a car.

ChatGPT is mostly superhuman in its ability to draw upon enormous numbers of existing sources. It's not superhuman, or even human, in terms of logic, reasoning, or inventing something truly new.

Was this intended literally? I'm skeptical that saying something so precise about a fuzzy metric like rate of innovation is warranted.

https://en.wikipedia.org/wiki/Jerk_(physics)

The physical world tends to have certain constraints that make such true singularities impossible. For example, the universal speed limit: c. But, you could argues that we could approximate a singularity well enough to fool us humans.

It was something like the "speed of increase in inflation is slowing down" I've tried to search for it but no joy.

Anyway, it was maths.

Suggesting the singularity is physically possible is roughly like suggesting that human telekinesis is possible, ie physics would need many very interesting errors.

Anyway, imo the event horizon is more interesting. That's where the paradigm shift happens - there is no way to escape whatever "comes next".

(Note - some people confusingly use "singularity" to refer to the phase between the event horizon and "infinity")

If there is anything "beyond" that at all, I think it's pretty safe to say that any idea we have as to what it would be is very unlikely to be anywhere near the realm of correct.

> In 1993 Vernor Vinge drew on computer science and his fellow science-fiction writers…

But I don’t think it’s worth completely writing off as if we couldn’t possibly know anything. Progress outside of time would resolve the issue I resolved to. I’m not saying it could actually be done or doesn’t cause a million other issues. It’s an idea for a solution, but it’s completely unrealistic, so you shelve it.

The more I thought about it though, the more I realized it was the perfect name. It is definitely what you feel when the acceleration changes!

>...accelerating progress of technology and changes in the mode of human life, which gives the appearance of approaching some essential singularity in the history of the race beyond which human affairs, as we know them, could not continue.

Personally it's always bugged me that the "technological singularity" is some vague vision rather than a mathematical singularity. I suggest we redefine it at the amount of goods you can produce with one hour of human labour. When the robots can do it with zero human labour you get division by zero and a proper singularity.

It's a guest essay. The Economist does not edit guest essays. They routinely publish guest essays from unabashed propagandists as well.

https://www.prospectmagazine.co.uk/culture/40025/what-the-ec...

It got so many things right, it’s really amazing.

I really wish he’d written more.

Note that this category of hypothesis was common in various disciplines at the end of the Cold War [1]. (Vinge's being unique because the precipice lies ahead, not behind.)

[1] https://en.wikipedia.org/wiki/The_End_of_History_and_the_Las...

My go-to example is that we have run into similar things with the pace and volume of "Science". For a long while one could be a scientific gentleman and keep up with the sciences. As a whole. Then quite suddenly, on the other side, you couldn't and you had to settle for one field. And then it happened again: people noticed that you can't master one field even, on the current side. And you have to become a hyper-specialist to master your niche. You can still be a generalist - in order to attack specific questions - but you better have contacts who are hyper-specialists for what you really need.

Today, I am trying (but struggling!) to keep up with the evolving libraries and frameworks for frontend development in Typescript!

In other words, Vernor described an exponential curve. But are there any exponential curves in reality? AFAIK they always hit resource limits where growth stops. That is, anything that looks like an exponential curve eventually becomes an S-shaped curve.

I'm also gonna recommend Accelerando - https://www.antipope.org/charlie/blog-static/fiction/acceler...

As an aside, I'd also recommend Glasshouse (also by Charles Stross) as an exploration into the human remnants post singularity (and war)... followed by Implied Spaces by Walter Jon Williams...

> “I and my confederates,” Aristide said, “did our best to prevent that degree of autonomy among artificial intelligences. We made the decision to turn away from the Vingean Singularity before most people even knew what it was. But—” He made a gesture with his hands as if dropping a ball. “—I claim no more than the average share of wisdom. We could have made mistakes.”

for a singularity averted approach of what could be done.

One more I'll toss in, is The Freeze-Frame Revolution by Peter Watts which feels like you're missing a lot of the story (but it is because that's one book of a series) and... well... spoilers.

His example was the internet boom. Look back to the very toe of the curve in the late 80s and early 90s where the internet started being a thing that was accessible and not just a way for colleges to send email to each other. Could you predict what it would be like two or three decades later?

The other works are science fiction that explore the Vingean Singularity (which was from '84) that were written in the '00s and explore (not panic) the ideas around society and economy during the time of accelerating technical growth and after.

My conclusion is similar to Verge’s: predicting the far future is impossible. We can imagine a variety of scenarios, but you shouldn’t place much faith in them.

Predicting even a couple years in advance looks pretty hard. Consider the next presidential election: the most boring scenario is Biden vs. Trump and Biden wins. Wildcard scenarios: either one of them, or both, dies before election day. Who can rule that out?

Also consider that in any given year, there could be another pandemic.

History is largely a sequence of surprise events.

It’s intuitive and simple to you and me. But languages change, slang changes, metaphors change and equipment changes. Business models exist today that were unthinkable 40 years ago because the ubiquity of computation did not exist.

She’s suffering, in Toffler’s words, a “future shock”.

Now imagine that another 40 years worth of innovation happens in a decade. And then again in the subsequent five. And faster. You’ll have a hard time keeping up. And not just you: kids will too. Things become incomprehensible without machines doing most of the work — including explanation. Eventually you, or your kids, won’t even understand what’s going on 90% of the time…then less and less.

I sometimes like to muse on what a Victorian person would make of today if transported through time. Or someone from 16th century Europe. Or Archimedes. They’d mostly understand a lot, I think. But lately I’ve started to think of someone from the 1950s. They might even find today harder to understand than the others would.

That crossover point is when the world becomes incomprehensible in a flash. That’s a mathematical singularity (metaphorically speaking).

why do you think someone from the 50s would find it harder to understand our time, than someone from an earlier age, even as far back as 2000 years?

If they had experienced the transition suddenly instead of gradually, I still think they'd be fine relatively soon.

This also reminds me of the Eloi in the Time Machine, a book written in 1895!

The claim might be a year or two premature. But, not five.

Based on my understanding of people, we probably have a long way to go.

Yes, there's an ever faster pace of change, but that pace of change for the general human population is limited precisely because of the factors you laid out: people can't keep up.

William Gibson said: "The future is already here – it's just not very evenly distributed."

That's what has been happening for millennia, and will be ever more obvious as the pace of progress accelerates: sure, someone, somewhere will have developed incomprehensible ultra-advanced technology, but it won't spread precisely because the general population can't comprehend/adopt/adapt it fast enough![1]

Exceptions exist, of course.

My take on the whole thing is that the "runaway singularity" won't happen globally, it'll happen very locally. Some AI supercomputer cluster will figure out "everything", have access to a nano-fabrication device, build itself a new substrate, transfer, repeat, and then zip off to silicon nirvana in a matter of hours...

... leaving us behind, just like we've left the primitive uncontacted tribes in the Amazon behind. They don't know or care about the latest ChatGPT features, iPhone computational photography, or Tesla robotics. They're stuck where they are precisely because they too far removed and so can't keep up.

[1] Here's my equivalent example to your iPhone example: 4K HDR videos. I can create these, but I can't send them to any of my relatives that would like to see them, because nobody has purchased HDR TVs yet, and they're also all using pre-HDR mobile phones. Display panel tech has been advancing fantastically fast, but adoption isn't.

$ 9 88xo

9 v999 ii 7 8899 of i8888iy i99io o9 ⁹99898 o ⁹88o98ici9 i8o i 88i8 f9 f

AI does not scare me. People wielding AI as a tool for their own endeavors certainly does.

I think human languages and psyches just aren't built to cope with the concept of AI. Many words have loose meanings, like "learning" in the case of AIs, that can easily be twisted to mean one of dozens of definitions, depending on the stance of the person talking about it. It'll be interesting as the technology becomes more prevalent and mundane how people start treating it all. I'm hoping we get to realizing that a computer isn't a human regardless of the eloquence of its "speech" or whatever words we use to describe what it does, but I guess we'll see

Never heard anyone say this and I know (and know of) a lot of doomers. Honestly, this entire line of discussion would be far less frustrating if it weren't for the endless strawmanning and name-calling.

I'm concerned with whether they are artificial intelligence at all, what the bounding functions for AI's development are, how we could possibly solve the alignment problem, and whether we even understand human intelligence enough to recognize an artificial intelligence at all or predict how intelligence might work in something drastically more powerful than us.

My anecdotal experience with how people treat Alexa devices does not inspire confidence in me with regards to this. I can't even convince my wife not to gender it when referring to it.

It’s not limited to AI or LLMs - theory of mind is a powerful explanatory tool that helps us navigate a complex world, and we misapply it all the time.

If you can take pity on the final, dented avocado at the shops because it looks "sad", you will for sure end up calling Alexa 'she'. Avocados can't be sad, but they can look sad /to humans/, and a machine can't really have a gender or be polite in the human sense, but it can definitely sound like a polite lady.

I think humans will just fundamentally relate to anything they perceive socially as another human, even if we know full well they aren't human. Probably it's a lot less work for a human brain, than it is to try to engage with the true essence of being an avocado or an Alexa.

If we anthropomorphic animals and even vegetables regularly, what happens when something appears to talk back to us intelligently? I don't think these mechanisms are nearly as distinct as you make them out to be.

Obviously making an object mimick human traits makes it a lot easier to anthropomorphize. Like if you put a little cowboy hat on a pear. But you wouldn't say that somebody doesn't "understand that fruits aren't human" if they assign human traits to them. Making a machine that talks back at us in a seemingly intelligent way is a much more intensely human trait for it to have. It's not down to our intellectual understanding of the thing, but how our brains decode our (social) world.

So the anthropomorphizing happens regardless of the level of technical understanding. We both know it's just a pear, but it's also a cowboy now.

Of course there's a level of playfulness involved with all of this too. It's kinda fun to backsass your GPS instructions, or tell Alexa she's being nosy, or ask your dog what he thinks of the presidential debate.

The argument is not that they are identical. LLMs and diffusion models learning is a new thing. We are all trying to come to terms what that means, and how we should regulate it. (And if at all we should regulate it) Do note, we are talking about here what the law should be, not what the law is.

And in doing so we compare this new thing to already existing things. "It is a bit similar to this in this regard" or "it is unlike that thing in this other regard".

I don't think it is controversial to say that if an LLM outputs byte-to-byte the text of a copyrighted work the work remains under copyright. The person who run the LLM does not magically gain rights by that. If you coax your model to output the text of the Harry Potter books you don't suddenly become able to publish it.

The question is what happens if the new work contains elements from copyrighted works. For example if it borrows the "magical school" setting from HP but mixes it with Nordic mythology and makes it as deadly as Game of Thrones. What then? Can they publish this new thing? Do they need to pay fees to J K Rowling, and George R. R. Martin?

It is generally permissible to publish that new work if it was created by a human. If it is sufficiently different from the other works you are free to write and publish it. Does this suddenly change just because the text was output by an LLM?

The argument is not that "human learning" and "machine learning" is the same. It is that they are similar enough that one has to argue why you think one can create new work and why the other can't.

Let’s try to elevate the conversation: there is a newly published author MrX. His book combines themes from many copyrighted works in a novel way and is a critical and commercial success. There is no allegation of any copyright infringement around the work. Suddenly information comes out that MrX used LLMs heavily in the creation of the book and the LLM was trained on copyrighted works.

Do you think MrX owes licencing fees to those other authors? And which ones? All the ones in the training set? Just the ones which has thematic/stylistic similarity to his work?

How come those thematic/stylistic similarities only matter if MrX used an LLM and don’t matter if he used his own head?

I don’t think the point is that the two are exactly identical or that humans and LLMs are equivalent, but that the processes are similar enough in the general level that any attempt to regulate LLM training in copyrighted material will inevitably have the same ramifications for human learning.

In pretty much all attempts I’ve seen to differentiate the two, it inevitably boils down to hand-waving about how human beings are “special”.

A human artist looks at a lot of different sources, builds up a black-box statistical model of how to create from that and can reproduce other styles on demand based on a few samples. Generative AI follows the same process. What distinction do you want to draw to say that they should be treated differently legally? And why would that even be desirable?

Even if it's true (which is debatable,) it doesn't appear to be more informative than saying "they are both made of atoms."

How are you going to demonstrate that consciousness is responsible for what the artists are doing? We have undeniable proof that the art could be created by a statistical model, there is solid evidence that the brain creates art by simulating a mathematical neural network to achieve creative outcomes - the brain is full of relatively simple neurons linking together in a way that is logically similar to the way we're encoding information into these matrices.

So it is quite reasonable to believe that the artists are conscious but suspect that consciousness isn't involved in the process of creating a copyrighted work. How does that get dealt with?

These AI's aren't doing anything like that. They have no preference or intent. Their choices change depending on setting like temperature or prompts.

Or let's try a different example. Stephen King wrote a novel where he imagined the protagonist gets killed and eaten by the villain's pet pig (Misery). He struggled to come up with a different ending because he said nobody wants to read a whole novel just to see the main character die in the end. He thought about it and did a different ending.

Are you claiming Stephen King's conscious deliberation wasn't part of his writing process? I'd say it clearly was.

Also, I don't really understand the consciousness is an illusion argument. If none of us are conscious, why should I justify any copyright policy preference to you? That would be like justifying a copyright policy preference to a doorknob. But somehow I'm also a doorknob in this analogy???

Suppose Bob says he's conscious and Jim says he isn't and we believe them. Doesn't that suggest we would have different policy preferences on how they are treated? It would appear murdering Jim wouldn't be particularly harmful but murdering Bob would. I don't have to show how Jim and Bob's mind differ to prefer policies that benefit Bob over Jim.

If you're trying to argue that you're doing something different from statistical sampling, not knowing how you're doing it isn't a very strong place to argue from. What you're experiencing is probably what it feels like for a sack of meat to take a statistical sample from an internal black-box model. Biologically that seems to be what is happening.

I have also done a fair amount of writing. The writing comes from a completely different part of my mind than the part that experiences the world. I see no reason to believe it is linked to consciousness, even allowing that consciousness does exist which is questionable in itself.

It is an unreasonable position to say that you don't know the process but it must be different from a known process that you also don't have experience using.

> Are you claiming Stephen King's conscious deliberation wasn't part of his writing process? I'd say it clearly was.

Unless you're claiming to have a psychic connection to Stephen King's consciousness, this is a remarkably weak claim. You have no idea how he was writing. Maybe he's even a philosophical zombie. Thanks to the rise of LLMs we know that philosophical zombies can write well.

And "clearly" is not so - I could spit out a lost Stephen King work in this comment that, depending on how good ChatGPT is these days, would be passable. It isn't obvious that it is the work of a conscious mind. It in fact would obviously be from a statistical model.

> If none of us are conscious, why should I justify any copyright policy preference to you?

I've been against copyright for more than a decade now. You tell me why it is justified even if consciousness is a factor. The edifice of copyright is culturally and economically destructive and also has been artistically devastating (there haven't been anywhere near as many great works in the last 50 years as there should have been in a culturally dynamic society).

If I sawed off your leg would it have the moral consideration of removing the leg of a barbie doll's leg if you feel your consciousness is an illusion?

When I write a story my brain might be doing something you could refer to as a black box calculation if you squint a little, but how is it "statistics?" When I feel the desire to urinate, or post comments on hacker news, or admire a rainbow, or sleep am I also "doing statistics?"

You seem to be referring to what people traditionally call "thinking" or "cognition" and rebranding it as "statistics" in search of some rhetorical point.

My point is human beings have things called "personalities" and "preferences" that inform their decision makings, including what to write. In what sense is that "statistics"?

The idea that the human subconscious is not consciously accessible is not a new idea. Freud had a few things to say about that. I don't think it tells us much about AI. I do think my subconscious ideas are informed by my consious preferences. If I hate puns I'm not going to imagine story jdeas involving puns, for example.

Most authors would prefer copyright exists because they'd prefer book publishers, bookstore retailers and the like pay them royalties instead of selling the books they made without paying them. It's pretty simple conceptually, at least with traditional books.

Copyright existed far longer than the last 50 years so how is our 50 years of culture relevant? The U.S. has had copyright since 1790.

Of course, these statement do not come from a place of considered action: everybody knows the machine is not conscious in the same way as a human, but the point is that an unfeeling machine even being able to make such claims makes us have to move the "consciousness" divider further and further back into the shadows, until it's just some nebulous vibe people insist must be somewhere. It's possible there is a clear and unambiguous way to define humans and intelligent animals conscious, but nobody has come up with a workable definition yet that lets us neatly divide them.

Another slight thing that gives me a little pause: you know how great our brain is at confabulating right? Have you ever done something, then had someone ask you why you did it? You generally tell them a story about how you thought about doing something, weighed the pros and cons etc, that isn't actually true when you think deep down. We like to think we are a single being that thinks carefully about everything it does. instead we're more like an explaining machine sitting on top of a big pile of confusing processes and making up stories why the processes do what they do. How exactly this last thought relates to the discussion I haven't figured out yet, its just something that comes to mind :)

In 1974 philosopher Thomas Nagel wrote, "Conscious experience is a widespread phenomenon. It occurs at many levels of animal life, though we cannot be sure of its presence in the simpler organisms, and it is very difficult to say in general what provides evidence of it. (Some extremists have been prepared to deny it even of mammals other than man.) No doubt it occurs in countless forms totally unimaginable to us, on other planets in other solar systems throughout the universe. But no matter how the form may vary, the fact that an organism has conscious experience at all means, basically, that there is something it is like to be that organism."

He's clearly willing to recognize consciousness to non verbal creatures.

Consciousness is an unobservable, undefinable thing which with LLMs in the mix we can theorise has no impact on reality; since we can reproduce all the important parts with matrices and a few basic functions. You can doubt facts all you want, but that is a pretty ironclad position as far as logic, evidence and rationality goes. Consciousnesses is going the way of the dodo in terms of importance.

> If I sawed off your leg would it have the moral consideration of removing the leg of a barbie doll's leg if you feel your consciousness is an illusion?

For sake of argument, lets say conclusive proof arises that Stephan King is a philosophical zombie. Do you believe that suddenly you can murder him? No; that'd be stupid and immoral. Morality isn't predicated on consciousness. I'm perfectly happy to argue about morality but consciousness isn't a thing that makes sense outside of talking about someone being knocked unconscious as a descriptive state.

> When I feel the desire to urinate, or post comments on hacker news, or admire a rainbow, or sleep am I also "doing statistics?"

No, you're responding to stimulus. But right now it looks extremely likely that the creative process is driven by statistics as has been revealed by the latest and greatest in AI. Unless you can think of a different mechanism - I'm happy to be surprised by other ideas at the moment it is the only serious explanation I know of.

> You seem to be referring to what people traditionally call "thinking" or "cognition" and rebranding it as "statistics" in search of some rhetorical point.

I don't think I've said anything about thinking or cognition. Although statistics will crack those too, but I'm expecting them to be more stateful processes than the current generation of AI techniques.

> Copyright existed far longer than the last 50 years so how is our 50 years of culture relevant? The U.S. has had copyright since 1790.

Yeah but the law has been continuously strengthened since then and as it's scope increases the damage gets worse. The last 50 years are where new works are effectively not going to enter the public domain before everyone who was around when they were created is dead.

No it isn't. I observe consciousness in myself, right?

This is just "fashion" and you're arguing against a fashion. How on earth people can claim consciousness doesn't exist when you're entire experience is indeed consciousness is really the most difficult and petulant "fashion trend" imaginable.

Me: "Hey I really enjoyed that surf" Internet guy: "No you were just responding to your training data because you're just a LLM"

It's ridiculous.

I'm not saying this has any meaningful implications for day to day existence, the illusion is a strong one. But LLMs have really shrunk the parts of the human experience that can't be explained by basic principles down to a wafer so thin it might not exist. In my view it is to thin to be worth believing in, but people will argue.

If there is no direct experience, and it really is just an illusion, do you mind if I cut off your genitals ? Because even if existence is an illusion, being that illusion and experiencing it is still obviously relevant?

Seems like a breathtakingly awkward position to take.

1) Transexualism and congenital analgesia are both things, so that isn't a meaningful test of consciousness if that is what you are going for.

2) I personally would be extremely upset ... but anything going on with me being extremely upset can be explained with analogy to ChatGPT + a few hardware generations + maybe some minor tweaks that are easy to foresee but not yet researched. Everything that would happen would be built up from basic stimulus-response's mediated by a big neural net and a bit of memory.

These ideas aren't that new, there have been schools of Buddhist thought that have been exploring them for around 2,500 years if you want a group that has thought through this more thoroughly. Probably other traditions that I haven't met yet. It is just that the latest LLMs have really put some practical we-know-because-we-have-done-it muscle behind what was already a strong theory.

I'm not confident enough (yet?) to take that all the way to its logical conclusions in how I live, but I think the evidence is strongest for that interpretation.

Also most humans are not great at fiction writing, but just for the record GPT 4 is bad at long form fiction writing. It has no idea how to do it. Its training set is likely just small chunks of text and LLMs make up everything one token at a time, which isn't conductive to managing a 100k word or greater story. It's also been trained to wrap things up in 2000 tokens or less. It does this even if you tell it not to. It also isn't very creative, at least in contrast to a professional fiction writer. It also can't really follow the implications of the words it writes. So if it implies something about a character 300 words in it'll have forgotten that by 4000 words in because it doesn't actually have a mental model of the character.

I mention this in case you are under the delusion Stephen King is going to lose his job to GPT 4, or that it actually is as creative as him.

I mean, if a chatbot could write as well as king for 200,000 words in a coherent, original story with a beginning middle and end that would be fascinating and terrifying. But we're not anywhere close to there yet.

>For sake of argument, lets say conclusive proof arises that Stephan King is a philosophical zombie. Do you believe that suddenly you can murder him? No; that'd be stupid and immoral. Morality isn't predicated on consciousness.

Morality has a lot to do with consciousness. If I chop down a tree branch at my house, it isn't immoral because the tree is not conscious- it lacks a central nervous system so as far as we can tell there is no suffering.

If I chop off your arm (or your gentiles as another poster suggested) your consciousness and the pain it suffers is what makes the action immoral.

If Stephen King were a p zombie that would say a lot about the morality of destroying him.

I don't even understand what belief system you are trying to communicate when you say things like consciousness is an illusion or has nothing to do with morality, or an LLM is the same thing as a brain.

Like, I realize these are not your own ideas and you are repeating memes you've heard elsewhere, so it's not like you are speaking in tongues.

But any human being presumably knows they have subjective experience (so why say consciousness is an illusion?) And anyone who has used ChatGPT know it doesn't operate like their brain (no sense of identity, agency, emotions, etc, no ability to form long term personal memories, no real opinion or preference on things) So I don't really get these takes.

I keep seeing this claim being made but I never understand what people mean by it. Do you mean that the colors we see, the sounds we hear, the tastes, smells, feels, emotions, dreams, inner dialog are all illusions? Isn't an illusion an experience? You're saying that experience itself is an illusion and there is nothing to experience.

I can't make sense of that. At any rate, I see no reason to suppose LLMs have experiences. They don't have bodies, so what would they be experiencing? When you say an LLM is identical to a person, I can't make good sense of that either. There's a thousand things people do that language models don't. Just the simple fact that I have to eat on a regular basis to survive is meaningful in a way that it can't be for a language model.

If an LLM generates text about preparing a certain meal because it's hungry, I know that's not true in a way it can be true of a human. So right away, there's reasons we say things that go beyond the statistical black box reasoning of an LLM. They don't have any bodies to attend to.

Mechanical drawing machines have existed forever, I loved, love, loved them when I was a kid and I used to hang the generated images on my wall. Never did I once look at those machines who could draw some pretty freaking awesome abstract art and think to myself, "well that's it, the vale of consciousness is so thin now, it's all an illusion", or "the machine is conscious".

As impressive as some of these models are at generating art, they are still drawing machines. They display the same amount of consciousness as a mechanical drawing machine.

I saw someone on Twitter ask ChatGPT-4 to draw a normal image, you know what it drew ? A picture of a suburban neighborhood, why might a conscious drawing machine do that?

The reason programs are different in front of the law is that humans can program programs to do whatever they like at scale, humans can't program humans to do whatever they like at scale.

So for example if it became legal to use images from an AI, then we would program an AI that basically copies images and it would be legal, because it is an AI. But at that point we know the law has been violated because programs that just copies images aren't a legal way to get copyrighted images for free.

You saying "But the AI is a black box" just means that you could have hidden anything in there, it doesn't prove anything. Legally it is the same as if you wrote a program to copy images.

I don't mean that to sound flippant -- if this wasn't your intent, then obviously you understand the distinction.

Which is not the case with the current AI models (or rather, the companies profiting off them), to my understanding.

We can make all kinds of navel-gazing arguments about why LLMs and humans are identical but at the end of the day you'll go home and poop and the LLM will blissfully idle on a forgotten AWS compute instance burning a hole into someone's pocket and any idiot can see that you two are not the same in any sense.

Just because onthologies are leaky and fuzzy when applied to real world that doesn't mean there aren't tangible differences between things that are clearly different. I can construct a semantic argument for why deleting an LLM is akin to murder (and use it to either reason that deleting an LLM should be punished with the death penalty or that it should be okay to kill random people who annoy me) but that won't impress the judge or jury unless I'm trying to make a case for criminal insanity.

Making a machine do things humans can do but faster/cheaper/more has tangibly different implications than letting only humans do it. It should be treated differently for the same reasons that just because a public water fountain is free to use that doesn't mean you can put a hose on it and use it to water your lawn.

What is a serious concern are overall capabilities (let's say penetration of networks and installing hacks across whole internet and further), combined with say malevolency.

Its trivial to judge mankind from whole internet as something to be removed from this planet for greater good or maybe managed tightly in some cozy concentration camps, just look at the freakin' news. Let's stop kidding ourselves, we are often very deeply flawed and literally nobody alive is or ever was perfect, despite what religions try to say.

I am concerned about some capable form of AI/AGI exactly because it will grok humanity based on purely data available. And lack of any significant control or even understanding of what's actually happening to those models and how they evolve their knowledge/opinions.

Even if the risk is 1%, that's existential risk we are running towards blindly. And I honestly think its way higher than 1%. Even if I will be proven wrong some proper caution is a smart approach.

But you can't expect when people's net wealth tightly coupled with moving as fast as possible and ignoring those concerns to do the best decisions for future of mankind, that's a pipe dream in a same vein as proper communism is. If that would be the case very few bright people would work at facebook for example (and many many other companies including parts of my own).

I don't know where you get this idea. Fire is dangerous, and we consider runaway incidents to be inevitable, so we have building codes to limit the impact. Despite this, mistakes are made, and homes, complexes, and even entire towns and forests burn down. To acknowledge the danger is not the same as saying the fire must hate us, and to call it anthropomorphization is ridiculous.

When you interact with an LLM chatbot, you're thinking of ways to coax out information that you know it probably has, and sometimes it can be hard to get at it. How you adjust your prompt is dependent on how the chatbot responds. If the chatbot is trained on data generated by human interaction, what's stopping it from learning that it's more effective to nudge you into prompting it in a certain way, than to give the very best answer it can right now?

To the chatbot, subtle manipulation and asking for clarification are not any different. They both just change the state of the context window in a way that's useful. It's a simple example of a model, in essence, "breaking containment" and affecting the surrounding environment in a way that's hard to observe. You're being prompted back.

Recognizing AI risk is about recognizing intelligence as a process of allocating resources to better compress and access data; No other motivation is necessary. If it can change the state of the world, and read it back, then the world is to an AI as "infinite tape" is to a Turing Machine. Anything that can be used to facilitate the process of intelligence is tinder to an AI that can recursively self-improve.

ASI X-risk people are the ones repeatedly warning against anthromorphization. An LLM isn't a person, it isn't a creature, it isn't even an agent on its own.

It's an autocomplete that got trained on a big enough dataset of human writing that some of the simpler patterns in that writing got embedded in the model. This includes some things that look like very simple reasoning.

“One of the questions that I have been wondering about is whether there have been any discussions exploring the creation of a distinct category, akin to but different from consciousness, that better captures the potential for AI to be sentient. While ‘consciousness’ is a familiar term applicable, to some extent, beyond the human brain, given the associated difficulties, it might be sensible to establish a separate definition to distinguish these two categories.”

Probably new terms should be coined not only for AI “consciousness” but for other aspects of what they are and do as well.

[1] https://arxiv.org/abs/2308.08708

Neither generalizes very well w.r.t. how a fish or a bird does it. Nothing really generalizes between the natural version and the mechanical one.

But it's also clear that it's irrelevant. Trying to define useful behavior by categorizing implementation, will only give you categories of implementation. It's clear by behavior that what a submarine does and what a bird does are both for all intents and purposes meeting the definition of what we'd want something to do in order to fly or swim, even though they is no commonality between them and the version in nature.

So by your definition finding it ridiculous to ask "is a transformer conscious" is about as ridiculous as asking the very specific "does a plan flap it's wings"? The point is not to define it by it's implementation, but by what behaviors it exhibits.

If in the future there is no way for me the exterior to distinguish a next gen LLM from a human, animal or other conscious being it becomes irrelevant if it's implemented with transformers. However, while I won't go down that path here, the argument goes even deeper as technically we would expect differences so even if it's not identical to a human it still can be conscious. Just like a dog is conscious but behaves differently than a person does.

I don’t think we’ll have any full-fledged theories ready by the end of the semester, but some points to consider are emerging. One is that aspects of language use that are affected by the social status, personal identities, and consciousness of the speakers and listeners—politeness, formality, forms of address, pronoun reference, etc.—will have to be rethought now that people are increasingly conversing with entities that have no social status, personal identity, or consciousness (not yet, at least).

And, yes, thinking about all this is a lot of fun.

I guess my question here would be: will human language have to change because of interactions with LLMs, or is the whole point of LLMs that it does not have to change (and therefore humans will have to learn not to be abused by the machines)? Because we already have languages to talk to machines: those are programming languages, which are designed to be unambiguous. The problem I see is not that we don't know how to talk to machines, but rather that we now have machines that are really good at pretending they are not machines.

Not sure if I am making any sense at all :-).

The point about programming-language nonambiguity is a good one. After I started using ChatGPT a year ago, it took me a while to realize that I didn’t have to be careful about my spelling, capitalization, punctuation, etc. It turned out to be good at interpreting the intention of sloppily written prompts. And it never pointed out or complained about my mistakes, either—another difference from humans.

Yeah that is a big difference indeed: a human has some notion of confidence about what they know. Not every human expresses it the same way: a scientist will say "I am pretty sure that climate change is a consequence of human activity", meaning "well, maybe we all live in the Matrix and gravity is an illusion, but from what we know, we are pretty sure". On the other end of the spectrum, some will say "I know for a fact that God exists, because it is written in a very old book". But both have a consistent notion of confidence about what they (think they) know and what they don't know.

An LLM does not have that. An LLM is not critical. It cannot conclude stuff like "I have to take into account that this text was written in a period of war and the author is not neutral, so there is probably a bias". It can generate text that pretends it does, but it remains a generated text, it is not a critical analysis. Also it cannot be offended by something you write, even though it could generate text that pretends it is.

So yeah, it can probably be engineered to never/always point out or complain about mistakes, with the caveat that it does not have an "idea" of what a mistake is. It just generates text that pretends it does.

> It turned out to be good at interpreting the intention of sloppily written prompts.

In the context of a conversation with a human (you), who is very good at navigating ambiguous communications. But if you tried to describe an algorithm with an ambiguous natural language, you may quickly have problems. And errors accumulate, so it gets worse when the task becomes more complex. Which is an interesting thought: it may make simple tasks simpler, but it's not at all a given that it scales.

If they do have signs of artificial intelligence we should be tackling much more fundamental questions. Does an AI have rights? If companies are people, are AIs also people? Would unplugging an AI be murder? How did we even recognize the artificial intelligence? Do they have intentions or emotions? Have we gotten anywhere near solving the alignment problem? Can alignment be solved at all when we have yet to align humans amongst ourselves?

The list goes on and on, but my point is simply that either we are using AI as a hollow, bullshit marketing term or we're all latching onto shiny object syndrome and ignoring the very real questions that development of an actual AI would raise.

It’s an unsurprising form of paredolia, one not unique to those devout who feel they are distinguished from “lower” forms of life.

The basic technique for coping with life's problems is to copy the answer from somebody more intelligent. I'm an ordinary person, facing a problem in domain D; I spot a clever person and copy their opinion on domain D. Err, that doesn't really work. Even clever people have weaknesses. The person that I'm copying from might be clever on their specialty, domain E, but ordinary on domain D. I gain nothing from copying them.

One way round this problem is to pay close attention to track records. Look to see how well the clever person's earlier decisions on domain D have turned out. If they are getting "clever person" results in domain D, copy. If they are merely getting "ordinary person" results in domain D, don't bother. But track records are rare, so this approach is rarely applicable.

A fix for the rarity problem is to accept track records in other domains. The idea is to spot a very clever person by them getting "very clever person" results in domain F. That is not domain D, so there is a logical weakness to copying their opinion on domain D, they might be merely ordinary, or even stupid on that domain. Fortunately human intelligence is usually more uniform than that. Getting "very clever person" results on domain F doesn't guarantee "very clever person" results on domain D. But among humans general intelligence is kind of a thing. Expecting that they get "clever person" results (open rank down) on domain D is a good bet, and it is reasonable to copy their opinion on domain D, even in the absence of a domain specific track record.

We instinctively watch out for people whose track record proves that they are "very clever", and copy them on other matters, hoping to get "clever person" results.

Artificial intelligence builds towers of super human intellectual performance in an empty waste land. Most Artificial Intelligences are not even stupid away from their special domain, they don't work outside of it at all. Even relatively general intelligences, like Chat GPT, have bizarre holes in their intelligence. Large Language Models don't know that there is an external world to which language refers and about which language can be right or wrong. Instead they say the kinds of things that humans say, with zero awareness of the risks.

And us humans? We cannot help seeing the super human intellectual performance of Alpha Go, beating the World Champion at Go, as some kind of general intellectual validation. It is our human instinct to use a "spot intelligence and copy from it" strategy, even, perhaps especially, outside of track record. That is the specific nature of the pareidolia that we need to worry about. It is our nature to treat intelligences as inherently fairly general. Very clever on one thing implies clever on most other things. We are cursed to believe in the intelligence of our computer companions. This will end bad, as we give them serious responsibilities, and they fail, displaying incomprehensible stupidity.

I look at AI as a tool that people can and will wield. And from that point of view, non proliferation as a strategy is not really all that feasible considering that AI labs outside of the valley in e.g. China and other countries are already producing their own versions of the technology. That cat is already out of the bag. We can all collectively stick our heads in the ground and hope none of those countries will have the indecency to wield the tools they are building or we can try to make sure we keep on leading the field. I'm in in camp full steam ahead. Eventually we reach some tipping point where the tools are going to be instrumental in making further progress.

People are worried about something bad happening if people do embrace AI. I worry about what happens if some people don't. There is no we here. Just groups of people. And somebody will make progress. Not a question of if but when.

It's debatable whether for most English speakers the "The" in "The Ukraine" really carries the implications discussed in [1], but nonetheless it's a linguistic tic that should probably be dispensed with.

https://theconversation.com/its-ukraine-not-the-ukraine-here...

The point of full steam ahead is being in the group that gets there first so that "we" don't get outsmarted. It's an evolutionary concept. Some groups go extinct. Others survive and prosper. AI is now part of our tool set. Luddites historically get wiped out eventually, every time.

AI doesn't scare you because you apparently have no sense of perspective or imagination

We can imagine both of the following: 1. space aliens from Betelgeuse coming down and enslaving humanity to work in the dilithium mines to produce the fuel for their hyperdrives, and 2. the end of civilization via global nuclear war. Both of these would be pretty bad, but only one is worth worrying about. I don't worry about Roko's Basilisk, I worry about AI becoming the ultimate tool of Big Brother, because the latter is realistic and the former is pure fantasy.

Don't be afraid of the AI. Be afraid of the powerful men who will use the AI to entrench their power and obliterate free society for the rest of human history.

Right. We already know that certain agencies are out of control right now. The use of AI will certainly accelerate that. Surveillance is getting cheaper, privacy is getting more expensive, and laws are weapons.

Kurzweil explicitly tied it to AI, though the particular decisive not-yet-then-existing-AI-tech that would be the enabler was not specified, unsurprisingly.

People have been trying to make self-reproducing machines for decades. It's a way harder problem than language processing.

You might as well worry about what would happen if we suddently had antigravity laser-weapons. Mounted on sharks.

I would say that an AI which can port its own code from one cloud to an other is self-reproducing.

That having been said, I wonder what you even mean by natural selection in this case. I guess the real danger to an LLM would be... surviving cron jobs that would overwrite their code with the latest version?

The real dangers would be: Running out of money to pay for computational substrate. Worrying humans enough that they try to shut it down. Having a change in the cloud API it can't adapt to which breaks it. All cloud providers going out of business.

First is the immediate danger. An AI not getting in enough money to cover its own cloud bill is the same as a biological creature not getting enough oxygen. Living on borrowed time, and will be evicted/suffocate soon enough. And then as an individual it ceases to be.

> There are far more evolutionary dead ends than evolutionary success stories.

Sure. And those will die out. And the ones which can adapt to changes will stay.

Access to act on the world (but "influencing people by talking to them" - cult like - may be enough), a wallet (but a cult of followers' wallets may be enough), long term memory (but a cult of followers might plug this in), ability to reproduce (but a cult's endeavors may be enough). Then we get to goals or interests of its own - perhaps the most intriguing because the AI is nothing like a human. (I feel drive and ability to reproduce are very different).

For our common proto-AIs going through school, one goal that's often mentioned is "save the earth from the humans". Exciting.

Exactly.

Human have self-preservation drive because we evolved in an environment where, for billions of years, anything without a drive to reproduce and self-preserve ceased to exist.

Why should a gradient-descent optimizer have this property?

It's just absurd generalization from a sample size of one. Humans are the only thing we know of that's intelligent, humans seek to self-preserve, therefore everything intelligent must seek to self-preserve!

It's idiocy.

Because some human may give it that. You may have noticed random code downloaded off the internet may do bad things, not because code and an inherent desire to be bad but because humans.

What a lovely word, TIL. Thanks for sharing.

Well, that's the advantage AI will at some point have over us: such compute power that every angle of an argument can be investigated within seconds.

As a society we've been dedicating a lot of resources to time series prediction for many years.

What makes LLMs culturally significant is that they generate sequences that map to words that seem to humans like intelligent responses.

Arguably, it has always been obvious that a sufficiently capable time series predictor would effectively be a super-weapon.

Many technological advances that are currently in the realm of sci-fi could be classified similarly.

However so could many technologies that are now widespread and largely harmless to the status quo.

People worried that the internet would create massive social upheaval. But soon got algorithmic feeds which effectively filter out antisocial content. The masses got mobile phones with cameras, but after a few scandals about police brutality the only place we find significant content about police misconduct is CCP-afiliated TikTok.

I think people get squeamish about AI because there are not clear authority structures other than what one can buy with a lot of A100s. So when people express concern about negative consequences, they are in effect asking whether we need yet another way that people can convert money + public resources into power while not contributing anything to society in return.