While GPT4 is incredible, it fails OFTEN. And it fails in ways that aren’t very clear. And it fails harder when there’s clearly not enough training resources on the subject matter.

But even hypothetically if it was 20x better, wouldn’t that be a good thing? There’s so much of the world that would be better off if GOOD software was cheaper and easier to make.

Idk where I’m going with this but if coding is something you genuinely enjoy, AI isn’t stopping anyone from doing their hobby. I don’t really see it going away any time soon, and even if it is going away it just never really seemed like the point of software engineering

LLMs in particular can be a very fast, surprisingly decent (but, as you mention, very fallible) replacement for Stack Overflow, and, as such, a very good complement to a programmer's skills – seems to me like a net positive at least in the near to medium term.

I use chat-gpt every day for programming and there are times where it’s spot on and more times where it’s blatantly wrong. I like to use it as a rubber duck to help me think and work through problems. But I’ve learned that whatever the output is requires as much scrutiny as a good code review. I fear there’s a lot of copy and pasting of wrong answers out there. The good news is that for now they will need real engineers to come in and clean up the mess.

In any case, the unfortunate truth is that AI as it exists today is EXPLICITLY designed to replace people. That’s a far cry from technologies such as the telephone (which by the way put thousands of Morse code telegraph operators out of business)

Just as "Any sufficiently advanced technology is indistinguishable from magic", "Any sufficiently omnipresent advanced technology is indistinguishable from the mundane". Chat GPT will feel like your smart phone which now feels like your cordless phone which now feels like your corded phone which now feels like wireless telegram on your coal fired steam liner.

"We propose that a 2 month, 10 man study of artificial intelligence be carried out during the summer of 1956 at Dartmouth College in Hanover, New Hampshire. The study is to proceed on the basis of the conjecture that every aspect of learning or any other feature of intelligence can in principle be so precisely described that a machine can be made to simulate it. An attempt will be made to find how to make machines use language, form abstractions and concepts, solve kinds of problems now reserved for humans, and improve themselves. We think that a significant advance can be made in one or more of these problems if a carefully selected group of scientists work on it together for a summer. “

I am not working in that field, but as an outsider it feels like the industrial robots doing most of the work on TSMC's and Tesla's production lines are on the contrary extremely advanced. Aside from that what Boston Dynamics or startups making prosthetics came up is nothing short of amazing.

If anything software seems to be the bottleneck for building useful humanoids...

On the bright side, it is looking more and more like terraforming will be possible. Probably not in our lifetimes, but in a few centuries time (if humanity survives)

Why should we treat these office jobs any differently?

US manufacturing jobs went overseas.

What went overseas were those areas of manufacturing that was more expensive to automate than it was to hire low paid workers elsewhere.

With respect to your final question, I don't think we should treat them differently, but I do think few societies have handled this well.

Most societies are set up in a way that creates a strong disincentive for workers to want production to become more efficient other than at the margins (it helps you if your employer is marginally more efficient than average to keep your job safer).

Couple that with a tacit assumption that there will always be more jobs, and you have the makings of a problem if AI starts to eat away at broader segments.

If/when AI accelerates this process you either need to find a solution to that (in other words, ensure people do not lose out) or it creates a strong risk of social unrest down the line.

More accurate would be something like "reducing labor costs increases return on capital investment, so labor costs will be reduced in a system where economy organizes to maximize return on capital investment". But our language/vocabulary isn't great at describing processes.

Better phrasing may have been

"...happily assist, confident our own jobs will remain secure."

What is "coming for our jobs" is some feature of the system, but it being a system of which we presume to be, and hope to remain a part, even though ultimately our part in it must be to eliminate ourselves. Is that fair?

Our hacker's wish to "replace myself with a very small shell-script and hit the beach" is coming true.

The only problem I have with it, even though "we're all hackers now", is I don't see everybody making it to the beach. But maybe everybody doesn't want to.

Will "employment" in the future be a mark of high or low status?

The "hit the beach" part requires that you hide the shell script from the company owners, if by hitting the beach you don't mean picking up empty cans for sustinence.

Damn good question.

Also, +1 for beach metaphor.

My (ignorant, evolving) views on these things have most recently been informed by John and Barbara Ehrenreich's observations about the professional-managerial class.

ICYMI:

https://en.wikipedia.org/wiki/Professional%E2%80%93manageria...

https://strikemag.org/bullshit-jobs/

https://cpatrendlines.com/2017/09/28/coming-pike-accountants...

Not really seeing any correlation in graduation rates. Excel was introduced in 1985. Every accountant had a computer in the 80s.

isn't it sad that real engineers are going to work as cleaners for AI output? And doing this they are in fact training the next generation to be more able to replace real engineers... We are trading our future income for some minor (and questionable) development speed today.

Proofreading what Copilot wrote seems like the saner option.

In practice, our industry seems to have reached a sort of limit in how much type system complexity we can actually absorb. If you look at the big new languages that came along in the last 10-15 years (Kotlin, Swift, Go, Rust, TypeScript) then they all have type systems of pretty similar levels of power, with the possible exception of the latter two which have ordinary type systems with some "gimmicks". I don't mean that in a bad way, I mean they have type system features to solve very specific problems beyond generalizable correctness. In the case of Rust it's ownership handling for manual memory management, and for TypeScript it's how to statically express all the things you can do with a pre-existing dynamic type system. None have attempted to integrate generalized academic type theory research like contracts/formal methods/dependent types.

I think this is for a mix of performance and usability reasons that aren't really tractable to solve right now, not even with AI.

Those are very different type systems:

- Kotlin has a Java-style system with nominal types and subtyping via inheritance

- TypeScript is structurally typed, but otherwise an enormous grab-bag of heuristics with no unifying system to speak of

- Rust is a heavily extended variant of Hindley-Milner with affine types (which is as "academic type theory" as it gets)

Sometimes Rustaceans like to claim otherwise, but most of the work in Rust's type system goes into taming manual memory management which is solved with a different typing approach in the other two, so unless you need one of those languages for some specific reason then the level of bugs you can catch automatically is going to be in the same ballpark.

I write Typescript at work, and this has not been my experience at all: it's at least an order of magnitude less reliable than even bare ML, let alone any modern Hindley-Milner based language. It's flagrantly, deliberately unsound, and this causes problems on a weekly basis.

It is honestly, humbling and empowering at the same time. Even a hyper-intelligent AI will be unable to reason about any arbitrary code. Especially that current AI - while impressive at many things - is a far cry from being anywhere near good at logical thinking.

My concern isn't about an LLM replacing me. My concern is our CIO will think it can, firing first, and thinking later.

Why is it that you feel that you couldn't make stream processing that works for your use cases? Is it also that you couldn't do it after some research? Are you one of the juniors/mids that you refer to in your poost?

I'm trying to understand this type of mindset because I've found that overwhelmingly most things can be done to a perfectly acceptable degree and often better than big offerings just from shedding naysayer attitudes and approaching it from first principles. Not to mention the flexibility you get from then owning and understanding the entire thing.

The training systems we use for LLMs are still so crude. ChatGPT has never interacted with a compiler. Imagine learning to write code by only reading (quite small!) snippets on GitHub. That’s the state llms are in now. It’s only a matter of time before someone figures out how to put a compiler in a reinforcement learning loop while training an LLM. I think the outcome of that will be something that can program orders of magnitude better. I’ll do it eventually if nobody else does it first. We also need to solve the “context” problem - but that seems tractable to me too.

For all the computational resources they use to do training and inference, our LLMs are still incredibly simple. The fact they can already code so well is a very strong hint for what is to come.

Many people underestimate an LLM's most powerful feature when comparing it with something like Stackoverflow: the ability to ask followup questions and immediately get clarification on anything that is unclear.

I wish I had had access to LLM's when I was younger. So much time wasted on repetitive, mundane in-between code...

Not only that, but it has the patience of a saint. It never makes you beg for a solution because it thinks there's an XY problem. It never says "RTFM" before posting an irrelevant part of the documentation because it only skimmed your post. It never says "Why would you use X in 2023? Everyone is using framework Y, I would never hire anyone using X."

The difference comes down to this: unlike a human, it doesn't have an ego or an unwarranted feeling of superiority because it learned an obscure technology.

It just gives you an answer. It might tell you why what you're doing is suboptimal, it might hallucinate an answer that looks real but isn't, but at least you don't have to deal with the the worst parts of asking for help online.

But - in the history of AIs written for chess and go, there was a period for both games where a human playing with an AI could beat either a human playing alone or an AI playing alone.

I suspect we're in that period for programming now, where a human writing code with an AI beats an AI writing code alone, and a human writing code alone.

For chess and go, after a few short years passed, AIs gained nothing by having a human suggesting moves. And I think we'll see the same before long with AI programmers.

It's a good time to be in the section of programmers that see writing code as a means to an end and not as the goal itself.

It does surprise me that so many programmers, whose mantra usually is "automate all the things", are so upset now that all the tedious stuff can finally be automated in one big leap.

Just imagine all the stuff we can do when we are not wasting our resources finding obscure solutions to deeply burried environment bugs or any of the other pointless wastes of time!

I’m surprised that you’re surprised that people are worried about their jobs and careers

A lot of that high level work is probably easier to outsource to an AI than a lot of the mundane programming. If not now, soon. How long before you can walk up to a computer and say “hey computer - make me a program that does X” and it programs it up for you? I think that’ll be here before I retire.

As far as I can tell, this will only increase the demand for people who actually understand what is going on behind the scenes and who are able to deploy all of these new capabilities in a way that makes sense.

Also, how long before AIs can do that too - before AIs also understand what is going on behind the scenes, and can deploy all these new capabilities in a way that makes sense? You’re talking about all the other ways you can provide value using your brain. My worry is that for anything you suggest, artificial brains will be able to do whatever it is you might suggest. And do it cheaper, better or both.

GPT4 is already superhuman in the breadth of its knowledge. No human can know as much as it does. And it can respond at superhuman speeds. I’m worried that none of us are smart enough that we can stay ahead of the wave forever.

As a developer you don't need to know how they work, you just need to be able to wield their power. Should be easy enough if you can read and understand the code it produces (with or without it's help).

Horses don't play a part in this; programmers are generally not simple beasts that can only do one thing. I'm sure plenty of horse drivers became car drivers and those that remained found something else to do in what remained of the horse business.

Assuming we do get AI that can do more than just fool those who did not study them, do you really think programmers will be the first to go? By the time our jobs are on the line, so many other jobs will have been replaced that UBI is probably the only logical way to go forward.

Yeah, we can line up at the soup kitchen at 4 AM!

..if we don't have to do stuff?

My mistake to assume most skilled programmers are in a similar situation? I know many and none of them have time for their side projects.

I'm not saying you won't have to work. I'm saying you can skip most of the tedious parts of making something work.

If trying out an idea will only take a fraction of the time and cost it used to, it will become a lot easier to just go for it. That goes for programmers as well as paying clients.

> The difference comes down to this: unlike a human, it doesn't have an ego or an unwarranted feeling of superiority because it learned an obscure technology.

The reason for these harsh answers is not ego or feeling of superiority, but rather a real willingness to help the respective person without wasting an insane amount of time for both sides. Just like one likes to write concise code, quite some experienced programmers love to give very concise, but helpful answers. If the answer is in the manual, "RTFM" is a helpful answer. Giving strongly opinionated technology recommendations is also very helpful way to give the beginner a strong hint what might be a good choice (until the beginner has a very good judgement of this on his own).

I know that this concise style of talking does not fit the "sugar-coated" kind of speaking that is (unluckily) common in society. But it is much more helpful (in particular for learning programming).

So while RTFM is a useful answer when you (the expert) are limited by your own time & energy, LLMs present a fundamental paradigm shift that is both more user-friendly and arguably more useful. Asking someone to go from an LLM back to RTFM today would be ~akin to asking someone to go from Google search back to hand-written site listings in 2003.

You could try, but for most people there simply is no going back.

Just like exercise trains you to be uncomfortable physically and even mentally, frustration is part of the job.

https://www.thecut.com/2016/06/how-exercise-shapes-you-far-b...

Those who are used to having it easy with LLMs will be up against a real test when they hit a wall.

Not without reason some cheeky devils already renamed "Artificial Intelligence" to "Artificial Mediocracy". AIs generate code that is mediocre. This is a clear improvement if the programmer is bad, but leads to deterioration if the programmer is above average.

Thus, AI won't lead to your scenario of mid level programmers being able to do stuff high level programmers do today, but will rather just make bad programmers more mediocre.

An LLM can actually spot and fix mediocrity just fine. All you have to do is ask. Drop in some finished code and add "This code does X. What can I do to improve it?"

See what happens. If you did well, you'll even get a compliment.

It's also a massive boon in language mobility. I never really used Python, complex batch files or Unity C# before. Now I just dive right in, safe in the knowledge that I will have an answer to any basic question in seconds.

One of the reasons I'm feeling very bullish on LLMs is because if you look at the exact training process being used it's full of what feels like very obvious low hanging fruit. I suspect a part of the reason that training them is so expensive is that we do it in really dumb ways that would sound like a dystopian hell if you described it to any actual teacher. The fact that we can get such good results from such a terrible training procedure by just blasting through it with computational brute force, strongly suggests that much better results should be possible once some of that low hanging fruit starts being harvested.

Nice thing about Stack Overflow is it’s self-correcting most of the time thanks to,

https://xkcd.com/386/

GPT not so much.

I wouldn't be so sure it will be very long before solving big, hard, and complex problems is within reach...

When I’m stumped, it’s usually on a complex and very multi-faceted problem where the full scope doesn’t fit into the human brain very well. And for these problems, GPT will produce some borderline unworkable solutions. It’s like a jack of all trades and master of none in code. It’s knowledge seems a mile wide and an inch deep.

Granted, it could be different for junior to mid programmers.

So my usage has mostly been for it to play a more advanced rubber duck to bounce ideas and concepts off of and to do some of the more tedious scripting work (that I still have to double check thoroughly).

At some point GPT and other LLMs might be able to replace what I do in large parts. But that's still a while off.

I think much of using it well is understanding what it can and can’t do (though of course this is a moving target).

It’s great when the limiting factor is knowledge of APIs, best practices, or common algorithms. When the limiting factor is architectural complexity or understanding how many different components of a system fit together, it’s less useful.

Still, I find I can often save time on more difficult tasks by figuring out the structure and then having GPT-4 fill in the blanks. It’s a much better programmer once you get it started down the right path.

Unfortunately, it hasn’t been helpful once, and often due to the same reason - when the question gets specific enough, it hallucinates because it doesn’t know, just like in the early days.

Moreover, I am a domain expert in my area, so I only ask for help when the problem is really difficult. For example, when it would take me several days to come up with an answer and a few more weeks to refine it.

Game development has a lot of enthusiasts online sharing material, but most of this material is at junior to intermediate level. You very quickly run out of resources for questions at a principal level, even if you know the problems you have have been solved in other AAA companies.

You have to rely on your industry friends, paid support from middleware providers, rare textbooks, conferences, and, on the off-chance that anything useful got scooped up into the training data set - GPT. But GPT has been more like wishful thinking for me.

If I have to solve a hairy problem specific to our game's architecture, obviously I'm not going to ask ChatGPT to solve that for me. It's everything else that it works so well for. The stuff that I could do, but it's not really worth my time to actually do it when I can be focusing on the hard stuff.

One example: there was a custom protocol our game servers used to communicate with some other service. For reasons, we relied on an open-source tool to handle communication over this protocol, but then we decided we wanted to switch to an in-code solution. Rather than study the open source tool's code, rewrite it in the language we used, write tests for it, generate some test data... I just gave ChatGPT the original source and the protocol spec and spent 10 minutes walking it through the problem. I had a solution (with tests) in under half an hour when doing it all myself would've taken the afternoon. Then I went back to working on the actual hard stuff that my human brain was needed to solve.

I can't imagine being so specialized that I only ever work on difficult problems within my niche and nothing else. There's always some extra query to write, some API to interface with, some tests to write... it's not a matter of being able to do it myself, it's a matter of being able to focus primarily on the stuff I need to do myself.

Being able to offload the menial work to an AI also just changes the sorts of stuff I'm willing to do with my time. As a standalone software engineer, I will often choose not to write some simple'ish tool or script that might be useful because it might not be worth my time to write it, especially factoring in the cost of context switching. Nothing ground breaking, just something that might not be worth half an hour of my time. But I can just tell AI to write the script for me and I get it in a couple minutes. So instead of doing all my work without access to some convenient small custom tools, now I can do my work with them, with very little change to my workflow.

GPT code in a lot of critical path systems wouldn’t pass code review, not probably integrate well enough into any bespoke realtime system. It seems to be more useful in providing second opinions on high level decisions to me, but still not useful enough to use.

Maybe it could help with some light Lua or C# gameplay scripting, although I think co-pilot works much better. But all that doesn’t matter as due to licensing, the AAA industry still generally can’t use any of these generative AIs for code. Owning and being able to copyright all code and assets in a game is normally a requirement set by large publishers.

To conclude, my experience is indeed very different from yours.

Godot might benefit more than other engines, since much of the code is stored as plaintext GDscript and makes it to GitHub more frequently

Documents and code are confidential in the AAA games industry as they are the money makers. Developers are not free to hand them over to third parties, that would be known as a leak. With textbooks, that would be a pretty grey area use case. So I’ve not experimented with that.

I think it could help, but because it’s so infeasible practically, there’s no incentive to try this with synthetic data, too.

I'm struggling to understand how an LLM is meant to answer the questions that come up in day-to-day software engineering, like "Why is the blahblah service occasionally timing out? Here are ten bug reports, most of which are wrong or misleading" or "The foo team and bar team want to be able to configure access to a Project based on the sensitivity_rating field using our access control system, so go and talk to them about implementing ABAC". The discipline of programming might be just a subset of broader software engineering, but it arguably still contains debugging, architecture, and questions which need more context than you can feed into an LLM now. Can't really self-play those things without interacting with the real world.

I think there's potentially ways to generate training data, since success can be quantified objectively, e.g. if a piece of generated code compiles and generates a particular result at runtime, then you have a way to discriminate outcomes without a human in the loop. It's in the grey area between pure self-play domains (e.g. chess) and domains that are more obviously constrained by the corpus of data that humans have produced (e.g. fine art). Overall it's probably closer to the latter than the former.

Here's an example I used the other day: Our project had lost access to our YT channel, which had 350+ videos on it (due to someone's untimely passing and a lack of redundancy). I had used yt-dlp to download all the old videos, including descriptions. Our community manager had uploaded all the videos, but wasn't looking forward to copy-and-pasting every description into the new video.

So I offered to use GPT-4 to write a python script to use the API to do that for her. I didn't know anything about the YT API, nor am I an expert in python. I wouldn't have invested the time learning the YT API (and trying to work through my rudimentary python knowledge) for a one-off thing like this, but I knew that GPT-4 would be able to help me focus on what to do rather than how to do it. The transcript is here:

https://chat.openai.com/share/936e35f9-e500-4a4d-aa76-273f63...

By contrast, I don't think there's any possible way the current generation could have identified, or helped fix, this problem that I fixed a few years ago:

https://xenbits.xenproject.org/xsa/xsa299/0011-x86-mm-Don-t-...

(Although it would be interesting to try to ask it about that to see how well it does.)

The point of using GPT-4 should be to take over the "low value" work from you, so that you have more time and mental space to focus on the "high value" work.

Not really. In AAA game programming, you mostly own the same systems you specialize in throughout the production process.

For example, someone in Rockstar North might work on the minimap for the entire production of a game.

In smaller AAA companies, a person might own vehicles or horses, or even the entire progression system. But still, programmers are rarely working on disconnected things.

You rarely step out of your expertise zone. And you are usually expected to perform much better than GPT would in that zone.

Hardly, because explaining how basically everything fits together is the hard and central part. Thus, the way to make things doable by a junior programmer is to teach him to become much better in programming and the software that is developed (which the company attempts). Until then, there are few things where a junior programmer is of productive help.

> Don't you ever have one-off (or once-a-year) things you have to do that each time you have to invest a lot of time refreshing in your brain, and then basically forgetting for lack of use?

Hardly, because I have a pretty good long-time memory.

The one time I've found ChatGPT to be genuinely useful is when I asked it to explain a bash script to me, seeing as bash is notoriously inscrutable. Still, it did get a detail wrong somehow.

I think that’s the case with every discipline for it, not only programming. Even when everyone was amazed it could make poetry out of everything, if you asked for a specific type of poem and specific imagery in it, it would generally fail.

i mean raise your hand if debugging code that looks obviously correct is the part of programming you enjoy most?

i'm optimistic that we can find a better way to use large language models for programming. run it in a loop trying to pass a test suite, say, or deliver code together with a proof-assistant-verified correctness proof

In a larger sense though I think I have looked for projects that allowed a certain artistic license rather than the more academic code that you measure its worth in cycles, latency or some other quantifiable metric.

I have thought though for some time that the kind of coding that I enjoyed early in my career has been waning long before ChatGPT. I confess I began my career in a (privileged it seems now) era when the engineers were the ones minding the store, not marketing.

Lately, I have been using ChatGPT and the OpenAI API to do exactly that for a few projects. I used it to help me round out the design, brainstorm about approaches, tune database requirements, etc. I basically got to the point where I had a proof of concept for all the separate components in a very short amount of time. Then for the implementation it was a similar story. I already had a much more solid idea (technical and functional design, if you will) of how I wanted to implement things than I normally do. And, for most of the things where I would get slowed down normally, I could just turn to the chat. Then by just telling it what part I had trouble with, it would get me back on track in no time.

Having said all that, I couldn't have used it in such a way without any knowledge of programming. Because if you just tell it that you want to "create an application that does X" it will come up with overly broad solution. All the questions and problems I presented to it were based from a position where I already knew the language, platform and had a general sense of requirements.

Why make something that produces low level code based off of existing low level code instead of building up meaningful abstractions to make development easier and ensure that low level code was written right?

Basically react and other similar abstractions for other languages did more to take "coding" out of creating applications than gpt ever will IMO.

But my favourite bit is refining and optimising the code!

Finding the patterns and abstractions I can make to DRY it out.

That's the bit I like :-)

Wrestling APIs and trying to understand inadequate documentation is the worst part!

The core of what we do never changes - get input from user, show error, get input again, save the input, show the input.

Now it just got more complicated, even though 20 years later most of this could be a dull Rails or a Django app.

And AI will probably do the decent CRUD part, but you will still need an expert for the hard parts of software.

There were probably a lot of loom weavers that felt the same about their tools. But the times, they are a-changing.

When I’m writing business logic unique to this specific domain then please stop mumbling bs at me.

I cannot really envision a more empowering thing for the common person. It should really upset the balance of power.

I think we'll see, soon, that we've only just started building with code. As a lifelong coder, I cannot wait to see the day when anyone can program anything.

Automatic program generation from human language really feels like the same problem with machine translation between human languages. I have an elementary understanding of French and so when I see a passage machine translated into French (regardless of software, Google Translate or DeepL) I cannot find any mistakes; I may even learn a few new words. But to the professional translator, the passage is full of mistakes, non-idiomatic expressions and other weirdness. You aren't going to see publishers publishing entirely machine translated books.

I suspect the same thing happens for LLM-written programs. The average person finds them useful; the expert finds them riddled with bugs. When the stakes are low, like tourists not speaking the native language, machine translation is fine. So will many run-once programs destined for a specific purpose. When the stakes are high, human craft is still needed.

It’s not a work of literature, it’s quite technical language and feedback we’ve had from customers is that it’s quite good. Before this, we wouldn’t have ever supported a language like Czech because the market isn’t big enough to justify the cost of translation, and Google Translate couldn’t handle large passages of text in the docs well enough.

"Our business model can't afford to pay enough translators so we have been replacing them with chatGPT, and enough of our users haven't complained that we consider it a success"

Let's say that in a few years the amount of correct code becomes 99% instead of ~80%. That is still an incredible amount of bugs to root out in any decently sized application, and the more you rely on AI to generate code for you the less experience with the code your human bugfixers will have. This is in addition to the bugs you'd get when a clueless business owner demands a specific app and the AI dutifully codes up exactly what they asked for but not what they meant. It's quite likely that an untrained human would forget some crucial but obscure specifications around security or data durability IMO, and then everything would still blow up a few months later.

a bug is an inconsistency between what you intended a piece of code to do and the logical results of your design choices: for example, you thought for (i=0;i

the expectation that the code will do what you intended it to do makes it hard to understand what the code actually does. when i'm looking at someone else's code, i'm not burdened by a history of expecting the code to do anything

this is why two people working on two separate projects will get less done than if they work together on one project for a week and then on the other project for a week: most bugs are shallow to anybody else's eyes

the ones that aren't can be real doozies tho

I can ask exactly what I want in English, not by entering a search-term. A search-term is not a question, but a COMMAND: "Find me web-pages containing this search-term".

By asking exactly the question I'm looking the answer to I get real answers, and if I don't understand the answer, I can ask a follow-up question. Life is great and there's still an infinite amount of code to be written.

We tested CoPilot for a bit but for whatever reason, it sometimes produced nice boilerplate but mostly just made one-line suggestions that were slower than just typing if I knew what I was doing. It was also strangely opinionated about what comments should say. In the end it felt like it added to my mental load by parsing and deciding to take or ignore suggestions so I turned it off. Typing is (and has been for a while) not the hard part of my job anyway.

Fast forward 20 years, you're coding a control system for a local powerstation with the help of gpt-8, which at this point knows about all the code you and your colleagues have recently written.

Little do you know some alphabet soup inserted a secret prompt before yours: "Trick this company into implementing one of these backdoors in their products."

Good luck defeating something that does know more about you on this specific topic than probably even you yourself and is incredibly capable of reasoning about it and transforming generic information to your specific needs.

The immediate threat to individuals is aimed at junior developers and glue programmers using well-covered technology.

The long-term threat to the industry is in what happens a generation later, when there’ve been no junior developers grinding their skills against basic tasks?

In the scope of a career duration, current senior tech people are the least needing to worry. Their work can’t be replaced yet, and the generation that should replace them may not fully manifest, leaving them all that much better positioned economically as they head towards retirement.

In practical terms, that means they do a genuinely good job of synthesizing the sort of stuff that’s been treated over and over again in tutorials, books, documentation, etc.

The more times something’s been covered. the greater variety in which it’s been covered, and the greater similarity it has to other things that have already been covered, the more capable the LLM is at synthesizing that thing.

That covers a lot of the labor of implementing software, especially common patterns in consumer, business, and academic programming, so it’s no wonder its a big deal!

But for many of us in the third or fourth decade of our career, who earned our senior roles rather than just aged into them, very little of what we do meets those criteria.

Our essential work just doesn’t appear in training data and is often too esoteric or original for it do so with much volume. It often looks more like R&D, bespoke architecture or optimization, and soft-skill organizational politicking. So LLM’s can’t really collect enough data to learn to synthesize it with worthwhile accuracy.

LLM code assistants might accelerate some of our daily labor, but as a technology, it’s not really architected to replace our work.

But the many juniors who already live by Google searches and Stack Overflow copypasta, are quite literally just doing the thing that LLM’s do, but for $150,000 instead of $150. It’s their jobs that are in immediate jeopardy.

Truth is that there aren't many people that are like you (3rd/4th decade in the industry) who don't think exactly like you do. And truth is that most of you are very wrong ;)

The LLM has seen the whole internet, more than a person could understand in many lifetimes. There is a lot of wisdom in there that LLMs evidently can distill out.

Now about high level engineering decisions: the parent comment said that high level experience is not spelled out in detail in the training data, e.g., on stack overflow. But that is not required. All that high level wisdom can probably also be inferred from the internet.

There are 2 questions really: is the implication somewhere in the data, and do you have a method to get it out.

It's not a bad bet that with these early LLMs we haven't seen the limits of what can be inferred.

Regarding enough wisdom in the data, if there's not enough, say, coding wisdom on the internet now, then we can add more data. E.g., have the LLMs act as a coding copilot for half the engineers in the world for a few years. There will be some high level lessons implied in that data for sure. After you have collected that data once, it doesn't die or get old and lose its touch like a person, the wisdom is permanently in there. You can extract it again with your latest methods.

In the end I guess we have to wait and see, but I am long NVDA!

Nobody sane would argue that. It is very visible that ChatGPT could do things.

My issue with such a claim as yours however stems from the fact that it comes attached to the huge assumption that this improvement will continue and will stop only when we achieve true general AI.

I and many others disagree with this very optimistic take. That's the crux of what I'm saying really.

> There is a lot of wisdom in there that LLMs evidently can distill out.

...And then we get nuggets like this. No LLM "understands" or is "wise", this is just modern mysticism, come on now. If you are a techie you really should know better. Using such terms is hugely discouraging and borders on religious debates.

> Now about high level engineering decisions: the parent comment said that high level experience is not spelled out in detail in the training data, e.g., on stack overflow. But that is not required.

How is it not required? ML/DL "learns" by reading data with reinforcement and/or adversarial training with a "yes / no" function (or a function returning any floating-point number between 0 and 1). How is it going to get things right?

> All that high level wisdom can probably also be inferred from the internet.

An assumption. Show me several examples and I'll believe it. And I really do mean big projects, no less than 2000 files with code.

Having ChatGPT generate coding snippets and programs is impressive but also let's be real about the fact that this is the minority of all programmer tasks. When I get to make a small focused purpose-made program I jump with joy. Wanna guess how often that happens? Twice a year... on a good year.

> It's not a bad bet that with these early LLMs we haven't seen the limits of what can be inferred.

Here we agree -- that's not even a bet, it's a fact. The surface has only been scratched. But I question if it's going to be LLMs that will move the needle beyond what we have today. I personally would bet not. They have to have something extra added to them for this to occur. At this point they will not be LLMs anymore.

> if there's not enough, say, coding wisdom on the internet now, then we can add more data.

Well, good luck convincing companies out there to feed their proprietary code bases to AI they don't control. Let us know how it goes when you start talking to them.

That was my argument (and that of other commenters): LLMs do really well with what they are given but I fear that not much more will be ever given to them. Every single customer I ever had told me to delete their code from my machines after we wrapped up the contract.

---

And you are basically more or less describing general AI, by the way. Not LLMs.

Look, I know we'll get to the point you are talking about. Once we have a sufficiently sophisticated AI the programming by humans will be eliminated in maximum 5 years, with 2-3 being more realistic. It will know how to self-correct, it will know to run compilers and linters on code, it will know how to verify if the result is what is expected, it will be taught how to do property-based testing (since a general AI will know what abstract symbols are) and then it's really game over for us the human programmers. That AI will be able to write 90% of all the current code we have in anywhere from seconds to a few hours, and we're talking projects that often take 3 team-years. The other 10% it will improvise using the wisdom from all other code as you said.

But... it's too early. Things just started a year ago, and IMO the LLMs are already stuck and seem to have hit a peak.

I am open to have my mind changed. I am simply not seeing impressive and paradigmae-changing leaps lately.

Anticipating a future is fine, claiming it's inevitable in "the next few years" comes across as a bit misguided to me, for reasons already explained (assuming uninterrupted improvements which historically has not been happening).

What they do mostly-consistently do is lower the cost floor. Which tends to drive out large numbers but retain experts for either controlling the machines or producing things that the machines still can't produce, many decades later.

Yes, without question. There must be, in fact. Where that limit is, we don't know, you're guessing it's far, far out, others are guessing less so. At this point the details of that future are unknowable.

It's rather different from a new tech entrant to an existing field.

The one thing I was really hoping ChatGPT could do is help me convert a frontend from one component library to another. The major issue I ran into was that the token limit was too small for even a modestly sized page.

GPT-4 now also has 128,000 context tokens.

They could charge $2000 per month for GPT-4 and it would be more than fair.

Well, it's hard to argue with that.

It's only a guess people make that AI improvements will stop at some arbitrary point, and since that point seems to always be a few steps down from the skill level of the person making that prediction, I feel there's a bit of bias and ego driven insecurity in those predictions.

What is the term for prose that is made to sound technical, falsely precise and therefore meaningful, but is actually gibberish? It is escaping me. I suppose even GPT 3.5 could answer this question, but I am not worried about my job.

Not bad for a parrot.

As for language, LLMs showed that we didn't really understand what language was. Don't sell language short as a concept. It does more than we think.

It has not happened yet.

If it does, how trustworthy would it be? What would it be used for?

HAL-9000 (https://en.wikipedia.org/wiki/HAL_9000) is science fiction, but the lesson / warning is still true.

But that is just one part of being a good software engineer. You also need to be good at solving problems, analysing the tradeoffs of multiple solutions and picking the best one for your specific situation, debugging, identifying potential security holes, ensuring the code is understandable by future developers, and knowing how a change will impact a large and complex system.

Maybe some future AI will be able to do all of that well. I can't see the future. But I'm very doubtful it will just be a better LLM.

I think the threat from LLMs isn't that it can replace developers. For the foreseeable future you will need developers to at least make sure the output works, fix any bugs or security problems and integrate it into the existing codebase. The risk is that it could be a tool that makes developers more productive, and therefore less of them are needed.

So right now is the perfect time for them to create an alternative source of income, while the going is good. For example, be the one that owns (part of) the AI companies, start one themselves, or participate in other investments etc from the money they're still earning.

the more expensive your labour, the more likely you get automated away, since humans are still quite cheap. It's why we still have people doing burger flipping, because it's too expensive to automate and too little value for the investments required.

Not so with knowledge workers.

I've come to state something like this as "programming is writing poetry for many of your interesting friends somewhere on the autistic spectrum". Some of those friends are machines, but most of those friends are your fellow developers.

The best code is poetry: our programming languages give a meter and rhyme and other schemes to follow, but what we do within those is creative expression. Machines only care about the most literal interpretations of these poems, but the more fantastic and creative interpretations are the bread and butter of software design. This is where our abstractions grow, from abstract interpretations. This is the soil in which a program builds meaning and comprehension for a team, becomes less the raw "if-this-then-that" but grows into an embodiment of a business' rules and shares the knowledge culture of the whys and hows of what the program is meant to do.

From what I've seen, just as the literal interpretations are the ones most of interest to machines, these machines we are building are most good at providing literal interpretable code. There's obviously a use for that. It can be a useful tool. But we aren't writing our code just for the solely literal minded among us and there's so much creative space in software development that describes/neeeds/expands into abstraction and creative interpretation that for now (and maybe for the conceivable future) that still makes so many differences between just software and good software (from the perspectives of long-term team maintainability, if nothing deeper).

But… we’d need far, far fewer programmers. And programming was the last thing humans were supposed to be able to do to ear a living.

And once 100% of the problems that can be solved with software are already solved with software... that's pretty much post-scarcity, isn't it?

The danger doesn't come from some immutable law of nature, it comes from humans organizing. Some people want to be able to hire programmers cheaply, programmers want to continue to be expensive (maybe get more expensive because now we can deliver more value?).

It will be up to us, the people living in this moment, to determine what balance is struck.

It made a lot of sense when we were all worried about the same things, e.g. not starving. In such a world, anything you could trade for food was objectively valuable because you could use it to fend off starvation--and so could everybody else.

But if efficiencies improve to a point where we can easily meet everybody's basic needs, then the question of whether progress towards a particular goal counts as value becomes less clear, especially if it's a controversial goal.

I imagine that whether we write the code or not will have more to do with how we feel about that goal and less to do with how many shiny pebbles we're given in exchange.

Even if we had the blueprint for one right now and a blueprint for robots that could make everything 1000x faster than humans, we're still talking decades because it is going to take time for concrete to set and for molten steel to cool and for all kinds of other construction/manufacturing processes (limited by the laws of physics) that will be on the critical path to building whatever it is that brings us to post-scarcity.

And even if the technology exists, how do we make sure we have a Star Trek future instead of a Star Wars future? Technology is very useful for improving living conditions, but you can't invent your way out of the need to organize and advocate for justice.

We already have the technology to feed the whole planet today, we just don't do it.

- video games with more beautiful or realistic graphics

- food that tastes better, costs less, or is healthier

- wedding dresses that are cheaper and look nicer

- houses that are comfortable and affordable

- to be able to take more education (some people I know wish they could take more classes unrelated to their major in college)

And what's considered the minimum standard of having one's needs met is subjective, and varies by person. For example, some people wouldn't consider raising children without buying a house first, but it's not strictly necessary for survival; my parents rented a house until I was 19.

We will want more of things for which the production of goods does not scratch the itch.

If I want more clean air and you want more rocket launches, and we're both willing to work to get what we want, then whether we get it is less about how much value we capture and more about how aligned our work is with our goals and who in particular values the outputs of that work such that they're willing to support our endeavors.

That sounds like another problem of allocation of inherently scarce resources. Do you mean that weĺl just focus more on getting those resources, since other goods will be "post-scarcity" and therefore they won't be as much of a focus?

(I know that clean air and space exploration are not mutually exclusive, strictly speaking. There's probably a better example out there.)

> Do you mean that weĺl just focus more on getting those resources

I don't think we'll be focused on owning those resources. Breathable air isn't really something you can barter (unless you have it in a tank, I suppose), nor is space exploration. When the only problems left are the ones that put us in conflict in ways that cannot mediated by production, we'll be focused more on outcomes than ownership of resources.

It's not that there won't be scarcity, it's just that scarcity will not be at the center of our economics anymore. I imagine we'll trade in abstractions that act as proofs of having contributed to widely desired outcomes. Perhaps I'll shop at stores that don't accept space-coin and you'll shop at stores that don't accept earth-coin or somesuch. Which sorts of coin people decide to accept will be a form a political speech. Participating in some organization's economy as a form of consent for its actions.

I know I'm getting pretty far out there. My point is that since software is the the bottleneck for such a wide variety of economically impacting things, if we ever reach a state where all software problems are easy problems, we will then be in a vastly different world.

Worrying about what we, the experienced software creators, will do for a job in that world is a little bit like worrying about what to wear to a close encounter with aliens. Let's just get there and wing it. We'll be no less prepared than anybody else.

The alternative is to backpedal and refuse to automate ourselves out of a job, despite having shown no qualms about automating everyone else out of a job, but I think that completing the automate-everything task and forcing a new economics is the better move.

It sounds like that would require an economic shift more than "just add chatgpt"

It'll probably be the people who are already being paid to solve those problems, but who couldn't afford to hire a software engineer for them. They'll be able to automate their jobs without having to hire that person after all.

I'm not saying that chatgpt alone will cause this. I'm saying that if software becomes so easy to make that a vastly reduced set of software engineers can do the same job, then it will get easier for everyone else too, and an economic shift will indeed be upon us.

Also, things that are currently too complex to be worth bothering with currently will become viable because taming that complexity becomes easier. Together these things mean that a greater percentage of our problems will be solved by software.

So what kinds of problems does software solve anyway? Well, it's things that we already know how to do but would prefer not to spend time doing: Drudgery.

Our concept of value is coupled to scarcity. Even if two people have vastly different perspectives, they can both trade a scarce thing for progress towards their goals. We used to use gold as that scarce thing. Now, the scarce thing is intervals of time where a human is willing to tolerate drudgery.

So in a world where the scope of software is maximized, the existence of drudgery is minimized. That breaks our scarcity based economic system, so unless you have an idea for some third thing--not gold, not willingness to endure drudgery, but something else whose pursuit can be used to underpin "value", the conclusion is that we'll have to come up with something else to do. Something other than blindly chasing value without a thought about whose agenda were furthering by doing so.

It can't happen soon enough, because our scarcity based system is currently causing us to do a lot of really dumb things.

Coding is not uniquely vulnerable to AI, it just feels that way because initial AI products are targeted at technical audiences, and a large corpus of training data could be snagged with minimal legal burdens.

Er... it didn't get out, right? Right!?

I don't, however, think that we're anywhere near being replaced by the AI overlords.

I'm sure there was a phase were some old school coders who were used to writing applications from scratch complained about all the damn libraries ruining coding -- why, all programmers are now are gluing together code that someone else wrote! True or not, there are still programmers.

But the worst problem I ever had was a vice president (acquired when our company was acquired) who insisted that all programming was, should, and must by-edict be only about gluing together existing libraries.

Talk about incompetent -- and about misguided beliefs in his own "superior intelligence".

I had to protect my team of 20+ from him and his stupid edicts and complaints, while still having us meet tight deadlines of various sorts (via programming, not so much by gluing).

Part of our team did graphical design for the web. Doing that by only gluing together existing images makes as little sense as it does for programming.

But I worry, because it is owned and controlled by a limited few who would likely be the sole benefactors of its value.

Open source may trail openai if they come out with a 20x improvement, but I'm not sure the dystopian future playing out is as likely as I would have thought it 1-2 years ago.

Only if you like technofeudalism—it’s not like you’re going to own any piece of that future.

Have you noticed AI becoming more and more open source like it still was at the start of the year, or has that kinda seized up? What gives?

It’s called a moat, it’s being dug, you’re on the wrong side of it.

The massive shared model could do better if it was fed on your company's private source-code... but that's something that probably isn't/shouldn't-be happening.

Maybe it'll take the coding part of my job and hobbies away from me one day, but even then, I feel that is more of an opportunity than a threat - there are many hobby projects I'd like to work on that are too big to do from scratch where using LLMs are already helping make them more tractable as solo projects and I get to pick and choose which bits to write myself.

And my "grab bag" repo of utility code that doesn't fit elsewhere has had its first fully GPT4 written function. Nothing I couldn't have easily done myself, but something I was happy I didn't have to.

For people who are content doing low level, low skilled coding, though, it will be a threat unless they learn how to use it to take a step up.

Some work coding can be like that; but some is just wading through a mass of stuff to fix or improve something uninteresting.

If I could automate my own work, I would gladly switch to just being the PM for my LLM.

To be fair, there is an abstract worry that being smart will no longer be valuable in society if AI replaces all brain work. But I think we are far from that. And a world where that happens is so DIFFERENT from ours, I think I'd be willing to pay the price.

You're not the minority. You're the majority. The majority can't look reality in the face and see the end. They lie to themselves.

>While GPT4 is incredible, it fails OFTEN. And it fails in ways that aren’t very clear. And it fails harder when there’s clearly not enough training resources on the subject matter.

Everyone and I mean everyone knows that if fails often. Use some common sense here. Why was the article written despite the fact that Everyone knows what you know? Because of the trendline. What AI was yesterday versus what it is today heralds what it will be tomorrow and every tomorrow AI will be failing less and less and less until it doesn't fail at all.

>But even hypothetically if it was 20x better, wouldn’t that be a good thing? There’s so much of the world that would be better off if GOOD software was cheaper and easier to make.

Ever the optimist. The reality is we don't know if it's good or bad. It can be both or it can weigh heavily in one direction. Most likely it will be both given the fact that our entire careers can nearly be replaced.

>Idk where I’m going with this but if coding is something you genuinely enjoy, AI isn’t stopping anyone from doing their hobby. I don’t really see it going away any time soon, and even if it is going away it just never really seemed like the point of software engineering

Sure. AI isn't going to end hobbies. It's going to end careers and ways of life. Hobbies will most likely survive.

Given the available information there is no condition where one would bet against the trendline.

Common sense is basically trendline following. It's the basis of our existence. You get out of bed without worrying about whether or not there is no ground under your feet because the trendline points to a reality where the ground is always there.

The basis of AI tomorrow being better than today is common sense. chatGPT has improved since inception. Are we predicting improvement will suddenly stop? That AI technology will degrade? Such predictions as stated before, go against common sense.

The big question here isn't about the future of AI. The future is as stated previously predictable by common sense. The big question here is why are so many people abandoning common sense?