However at some point you have to admit the LLM does generate things that are good answers. They might be good answers that happen to pass the smell test, but they are nonetheless good answers. For instance when you ask it for a snippet of code and it gets it right.

And here is the crucial thing: you need to already know what you're doing to know whether the LLM got it right. I'm no historian, and I can ask cGPT for an essay about the causes of the Great War. When I get the answer, it sounds right to me. I don't know if the essay talks about the things an actual historian would find important, all I know is that it gives me the vanilla answer that some layman who has read a little bit would think was the right answer.

Now there's another issue this brings up. Most of us are experts in one field only. What is stopping the LLM from fooling me in every field that I don't know anything about? I best be wary of using it outside of my area of expertise.

So in the current iteration, I think LLMs are a shortcutting tool for experts. I can tell when it spits out a snippet of code that is correct, and when it's wrong. Someone who wasn't working in my domain would get fooled.

Isn't the hint in the name? It's a language model, not a knowledge model. LLMs are exceptional at generating stuff that passes as coherent language (like, the parts of speech are all where one would expect them to be in the sentences). The trouble is people think it goes deeper when the knowledge modeling part only happens incidentally because language and knowledge modeling are so closely related.

Obviously, they're better at grammar than many other things, but it's clear that they contain a vast amount of knowledge. They contain the information that the sky is blue and firetrucks are red, which isn't a matter of grammar or coherence, but is simply information about the world.

Correct. But it does mean they're about "language" rather than knowledge. It's meant to read, parse and generate language, no matter if it includes prose, facts, knowledge or whatever.

> Clearly, they aren't as likely to generate "Colorless green ideas sleep furiously" as "It's important to feed your dog".

Clearly, as the training set probably contains much more instances of the latter, than the first example. And if it's more probable to generate, that's the text it'll generate. It doesn't understand why or how, just that it's more likely, so that's the way it's going.

Not really. It was invented because they were trained on natural language corpuses. It wasn't a deep, philosophical concept.

It continues to be used for models that are doing completely non-languagy stuff, such as robotics, although I expect the growing use of 'foundation model' or some other less-confusing term will likely replace it.

    it does mean they're about "language" rather than knowledge
    

To some extent, those are kind of the same thing.

They are not the same thing. They are not even "kind of" the same thing, for any reasonable definition of "kind of." That should be obvious, because a lie is language, which is pretty close to the opposite of knowledge.

I mean no offense, but your hedging words are doing so much work that I think your statement counts as bullshit.

Edit: thinking about it more, I think your comment shows the particular kind of bullshit that often characterizes tech people's thinking and is so useful for hyping shit (e.g. take something hard, lossily reformulate it into something easier but vaguely similar, solve the easy thing, then declare you solved the hard thing).

Knowledge is... concepts and their relationships, typically with the implication of being grounded in objective reality (although one can be knowledgeable about things like, say, Tolkien, or Star Wars, or fantasy tropes more generally).

.

Language is (a particular way of representing) knowledge, in about the same way that a particular .cpp file is (a particular way of representing) a linked list or whatever.

So, in your view, are .cpp files and linked lists "kind of" the same thing?

Language can be used to express some kinds of knowledge, but they're not even "kind of" synonymous. Language can express a lot of things that aren't knowledge, and there's knowledge that can't be expressed with language (qualia, at least).

All the more and less formal models we invent for describing "language"? This is just fitting curves to a snapshot of reality. So sure, "a lie is language, which is pretty close to the opposite of knowledge" - a lie can be expressed in the formal model. But the truth (as subjectively understood) is much more likely to be spoken or written down in practice.

Communication isn't random. Language isn't random. Learning it is picking up the patterns.

>>>> But it does mean they're about "language" rather than knowledge.

>>> To some extent, those are kind of the same thing.

>> They are not the same thing.

> Of course they are the same thing.

Language and knowledge (both broadly construed) are obviously not the same thing. Language can encode non-knowledge like nonsense or lies, and there is knowledge (e.g. of the experience of qualia) that can't be expressed in language.

I think the point up-thread is true: something that knows only about how language is unreliable source of knowledge. Even if all its input is true knowledge, it can still blindly combine that input in linguistically plausible but false ways.

you can use language as a proxy for knowledge in some cases, but doing so guarantees generating bullshit, which is the entire point of the OP.

They can emit statements that happen to be true because the probabilities fall that way (as a sibling comment points out.) But it's not "justified", because models don't have any ability to remember or cite sources, or even internally classify tokens as true or false (it's all just weights.)

(And it's not a "belief", either, because there's no intentionality, but that's a more slippery concept.)

Therefore, philosophically speaking, it is fundamentally impossible to acquire knowledge from a LLM unless you verify ("justify") a statement via some secondary, more trustworthy channel.

FWIW, although 'justified true belief' is classic in the sense of being an old definition of knowledge, I don't think any significant number of modern philosophers would use it.

But yeah the problem what you mean by "knowing." For example, I am perfectly happy to say that a LLM "knows" words just like I "know" words, or that it "knows about" very popular topics in a loose sense.

But that's a very different sense of "knows" from the kind of specific propositional knowledge we're usually interested in when using a LLM as a search engine, fact checker or source of information!

I’m clearly keeping track of how strongly I believe a variety of things. Not how I started believing them to begin with, however, so I lack the justification. And in a lot of cases, the lost justification would have been: “Because someone else said it, so now I’m aping them.”

But I think almost none of them would ascribe "knowledge" to a statistical engine that output propositions of random truth values weighted by how many examples of that proposition it's seen in it's training data.

You can view the logprobs in the API and literally watch it choose to say different things based on it's internal dice rolls.

Knowledge without a feedback mechanism for verification is prone to veer off course.

Right now the feedback mechanism is based on human reinforcement.

What if we put an LLM in an android’s body and let the machine interact with the world through the same senses that we do?

Ultimately though this is all about the definition of words. Humans find the distinction between "true" and "false" to be important and meaningful; animals don't care. AIs don't care (being incapable of caring), but humans care very much about the propositional veracity of statements that AI makes.

The important thing is that we're on the same page about what's actually happening with these systems and how they work and don't slip into magical thinking or assume that just because a system is facile with language it automatically has a sense of truth or right or wrong.

This is only going to get more complicated as models get more sophisticated.

https://www.youtube.com/watch?v=yhfl7kasjZc

There are a quite a few videos with the same reaction. There does seem to be something going on.

Is it though?

There's nuance in what we call knowledge, but to me, knowledge is not uni-dimensional. What I mean by that is, I know when I _know_ something, and I have learned that that is not a trivial skill.

How do I know I _know_ something? Usually because I have searched for evidence from multiple angles / dimensions / senses. When multiple independent observations of quality agree, and my conclusion matches the result, it is safe to say that I know that fact.

An LLM may have multiple 'dimensions' in a linear algebra sense, but does it have independent information?

This is technically true, but no one actually does much of it consciously. But this is how you picked up every skill and understanding, including language, since you've been born.

And this - the unconscious fusion of correlated patterns - is pretty much what LLMs undergo in training.

EDIT:

Also, a lot of confusion is created by using terms like "a fact" or "a conclusion" or "a result", as if they were fixed points in space, fully defined mathematical symbols. But there exist no such thing. There is no binary "know/don't know", there is no binary "result match/does't match the conclusion". Those are all continuous quantities, that get rounded off for convenience.

> How do I know I _know_ something? Usually because I have searched for evidence from multiple angles / dimensions / senses. When multiple independent observations of quality agree, and my conclusion matches the result, it is safe to say that I know that fact.

I wanted to thank you for laying out this perspective. In my experience, in discussions about LLMs, a lot of time people get caught up on words like 'know' which work well for talking about humans, but don't cash them out in definitions I know what to do with. This is a really content-ful of sharing what you mean by your claim.

There is a setting right there called temperature, no?

However, stating that LLMs predict text, not facts - and that LLMs don’t “think”, is enough to start a debate.

Yeah the issue is anthropormphization and the “cool” factor of demos, vs actual experience.

I think there's a strong argument to be made (that has been made) that much of our knowledge capability is built on language - that language did not simply allow us to communicate our knowledge, but actually allowed individuals to think in a new way.

I certainly agree that current LLMs have no regard for truth and logic at their core, and there are clearly areas where their ability to read and write does not translate to producing true statements. LLMs can't build things out of language like humans can.

I am less sure that LLMs don't have a long way to go before we reach the limits of what modeling language can do.

True. But most of the training corpus was not linguistically-sound gibberish; it was actual text. There was knowledge encoded in the words. LLMs are large language models - large enough to have encoded some of the knowledge with the words.

Some of. And they only encoded it. They didn't learn it, and they don't know it. It's just encoded in the words. It comes out sometimes in response to a prompt. Not always, not often enough to be relied on, but often enough to give users hope.

What we really need is a language model coupled to a knowledge model.

>Some of. And they only encoded it. They didn't learn it, and they don't know it. It's just encoded in the words. It comes out sometimes in response to a prompt. Not always, not often enough to be relied on, but often enough to give users hope.

And some of the pieces of "knowledge" in the training corpora were wrong, lies, or bullshit themselves.

Garbage in, garbage out.

People say stuff like this a lot, and I never know what to do with it. I'm not sure what specifically I'm supposed to get out of these distinctions.

And that's enough to be a problem, given how people are trying to use LLMs. They're trying to use it like a human who "knows" (whatever that means). And whatever an LLM actually does, it doesn't do that.

If the claim is only "LLMs don't work like humans" and "LLMs make a lot of factual and logical errors", then that seems to be correct as far as it goes, but it doesn't go very far.

Well yes, but how is this insight helpful? People aren't just looking at the name. Have your missed the last 10 months of ridiculous breathless coverage of LLMs?

> The trouble is people think it goes deeper when the knowledge modeling part

People are literally calling this shitty text prediction algorithm "AI". The discourse is completely out of hand.

https://en.m.wikipedia.org/wiki/Dartmouth_workshop

> An attempt will be made to find how to make machines use language, form abstractions and concepts, solve kinds of problems now reserved for humans, and improve themselves.

LLMs fit that definition well. They're not AGI, but I think it's fair to call them AI.

What I think is the problem is that when people hear "artificial intelligence," they don't think of chess programs anymore; they think of something like Skynet from the Terminator. Or, maybe now they think of ChatGPT.

If they don't get to define "AI", who does?

Actually the hint is not in the name, that only tells a portion of the truth. If LLM were trained on textbooks teaching foreign languages from every other language, the name would suffice. But it does not, LLMs are trained on wide variety of texts, many of them social media, blog, and general web posts. The larger web is an ocean of bullshit, and that is where the bullshit within LLMs comes from.

'Textbooks are all you need' ought to be how LLMs are trained so people can use them for their "knowledge work" without bullshit ramifications.

Not only that: there's also the fact that LLMs do not "fact check" their output against any data source. They just generate text based on relative frequencies of different kinds of text in their training data. They don't even have the concept of text being related to other things in the world; in fact they don't even have the concept of "other things in the world".

Any text generator built that way will output bullshit, no matter how accurate its training data is. "Bullshit" does not mean the output is necessarily wrong; it means, just as Frankfurt said, that the thing generating the output doesn't care about whether it's true or not. LLMs meet that criterion: they don't care about anything other than relative frequencies in their training data.

But in many spheres of knowledge there are still a lot of contradictory opinions in textbooks. Why did Rome fall?, What's the best X? In science there have been many schisms: quantum theory, tectonic plates, GMO/Organic etc.

Writing code is probably one of the few areas where just about all content is 'good' - there may be a lot of bad or poor performing code in books and online but the vast majority of it 'works'.

For many other fields, introductory textbooks even at the university level may abound in oversimplifications, and it is only in later years of study that one begins, by reading specialized papers and monographs, to grasp how complicated the facts really are.

For a long time there have been "machine learning" techniques where getting the right answer 80% of the time is a win, what's interesting about LLMs is they sometimes "raise the bar" for the machine learning problems (get it right 90% of the time) but have also lowered the bar for other things that can be done algorithmically 100% of the time.

It’s a powerful tool but like any powerful tool, you will use it wrongly if you don’t acknowledge its strengths and weaknesses.

Judging a LLM because he can’t act as a program is a fundamental error because it means you are wasting an enormous amount of resources for nothing.

The novelty of LLMs and where they are good at is helping you thinking about things. Helping you to complete reasonings about your thoughts or about your data. They are not good at doing things but they are good at « feeling » things.

If a ML algorithm with a 90% success rate and a "sanity checker" might be much easier to implement than a ML algorithm with a 100% success rate.

My take has been that LLMs aren't valuable for me because I'm an expert and writing code is no longer very difficult. And searching for the knowledge when I need it is faster on my own than having a conversation with a computer because true expertise also involves having mastered and streamlined that process.

Even among senior developers it can be shockingly rare for people to actually understand and implement something like DI/IoC in a correct and useful way.

I've had trouble finding it, but years ago, I read the abstract of a study which looked at chess performance, pitting people who had access to some basic chess software against those who didn't. What they found is that the strongest predictor of success was whether a player could utilise the software effectively, rather than pure chess skill.

Another area where I don't think there are any published studies, but there's a lot of anecdotal evidence, is algorithmic trading, where successful firms almost always have human traders monitoring the algorithmic behaviour. This means that humans can spot when the algorithm is doing something a bit odd, or when conditions aren't conducive to the algorithm performing well, and they can tweak some parameters on the algorithm to get a better result, shut it down if it looks like it's lost the plot, etc. (And let's be honest, a lot of this oversight is probably a result of Knight Capital.)

My point here is: You're entirely right, LLMs are currently a tool for people with experience in the field they're working with. But that's still a pretty big step forward -- just like top quant hedge funds consistently outperform human-managed funds (see: Rentech), smart people who learn how to leverage these tools are going to outperform. The key is learning where they perform well, where they perform poorly, and how to validate their output.

There was a time when this was true - a grandmaster and a computer was stronger than either alone. But that window closed pretty quickly. Today, even the strongest GMs' best strategy is to follow the computer blindly.

Gray-box trading and humans in a control loop over automated trading have both been happening for decades. Knight Capital makes for a pretty spectacular example, but that's about it.

Most examples I have seen are about basic code that is highly likely to be already present in the training data - making the language model an expert system, but with a worse rate of failure.

Meanwhile it has a hard time creating something it hasn't already seen, or a close variant.

How much time have you spent with the strongest models, such as GPT-4?

I've been using it on a daily basis for 6+ months now and your statement there doesn't fit my experience at all.

Just yesterday I pasted in a hastily written function and told GPT-4 to "please evaluate this function and make sure it's bulletproof". It outputted a bulleted list of criticisms (mostly true) and then spit out a version of the function that was shorter, easier to understand, and was safer.

That's not just the "next most likely token". Or if it is, then that's all any of us are doing.

It does not hand recursive thinking well at all.

The trick to using it effectively is to figure out the ways of prompting it that produce GOOD results.

At this point I'm pretty confident it helps them. They need to understand that it's not infallible, but I think most people figure that out pretty quickly after using it for more than a few days.

There's absolutely a skill to using these things well, and it's a difficult one to teach because it's based more on intuition that you develop over time.

The key thing is helping people understand the strengths and limitations of these tools. That's something I do think we can teach.

I was listening to a podcast yesterday where the purported expert gave an answer that sounded right but based on my direct experience - very wrong.

Of course I could be wrong as well.

Or the experts answer was the same kind of BS the OP was referring to and is so oft generated by LLMs.

Many philosophers argue that knowing whether something which appears to be conscious really is conscious is a meaningless question. If there is no way to distinguish consciousness from non-consciousness from the outside, then you may as well just say that the thing is conscious.

I think we may be approaching the same question with regard to truth. If it appears to be truth from the outside, is that the same thing as being truth? Is there any point in knowing wether it was produced from a thinking mind or not?

If you haven't seen Gettier problems before, I'd start with the generalized problem and work out from there: https://en.wikipedia.org/wiki/Gettier_problem#The_generalize...

Edit: #if directives, like static_assert.

  ~ $ cat sizeofmacro.c
  #include 

  #define pintsize(T) printf("sizeof(%s) is %zu. Big or small? You decide.\n", #T, sizeof(T))

  int main(int argc, char **argv) {
          pintsize(int);
          pintsize(argv);
          return 0;
  }
  ~ $ gcc sizeofmacro.c && ./a.out
  sizeof(int) is 4. Big or small? You decide.
  sizeof(argv) is 8. Big or small? You decide.

the preprocessor doesn't know anything about structs or types

>The sizeof in C is an operator, and all operators have been implemented at compiler level; therefore, you cannot implement sizeof operator in standard C as a macro or function. You can do a trick to get the size of a variable by pointer arithmetic.

> Certainly, you can use a combination of sizeof and #error directives for conditional compilation to achieve a similar effect without static_assert.

  // Compile-time check for structure sizes
  #if sizeof(Elf32_Phdr) != sizeof(((Elf32_Ehdr*)0)->e_phentsize)
      #error "32-bit structure size mismatch"
  #endif

  #if sizeof(Elf64_Phdr) != sizeof(((Elf64_Ehdr*)0)->e_phentsize)
      #error "64-bit structure size mismatch"
  #endif

> I didn't know sizeof could be used with the C preprocessor

> Yes, sizeof can be used in certain contexts within the C preprocessor.

> It's commonly employed in combination with the #if and #error directives to perform compile-time checks on sizes or other constants.

> This usage allows you to catch potential issues during compilation rather than at runtime.

I should probably go sleep a bit before I embarrass myself any further. After nearly 24 hours awake it seems I have lost the ability to tell macros and directives apart.

What's stopping any source from doing this? In the case of encyclopedias or news websites, I guess you could say reputation. But that's hardly reliable either.

So I guess that's my big pushback in regards to correctness complaints:

1) "Official" sources have always lied, or at least bent the truth, or at least pushed their subjective viewpoint. It has always been the readers job to think critically and cross-reference. On this point, LLM don't fundementally change anything, they just bring this long-running tension closer to our attention.

2) "LLM are inaccurate". Ok, inaccurate compared to what? Academic journals (see the reproducibility crisis), encylopedias (they tend to be accurate by nature of leaving out contentious facts), journalists (big laugh from me)?

I think outright hallucinations are a valid concern to bring up, but I would refer to my point about cross-referencing. However, I'd still point out that oftentimes a person might have read a perfectly factually accurate encyclopedia article but remembered hallucinations via motivated reasoning. Is the end result (what the person thinks and remembers) really different between LLM and traditional sources? This seems like more of a human problem than a LLM problem.

Do we have any factual evidance showing that people learning via LLM+traditional methods are actually less informed than people who learn from traditional methods alone? Right now, there's a lot of fear mongering and charged rhetoric, and not a lot of facts and studies. The burden of proof needs to fall on the people who want to regulate and restrict access to these models.

Finally, what I think this is really about is the continued transfer from the "industrial age" to the "information age". 100 years ago, your average person wasn't expected or really even able to "do their own research", and instead relied on top-down elite driven institutions to disseminate their version of the information. De-industrialization, then the internet, then social media, and now LLM are cracking this (now) outdated social order, and our old elites are understandably threatened.

I think this is another reason why we need to make this debate more rigorous and fact based: are LLM actually dangerous or is this just an example of elite preference?

I think the major difference that makes in not just "elite preference" is that the LLM output is just confidently wrong. If I read a report from an elite institution I'm at least reasonably confident that they're not going to base their entire argument on the premise that 2+2=5. Someone is going to call them on it, there may be reputational damage, there may even be legal repercussions in certain cases.

LLMs have no such protections. GPT recently, when asked to "write a function using the ELixir programming language that..." wrote a function in Elixir syntax using Python libraries and function names. That's a class of error that makes it actually dangerous if you're asking it about anything you can't fully check on your own, and there's no checks-and-balances for the content it generates that's only ever visible to a single user.

I agree with you that "official" sources can't be trusted and there are inaccuracies everywhere, but you have to admit that the craziest sources who say the craziest things (flat-earthers and such for example) get pretty easily dismissed and everything else they say becomes suspect. LLM's bypass that and can say the most outrageous things without ever getting caught or being forced to make a retraction/correction. That feels like a problem worth worrying about.

Here's Nvidia showing they can use LLMs to teach robots complex skills - https://eureka-research.github.io/

lots of game changing technology starts off being used for toy or bullshit use cases, the GPUs being used to train AI were originally designed for allowing video games to have better graphics

In my experience so far, more often than not, it doesn’t get it right. Brings to mind the old maxim, “a broken clock is right twice a day”.

My new thing: Testing LLM critique against humans.

The results are sobering.

In my experience, there is a wild amount of finding absolutely nothing wrong with what is clearly going wrong and me having to prompt some variation of "so... what were you planning to do about that thing?" for something to happen.

One thing to add - we should not be comparing LLMs to experts in any field (where expertise can be objectively determined). We should be comparing LLMs to an average (in some cases better than average) layperson. The same level skepticism is warranted but it does not mean that they are not useful.

My guess is that if you created a GPT of a well-regarded book on WW1, you’d be able to have an enlightening conversation about the topic.

So they run the code, it breaks, they paste the error message back into chatGPT and try again. What's wrong with that? Don't know about you, but I don't get code right the first time, either.

SQL injection is bad code that "works". Most security issues come from bad code that just "works". I can't believe I have to say this, on HN of all places, but you have to actually know what you are doing when writing code. Some of you scare the fucking shit out of me.

Hawing a noise generator sometimes generate something that looks like a valid signal doesn’t mean it’s not a noise generator.

It is a language model, so it’s best use is language things (emails, motivational letter, summaries)

For everything else, you need to be enough of an expert to validate the output.

Does this happen often for you?

I've logged my success rate with Go and C.

32/37 responses failed to meet the requested criteria. 15 wouldn't even compile.

Getting the bot to do it right takes longer than writing it myself.

Maybe that's just me?

I wouldn't think so. Word prediction and writing code aren't nearly the same thing, when I think about them. Sure, they both require language. But one of those also requires reasoning. As worded in this paragraph, could an LLM tell me which one? Sure. Sometimes...

+1, though it took some time to figure out what questions work well. Yesterday, I asked chatGPT to help me document some R functions, and in my opinion, it did a great job [0]. I asked it to summarize what my functions were doing, and it gave me nice, plain-language summaries, and then reformatted my notes into the the form that Roxygen2 [1] expects, and added some useful inline comments. Moreover, it was fast. It read and understood my code in a few seconds. No human can compete with that.

I don't think this is bullshit. I think that chatGPT shows expertise with some formal conventions that can be kind of a pain to memorize and work with. You might think those conventions were BS in the first place, but regardless, they're what we settled on, and it's really nice to have a coding assistant do the tedious work.

I don't think chatGPT could have written the functions in the first place. But who knows what GPT [5,6...N] will be capable of

[0] https://github.com/setgree/sv-meta/commit/5f71e7c251b38e1981...

[1] https://cran.r-project.org/web/packages/roxygen2/vignettes/r...

Ever seen mainstream media write something about your field of expertise that wasn't complete horseshit? No? Then why believe anything else they say.

The real horror is when there's going to be an entire generation that grew up not by relying on experts/historic facts and reality, but on something that AI generated that looks good enough and is a neat answer.

Imagine this generation in positions of power over actual experts, or equating AI opinion with the opinion of an older-generation expert.

Progress in all of human history has always hinged on our ability to disregard idiots in favor of experts. Now, the position of the idiot is stronger than ever, because anyone can type a prompt into GPT without real understanding.

That's the real scary part.

For the example, causes of the Great War, I'd first ask it to provide a list of the top ten reasons reputable historians cite as primary causes of World War One. Then, for each reason that looked interesting, I'd ask the LLM to provide arguments for and against, including one primary historical document in support of each side. Then, go to Google Scholar or similar and look up that document and see what different reputable historians have said about it. A few iterations of this process should get you to some reliable information even in a field you know nothing about.

This is a time-consuming process, and requires active engagement rather than what many people have been trained to do by our atrocious education system and watching television, i.e. passive absorption of content without skepticism.

Now there might be outliers that are missing from both lists but which might be worth looking at, as it's not that unusual for consensus viewpoints to be overturned by new information and so on, and that's not something LLMs are likely to be very helpful with.

Fundamentally, my point is that if skeptical thinking, rational analysis, research strategies etc. aren't taught to people from a young age than it really doesn't matter whether the BS they're subjected to is human-generated or AI-generated.

I want to call out that this is not necessarily true. You can interact with the LLM using agents, feedback loops, and some scoring function on randomized test inputs to produce novel code that you don't know the structure of beforehand.

You start with a set of example inputs and desired outputs, and prompt the llm for a function in your programming language of choice. You then take the LLM's response and feed it into an agent that executes the code against the inputs and outputs, and report the discrepancies back to the LLM. It responds with a new completion, which you feed to the agent until all your example inputs produce the expected outputs. Finally, you can use property-based testing to produce new examples for testing by the llm, resolving the answers until the produced code is correct to within some margin of acceptable correctness.

You can do this to produce code without needing to know anything besides the desired properties of generated code and the properties of the inputs. You can further automate this by using a separate LLM to produce the examples instead of the typical generation and shrinking functions to produce examples.

This doesn't require any prior knowledge of the target language. You can expand beyond programming into any domain for which you can produce a scoring function and automate input generation.

I suspect that you can use control theory (PID loops, behavior choice loops scored by Eigenvalues) to model complex scenarios spanning multiple domains as well, choosing the evaluation agent as the behavior, each behavior of which has a separately defined scoring function. All of this can be automated by using the LLM generator without prior knowledge of the algorithmic structure of the solution and knowledge of all possible inputs to all possible behaviors.

If that all sounds very familiar, it's because it's essentially just doing test-driven development, but with the LLM machine as the developer.

It would be expensive to run an entire project development that way, due to the high costs of executing LLMs and/or querying LLM apis, but if the cost is less than that of employing a junior developer or teams of developers, it might be worth it. And human intervention can be kept in the loop at any stage, making it very feasible for rapid prototyping - where you don't necessarily need the entirely correct answer from the machine, just a good enough starting point to allow the human to take over to produce a result.

Reminder, you started with this:

> You can interact with the LLM using agents, feedback loops, and some scoring function on randomized test inputs

Maybe those are fancy words for simple things which I’m not familiar with or learning programming language seems a simpler option :)

Of course if you feed an AI scientific papers and books of some quality, information will be of high quality. THAT... is how you "filter for truth". It's not supposed or required to be perfect. But of course you can filter the input.

The fact you also add conversational input from various sources like Reddit and Twitter doesn't change this balance significantly. Also even those sources are filtered to exclude vulgarities, certain topics and so on.

Also you're not thinking this through. If a question is on average answered incorrectly 'round the world, it means you're over 50% likely to also give the incorrect answer when asked. Who equipped you to know the truth?

When/where/how?

Perhaps in the future, the training data will be more carefully curated to eliminate as many 'wishful truths that do not exist' and 'misinformed conversations between parties with no correct voice' as possible.

Anyway, yes, the "web" in general is full of garbage and propaganda. But then again we're exposed to the same web, and without the pre-processing which OpenAI etc. apply to filter out certain words, phrases, topics (including lots of manual human labor). So it's funny how we're living on that same input, but we're so confident we know what's true and what's wrong.

But LLMs aren't just trained on literally anything on the web. For example consider Wikipedia. That's not one's average drivel for sure. And next to the general web, you need higher quality sources of data in the mix to bias the AI for truth. Scientific papers, good books, transcripts from conferences and presentations. Wikipedia to mention again. So on.

Of course, sometimes LLM make technical mistakes which you can clearly rule as incorrect. Say, plumbus.com's API never had a method getDescription(). But the AI saw many APIs of services like this, which have API getDescription() and given its small network, it's forced to fill-in gaps by inferring the answer based on a vast assortment of contextual clues. Which we call hallucinations (incorrectly). But even the correct answers are inferred. It's all inference. And we all think by inference. We just have 50 times more capacity than the biggest LLM at the moment, so our inference tends to be much more sophisticated, and therefore more likely to be correct. I mean in theory. After all, it's humans and their big brains who produced the garbage that's the web, so.

Children's initial years, being basically lied to about the nature of the adult world, is where their sophisticated inference comes from. The internal intellectual conflict when one realizes it is actually good to tell children the world is a nice, fair, happy place - once figuring out it is not. That right there, I theorize, is where our sophisticated inference is created: we realize our own dynamic minds. We realize the necessity to seat concepts of "a fair world" into people before allowing them to mentally mature and learn it is not.

Hah, sure. Okay. How? You just read it and think really hard?

Bertrand Meyer, the inventor of Eiffel, who rants about software correctness all the time -- didn't notice an error in a 1-line expression in Eiffel that was generated by ChatGPT [0].

If you spend a little time formalizing correctness in a theorem prover or model checker you may not be so confident that you can read a snippet of code and know that it's correct. In order to know that you have to be able to write the specification precisely. Natural language is not precise enough.

Update: You may be able to write the specification precisely either by modelling the system and checking the model or writing a theorem and proving it... but so far LLM's cannot reason on that level.

[0] https://buttondown.email/hillelwayne/archive/programming-ais...

> AI systems like ChatGPT are trained with text from Twitter, Facebook, Reddit, and other huge archives of bullshit, alongside plenty of actual facts (including Wikipedia and text ripped off from professional writers). But there is no algorithm in ChatGPT to check which parts are true. The output is literally bullshit.

Well. Any philosopher, mathematician, scientist, or novelist was trained on plenty of actual facts (established works of other people, school learning, experiments) and huge archives of bullshit (individual day-to-day experiences that cannot be called scientific, casual conversations just like Twitter conversations, propaganda, lies). Their output then would also be bullshit, and by extension, everything humanity ever created.

Where I found ChatGPT to be really helpful is when I don't know anything regarding something, and ChatGPT can give me something I could at least verify as true or false.

Compared to before, where I just got stuck on some things, as I couldn't find out what I was actually looking for at all.

That said, there are many people who don't bother to verify anything, as can be observed in comments sections where one can in many cases find people who clearly only read the headline and not the article.

Anyone who uses ChatGPT for a while will learn to take everything with a grain of salt.

If I ask ChatGPT "why is fluoride bad?" it doesn't give me hours of conspiracy theory content^. It doesn't try to sell me water.

Google is not so aligned towards neutral. (Of course, people have been busy sabotaging the Google dataset for over a decade)

^ although I did ask it "what do I do about a ghost that lives in my house" and got this: > Dealing with a ghost can be unsettling. You could consider consulting a paranormal expert or a spiritual advisor for guidance on how to address the presence of a ghost in your house.

Context matters - if you begin with saying that you're 6 years old, it will not be willing to admit that Santa isn't real.

* Ignoring it. Not because ghosts aren’t real, but because “some people choose to cohabit and not engage unless it becomes intrusive”.

* Politely ask the ghost to stop disturbing you.

* Purify your home by burning herbs.

* Hire a medium or psychic.

* Get a spiritual leader “from your respective religion” to bless your home.

* Seek help from paranormal investigators.

If I make the exact same question but give it the system prompt “You are James Randi, the world-famous skeptic”, it gives a reasonable answer to help identify the true cause of whatever is making you think there is a ghost.

Which just goes to show how much of a bullshit generator this is, as you can get it to align with whatever preconceived notions you—or, more importantly, the people who own the tool—have.

Then it suggested, amongst other things, ghost expelling rituals, seeking professional psychological help, and moving out of the house.

Using the James Randi prompt, the answer is comical. It says “that would be quite a twist” since he had spent his life disproving supernatural claims.

This, to me, seems to be an amusing reversal of trends. Before the internet there was the "mainstream media". NBC, CBS, ABC run by the big conglomerates such as GE. Which would mold public consensus in the US. Now there is a desire to go back to that. Let LLM do the thinking and the managing of biases and just tell me what to think.

The unfiltered internet is too much. It's overwhelming. It's the raw sewage of the Facebook feed. We need someone to coddle us. And that champion, today, is OpenAI.

Studying political polarization in the US, the end of the Cold War, globalization, the end of the fairness doctrine, the evolution of major media ownership over time, social media algorithms, echo chambers and confirmation bias is left as a starting exercise for the reader.

This is true, yet I wonder how much of this "ChatGPT is way better than Google at Googling" effect is due simply to how bad Google has gotten over the last decade.

ChatGPT and similar models seem to be uncontroversially best at this kind of task -- just being a replacement for Google Search.

I think you answered your own question there. It doesn’t say your question makes no sense, or that’s incorrect. It just makes up a bullshit answer.

Most of negative sentiment is not “world would be better without chatgpt“, but “one must be very cautious and always validate the answer”.

It’s valuable tool, but it has a LOT of hype, which led to things like people trusting chatgpt output in a court case.

Second, the fundamental issue here is that as long as verification is needed, this technology is only ever useful for domains in which you're already an expert. That severely limits the technology, and yet this keeps being touted as some kind of general knowledge base when clearly it's at best a very narrow one.

And it seems to me like this is unlikely to be solved in the current paradigm. This is just intrinsic to the architecture of LLMs. There is no getting away from it.

So I guess I am overestimating the average user by assuming that people read the text written on every single page they use to query the system.

Even so, this entire comments section reads like a giant saltmine of people that are terrified of embracing tech that in every sense of the word increases turnaround time on productivity.

The ability to literally feed a screenshot into it and ask questions regarding it, with insane results is astounding.

I have input a picture of baby rabbits and asked it to rate my chickens, and got a detailed explanation of how the picture contains baby rabbits and not chickens.

There are things the system is great at, and others it sucks at currently. But you are all mistaken if you think its going to stay this bad. If you haven't even bothered to check how quickly the field has moved from single-modal to multi-modal, well, damn guys, I am sorry, but you are getting automated first.

The biggest problem with current models is that they don't know how to verify their own answers, and there's no way to allow them to without putting a ministry of truth on it (similar to what twitter was during covid).

To combat this, we need to figure out how to let the model generalize truth, which is something it can only do if fed enough (unfiltered, unbiased) data. With enough data, it will find the patterns underlying false information and eventually figure out how to use it.

In fact, most likely, the future models will simply be fed unlabeled data, in any format (audio, video, telemetry, analytics) and generalize from that.

The concept of feeding normalized data to a model for training is archaic and stupid. The world is not normalized. Data is not always the same, and will not always even be guaranteed to be present.

Anyways, what does it matter.

Why am I trying to sell sugar to people addicted to salt?

Yes, you definitely are. Speaking from experience of supporting users for many years. People will not only ignore such text, they will repeatedly make mistakes or ask questions the text answers directly, even after you’ve explained it five different ways.

The rest of your post is full of assumptions and dismissals, so I think I may be wasting my time, but let’s please stop with the rhetoric that someone who doesn’t agree with you is somehow afraid or has an agenda. As way of example, your comment ignores the bigger picture of bad actors using LLMs to push propaganda, sow discord, or scam people. The “bullshit” part of the problem isn’t limited to “it gave me a wrong answer on my homework”.

Why is verification part of the process if not because ChatGPT generates bullshit? I don’t think the author is disregarding that it is sometimes correct.

What authority does random information on the internet have? How do I verify the information without looking at random information on the internet?

People don’t do that now and we just made it easier to get confident answers that no one will check.

No, everyone very much does not know that. Including the lawyers who though it made sense to ask ChatGPT questions, than asking it if the answers were correct.

https://www.washingtonpost.com/technology/2023/11/16/chatgpt...

Proof that LLMs aren't the only bullshit generators, humans promoting LLMs are too!

No - not everyone using ChatGPT knows that verification is part of the process.

In fact I'd wager that the majority of people using ChatGPT aren't doing any verification on the output at all.

lol. lmao

On the other hand:

"Things that try to look like things often do look more like things than things. Well-known fact," said Granny. "But I don’t hold with encouraging it." - Terry Pratchett, Wyrd Sisters

This makes LLMs much better than claimed by those who dismiss it as BS or stochastic parrot or whatever, and also not as good as domain experts.

Humans don’t learn the same way AIs are trained. Equating the two is a fallacy.

So, they have those two modes of operation, and most of them are built in a way that won't let you know which mode they are working on. That can be useful depending on what you want to do, but it doesn't strike me as the most effective architecture for them.

(They are also reasonably good pattern fitters, what is kinda nice for code autocompletion on some languages. But they are way too inefficient here. Yet, nobody created any good high-efficiency autocomplete that merges searching and pattern fitting, so people use them.)

Can you share which obscure 18th century historical fact you were looking for? Maybe you happen to be exceptionally bad at google queries?

Would you mind telling what's the fact you were searching for? (and how did you search for it?)

It definitely took you more than 10 seconds, though I guess a bit of hyperbole does not hurt.

You validated findings against other source, which is obligatory step with chat gpt.

It's a little surprising how hard it is to convince people that it's worth $20 to use v4.0. It is very apparent how much better it is.

It's not extremely convincing.

It also seems odd that you had the materials available to research and validate the response ChatGPT gave you, but somehow no ability to glean the necessary answer from those materials to begin with. Obviously, because Google was completely useless to you, you didn't simply use Google to do that validation, nor could you have used another search engine, because then you could have found the result you were looking for to begin with. Did you just use ChatGPT to validate its own responses?

Exactly. You answered your own question and that is the author's point.

Do you even know why it gave the incorrect answer at first? Not even ChatGPT can explain to you why it did that or keeps doing this.

How can you trust it to give the right answer (without you Googling to check) if you really don't know the answer? It already means you don't trust it an you know it bullshits as the author has claimed.

It can confidently convince someone outside of one's own expertise that it can give a soundly correct answer but can easily be bullshitting nonsense.

> Well. Any philosopher, mathematician, scientist, or novelist was trained on plenty of actual facts (established works of other people, school learning, experiments) and huge archives of bullshit (individual day-to-day experiences that cannot be called scientific, casual conversations just like Twitter conversations, propaganda, lies). Their output then would also be bullshit, and by extension, everything humanity ever created.

The difference is with humans, they can be transparently held to account on whatever they say. An AI system cannot. So this actual whataboutism towards humans is incredibly weak here and a common excuse by AI proponents supporting the nonsense that AI systems like ChatGPT can uncontrollably generate in a black box system.

One is analogous to how LLMs operate and is essentially next token prediction. Think of your mental state when you’re very engrossed in conversation. It’s not a truly conscious act. Sometimes you even surprise yourself. If you’re a child, or childish, you may just confabulate things in the moment, maybe without realizing that you’re doing it in the moment.

Now think of some very difficult problem you’ve had to solve. It’s not the same, right? It’s a very conscious act, directing your focus here and there, trying to reason on how everything fits together and what you might change to fix your problem. Odds are good that you’re not even using language to model the problem in your head.

LLMs are doing the first thing, and are exceptionally good at it even in this early stage. The surprising thing to me is how far this can get you. If you have an inhuman level of knowledge to work from then in conversation mode you can actually solve some moderately difficult problems.

I think that maps to our own experiences as well. For the things that you have deep knowledge on you will sometimes find yourself solving a problem just by constructing sentences.

knowledge structures as we construct it are symbolic, with symbols representing abstractions (ie classes), along with relations between these symbols. human ingenuity consists of coming up with new symbols or new relations between existing symbols (which is a process of abduction) based on new perceptual inputs (either our senses or instruments). Such knowledge structures are powerful because, they allow us to build giant towers based on solid foundations.

> For the things that you have deep knowledge on you > will sometimes find yourself solving a problem > just by constructing sentences.

This is another way of saying that you have clarity in that subject, and so your stream of thought aligns with knowledge structures - which is another to say that you really understand something. However, in my experience very few people are able to stay within their lanes (competence), and most of us tend to babble on topics we really dont have knowledge structures for. also, few people have the self-awareness of what they really have knowledge structures for (ie, know what they dont know).

That's what rubber duck debugging is, no?

LLMs are rubber ducks that can talk back, for better or worse.

There was that study a while ago that triggered the NPC meme. The common interpretation was that most people only think in the first way you described.

https://hurlburt.faculty.unlv.edu/heavey-hurlburt-2008.pdf

If so, the interpretation you describe seems pretty far off base.

Also, my impression is that that meme was not sparked by a psychological study, but that a few people did draw on this study to justify it.

And I thought what kicked it off was from only a few years before at most.

I feel like we should have all moved on from the fact that LLMs don't have a very precise memory? This is (not entirely, but mostly) fixed by including the information you want to query in the prompt itself, like Bing does. ChatGPT is much better at summarization than precise fact recall.

And most of the time, it doesn't matter. When I'm using ChatGPT to learn or troubleshoot something, it's a jumping off point, not the end of the story. As long as everyone understands that, there's no problem. I have yet to see anyone who treats its output as 100% fact, and no AI company claims it either.

But everyone doesn’t understand that, so there is a problem.

> I have yet to see anyone who treats its output as 100% fact

Yet they exist.

https://www.nytimes.com/2023/06/08/nyregion/lawyer-chatgpt-s...

Not every human being on Earth knows what ChatGPT is, so that’s a non-starter.

> or everyone as in most reasonable people

That qualifier is doing some heavy lifting. It’s so vague it can be used to excuse anything¹. Not only reasonable people (however we’re going to define that) have access to the tool.

> ChatGPT tells you it can lie before you can even ask it anything.

And what you learn fast by doing user support of any tech tool is that most users don’t read instructions even if it’s shoved in their faces. I personally know people that think ChatGPT is the source of truth, despite that disclaimer.

¹ https://en.wikipedia.org/wiki/No_true_Scotsman

Mimicking reasoning abilities is not the same thing as actually having/developing them.

That’s like saying “Oh, you didn’t get the right answer to that math question, you just mimicked giving me the right answer”. Yeah, but if I gave you the right answer, then I gave you the right answer, and I must’ve gotten it somehow.

In the same way that with a calculator, if you put bullshit in you will get bullshit out. In many cases you need to INPUT facts. This is automated with things like RAG.

I think this is why ChatGPT is extending deeply into RAG, because it matches more closely how users are expecting the tool to work.

the humans I work with were taught by people that spoke facts backed up by empirical evidence

meanwhile chatgpt was trained on reddit and twitter

So far, the yi model is actually not bad, and by two weeks from now two new models will be out that are better. So it will continue until Altman figures out which politicians to bribe the hardest.

Given that the gold standard for rubberducking neither knows nor communicates anything, this is an easy win.

How you deal with that is up to you. You can either try to work with what it gives you, setup tooling to easily verify things, and increase your productivity, or you can dig your head down in the sand and say it's all incorrect and you want nothing with it.

Both are valid approaches :)

AIs don't "sometimes produce bullshit", they are always producing bullshit. Sometimes what they produce is accurate, as a consequence of the initial corpus being accurate. But to the AI, much like to the demagogue, this is totally irrelevant - it only has to sound good.

And if it's "easy" to verify what AIs say, then it should be "easy" to teach AI to verify before it speaks, which would automatically counter this criticism of AI as bullshit machines. But the real threat is that AI can produce bullshit which is hard to verify, and can do so at a speed that puts professional gishgallopers to shame.

The Cambridge Dictionary says it has two definitions[1]:

1. a rude word for complete nonsense or something that is not true

2. a rude word meaning to try to persuade someone or make them admire you by saying things that are not true

I don't think thise align with your definition and your idea that AIs always produce bullshit. They don't always produce nonsense or something that isn't true, and they don't intentionally use untrue statements to try and persuade someone or gain admiration.

[1] https://dictionary.cambridge.org/dictionary/english/bullshit

Under the above definition, AIs do always produce bullshit, because their content is always truth-irrelevant.

[0]: https://en.wikipedia.org/wiki/On_Bullshit

On one axis we have the truth value of a statement, on the other, the intent (sadly no tables on HN)

Good intent

- Factually truthful: 'Roses are red'

- No inherent truth Value: Maybe surreal humor, like 'this statement is humor'

- Factually false: Satire, parody, sarcasm, 'Great HN comment Einstein'

Bullshit: Statements for effect, indifferent to truth value

- True: 'More doctors smoke Camels'; 'My response to the Covid pandemic was amazing!';

- No truth value - ill-formed nonsense, oxymoron, non sequitur or empty puffery: 'Our Founding Fathers under divine guidance created a new beginning for mankind.'; 'Starbucks provides an immersive ultra-premium coffee-forward experience.'; 'Nobody’s bigger or better at the military than I am.'

- False: 'Obama was born in Kenya'; 'We had the biggest inauguration crowd ever.'

Bad intent, to defraud or mislead:

- True: 'I did not have sexual relations with that woman, Miss Lewinsky'

- No truth value: 'We’re not going to sit here and listen to you bad-mouth the United States of America!'

- False: 'Clinically proven to boost genes and make your skin visibly younger in just a week'; Gish Gallop

Of course this is mostly bullshit:

- Hard to discern intent when something turns out to be false, was Bush's WMD claim bullshit, or a lie; similarly Obama, 'if you like your insurance, you can keep it'. One man's benign public health simplification about masks or food pyramids is another's nefarious conspiracy.

- Even hard science isn't free of bullshit, Feynman's cargo cult speech notwithstanding, any sufficiently complex science and engineering devolves to some amount of cargo cult bullshit in practice

- Let's not even get started on organized religion, Washington and apple trees and whatnot. Communication takes place on multiple levels, fruit flies like a banana, something can be bullshit on one level and good (or bad) intent on another.

In the words of George Carlin, "Bullshit is the glue that binds us as a nation." Or Napoleon, "History is a set of lies agreed upon."

GPT's ability to bullshit at a human level is a truly monumental achievement.

So what I don't understand is how "drunk driving ... helps people to get to work on time" is close to true and what I said that made it seem like "impossible to say if it's good or bad" about ChatGPT?

In the context of the bullshit economy (e.g., the bureaucrat who writes reports that nobody [reads] because he is paid to write reports that nobody reads because this inflates his manager's headcount and therefore inflates his manager's salary) this is all that needs to be said.

It's tangential to uses of ChatGPT by someone who examines the output and selects what is useful via some kind of expert judgement, or who refines his use of the tool via API calls rather than naive text prompts to the default GUI.

Code that runs and works (as in, does what it's asked to do) isn't "bullshit" at all.

Neither is search with references to custom sources in a specific corpus ("RAG").

So yes, AI is in many ways a bullshit generator (as are we); but it can be used in useful and non-bullshit ways.

There’s a pretty big gulf between functional and correct in a world where optimization matters. This is not always the case (strictly, i.e. algorithmically speaking), but you should be optimizing for things like reusability, testability, etc. which GPT-generated code can miss.

In French, we have a word that I find more accurate, which is "baratineur", which means "someone who tells you what you want to hear, regardless of the concepts of truth or lies"

(With apologies to Einstein)