> I learned how to write math kernels by renting Vast VMs and watching Gautham Venkatasubramanian and mrdomino develop CUDA kernels in a tmux session. They've been focusing on solving a much more important challenge for llamafile, which is helping it not have a mandatory dependency on the cuBLAS

If I'm reading this right, they're trying to rewrite cuBLAS within CUDA itself. I'm guessing the next step would be removing CUDA dependency and go with directly using Vulkan or Metal compute shaders. Am I correct?

It can then after compilation benchmark these, generate a wisdom file for the hardware and pick the right implementation.

Compared with that "a few" implementations of the core math kernel seem like an easy thing to do.

I wish we had superoptom

I'm sitting in an office with a massive number of Macbook Pro Max laptops usually sitting idle and I wish Apple would realize the final coup they could achieve if I could also run the typically-NVIDIA workloads on these hefty, yet underutilized, Mx machines.

Sounds kludge, but introduce enough constraints and you end up with this as the best solution.

Good question, the account is muddy --

1. Electricity is a parent company responsibility, so while that is a factor in OpEx price, it isnt a factor for us. I dont think it even gets submetered. Obviously, one wouldnt want to abuse this, but maxing out Macbooks dont seem close to abuse territory

2. The M1/M2/M3 machines are already purchased, so while that is major CapEx, it is a sunk cost and also an underutilized resource most of the day. We assume no wear and tear from maxing out the cores, not sure if that is a perfect assumption but good enough.

3. Local servers are out of the question at a big company outside of infra groups, it would take years to provision them and I dont think there is even a means to anymore.

The real question is cloud. Cloud with RTX/A100 would be far more expensive, though I'm sure performant. (TPM calculation left to the reader :-) I'd leave those for fine tuning, not for inference workloads. Non-production Inference is particularly bad because you cant easily justify reserved capacity without some constant throughput. If we could mix environments, it might make sense to go all cloud on NVIDIA but having separate environments with separate compliance requirements makes that hard.

Jokes aside, I think a TPM calculation would be worthwhile and perhaps I can do a quick writeup on this and submit to HN.

In such a case, my guess is that the machines being free would be trumped by the increased cost of electricity.

Of course this knowledge is superficial and probably outdated, but if I'm not too far off base, it's probably more work to translate a general CUDA-like layer or CUDA libs to OpenCL.

In practice, OpenCL became a giant mess. Some vendors put speed bumps by not supporting the transition from 2 to 3, or having shitty drivers for it.

It also sat at the wrong level of abstraction for high performance compute, which is why CUDA ended up being used.

Vulkan would have been reasonable to write compute shaders in, if there wasn't a ton of alternatives out there already now

    https://github.com/ggerganov/llama.cpp/pull/6414
    https://github.com/ggerganov/llama.cpp/pull/6412

The unrolling optimization is also just another flag away (`-funroll-all-loops`). The Intel Compiler will even do this without prompting. In fact, it appears to only do a modest 2x unroll on my machine, suggesting that the extreme unroll in this article would have been overkill.

Parallelization certainly a lot to ask of Fortran 77 source, but there there is little stopping you from adding OpenMP statements to the `SGEMM` function. In fact, modern Fortran even offers its own parallelization constructs if you're willing to go there.

Which is to say: Let's not belittle this old Fortran 77 function. Yes it is old, and does not even resemble modern Fortran. But the whole point of Fortran is to free the developer from these platform-specific details, and hand the job off to the compiler. If you don't like that approach, then you're welcome to go to C or C++. But this little block of Fortran code is already capable of doing just about everything in this article.

I am basing these comments on quick inspection of the assembly output. Timings would be equally interesting to compare at each stage, but I'm only willing to go so far for a Hacker News comment. So all I will say is perhaps let's keep an open mind about the capability of simple Fortran code.

[0] https://www.cs.utexas.edu/users/rvdg/tmp/TSoPMC.pdf

Anyway, I have come to eat crow. Thank you for your insight and helping me to get a much better perspective on this problem. I mostly work with scalar and vector updates, and do not work with matrices very often.

I am not trying to say that a simple 50+ year old matrix solver is somehow competitive with existing BLAS libraries. But I disagreed with its portrayal in the article, which associated the block with NumPy performance. Give that to a 2024 Fortran compiler, and it's going to get enough right to produce reasonable bytecode.

DO CONCURRENT is a serial construct with an unspecified order of iterations, not a parallel construct. A DO CONCURRENT loop imposes requirements that allow an arbitrary order of iterations but which are not sufficient for safe parallelization.

The ones you can run on your own machine tend to be bad - really bad. They hallucinate wildly and fail at all sorts of tasks that the larger hosted ones succeed at.

This makes them a fantastic tool for learning more about how LLMs work and what they're useful for. Interacting with a weak-but-functional LLM that runs on your own computer is a great way to get a much more solid mental model for what these things actually are.

if you accidentally use the word"hack" when trying to get ChatGPT to write some code for you. it'll stop and tell you that hacking is bad, and not a colloquial expression, and refuse to go further.

privacy reasons are another reason to try a local LLM. for the extremely paranoid (justified or not), a local LLM gives users a place to ask questions without the text being fed to a server somewhere for later lawsuit discovery (Google searches are routinely subpoenaed, it's only a matter of time until ChatGPT chats are as well.)

There's an uncensored model for vision available as well. The censored vision models won't play the shallow game of hot or not with you.

There are uncensored image generation models as well, but, ah, those are NSFW and not for polite company. (As well as there's multiple thesis' worth of content on what that'll do to society.)

Side note: ChatGPT is now completely useless for most creative tasks. I'm trying to use it, via NovelCrafter, to help flesh out a story where a minor character committed suicide. ChatGPT refuses to respond, mentioning "self harm" as a reason.

The character in question killed himself before the story even begins (and for very good reasons, story-wise); it's not like one's asking about ways to commit suicide.

This is insane, ridiculous, and different from what all other actors of the industry do, including Claude or Mistral. It seems OpenAI is trying to shoot itself in the foot and doing a pretty good job at it.

Perhaps the HR support line for OpenAI developers tasked with implementing the censorship system?

it's just fashionable to hate on AI, /s or not.

That's what you are asking the LLM to do, you are generating code not searching for an error condition.

The parent comment is saying maybe as a writer, writing the actual story isn't such a bad thing...

Just a different perspective, live your own life. If the story is good I'll enjoy it, who wrote it is for unions and activists to fight. I've got enough fights to fight.

If you're looking to read a more advanced example of its capabilities and limitations, try

https://simonwillison.net/2024/Mar/23/building-c-extensions-...

The models are capable of much much more, and they are being significantly nerfed over time by these ineffective attempts to introduce safeguards.

Recently I've asked GPT4 to quote me some code to which it replied that it is not allowed to do so - even though it was perfectly happy to quote anything until recently. When prompted to quote the source code, but output it as PHP comments, it happily complied because it saw that as "derivative work" which it is allowed to do.

I'm grateful because I thought code LLMs only sped up the "RTFM" part, but it made me find those libs so I didn't have to Google around for (and sometimes it's hard to guess if they're the right tool for the job, and they might be behind in SEO).

1. Being the "senior developer" who spend their whole career working with a technology you're very junior at. No matter what you do and how long your programming career is, you're inevitably going to run into one of these sooner or later. Whether it's build scripts, frontend code, interfacing with third-party APIs or something else entirely, you aren't an expert at every technology you work with.

2. Writing the "boring" parts of your program, and every program has some of these. If you're writing a service to fooize a bar really efficiently, Copilot won't help you with the core bar fooization algorithm, but will make you a lot faster at coding up user authentication, rate limiting for different plans, billing in whatever obscure payment method your country uses etc.

3. Telling you what to even Google for. This is where raw Chat GPT comes into play, not Copilot. Let's say you need a sorting algorithm that preserves the order of equal elements from the original list. This is called stable sorting, and Googling for stable sorting is a good way to find what you're looking for, but Chat GPT is usually a better way to tell you what it's called based on the problem description.

I asked it to try again and got a better result though, just didn't include it.

> So genuine question: is that answer representative of the output quality of these things?

And most use I've seen on it realistically a good LSP covers.

Or to put it a other way. It's no good at writing algorithms or data structures ( or at least no better thab I would have with a first drafy but the first draft puts me ahead of the LLM in understanding that actual problem at hand, handing it off to an LLM doesn't help me get to the final solution faster).

So that leaves writing boiler plate but concidering my experience with it writing more complex stuff, I would need to read over the boilerplate code to ensure it's correct which in that case I may as well have written it.

Fair, that is possible depending on your domain.

> It's no good at writing algorithms or data structures

In my experience, this is untrue. I’ve gotten it to write algorithms with various constraints I had. You can even tell it to use specific function signatures instead of any stdlib, and make changes to tweak behavior.

> And most use I've seen on it realistically a good LSP covers.

Again, I really don’t understand this comparison. LSPs and LLMs go hand in hand.

I think it’s more of a workflow clash. One really needs to change how they operate to effectively use LLMs for programming. If you’re just typing nonstop, maybe it would feel like Copilot is just an LSP. But, if you try harder, LLMs are game changers when:

- maybe you like rubber ducking

- need to learn a new concept and implement it

- or need to glue things together

- or for new projects or features

- or filling in boilerplate based on existing context.

https://godbolt.org/z/s9Yvnjz7K

I mean I could write the algorithm by hand pretty quickly in C++ and would follow the exact same thought pattern but also deal with the edge cases. And factoring in the loss of productivity from the context switch that is a net negative. This algorithm is also not generic over enough cases but that is just up to the prompt.

If I can't trust it to write `strip_whitespace` correctly which is like 5 lines of code, can I trust it to do more without a thorough review of the code and writing a ton of unit tests... Well I was going to do that anyway.

The argument that I just need to learn better prompt engineering to make the LLM do what I want just doesn't sit with me when instead I could just spend the time writing the code. As I said your last point is absolutely the place I can see LLMs being actually useful but then I need to spend a significant amount of time in code review for generated code from an "employee" who is known to make up interfaces or entire libraries that doesn't exist.

> Write me an algorithm in C++ which finds the begin and end iterator of a sequence where leading and trailing whitespace is stripped. Please write secure code that handles any possible edge cases.

It gave me this:

https://chat.openai.com/share/55a4afe2-5db2-4dd1-b516-a3cacd...

I'm not sure what other edge cases there might be, however. This only covers one of them.

In general, I've found LLMs to be marginally helpful. Like, I can't ever remember how to get matplotlib to give me the plot I want, and 9 times out of 10 GPT-4 easily gives me the code I want. Anything even slightly off the beaten path, though, and it quickly becomes absolutely useless.

Free GPT I get https://chat.openai.com/share/f533429d-63ca-4505-8dc8-b8d2e7... which has exactly the same problem as my previous example and doesn't consider the string of all whitespace.

Sure GPT4 is better at that, it wasn't the argument made.

The example you gave absolutely was the code I would write on a first draft since it does cover the edge cases (assuming we aren't dealing with the full UTF charset and all that could be considered a space there).

However this is code that is trivial to write in any language and the "Is it that hard to input a prompt into the free version of ChatGPT and see how it helps with programming? " argument doesn't hold up. Am I to believe it will implement something more complex correctly. This is also code that would absolutely be in hundreds of codebases so GPT has tons of context for it.

> My guess is that this was generated using GPT4?

This is a good guess, since I stated outright that I used GPT-4, and then mentioned GPT-4 later on in the comment.

I'm not sure there isn't a buffer overflow in the vector_decode code he showed there, likewise I don't see any error checks on the code and I am not familiar with the sqlite api to even know whether errors can be propagated upwards and what error conditions would mean in that code.

This code is probably fine for a quick side project but doesn't pass my smell test for anything close to production ready code.

I definitely would want to see a lot of unit tests around the decode and encode functions with fuzzing and to be honestly that would be the bulk of the work here. That and documentation on this code. Even though he encode function looks correct at first glance.

I also don't see an easy way to actually unit test this code as it is without actually running it through sqlite which outs a lot of dependencies on the unit test.

I would either need to spend a lot more time massaging gpt to get this to a point where I would be fine shipping the code or you know just write it myself.

Regarding your point around being able to whip up the code yourself - the point is to have a decent starting point to save time and energy. Like you said, you know the edge cases so you could skip the boring parts using GPT and focus purely on fixing those. Though, with more prompting (especially providing examples), GPT can also handle that for you.

I have nearly 2 decades of experience as a developer and it took me a while to reorient my flow around LLMs. But now that I have, it’s truly gamechanging.

And since you asked, here’s my system prompt:

You are an experienced developer who follows industry standards and best practices. Write lean code and explain briefly using bullet points or numbered lists. Elaborate only when explaining concepts or making choices. Always mention which file and where to store provided code.

Tech Stack: < insert all the languages, frameworks, etc you’d like to use >

If I provide code, highlight and explain problematic code. Also show and explain the corrected code.

Take a deep breath and think step by step.

Also, always use GPT4 and customize the above to your style and liking.

There is some code I would really prefer not to write that is a decent test case for this and won't expose company code to GPT. Will give feedback when I am done. Maybe you are correct.

Though it does come a little off as a comparison. How about programming assistance via asking a colleague for help, Stack Overflow, or online references, code examples, and other such things, which are closer to what the LLM would provide than LSP and treesitter?

That's funny. It makes me wonder how these statistical mad libs machines will handle the gradual boundaries nature gives us. Almost all mammals give birth live, but not all. Nearly all mammals had mammalian parents, but not all.

Daniel Dennett was making this argument for why we haven't developed reasonable models for the nature of consciousness. It's because we're so sure there will be an absolute classification, and not a gradual accumulation of interacting systems that together yield the phenomenon.

For everyone else, I wish they experience these (locally or elsewhere) weak LLMs atleast once before using the commercial ones just to understand various failure modes and to introduce a healthy dose of skepticism towards the results instead of blindly trusting them to be the facts/truth.

In my case I got the runtime detection issue (explained in the README "gotcha" section). Solved my running "assimilate" [3] on the downloaded llamafile.

    [1] https://github.com/Mozilla-Ocho/llamafile/
    [2] https://huggingface.co/jartine/Mixtral-8x7B-Instruct-v0.1-llamafile/resolve/main/mixtral-8x7b-instruct-v0.1.Q5_K_M.llamafile?download=true
    [3] https://cosmo.zip/pub/cosmos/bin/assimilate

To a first order approximation, LLMs are bandwidth constrained. We can estimate single batch throughput as Memory Bandwidth / (Active Parameters * Parameter Size).

An 8-bit quantized Llama 2 70B conveniently uses 70GiB of VRAM (and then some, let's ignore that.) The M3 Max with 96GiB of VRAM and 300GiB/s bandwidth would have a peak throughput around 4.2 tokens per second.

Quantized models trade reduced quality for lower VRAM requirements and may also offer higher throughput with optimized kernels, largely as a consequence of transfering less data from VRAM into the GPU die for each parameter.

Mixture of Expert models reduce active parameters for higher throughput, but disk is still far too slow to page in layers.

If nothing else it does a great job of demystifying them. They feel a lot less intimidating once you've seen a small one running on your computer write a terrible haiku and hallucinate some non-existent API methods.

Totally. I recently asked a locally-run "speed" LLM for the best restaurants in my (major) city, but it spit out restaurants opened by chefs from said city in other cities. It's not a thing you'd want to rely on for important work, but is still quite something.

Between that and dirt cheap storage prices, it is possible to have a local, offline copy of more human knowledge than one can sensibly consume in a lifetime. Hell, it's possible to have it all on one's smartphone (just get one with an SD card slot and shove a 1+ Tb one in there).

First there is a leaderboard for embeddings. [1]

Even then, it depends how you use them. Some embeddings pack the highest signal in the beginning so you can truncate the vector, while most can not. You might want that truncated version for a fast dirty index. Same with using multiple models of differing vector sizes for the same content.

Do you preprocess your text? There will be a model there. Likely the same model you would use to process the query.

There is a model for asking questions from context. Sometimes that is a different model. [2]

Are LLMs unable to distinguish between real life and fantasy? What prompts have you thrown at them to make this determination? Sending a small fairy tale and asking the LLM if it thinks it's a real story or fake one?

There’s so much subtle knowledge like the way a mother learns to calm her child or the way a carpenter learns to work different kinds of wood which may be written down in part, but may also be learned through lived experience or transferred from human to human such that little of it gets written down and posted online.

My worst case was with a guy who asked me to write an arbitrage betting bot. When I asked how to calculate coeffs, he pointed at two values and said "look, there , there thinks for a minute then it's !". When I asked how exactly did he calculate it, he simply repeated with different numbers.

Once upon a time I was good at a video game. Everyone realized that positioning is extremely important in this game.

I have good positioning in that game and was asked many times to make a guide about positioning. I never did, because I don't really know how. There is too much information they you need to convey to cover all the various situations.

I think you would first have to come up with a framework on positioning to be able to really teach this to someone else. Some kind of base truths/patterns that you can then use to convey the meaning. I believe the same thing applies to a lot of these processes that aren't verbalized.

Imagine someone learning English has written "The experiment reached it's conclusion" and you have to correct their grammar. Almost any english speaker can correct "it's" to "its" but unless they (and the person they're correcting) know a bunch of terms like 'noun' and 'pronoun' and 'possessive' they'll have a very hard time explaining why.

Language is so good, that an artificial language tool, without any understanding of these mental processes, can appear semi-intelligent to us.

A few people unable to do this serialization doesn't mean much on the larger scale. Just that their ideas and mental processes will be forgotten.

From oral tradition to tablets to scrolls to books to mass produced books to digital and now these LLMs, I think it’s still a good idea to preserve what we have the best we can. Not as a replacement, but a hedge against a potential library of Alexandria incident.

I could imagine a time in the near future where the models are domain-specific, and just like there are trusted encyclopedia publishers there are trusted model publishers that guarantee a certain level of accuracy.

It’s not like reading a book, but I for sure had an easier time learning golang talking with ChatGPT than a book

What would cause a Library of Alexandria incident wiping out all human knowledge elsewhere, that would also allow you to run a local LLM?

What you're really asking is, what could cause a modern Library of Alexandria incident? But the fact is we keep the only copy of too many things on the servers of the major cloud providers. Which are then intended to have their own internal redundancy, but that doesn't protect you against a targeted attack or a systemic failure when all the copies are under the same roof and you lose every redundant copy at once from a single mistake replicated in a monoculture.

Or more Sci-fi related, some hyper computer virus that ends up infecting all internet connected devices.

Not too far fetched if we can conceive of some AI enabled worm that mutates depending on the target, I could imagine a model of sorts being feasible within the next 5-10 years

Venture capital in tech is a good example of this. The book knowledge is effectively globally distributed and almost free, effectively success happens in a few geographically concentrated counties.

Yes - the available training data is essentially mostly a combination of declarative knowledge (facts - including human-generated artifacts) and procedural knowledge (how to do things). What is missing is the learning process of taking a description of how to do something, and trying to apply that yourself in a specific situation.

No amount of reading books, or reading other people's blogs on how they did something, can avoid the need for hands-on experience if you want to learn how to do it yourself.

It's not just a matter of information that might be missing or unclear in instructional material, including how to cope with every type of failure and unexpected outcome, but crucially how to do this yourself - if you are to be the actor, then it's the predictive process in your mind that matters.

Partly for this reason, and partly because current AI's (transformer-based LLMs) don't support online learning (try & fail skill acquisition), I think we're going to see two distinct phases of AI.

1) The current "GenAI" phase where AI can only produce mash-ups of things it saw in it's pre-training data, augmented by similar "book learning" provided in-context which can be utilized by in-context learning. I'd characterize what this type of AI to be useful for, and capable of, as "automation". Applying that book (incl. anecdotal) knowledge to new situations where mash-up is all you need.

2) The second phase is where we have something closer to AGI, even if still below human level, which is no longer just a pre-trained transformer, but also has online learning and is agentic - taking actions predicated on innate traits like curiosity and boredom, so that given the book knowledge it can (& will!) then learn to apply that by experimentation/practice and learning from its own mistakes.

There will no doubt be advances beyond this "phase two" as well, but it seems we're likely to be stuck at "phase one" for a while (even as models become much better at phase one capabilities), until architectures fundamentally advance beyond transformers to allow this type of on-the-job training and skill acquisition.

In general AI in 2024 is not even close to understanding these ideas, nor does any AI developer have a clue how to build an AI with this understanding. The best we can do is imitating object permanence for a small subset of perceptible objects, a limitation not found in dogs or spiders.

I can't help but feel we're at the "aliens watch people eat from space and recreate chemically identical food that has no taste" phase of AI development.

However, there is a broader context: this is supporting an argument against physicalism, and in this light it falls apart. There are a couple of missing bits required to complete the experiment in this context. The understanding that knowledge comes in 2 varieties: direct (actual experience) and indirect (description by one with the actual experience using shared language). This understanding brings proper clarity to the original argument, as we are aware - I think - that language is used to create compressed representations of things; something like a perceptual hash function.

The other key bit, which I guess we've only considered and extensively explored after the argument was formulated, is that all information coming in via the senses goes to the brain as electrical signals. And we actually have experimental data showing that sensory information can be emulated using machines. Thus, the original argument, to be relevant to the context, should be completed by giving Mary access to a machine that she can program to emulate the electrical signals that represent color experience.

I posit that without access to that hypothetical machine, given the context of the experiment, it cannot be said that Mary has "learned everything there is to learn about color". And once she has comprehensively and correctly utilized said machine on herself, she will gain no new knowledge when she is exposed to the world of color. Therefore this experiment cannot be used as an argument against physicalism as originally intended.

And so without the correct amount of salt and/or spices, we're talking about food that's very similar, and not identical.

But, fair point. The gist I was trying to get across is that I don't even know what a plant smells like to you, and you don't know what a plant smells like to me. Those aren't comparable with any objective data. We make guesses, and we try to get close with our descriptions, which are in words. That's the best we can do and we share our senses. Asking more from computers seems overly picky to me.

It gave me a new mental model for LLMs rather than a "spicy autocomplete" or whatever, I now think of it as "a lossy compressed database of knowledge". Like you ran the internet through JPEG at 30% quality.

In this scenario you’d need to also be left with a big chunk of compute, and power infrastructure. Since ChatGPT is the front end of the model you’d also need to have the internet still going in a minimum capacity.

But really, didn't you catch the meaning of parents message, or are you being purposefully obtuse?

Given the way LLMs are right now, I suspect there will be lot of failed experiments and the kingdom of science will not advance that quick.

It’s more likely that it wouldn’t even start. The first step to any development was figuring out nitric acid as the cure to the petrification. Good luck getting any LLM to figure that out. Even if it did, good luck getting any of the other characters to know what to do with that information that early on.

Both the context window and output structure are, in my opinion, massive impedance mismatches for the emergent intellect embedded in the weights of the model.

If there were a way to match the impedance, I strongly suspect we'd already have AGI on our hands.

This is a fundamental limitation, much more fundamental than appears at first. It means that the only way to touch the model, and for the model to touch the world, is through the tokenizer (also, btw, why tokenizer is so essential to model performance). Touching the world through a tokenizer is actually quite limited.

So there is an intelligence in there for sure, but it is locked in an ontology that is tied to its interface. This is even more of a limitation than e.g. weights being frozen.

That's not familiar behavior. Nor is the counting reddit derived output. It's also not familiar for a single person to have the breadth and depth of knowledge that ChatGPT has. Sure, some people know more than others, but even without hitting the Internet, it has a ridiculous amount of knowledge, far surpassing a human, making it, to me, alien. though, it's inability to do math sometimes is humanizing to me for some reason.

ChatGPT's memory is also unhuman. It has a context window which is a thing, but also it only knows about things you've told it in each chat. Make a new chat and it's totally forgotten the nickname you gave it.

I don't think of HR Geiger's work, though made by a human, as familiar to me. it feels quite alien to me, and it's not just me, either. Dali, Bosch, and Escher are other human artists who's work can be unfamiliar and alien. So being created by our species doesn't automatically imbue something with familiar human processes.

So it dot products, it matrix multiplies, instead of reasoning and understanding. It's the Chinese room experiment on steroids; it turns out a sufficiently large corpus on a sufficiently large machine does make it look like something"understands".

Math is mostly impeded by the tokenization, but it would still make more sense to adapt them to use RAG to process questions that are clearly calculations or chains of logical inference. With proper prompt engineering, they can process the latter though, and deviating from strictly logical reasoning is sometimes exactly what we want.

The ability to reset the text and to change that history is a powerful tool! It can make the model roleplay and even help circumvent alignment.

I think that LLMs could one day serve as the language center of an AGI.

"Alien" still works, but then you might have to add all the context at length, as you've done in this last comment.

    AI researchers unveil alien intelligence

The chat interfaces (Claude, ChatGPT) certainly have a particular style of writing, but the underlying LLMs are definitely capable of impersonating as our species in the medium of text.

I saw this talk with Geoffrey Hinton the other day and he said he was astonished at the capabilities of ChatGPT-4 because he asked it what the relationship between a compost heap and a nuclear bomb was, and he couldn't believe it answered, he really thought it was proof the thing could reason. Totally mind blown.

However I got it right away with zero effort.

Either I'm a super genius or this has been discussed before and made it's way into the training data.

Usual disclaimer: I don't think this invalidates the usefulness of AI or LLMs, just that we might be bamboozling ourselves into the idea that we've created an alien intelligence.

If an LLM can tell you the relatonship between a compost heap and nuclear bomb, that doesn't mean that was in the training data.

It could be because a compost heap "generates heat", and a nuclear bomb also "generates heat" and due to that relationship they have something in common. The model will pick up on these similar patterns. They tokens are positioned closer to each other in the high dimensional vector space.

But for any given "what does x have in common with y", that doesn't necessarily mean someone has asked that before and it's in the training data. Is that reasoning? I don't know ... how does the brain do it?

That answer was close btw !

Then after all is indexed you can pursue RAG on a richer set of metadata. Though i've got no idea what that stateful world view is.

> Australia won the 1987 Cricket World Cup. The 1986 date is incorrect; there was no Cricket World Cup in 1986. The tournament took place in 1987, and Australia defeated England in the final to win their first title.

https://chat.openai.com/share/e9360faa-1157-4806-80ea-563489...

I'm no cricket fan, so someone will have to correct Wikipedia if that's wrong.

If you want to point out that LLMs hallucinate, you might want to speak plainly and just come out and say it, or at least give a real world example and not one where it didn't.

    >>> who won the 1986 Cricket World Cup
    India
    
    >>> who won the 1986 Cricket World Cup
    Australia
    
    >>> who won the 1986 Cricket World Cup
    New Zealand
    
    >>> who won the 1986 Cricket World Cup
    West Indies
    
    >>> who won the 1986 Cricket World Cup
    England

> Brooklyn Nets won the 1986 Cricket World Cup by scoring 46 yards in only 3 frames

LLM's hallucinations are insidous because they have the ring of truth around them. yards and frames aren't cricket terms, so we're off to the races with them.

For fact retrieval you need to use temperature 0.

If you don't get the right facts then try 34b, 70b, Mixtral, Falcon 180b, or another highly ranked one that has come out recently like DBRX.

It's like how you have to set x264 to the anime tuning or the film tuning depending on what you run it on.

A year ago, a lot of this information only existed on the internet, and would have been nearly impossible to recover in any cohesive unfragmented form if the lights were to ever go out on our civilization.

Now the problem space has moved simply to "find a single solitary PC that will still boot up", and boom, you have access to everything.

I think we just created our Rosetta stone.

If you want a friendly but fallible UI to that dump, download an LLM and build a simple ReAct framework around it with prompting to use the wikipedia dump for reference.

https://www.cs.utexas.edu/users/pingali/CS378/2008sp/papers/...

>> I believe the trick with CPU math kernels is exploiting instruction level parallelism with fewer memory references

It's the collection of tricks to minimize all sort of cache misses (L1, L2, TLB, page miss etc), improve register reuse, leverage SIMD instructions, transpose one of the matrices if it provides better spatial locality, etc.

This is great. I love the idea of measuring performance differences in “years of Moore’s law.”

Twenty years puts the delta in an easy to understand framework.

I have come to appreciate "slowness" of Python. It trades speed for legibility, which is a great compromise once you have really fast native libraries one import away. Best of both worlds.

(I've only tested this once, and my naive hand-written C++ was still twice as fast as Numpy, but that was only on one specific task.)

Being smart in how I use Postgres indexing (and when to disable it outright) has more performance impact than the actual language doing the plumbing.

While running tiny llama does indeed count as running a language model, I’m skeptical that the capabilities of doing so match what most people would consider a baseline requirement to be useful.

Running 10 param model is also “technically” running an LM, and I can do it by hand with a piece of paper.

That doesn’t mean “you don’t need a computer to run an LM”…

I’m not sure where LM becomes LLM, but… I personally think it’s more about capability than parameter count.

I don’t realllly believe you can do a lot of useful LLM work on a pi