With LLM-produced materials, we are currently forced to rely on belief that what we ask for is what we will get. We don't know. We can't prove it. For example, image generators have been given billions and billions of images and pictures of people and still they generate material with people that have three arms and eight fingers. There are guardrails in place in most image generators, but the LLM at the core of the machine doesn't know anything. It doesn't know that people don't have three arms. A child learns very quickly how many fingers and arms a person has. An AI does not. It has to be explicitly taught these things. This is fundamentally different. You put a human in the world for a while and it learns a tremendous number of things that we've taken for granted. An AI does not do this. You have to hold its "hand". This is OK with stuff that we can easily verify. But what happens when it's not easily verified? What happens when it's an MRI output or something much more complex and difficult to verify? Then we have to take it on faith that each of those shadows on the MRI is really a potential tumor and not just three extra fingers. I just don't think this technology is a viable path forward. I think that this technology base doesn't scale. We've managed to scale it pretty far with bloody-mindedness, but we can't seem to refine it. Well, we can refine it, but we have to teach these tools every single little detail about the world so that we can better believe their output. But just that we have to "believe" as part of our use of a technology is very, very different than scientific approaches. It's fundamentally different than other tools we've built. In other cases, most people had no idea how their tools worked, but at least <i>someone did</i>. It was possible to learn. With LLMs, the black box is a black box for everyone. On top of that, an emergent feature of these things is hallucinations---like the eight fingers and three arms---but also a tremendous inherent bias in the input, for which there is no reasonable solution. Because how do you correct a bias? To what would you correct it? If I ask an image generator for a a hot slut, it will 100% produce a nearly-naked woman, not a man. It will also not produce androgynous content. It dreams, but it dreams the dreams we taught it to dream. As noted in the tweet <a href="https://twitter.com/karpathy/status/1733299213503787018" author="Andrej Karpathy" source="Twitter">On the hallucination "problem"</a>, <bq>[LLMs] are dream machines. We direct their dreams with prompts. The prompts start the dream, and based on the LLM's hazy recollection of its training documents, most of the time the result goes someplace useful. It's only when the dreams go into deemed factually incorrect territory that we label it a "hallucination". It looks like a bug, but it's just the LLM doing what it always does. [...] An LLM is 100% dreaming and has the hallucination problem. A search engine is 0% dreaming and has the creativity problem. [...] An LLM Assistant is a lot more complex system than just the LLM itself, even if one is at the heart of it. [...] the LLM has no "hallucination problem". Hallucination is not a bug, it is LLM's greatest feature. The LLM Assistant has a hallucination problem, and we should fix it.</bq> Everything it does is hallucination. Some of it happens to hit close to what we consider to be a bullseye. The discussion in the video below expands on that point. <media href="https://www.youtube.com/watch?v=6LXw2beprGI&t=1526s" src="https://www.youtube.com/v/6LXw2beprGI&t=1526s" source="YouTube" width="560px" author="Outerbounds" caption="Making Large Language Models Uncool Again"> The following transcription is from about <b>25:30</b>: <bq>So something everybody I think pretty much agrees on, including Sam Altman, including Yann LeCun, is LLMs aren't going to make it. The current LLMs are not a path to AGI. They're getting more and more expensive, they're getting more and more slow, and the more we use them, the more we realize their limitations. We're also getting better at taking advantage of them, and they're super cool and helpful, but they appear to be behaving as extremely flexible, fuzzy, compressed search engines, which when you have enough data that's kind of compressed into the weights, turns out to be an amazingly powerful operation to have at your disposal.

[...] And the thing you can really see missing here is this planning piece, right? So if you try to get an LLM to solve fairly simple graph coloring problems or fairly simple stacking problems, things that require backtracking and trying things and stuff, unless it's something pretty similar in its training, they just fail terribly.</bq> We'll have to reevaluate the tech and try again. I imagine we'll futz around for a while first, letting some fools get spectacularly rich on it first, as we always do.