What i meant was: if we are looking for something like intent or true agency in these large language models, i don’t know why they would have it but a model fit in excel would not. they are both essentially fitted curves.
the applications of a statistical model that can predict future words is, of course, very different from the applications of excel, or a curve fit in excel.
similarly, a computer programmed to solve differential equations and predict future behaviors of solar bodies is an amazing accomplishment that also has some amazing applications. but i dont imagine the program itself is intelligent, has intent or agency.
I don’t think we’re close to seeing the full breadth of what the current crop of LLMs can do.
But they don’t know “reality”, all they are doing is guessing the next best word.
We evolved imagination, then language to describe imagination to each other.
Llm are trying to skip the imagination part so they’ll be constrained.
The models don’t “hallucinate”, they don’t understand the difference between true and false so they’ll always produce false output, they can’t differentiate what false is.
Human intelligence has similar flaws though. We have all been there - believing something as absolutely correct…until we learn something new that causes us change our mind.
It does change how we think about compute. We know computers to be precise given the inputs provided. We expect humans to be biased. AI results in biased compute.
An interesting experiment I just tried - I sent gemini an outdoor picture with a blue sky and after three prompts, convinced it to generate a caption for the photo for a deep-red crimson sky. I lied, of course, and told Gemini it was wrong, and the picture was taken during a smoky sunset.
The issue is that many of us do not think there is anything magical going on inside our own heads. If we assume there is no soul, then there is no non-deterministic free will. Our agency is a sort of pde of our genes and our environment and itself. And on some level, everything we perceive, understand, think, believe, etc. are mathematical vectors on some level.
Of course, at the very least, the training process is very different…
–Humans are derived from animals, basically? Then educated by a smattering of language? And trained to be civilized?
–LLMs are a copy of a massive amount of language? Then trained to be civilized? And then are told to act like they have a fixed personality?
There are some really big differences there. If nothing else we can agree that LLMs are sometimes approximating a thing that animals have for real-- like feelings. But I find myself very skeptical that the “real” things we have mean very much in the end.
Specifically, I wonder if human brains are kind of like LLMs strapped onto animal brains. So we have “intent”, but our intent is also kind of bullshit.
What’s agonizing is that the do understand the idea of truth and falsehood. Like you can tell an LLM a complex story where some parts are true, some are false, some people are lying, some believe the lies, some don’t believe the lies, some are just pretending, etc. and the LLM can make sense of it all.
It can also fabricate lies, and has a sense of which ones you might believe.
But yeah, it can’t tell whether it is telling the truth, because like you say, its truth is always a guess.
But the way our thinking works is about as far away from llms as it is from excel. Our thinking is embodied and has a very different kind of relationship to the environment and to other people.
Even if our minds can be described as algorithms, our brains are not necessarily turing machines and so those algorithms may not be able to be run on computers.
I think this is a common way to think about it. i don’t think there is any good scientific reason for thinking that is true. in particular, a lot of effort by brilliant people was put into trying imagine science as a kind of logical language (logicial positivism in the 1930s) and this effort failed completely.
The point though is that science doesn’t seem to be able to be expressed as an algorithm. There is no calculus of knowledge as far as anybody can tell.
For these llms to really be able to think, thinking has to be the same thing as language. to know something is to be able to write it and vice versa. there is maybe an argument that this is true for math. it seems completely untrue for knowledge in general.
I think that’s a different problem than positivism. With positivism you’re trying to squeeze science down into perfectly exact and provable boxes. But LLMs don’t do that. They are loose with their words.
The problems you’re talking about are more like Qualia, and the Chinese Room Experiment, and P-Zombies. Where there’s a good case to be made, but it’s quite muddled.
I think positivism is a form of rationalism. the idea of rationalism is to turn science into a way of thinking, or a particular kind of method. descartes had his own way of doing it. so did the positivists. now we are saying it is the new kind of algorithm, namely the deep neural net, etc., determining the llm output. i’d argue the real “algorithm” involved is not the output of the llm, but the algorithm determining that output. it is highly logical and mathematical, i’d say. the language is just the empirical “data” upon which the training algorithm executes its calculus.
I can kind of see what you’re saying here, but I feel like there’s a difference between a ‘simple’ rationalist system and a irrational system made of rational atoms and external inputs. I think I’d need to revisit why the rationalist system is said to fail.
I think (and i am not a very good philosopher) that the idea of rationalism is to find a core set of principles or steps or method that you know is right. This “core” ignores all context, including experience, values, culture, etc.
I think the reason rationalism fails is because good thinking can never be completely decontextualized from experience, values and culture.
In the case of LLMs, the decontextualization is literal. They are cut off from the context of experience because that data does not exist and is not given to them. All they know of context is words, which is the prompt. And all they know of that prompt is other words.
I can show that the biomass of my fish species is related to the North Atlantic Oscillation. Unfortunately, I can’t figure out what the shape of that relationship is. I also have to cheat a bit and use different curves for different time periods.
Part of the trouble is a limited time series. Part is interactions with other parts of the environment. Part is a changing environment. A huge chunk is incomplete knowledge.
A month or two ago, I was at a framework meeting that kind of failed. One of the tasks for the meeting was to decide on a model that determined the age at which 50% of these fish were maturing at for each year. I think the team tried 30+ models (various forms of a logistic regression model or possibly a beta binomial model) and had a few solid options. Unfortunately the reviewers wanted the models to better account for some sampling factors. Most of the models that were shortlisted produced reasonable results, but they all created varying versions of their truth using the same dataset.
crossposted to annoyed thoughts.
We’ve been having problems for like weeks on the AI server. Stuff is just not working. All sorts of stuff. Weird results. Just a complete mess.
So I’ve been handing it off to the dev that’s running that half of things. He’s a young guy, out of school for a year or two, wants to code so he connected with me. But I think he doesn’t actualy know how to vibe code. Because, he hasn’t fixed the problems in weeks.
So this morning at about 7am I grabbed a coffee and seized control of the factors of production or something. Between a coffee and making breakfast, I vibe coded and fixed everything. All problems diagnosed, everything solved, everything working smoothly. ALl in about 2.5 hours.
I’m gonna have a conversatoin, I think he’s using Claude code like a scalpel instead of a nuclear warhead.
Well, going back to my original random speculation-- “what if human brains are kind of like LLMs strapped onto animal brains?”
It wouldn’t be that we have no data. Rather, we have all the same raw data that a deer or a rabbit has. So we “know” perceptions and feelings and objects and relationships. We have a map to transform the raw data into words. It is a mathematical function. We transform the data into words, and then process those words through an LLM.
The LLM is always on, running on a loop, generating words. Some of the words have to do with our animal-data, but sometimes the LLM is just yammering. The LLM also has a way to send the animal-brain instructions-- where to go, what to do, what to be afraid of, what words to say out loud. The animal brain listens, but not perfectly. There’s an id/ego/superego split between the two.
Along this speculation, the LLM is doing a lot of heavy lifting.
It is how we created the language map.
It is how we learn language-- passively training on word distribution.
It is how we come up with ideas, stories, jokes, etc.
It is how we reason-- and also why our reasoning tends to suck.
It’s how we hold and also share complicated ideas, values, art, math, science, culture, religion, money, goals, etc.-- anything that’s not just a feel.
Under this speculation, the “complicated ideas” are encoded in the same way they are in an artificial LLM. That is, they don’t need to correspond to anything in the outside world, but they are internally represented, and they inform our behavior. Creating the illusion (or reallity?) of agency. The agency is tied to our animal selves, but only remotely.
good thinking
Anyway, my speculation here is less about “good thinking”, and more about why our thinking is so rambling, stupid, fantastical, and nonsensical by default.
One big objection i have to this (or at least one possible version of this claim) is that, for us, words signify something. This is so true that we often forget the words and pay attention only to what is signified. For example, multilingual people might forget what language something was written in because they only remember the content.
For LLMs, words signify nothing. Instead they are objects in themselves that appear with other words in some statistical regularity.
In this way, LLMs treat words similarly to how people sometimes treat terms in formal languages that appear in logic and math. A “point” and “line” mean nothing except for the rules in the formal system relating them to each other. Such formal languages can then be mechanically applied with computation.
However, I think that even when we work with formal languages like that, most of us cannot help but imagine the words as signifying something. A point and a line are these pictures in our heads.
So the “language” used by LLMs seems fundamentally incompatible with the “language” used by us. I think this helps explain why LLMs can do some tasks we find very hard, but struggle with tasks we find easy.
I know we’re not neurologists here, but if you have a link, I’ll read it.
Mammals and birds can imagine a future state to assess next action. LLMs lack that.
LLMs don’t usually have to think about with future states. If you prompt them with some kind of time-based environment, I expect they’d do fine.
I want to add here that I don’t think humans are exactly LLMs attached to animal brains. Just that they’re closer than we suppose.
Obviously,
Animal brains exist on something of a continuum. Mammals can do all sorts of things, including learning some language.
Large Language Models are really, really large. They’ve been trained on trillions? of words and humans are raised on like millions. You can make an LLM smaller, but not that much smaller.