> Any project trying to use deterministic symbolic logic to represent the world fundamentally misunderstands cognition.
The counterposition to this is no more convincing: cognition is fuzzy, but it's not really clear at all that it's probabilistic: I don't look at a stump and ascertain its chairness with a confidence of 85%, for example. The actual meta-cognition of "can I sit on this thing" is more like "it looks sittable, and I can try to sit on it, but if it feels unstable then I shouldn't sit on it." In other words, a defeasible inference.
(There's an entire branch of symbolic logic that models fuzziness without probability: non-monotonic logic[1]. I don't think these get us to AGI either.)
Which word will I pick next in this sentence? Is it deterministic? I probably wouldn’t respond the same way if I wrote this comment in a different mood, or at a different time of day.
What I say is clearly not deterministic for you. You don’t know which word will come next. You have a probability distribution but that’s it. Banana.
I caught a plane yesterday. I knew there would be a plane (since I booked it) and I knew where it would go. Well, except it wasn’t certain. The flight could have been delayed or cancelled. I guess I knew there would be a plane with 90% certainty. I knew the plane would actually fly to my destination with a 98% certainty or something. (There could have been a malfunction midair). But the probability I made it home on time rose significantly when I saw the flight listed, on time, at the airport.
Who I sat next to was far less certain - I ended up sitting next to a 30 year old electrician with a sore neck.
My point is that there is so much reasoning we do all the time that is probabilistic in nature. We don’t even think about it. Other people in this thread are even talking about chairs breaking when you sit on them - every time you sit on a chair there’s a probability calculation you do to decide if the chair is safe, and will support your weight. This is all automatic.
Simple “fuzzy logic” isn’t enough because so many probabilities change as a result of other events. (If the plane is listed on the departures board, the prediction goes up!). All this needs to be modelled by our brains to reason in the world. And we make these calculations constantly with our subconscious. When you walk down the street, you notice who looks dangerous, who is likely to try and interact with you, and all sorts of things.
I think that expert systems - even with some fuzzy logic - are a bad approach because systems never capture all of this reasoning. It’s everywhere all the time. I’m typing on my phone. What is the chance I miss a letter? What is the chance autocorrect fixes each mistake I make? And so on, constantly and forever. Examples are everywhere.
To be clear, I agree that this is why expert systems fail. My point was only that non-monotonic logics and probability have equal explanatory power when it comes to unpredictability: the latter models with probability, and the former models with relations and defeasible defaults.
This is why I say the meta-cognitive explanation is important: I don’t think most people assign actual probabilities to events in their lives, and certainly not rigorous ones in any case. Instead, when people use words like “likely” and “unlikely,” they’re typically expressing a defeasible statement (“typically, a stranger who approaches me on the street is going to ask me for money, but if they’re wearing a suit they’re typically a Jehovah’s Witness instead”).
> I don’t think most people assign actual probabilities to events in their lives, and certainly not rigorous ones in any case.
Interesting. I don't think I agree.
I think people do assign actual probabilities to events. We just do it with a different part of our brain than the part which understands what numbers are. You can tell you do that by thinking through potential bets. For example, if someone (with no special knowledge) offered a 50/50 bet that your dining chair will break next time you sit on it, well, that sounds like a safe bet! Easy money! What about if the odds changed - so, if it breaks you give them $60, and if it doesn't break they give you $40? I'd still take that bet. What about 100-1 odds? 1000-1? There's some point where you start to say "no no, I don't want to take that bet." or even "I'd take that bet if we swap sides".
Somewhere in our minds, we hold an intuition around the probability of different events. But I think it takes a bit of work to turn that intuition into a specific number. We use that intuition for a lot of things - like, to calibrate how much surprise we feel when our expectation is violated. And to intuitively decide how much we should think through all the alternatives. If we place a bet on a coin flip, I'll think through what happens if the coin comes up heads or if it comes up tails. But if I walk into the kitchen, I don't think about the case that I accidentally stub my toe. My intuition assigns that a low enough probability that I don't think about it.
Talking about defeasible statements only scratches the surface of how complex our conditional probability reasoning is. In one sense, a transformer model is just that - an entire transformer based LLM is just a conditional probability reasoning system. The entire model of however many billions of parameters is all a big conditional probability reasoning machine who's only task is to figure out the probability distribution over the subsequent token in a stream. And 100bn parameter models are clearly still too small to hit the sweet spot. They keep getting smarter as we add more tokens. If you torture an LLM model a little, you can even get it to spit out exact probability predictions. Just like our human minds.
I think these kind of expert systems fail because they can't do the complex probability reasoning that transformer models do. (And if they could, it would be impossible to manually write out the - perhaps billions - of rules it would need to accurately reason about the world like chatgpt can.)
> I think people do assign actual probabilities to events. We just do it with a different part of our brain than the part which understands what numbers are. You can tell you do that by thinking through potential bets.
I think these are different things! I can definitely make myself think about probabilities, but that's a cognitive operation rather than a meta-cognitive one.
In other words: I think what you're describing as "a bit of work" around intuitions is our rationalization (i.e., quantification) of an process that's internally non-statistical, but defeasible instead. Defeasibility relationships can have priorities and staggerings, which we turn into fuzzy likelihoods when we express them.
My intuition for this comes from our inability to be confidently precise in our probabilistic rationalizations: I don't know about you, but I don't know whether I'm 57.1% or 57.01983% confident in an expression. I could make one up, but as you note with torturing the LLM, I'm doing it to "make progress," not because it's a true statement of probability.
(I think expert systems fail for a reason that's essentially not about probability reasoning, but dimensionality -- as the article mentions Cyc has at least 12 dimensions, but there's no reason to believe our thoughts have only or exactly these 12. There's also no reason to believe we can ever model the number of dimensions needed, given that adding dimensions to an encoded relation set is brutally exponential.)
>My intuition for this comes from our inability to be confidently precise in our probabilistic rationalizations: I don't know about you, but I don't know whether I'm 57.1% or 57.01983% confident in an expression.
LLMs are probabilistic and notoriously unable to be confidently precise in their probabilistic rationalizations.
> LLMs are probabilistic and notoriously unable to be confidently precise in their probabilistic rationalizations.
Sure. To tie these threads together: I think there are sufficient other different properties that make me reasonably confident that my thought process isn't like an LLM's.
(Humans are imprecise, LLMs are imprecise, thermometers are imprecise, but don't stick me or my computer in an oven, please.)
>Sure. To tie these threads together: I think there are sufficient other different properties that make me reasonably confident that my thought process isn't like an LLM's.
Doesn't have to be like an LLM's to be probabilistic
>I don't look at a stump and ascertain its chairness with a confidence of 85%
But i think you did. Not consciously, but i think your brain definitely did.
https://www.nature.com/articles/415429a https://pubmed.ncbi.nlm.nih.gov/8891655/
These papers don't appear to say that: the first one describes the behavior as statistically optimal, which is exactly what you'd expect for a sound set of defeasible relations.
Or intuitively: my ability to determine whether a bird flies or not is definitely going to be statistically optimal, but my underlying cognitive process is not itself inherently statistical: I could be looking at a penguin and remembering that birds fly by default except when they're penguins, and only then if the penguin isn't wearing a jetpack. That's a non-statistical set of relations, but its external observation is modeled statistically.
>which is exactly what you'd expect for a sound set of defeasible relations.
This is a leap. While a complex system of rules might coincidentally produce behavior that looks statistically optimal in some scenarios, the paper (Ernst & Banks) argues that the mechanism itself operates according to statistical principles (MLE), not just that the outcome happens to look that way.
Moreover, it's highly unlikely, bordering on impossible, to reduce the situations the brain deals with even on a daily basis into a set of defeasible statements.
Example: Recognizing a "Dog"
Defeasible Attempt: is_dog(X) :- has_four_legs(X), has_tail(X), barks(X), not is_cat(X), not is_fox(X), not is_robot_dog(X).
is_dog(X) :- has_four_legs(X), wags_tail(X), is_friendly_to_humans(X), not is_wolf(X).
How do you define barks(X) (what about whimpers, growls? What about a dog that doesn't bark?)? How do you handle breeds that look very different (Chihuahua vs. Great Dane)? How do you handle seeing only part of the animal? How do you represent the overall visual gestalt? The number of rules and exceptions quickly becomes vast and brittle.
Ultimately, the proof as they say, is in the pudding. By the way, the CyC we are all talking about here is non-monotonic. https://www.cyc.com/wp-content/uploads/2019/07/First-Orderiz...
If you've tried something for decades and it's not working, and it doesn't even look like it's working and experiments with the brain suggest probabilistic inference and probabilistic inference machines work much better than the alternatives ever did, you have to face the music.
> How do you define barks(X) (what about whimpers, growls? What about a dog that doesn't bark?)? How do you handle breeds that look very different (Chihuahua vs. Great Dane)? How do you handle seeing only part of the animal? How do you represent the overall visual gestalt? The number of rules and exceptions quickly becomes vast and brittle.
This is the dimensionality mentioned in the adjacent post, and it's true of a probabilistic approach as well: an LLM trained on descriptions of dogs is going to hallucinate when an otherwise sensible query about dogs doesn't match its training. As others have said more elegantly than I will, this points to a pretty different cognitive model than humans have; human beings can (and do) give up on a task.
(I feel like I've had to say this a few times in threads now: none of this is to imply that Cyc was a success or would have worked.)
>This is the dimensionality mentioned in the adjacent post
LLMs are only a few years old but symbolic ai was abandoned for NLP, Computer Vision etc long before that. Why ? Because the alternative was just that bad and more importantly, never seemed to really budge with effort. Companies didn't wake up one morning and pour hundreds of millions into LMs. In fact, NNs were the underdog for a very long time. They poured more and more money into it because it got better and better with investment.
There is zero reason to think even more dimensionality would do anything but waste even more time. At least the NN scalers can look back and see it work in the past. You don't even have that.
>an LLM trained on descriptions of dogs is going to hallucinate when an otherwise sensible query about dogs doesn't match its training. As others have said more elegantly than I will, this points to a pretty different cognitive model than humans have; human beings can (and do) give up on a task.
It doesn't take much to train LLMs to 'give up'. Open AI talk about this from time to time. It's just not very useful with a tendency to overcorrect. And humans hallucinate (and otherwise have weird failure modes) all the time. We just call them funny names like dunning kruger and optical illusions. Certainly less than current SOTA LLMs but it happens all the same.
>I feel like I've had to say this a few times in threads now: none of this is to imply that Cyc was a success or would have worked.
The point is not about Cyc. They're hardly the only attempt at non-monotonic logic. The point is that they should work much better than they do if there's anything to it. Again, forget recent LLMs. Even when we were doing 'humble' things like spelling error detection & correction, text compressors, voice transcription boosters, embeddings for information retrieval, recommenders, knowledge graph creation (ironically enough), machine translation services, etc these systems were not even in the conversation. They performed that poorly.
I think we're talking past each other. I'm not interested in defending symbolic AI at all; it's clear it's failed. All told, I would not say I'm particularly interested in any kind of AI.
I'm interested in theory of mind, and I think defeasibility with a huge number of dimensions is a stronger explanation of how humans behave and think than something resembling an LLM. This doesn't somehow mean that LLMs haven't "won" (they have); I just don't think they're winning at human-like cognition. This in turn does not mean we could build a better alternative, either.