By Rohit Krishnan, he and I are both interested in the question of what LLMs can’t do and why. Here is an excerpt:
It might be better to say that LLMs demonstrate incredible intuition but limited intelligence. It can answer almost any question that can be answered in one intuitive swipe. And with enough training data and enough iterations, it can result in a facsimile of reasoned intelligence.
The fact that adding RNN-like binding seems to make a small difference, but by no means sufficient to solve the problem, at least in toy models, is an indication in this direction. But this is not enough to solve the problem.
In other words, there is a “goal drift” where, as more steps are added, the overall system starts doing the wrong things. As contexts increase, even given the history of conversations, LLMs struggle to determine what to focus on and what the goal actually is. Attention is not precise enough for many problems.
A closer answer is that neural networks can learn all kinds of irregular patterns. once you add external memory.
And:
In LLMs as in humans, context is what is rare.
Interesting everywhere.
