Since writing this recent post, I came across an article in Quanta which explores a related topic: “The Polyglot Neuroscientist Resolving How the Brain Parses Language.” Its protagonist, Ev Fedorenko, studies “how language works in the adult human brain,” and has explored the neurological mechanism which translates “between external perceptions…and representations of meaning.” Put another way, Fedorenko asserts that the part of the human brain solely responsible for language itself does not actually understand that language. Rather, it mediates between the brain regions responsible for higher level thought, which can be expressed in language, and the brain regions responsible for converting language into an external form, like speech or writing.
Not that her research suggests the human brain performs the same kind of pattern recognition and generation as LLMs; rather, that there is a region of the brain that functions similarly to the tasks to which LLMs are trained, mediating between ideas and their presentation. What I find most convincing about her research is the discovery that this “LLM-like” language processing region in the brain reacts to nonsense sentences the same way it does to meaningful sentences. It’s entirely possible to string a series of words together into a grammatically and syntactically correct sentence that is utterly meaningless – I’ve read supposedly serious prose which sometimes seems to exemplify this capacity of language – and Fedorenko’s work establishes that the actual language processing part of our brain can’t distinguish these meaningless applications of language from meaningful ones. That step requires the mediation of other brain regions associated with complex, abstract thought and higher reasoning.
This is a fascinating addition to the conversation about whether or not LLMs really “understand.” Based on the kinds of errors they tend to make, it seems very likely they are currently fulfilling a role similar to the one our own language-processing brain region fills: processing language without actually parsing and understanding it. If the human language processing region translates from thought to expression, then what LLMs and chatbots are doing is essentially adding another layer: abstraction->human language processing->expression to LLM->LLM language processing->expression from LLM. No wonder, then, that the results are not what we expect, and that the failure modes can be so strange and unexpected compared to what we might expect from a human.
The Quanta article explores Fedorenko’s work in much more depth, including an interview with her; I encourage you to give it a read and consider how it might relate to the discussion we began with our previous post about linguistics and understanding.
IGC Publishing 