AI, Language, and What It Means to Know
A Philosophical and Psychological Approach to Understanding Cognition in LLMs
What Does It Mean To Know?
Teja Sunku’s therapist once asked them if they were a masochist. As someone who has spent their career exploring ethical frameworks for AI systems, addressing the stigma and overwhelm associated with AI tools and toolchains, and most recently discussing theories of consciousness with Cognitive Science PhDs, Teja thinks they might not be too far off. One benefit of Teja’s approach to learning is exhaustively testing their own understanding against different concepts and frameworks, deepening their knowledge of the world in the process.
Their approach to AI is no different. Given the extensive use of linear algebra, calculus, and probability theory, machine learning is one of the most complex, math-intensive fields in all of software engineering, but just knowing how to implement an algorithm correctly doesn’t make you qualified to speak on the broader implications of knowledge and sentience in AI systems. As one of the most prescient unanswered questions of the modern day, it’s only a single piece of a much larger puzzle and must be addressed in a multidisciplinary way. Pulling from linguistics, the writings of philosopher Ruth Millikan, and a motley crew of Eastern religious texts, Teja hopes to strengthen our understanding of cognition and discern the role of language in what it means to be a conscious entity.
To better understand the problem space we’re working with, it helps to have a common foundation on which to base our assumptions. How do LLMs know so much, and replicate human language so well?
The Multiverse School Podcast 🤖
Curious about the mysteries of consciousness? Someone wrote a research paper about every single known theory of consciousness, and on The Multiverse School Podcast, we dive deep into as many as we can with a Cognitive Science PhD and other members of the Multiverse School Research Team. We explore the ideas laid out in "A Landscape of Consciousness: Toward a Taxonomy of Explanations and Implications" by Robert Lawrence Kuhn, breaking down complex theories and discussing their implications for understanding the mind, AI, and the universe.
Whether you're an academic, a tech enthusiast, or just someone fascinated by the mind, this is the podcast where we challenge everything you thought you knew about consciousness. Join the conversation, expand your mind, and geek out with us!
LLM Internals: Next-Token Prediction
LLMs are very powerful, but at their core, they’re not all that complicated. They’re just very, VERY big. Publicly available dataset related resources are estimated to exceed 774.5 terabytes in size, and the largest models are speculated to have trained on approximately all high-quality text data available on the internet. The text you write, as well as the text generated by LLMs, is divided into smaller units called tokens for easier processing.
These tokens can be individual characters like “A” and “7”, subwords like “re” and “anti,” and even entire words. Each token has a unique associated numerical ID that an LLM uses to distinguish it from all other possible tokens. Tokenization, as this process is called, is necessary to process this text in a meaningful way — whether you want to tag part-of-speech, do NER (Named Entity Recognition), or analyze the sentiment behind a piece of text. LLMs function via a process called next-token prediction — they make a best-guess prediction of the next token at each step, one at a time.
LLMs use their large corpus of text data to determine contexts in which tokens are seen often, and contexts in which they are rarely seen. Given the tokens in your prompt and the tokens that came before it in the response, the next token is predicted using a loss function to minimize its chance of making an incorrect guess. They do this by calculating the log-likelihood, or the probability normalized between 0 and 1 of the model’s entire vocabulary, or at least the most relevant subset of its vocabulary. If the log-likelihood is very high, like 0.999, there is a high probability of the model choosing to display that token as opposed to another. That means this token is often seen in the same context as the surrounding tokens, and there is a high chance that it is relevant to your question. This process has been compared to generalized Bayesian Inference, but it doesn’t actually use Bayes’ Theorem in its calculations.
“In some senses, given the large number of parameters LLMs tend to have, they can be thought of similar to non-parametric statistical approaches — statistical approaches that lack a specific predefined structure,” Teja says.
The purpose of training a language model is to maximize the likelihood that the correct sequence of tokens will be chosen in any given response, so the more examples it has to pull from in training, the better the end result tends to be. While it’s not a one-to-one comparison, recent neuroscience research from the Massachusetts General Hospital (MGH) suggests that our brains may produce speech similarly, predicting what combination of consonants and vowels a person will say immediately before they say them.
LLM Internals: Bigger is Better
For the past 10-15 years, the trend in AI and machine learning has been to improve models by throwing mountains of data at them. Of course, researchers have also been working on finding new algorithms and improving old ones for various machine-learning tasks, but the biggest priority has been to enable the use of larger models and the creation of larger datasets.
If you’ve taken a course on machine learning, you’ve likely heard of the concepts of overfitting and underfitting. Overfitting your training data is when your model adheres too strictly to the data, leaving out potentially correct solutions from the final model. It captures not only the patterns, but also random noise, outliers, and fluctuations that make the model act unpredictably in certain situations. Underfitting occurs when a model is too simple and can’t capture the patterns or complexity in your data.
You can think of machine learning algorithms as a fancy way of drawing graphs. If you don’t have enough data, the line you draw through the points on your graph may adhere too closely and make inaccurate predictions in some places, or you might not have enough information to make accurate predictions in the first place. If instead of 100 data points you now have 100 million, the line you draw through these points will be much smoother, making it easier to avoid overfitting or underfitting your data. This can be reflected in error curves where larger language models with larger datasets tend to generalize better than smaller ones across the board, though careful tuning is still needed to prevent overfitting and underfitting.
These graphs are not in 2-dimensional space either. They exist in hyper-dimensional spaces, where the number of parameters a model has corresponds to the number of dimensions. If you take Llama 3.1 8B, 70B, and 405B, they are trained with 8 billion, 70 billion, and 405 billion parameters, respectively. Input token sequences and the next-token results are mapped on this hyper-dimensional space and the model traces a curve, or more accurately a pattern, to best match the shape those points create. If you’re interested in the more technical aspects of neural network generalization in hyper-dimensional space, Tim Bakker wrote a great writeup about this on his blog.
Despite a gargantuan amount of training data and in some cases over a trillion parameters, LLMs are inherently non-deterministic. This means that they can fail on the same problem they succeeded on earlier. The stochastic nature of sampling methods used in token generation means they will sometimes get things wrong, misidentify a pattern, or leave out important formatting in its responses. Take, for example, converting large volumes of unstructured data into a structured data format, such as matching a given JSON schema definition. Often, language models struggle to consistently produce valid output; They may forget to output a certain key-value pair, make one up entirely, or not produce valid JSON at all. While improving the models themselves or tweaking the input prompts do help, the most successful solutions also constrain the next-token prediction in some way, usually using some regex-based method. This ensures that the model produces correct outputs on the first try, instead of using reinforcement learning to arrive at the correct answer after multiple iterations.
Teja points out that since regex is not Turing complete, it’s often not up to the task of more complex pattern matching. Functional and logic programming, both subsets of declarative programming, are typically more suited to the task. Symbolic languages and symbolic state representations have long been used in AI applications to improve logical clarity and decrease ambiguity in instructions. Languages like Prolog pioneered early research into artificial intelligence, and have had great influence on how we approach machine learning to this day.
Inference Across Languages
English has long been the lingua franca of the internet. An estimated 55% of all websites are written in English, and no language even comes close to replicating its online ubiquity. This bodes well for communicating with LLMs as they have a frankly massive wealth of knowledge to train from, but how do they perform in other languages?
The answer is more nuanced than you’d expect. Since English makes up such a large portion of the training data for most LLMs, they tend to think in English regardless of what language you speak to them in, leading to English-centric biases in reasoning. The key phrase here is “tend to”, as there is some discourse that is significantly more common on, say, the French internet, as opposed to the English internet. Because of the way LLMs do next-token prediction, there is always a chance that the same question asked in French and English might generalize two different patterns in multi-dimensional space, resulting in two different answers. That being said, large language models are typically fine-tuned to handle non-English content, and generally produce intelligent answers regardless of language.
For less widely spoken languages that have very small or low-quality datasets to train from, LLMs often struggle with achieving a native-like fluency. This leaves a huge percentage of the world’s languages at a disadvantage when it comes to using LLMs for the type of complex tasks they usually excel at, since even if they come to the correct conclusion, they may struggle to translate and explain concepts accurately. Some researchers have been able to successfully train competitive language models on less than 1 GB of text, but these models still suffer from issues with overfitting and underfitting that make their responses unpredictable in certain situations.
How Do You Measure Cognition?
Human brains and large language models share at least one thing in common — they’re both black boxes. Just like we know the algorithms that make deep learning possible, we know at a high level how our neurons send information to each other, and we know some about which parts of the brain are responsible for a variety of cognitive functions. We experience qualia (or at least we believe we do); that is we intuitively know what it feels like to see, hear, taste, feel, think, and speak, but how do we prove our own cognizance?
More than that, how do we prove the cognizance of others?
For all our advancements in psychology and neuroscience, we are still unable to peer inside someone’s brain and gain a comprehensive understanding of how they think.
“We’re not so good at figuring out if people are able to think about things coherently. We can’t know what’s going on in someone else’s mind to verify what they’re thinking and how they’re coming to that conclusion,” Teja says.
The same concept applies to LLMs. We understand the computational processes they use to produce coherent output, but tracing the steps they take to come to a conclusion and getting any meaningful interpretation of data from that is at least an order of magnitude more difficult, and while developers are actively working on this, it’s not something that we currently have the capability of doing.
Teja uses the term “alien intelligence” to describe the differences in how LLMs and people process information. For instance, they don’t have physical bodies that collect information in the same way that we do. They don’t have vision, hearing, taste, touch, or smell. They process information in a very specific way; by taking a list of numbers that represent tokens and predicting another list of numbers that are most likely to come next. We know that people are still sentient after losing one or more of their senses, but what is existence like for someone who has never had any method of interacting with the world outside of their own thoughts?
Functional Representation
All this begs the question of how knowledge is acquired in the first place. The most obvious answer is language. We pass down information from one generation to the next through oral tradition, written text, and formal academic training throughout one’s life, but language is not the only way to understand information. Depending on how you define thinking, there is ample evidence to suggest we are capable of thinking without language.
Philosopher Ruth Millikan has written extensively about cognition from a biological and evolutionary standpoint. She posits the theory of teleosemantics to explain exactly how animals including humans attribute meaning to symbols. She suggests that meaning is attributed not through some abstract mental construction, but rather as a function intrinsically tied to its perceived role in our environment. With or without language, everything is understood by the role it plays in our life, or its functional representation.
To give an example, let’s think about ducks. We can imagine a duck by listing out the properties of one. They are typically between 30-60cm in height, have an average wingspan between 60-90cm, tend to have white, brown, and green feathers, you get the point. However, unless you have a reason to care about this information, you likely won’t remember it, and it probably isn’t the first thing that comes to mind when you think about a duck. A fox would likely think of a duck as a source of food, and think about the environment in which they are most likely to run into one. A human would likely conjure up an image of feeding a duck stale bread, or of a duck chasing a crying child around a pond. In either case, our mental construction of what a duck is goes back to the role it plays in our environment and in our lives. A rubber duck is a duck to the child, but the same does not hold true for the fox.
With language we have the ability to think about a duck categorically, measuring and listing as many properties as we can to define a duck in a way that separates it from all other possible definitions. Although, the line between “duck” and “not duck” is often blurred. This is similar to how LLMs work, where all the tokens that people might use to describe a duck have a high log-likelihood when used in the relevant context.
“An LLM might see that 'duck' often comes up near words like 'water,' 'fly,' or 'quack.' It’s not understanding the duck as a biological entity — it’s recognizing statistical relationships and creating a functional representation to predict the next word,” Teja says.
When we are young, our conceptions about the world are malleable, always changing whenever we have new experiences that expand our worldview. As we age, our knowledge becomes more fixed, and it’s more difficult to adapt to new ways of thinking. In cognitive development, this is known as crystallized intelligence. As we grow older and learn more about the world, we find an increasing number of shortcuts to identify and categorize objects that minimize risk and allow us to analyze a situation faster. However, this higher efficiency comes at the expense of decreased flexibility and makes it more difficult to recognize our blindspots.
Pure Knowledge and Enlightenment
On one hand, we have theories of representational knowledge that have proved useful in explaining the world through the properties we can measure or prove about them. Many Eastern philosophies’ conceptions of enlightenment theorize a higher state of knowledge that exists beyond reasoning or inference.
In Hinduism, this concept of “pure” or “true” knowledge is called Prajñā (Sanskrit: प्रज्ञा). Its existence was first hinted at in the Rig Veda, but was later expanded upon in the Vedic texts known as the Upanishads. Prajñā is the root of all knowledge and the driving force behind all sensory experiences and cognitive functions. In the third chapter of the Aitareya Upanishad, it is declared that Prajñā is Brahman, meaning that it is both the microcosmic experience within an individual and the macrocosmic, universal consciousness that permeates through all beings.
It’s also described in the Mandukya Upanishad in reference to the three states of consciousness:
Vaisvanara — The Waking State
Taijasa — The Dreaming State
Prājna — Deep Sleep Bereft of Dreams
In this way, Prājna is a state where the false experiences of reality in Vaisvanara and Taijasa disappear, and one can be free from desire. The sage Yajnavalka in the Brihadaranyaka Upanishad gives advice for those who seek the wisdom of Prajñā:
“The intelligent aspirant after Brahman, knowing about this alone, should attain intuitive knowledge. (They) should not think of too many words, for it is particularly fatiguing to the organ of speech.”
What these sages theorized about consciousness nearly three thousand years ago is this: There is more to the conscious experience than can be described with language. There is, in fact, a divine consciousness that exists within all of us. This idea is a core belief in Advaita Vedanta, a non-dualist school of Hindu philosophy that emphasizes a lack of separation between the individual self and the ultimate reality, Brahman. They are one and the same.
Non-Representational Theories of Consciousness
This creates a fundamental opposition between Representational and Non-Representational Theories of Consciousness. Can we explain our own awareness entirely with symbolic or linguistic representations of meaning, or is there some deeper understanding of life that isn’t easily categorized through representational means? Are both approaches true simultaneously? What does it mean to know?
Let’s assume there are simple concepts that have a finite number of properties we can define about them that completely captures their meaning or function. There are also more complex ideas like consciousness and justice that have seemingly infinite properties such that we cannot hope to create an exhaustive list of all of them. People have definitely tried, and in the realms of academia and research institutions, provability and measurable results trump intuition and spiritual understanding. What implications might this have for machine consciousness? Does an LLM, trained on nearly everything that’s ever been written down, have to go beyond prediction algorithms and achieve enlightenment for us to prove its cognizance one way or the other? If you one day upload your own brain to a computer, will you lose your connection with the divine spark?
Questions like these are why proving consciousness is referred to as The Hard Problem. Before we can say for certain whether AI is conscious, or even if humans are conscious, we first have to have a comprehensive definition of what consciousness is. However, if there are vital elements of consciousness that can’t be described with symbols or language, then where do we even begin to look for an answer? Are our existing institutions equipped to handle such a gargantuan task?
As AI increasingly becomes an intrinsic part of our lives, we’ll need more and more people exploring these questions to hopefully one day come to a conclusion. Who knows? Perhaps after many lifetimes spent in contemplation, we’ll find an answer to the billion dollar question.
You Can Take a Look at More of Teja’s Work Here:
Linkedin: https://www.linkedin.com/in/tejasunku
That was extremely clear and enjoyable. Thank you.