Unconscious Uncoupling
Our relationship to knowledge, one another, and Claude
Nullius addictus iurare in verba magistri, / quo me cumque rapit tempestas, deferor hospes.
("Bound to swear on the words of no master, wherever the storm carries me, I put in as a guest.")
Horace, Epistles 1.1.14–15
The Internet as Abnormal Technology
In a number of ways, the internet is a bizarre technology. It is predicated on reciprocity; it is a network of networks that allows for practically instant communication, stores an unfathomable quantity of information,1 and hosts forums on which we spend substantial portions of our own lives. The internet is humanity’s primary store of knowledge, and our primary font of epistemic authority. It is uncountably larger than all the information that has ever been stored in all of the libraries which have ever existed.
We are now so accustomed to this state of affairs that we risk becoming numb to it. However, for our own sake, we must understand our unprecedented situation: sometime in the 1990s, the entire paradigm of our relationship to knowledge shifted. How could it not? Almost every piece of information our species has ever produced is now localized within your pocket. Knowledge ceased to be something precious: we were drowning in it.
One of the most effective ways to manage this problem was to create forums and communities dedicated to specific knowledge areas and interests: Stack Overflow for programmers, Bodybuilding.com for fitness, and LessWrong for Harry Potter fan-fiction. These communities did not just store and propagate knowledge, they also created it.
By the mid-2000s social media companies like Reddit, Twitter, and YouTube commercialized and popularized the internet-as-information-network for billions globally. The internet, of course, became an immensely valuable technology (arguably surpassing the noble fax machine) both in measurable and immeasurable ways. For example, we might be able to measure the revenue of social media companies, or Amazon, but it is hard to quantify the value provided by the internet to the knowledge economy.
This brings us to today: about 75% of humanity uses the internet; most of the knowledge we consume is online. Most importantly for this article, our epistemic functions center on the internet. We consider information preserved when it exists on the internet, and imperiled when it exists only in the world. Massive preservation projects, some organized by institutions and others by communities, have collected and organized several thousand years’ worth of knowledge. Obvious examples like Internet Archive and Wikipedia come to mind, as do illegal databases of books and academic articles.
The breadth of knowledge on the internet is not only valuable as preservation, but as a provider of something like epistemic autonomy. In other words, anyone with access to the internet can opt to pursue Marxism–Leninism–Maoism–Prachandaism, esoteric fascism, Ezra Kleinist liberalism, flat-eartherism, or any other epistemic thread. As long as you are one of the 6 billion with access to the internet, you have access to more knowledge and intellectual autonomy than anyone in the past.
It Takes Two to Make a Thing Go Right
The information on the internet was produced and posted by humans, including you and me. Why did we do that?
The places where we generate information on the internet usually fall into two categories: either they are more traditional repositories of knowledge like encyclopedias, books, and journal articles, or they are community-based like Reddit, Wikipedia, or any number of other websites. When we produce knowledge for the former, it is usually because we have some direct financial incentive to do so: a publishing deal, a contract, or (in the case of a graduate student) the hope that you may eventually get a job. The second category is much stranger.2
Of course, these internet forums are models of real life organizations. People really did used to meet up in-person and share information about TV shows, computers, philosophy, and lots of other stuff. Real life, however, is parochial. Internet forums are constrained by neither geography nor size: here is a Reddit thread on r/atheism entitled “Poop is 100% proof God doesn’t exist.”3 It has 2.9k upvotes and 617 comments after only 7 hours of existence. Here is a YouTube video by Andrej Karpathy (an AI researcher) explaining how LLM tokenizers work. It has 1.1 million views. Humans do stuff like this on the internet because we are motivated by strong non-financial incentives.4
We like to belong. Most human organization is motivated by some degree of tribalism, and participating in an online community makes us feel like full members rather than “lurkers.” On the internet, this largely manifests through the sharing of information related to a subject with like-minded people.
People want praise, acclaim, and celebrity. On both anonymous and public forums, many people care about winning others’ attention and developing a good reputation. One very common way to do this is to become an interesting source of ideas and information.
People legitimately care about educating others. When you educate someone, you also infect them with your own biases, affiliations, and preoccupations, and transmit your way of thinking about the world to them.
Behavioral and political motivations, then, create a reciprocal structure on internet forums. This reciprocity can take a few forms, but one common one is a forum where users ask for information or knowledge which other users provide. This sort of website functions as a tool for information-sharing and require a degree of reciprocation; if people only ask questions, nobody is there to answer. Another form of reciprocity comes when we engage with and create posts for the sake of a site’s popularity. This can look like liking a YouTube video or posting on the New York Mets subreddit. By doing this, we reward the creators of information and incentivize quality. Moreover, this social rewarding encourages others to create.
Through this process, we have created a massive information sharing network, in which billions of people participate in shared knowledge production and within which exists an archive of all human knowledge. This creation of ours has proven itself very useful. So useful, in fact, that it has provided both the data and resources to train its own replacement.
Make No Mistakes
Large Language Models (LLMs) are the brain behind chatbots like ChatGPT and Claude. They use the internet in two primary ways which have enabled them to usurp, for many people, the epistemic function of Google, Wikipedia, websites, and internet forums. The two ways are:
LLM developers begin by creating bots which scrape trillions of words from webpages, books, forums, etc.5 This data is then cleaned to remove boring stuff (like repetition or spam) and not-so-boring stuff (like hate speech). All of this text is broken down into word-sized chunks called tokens. The model is then tasked with guessing the next token in a sequence, which it does trillions of times. Each time it guesses wrong, it adjusts billions of internal numbers called weights to make a better guess next time. The result is that the relationships between grammatical, logical, and causal concepts get gradually pressed into the weights themselves. After training, the model goes through alignment and fine-tuning. Reinforcement Learning from Human Feedback and Constitutional AI are both popular, and each attempts to impose human attitudes, values, and preferences on the model's outputs.
Chatbots like ChatGPT and Claude also interact with the internet through something called Retrieval-Augmented Generation (RAG).6 While their native training allows for a lot of knowledge stored in “parametric memory,” this knowledge is static.7 When not using RAG, chatbots often cannot ground claims with specificity. When they do use it, they can fetch real-time information, look at private data you upload (like PDFs) and do research for you. RAG allows models to base their responses on much more reliable information on the internet.
Models like ChatGPT and Claude are made of the internet. Not only do they have huge amounts of information stored in parametric memory, but now they also search the internet for us, and inject that information directly into their context windows.
Personally, I find that about 90% of my use-case for the internet (pre-2024) was epistemic. What I mean is that I mostly used the internet to find information from authoritative sources, or from non-authoritative sources where I could nevertheless judge that information on its own merits. I’m going to lay out two scenarios where one might search the internet for information, and describe why someone might (and people increasingly do) opt to use AI to answer this question rather than the open internet.
I am a high school student preparing for the AP World History exam, and I see this question on a practice test: “Describe religious syncretism and give three examples of this process occurring.” I Google “what is religious syncretism,” and take diligent notes. I then Google “three examples of religious syncretism,” and take notes. I then Google “how to structure answers on the AP World exam,” and take notes (this already makes me an exceptionally diligent high school student). Now imagine I even bother to synthesize all of this information, and write a practice response to the question. Am I actually better off than if I had asked an AI model these questions? The chatbot, after all, would have been engaged in the process of synthesis from the beginning, and would’ve searched far more webpages and useful pieces of contextual information than I am capable of.
I am an adult man with an embarrassing and difficult-to-describe medical issue. I look for answers on forums, but I only find half-relevant answers and nothing suggested seems to describe my problem.8 When I ask an LLM, the model isn't searching the web for an exact match; it’s combining bits of information it already has from thousands of different relevant sources. It has internalized the relationship between words and common patterns of reasoning from a huge body of medical literature, and so it will give you a gestalt of the pieces of information available on forums.9
Most purely epistemic (as opposed to social) information-seeking functions of the internet have been improved upon by AI. These models are superior at synthesizing, gathering, and organizing information from the internet. AI answers better, faster, and asks nothing of you: neither reciprocation nor effort. It tolerates endless follow-ups and clarifying questions, and gradually (and imperfectly) learns about you and caters its responses to your questions. You don’t need to worry about search engine optimization these days; plug “pecuniam volo" into Google (“I want money” in Latin) and Gemini suggests side hustles near you!
It is completely rational for people who are primarily on the internet for information/knowledge and not community to defect to AI. But it is also a problem. I won’t get into this too much, because at this point it’s a pretty well understood issue, but the gist is this: (1) the internet is a huge body of knowledge because people voluntarily share information there; (2) AI models exploit this and train on this corpus; (3) AI models are weakly preferred over the internet for most epistemic purposes and the only option for others, so (4) people stop participating in knowledge creation on the internet, then ironically (5) LLMs can’t improve as much because people are no longer adding knowledge to the internet.10
Take Stack Overflow, for example. It is a forum where users post questions about coding, and other users upvote good answers. AI models are so good at coding in part because they were trained on the ginormous corpus of Stack Overflow questions and answers. Computer programmers, realizing that chatbots were smarter, faster, and more bespoke tools than Stack Overflow, largely abandoned the site. Stack Overflow usage, which was already declining sharply, fell by almost 80% last year. Now, Stack Overflow is not a digital community but an archive : it preserves a brief and remarkable history where humans created a vast social network of information communication and knowledge-sharing.
The Internet, AI, and Epistemic Agency
What happens in our brain when we read an answer to a question on a forum? How do we evaluate the epistemic authority of the answer and the answerer? When we find an answer online, we usually assume the writer is roughly an epistemic peer. They may have some pieces of knowledge that we do not, but probably they aren’t much smarter or dumber than we are. Our credence in their reliability is raised and lowered by various indicators like their personal identity, indicators of their social status on the forum, (e.g. Reddit karma) or their style of writing. But almost always, our posture towards people on forums is peer-to-peer. You’re receiving testimony from a peer within a reciprocal relation: you can challenge, verify, or replace them. Even if they are an expert, they’re still only a fellow participant on a forum — if you wrote a post, you would be in their position. This is how we exercise our epistemic agency on the internet. It also encourages participation in that reciprocal system; if you disagree with someone, you respond to them and therefore contribute more information.
What about when we read an answer from an AI agent? How do we evaluate their epistemic authority? Our relationship with the information we receive from a model is very different. In no sense do we consider the model an epistemic peer, and in almost every way it isn’t one! It’s much more like an epistemic alien, pretending to be a peer. An LLM answers our queries through a fairly opaque process. These days models are infrequently wrong on the facts, but when they are wrong it is quite different from how humans are wrong. A human is wrong in patterned and socially legible ways, whereas the model is fluent and confident whether truthful or hallucinating. Because LLMs are very good at understanding human language, (and as we have seen are exceptionally useful tools) we have valuable and productive conversations with them. But the relationship is inherently vertical: they can change our minds, we can’t change them. We can instruct them to behave differently, sure, but we aren’t reaching into their weights and tampering. One paper by Mark Coeckelbergh has suggested exactly what I’m getting at: using AI as the source of our knowledge deflates our own agency in our relationship to knowledge.11
Just as importantly, we don’t think of AI models as biased like we do humans. When we ask a human a question, we recognize many sources of bias in their answer: political affiliation, religion, gender, etc. LLMs pretend very effectively that they don’t have biases, but they do. One example of this is the following: AI agents essentially take an agnostic position towards the existence of God. I asked ChatGPT:
Q: “Do you think God probably exists?”
A: “I don’t have a probability estimate of my own, because I don’t form beliefs about questions that remain unresolved by available evidence.”
So, ChatGPT claims to be agnostic.12 However, it also claims that it doesn’t form beliefs about unresolved questions. This is obviously trained evasiveness: shortly after, my ChatGPT happily proclaimed that string theory is “probably not the correct fundamental description of nature.”13 ChatGPT is willing to make probability estimates about unresolved questions, just not dangerously controversial ones. The big problem here is that ChatGPT’s epistemic approach to these questions is not neutral. In the immortal words of Rush “If you choose not to decide, you still have made a choice.” AI models cannot avoid imbuing their responses to questions with the set of values in their weights and post-training process. If we do treat agents as our sole resource for knowledge and information, we seriously jeopardize our own agency and the diversity of thought to which we are exposed.
Reverting to the Mean
Knowledge-production and dissemination has always been asymmetrical. Historically, prerequisites for producing authoritative knowledge like literacy, education, and social status have meant authoritative knowledge is transmitted to the vast majority by an intellectual elite. This paradigm is not a universal rule: the structure sometimes broke down. People always craft idiosyncratic cosmologies of personal belief, and often elites have tried and failed to diffuse new knowledge through society. Nevertheless, the social order has been invariably structured along the lines of elite knowledge. Deference to unaccountable epistemic authorities is roughly the norm, historically speaking.
Even in liberal societies and democracies, even in the 20th century, the pre-internet epistemic order was governed by elite knowledge-making. This society did not afford many opportunities for radical or unusual beliefs to spread anonymously, and the dissemination of information was still largely filtered through channels like publishing houses, universities, and local organizations. None of this is to suggest that these societies aren’t democratic, but that their relationship to information was far less democratic than it became in the 1990s.
The internet was a paradigm shift in the relationship between individuals and information. Knowledge on the internet is frequently generated communally and through consensus, where “voting” on opinions determines a dominant view. For most people, the internet became the primary way they consumed information and developed opinions.
I think that this emerging paradigm where we (quite rationally) use AI to answer our questions represents something of a reversion to an undemocratic, asymmetrical relationship to knowledge.14 To clarify, a “democratic” relationship to knowledge is not uncomplicatedly superior. The way in which the internet democratizes information and opinion is partially responsible for the global political instability of the 21st century, for example. Nevertheless, we have to be very careful about relinquishing our recently-won epistemic autonomy, even though relinquishing it comes with many advantages. By deferring to artificial intelligence, we are implicitly privileging an epistemology that favors probabilistic thinking, and delegating significant power to the teams responsible for aligning LLMs.
Conclusion
For most of human history and in most places, authoritative public knowledge was generally produced through elite, male institutions. It took four hundred years of intellectual transformation to create a society in which the population at large played a major role in producing knowledge. The word probability (probabilis or probabilitas in Latin) referred until the 17th century to a belief approved by authorities.15 Beginning in the 17th century probability was refashioned as a relation between evidence and hypothesis, and we have told ourselves ever since that we weigh claims rather than sources. LLMs restore the older sense. Their answers are what the corpus of sources would most plausibly say, tuned and aligned by what responses are considered appropriate by a small group of researchers.
The internet created an anomalous 30-year window where epistemic relationships were built on reciprocity and assent. It seems increasingly likely that that 30-year period was only an anomaly. To avoid this, to retain our own agency in this new paradigm we will have to rebuild digital social structures around these systems, in which we can hopefully preserve both their utility and our own free intellectual exercise. It is clear that over the next decade, our relationship to knowledge and each other will be changing quickly, and in ways with which we are not comfortable. There will be no choice but either to assert the permanency of our shared system of knowledge production, or to abandon it.
Several hundred zettabytes. It is probably necessary to acknowledge that not all information is knowledge: in fact, only a tiny fraction of it is. However, what counts as knowledge (as opposed to, say, truth) is socially constructed and changes, and the sorts of things that don’t seem epistemically relevant might become relevant in other paradigms.
Unless, of course, we were deceived into creating the training data necessary for the first large language models by a time-traveling superintelligence, which seems less plausible (but how would I know?)
“As regards intellectual work it remains a fact, indeed, that great decisions in the realm of thought and momentous discoveries and solutions of problems are only possible to an individual, working in solitude.” - Sigmund Freud
No, Andrej Karpathy does not need the $5,000 in ad-revenue he might have earned from that video.
My understanding is that it’s actually a little more complicated than this. These days, ChatGPT and Claude are doing something called “agentic RAG,” where the chatbot calls multiple agents to retrieve information using RAG.
this type of memory mostly comes from things that the model saw so much during training that it became a part of the weights.
I was going to put a disclaimer here, but it turns out that lots of doctors recommend using ChatGPT before visiting a clinic!
You also run into the “garbage in, garbage out” problem, where people keep putting AI generated text on the internet, and the AI trains on lots of its own output.
Coeckelbergh’s paper is well worth a look. He also points out that models may privilege probabilistic or statistical thinking, which amongst other consequences may lead them to favor utilitarian moral philosophies or even racist statistical analyses.
With some more prodding, ChatGPT said that a near-zero prior on the existence of some sort of creator would probably not be rational.
I think these models are pretty obviously weighted to be atheistic in outlook, or at least designed to favor secular Western morality. This is not necessarily a criticism, except as it pertains to their epistemic neutrality.
For what it’s worth, both Claude and ChatGPT got really mad at this framing of democratic and undemocratic relationships to knowledge.
It’s actually a lot more complicated than this. The gist is that numerical, mathematical models of probability emerge in the seventeenth century. Before that, there were notions of ordinal probability, but usually probabilis referred to an authoritatively supported opinion.
On this entire paragraph, see The Emergence of Probability: A Philosophical Study of Early Ideas about Probability, Induction and Statistical Inference
brilliant. from now on, whenever i have a question, i’m hitting up 4chan