The first and by far most expensive stage of creating an AI language model, which we call \u201cpre-training,\u201d does not produce the kind of useful assistant (like ChatGPT) we use today. The result of that first stage is the so-called base model.<\/p>\n\n\n\n
Experts like Andrej Karpathy describe this form of AI very simply: it is not a smart conversationalist, but just an \u201cextremely expensive text completion system.\u201d<\/p>\n\n\n\n
Although this model has \u201cread\u201d the entire Internet and possesses a vast amount of knowledge, it still cannot talk to humans at this stage.<\/p>\n\n\n\n
To understand the nature of the base model, Karpathy offers a brilliant analogy: imagine it as a compressed digital archive of the entire Internet. In its hundreds of billions of parameters, the model has condensed statistical patterns from an incredible 15 trillion basic language units (tokens).<\/p>\n\n\n\n
However, it is important to understand that this islossy compression.<\/em>The model does not memorize texts from the Internet verbatim, like a digital encyclopedia or database, but learns the style of expression. It does not store facts as fixed records, but as statistical probabilities. Therefore, it cannot \u201ccite\u201d the Internet exactly, but generates text that sounds like something that could be found online.<\/p>\n\n\n\n The main reason why the basic language model cannot be a useful assistant is that it does not understand the concept of conversation. It is not programmed to help, but to continue the beginning of a series of words.<\/p>\n\n\n\n If we ask the basic model a simple question, \u201cWhat is 2 + 2?\u201d it will most likely not answer \u201c4\u201d. Instead, it will continue the series as an online document would continue (e.g. a school quiz or knowledge test). It will probably generate new questions such as \u201e<\/em>\u201cWhat is 3 + 3?\u201c<\/em> \" or \u201e<\/em>\u201cWhat is the definition of addition?\u201c<\/em>\u201d. In the basic model, the question is not the beginning of a dialogue, but the beginning of a document that it tries to complete mathematically precisely by imitating the styles it has encountered during its learning.<\/p>\n\n\n\n Despite possessing an unimaginable amount of information, the basic model is not usable in everyday work by itself. It has no \u201cpersonality\u201d, does not follow instructions and can often wander in unexpected directions because it only imitates samples from the Internet, including those that are less useful or incorrect.<\/p>\n\n\n\n This insight is crucial for the education system because it explains the origin of so-called \u201challucinations\u201d. AI does not \u201clie\u201d intentionally, but simply tries to generate the most statistically likely continuation of a sentence. If that continuation sounds convincing, the model will print it, regardless of whether it is factually based.<\/p>\n\n\n\n It is only in the second, much cheaper and shorter phase, that the simulator turns into a useful assistant that understands instructions and knows how to conduct a meaningful conversation.<\/p>\n\n\n\nWhy does the basic model not participate in a conversation?<\/h3>\n\n\n\n
From imitator to digital assistant<\/h3>\n\n\n\n
Read the previous articles in the series:<\/h3>\n\n\n\n