Hi, do you use ChatGPT on a daily basis? Maybe you even know about Claude or Gemini and use them as a helper writer or as a tutor, asking them a lot of questions. I certainly hope you don’t use them as a friend to tell your problems to, or as a serious advisor on important topics in your life. Here is the executive brief on why.

Remember the auto-completer on your phone keyboard that, some years ago, you hated because it always made you write something wrong? Well, that little creature grew up, discovered red-pill culture, put itself on steroids, and became a gym bro.

Now the technical brief. It’s the same principle as auto-completers. You find a way to turn words into numbers, then put those numbers into matrices. Humanity found new algorithms that made this better, and also figured out how to train them using almost all high-quality text from the first world. Then GPUs were used to run that training, because they are really good at matrix operations.

At the end, the algorithm has a probability for each possible next word. It doesn’t always take the one with the highest probability; that choice is controlled by what is called “temperature” or “creativity.” Then it auto-completes with a word and repeats the process many times, until training constraints tell it to stop. The first input was your prompt; the next is your prompt plus one word, and so on.

It is an echo chamber, with the same biases humanity put into the training data. If you want good answers, you need to be good with language, specific, and knowledgeable in the domain. The same applies if you are using more complex models that work with audio, images, or video.

If you are not interested in more technical details, you can stop reading here. Now we will get into how the guts of the beast work.

When I was a kid, I loved to sit at the back of the classroom and talk with my friends.
The teacher didn’t love it, so we ended up passing little pieces of paper with messages.
We changed consonants into numbers so that, if we got caught, nobody would know what we wrote about.

Imagine my surprise when I found out that this is essentially how we teach these creatures called neural networks about our language.
Of course, for them, they are just numbers—they don’t get the meaning, just like the teacher when he intercepted our messages.

Artificial neuron

There was once a creature that looked like a pawn on a chessboard.
This creature had thoughts that were both very basic and complex at the same time.

First, the only language it understood was mathematics—nothing else.
So it only spoke numbers and heard numbers.

The thoughts it had were either a linear regression or a logistic regression.
So if you gave it numbers, it would “imagine” a line that best fits those numbers,
or a curve that puts a limit on which two classes of numbers are separated.

Either way, in a mathematical sense, these thoughts were linear.

Neural networks

Now imagine two lines of pawns.The pawns don’t talk to the ones beside them, but each pawn talks to all the pawns in the line in front of it. A neural network is this kind of arrangement of pawn-like creatures.

Types of neural networks

  • FNN (Feedforward Neural Networks):
    Used for classification problems with structured data.
    They are essential to almost all other types.

  • CNNs (Convolutional Neural Networks):
    Used to process spatial information that comes in grid-like formats by default,
    such as images and videos.
    The network of pawns, as a whole, performs a more complex operation called convolution.

  • RNNs (Recurrent Neural Networks):
    Very good for sequential time data, like audio.
    LSTMs, for example, have feedback loops in their neurons that allow short-term memory.
    How? The last line of pawns produces an output that becomes an input for the first line.

  • Autoencoders:
    Compress and decompress information, and they can be deep (many layers).
    The idea is to filter noise from information to capture its essence,
    manipulate it, and then reconstruct the information more or less as it was given.
    Imagine a first line of 10 pawns, a second of 8, and a third of 4—forming a funnel-like shape.

  • GANs (Generative Adversarial Networks):
    A discriminator tries to figure out whether what the generator created is a fake sample or not.

  • Transformers:
    Just wait for it!!!

Now, just changing consonants to numbers is not enough anymore.Here we need vector-like representations: arrays of numbers that represent words.

This allows us to:

  • Add words when a concept is represented by two words.
  • Multiply vectors to get a sense of semantic similarity between words,
    using an operation called the dot product.

So here is what happened before ChatGPT:

We performed:

  • Stemming
  • Tagging
  • Lemmatization
  • Tokenization
  • N-gram extraction (sequences of tokens)
  • Tokenization again

Then we trained a neural network just to create embeddings based on the final tokens. Your face says a lot—let me explain some of the words you just read.

Stemmers

Algorithms that reduce words to their closest root form.
For example, running becomes run.
They use rules, not machine learning, and sometimes they chop words into non-existing forms,
like passing becoming pas.

Lemmatization

This goes deeper.
It returns the dictionary base form of a word.
For example, better becomes good.

It needs tagging.
Remember in school when you had to recognize whether a word was a noun, verb, or adjective?
That’s tagging.

Tokenization

Tokenization is the process of dividing text into units:

  • Words
  • Syllables
  • Lemmas (from lemmatization)
  • Stems (after stemming)

Once tokenized, these units can be converted into numbers or arrays of numbers.

N-grams

An n-gram is a composition of tokens:

  • 2 tokens (bigrams)
  • 3 tokens (trigrams), etc.

They help capture frequent word combinations and reduce vocabulary size
by preferring common token sequences.

Embeddings or frequency vectorizers

The difference between an embedding and a frequency vector
is whether contextual meaning is preserved.

We are still converting words into numbers.

One approach is to:

  • Lemmatize
  • Tokenize
  • Take a large bag of words (a corpus)
  • Measure how often each word appears across documents

This is a frequency base method, no semantic meaning asociated to the vectors, only numerical representation

Other method is to use pre-trained models that were trained to change the value of the vector depending on what it meant and how close the vectors that represent similar words would end up in regard to that first word. So the vectors for words like snow, cold, ice, winter will be relatively closer to each other, and not so close to the ones that represent words like kangaroo, monkey, tiger.

These pre-trained models are called tokenizers, they use all that we have discussed above, and they are the first step before ChatGPT algorithms can be used. See, no magic—just a lot of code and statistics.

Attention position matters and it is not new

The first application of the concept of attention—the idea that a word can affect or refer to a previous word—was used in feed-forward networks that aimed to compress the whole sentence, but it was a bottleneck. Then they used additive attention, where a small neural network computed a score of how much each token mattered in each state of the neural network. Then they realized that they could use dot product to see how similar two vectors were, and this would be more efficient. In 2016, self-attention was born, where through a series of matrices (Key, Query, and Value) they computed how all tokens in a sequence related to each other inside an RNN, which was until that time the default network for language.

Then multi-head self-attention came along, and since then the concept has been evolving, solving problems of long context (the amount of text you can pass before it starts forgetting).

Now, as you already know in such an obvious sense,** “attention is not new and position matters”** is not the same as what the title above says. Positional ordering is part of how we understand language. To encode the position of each word into its numerical representation, nowadays a method called RoPE is the default, where sine and cosine waves of different frequencies are used to encode each position.

Attention, context window and position where the concepts that blew my mind, because, when you read or talk with some one,
of course they are present, but you never stopped to think about them, completely unconscious behaviours, and yet so important,
our hole communication depends on them ….so common and yet so elegant human language ….

Anyhow, I think this is enough for one reading. So go take a walk, be with frendly people and see you next year, because transformers are already here.