By the release of ChatGPT, the conception of a better model was how big it could be, but a paper presented a model named Chinchilla, Training-Compute-Optimal Large Language Models, where this small model had the same performance as the big ones. How? It was trained on more data in proportion to the number of parameters the model had. What the paper found was groundbreaking, because that meant just a bigger model was a waste of resources. You had to increase all aspects of training at the same time, and they proposed a proportion:
training tokens = 20–40 times the number of parameters
Important not just for research, but for industry applications, because this meant you don’t really need that many GPUs for the model you want to fine-tune (quality and domain specificity first) or RAG in your company. Just a smaller investment in data, so now you know what model to pick for sure: the one that matches your data budget. If what you want is to distill, the teacher should be between 3 and 10 times bigger than the student, and same ratio for data — the student should see 3 to 10 times more data.
Imagine that you have a model that understands text, and you could see the vectors that represent each word in a 3D space: a white room filled with arrows that come from the same corner and go in every imaginable direction. Then the same, but for a model that understands images, and for another model that understands sound. Then somehow you put the text 3D space over the image 3D space, and then over the two of them the sound 3D space, ¡¡¡AND THEY FIT!!!
That is the essence of a multimodal model: the arrow that represents a dog in the text space is the same as the image of a dog in the image space, or the sound of a dog in the sound space. So yes, three different models are connected, but in a way that allows them to learn shared representations of the same concept. Of course this is not automatic; the models have to be trained using paradigms like contrastive learning and discriminative learning. This area of deep learning is a huge field that started around 2015, but with the transformer architecture implementation it started to shine with papers like ImageBind by Meta.
First you want to build pre-tokenization and tokenization pipelines. This involves heuristics and some ML methods based on frequency, as far as I know. This part is crucial to how expensive and accurate inference will be. The main objective here is to deal with spaces, typos, capitalization, and punctuation, and normally — if you need it — you can use lemmatization and stemmers. To build the tokenization algorithm you can start by using every character as a token, then merge the most common ones, and repeat until you get an average length per token desired. Shorter tokens, more flexibility, but you will have a higher computation cost; longer tokens, the other way around. The optimal trade-off is around the subword length. If done right, the training can be 3 times faster than if not, and the dataset on which it is trained should be almost the same dataset the LLM will be trained on. So here comes the central question.
The internet is actually kind of dirty to just take all documents in it, and since you will need to use a web crawler to scrape the whole internet, you will not get clean, beautiful, ready-to-use documents, but a bunch of HTML tags, attributes, and a lot more. So after the web crawling phase, you need to do something called HTML extraction, which is to keep the content of the page and not all the other stuff. Then you will need to filter the content. How? Well, blacklisted pages will be banned, so you have to have access to a big list like this one, then implement heuristic filters to eliminate low-quality documents, after that do deduplication (eliminate things that are too similar to each other). After that you could split the remaining data into validation and train datasets. Then with the train dataset you should somehow train a classifier model that will help you find which of your documents have been referenced by a high-credibility source, to give those documents more importance, and also a clustering model to group the content into domains. And if you want to give your model a better performance in a specific domain, well, upweight the documents in that domain. At the end you have a training set that is kind of stratified by quality, and you want to train on the whole dataset but kind of overfit on the higher-quality data.
This is just what I could find out, but there is a lot of secrecy around this area, and we did not mention synthetic data or multimodal data.
Now you have a probabilistic model of language that is really good at predicting the next token. Now let’s train for alignment (a fancy word to say: teach the LLM to act as a human).
50k to 100k high-quality examples of prompts + ideal answers should be enough to do supervised fine-tuning, I think. Now it is possible to combine this with LLM-generated examples. After this, the RLHF stage starts, where the main goal is to clone human behavior. You can use Proximal Policy Optimization (PPO); this meant training a reward model on human preferences to reward the LLM when it had a good answer. But now Direct Preference Optimization is being used, where you show the model a good answer vs. a bad answer. And how does the human get in? Well, human annotators first rank the answers that the fine-tuned model provides. The ratings are converted into a continuous function (Bradley-Terry), and that function is used to train PPO and DPO models.
After all that work (and 50–100 million dollars), now you have your model. Congrats!! Let’s call him Thoughtless =), now thougthless will have to answer standarize test, like ast or saber pro in diferent domains, also human preference test, adversarial tests, agentic tests, to see if its helpful, complient, and if it can generate harmful content .
The Mixture of Experts intuition is easy to grasp. Instead of having a huge network that activates the same when solving cancer questions and when asking for a Duff, wouldn’t it be better to just activate the part of it that is good at asking for fiction beers when it needs to? Obviously it can’t be as demanding as solving cancer, right? That way the scaling laws are not going to limit us as much, and we can still grow our models without making inference slower and more expensive.
MoE was originally conceived in 1991, and that was the whole architecture. You had experts made of one MLP, then a non-linear function like hyperbolic tangent, and again a linear layer. There was a smaller network with the same architecture whose only job was to decide which token should go to each expert in a probabilistic way. Then the token was sent to all experts and their answers were weighted on the previously calculated probabilities — a lot like a random forest if you ask me (Decision Trees, Dragees, and Forests). The concept was applied again in 2017 with RNNs, in 2019 with transformers, and finally DeepSeek used it in 2024, which eventually materialized in January 2025 in the DeepSeek V3 model. What changed between 1991 and 2025?
A linear relationship was established between the size of the expert and the number of experts; a load balancing method that consisted in taking the 2 most probable experts to send the token and averaging the answers; a balancing loss that “punishes” the default network (router) when too many tokens are sent to the same expert (or when too few tokens go to another expert); and now the experts have a limited capacity for tokens to bound compute.
Because these experts are essentially the feed-forward ,here we explained what a feed forward network is ,part of the network chopped into smaller pieces, you can put them after the tokens are mixed, or inside each modality encoder before the tokens are mixed, or inside the fusion head where the tokens are mixed. The most popular is cross attention + MoE.
Yes, you will need fewer GPUs for inference, and yes, latency can go down, but have you heard about the KV cache and memory bandwidth?
If you don’t have a very good KV cache or high-speed bandwidth memory, maybe MoE is not for you — maybe distillation or quantization, but that will be for another blog. Stay fine-tuned. And go outside.
]]><div> FOR EVERYTHING — CSS DOESN’T REALLY FIX IT. Oh, you have no idea what CSS and <div> are? Stay tuned. And if you do, you may also want to stay.
The way I understand the web is: a bunch of databases with interfaces made for humans, so we can interact with those databases. It makes sharing ideas, services, and daily-life bureaucracy easier — so much so that now not having access to it puts you at a big disadvantage. A lot of people pay for their internet connection, have their device to access it, and still can’t access the knowledge or the services. And it is because the developer did not know how to create a good interface, so they made a beautiful render that was useless for these people: people with mobility problems, vision problems, or just with no mouse and only a keyboard to navigate, to name a few.
That’s the purpose of HTML5: a markup language with a lot of semantic tags that allow the browser to build a structured hierarchy of what is going to be on the screen. It has tags to identify all the elements you could have — button tags, section tags, links, lists, headings, images, forms, and many more semantic tags that give attributes and functionality to everything you want to put in those categories. If your interface is organized, meaningful, and easy to navigate with keyboards and other devices… imagine all the work that building that infrastructure took for the developers of HTML5, for you to go and take a non-semantic tag and use it for everything (that would be a <div>). The reality is that if you use semantic HTML, at least 40% of accessibility issues will never exist.
Another 20% will be fixed if you put meaningful text inside the tags and the attributes of the tags. Images have an attribute called alt tex, fill it with something that communicates what you wanted to communicate with the image. That is the challenge: not just filling it with whatever. And hide purely decorative images from the focus (what allows you to navigate a web page using tab or a screen reader). Buttons should have text that explains what they will do when pushed, and links should have text that communicates what the link is about — a URL can be painful to hear when a screen reader reads it.
CSS is another language that allows you to style that HTML. Communicating things only with color is a bad idea (color-blind people will not get it). Styling text so it looks like a heading when the HTML is not a heading, or communicating something using a visual flow of information that does not respect the tab order of the keyboard — that is misleading and awful for a lot of people. CSS is amazing (I don’t really like it) for a lot of things, but you can destroy accessibility with it, so these are important concepts to understand so you don’t break it.
It has a box model where every element is a box with content, padding, border, and margin. The main issue with CSS is to position and lay out all these boxes in a way that not only looks good but makes the information more understandable visually — again, without damaging accessibility for non-visual users. You can create a set of rules for different elements or groups of elements using specificity selectors, and the hierarchy of how those selectors get picked is top to bottom in the style sheet.
The rules or properties that control positioning and layout are the following:
DISPLAY:
block: a new line on the screen for each element; takes as much horizontal space as it can.inline: never a new line for each element; only takes as much horizontal space as it needs. Width and height do not exist here.inline-block: inline, with height and width.grid: the idea is that you divide the available space of the container into a grid, deciding how many columns and rows. Inside that grid you can make elements take more than one cell at a time, so at the most basic template you can imagine a numpad-like grid (yes, you can define gaps between the cells).flex: automatically a two-axis system is used to divide the container’s available space. You can change the main axis to vertical with the flex-direction property.justify-content will align the items across the main axis and align-items will do it on the secondary axis — but only visually. So if a user uses keyboard focus to navigate the content, or screen readers, and the meaning of the information depends on the order you set visually with flexbox… well, Houston, we got a problem.
POSITION:
relative: to where it should be by default.fixed: will follow the user.sticky: will follow the user from a particular position.Media queries are for applying certain styles depending on which device you are on, using the size of the screen. Whatever you do, the information and usability have to be accessible — it doesn’t matter if you are on a mobile or a PC.
We have not talked much about it, but the browser is a very sharp piece of software that creates object models from these files (like hierarchy trees), overlaps them and renders them, creating a render DOM that is the representation of both. After this, the layout of each of the elements on the page has to be calculated — a demanding task — and it has to be done every time you change font sizes, add or remove elements from the DOM, or change sizes or box-model properties. Then there is the repainting, triggered by the change of all properties that are not layout-oriented: less expensive, but still worth keeping in mind when CSS animations are used.
Why is this important? Well, because all this is triggered when JS enters the picture to dynamically create and inject HTML into your page, and the result will afect the position of your screen reader and can mak information imposible to navigate.
An event is an action by the user on the page: click, enter, spacebar, a particular key. If you use events that are multi-device (activated by the mouse or the keyboard or other devices), everything goes great. When you don’t… well, no accessibility, right?
Components are these kinds of personalized HTML tags that are crafted by wrapping HTML and CSS in JS code, and all the accessibility stuff we saw earlier on CSS and HTML is really important when building them. But if you don’t properly expose the semantics, component accessibility breaks. By the way, developer: don’t access innerHTML, use innerText, so you are not exposed to XSS attacks.
All frameworks are component-based, React being the most accessible I know of. You can also use dynamic updates to a page using JS — like when you committed an error filling out a form and an error message appears below the field. Make sure that update is announced by the screen reader. But remember about re-rendering the page’s DOM when we manipulate the tree with JS? Well, this can cause the focus to be lost, so avoid unnecessary changes to the DOM, and ALWAYS BE AWARE OF MANAGING FOCUS. If a modal appears on the page but is not announced and the focus is not moved to it, it is a ghost — can’t be perceived or interacted with as you intended in the moment you intended.
That will be all for today. You may go to recess and play accessible games with your friends, made with proper HTML5, CSS, and focus management.
]]>I had to work on a project where I would fix a tool that generated insights for the hospitality industry, and that’s when I realized 95% of tutorials about web scraping are irrelevant, at least for industry purposes they simply don’t work, and here I will tell you why and what does normally, but not always, work, and especially how web scraping is a mix of everything, so if you have a friend that is getting serious with learning software fundamentals, tell them to scrape Amazon, if they can.
Before that, a warning, this is for people that are interested in learning the technical side behind or to at least realize why the data guy I hire charges me so much when I ask him for a full house solution to scrape data from the web, but because the idea of this blog is to put in simpler terms the tech stuff, here a brief: you need to reverse engeneer the site you want to scrape , prepare for an armamentistic war were you create spies and they anty spy methods, and after that manage the hole storage cleanning and probably presenting of the data, so it combines knowing automation , a little of ciber security, network protocols, proxies, some web development tecnologies, data bases orchestraition with containers probably and to keep updating the system. IT’S A LOT OF WORK.End of the brief.
First you want to enter the main page of the portal you want to scrape, and add this to the URL “/robots.txt”, this is normally for crawlers but it can give you a perspective of the mission you are about to embark on.
When it tells that specific user agents are disallowed, now you know they monitor user agents and will spot you if your user agent makes no sense.
User agent string: is information about your system and is sent in the user agent header when you send an HTTP request.
robots.txt also specifies sensitive endpoints, when you see they are disallowed, and the sitemaps that are safe to just scrape with no use of UI. If they say something like crawl-delay 3, that can mean you can only send a request every 3 seconds. It also tells you how complex the site is, depending on how many tracking endpoints, disallows, user agents, and sitemaps. More complex sites, more strict anti-bot measures.
You have to infer other things from the names of those sensitive endpoints: /tracking/: have anti-bot systems with behavior tracking /js_tracking: telemetry and performance tracking and detection of headless browsers /fragment.: the site is dynamic, so the HTML is partial and it probably uses AJAX /anysearch.: the results on a search are also loaded dynamically /event: it tracks user action logic /xfg: or weird names like that could be an anti-bot system, do they have many user agents blocked?
Now you want to go again to your main page and put a lot of attention to:
How you are interacting with it, so you can then replicate that behavior (clicks, scrolls, searches, idle times, mouse movements). Use the developer tools to find out how the structure that renders the info you are interested in is built (HTML, CSS, XPath), and maybe go to the network section, if you are in luck you can find a hidden API that will make everything much easier, but from here we will presuppose you did not find it.
As we have established, this is warfare, so your scraper needs to be maintainable and robust, because changes will be needed as the website changes their UI or updates their anti-bot measures, this is why I will suggest you use a layered-based approach and a modular architecture thinking, where you define a core module from which you can import for your specific scrapers, if there is a better way you guys are encouraged to contact me via LinkedIn and tell me.
layers:
Orchestrator: the idea is that here you decide how and when to split in batches the scraper jobs, when and how the scraper will run, and for this here you will connect all the instances of your other layers, and handle failures, concurrency, and rate limits (how much to scrape in one run).
Automation driver: all the interaction that a human will do to get to the URL you want to scrape, this module is responsible for that, if you are using Playwright or Camoufox, or decided to build the user behavior controller yourself using playwrigth or selenium .
Network/fingerprint/proxy: Services from IP Royal or Bright Data will be useful, especially the ones that allow you to rotate in residential proxies, because if the server sees you (your IP) too often you are blocked. You also want to rotate your session, you know, that thing that is built with which cookies you are using and the data of the hardware and browser you use to navigate to a page, I think Bright Data has something called secure browser that helps with that.
Parsing/extraction: here you use the selectors you previously found using the developer tools to know what to extract and actually do it, all the cleaning, normalization, and validation of the data goes here. Beautiful Soup is one of the standards here.
Storage: I don’t have much experience here, still, not only a SQL database will be needed, an entire data pipeline is in order where you do data quality functions like normalization, validation, deduplication, idempotency, you need to decide if the data pipeline will process the data in a stream or in batches, I think this depends on how much RAM is available and how much data you have, or you can just dump the data in a CSV, your call.
Infrastructure: where and how the system runs. Docker is a must here, you can put it in a GitHub runner or an EC2, you want a way to handle logging and monitoring protocols, and maybe a way to destroy and create instances of your scraper system dynamically to better handle fingerprinting if you do so by yourself, and I haven’t talked about concurrency or observability because, being honest, I don’t really understand how that works yet, I just do a logging config, where I manage logs in files with the handlers it has by default and try not to let them grow too much.
As you can see, the anti-bot strategies are specially distributed in the network, automation, infrastructure, and orchestration modules or layers of your core.
Implementation will take its time, I hope there were a library or framework that had all these modules and I could just import and use out of the box, like scikit-learn for machine learning, but the closest thing I found is Scrapy and it falls short in some aspects apparently, but still a good thing to use.
Because you will have to implement a lot of this yourself, it is a good idea to have the next concepts clear:
interface: this is a template that is standardized for what a file should have in terms of behaviors (methods and functions), does not provide implementation factory pattern: imagine a super machine (class) that creates entire lines of production (other classes that actually create objects) strategy pattern: the idea is that it is a decision-maker algorithm, you give it a context (arguments to a function) and based on that it decides the best tool for the job abstract class: a mental group that is normally only used to gather classes that have some similarity between them. Humans, dogs, and whales (classes) are all mammals (abstract class), and inside there can be implementations of methods or variables that all classes inside of mammals have, like breathing or number of eyes difference between instance, static, and class methods: an instance method only makes sense if there is an object to perform it (to talk only can be done if a person exists and knows how to). A static method can be used even if there is no instance of the class (imagine a validation strategy to check if it’s a person that is not performed by another person). And finally, a class method is pretty similar to a static method but can edit variables that are part of the class, not the individuals in that class (number of persons in the world).
Now my dear spy, go and scrape the world.
]]>Well, if so, the first step for this introduction is for you to go and use NVDA on your PC if you have Windows, or TalkBack if you have an Android phone. These are assistive technologies that not only blind people but also other people with disabilities can use (the most common worldwide). I will leave you the links for basic tutorials here. You don’t have to be an expert, only to have a grasp of how other people interact with a computer.
A11Y is the symbol of digital accessibility because there are 11 letters between the first one and the last one, symbolizing how accessibility is about understanding the different types and colors of disability, which, by the way, is not inherent to the individual but appears in the relationship between the individual and their environment. You can think about it like this:
Michael Phelps is one of the greatest swimmers of our time. Physically, he is nowhere near to be thought of as a disabled person around the water, right? Now imagine he is abducted by an alien whale-based form and is forced to live with them on their vast ocean planet, where whales work and live like us but underwater, going to the surface two or three times a day for oxygen. How do you think Michael will do in that environment? Will he be able to do anything with no accommodations or tech that supports him?
It is the same individual, but his environment changed, and so the relationship he has with the environment changed. Now, if he tries to use a cellphone here on Earth while having shivers and blurry vision because of a horrible, horrible flu, the individual changed (he got the flu). So although the environment stayed the same, the relationship also changed.
This is important because we are all changing all the time, and also our environment — sometimes small changes, sometimes really big changes. The probability of a mismatch between the two (a disability) is always there, and actually increases with age: around 34% after 65, unless you already have a chronic illness or any comorbidity, then the 34% probability is reached way sooner.
According to the World Health Organization, 16% of people around the world have a significant disability. If you cross-reference that with the fact that natality rates are dropping, retirement age will increase, and chronic autoimmune and mental illnesses are on the rise, then it is not a surprise that the USA (ADA / Section 508) and the EU (Directive 2016/2102, Directive 2019/882) already have laws that enforce accessibility across digital products. The legal frameworks of these countries, and of countries like Argentina and Brazil, refer to WCAG AA directly or indirectly as the technical standard.
So let’s understand how the WCAG technical framework works.
It is like a building with three floors (A, AA, AAA) that is supported by four thick columns, or principles, that every interface should accomplish:
Around these columns there are different rooms supported by each column. These rooms are called guidelines. There are 13 in total, and this is how they are distributed:
To enter each floor, you need to meet the success criteria for each guideline, and inside each guideline you can find how to implement that success criteria. As I told you earlier, AA is the minimum from an economic, legal, ethical, and functional point of view (it is the one that eliminates most of the friction).
Now, don’t get afraid of the different versions of WCAG. They are cumulative: if you comply with 2.2, you will comply with 2.1.
To build or design an interface, you first need to understand what kind of information you are displaying.
You should take WCAG as good practices that are a must.
With this knowledge, you can navigate the guidelines more effectively and apply them with intention.
However, to make accessibility truly effective, you need to know your users and understand the different body–environment mismatches that exist ( not only the ones that are easily perceivable.)
Below are some examples. You can research each of them further on your own:
This is the beginning of a journey.
Use this knowledge wisely.
That will be all for today.
Call a friend and teach them what you learned here.
Share this post so many more people can become aware.
Remember the auto-completer on your phone keyboard that, some years ago, you hated because it always made you write something wrong? Well, that little creature grew up, discovered red-pill culture, put itself on steroids, and became a gym bro.
Now the technical brief. It’s the same principle as auto-completers. You find a way to turn words into numbers, then put those numbers into matrices. Humanity found new algorithms that made this better, and also figured out how to train them using almost all high-quality text from the first world. Then GPUs were used to run that training, because they are really good at matrix operations.
At the end, the algorithm has a probability for each possible next word. It doesn’t always take the one with the highest probability; that choice is controlled by what is called “temperature” or “creativity.” Then it auto-completes with a word and repeats the process many times, until training constraints tell it to stop. The first input was your prompt; the next is your prompt plus one word, and so on.
It is an echo chamber, with the same biases humanity put into the training data. If you want good answers, you need to be good with language, specific, and knowledgeable in the domain. The same applies if you are using more complex models that work with audio, images, or video.
If you are not interested in more technical details, you can stop reading here. Now we will get into how the guts of the beast work.
When I was a kid, I loved to sit at the back of the classroom and talk with my friends.
The teacher didn’t love it, so we ended up passing little pieces of paper with messages.
We changed consonants into numbers so that, if we got caught, nobody would know what we wrote about.
Imagine my surprise when I found out that this is essentially how we teach these creatures called neural networks about our language.
Of course, for them, they are just numbers—they don’t get the meaning, just like the teacher when he intercepted our messages.
There was once a creature that looked like a pawn on a chessboard.
This creature had thoughts that were both very basic and complex at the same time.
First, the only language it understood was mathematics—nothing else.
So it only spoke numbers and heard numbers.
The thoughts it had were either a linear regression or a logistic regression.
So if you gave it numbers, it would “imagine” a line that best fits those numbers,
or a curve that puts a limit on which two classes of numbers are separated.
Either way, in a mathematical sense, these thoughts were linear.
Now imagine two lines of pawns.The pawns don’t talk to the ones beside them, but each pawn talks to all the pawns in the line in front of it. A neural network is this kind of arrangement of pawn-like creatures.
FNN (Feedforward Neural Networks):
Used for classification problems with structured data.
They are essential to almost all other types.
CNNs (Convolutional Neural Networks):
Used to process spatial information that comes in grid-like formats by default,
such as images and videos.
The network of pawns, as a whole, performs a more complex operation called convolution.
RNNs (Recurrent Neural Networks):
Very good for sequential time data, like audio.
LSTMs, for example, have feedback loops in their neurons that allow short-term memory.
How? The last line of pawns produces an output that becomes an input for the first line.
Autoencoders:
Compress and decompress information, and they can be deep (many layers).
The idea is to filter noise from information to capture its essence,
manipulate it, and then reconstruct the information more or less as it was given.
Imagine a first line of 10 pawns, a second of 8, and a third of 4—forming a funnel-like shape.
GANs (Generative Adversarial Networks):
A discriminator tries to figure out whether what the generator created is a fake sample or not.
Transformers:
Just wait for it!!!
Now, just changing consonants to numbers is not enough anymore.Here we need vector-like representations: arrays of numbers that represent words.
This allows us to:
So here is what happened before ChatGPT:
We performed:
Then we trained a neural network just to create embeddings based on the final tokens. Your face says a lot—let me explain some of the words you just read.
Algorithms that reduce words to their closest root form.
For example, running becomes run.
They use rules, not machine learning, and sometimes they chop words into non-existing forms,
like passing becoming pas.
This goes deeper.
It returns the dictionary base form of a word.
For example, better becomes good.
It needs tagging.
Remember in school when you had to recognize whether a word was a noun, verb, or adjective?
That’s tagging.
Tokenization is the process of dividing text into units:
Once tokenized, these units can be converted into numbers or arrays of numbers.
An n-gram is a composition of tokens:
They help capture frequent word combinations and reduce vocabulary size
by preferring common token sequences.
The difference between an embedding and a frequency vector
is whether contextual meaning is preserved.
We are still converting words into numbers.
One approach is to:
This is a frequency base method, no semantic meaning asociated to the vectors, only numerical representation
Other method is to use pre-trained models that were trained to change the value of the vector depending on what it meant and how close the vectors that represent similar words would end up in regard to that first word. So the vectors for words like snow, cold, ice, winter will be relatively closer to each other, and not so close to the ones that represent words like kangaroo, monkey, tiger.
These pre-trained models are called tokenizers, they use all that we have discussed above, and they are the first step before ChatGPT algorithms can be used. See, no magic—just a lot of code and statistics.
The first application of the concept of attention—the idea that a word can affect or refer to a previous word—was used in feed-forward networks that aimed to compress the whole sentence, but it was a bottleneck. Then they used additive attention, where a small neural network computed a score of how much each token mattered in each state of the neural network. Then they realized that they could use dot product to see how similar two vectors were, and this would be more efficient. In 2016, self-attention was born, where through a series of matrices (Key, Query, and Value) they computed how all tokens in a sequence related to each other inside an RNN, which was until that time the default network for language.
Then multi-head self-attention came along, and since then the concept has been evolving, solving problems of long context (the amount of text you can pass before it starts forgetting).
Now, as you already know in such an obvious sense,** “attention is not new and position matters”** is not the same as what the title above says. Positional ordering is part of how we understand language. To encode the position of each word into its numerical representation, nowadays a method called RoPE is the default, where sine and cosine waves of different frequencies are used to encode each position.
Attention, context window and position where the concepts that blew my mind, because, when you read or talk with some one,
of course they are present, but you never stopped to think about them, completely unconscious behaviours, and yet so important,
our hole communication depends on them ….so common and yet so elegant human language ….
Anyhow, I think this is enough for one reading. So go take a walk, be with frendly people and see you next year, because transformers are already here.
]]>So, you plant your first yes/no question:
Is the dragee pink?
Whatever the answer, the next step is to ask the second yes/no question:
Does it taste bad?
graph TD
A[Is the dragee pink?] -->|Yes| B[Does it taste bad?]
A -->|No| C[Does it taste bad?]
B -->|Yes| D["Record as 'Pink & Bad' on left sheet"]
B -->|No| E["Record as 'Pink & Good' on right sheet"]
C -->|Yes| F["Record as 'Not Pink & Bad' on left sheet"]
C -->|No| G["Record as 'Not Pink & Good' on right sheet"]
The first question is called the root. Other questions are called nodes, and the final answers are called leaves.
This is a very simple decision tree to predict the flavor of a dragee based on whether it’s pink or not. Of course, we could include more criteria, like other colors, shape, or even the shop where it was bought. No matter how many features you have, always think about asking questions with binary answers (yes/no, true/false, 1/0, up/down).
Let me show you the Gini in the bottle 🧞
“Impurity” refers to how mixed the votes (or classes) are in a leaf.
We can also talk about node impurity, which is the weighted average of the impurity of each leaf. There are different ways to calculate it, but the most common is Gini Impurity:
[ Gini = 1 - \sum_{i=1}^{C} p_i^2 ]
Where:
Overfitting is like studying for a test by memorizing all past questions.
Depending on which stopping criteria you pick, your tree can get really big —
but you can still prune it using your favorite library’s pruning algorithms.
Let’s suppose you have a list of dragee radii, and you want to predict flavor based on that.
In essence, they work very similarly.
Decision trees are prone to overfitting, which brings us to democracy, but first …
From your original dataset, take a random sample with the same number of data points (allowing repetition).
This is called bootstrapping.
Then:
Now repeat the process many times.
You’ll get many trees of different shapes and sizes — a forest full of variety.
Each tree gets a vote, and the forest decides by majority.
The category with the most votes wins.
The important thing here is variety.
Without it, it’s not a democracy… it’s a demo‑crazy 🪓
Now get out and plant some trees, invite your friends , tell them what you learn here.
If you want to see code, chek here
]]>Machine learning algorithms have been used since the 1960s in the military, in medical image processing, and in science and manufacturing. They just weren’t very visible to the average person, even when we started “using them” all the time—search engines like Google, spam filters in email, recommendations in e-commerce like Mercado Libre, facial recognition in phones, and even the camera app use ML algorithms to improve performance. So isn’t it about time we demystify it?
As I have said (and will say again) in other publications: these are just statistical methods applied to a large amount of data to tune the parameters (dials) of a mathematical equation there’s no consciousness involved.
Nowadays, the algorithm alone is not enough. It’s still crucial, but we need to embed it into a system, just as mitochondria are embedded in the cell, and the cell is part of something bigger. This doesn’t mean it should always be used; in fact, if the pattern is too simple or there’s an easier solution that works well, it shouldn’t be used. These are magnifiers of our human capabilities and our biases so their application needs ethical scrutiny; Imagine a loan wordiness classification system that was fed with raze or gender as an input data, What about the algorithm in social platforms that encloses it’s users in a opinion bubble?
The applications of machine learning are quite diverse, but they can be grouped under two points of view:
Finally, we need a function that helps us measure success in our objective. This is called a loss function ( like a Progress tracker), and a system, unlike a single model, can have more than one loss function, sometimes one per functional requirement (objective). A system can therefore integrate many ML algorithms.
They must understand the business metrics that define success, grasp the problems to be solved, and build a semantic bridge between real-world success metrics and those that measure an ML model’s success, and more times than others finding middle ground between espectations that are contrary to each other.
A very important part of this process—something we’ll discuss later—is knowing what data is needed or available, exploring it, and understanding it to build models.
The mathematics and code for most of these models are already developed and documented in libraries like scikit-learn or PyTorch. So, as you might predict, the heart of a data scientist’s work is communication and learning quickly to propose and develop possible solutions.
Models are nothing without data—and there’s no ML without models. This is why ML system development isn’t like traditional software development: it’s not just the code that needs version control. The data and the data artifacts used to transform it also need version control, so the system requires proper data infrastructure.
The data engineer and data architect must make many decisions about how models will consume data, how the data will arrive, and how to make it clean and usable. In real life, data is usually messy and often lacks labels. Without labels, how can you use a loss function to tell your model whether it’s doing well or not? It’s like shooting arrows without a target. Again, data scientists and data engineers need to communicate a lot.
Let’s say your team has already found a model that best solves your problem. Now it’s time to deploy it. Nowadays, deploying it in the cloud is the most popular option. Your The model or models will be placed inside containers (like fishbowls that have everything your fish needs to live) and connected to the rest of your system or apps through an API, which will have to be built—a set of rules that standardizes communication.
The system’s performance must be monitored to detect when it’s time to revisit the data, perhaps there have been statistical changes that explain a decline in performance. These models stop working as soon as the underlying patterns change, and if those patterns are linked to human behavior, that happens more often than you might think.
The people responsible for this are called ML engineers or MLOps professionals. They help build infrastructure not only for deployment but also for monitoring and tracking models and data, with the goal of identifying and mitigating performance decay due to pattern changes. Once again, communication is key.
The data your system makes predictions on can be use to retrain your models, specially when the pattern changes, so the loop closes, again the data infra structure has to be tune, also the models , it’s an alive process, where people is the key ingridient.
Now that you understand better how ML is and will keep shaping our lives in the shadows, and how, even when it seems radically advanced, is just the product of well-educated communicators who happen to know a lot of statistics, linear algebra, and software, how are you going to use this knowledge?.
]]>Then virtual machines became popular and evolved. If you had a lot of money, you could make a huge investment in buildings and servers, build virtual machines on those servers, and then rent those virtual machines.
That’s how cloud computing was born. The cloud is just a bunch of servers you can use, but that aren’t yours.
The average Joe normally just uses storage services like Google Drive, Google Photos, Netflix, Gmail, or Dropbox, with a cute interface and no need of knolege on how they work. But you can also use computing services (e.g., running your programs on someone else’s computer using the internet as a communication system).This is different from the 1950s, where you also used someone else’s computer (a mainframe) but had to be physically present to use it.
This business model allows small players to have access to top-of-the-line hardware at lower costs, in an easy-to-scale fashion.
Now let’s talk about some acronyms:
IaaS: You pay for the virtual machines but you are in charge of managing how you use them (OS, storage, networks).
Example: EC2 from AWS. Use it when you need to train big models or do heavy computing (training).
PaaS: The provider also handles OS, storage settings, and security patches.
It’s like hiring a catering service: they provide the kitchen, the cooking, and the ingredients, and you pick the menu (what to run in that environment).
Use it when you need to serve models as APIs quickly (deploying).
SaaS: You go out to a restaurant, the menu is set, and you pick what you want.
Example: You pay for a prebuilt monitoring service for an ML model that you run in a PaaS service and that was trained in an IaaS virtual machine.
When you create a new AWS account for the first time, you will probably be overwhelmed because it’s a huge site with many options and panels, which change depending on some factors. It’s not very comprehensible really—I would say WCAG 2.2 accessibility guidelines are not met.
AWS has more than 200 services, and they are not the only cloud provider. So I asked GPT for an easy-to-map classification of the services. This makes it almost a 1-to-1 mapping with the other two big providers (GCP and Azure).
Connecting resources with safety precautions and managing users and permissions.
Like you would on a PC or local network.
Where your code runs.
Any kind of data. Like a folder in your local PC, with a flat structure where you use keys or paths to upload/download.
Structured or semi-structured data built for efficiency in information access.
Not flat like storage. If you’re a data analyst, this is your niche.
ML pipelines, AI services, and tools to manage applications.
Regions represent the physical locations where the data centers are. Depending on the region and services, prices will change. This applies to all providers.
When picking a region, you want to select the one that:
The AWS console (web page) changes depending on which region you have selected.
Here’s a quick guide for ML engineers starting with cloud and not sure where to begin:
| Category | AWS (most known for ML beginners ⭐) | Azure | GCP |
|---|---|---|---|
| Compute (Run code) | Lambda ⭐ (serverless, easy to deploy small models/APIs) | Azure Functions | Cloud Functions |
| Storage (Save data/models) | S3 ⭐ (store datasets, models, logs) | Blob Storage | Cloud Storage |
| Databases & Analytics | RDS ⭐ (managed SQL, great for ML feature storage) | SQL Database | Cloud SQL |
| Networking & Security | IAM ⭐ (manage roles/permissions, critical for safe deployments) | Azure Active Directory | IAM |
| AI/ML & Dev Tools | SageMaker ⭐ (train, deploy, monitor ML models) | Azure ML | Vertex AI |
I hope this broad schema has helped make your first encounter with a cloud platform more digestible. You are doing great, keep up the good work.
]]>Since 1971, under Richard Nixon’s Republican period, the dollar stopped having an amount of gold standing behind it. From that year, all around the world, money stopped being something with a material meaning and started to be the reflection of trust in a system of government and private entities (national and private banks among them, but not the only ones). National banks can decide to create more money (papers or bills: dollars, euros, pesos). More papers, same trust,less value per paper, more inflation. Seems like a lot of power concentrated in a couple of hands, right? If there was only another way…
When it’s guaranteed, you have to do some kind of effort to acquire a result, not take easy shortcuts. For example, having a strong, muscular body can be easy to check. And if the only place to get steroids was Mount Everest, you could either train or try to get to Mount Everest. It’s harder to cheat than to do the task.
It is just a list of instructions that leaves no room for ambiguity. The language or structures that allow you to create a set of instructions like this, so a PC can interpret them, are normally known as programming languages.
They used to keep a register of financial operations. Later, they changed to the form of Excel spreadsheets and books. At some point, someone decided to encrypt (disguise, like when you and your friend decided all “a” in your messages would be a “y” and only you could understand the meaning of them) a huge spreadsheet notebook to ensure the information in it was not changed.
So, that person found a machine where they put all the text they wanted to encrypt and the output was 256 zeros and ones. This array is called a hash, and the machine a hash function. It’s easy to know if a machine like this is a hash function, a funny one, or a broken one.
It is a way to transfer trust in the system itself, just like money since 1971. But this system does not depend on a central authority serving as referee for the transactions. Instead, it implements a mechanism where all the participants can be referees. How? Well, it runs over something called a blockchain. Every block is just an accounting book, and all accounting books have a reference to the last one that was filled up. Now, an algorithm groups a bunch of transactions for the new block, but in order to add this block to the blockchain, the persons (or machines) in the blockchain have to find the correct hash, putting in use the proof-of-work mechanism. They pass as input the transactions, the reference to the previous block, and a random number. What they change is the random number. Each guess can take at least 10 minutes to compute. The machines that do this are called miners and the process is called mining.
If a miner guesses correctly, they earn the right and the pay for being the intermediary in that transaction. They get paid the transaction fees and newly created bitcoin. Bitcoins have a limit. The system was created with only 21 million bitcoins in existence, so every 210,000 blocks (about 4 years), the amount of bitcoin created when a block is added is reduced by half. No inflation.
The popularization of these technologies and the massification of the use of LLM-based applications like ChatGPT, together with the necessity of less battery consumption, have made necessary changes in chip architectures. Becoming more popular is what is known as a “system on a chip,” which is the integration on a single chip of the CPU, TPU, GPU, NPU, Wi-Fi adapter, barometers, thermometers, etc.
I know, I know — in this series we have only covered a couple of ‘U’s so far, so let’s review (take a little walk, let what you have learn settle and keep reading).
The most common processing unit. It processes things in series and switches between processes to give a sensation of parallelism.
Intel, AMD, Snapdragon, M1 from Apple… Every operation occurs in what is called a cycle, controlled via a little crystal inside the PC.
A 2.1 MHz frequency means 2.1 million cycles per second. An operation — the smallest one — needs about 4 cycles (just to save something like a number in memory). A click or writing a letter is a compilation of these small operations.
A Tensor Processing Unit is a chip designed to do tensor math, where a tensor is a special kind of number array, like a matrix but with more than two dimensions. It uses 8-bit arithmetic to be precise and energy-saving. It is especially used for neural networks. It was invented by Google and has its own low-level programming language called CUDA. Imagine it like a monk that only does linear algebra.
A GPU uses parallelism in processing; it does not just emulate it like a CPU would(real multitasking, not changing focus every 3 seconds). GPUs were developed especially because of the gaming industry. They usually have their own RAM memory called VRAM and their own cooling system, all minners are usually GPUs.
We still use CPUs because GPUs are awful with general tasks.
A Neural Processing Unit is a chip designed to accelerate AI workloads, particularly for deep learning models. It optimizes matrix multiplications and tensor operations, similar to a TPU, but is more general-purpose and designed for integration into consumer devices such as smartphones and laptops. An NPU makes AI applications like image recognition, speech processing, and natural language models run faster and more efficiently with less battery consumption.
congratulations, it’s been a long journey, why dont you give it a try and invite some one to hang out and share what you have learn here.
]]>First, you and your friend exchanged keys when you two added each other in (message app). Now, when she pressed send, the message she wrote was encrypted using the key you gave her — your public key. When the message arrives at your phone, before you even open it, it is decrypted using the key you did not share — your private key.
This protocol of having 2 keys, one private and one public — to share the public so others encrypt the messages they send you, and to decrypt with your private key — is called SSH. Now, this only accounts for the security part.
A bit is an interrupter that can be either on or off. In electronics, this normally relates with having a low voltage or current (represented by a 0) or a high voltage or current (represented by a 1). A bit is either a 0 or a 1.
If you group 8 bits, you have a byte. And with a byte, you can represent not 8 numbers but 256 numbers… how?
Imagine you have powers of 2 arranged from 2^0 to 2^7, and under this array, you have an interrupter that decides if the power of 2 over it is summed to the grand total or not. That is a binary system. All 8 bits on and you have 255.
Then you can add other codification systems on top of this one to use bytes to represent characters, like UTF-8 or ASCII.
So, your friend’s message is converted to a series of bytes that are sent via waves to the Wi-Fi router at your friend’s house, or to one of the cell phone antennas her cell phone is connected to (more than one at the same time). Then these devices convert it to an electrical signal that is sent via cable to an Internet Service Provider that is the company that conects you to the rest of the world. in colombia (Movistar,claro)
Just as your house has an address, your internet connection does too. All internet connections have an address. For this addressing, there are 2 protocols — IPv4 and IPv6. The first one uses 4 bytes to represent addresses. I’m sure you are used to seeing numbers like 127.0.0.1 (this is set as localhost).
Nowadays, IPv4 is used to represent some form of local network, and it’s being used at the same time as IPv6, which has 8 bytes to represent addresses. IPv6 uses the hexadecimal system to represent them, so they look like AAAB:FFF::1234, where :: represents a space where all numbers inside are 0.
Now, remembering all these numbers is hard. Imagine — wouldn’t it be so much simpler to say to the Uber driver, “Take me to the house of Fulanito de Tal,” and he immediately knew where to go? Well, the Uber driver is your browser, and there is an actual translator that allows this to happen. They are called Domain Name Servers. These servers translate the name of the place you want to go — e.g., minijuegos.com — to the IP address of the server in which minijuegos.com is hosted. This is how you and facebook can comunicate via letters using as delivery system something call TCP protocol…. wait you did not know you and facebook sended each other letters? well kind of, keep readding.
You log in to your favorite social network—it doesn’t matter which one, what I’m about to explain is the same for all web pages.
You pick a photo, fill out the description, and tag your friends’ accounts in it. Then, when you press upload photo, something amazing happens. Your device (the client) writes a letter to the social network’s backend (the server).
What does the letter contain?
It contains all the information you just filled out, plus more, in an already established format called a POST request.
The server receives and processes all the information in the letter, then responds to your device with a 200 status code, which means everything went fine and your post was created.
These established formats for the letters the client and server send to each other are called HTTP methods. Besides POST, there are others such as GET, PATCH, and DELETE, each one with a specific purpose.
The way the backend answers is defined by HTTP status codes. There are five main types:
In short, HTTP sets the rules for how the structure or format of each “letter” should look. These letters are then encapsulated in a package that TCP (like FedEx in the real world) delivers to each endpoint.
Now, imagine setting up all these protocols in pure binary. For you and me, writing and understanding how we are manipulating machines this way would be hell. Sharing that knowledge with others so they could help would be extremely difficult.
This is why different programming languages exist. Each one was created to solve the need of giving a set of instructions to a particular hardware architecture (phones, old and new PCs, routers, antennas—if it has a CPU, it counts) in a way that allows people to express complex ideas and share them with others, without needing to rewrite everything whenever hardware changed.
Because silver bullets don’t exist, programming evolved in layers: first came assembly, built over binary, then C and C++ over assembly. From there, more languages were built, branching off when someone wanted to prove a new idea or paradigm (like object-oriented programming, functional programming, inheritance, etc.—don’t worry about the details for now).
The goal has always been to make speaking to the machine as close as possible to speaking to a human—while staying precise.
But all programming languages share some things in common:
Now i hope you have a great end of the day, with some friends to tell them about what you have read here, and while you do it ¿ Does the language you guys are speaking have things in cummon whith programming languages?
]]>