Radek Osmulski

How to train an Alpaca?

There used to be a time when fine-tuning LLMs on off-the-shelf hardware wasn't a thing. Then the Llama weights got leaked, Stanford Alpaca was released, and the rest is history. So how was Alpaca fine-tuned? And why might we care? On one hand, Alpaca is where the Cambrian explosion of

How to fine-tune a Transformer (pt. 2, LoRA)

In part 1 of this series, I fine-tuned a Transformer using techniques straight from Universal Language Model Fine-tuning for Text Classification published in 2018. But so much has happened in the last 5 years! My plan was to read a couple of papers next, but I stumbled across LoRA: Low-Rank

How to fine-tune a Transformer?

I only started to learn about LLMs and in this blog post, I share how I would approach fine-tuning a Transformer today. Which of the techniques I learned years ago still work in the era of the Transformer? Also, toward the end of the blog post, I address the training

SLURM survival guide

This post is written from the perspective of someone learning to run SLURM jobs. There might be some inaccuracies but the idea is to get you up and running fast. Unfortunately, the only material on SLURM I have been able to find was written by MLOps folks for MLOps folks

How to evaluate an LLM on your data?

Being able to evaluate the outputs of an LLM model on your test set is a very valuable problem to solve. Imagine this scenario. You work at ACME Inc and one morning you pour yourself a great cup of coffee, open Slack, and see this: Employee #342526, we need you

personal project

Use ChatGPT inside Jupyter Notebook

Bringing the new tool as close as possible to where people already do their work is key.

An IDE for the era of AI

So much code that I would have to write by hand automagically appears on my screen!

There is something weird about the current generation of AI — better pay attention

Hype aside, there is something very uncanny about the most recent generation of AI models.

Featured

How did I get here?

An introductory chapter to a book on learning machine learning that I wrote.

How to reach the top of the imagenette leaderboard?

How to make your NNs more shift-invariant? What are some hyperparameter changes worth considering when training with a limited budget of epochs?

Going From Not Being Able to Code to Deep Learning Hero

A detailed plan for going from not being able to write code to being a deep learning expert. Advice based on personal experience.

How to build a Deep Learning system that will answer questions about the Harry Potter universe?

Riva is a set of APIs into a very complex, very well staffed AI research organization.

notes

20 Years of Tech Startup Experiences in One Hour by Jeremy Howard

There is no such thing as business... there is only such a thing as making things people want and selling them to them.

How to use the power of the community to learn faster

Community is the most powerful force behind online learning. It is the reason why MOOCs have a limited impact and tight-knit communities like fast.ai consistently produce unbelievable results.

howto

How to train and validate on Imagenet

Training on Imagenet is something that is completely trivial after you do it once, but if you are just someone on the Internet without such prior experience, it is an insurmountable task. Up until a couple of days ago, I didn't even know how to get the data! And then

Machine Learning and Testing

The rewards of testing can be immense, but so can be the price that one would need to pay for testing poorly.

How to train your neural network

Evaluation of cosine annealing.

Why take the log of a continuous target variable?

In this article, we’ll look at a simple but useful concept that often gets overlooked.

How to do machine learning efficiently

The only way to maintain your sanity in the long run is to be paranoid in the short run.

Do smoother areas of the error surface lead to better generalization?

An experiment inspired by the first lecture of the fast.ai MOOC

Introduction to data augmentation and pseudo-labeling

A closer look at two techniques that can help you make the most of your training data.

Can we beat the state of the art from 2013 with only 0.046% of training examples?

If we take the CNN layers of a pretrained model, how much data would we need to recover some of the mappings from features to classes? As it turns out, very little!

howto

Automated AWS spot instance provisioning with persisting of data

After following this guide, you will be able to spin up an AWS EC2 spot instance by executing a single command from your terminal. The instance will have a volume attached that will be persisted across shutdowns.