From API to GPU, Week 6 (Part 1): A Model That Predicts, and How Wrong It Is

Phase 2 of 8: Enough ML to understand inference. Week 6 of 32, part 1 of 2.

Every week so far, the model already existed. I ran Phi-4, read Qwen's files, and
measured tensors, but I never made a model learn anything. This week I build one
from scratch and train it. That is a lot for one sitting, so I split the week into
two posts and go slowly.

The goal: the model starts with a random weight and bias, which are just
wrong guesses. Training is what adjusts them, step by step, until they land on
the values that actually convert Celsius to Fahrenheit: a weight of 1.8 and a
bias of 32. Everything in these two posts is in service of that one idea.

This first post builds the model and answers two questions, nothing more: how does
the model make a prediction, and how do we measure how wrong that prediction is?
No learning yet. Part 2 takes the error measured here and turns it into learning.

The model is a one-neuron linear model: one output value computed from one weight
and one bias. It is deliberately tiny, so nothing hides the machinery. It runs on
the CPU through ssh spark, because the problem is far too small to need a GPU.

What "learning" means here

If you have never trained a model before, "learning" can sound like magic. It is
not. It is trial and error run in a tight loop, the same way you would
reverse-engineer an unknown function from its input and output logs.

Picture a black box that turns numbers in into numbers out. You cannot see inside
it, but you have a few examples of what it did: it turned -40 into -40, 0 into 32,
and 100 into 212. You want a formula that reproduces those outputs. So you guess
one, check how far each guess is from the real answer, nudge the guess in the
direction that shrinks the error, and repeat until the answers line up. That loop
is all that "training" is.

A neural network does exactly this, with two differences. First, the "formula" is
a set of adjustable numbers called parameters (the weights and biases from Week
4), and learning means finding good values for them. Second, the nudging is
automatic: the network measures its own error and works out which way to move each
parameter, so nobody tweaks anything by hand.

That loop has four moves: make a guess, measure how wrong it is, work out which way
to adjust, and take a small step, then repeat. This post covers only the first
two, making a guess (the forward pass) and measuring how wrong it is (the
loss). Part 2 covers the adjusting. Splitting it this way keeps each idea tied
to a real number you can see, instead of a pile of new words up front.

The black box in this post is Celsius to Fahrenheit. The true rule is
F = C * 1.8 + 32. The model never sees that rule. I only use it to build the
correct answers, then show the model input-output examples and let it discover on
its own that the weight should be 1.8 and the bias should be 32.

The model: one neuron

Week 4 said a weight multiplies an input and a bias is added after. This
model is exactly that, one weight and one bias:

That single weight-and-bias unit is a neuron, the smallest building block of a
neural network. A neural network is just many neurons wired together, so the
output of some becomes the input of others. Real networks stack thousands or
millions of them; this week's has exactly one, which is why nothing hides the
mechanics.

In PyTorch that is a single linear layer, nn.Linear(1, 1): one input
feature (the one input value) and one output value per example. A layer is
one processing step, and a linear transformation is the multiply-and-add above.
(Because there is a nonzero bias, a mathematician would call this an affine
transformation, but PyTorch names the layer Linear, so I use that word.) If the
weight lands on 1.8 and the bias on 32, the layer computes Fahrenheit exactly.

Why is this called linear? Picture plotting the model's output against its
input on a graph. A weight-times-input-plus-bias rule always traces a straight
line. The weight sets the line's slope: how steeply the output climbs as the
input grows. The bias slides the whole line up or down. Celsius to Fahrenheit is a
straight line with slope 1.8, because every extra degree Celsius adds exactly 1.8
degrees Fahrenheit, the same amount at every temperature. That fixed,
never-changing slope is what "linear" and "straight line" mean here, and a single
linear layer can match it perfectly.

This model has no activation function, the extra piece that would let it bend a
straight line into a curve. Because Celsius to Fahrenheit is already straight, it
does not need one. What an activation function is, and what a curve looks like, I
unpack at the end, in Wait, why is there no activation function?, so the main
thread stays on the model.

Making a prediction: the forward pass

A forward pass runs the inputs through the layer to get predictions. In
PyTorch it is the single expression model(celsius). Before any training the
weight and bias are random, so the predictions will be nonsense, which is exactly
what I want to see first.

The six Celsius inputs are paired with their correct Fahrenheit answers. Each
correct answer is called a label or target, and the line
fahrenheit = celsius * 1.8 + 32.0 builds all six at once. There is no loop here.
celsius is a tensor, the small array type from Week 5, and multiplying a tensor
by a single number multiplies every element by that number; adding 32 then adds it
to every element. That is an element-wise operation, and it is why the result is
another tensor of six answers, not one number. Note that celsius is the input
data, not the model. The model is the nn.Linear layer built a few lines down.

Each value in celsius is written with a trailing dot, so -40. means the
floating-point number -40.0, not the integer -40. The dot keeps the tensor in
decimals, which is what the model's math needs. And each value sits in its own
brackets, like [-40.], so stacking the six of them makes a tensor shaped 6 rows
by 1 column: six examples, one value each.

To be clear, those six numbers are six separate Celsius inputs (-40, -10, 0, 20,
37, and 100 degrees). [-40.] and [-10.] are two different examples, not an
input-and-answer pair. Each input's Fahrenheit answer lives at the matching
position in the separate fahrenheit tensor, and the run below prints them side
by side. The value -40 shows up as both the first input and its own answer only
because -40 C equals -40 F, the one temperature where the Celsius and Fahrenheit
scales meet, which is a coincidence of the scales, not the data layout.

The command below builds the model, prints the device it runs on and its random
starting weight and bias, then does one forward pass and lays each prediction next
to its target:

ssh spark '~/venvs/w1/bin/python - <<PY
import torch  # PyTorch: the tensor and neural-network library
from torch import nn  # nn holds ready-made layers such as Linear

# Fix the random starting values so this run is repeatable and matches the post.
torch.manual_seed(0)

# The six training inputs, in Celsius. Each inner [ ] is one example holding one
# value, so the tensor shape is 6 rows by 1 column: six examples, one feature.
celsius = torch.tensor([[-40.], [-10.], [0.], [20.], [37.], [100.]])

# The correct answer for each input, built from the real rule F = C * 1.8 + 32.
# The model never sees this rule; it only sees the inputs and these answers.
fahrenheit = celsius * 1.8 + 32.0

# One linear layer with 1 input feature and 1 output: prediction = w * C + b.
model = nn.Linear(1, 1)

# weight and bias start at random values; .device shows they live on the CPU.
print(f"device={model.weight.device}")
print(f"initial weight={model.weight.item():.6f} bias={model.bias.item():.6f}")

# The forward pass: run all six inputs through the layer to get predictions.
# no_grad() means "just predict, do not record anything for training".
with torch.no_grad():
    preds = model(celsius)

# Show each prediction next to the answer it should have produced.
print("initial predictions vs targets:")
for c, p, f in zip(celsius.tolist(), preds.tolist(), fahrenheit.tolist()):
    print(f"  C={c[0]:>6.1f}  pred={p[0]:>8.3f}  target={f[0]:>7.1f}")
PY'

device=cpu
initial weight=-0.007487 bias=0.536444
initial predictions vs targets:
  C= -40.0  pred=   0.836  target=  -40.0
  C= -10.0  pred=   0.611  target=   14.0
  C=   0.0  pred=   0.536  target=   32.0
  C=  20.0  pred=   0.387  target=   68.0
  C=  37.0  pred=   0.259  target=   98.6
  C= 100.0  pred=  -0.212  target=  212.0

Two things to notice. device=cpu confirms this runs on the CPU, the .device
attribute from Week 5, and the problem is far too small to need a GPU. And the
random start makes every prediction wrong: the weight is about -0.007 and the bias
about 0.54, straight from torch.manual_seed(0) (a fixed seed, so you get these
same numbers), so at 100 C the model guesses -0.212 instead of 212. The model has
no idea what Fahrenheit is yet.

So why are the predictions wrong when the correct answers are sitting right there
in fahrenheit? Because the model never looks at those answers when it predicts.
A prediction is only weight * celsius + bias, computed from the model's own
weight and bias, which right now are the random -0.007 and 0.54. The fahrenheit
values are a separate answer key: they are used to score the predictions, and in
Part 2 to teach the model, but the model does not see them while it predicts. It
is like a student taking a test with the answer key face down. Until they learn
the rule, the answers are guesses, no matter that the key is in the room.

The answers are not inside the input tensor at all. celsius and fahrenheit
are two separate tensors, two different variables. The call model(celsius)
passes only celsius, so fahrenheit never enters the model. Inside, the linear
layer computes exactly weight * celsius +
bias from its own two numbers. The one and only place fahrenheit is handed to
anything is the loss, loss_fn(preds, fahrenheit), in the next section. You can
prove the model uses only its weight, bias, and the input by recomputing its
prediction by hand and comparing:

ssh spark '~/venvs/w1/bin/python - <<PY
import torch
from torch import nn
torch.manual_seed(0)
celsius = torch.tensor([[-40.], [-10.], [0.], [20.], [37.], [100.]])
fahrenheit = celsius * 1.8 + 32.0   # the answer key: a separate tensor

model = nn.Linear(1, 1)
w = model.weight.item()             # weight and bias: the model owns these
b = model.bias.item()

with torch.no_grad():
    preds = model(celsius)          # pass ONLY celsius, never fahrenheit
by_hand = celsius * w + b           # weight * input + bias, computed by hand

print(f"weight={w:.6f} bias={b:.6f}")
print("model(celsius) matches weight*celsius+bias:",
      torch.allclose(preds, by_hand))
print("prediction for 100 C:", round(preds[-1].item(), 4))
PY'

weight=-0.007487 bias=0.536444
model(celsius) matches weight*celsius+bias: True
prediction for 100 C: -0.2122

model(celsius) and the hand-computed weight * celsius + bias come out
identical, so the model's output depends only on its two parameters and the
Celsius input. The value -0.2122 is the same wrong guess for 100 C from the
table above. fahrenheit plays no part in producing it.

Measuring how wrong it is: the loss

Now that the model can make predictions, we need a single number that says how good
or bad those six predictions are as a group. That number is the loss. A smaller
loss means better predictions; a loss of zero means every prediction is exactly
right.

Building it starts with the error for one example: how far a single prediction
is from its target, which is just prediction - target. For the 100 C example the
prediction was -0.212 and the target is 212, so the error is about -212. Some
errors come out negative (the guess was too low) and some positive (too high).

To turn six errors into one score, I use mean squared error (MSE). The name is
the recipe read backwards: take each error, square it, then take the mean, which is
the average.

Squaring does two useful things. It makes every error positive, so a guess that is
too low and one that is too high both count as wrong instead of cancelling out. And
it punishes big misses far more than small ones: an error of 200 becomes 40000,
while an error of 2 becomes just 4. The command below does this by hand, one step at
a time, then checks the result against PyTorch's built-in nn.MSELoss:

ssh spark '~/venvs/w1/bin/python - <<PY
import torch
from torch import nn
torch.manual_seed(0)
celsius = torch.tensor([[-40.], [-10.], [0.], [20.], [37.], [100.]])
fahrenheit = celsius * 1.8 + 32.0     # the correct answers

model = nn.Linear(1, 1)
with torch.no_grad():
    preds = model(celsius)            # the six predictions

# Step 1: the error for each example is prediction minus target.
errors = preds - fahrenheit
# Step 2: square each error so big misses count more and signs do not cancel.
squared = errors ** 2
print("per-example error and squared error:")
for e, s in zip(errors.flatten().tolist(), squared.flatten().tolist()):
    print(f"  error={e:>10.3f}  squared={s:>12.3f}")

# Step 3: the loss is the average of those six squared errors.
by_hand = squared.mean().item()
builtin = nn.MSELoss()(preds, fahrenheit).item()
print(f"average of squared errors (by hand) = {by_hand:.4f}")
print(f"nn.MSELoss() gives the same number  = {builtin:.4f}")
PY'

per-example error and squared error:
  error=    40.836  squared=    1667.572
  error=   -13.389  squared=     179.257
  error=   -31.464  squared=     989.955
  error=   -67.613  squared=    4571.558
  error=   -98.341  squared=    9670.867
  error=  -212.212  squared=   45034.031
average of squared errors (by hand) = 10352.2070
nn.MSELoss() gives the same number  = 10352.2070

Read the output top to bottom and the loss stops being a mystery number. Each row
is one example's error and that error squared. The 100 C example dominates: its
error of -212 squares to 45034, more than all the other five combined, which is
exactly the effect of punishing big misses. Add the six squared errors and divide
by six and you get 10352.2070, and PyTorch's nn.MSELoss returns the identical
value. So the built-in loss really is just the average of the squared errors,
nothing more.

One consequence of squaring: the loss is in squared Fahrenheit units, not degrees.
A loss of 10352.21 does not mean the model is off by 10352 F. The number is not a
temperature to read; it is a score to drive down. Right now it is huge, which fits,
because the predictions are still random. In Part 2, every training step pushes this
number lower.

This is the whole job of Part 1. The model can make a prediction (the forward
pass), and we can score how wrong it is with one number (the loss). Part 2 uses
that one number to improve the model.

Results so far

What Part 2 covers

Part 2 takes the loss from here and turns it into learning. It introduces the
gradient (which direction to move each parameter), backpropagation (how
PyTorch computes those directions), the optimizer and learning rate (how
big a step to take), and epoch and batch (how the loop repeats). Then it
runs the full training loop and watches the weight and bias climb from random noise
to 1.8 and 32.

Wait, why is there no activation function?

Most neural networks include an activation function: a step that reshapes a
layer's output so the network can learn relationships that are not straight lines.
This model leaves it out, and seeing why is a good way to learn what one does.

Many relationships are not straight. When the output climbs quickly in one place
and slowly in another, plotting it draws a curve: a line whose slope keeps
changing instead of staying fixed. A bend is just a point where the slope
changes. A few everyday examples make it concrete:

• A savings account with compound interest. The balance grows slowly at first, then faster and faster, because the interest itself earns interest. Over time the line curves upward.
• A phone battery charging. It races from 0 to 80 percent, then crawls through the last stretch to 100. The rate keeps changing, so the line bends.
• A web server's response time as load rises. Lightly loaded, a few more requests barely change it. Near full capacity, each extra request adds much more delay and the response time shoots up. Nearly flat, then steep.

Straight-line relationships, by contrast, hold one constant rate: total cost is the
price per item times the number of items, distance is speed times time at a steady
speed, and Celsius to Fahrenheit adds the same 1.8 for every degree. A single
linear layer matches those exactly. The moment the rate has to change, a straight
line cannot follow it, and that is the job an activation function does.

The most common activation function is ReLU. The rule is simple: replace every
negative value with zero, and leave positive values unchanged. Here it is applied
to five sample numbers:

ssh spark '~/venvs/w1/bin/python -c "
import torch; from torch import nn
# Five sample numbers, two negative, a zero, and two positive.
x=torch.tensor([-2.0, -0.5, 0.0, 1.5, 3.0])
# ReLU replaces every negative with 0 and leaves the rest unchanged.
print(nn.ReLU()(x))"'

tensor([0.0000, 0.0000, 0.0000, 1.5000, 3.0000])

The two negative inputs came out as zero, the zero stayed zero, and the two
positive ones passed through unchanged. That bend at zero is the ingredient that,
stacked across many neurons and layers, lets a network trace curves like the ones
above. Celsius to Fahrenheit is a straight line, though, so this model needs no
activation function. Adding a ReLU here would actually break it: its Fahrenheit
answers go negative (-40 C is -40 F), and ReLU would clamp those to zero. When a
task does need curves, an activation function is what provides them, and a later
week puts one to work.

So what does a curve mean for a model in practice? It sets a ceiling on what the
model can learn. A model built only from linear layers can draw straight lines and
nothing else, no matter how its weights are set or how many layers you stack,
because stacking straight-line layers just produces another straight line. If the
real relationship bends, that model can never fit it well. An activation function
is what lifts the ceiling: the bends let the network shape its output to follow
curved data.

That gives a simple rule for when to use one. If the relationship you are modeling
really is a straight line, like this Celsius-to-Fahrenheit toy or a plain linear
regression, you do not need an activation function. For almost everything else,
recognizing an image, predicting a price from many features, understanding text,
the relationship bends, so real networks add an activation function after every
hidden layer by default. This model is the rare exception, which is exactly why it
is a clean place to see what an activation function does by watching a model that
works fine without one.

Run it yourself

The public Week 6 lab has the training script used in Part 2, the captured runs,
observations, and troubleshooting notes.¹

1. Week 6 companion lab:
   https://github.com/dramasamy/from-api-to-gpu/tree/main/week-06-neural-network-basics ↩