You've tried to understand how Neural Networks actually learn, right?

You open a textbook... you click on a video...

You open a textbook... you click on a video...

and within minutes, you're drowning.

Backpropagation!

Stochastic Gradient Descent!

The Chain Rule!

It feels like an impossibly complex black box.

A machine that just... performs magic.

A machine that just... performs magic.

And you're told to just accept that it works.

Here's the secret:

Here's the secret:

You've been taught this the wrong way around.

The entire engine that powers all of modern AI...

The entire engine that powers all of modern AI...

...is built on just a few simple, incredibly intuitive ideas.

Idea #1: The Valley

How do you find the bottom of a valley when you’re stuck in a thick fog?

Idea #2: The Blame Game

And how do you figure out who to "blame" when a team project goes wrong?

Once you truly grasp these two concepts...

Once you truly grasp these two concepts...

...all of that scary math suddenly clicks into place.

It’s not a barrier; it's just the language we use to describe a logic you already understand.

So here’s my promise to you.

Give me 40 minutes, and you will MASTER how Neural Networks learn.

We will go step-by-step, building the entire learning process from scratch.

No skipped steps. No magic.

By the end of this video, you will have a deep, foundational understanding of Neural Networks.

By the end of this video, you will have a deep, foundational understanding of Neural Networks.

You won't just know the buzzwords.

You will finally get that 'Aha!' moment.

Alright, let's get straight to the code.

INPUT: function f(x)
OUTPUT: argmin_x f(x)

FOR 100 iterations:
  gradient = f'(x)
  x = x - η × gradient
RETURN x
    

This tiny loop right here?

This tiny loop right here?

...is the beating heart of EVERY single AI system you have ever heard of.

ChatGPT, Midjourney, Self-Driving Cars... they ALL run on this exact logic.

So, what is it doing?

Every neural network is trying to get better by minimizing its "error"

Every neural network is trying to get better by minimizing its "error"

—the gap between its guess and the right answer.

This algorithm is the engine that drives that error down to zero.

The intuition is dead simple.

Remember that foggy hill I mentioned?

You're lost, you need to get to the bottom of the valley

but you can only see your own two feet.

You're lost, you need to get to the bottom of the valley

but you can only see your own two feet.

What do you do?

You don't need to see the whole map. You just:

That's it.

That's Gradient Descent.

You just feel the slope, take a step

and do it again and again until you reach the bottom where the ground is flat.

Our algorithm does the exact same thing, but with math.

Let's make this crystal clear with an example.

The gradient is found with the derivative:

The gradient is found with the derivative:

At x = 3, slope is f'(3) = 2×3 = 6. Positive slope means "downhill" is to the left.

The gradient is found with the derivative:

At x = -2, slope is f'(-2) = 2×(-2) = -4. Negative slope means "downhill" is to the right.

The gradient is found with the derivative:

At x = 0, slope is f'(0) = 0. The ground is flat. You've arrived!

The Update Rule

This just automates the process. It subtracts the slope, forcing x to always move downhill towards the minimum.

Look at that! We started at x=3 with a huge error of 9. After just a few steps, it’s plummeting.

The algorithm is literally sliding down the curve of the parabola

But... here comes the plot twist.

Gradient descent has tunnel vision.

It only sees the slope directly under its feet.

What if the landscape isn't a simple valley?

If you start at x = 0.5...

If you start at x = 0.5...

If you start at x = 0.5...

The algorithm slides into the shallow valley and gets stuck.

It found an answer, but not the best answer.

But if you start at x = -0.5...

But if you start at x = -0.5...

But if you start at x = -0.5...

It finds the true, deep valley perfectly.

Same algorithm, different starting points, wildly different results.

This raises a terrifying question.

A neural network has millions of parameters.

Creating an error landscape with billions of traps.

How could this work for GPT-4?

Here's the surprising, almost unbelievable answer:

...for huge neural networks, it almost doesn't matter.

The Surprising Luck of High Dimensions

It's one of the luckiest coincidences in the history of AI,

and we're still trying to fully understand why.

Empirically, gradient descent just... works.

Now you understand the core engine.

But so far, our valley only has one dimension. We can only move left or right.

A real neural network is like a massive soundboard with a million knobs to tune.

How do you adjust all of them at once to find the perfect sound?

How do you figure out the slope in a million different directions simultaneously?

Alright, so we've mastered finding the bottom of a 1D valley. We can move left and right.

But a real neural network isn't a single slider.

It's a massive soundboard with millions of knobs.

How do you find the steepest downhill path...

How do you find the steepest downhill path...

...when 'downhill' is in a million different directions?

The answer is surprisingly elegant.

You don't.

Instead, you figure out the slope for each knob individually...

Instead, you figure out the slope for each knob individually...

...as if it were the only one you were turning.

You focus on one knob at a time...

listen to its effect...

adjust it... and move to the next.

This is the core intuition behind one of the most important tools in machine learning:

The Partial Derivative.

When our function has multiple variables...

We can't just ask for "the slope"

We need to specify which direction

This uses the partial derivative symbol:

"Slope in the x₁ direction only"

And for the other direction:

"Slope in the x₂ direction only"

So how do we calculate these partial derivatives?

Here is the one magic rule you need to remember.

To find the partial derivative with respect to one variable...

...you treat ALL OTHER variables as if they are just constant numbers.

You've just calculated a multi-dimensional gradient.

With this tool, our Gradient Descent algorithm gets a simple upgrade.

Instead of one update rule, we now have one for each variable...

and here's the key: we apply them all at the same time.

INPUT: function f(x1,x2)
FOR 100 iterations:
  grad_x1 = ∂f/∂x1
  grad_x2 = ∂f/∂x2

  x1 = x1 - η × grad_x1
  x2 = x2 - η × grad_x2
RETURN (x1,x2)
    

Look at that beautiful convergence!

In the first step, the gradient tells it to move 0.6 in the `x1` direction and 0.8 in the `x2` direction...

...slashing the error in half.

As it gets closer to the bottom, the slopes get smaller, so it takes smaller, more careful steps.

`x1` and `x2` both spiral down towards zero, perfectly finding the minimum of our 3D bowl.

This is it. This is the fundamental technique for training a neural network.

We treat every single weight as its own "knob".

We calculate its partial derivative—its individual contribution to the total error...

...and then we nudge it slightly in the right direction.

Scale this up from 2 knobs to 2 million, and you have modern machine learning.

But this raises a new, much more subtle problem.

In our bowl example, `x1` and `x2` directly affected the final error.

In a deep neural network...

A weight in the first layer doesn't directly touch the final error.

Its influence travels through a long, complex chain.

How do you calculate the "blame" for a single knob when its effect is buried 20 layers deep?

This is the single biggest problem in deep learning...

...and its solution is one of the most elegant ideas in all of mathematics.

The Chain Rule.

The Formula

This simple formula is the masterstroke.

It's what makes deep learning possible.

And the intuition behind it?

It's literally a blame game.

Imagine your team's final presentation fails. That's the error.

To find out why, you trace the problem backward:

To find out why, you trace the problem backward:

To find out why, you trace the problem backward:

To find out why, you trace the problem backward:

To find out how much the initial data collector is responsible for the final failed presentation...

...you just multiply the blame at each step.

90% × 80% × 50% = 36%

The Chain Rule does exactly this.

It multiplies the influence at each link in the chain to find the total impact of a variable far, far away.

Let's see this mathematical blame game in action.

\(x_2\) influences \(v\) in two ways: indirectly through \(u\), and directly.

The total blame is just the sum of the blame from all paths.

And just like that...

the Chain Rule has untangled that complex nested function for us.

Now, look what we can do.

FOR 100 iterations:
  # Calculate current values
  u = 2*x1 + 3*x2
  v = x1 + x2**2
  
  # Calculate the blame for each variable
  grad_x1 = 12 * u * v**2
  grad_x2 = (3 * v**2) * (2*u + 6*x2)

  # Nudge each variable in the right direction
  x1 = x1 - η * grad_x1
  x2 = x2 - η * grad_x2
RETURN (x1,x2)
    

This is the key.

The Chain Rule gives us a systematic way to find the gradient for any variable...

This is the key.

The Chain Rule gives us a systematic way to find the gradient for any variable...

...no matter how deeply it's buried inside a complex function.

This is the final piece of the puzzle.

We have all the mathematical tools we need.

Now, it's time to stop playing with abstract functions.

Let's use these tools to build our very first, functioning AI brain...

...and watch it actually learn, right before your eyes.

So, what even *is* a 'Neural Network'?

Forget the hype. Forget the sci-fi.

It's just a collection of simple functions, called "neurons," organized in "layers."

Each neuron is just a tiny calculation. Nothing magical.

f(a,b) = a + b²
    

We stack them in layers, like an assembly line.

This creates a Forward Pass—a one-way flow of information from data to answer.

Let's build one right now.

This is the Forward Pass.

We just feed inputs through and see what comes out.

No learning yet. Just pure calculation.

Ouch. Not great.

For the second example, it was off by a massive 19 points.

But how do we quantify this failure?

We need a single number that tells us, overall, how wrong our network is.

We can't just add them up...

...because a -19 and a +19 would cancel out, making it look perfect when it's terrible.

The solution?

We SQUARE the differences.

Why Squaring Works:

Let's calculate the Total Squared Error.

There it is: 534

That is our antagonist. Our enemy.

Our goal for the rest of this video...

...is to make this number—534—

...as close to zero as possible.

We've completed the Forward Pass.

Our network has made its first, terrible predictions, and we've measured its failure.

Now for the 'Aha!' moment.

We're going to take all the tools we've learned

—Gradient Descent, Partial Derivatives, and the Chain Rule—

and unleash them on this network.

Remember the "Blame Game" with the Chain Rule?

Tracing responsibility back through a complex chain?

And remember our complex nested function from that section?

Here is the single most important insight of this entire video.

A Neural Network IS just a giant, nested function.

That's it. That's the whole secret.

Look at the comparison.

This raises the most critical question:

What are the "knobs" we are allowed to tune in our neural network? What are the variables?

Is it the inputs, x1 and x2?

NO!

The inputs are the data we're given.

We can't change the fact that we were given (x1=3, x2=2).

That's the problem we have to solve.

The ONLY things we have control over...

... are the weights. `w1, w2, w3, w4, w5`.

This is the breakthrough.

The Real Comparison

We're not trying to find the best inputs.

We're trying to find the best weights that turn the inputs into the correct outputs.

We already know exactly how to do this!

We use Gradient Descent to find the values of our variables...

...that minimize a function's output.

The process is identical.

It's time to play the Blame Game.

Let's focus on that first terrible prediction.

Now stop.

Look at what we just did. Look closer.

Do you see it? The term ∂Error/∂y_pred = 26 is in BOTH calculations!

This is the multi-trillion dollar trick that makes deep learning efficient.

We don't recalculate everything from scratch for every weight.

We calculate the error gradient once...

...then propagate it backwards, reusing calculations.

This is Backpropagation.

It's not magic.

It's just being clever and not re-doing work you've already done.

Systematically Finding the Blame

Forward Pass: Computing the Output

# Inputs: x1, x2
# Weights: w1, w2, w3, w4, w5

# Hidden Layer
h1 = (x1 + w1*x2)²
h2 = w2*x1*x2

# Output Layer  
y_pred = w3*h1 + w4*h2 + w5

# Error
Error = (y_true - y_pred)²
            

Backward Pass: Computing Gradients

# Start with error gradient
∂E/∂y_pred = 2(y_pred - y_true)

# Output layer weights
∂E/∂w5 = ∂E/∂y_pred * 1
∂E/∂w4 = ∂E/∂y_pred * h2  
∂E/∂w3 = ∂E/∂y_pred * h1

# Hidden layer gradients
∂E/∂h1 = ∂E/∂y_pred * w3
∂E/∂h2 = ∂E/∂y_pred * w4

# Hidden layer weights  
∂E/∂w1 = ∂E/∂h1 * 2(x1+w1*x2)*x2
∂E/∂w2 = ∂E/∂h2 * x1*x2
            

That's it. The learning has happened.

We played the Blame Game, and we commanded each weight to adjust itself.

But... did it work?

This is the moment of truth.

Let's compare.

YES! It worked!

The prediction moved from 37 down to 33.95.

It got closer to the target of 24. The error got smaller.

This is not a guess. This is not magic.

This is intelligence emerging directly from mathematics.

Following the chain of blame backwards

Nudging parameters downhill

The network learns automatically

Repeat this process a thousand times, and it will get even closer.

A million times, and it will be nearly perfect.

But hang on.

Think about what we just did.

We trained our network on one single example.

`(3, 2) -> 24`

That's like studying for a final exam by memorizing the answer to a single practice question.

You might ace that one question, but you haven't learned the concept.

You'll fail the actual test.

To become truly intelligent, our network can't just memorize one data point.

It needs to find the underlying pattern that works for ALL of our examples.

So, the next logical question is:

How do you learn from the entire crowd of data at once?

There are two main strategies for this.

The Algorithms Side-by-Side

Let's say we have N training examples: {(x₁, y₁), (x₂, y₂), ..., (xₙ, yₙ)}

For each example i = 1 to N:
  // Calculate gradient for ONE example
  gradient_i = ∇Loss(θ, xᵢ, yᵢ)
  // Update weights IMMEDIATELY
  θ = θ - η × gradient_i

total_gradient = 0
For each example i = 1 to N:
  // Calculate gradient for this example
  gradient_i = ∇Loss(θ, xᵢ, yᵢ)
  // ADD to the running total
  total_gradient += gradient_i
// Update weights ONCE at the end
θ = θ - η × total_gradient

Online learning is impulsive.

It learns from one example and immediately changes its mind.

Online learning is impulsive.

Batch learning is more deliberate.

It listens to the opinion of every single example before making a collective, democratic decision.

Let's try it.

Let's perform one update using Batch Learning and see what happens.

This update direction is a compromise.

It’s the average best direction that helps reduce the error across ALL our data, not just one example.

But did it work?

Let's verify.

ALL examples improved!

One update. Works for everyone.

This is how a network learns to generalize.

So, which one is better?

So what do we use in the real world?

We compromise.

We use Mini-batch Learning.

This is the sweet spot. It's the industry standard that powers virtually all modern AI.

Mini-batch:

Mini-batch Learning Algorithm

For each mini-batch of size B (where B < N):
  total_gradient = 0
  For each example j in mini-batch:
    // Calculate gradient for this example
    gradient_j = ∇Loss(θ, xⱼ, yⱼ)
    // ADD to mini-batch total
    total_gradient += gradient_j
  // Update weights using mini-batch average
  θ = θ - η × (total_gradient / B)

It's the best of both worlds.

The gradients are stable enough from the small crowd, and the memory usage is perfectly manageable.

This is how you train a network on a dataset of billions of images.

This is how you train a network on a dataset of billions of images.

You feed it one small handful at a time.

So now we have a complete, practical learning loop that can handle massive datasets.

We have all the pieces.

The only question left is... how does this simple process, which we ran on a tiny network with only 5 weights...

...possibly scale up to train a monster like GPT-4 with over a trillion weights?

Is it really the same algorithm?

The answer is the most beautiful part of this entire story.

It's the same three-step dance.

Scale changes everything, and yet, it changes nothing.

The stunning truth is this:

You just mastered the core algorithm running inside every AI system on Earth.

ChatGPT, autonomous vehicles, medical diagnosis AI...

...they are all just scaled-up, engineered versions of what we just built together from scratch.

So, how do we get from our simple model to these massive ones?

The core principles don't change. Same engine, bigger scale.

1. More Layers and More Neurons

This just means the "chain" for the Chain Rule gets much, much longer. The process is identical.

2. More Practical Activation Functions

3. Better Loss Functions for Different Jobs

The specific formula changes, but its job is always the same: give us a number to kick off the blame game.

And that's it. That's the whole secret.

You have seen the entire process.

From Midjourney to Tesla...

...they all use the exact same loop.

The Universal Pattern

You started this hour thinking neural networks were an impenetrable black box.

But now you know the truth.

It's not magic.

It's just a beautiful cascade of simple, intuitive ideas:

finding the bottom of a valley, tuning one knob at a time, and playing a clever game of blame.

You've mastered the fundamentals.

Welcome to the world of AI.