Regularised Cost Function
If we have overfitting from our hypothesis function, we can reduce the weight that some of the terms in our function carry by increasing their cost.
Say we wanted to make the following function more quadratic:
We’ll want to eliminate the influence of θ3x3 and θ4x4. Without actually getting rid of these features or changing the form of our hypothesis, we can instead modify our cost function:
We‘ve added two extra terms at the end to inflate the cost of θ3 and θ4. Now, in order for the cost function to get close to zero, we will have to reduce the values of θ3 and θ4 to near zero. This will in turn greatly reduce the values of θ3x³ and θ4x⁴ in our hypothesis function. As a result, we see that the new hypothesis (depicted by the pink curve) looks like a quadratic function but fits the data better due to the extra small terms θ3x³ and θ4x⁴.
We could also regularize all of our theta parameters in a single summation as:
The λ, or lambda, is the regularization parameter. It determines how much the costs of our theta parameters are inflated.
