Gradient Cost Function

We experimented with a gradient cost function, which evaluates the "cost" of an AI prediction (how close the predictions are to actual values) and the "gradient" of the prediction (which direction to go in to improve the predictions).

We changed the iteration and learning rate values of the function to see what impact this had on the "cost".

We began with a simple set of data points in a straight line.

x = [1, 2, 3, 4, 5]
y = [5, 7, 9, 11, 13]

The graph above shows the line of best fit has slope 2 and y-intercept 3.

We used "gradient descent" to try to find the line of best fit. The line is described by the slope "m" and the y-intercept "b".

Here is an explanation of the variables in the code of the gradient cost function below.

m_curr	current predicted value of slope "m"
b_curr	current predicted value of y-intercept "b"
iterations	how many times "m_curr" and "b_curr" will be updated
n	number of data points (5 in this example)
learning_rate	percentage of error gradient to move for each iteration
y_predicted	predicted values for y (using "m_curr" and "b_curr")
cost	how close the predictions are to actual values
md	error gradient of the "m" value (which direction "m" needs to go in to reduce the "cost")
bd	error gradient of the "b" value (which direction "b" needs to go in to reduce the "cost")

In [ ]:

import numpy as np

def gradient_descent(x,y):
    m_curr = b_curr = 0
    iterations = 100
    n = len(x)
    learning_rate = 0.08

    for i in range(iterations):
        y_predicted = m_curr * x + b_curr
        cost = (1/n) * sum([val**2 for val in (y-y_predicted)])
        md = -(2/n)*sum(x*(y-y_predicted))
        bd = -(2/n)*sum(y-y_predicted)
        m_curr = m_curr - learning_rate * md
        b_curr = b_curr - learning_rate * bd

    print ("iterations {}, learning_rate {}".format(iterations,learning_rate))
    print ("m {}, b {}, cost {}".format(m_curr,b_curr,cost))

x = np.array([1,2,3,4,5])
y = np.array([5,7,9,11,13])

gradient_descent(x,y)

Here is the initial result with iterations 100 and learning_rate 0.08. The value of "m" is already close to 2 and the value of "b" is getting close to 3. The "cost" is low.

iterations 100, learning_rate 0.08
m 2.0405161418256377, b 2.8536001910078013, cost 0.004120600119124239

I tried more iterations and a higher learning rate - a very small increase in the learning rate caused the predictions to fluctuate dramatically.

iterations 1000, learning_rate 0.1
m -8.532027941246304e+135, b -2.3632349282172212e+135, cost 4.968263354798973e+272

I tried more iterations but kept the learning rate at 0.08. This gave very good results - the slope "m" is almost exactly 2 and the y-intercept "b" is almost exactly 3.

iterations 1000, learning_rate 0.08
m 2.0000000000007776, b 2.999999999997196, cost 1.5126502520921528e-24

I learned from the exercise that this very simple gradient cost function is not reliable - small changes in the learning rate caused erratic results. This shows why more stable and efficient methods such as Mini-Batch Gradient Descent are used in the AI industry.

I can definitely take what I learned about gradient descent functions from this practical example into my future career as a data scientist!