Forward Propagation for Neural Network

I am trying to create a forward-propagation function in Python 3.8.2. The inputs look like this:

Test_Training_Input = [(1,2,3,4),(1.45,16,5,4),(3,7,19,67)]
Test_Training_Output = [1,1,0]

I am not using biases (not sure if they are that important and it makes my code very complicated) but I am using weights. The weights are stored in a list, Layer1W, I’m not sure how long to make it, but I think, len(Test_Training_Input)+len(Test_Training_Output) should work.

So far, my function looks like this:

def forwardprop():
    global Layer1O
    Layer1O = []
    for init in range(0,len(Layer1W)):
        total = sum(Test_Training_Input[1][1])*Layer1W[init]
        Layer1O.append(relu(total))
    return Layer1O

I think this is very wrong… Any suggestions?

Answer

This is a very rough explanation of a considerably complex topic, I highly recommend this e-book

import numpy as np
import numpy.random as rnd

#inputs
x = np.array([[0, 1, 0],
             [1, 0, 1]])

# x is of shape 2 x 3

#Here is a 3 -> 2 network, layer 1 (L1) will have 3 neurons, layer 2 (L2) will have 2
#   n00 is the first neuron of layer 1
#   n01 is the second neuron of layer 2
#   n02 is the third neuron of layer 3

#   n10 is the first neuron of layer 2
#   n11 is the second neuron of layer 2

#so there will be 6 weights, (let us assume their values also)
#   W(n00->n10) = 0.1, W(n00->n11) = 0.2
#   W(n01->n10) = 0.3, W(n01->n11) = 0.4
#   W(n02->n10) = 0.5, W(n02->n11) = 0.6

# we will store the corresponding weights in this fashion
# you might wonder why make it 2 x 3 matrix, and not 3 x 2
# this is to make use of matrix multiplication

#                 n00   n01  n02
w =    np.array([[0.1, 0.3, 0.5],  #n10
                 [0.2, 0.4, 0.6]]) #n11

# so the weight between n02 and n11 = w[1, 2] = 0.6

# note the input should be a vector,
# for example if the input is like this x = [1,1,0], shape = 3
# we will feed it as a vector, i.e
# [[1], <- x0
#  [1], <- x1
#  [0]] <- x2
# shape =  3 x 1

# so there will be 2 outputs from the feedforward, given by
#o1 = w00*x0 + w01*x1 + w02*x2
#o2 = w10*x0 + w11*x1 + w12*x2

# this is just the matrix multiplication between w and x,
#
#   [[0.1, 0.3, 0.5],   X  [[1],
#    [0.2, 0.4, 0.6]]      [1],     =    [[o1],
#                          [0]]           [o2]]
#

def feedforward(w, x):
    z = np.matmul(w, x)
    return z

# we will make x a vector, i.e change the shape as 2 x (3 x 1)
x = x[:, :, np.newaxis]
print(x.shape)
print(feedforward(w, x[0]))


# for your example
x = np.array([[1,2,3,4],[1.45,16,5,4],[3,7,19,67]])
# make it to 3 x 4 x 1
x = x[:, :, np.newaxis]

#lets try a 4 -> 2 -> 1 network

w1 = rnd.normal(size=(2, 4))
w2 = rnd.normal(size=(1, 2))

wgt = [w1, w2]

def feedforward2(wght, x):
    # best practice would be to put this in a class and store the outputs of each layer
    for w in wght:
        x = np.matmul(w, x)
    return x 

for i in x:
    print(feedforward2(wgt, i))