Is the linear model trainable in Convolution or NN?

Tags: , ,



If I’ve an following layer

x = Conv2D(x, activation='linear')

Is this layer trainable? As we know derivative of linear function is constant, so in this cases will the weight get ever updated? Situation like also

tf.keras.activation.linear (x) # no changes 
tf.keras.activation.relu (x) # will change 

Answer

The layer is trainable. Your data will be approximated by linear function.

Training process is finding a function which is the best approximation of your data. If you don’t use activation – your data will be approximated by linear function.

E.g. if your layer is Dense(1) – your data will be approximated by line. If your data is 2D – you can draw the points, run training and see that your data will be approximated by line: dense.w * x + dense.b.

The finction should be differentiable (for backpropagation). Linear finction is differentiable, so it is fine.

Loss function can not be linear because it should have minimum. But it is not required for layer.



Source: stackoverflow