If I’ve an following layer
x = Conv2D(x, activation='linear')
Is this layer trainable? As we know derivative of linear function is constant, so in this cases will the weight get ever updated? Situation like also
tf.keras.activation.linear (x) # no changes tf.keras.activation.relu (x) # will change
Advertisement
Answer
The layer is trainable. Your data will be approximated by linear function.
Training process is finding a function which is the best approximation of your data. If you don’t use activation – your data will be approximated by linear function.
E.g. if your layer is Dense(1) – your data will be approximated by line. If your data is 2D – you can draw the points, run training and see that your data will be approximated by line: dense.w * x + dense.b.
The finction should be differentiable (for backpropagation). Linear finction is differentiable, so it is fine.
Loss function can not be linear because it should have minimum. But it is not required for layer.