why does the mse loss had a sudden jump?

i’m working on a regression problem using neural network. the mse loss would decrease at the beginning of train and the accuracy is satisfactory, yet, when the train process goes on, the loss had a huge jump, and maintain at a certain value,like the curve in the picture. i don’t know why and how to fix it? and i wanna ask if i could use the train coefficient before the jump, like train step at 8000, as my final result?

Answer

This is a typical case of model training where the value of the accuracy metric stops improving (and even get worse) from a certain number of training epochs. I’ll suggest you to implement Early Stopping meaning that, “yes”, you can take the training accuracy at step 8000 as you final result if your only goal is to minimize the training loss.

This TF documentation explains how to implement Early Stopping with Tensorflow’s tf.keras.callbacks.EarlyStopping() function.

However if your goal is a model that generalizes well on unseen data (test/validation data) as this is generally the case, you might want to evaluate your model’s test accuracy in order to take it into account when implementing Early Stopping.

This article gives a very good example of end-to-end implementation of early stopping with Tensorflow.