Skip to content
Advertisement

Save and load model optimizer state

I have a set of fairly complicated models that I am training and I am looking for a way to save and load the model optimizer states. The “trainer models” consist of different combinations of several other “weight models”, of which some have shared weights, some have frozen weights depending on the trainer, etc. It is a bit too complicated of an example to share, but in short, I am not able to use model.save('model_file.h5') and keras.models.load_model('model_file.h5') when stopping and starting my training.

Using model.load_weights('weight_file.h5') works fine for testing my model if the training has finished, but if I attempt to continue training the model using this method, the loss does not come even close to returning to its last location. I have read that this is because the optimizer state is not saved using this method which makes sense. However, I need a method for saving and loading the states of the optimizers of my trainer models. It seems as though keras once had a model.optimizer.get_sate() and model.optimizer.set_sate() that would accomplish what I am after, but that does not seem to be the case anymore (at least for the Adam optimizer). Are there any other solutions with the current Keras?

Advertisement

Answer

You can extract the important lines from the load_model and save_model functions.

For saving optimizer states, in save_model:

JavaScript

For loading optimizer states, in load_model:

JavaScript

Combining the lines above, here’s an example:

  1. First fit the model for 5 epochs.
JavaScript
  1. Now save the weights and optimizer states.
JavaScript
  1. Rebuild the model in another python session, and load weights.
JavaScript
  1. Continue model training.
JavaScript
User contributions licensed under: CC BY-SA
3 People found this is helpful
Advertisement