How to train keras models consecutively

I’m trying to train different models consecutively without needing to re-run my program or change my code all the time, so this way I can let my PC training different models.

I use a for loop while feeding different information from a dictionary for building different models each time, and so I can train a new model each time de function gets called, for testing the accuracy on different setups to understand which one is the best on each case.

def create_model(modeltoload):
    model = Sequential()
    previsores, alto, baixo, fechado, aberto = get_train_data(modeltoload)
    if modeltoload['Type'] == 'LSTM':
        if len(modeltoload['Layers']) == 1:
            model.add(LSTM(units=modeltoload['Layers'][0], activation='tanh',
                           input_shape=(previsores.shape[1], modeltoload['Entry'])))
            model.add(Dropout(0.3))
        else:
            model.add(LSTM(units=modeltoload['Layers'][0], activation='tanh', return_sequences=True,
                           input_shape=(previsores.shape[1], modeltoload['Entry'])))
            model.add(Dropout(0.3))
        for i in range(1, len(modeltoload['Layers'])):
            if i == (len(modeltoload['Layers'])-1):
                model.add(LSTM(units=modeltoload['Layers'][i], activation='tanh'))
            else:
                model.add(LSTM(units=modeltoload['Layers'][i], activation='tanh', return_sequences=True))
        model.add(Dense(units=1, activation='relu'))
    if modeltoload['Type'] == 'DENSE':
        model.add(Dense(units=modeltoload['Layers'][0], activation='relu', input_dim=modeltoload['Entry']*5+1))
        model.add(Dropout(0.1))
        for i in range(1, len(modeltoload['Layers'])):
            model.add(Dense(units=modeltoload['Layers'][i], activation='relu'))
            model.add(Dropout(0.1))
        model.add(Dense(units=1, activation=modeltoload['Activation']))
    model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['binary_accuracy'])
    return model

and then

def train_model(modeltoload):
    previsores, classe, baixo, fechado, aberto = get_train_data(modeltoload)
    model = create_model(modeltoload)
    history1 = model.fit(previsores, classe, epochs=1000, batch_size=modeltoload['Batch'],
                         callbacks=[es, rlr, mcp, csv], shuffle='batch', verbose=2, validation_split=0.1)
    k.clear_session()
    del model
    return history1

The problem is that when I start the first training session, it goes all good like this:

Training: DENSE/60N-14B-190E-tanh.h5
Train on 2575 samples, validate on 287 samples
Epoch 1/1000
Epoch 00001: loss improved from inf to 2.50127, saving model to DENSE/60N-14B-190E-tanh.h5
 - 1s - loss: 2.5013 - binary_accuracy: 0.4711 - val_loss: 1.1434 - val_binary_accuracy: 0.5017
Epoch 2/1000 
.
.
.
Epoch 307/1000
Epoch 00307: loss did not improve
 - 0s - loss: 0.5200 - binary_accuracy: 0.7522 - val_loss: 0.8077 - val_binary_accuracy: 0.5401
Epoch 00307: early stopping

But when the second and so on model gets created, the loss start not on [inf] but at the last value of the prior training:

Training: DENSE/60N-14B-220E-tanh.h5
Train on 2548 samples, validate on 284 samples
Epoch 1/1000
Epoch 00001: loss did not improve
 - 1s - loss: 1.3203 - binary_accuracy: 0.5063 - val_loss: 0.7724 - val_binary_accuracy: 0.5246
Epoch 2/1000
Epoch 00002: loss did not improve
 - 0s - loss: 0.7366 - binary_accuracy: 0.4945 - val_loss: 0.7247 - val_binary_accuracy: 0.5000

Even when using

k.clear_session() 
del model 

It seems that I am loading some previous information about the last trained model. Does someone have an insight about this issue?

Answer

I assume from the training progress output you included that you are using Keras’s ModelCheckpoint callback. If you use the same ModelCheckpoint for multiple training runs, it will only save your new model if the loss of the new model is an improvement on the previous saved model.

To fix this issue, just generate the ModelCheckpoint object each time within your train_model function.