I’m trying to train different models consecutively without needing to re-run my program or change my code all the time, so this way I can let my PC training different models.
I use a for loop while feeding different information from a dictionary for building different models each time, and so I can train a new model each time de function gets called, for testing the accuracy on different setups to understand which one is the best on each case.
def create_model(modeltoload): model = Sequential() previsores, alto, baixo, fechado, aberto = get_train_data(modeltoload) if modeltoload['Type'] == 'LSTM': if len(modeltoload['Layers']) == 1: model.add(LSTM(units=modeltoload['Layers'][0], activation='tanh', input_shape=(previsores.shape[1], modeltoload['Entry']))) model.add(Dropout(0.3)) else: model.add(LSTM(units=modeltoload['Layers'][0], activation='tanh', return_sequences=True, input_shape=(previsores.shape[1], modeltoload['Entry']))) model.add(Dropout(0.3)) for i in range(1, len(modeltoload['Layers'])): if i == (len(modeltoload['Layers'])-1): model.add(LSTM(units=modeltoload['Layers'][i], activation='tanh')) else: model.add(LSTM(units=modeltoload['Layers'][i], activation='tanh', return_sequences=True)) model.add(Dense(units=1, activation='relu')) if modeltoload['Type'] == 'DENSE': model.add(Dense(units=modeltoload['Layers'][0], activation='relu', input_dim=modeltoload['Entry']*5+1)) model.add(Dropout(0.1)) for i in range(1, len(modeltoload['Layers'])): model.add(Dense(units=modeltoload['Layers'][i], activation='relu')) model.add(Dropout(0.1)) model.add(Dense(units=1, activation=modeltoload['Activation'])) model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['binary_accuracy']) return model
and then
def train_model(modeltoload): previsores, classe, baixo, fechado, aberto = get_train_data(modeltoload) model = create_model(modeltoload) history1 = model.fit(previsores, classe, epochs=1000, batch_size=modeltoload['Batch'], callbacks=[es, rlr, mcp, csv], shuffle='batch', verbose=2, validation_split=0.1) k.clear_session() del model return history1
The problem is that when I start the first training session, it goes all good like this:
Training: DENSE/60N-14B-190E-tanh.h5 Train on 2575 samples, validate on 287 samples Epoch 1/1000 Epoch 00001: loss improved from inf to 2.50127, saving model to DENSE/60N-14B-190E-tanh.h5 - 1s - loss: 2.5013 - binary_accuracy: 0.4711 - val_loss: 1.1434 - val_binary_accuracy: 0.5017 Epoch 2/1000 . . . Epoch 307/1000 Epoch 00307: loss did not improve - 0s - loss: 0.5200 - binary_accuracy: 0.7522 - val_loss: 0.8077 - val_binary_accuracy: 0.5401 Epoch 00307: early stopping
But when the second and so on model gets created, the loss start not on [inf] but at the last value of the prior training:
Training: DENSE/60N-14B-220E-tanh.h5 Train on 2548 samples, validate on 284 samples Epoch 1/1000 Epoch 00001: loss did not improve - 1s - loss: 1.3203 - binary_accuracy: 0.5063 - val_loss: 0.7724 - val_binary_accuracy: 0.5246 Epoch 2/1000 Epoch 00002: loss did not improve - 0s - loss: 0.7366 - binary_accuracy: 0.4945 - val_loss: 0.7247 - val_binary_accuracy: 0.5000
Even when using
k.clear_session() del model
It seems that I am loading some previous information about the last trained model. Does someone have an insight about this issue?
Advertisement
Answer
I assume from the training progress output you included that you are using Keras’s ModelCheckpoint callback. If you use the same ModelCheckpoint for multiple training runs, it will only save your new model if the loss of the new model is an improvement on the previous saved model.
To fix this issue, just generate the ModelCheckpoint object each time within your train_model
function.