Why is my validation accuracy so much lower when I switch from doing all in-memory learning to a dada generator?

I have a data set that contains 2 columns:

1.) A string column consisting of 21 different letters. 2.) A classification column: Each of these strings is associated with a number from 1-7.

Using the following code, I first perform integer encoding.

codes = ['A', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'K', 'L',
     'M', 'N', 'P', 'Q', 'R', 'S', 'T', 'V', 'W', 'Y']

def create_dict(codes):
    char_dict = {}
    for index, val in enumerate(codes):
        char_dict[val] = index+1
 return char_dict

def integer_encoding(data):
"""
  - Encodes code sequence to integer values.
  - 20 common amino acids are taken into consideration
    and rest 4 are categorized as 0.
"""
    encode_list = []
    for row in data['Sequence'].values:
        row_encode = []
        for code in row:
            row_encode.append(char_dict.get(code, 0))
            encode_list.append(np.array(row_encode))
    return encode_list

Using this code, I am performing integer and then one-hot encoding all in memory.

char_dict = create_dict(codes)
train_encode = integer_encoding(balanced_train_df.reset_index()) 
val_encode = integer_encoding(val_df.reset_index()) 
train_pad = pad_sequences(train_encode, maxlen=max_length, padding='post', truncating='post')
val_pad = pad_sequences(val_encode, maxlen=max_length, padding='post', truncating='post')
train_ohe = to_categorical(train_pad)
val_ohe = to_categorical(val_pad)

Then I train my learner like so.

es = EarlyStopping(monitor='val_loss', patience=3, verbose=1)

history2 = model2.fit(
    train_ohe, y_train,
    epochs=50, batch_size=64,
    validation_data=(val_ohe, y_val),
    callbacks=[es]
)

This gives me validation accuracies that are decent for a first stab at about 86%.

Even the first epoch looks like this:

 Train on 431403 samples, validate on 50162 samples
 Epoch 1/50
 431403/431403 [==============================] - 187s 434us/sample - loss: 1.3532 - accuracy: 0.6947 - val_loss: 0.9443 - val_accuracy: 0.7730

Note the validation accuracy of 77% on first round.

But because my dataset is relatively big, I end up consuming about 50+Gb. This is so because I am loading the entire dataset into memory and convert the entire dataset and data transformations in memory.

To do my learning in a more memory efficient way, I am introducing a data generator like so:

class DataGenerator(Sequence):
    'Generates data for Keras'
    def __init__(self, list_IDs, data_col, labels, batch_size=32, dim=(32,32,32), n_channels=1,
             n_classes=10, shuffle=True):
    'Initialization'
    self.dim = dim
    self.batch_size = batch_size
    self.data_col_name = data_col
    self.labels = labels
    self.list_IDs = list_IDs
    self.n_channels = n_channels
    self.n_classes = n_classes
    self.shuffle = shuffle
    self.on_epoch_end()

def __len__(self):
    'Denotes the number of batches per epoch'
    return int(np.floor(len(self.list_IDs) / self.batch_size))

def __getitem__(self, index):
    'Generate one batch of data'
    # Generate indexes of the batch
    indexes = self.indexes[index*self.batch_size:(index+1)*self.batch_size]

    # Find list of IDs
    list_IDs_temp = [self.list_IDs[k] for k in indexes]

    # Generate data
    X, y = self.__data_generation(list_IDs_temp)

    return X, y

def on_epoch_end(self):
    'Updates indexes after each epoch'
    self.indexes = np.arange(len(self.list_IDs))
    if self.shuffle == True:
        np.random.shuffle(self.indexes)

def __data_generation(self, list_IDs_temp):
    'Generates data containing batch_size samples' # X : (n_samples, *dim, n_channels)
    # Initialization
    X = np.empty((self.batch_size, *self.dim))
    y = np.empty(self.batch_size, dtype=int)

    # Generate data
    for i, ID in enumerate(list_IDs_temp):
        # Store sample
        # Read sequence string and convert to array 
        # of padded categorical data in array
        int_encode_dt = integer_encoding(integer_encoding([balanced_train_df.loc[ID, self.data_col_name]]))
        padded_dt = pad_sequences(int_encode_dt, maxlen=660, padding='post', truncating='post')
        categorical_dt = to_categorical(padded_dt)
        X[i,] = categorical_dt
        # Store class
        y[i] = self.labels[ID]-1
    return X, to_categorical(y, num_classes=self.n_classes)

The code was adapted from here: https://stanford.edu/~shervine/blog/keras-how-to-generate-data-on-the-fly

Learning is then triggered like so:

params = {'dim': (660, 21), # sequences are at most 660 long and are encoded in 20 common amino acids,
      'batch_size': 32,
      'n_classes': 7,
      'n_channels': 1,
      'shuffle': False}

training_generator = DataGenerator(balanced_train_df.index, 'Sequence', balanced_train_df['ec_lvl_1'], **params)
validate_generator = DataGenerator(val_df.index, 'Sequence', val_df['ec_lvl_1'], **params)

# Early Stopping
es = EarlyStopping(monitor='val_loss', patience=3, verbose=1)

history2 = model2.fit(
    training_generator, 
    validation_data=validate_generator,
    use_multiprocessing=True,
    workers=6,
    epochs=50,
    callbacks=[es]
    )

The problem here is that my validation accuracies never exceed 15% using the data generator.

Epoch 1/10
13469/13481 [============================>.] - ETA: 0s - loss: 2.0578 - accuracy: 0.1427
13481/13481 [==============================] - 242s 18ms/step - loss: 2.0578 - accuracy: 0.1427 - val_loss: 1.9447 - val_accuracy: 0.0919

Note the validation accuracy of only 9%.

My question is why that is occurring? One thing I cannot explain is this:

When I do all in memory learning, I set the batch size to 32 or 64, but the number of steps remains roughly 413k (the total number of training samples). But when I use the data generators, I get much smaller numbers generally 413k samples/batch size. Is this telling me that I am not really using the batch size parameter in the in-memory learning case? Explanations appreciated.

Answer

A series of stupid errors cuased this discrepancy and they are all located in this one line here:

int_encode_dt = integer_encoding(integer_encoding([balanced_train_df.loc[ID, self.data_col_name]]))

Error 1: I should pass in the dataframe I want to process which allows me to feed in training and validation error. The way I did this before…even if I thought I passed in validation data, I would still use training data.

Error 2: I was double integer encoding my data (duh!)