I have a data set that contains 2 columns:
1.) A string column consisting of 21 different letters. 2.) A classification column: Each of these strings is associated with a number from 1-7.
Using the following code, I first perform integer encoding.
codes = ['A', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'K', 'L',
'M', 'N', 'P', 'Q', 'R', 'S', 'T', 'V', 'W', 'Y']
def create_dict(codes):
char_dict = {}
for index, val in enumerate(codes):
char_dict[val] = index+1
return char_dict
def integer_encoding(data):
"""
- Encodes code sequence to integer values.
- 20 common amino acids are taken into consideration
and rest 4 are categorized as 0.
"""
encode_list = []
for row in data['Sequence'].values:
row_encode = []
for code in row:
row_encode.append(char_dict.get(code, 0))
encode_list.append(np.array(row_encode))
return encode_list
Using this code, I am performing integer and then one-hot encoding all in memory.
char_dict = create_dict(codes) train_encode = integer_encoding(balanced_train_df.reset_index()) val_encode = integer_encoding(val_df.reset_index()) train_pad = pad_sequences(train_encode, maxlen=max_length, padding='post', truncating='post') val_pad = pad_sequences(val_encode, maxlen=max_length, padding='post', truncating='post') train_ohe = to_categorical(train_pad) val_ohe = to_categorical(val_pad)
Then I train my learner like so.
es = EarlyStopping(monitor='val_loss', patience=3, verbose=1)
history2 = model2.fit(
train_ohe, y_train,
epochs=50, batch_size=64,
validation_data=(val_ohe, y_val),
callbacks=[es]
)
This gives me validation accuracies that are decent for a first stab at about 86%.
Even the first epoch looks like this:
Train on 431403 samples, validate on 50162 samples Epoch 1/50 431403/431403 [==============================] - 187s 434us/sample - loss: 1.3532 - accuracy: 0.6947 - val_loss: 0.9443 - val_accuracy: 0.7730
Note the validation accuracy of 77% on first round.
But because my dataset is relatively big, I end up consuming about 50+Gb. This is so because I am loading the entire dataset into memory and convert the entire dataset and data transformations in memory.
To do my learning in a more memory efficient way, I am introducing a data generator like so:
class DataGenerator(Sequence):
'Generates data for Keras'
def __init__(self, list_IDs, data_col, labels, batch_size=32, dim=(32,32,32), n_channels=1,
n_classes=10, shuffle=True):
'Initialization'
self.dim = dim
self.batch_size = batch_size
self.data_col_name = data_col
self.labels = labels
self.list_IDs = list_IDs
self.n_channels = n_channels
self.n_classes = n_classes
self.shuffle = shuffle
self.on_epoch_end()
def __len__(self):
'Denotes the number of batches per epoch'
return int(np.floor(len(self.list_IDs) / self.batch_size))
def __getitem__(self, index):
'Generate one batch of data'
# Generate indexes of the batch
indexes = self.indexes[index*self.batch_size:(index+1)*self.batch_size]
# Find list of IDs
list_IDs_temp = [self.list_IDs[k] for k in indexes]
# Generate data
X, y = self.__data_generation(list_IDs_temp)
return X, y
def on_epoch_end(self):
'Updates indexes after each epoch'
self.indexes = np.arange(len(self.list_IDs))
if self.shuffle == True:
np.random.shuffle(self.indexes)
def __data_generation(self, list_IDs_temp):
'Generates data containing batch_size samples' # X : (n_samples, *dim, n_channels)
# Initialization
X = np.empty((self.batch_size, *self.dim))
y = np.empty(self.batch_size, dtype=int)
# Generate data
for i, ID in enumerate(list_IDs_temp):
# Store sample
# Read sequence string and convert to array
# of padded categorical data in array
int_encode_dt = integer_encoding(integer_encoding([balanced_train_df.loc[ID, self.data_col_name]]))
padded_dt = pad_sequences(int_encode_dt, maxlen=660, padding='post', truncating='post')
categorical_dt = to_categorical(padded_dt)
X[i,] = categorical_dt
# Store class
y[i] = self.labels[ID]-1
return X, to_categorical(y, num_classes=self.n_classes)
The code was adapted from here: https://stanford.edu/~shervine/blog/keras-how-to-generate-data-on-the-fly
Learning is then triggered like so:
params = {'dim': (660, 21), # sequences are at most 660 long and are encoded in 20 common amino acids,
'batch_size': 32,
'n_classes': 7,
'n_channels': 1,
'shuffle': False}
training_generator = DataGenerator(balanced_train_df.index, 'Sequence', balanced_train_df['ec_lvl_1'], **params)
validate_generator = DataGenerator(val_df.index, 'Sequence', val_df['ec_lvl_1'], **params)
# Early Stopping
es = EarlyStopping(monitor='val_loss', patience=3, verbose=1)
history2 = model2.fit(
training_generator,
validation_data=validate_generator,
use_multiprocessing=True,
workers=6,
epochs=50,
callbacks=[es]
)
The problem here is that my validation accuracies never exceed 15% using the data generator.
Epoch 1/10 13469/13481 [============================>.] - ETA: 0s - loss: 2.0578 - accuracy: 0.1427 13481/13481 [==============================] - 242s 18ms/step - loss: 2.0578 - accuracy: 0.1427 - val_loss: 1.9447 - val_accuracy: 0.0919
Note the validation accuracy of only 9%.
My question is why that is occurring? One thing I cannot explain is this:
When I do all in memory learning, I set the batch size to 32 or 64, but the number of steps remains roughly 413k (the total number of training samples). But when I use the data generators, I get much smaller numbers generally 413k samples/batch size. Is this telling me that I am not really using the batch size parameter in the in-memory learning case? Explanations appreciated.
Advertisement
Answer
A series of stupid errors cuased this discrepancy and they are all located in this one line here:
int_encode_dt = integer_encoding(integer_encoding([balanced_train_df.loc[ID, self.data_col_name]]))
Error 1: I should pass in the dataframe I want to process which allows me to feed in training and validation error. The way I did this before…even if I thought I passed in validation data, I would still use training data.
Error 2: I was double integer encoding my data (duh!)