It’s a bit annoying that tf.keras generator still faces this issue, unlike pytorch. There are many discussions regarding this, however, still stuck with it. Already visit:
Problem
I have a data set consist of around 21397. I wrote a custom data loader which returns the total number of samples as follows:
class NGenerator(Sequence):
...
def __len__(self):
return int(np.ceil(float(len(self.data)) / float(self.batch_size)))
...
From the data, I’ve made 5 fold subset of it. Each fold contains as follows:
Fold Amount 1.0 4280 0.0 4280 2.0 4279 3.0 4279 4.0 4279
For each fold, I’ve set step_per_epoch and validation_per_epoch as follows:
# here, train_labels is the df of the subset based on fold steps_per_epoch = np.ceil(float(len(train_labels)) / float(batch_size)) validation_steps = np.ceil(float(len(val_labels)) / float(batch_size))
Now, to make an OOF score, we predict on the validation set and wanted to store results as follows:
batch_size = 64 oof = np.zeros(len(df)) ... for each_fold, (trn_idx, val_idx) in enumerate(skf...): train_labels = df.iloc[self.trn_idx].reset_index(drop=True) val_labels = df.iloc[self.val_idx].reset_index(drop=True) .... train_gen, val_gen = ..., model.fit() pred = model.predict(val_gen, steps=validation_steps) oof[self.val_idx] = np.argmax(pred, axis=1) < --------- HERE
After training, at indexing time (oof), it throws a size mismatch of shape between 4280 and 4288. So, it looks like, with this step size and batch size, the model is predicting 8 samples of the next batch. Next, we set batch_size equal to 40 which dividable by the total number of the subset (4280). Good enough but (of course) faced again size mismatch in Fold 2 of shape between 4279 and 4280. One of the simple workarounds is to add 3 samples in fold 2,3,4 -_-
Any general tips to get rid of it? Thanks.
Advertisement
Answer
Did not have time to go through all your code however I thought the code below might be useful to you. The variable-length should be set to the number of samples. Then the code determines a batch size and steps per epoch such that length = batch_size*steps per epoch. Variable b_max should be set to the maximum batch size you will allow based on memory capacity. Note if the length is a prime number batch size will end up as 1 and steps will end up as length.
def get_bs(length, b_max): batch_size=sorted([int(length/n) for n in range(1,length+1) if length % n ==0 and length/n<=b_max],reverse=True)[0] steps=(length/batch_size) return batch_size, steps
I use this to set validation steps so during validation the samples in the validation set are processed exactly once. An example is shown below.
batch_size, steps = get_bs(2048, 90)
print ('batch_size = ', batch_size, ' steps = ', steps)
# result is batch_size = 64 steps = 32.0