It’s a bit annoying that `tf.keras`

generator still faces this issue, unlike `pytorch`

. There are many discussions regarding this, however, still stuck with it. Already visit:

### Problem

I have a data set consist of around **21397**. I wrote a custom data loader which returns the total number of samples as follows:

class NGenerator(Sequence): ... def __len__(self): return int(np.ceil(float(len(self.data)) / float(self.batch_size))) ...

From the data, I’ve made 5 fold subset of it. Each fold contains as follows:

Fold Amount 1.0 4280 0.0 4280 2.0 4279 3.0 4279 4.0 4279

For each fold, I’ve set `step_per_epoch`

and `validation_per_epoch`

as follows:

# here, train_labels is the df of the subset based on fold steps_per_epoch = np.ceil(float(len(train_labels)) / float(batch_size)) validation_steps = np.ceil(float(len(val_labels)) / float(batch_size))

Now, to make an **OOF** score, we predict on the validation set and wanted to store results as follows:

batch_size = 64 oof = np.zeros(len(df)) ... for each_fold, (trn_idx, val_idx) in enumerate(skf...): train_labels = df.iloc[self.trn_idx].reset_index(drop=True) val_labels = df.iloc[self.val_idx].reset_index(drop=True) .... train_gen, val_gen = ..., model.fit() pred = model.predict(val_gen, steps=validation_steps) oof[self.val_idx] = np.argmax(pred, axis=1) < --------- HERE

After training, at indexing time (`oof`

), it throws a size mismatch of shape between `4280`

and `4288`

. So, it looks like, with this `step size`

and `batch size`

, the model is predicting `8`

samples of the next batch. Next, we set `batch_size`

equal to `40`

which dividable by the total number of the subset (`4280`

). Good enough but (of course) faced again size mismatch in **Fold 2** of shape between `4279`

and `4280`

. One of the simple workarounds is to add `3`

samples in fold `2,3,4`

-_-

Any general tips to get rid of it? Thanks.

## Advertisement

## Answer

Did not have time to go through all your code however I thought the code below might be useful to you. The variable-length should be set to the number of samples. Then the code determines a `batch size`

and `steps per epoch`

such that `length = batch_size*steps`

per epoch. Variable `b_max`

should be set to the **maximum batch size** you will allow based on memory capacity. Note if the `length`

is a prime number batch size will end up as `1`

and `steps`

will end up as `length`

.

def get_bs(length, b_max): batch_size=sorted([int(length/n) for n in range(1,length+1) if length % n ==0 and length/n<=b_max],reverse=True)[0] steps=(length/batch_size) return batch_size, steps

I use this to set `validation steps`

so during validation the samples in the validation set are processed exactly once. An example is shown below.

batch_size, steps = get_bs(2048, 90) print ('batch_size = ', batch_size, ' steps = ', steps) # result is batch_size = 64 steps = 32.0

**9**People found this is helpful