Tag: out-of-memory

How to avoid memory error in Python while zipping list with large number of permutations from another list to get unique combinations?

I have the code below to generate all unique combinations between trucks and the permutations of tripares. The way I do it (below) gives me correct results. However it works only when len(tripares) < 10. When len(tripares) > 11 I get a memory error. Is there a way to avoid this memory error? Answer If y…

pandas out of memory error after variable assignment

dataframe out-of-memory pandas python

I have a very large pandas data frame and want to sample rows from it for modeling, and I encountered out of memory errors like this: MemoryError: Unable to allocate 6.59 GiB for an array with shape (40, 22117797) and data type float64 This error is weired since I don’t need allocate such large amount o…

Importing large IDL files into Python with SciPy

out-of-memory python

I currently use scipy.io.readsav() to import IDL .sav files to Python, which is working well, eg: However, if the .sav file is large (say > 1 GB), I get a MemoryError when trying to import into Python. Usually, iterating through the data would of course solve this (if it were a .txt or .csv file) rather th…

Python 3 causes memory error at shuffle(X,Y) where X is 36000 3-channel images (36000, 256,256,3) and Y is 3-channel normal data (36000, 256,256,3)

deep-learning out-of-memory python

Following image showing Memory Usage: Memory error occurs. I am using Numpy and Python3. I have two numpy arrays of shape (36000,256,256,3) each as X and Y and memory error occurs when I do following code. They are code to prepare training data. Is there another way to do it which uses lesser memory? This is …