I have an initial NumPy array of size (512,512,100)
of type np.float64
and then I use view_as_windows function to get an array of size (499,499,100,64,64)
. This function returns a view that consumes much less memory than an actual NumPy array. I want to reshape the view to (499*499*100,64,64)
. Using the regular np.reshape
function, it takes too much time trying to reshape the array, and then it breaks due to trying to convert it to an actual NumPy array. I tried the following, and none of them works:
#gs.shape gives (499,499,100,64,64) gs.shape = (gs.shape[0]*gs.shape[1]*gs.shape[2],64,64,) gs = np.reshape(gs,(gs.shape[0]*gs.shape[1]*gs.shape[2],64,64,)) gs = gs.reshape(gs.shape[0]*gs.shape[1]*gs.shape[2],64,64,)
What is the correct way to change the view without actually causing memory overload?
Advertisement
Answer
Why do you want to do this reshape?
Your starting array (with a small dtype):
In [100]: arr = np.ones((512,512,100), 'int8') In [101]: arr.nbytes, arr.strides Out[101]: (26214400, (51200, 100, 1))
Using the view_as_windows
as ported to the numpy core:
In [102]: new = np.lib.stride_tricks.sliding_window_view(arr,(64,64),(0,1)) In [103]: new.shape, new.strides Out[103]: ((449, 449, 100, 64, 64), (51200, 100, 1, 51200, 100)) In [105]: new.nbytes/1e9 # the number of elements (memory if copy) Out[105]: 82.5757696
Note that the arr
final dimension, 100, is now 3rd; in the view
is still has a single element stride.
Effectively the first dim, 512, has been ‘split’ between axis 0 and 3; and the 2nd 512, axis (1,4).
To get the shape, (499499100,64,64), strides would have to be (? , 51200, 100). I’m not an expert on as_strided
, but have enough experience to say there’s no way of reworking the first 3 strides (51200, 100, 1) into one.
Let’s remove the size 100 dimension to simplify things a bit.
In [106]: arr = np.ones((512,512), 'int8') In [107]: arr.strides Out[107]: (512, 1)
The windowed:
In [108]: new = np.lib.stride_tricks.sliding_window_view(arr,(64,64),(0,1)) In [109]: new.shape, new.strides Out[109]: ((449, 449, 64, 64), (512, 1, 512, 1))
and its 3d reshape (a copy):
In [110]: new1 = new.reshape(-1,64,64) In [112]: new1.shape, new1.strides Out[112]: ((201601, 64, 64), (4096, 64, 1))