When I recreate and reassign a JAX np array to the same variable name, for some reason the GPU memory nearly doubles the first recreation and then stays stable for subsequent recreations/reassignments. Why does this happen and is this generally expected behavior for JAX arrays? Fully runnable minimal example:…
Tag: memory
How are small sets stored in memory?
If we look at the resize behavior for sets under 50k elements: This pattern is consistent with quadrupling of the backing storage size once the set is 3/5ths full, plus some presumably constant overhead for the PySetObject: A similar pattern continues even for larger sets, but the resize factor switches to do…
Optimal way of storing stream content on disk using Python
I’d like to stream data directly to disk. One way of doing that is simply to read data and write to file, but I also want to minimalize RAM usage. I’ve figured out some manual way of doing that: Do I really have to manually load stream part by part? If so, how can I determine optimal value of BUFF…
Why list([]) weighs less than []
I have such code: With the next result: The question is: Why does a list created with list([]) weigh less than a list created with just [] or for _ in condition? Answer (The details in this answer depend on the implementation, they are written to match CPython 3.10.0. Other versions or other implementations o…
Python memory release when using a for loop inside a class
I have some troubles after I created a class to process raster images. The class includes different methods for checking DBs and processing the images. The usage script is super simple: The method run_extraction() is the following: The method does several steps for getting an observation for a given variable.…
Time & memory complexity management with multi-dimensional matrices using parallelisation and numpy
I have a time series of very large matrices. I am trying to speed up the process and was wondering the most optimal way to do this. The two things that came to mind are to parallelize the process using numba or to apply a function to the matrices such as with np.apply_along_axis. Speed and memory complexity a…
Closing figures from previous sessions
I am running a python code that produces some figures with Matplotlib and Pandas. After a few runs of the code, I am getting to following error: RuntimeWarning: More than 20 figures have been opened. Figures created through the pyplot interface (matplotlib.pyplot.figure) are retained until explicitly closed a…
How to make this code not to consume so much RAM memory?
I have these two function and when I run them my kernel dies so freaking quickly. What can I do to prevent it? It happens after appending about 10 files to the dataframe. Unfortunately json files are such big (approx. 150 MB per one, having dozens of them) and I have no idea how to join it together. EDIT: Due
Why tensor size was not changed?
I made the toy CNN model. Then, I had checked model.summary via this code And I was able to get the following results: I want to reduce model size cuz i wanna increase the batch size. So, I had changed torch.float32 -> torch.float16 via NVIDIA/apex As a result, torch.dtype was changed torch.float16 from to…
What for we call for the typing import List from the Python Standard Library?
I am solving some questions from the Leetcode: https://leetcode.com/problems/remove-duplicates-from-sorted-array/ I find the answers: Given a sorted array, remove the duplicates in-place such that each element appear only once and return the new length The answer for this question can be, too: I can’t f…