Skip to content
Advertisement

Tag: memory-management

High memory allocation when using dask.bag.map

I am using dask for extending dask bag items by information from an external, previously computed object arg. Dask seems to allocate memory for arg for each partition at once in the beginning of the computation process. Is there a workaround to prevent Dask from duplicating the arg multiple times (and allocating a lot of memory)? Here is a simplified

Python memory release when using a for loop inside a class

I have some troubles after I created a class to process raster images. The class includes different methods for checking DBs and processing the images. The usage script is super simple: The method run_extraction() is the following: The method does several steps for getting an observation for a given variable. The code works, but it doesn’t release memory. Since it’s

What for we call for the typing import List from the Python Standard Library?

I am solving some questions from the Leetcode: https://leetcode.com/problems/remove-duplicates-from-sorted-array/ I find the answers: Given a sorted array, remove the duplicates in-place such that each element appear only once and return the new length The answer for this question can be, too: I can’t find the answer to the question of why in the code should be applied typing. On the

Are there disadvantages of using __slots__?

I’m using Python 3.7 and Django. I was reading about __slots__ . Evidently, __slots__ can be used to optimize memory allocation for a large number of those objects by listing all the object properties ahead of time. My perhaps obvious question is why wouldn’t we want to do this for all objects? Are there disadvantages for using __slots__? Answer Fluent

How to cut my addiction to Python dictionaries

So, I have a large 30k line program I’ve been writing for a year. It basically gathers non-normalized and non-standardized data from multiple sources and matches everything up after standardizing the sources. I’ve written most everything with ordered dictionaries. This allowed me to keep the columns ordered, named and mutable, which made processing easier as values can be assigned/fixed throughout

Advertisement