I’m populating a database using Django from an Excel sheet. In the example below, the purchases list simulates the format of that sheet. My problem is that I’m repeatedly using Customer.objects.get to retrieve the same Customer object when creating new Purchase entries. This results in repeated da…
Tag: performance
Most efficient way to split up a dataframe into smaller dataframes
I am writing a python program that will parse a large dataframe (tens of thousands of lines) into smaller dataframes based on a column value, and it needs to be fairly efficient, because the user can change the ways they break up the dataframe, and I would like the output to update dynamically. Example input:…
Fastest way to transform a continuous string of hex into base 10 in Python
I have a 100M 3600 character length strings of hexadecimal digits that I want to split into blocks of three and then convert into base 10. Strictly speaking, I want to transform these into signed 4 byte numbers. As I have 100M of these strings to process, my main aim is code efficiency/speed. For splitting th…
The most efficient way rather than using np.setdiff1d and np.in1d, to remove common values of 1D arrays with unique values
I need a much faster code to remove values of an 1D array (array length ~ 10-15) that are common with another 1D array (array length ~ 1e5-5e5 –> rarely up to 7e5), which are index arrays contain integers. There is no duplicate in the arrays, and they are not sorted and the order of the values must b…
Fastest way to get all first-matched rows given a sequence of column values in Pandas
Say I have a Pandas dataframe with 10 rows and 2 columns. Now that I am given a sequence of ‘col1’ values in a numpy array： I want to find the rows that have the first occurence of 3, 1 and 2 in ‘col1’, and then get the corresponding ‘col2’ values in order. Right now I am u…
How to improve performance of dataframe slices matching?
I need to improve the performance of the following dataframe slices matching. What I need to do is find the matching trips between 2 dataframes, according to the sequence column values with order conserved. My 2 dataframes: Expected output: This is the following code I’ m using: Despite working, this is…
Is it possible to make this function on numpy array more efficient?
Here a is a 1D array of integer indices. To give some context, a is the atom indices of the 1st molecules. The return is the atom indices for n identical molecules, each of which contains step atoms. This function basically applies the same atom selection to many molecules For example Answer It looks like bro…
Detect if all pixels in an image are zero/black (Maya Python)
Newbie. Using the following code to check if a grayscale image (a mask texture in Maya) has all black pixels in the RGB channels (meaning it is empty). This works, but is a bit slow on large images (2048×2048 is around 15 seconds). Looking for how I can speed this up / do this more efficiently. EDIT: Thi…
python-mysql-connector: I need to speed up the time it takes to update multiple items in mySQL table
I currently have a list of id’s approx. of size 10,000. I need to update all rows in the mySQL table which have an id in the inactive_ids list that you see below. I need to change their active status to ‘No’ which is a column in the mySQL table. I am using mysql.connector python library. Whe…
Improve the speed of code with if else statements
I am working on building a classs. But I realized that one part is extremly slow. I believe it is this part below, because when I call it, it takes couple of minutes which is too long. How can the speed of the function be improved? Not sure how the code can be improved to increase speed. There is a