Currently trying to write code to check for data quality of a 7 gb data file. I tried googling exactly but to no avail. Initially, the purpose of the code is to check how many are nulls/NaNs and later on to join it with another datafile and compare the quality between each. We are expecting the second is the …
Tag: dask
Error comparing dask date month with an integer
The dask map_partitions function in the code below has a dask date field where its month is compared to an integer. This comparison fails with the following error: ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all(). What is this error and how to fix it? A…
Extracting latest values in a Dask dataframe with non-unique index column dates
I’m quite familiar with pandas dataframes but I’m very new to Dask so I’m still trying to wrap my head around parallelizing my code. I’ve obtained my desired results using pandas and pandarallel already so what I’m trying to figure out is if I can scale up the task or speed it up…
Dask dataframe: Can `set_index` put a single index into multiple partitions?
Empirically it seems that whenever you set_index on a Dask dataframe, Dask will always put rows with equal indexes into a single partition, even if it results in wildly imbalanced partitions. Here is a demonstration: However, I found no guarantee of this behaviour anywhere. I have tried to sift through the co…
Dask: Continue with others task if one fails
I have a simple (but large) task Graph in Dask. This is a code example Here SomeIterable is a list of dict, where each are arguments to my_function. In each iteration b depends on a, so if the task that produces a fails, b can’t be computed. But, each element of results are independent, so I expect if o…
Unable to expand cluster by dask
I am very new to kubernetes & dask and trying to implement some kube cluster and have created minikube cluster with some services, further want to expand it with flexible dask functionality. I am planning to deploy it to gcloud somehow later, so i am trying to initialize dask cluster (scheduler and worker…
Build a dask dataframe from a list of dask delayed objects
I have a list of dask delayed objects Portfolio_perfs: Each delayed object is a numpy array of length 2 I want to build the following dataframe without using dask.compute: How can I build this dask dataframe without going through dask.compute? Thank you Answer Since each delayed object is a numpy array, you a…
Where does dask store files while running on juputerlab
I’m running dask on jupyterlab. I’m trying to save some file in home directory where my python file is stored and it’s running properly but I’m not able to find out where my files are getting saved. So I made a folder named output in home directory to save file inside, but when I save …
Getting very slow iterations in a loop run over a Datarray using Xarray and Dask
I am trying to calculate windspeed from u and v components for 1 year data at hourly timestep and 0.1 x 0.1 Degree resolution for a total of 40 years. The individual u and v netcdf files for 1 year is about 5GB each. I have implemented a basic for loop where the u and v netcdf files for each
Dask dataframe crashes
I’m loading a large parquet dataframe using Dask but can’t seem to be able to do anything with it without the system crashing on me or getting a million errors and no output. The data weighs about 165M compressed, or 13G once loaded in pandas (it fits well in the 45G RAM available). Instead, if us…