Skip to content
Advertisement

Build a dask dataframe from a list of dask delayed objects

I have a list of dask delayed objects Portfolio_perfs:

type(Portfolio_perfs)
<class 'list'>
#print until 3
Portfolio_perfs[:3]
[Delayed('getitem-b7fd8629e2a0ecfe4e61ae6f39926140'), Delayed('getitem-af3225459229d541b73dc79319edaec2'), Delayed('getitem-0555389e6dd01031de85e293b8c42b85')]

Each delayed object is a numpy array of length 2

Portfolio_perfs[0].compute()
array([0.75620425, 0.1835988 ])

I want to build the following dataframe without using dask.compute:

pd.DataFrame(dask.compute(*Portfolio_perfs))
            0         1
0    0.756204  0.183599
1    0.825101  0.195705
2    0.792804  0.189422
3    0.786267  0.178194
4    0.860377  0.220204
..        ...       ...
595  0.636857  0.139955
596  0.925144  0.218462
597  0.925077  0.213963
598  0.922016  0.206081
599  0.770950  0.170273

[600 rows x 2 columns]

How can I build this dask dataframe without going through dask.compute? Thank you

Advertisement

Answer

Since each delayed object is a numpy array, you are interested in da.from_delayed():

dask_array = da.from_delayed(Portfolio_perfs)

Alternatively, it’s possible to convert numpy arrays to pandas dataframes and then use:

dd.from_delayed()

Note that it’s not possible to do it with pd.DataFrame because pandas will not know what to do with the delayed objects, so you will need to use dask.dataframe for this task.

Advertisement