Unable to expand cluster by dask

I am very new to kubernetes & dask and trying to implement some kube cluster and have created minikube cluster with some services, further want to expand it with flexible dask functionality.

I am planning to deploy it to gcloud somehow later, so i am trying to initialize dask cluster (scheduler and workers to my minikube cluster) from a pod with RBAC with python script, but my attempts wasn’t successful.

I’ve tried both local and remote deploy_mode in KubeCluster just for find some working solution, but i guess it’s must be remote.

After running this python script

pod_spec = make_pod_spec(image='daskdev/dask:latest',
                         memory_limit='2G', memory_request='2G',
                         cpu_limit=1, cpu_request=1, threads_per_worker=1,
                         env={'EXTRA_PIP_PACKAGES': 'git+https://github.com/dask/distributed'})

scheduler = make_pod_spec(image='daskdev/dask:latest',
                          memory_limit='1G', memory_request='1G',
                          cpu_limit=1, cpu_request=1, threads_per_worker=1,
                          env={'EXTRA_PIP_PACKAGES': 'git+https://github.com/dask/distributed'},
                          )

cluster = KubeCluster(scheduler_pod_template=scheduler, pod_template=pod_spec, n_workers=2,
                      auth=auth.InCluster())

cluster.adapt(minimum=1, maximum=4)

I get ValueError: missing port number in address '$(DASK_SCHEDULER_ADDRESS)' in scheduler pod logs

Full logs of a scheduler pod

Am i missed setting some env vars in pod specifications? Isn’t it must be set automatically?

If it will be helpful, when i set deploy mode as local, Client constructor raise error, which says scheduler port is not set or something like this. I tried to set it manually, but it keeps raising this error.

I will be very grateful for any advice to implement this

Answer

I figured out that it is impossible to use make_pod_spec as scheduler_pod_template. For those who want that function, I have opened pull request.

Advertisement

Answer