How to create sparkmagic session automatically (without having to manually interact with widget user-interface)?

I am using sparkmagic to connect Jupyter notebooks to a remote spark cluster via Livy.

The way it is now, I need to execute a notebook cell to bring up the %manage_spark user-interface widget, and manually select the language and click “create-session” in order to establish the spark context for the notebook.

Is there a way to automatically generate the session when executing the cell, instead of having to manually select the language and hit “create-session”?

Can one provide arguments to the sparkmagic somehow for instance?

I’m imagining being able to do Kernel->Restart & Run All, and have the notebook execute completely.

Answer

Kind of late, but here is how I pass the parameters for the spark session in the cell

%spark add -s session -l python -t None -u http://localhost:8998