I am trying to run a spider on a vps (using scrapyjs which uses python-gtk2). On running the spider I am getting the error
/root/myporj/venv/local/lib/python2.7/dist-packages/gtk-2.0/gtk/__init__.py:57: GtkWarning: could not open display
How do I run this in a headless setup?
Advertisement
Answer
First of all, you didn’t specify if you have a desktop environment (or X) installed on your server?
Regardless of that, you can achieve headless setup for your spider by using xvfb:
Xvfb or X virtual framebuffer is a display server implementing the X11 display server protocol. In contrast to other display servers Xvfb performs all graphical operations in memory without showing any screen output. From the point of view of the client, it acts exactly like any other X display server, serving requests and sending events and errors as appropriate. However, no output is shown. This virtual server does not require the computer it is running on to even have a screen or any input device.
First, install xvfb on your server. Assuming you are using apt-get:
sudo apt-get install xvfb
Now, run the xvfb service at a custom display number, like:
sudo Xvfb :5
After that, you can run any application that requires X inside your virtual frame buffer by exporting the display number and running your application:
export DISPLAY=:5 run_my_application
The DISPLAY is set only for the current terminal session, so keep that in mind when implementing automation for this step.