Skip to content
Advertisement

Tag: scrapy

GtkWarning: could not open display

I am trying to run a spider on a vps (using scrapyjs which uses python-gtk2). On running the spider I am getting the error How do I run this in a headless setup? Answer First of all, you didn’t specify if you have a desktop environment (or X) installed on your server? Regardless of that, you can achieve headless setup

Scrapy: Passing item between methods

Suppose I have a Bookitem, I need to add information to it in both the parse phase and detail phase Using the code as is would led to undefined item in the detail phase. How can I pass the item to the detail? detail(self,response,item) doesn’t seem to work. Answer There is an argument named meta for Request: then in function

Recording the total time taken for running a spider in scrapy

I am using scrapy to scrap a site I had written a spider and fetched all the items from the page and saved to a csv file, and now i want to save the total execution time taken by scrapy to run the spider file, actually after spider execution is completed and when we have at at terminal it will

Advertisement