I am using scrapy to scrap a site
I had written a spider and fetched all the items from the page and saved to a csv file,
and now i want to save the total execution time
taken by scrapy to run the spider file,
actually after spider execution is completed and when we have at at terminal it will display some results
like starttime, endtime
and so on …. so now in my program i need to calculate the total time taken by scrapy to run the spider and storing the total time some where….
Can anyone let me now how to do this through an example……..
Thanks in advance………..
Advertisement
Answer
This could be useful:
from scrapy.xlib.pydispatch import dispatcher from scrapy import signals from scrapy.stats import stats from datetime import datetime def handle_spider_closed(spider, reason): print 'Spider closed:', spider.name, stats.get_stats(spider) print 'Work time:', datetime.now() - stats.get_stats(spider)['start_time'] dispatcher.connect(handle_spider_closed, signals.spider_closed)