I have different spiders that scrape similar values and I want to store the scraped values in different slite3 tables. I can do this by using a different pipeline for each spider but, since the only thing that changes is the table name, would it be possible to pass somehow the table name from the spider to the pipeline? This is the code I use to create the table:
def open_spider(self, spider): self.connection = sqlite3.connect("database.db") self.c = self.connection.cursor() try: self.c.execute(''' CREATE TABLE IF NOT EXISTS table_name( value1 TEXT, value2 TEXT, value3 TEXT ) ''') self.connection.commit()
Advertisement
Answer
yes , something like this?
def open_spider(self, spider): self.connection = sqlite3.connect("database.db") self.c = self.connection.cursor() querystring = f" CREATE TABLE IF NOT EXISTS {spider.name} ( value1 TEXT, value2 TEXT, value3 TEXT ) " try: self.c.execute(querystring) self.connection.commit()