Skip to content
Advertisement

passing table name to pipeline scrapy python

I have different spiders that scrape similar values and I want to store the scraped values in different slite3 tables. I can do this by using a different pipeline for each spider but, since the only thing that changes is the table name, would it be possible to pass somehow the table name from the spider to the pipeline? This is the code I use to create the table:

def open_spider(self, spider):
        self.connection = sqlite3.connect("database.db")
        self.c = self.connection.cursor()
        try:
            self.c.execute('''
                CREATE TABLE IF NOT EXISTS table_name(
                    value1 TEXT,
                    value2 TEXT,
                    value3 TEXT
                )
            ''')
            self.connection.commit()

Advertisement

Answer

yes , something like this?

def open_spider(self, spider):
        self.connection = sqlite3.connect("database.db")
        self.c = self.connection.cursor()
        querystring = f"
                CREATE TABLE IF NOT EXISTS {spider.name} (
                    value1 TEXT,
                    value2 TEXT,
                    value3 TEXT
                )
            "
        try:
            self.c.execute(querystring)
            self.connection.commit()
7 People found this is helpful
Advertisement