Scrapy not sending post request

I was trying to make a post request to a URL but scrapy isn’t sending the post request. I am not getting the correct response.

Below is my code.

import scrapy

class MainSpider(scrapy.Spider):
    name = 'main'
    directory = 'https://www.constructionenquirer.com/directory'
    start_urls = [directory]

    def parse(self, response):
        sectors = response.xpath('(//select[@name="sector"]/option)[position()>1]')
        for sector in sectors:
            vals = sector.xpath('.//@value').get()

            data = {
                'ce-directory-action': 'ce-directory-action',
                'sector': vals,
                'action': 'find-firms-by-sector'
            }

            yield scrapy.FormRequest(url=self.directory, formdata=data, callback=self.parse_sectors)
    
    def parse_sectors(self, response):
        yield {
            "Name": response.xpath('//h3/a/text()').get()
        }

Answer

You have a typo in your code here:

'ce-directory-action': 'sector-search',

Advertisement

Answer