I was trying to make a post request to a URL but scrapy isn’t sending the post request. I am not getting the correct response.
Below is my code.
JavaScript
x
25
25
1
import scrapy
2
3
class MainSpider(scrapy.Spider):
4
name = 'main'
5
directory = 'https://www.constructionenquirer.com/directory'
6
start_urls = [directory]
7
8
def parse(self, response):
9
sectors = response.xpath('(//select[@name="sector"]/option)[position()>1]')
10
for sector in sectors:
11
vals = sector.xpath('.//@value').get()
12
13
data = {
14
'ce-directory-action': 'ce-directory-action',
15
'sector': vals,
16
'action': 'find-firms-by-sector'
17
}
18
19
yield scrapy.FormRequest(url=self.directory, formdata=data, callback=self.parse_sectors)
20
21
def parse_sectors(self, response):
22
yield {
23
"Name": response.xpath('//h3/a/text()').get()
24
}
25
Advertisement
Answer
You have a typo in your code here:
JavaScript
1
2
1
'ce-directory-action': 'sector-search',
2