My css selectors response.css('div.jhfizC') and ('a[itemprop="url"]') show 97 items in the web page, but my code is only scraping 35 items. Where is the fault?
Here is my code:
JavaScript
x
17
17
1
import scrapy2
3
class EbayfootSpider(scrapy.Spider):4
name = 'exebay'5
6
start_urls = ['https://www.mercari.com/search/?brandIds=4578&categoryIds=724&facets=2&length=30']7
8
9
def parse(self, response):10
big_div = response.css('div.jhfizC')11
12
for all_div in big_div:13
14
item_url = all_div.css('a[itemprop="url"]').css('::attr(href)')[0].extract()15
if item_url is not None:16
print(item_url)17
Advertisement
Answer
In the end of the url just put length 90 instead of 30 , length indicate 30 item per page.
JavaScript
1
115
115
1
In [1]: url = 'https://www.mercari.com/search/?brandIds=4578&categoryIds=724&facets=2&length=90' 2
3
fetch(url) 4
5
In [4]: response 6
Out[4]: <200 https://www.mercari.com/search/?brandIds=4578&categoryIds=724&facets=2&length=90>7
8
In [6]: big_div = response.css('div.jhfizC') 9
10
In [7]: item_url = all_div.css('a[itemprop="url"]').css('::attr(href)')11
12
In [11]: for i, all_div in enumerate( big_div): 13
: item_url = all_div.css('a[itemprop="url"]').css('::attr(href)')[0].extract() 14
: print(i,'https://www.mercari.com/'+item_url) 15
: 16
17
0 https://www.mercari.com//us/item/m27379701884/?ref=search_results18
1 https://www.mercari.com//us/item/m87408459178/?ref=search_results19
2 https://www.mercari.com//us/item/m31639451091/?ref=search_results20
3 https://www.mercari.com//us/item/m55461396239/?ref=search_results21
4 https://www.mercari.com//us/item/m61065950216/?ref=search_results22
5 https://www.mercari.com//us/item/m59003878460/?ref=search_results23
6 https://www.mercari.com//us/item/m42114019532/?ref=search_results24
7 https://www.mercari.com//us/item/m60410492125/?ref=search_results25
8 https://www.mercari.com//us/item/m61707057858/?ref=search_results26
9 https://www.mercari.com//us/item/m77824553207/?ref=search_results27
10 https://www.mercari.com//us/item/m72401412407/?ref=search_results28
11 https://www.mercari.com//us/item/m83737664911/?ref=search_results29
12 https://www.mercari.com//us/item/m31513565581/?ref=search_results30
13 https://www.mercari.com//us/item/m53686770401/?ref=search_results31
14 https://www.mercari.com//us/item/m31687661458/?ref=search_results32
15 https://www.mercari.com//us/item/m73565122351/?ref=search_results33
16 https://www.mercari.com//us/item/m54774041977/?ref=search_results34
17 https://www.mercari.com//us/item/m63815616689/?ref=search_results35
18 https://www.mercari.com//us/item/m72900819243/?ref=search_results36
19 https://www.mercari.com//us/item/m38982779224/?ref=search_results37
20 https://www.mercari.com//us/item/m16950460012/?ref=search_results38
21 https://www.mercari.com//us/item/m96072811718/?ref=search_results39
22 https://www.mercari.com//us/item/m98926246504/?ref=search_results40
23 https://www.mercari.com//us/item/m22674930214/?ref=search_results41
24 https://www.mercari.com//us/item/m76505587116/?ref=search_results42
25 https://www.mercari.com//us/item/m81533546144/?ref=search_results43
26 https://www.mercari.com//us/item/m71881974396/?ref=search_results44
27 https://www.mercari.com//us/item/m57251522596/?ref=search_results45
28 https://www.mercari.com//us/item/m85616167131/?ref=search_results46
29 https://www.mercari.com//us/item/m66592587980/?ref=search_results47
30 https://www.mercari.com//us/item/m16222098730/?ref=search_results48
31 https://www.mercari.com//us/item/m30516632346/?ref=search_results49
32 https://www.mercari.com//us/item/m82678279409/?ref=search_results50
33 https://www.mercari.com//us/item/m22459472107/?ref=search_results51
34 https://www.mercari.com//us/item/m49112577494/?ref=search_results52
35 https://www.mercari.com//us/item/m89573947301/?ref=search_results53
36 https://www.mercari.com//us/item/m15348612036/?ref=search_results54
37 https://www.mercari.com//us/item/m11392827625/?ref=search_results55
38 https://www.mercari.com//us/item/m60718086586/?ref=search_results56
39 https://www.mercari.com//us/item/m57133492284/?ref=search_results57
40 https://www.mercari.com//us/item/m73858839710/?ref=search_results58
41 https://www.mercari.com//us/item/m77945239665/?ref=search_results59
42 https://www.mercari.com//us/item/m13356901817/?ref=search_results60
43 https://www.mercari.com//us/item/m63603936227/?ref=search_results61
44 https://www.mercari.com//us/item/m49178440576/?ref=search_results62
45 https://www.mercari.com//us/item/m14806568451/?ref=search_results63
46 https://www.mercari.com//us/item/m67151874325/?ref=search_results64
47 https://www.mercari.com//us/item/m90580799274/?ref=search_results65
48 https://www.mercari.com//us/item/m66942995364/?ref=search_results66
49 https://www.mercari.com//us/item/m90049596871/?ref=search_results67
50 https://www.mercari.com//us/item/m25892617355/?ref=search_results68
51 https://www.mercari.com//us/item/m11787855892/?ref=search_results69
52 https://www.mercari.com//us/item/m10379012995/?ref=search_results70
53 https://www.mercari.com//us/item/m28508818974/?ref=search_results71
54 https://www.mercari.com//us/item/m23474551312/?ref=search_results72
55 https://www.mercari.com//us/item/m99385515312/?ref=search_results73
56 https://www.mercari.com//us/item/m13303803841/?ref=search_results74
57 https://www.mercari.com//us/item/m36605382938/?ref=search_results75
58 https://www.mercari.com//us/item/m27122502813/?ref=search_results76
59 https://www.mercari.com//us/item/m58324967907/?ref=search_results77
60 https://www.mercari.com//us/item/m88943761510/?ref=search_results78
61 https://www.mercari.com//us/item/m73769298655/?ref=search_results79
62 https://www.mercari.com//us/item/m60815399306/?ref=search_results80
63 https://www.mercari.com//us/item/m48872595825/?ref=search_results81
64 https://www.mercari.com//us/item/m53519878176/?ref=search_results82
65 https://www.mercari.com//us/item/m59524853672/?ref=search_results83
66 https://www.mercari.com//us/item/m72346338496/?ref=search_results84
67 https://www.mercari.com//us/item/m86496319979/?ref=search_results85
68 https://www.mercari.com//us/item/m54138309188/?ref=search_results86
69 https://www.mercari.com//us/item/m61559200881/?ref=search_results87
70 https://www.mercari.com//us/item/m42596652475/?ref=search_results88
71 https://www.mercari.com//us/item/m54065649678/?ref=search_results89
72 https://www.mercari.com//us/item/m87125447888/?ref=search_results90
73 https://www.mercari.com//us/item/m26421150316/?ref=search_results91
74 https://www.mercari.com//us/item/m72992367701/?ref=search_results92
75 https://www.mercari.com//us/item/m46688464169/?ref=search_results93
76 https://www.mercari.com//us/item/m34842804531/?ref=search_results94
77 https://www.mercari.com//us/item/m80806378583/?ref=search_results95
78 https://www.mercari.com//us/item/m39280119583/?ref=search_results96
79 https://www.mercari.com//us/item/m48130903632/?ref=search_results97
80 https://www.mercari.com//us/item/m57955016021/?ref=search_results98
81 https://www.mercari.com//us/item/m54360880136/?ref=search_results99
82 https://www.mercari.com//us/item/m29432458124/?ref=search_results100
83 https://www.mercari.com//us/item/m81313993828/?ref=search_results101
84 https://www.mercari.com//us/item/m34469644752/?ref=search_results102
85 https://www.mercari.com//us/item/m26953235217/?ref=search_results103
86 https://www.mercari.com//us/item/m29197336718/?ref=search_results104
87 https://www.mercari.com//us/item/m24460351728/?ref=search_results105
88 https://www.mercari.com//us/item/m95427865100/?ref=search_results106
89 https://www.mercari.com//us/item/m29184483225/?ref=search_results107
90 https://www.mercari.com//us/item/m47761827365/?ref=search_results108
91 https://www.mercari.com//us/item/m333400415/?ref=search_results109
92 https://www.mercari.com//us/item/m222574675/?ref=search_results110
93 https://www.mercari.com//us/item/m916567471/?ref=search_results111
94 https://www.mercari.com//us/item/m516644163/?ref=search_results112
95 https://www.mercari.com//us/item/m30663599997/?ref=search_results113
96 https://www.mercari.com//us/item/m935881167/?ref=search_results114
97 https://www.mercari.com//us/item/m645603485/?ref=search_results115