I’m trying to get every products individual URL link from this link https://www.goodricketea.com/product/darjeeling-tea .How should I do that with beautifulsoup? Is there anyone who can help me?
Advertisement
Answer
To get product links from this site, you can for example do:
JavaScript
x
10
10
1
import requests2
from bs4 import BeautifulSoup3
4
5
url = "https://www.goodricketea.com/product/darjeeling-tea"6
soup = BeautifulSoup(requests.get(url).content, "html.parser")7
8
for a in soup.select("a:has(>h2)"):9
print("https://www.goodricketea.com" + a["href"])10
Prints:
JavaScript
1
15
15
1
https://www.goodricketea.com/product/darjeeling-tea/roasted-darjeeling-tea-250gm2
https://www.goodricketea.com/product/darjeeling-tea/thurbo-darjeeling-tea-whole-leaf-250gm3
https://www.goodricketea.com/product/darjeeling-tea/roasted-darjeeling-tea-organic-250gm4
https://www.goodricketea.com/product/darjeeling-tea/roasted-darjeeling-tea-100gm5
https://www.goodricketea.com/product/darjeeling-tea/thurbo-darjeeling-tea-whole-leaf-100gm6
https://www.goodricketea.com/product/darjeeling-tea/thurbo-darjeeling-tea-fannings-250gm7
https://www.goodricketea.com/product/darjeeling-tea/castleton-premium-muscatel-darjeeling-tea-100gm8
https://www.goodricketea.com/product/darjeeling-tea/castleton-vintage-darjeeling-tea-250gm9
https://www.goodricketea.com/product/darjeeling-tea/castleton-vintage-darjeeling-tea-100gm10
https://www.goodricketea.com/product/darjeeling-tea/castleton-vintage-darjeeling-tea-bags-50-tea-bags11
https://www.goodricketea.com/product/darjeeling-tea/castleton-vintage-darjeeling-tea-bags-100-tea-bags12
https://www.goodricketea.com/product/darjeeling-tea/badamtam-exclusive-organic-darjeeling-tea-250gm13
https://www.goodricketea.com/product/darjeeling-tea/badamtam-exclusive-organic-darjeeling-tea-100gm14
https://www.goodricketea.com/product/darjeeling-tea/seasons-3-in-1-darjeeling-leaf-tea-150gm-first-flush-second-flush-pre-winter-flush15