I’m trying to get every products individual URL link from this link https://www.goodricketea.com/product/darjeeling-tea .How should I do that with beautifulsoup? Is there anyone who can help me?
Advertisement
Answer
To get product links from this site, you can for example do:
import requests
from bs4 import BeautifulSoup
url = "https://www.goodricketea.com/product/darjeeling-tea"
soup = BeautifulSoup(requests.get(url).content, "html.parser")
for a in soup.select("a:has(>h2)"):
print("https://www.goodricketea.com" + a["href"])
Prints:
https://www.goodricketea.com/product/darjeeling-tea/roasted-darjeeling-tea-250gm https://www.goodricketea.com/product/darjeeling-tea/thurbo-darjeeling-tea-whole-leaf-250gm https://www.goodricketea.com/product/darjeeling-tea/roasted-darjeeling-tea-organic-250gm https://www.goodricketea.com/product/darjeeling-tea/roasted-darjeeling-tea-100gm https://www.goodricketea.com/product/darjeeling-tea/thurbo-darjeeling-tea-whole-leaf-100gm https://www.goodricketea.com/product/darjeeling-tea/thurbo-darjeeling-tea-fannings-250gm https://www.goodricketea.com/product/darjeeling-tea/castleton-premium-muscatel-darjeeling-tea-100gm https://www.goodricketea.com/product/darjeeling-tea/castleton-vintage-darjeeling-tea-250gm https://www.goodricketea.com/product/darjeeling-tea/castleton-vintage-darjeeling-tea-100gm https://www.goodricketea.com/product/darjeeling-tea/castleton-vintage-darjeeling-tea-bags-50-tea-bags https://www.goodricketea.com/product/darjeeling-tea/castleton-vintage-darjeeling-tea-bags-100-tea-bags https://www.goodricketea.com/product/darjeeling-tea/badamtam-exclusive-organic-darjeeling-tea-250gm https://www.goodricketea.com/product/darjeeling-tea/badamtam-exclusive-organic-darjeeling-tea-100gm https://www.goodricketea.com/product/darjeeling-tea/seasons-3-in-1-darjeeling-leaf-tea-150gm-first-flush-second-flush-pre-winter-flush