I am trying to scrape the href attribute from links from a page, but I end up with []
as the output
My desired output is: https://www.pigiame.co.ke/listings/nissan-latio-2016-36000-kms-5300124
Advertisement
Answer
You can try:
import re import requests import urllib.parse from bs4 import BeautifulSoup url = "https://www.pigiame.co.ke/cars" headers = { "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:107.0) Gecko/20100101 Firefox/107.0", } soup = BeautifulSoup(requests.get(url, headers=headers).content, "html.parser") links = [ item.a["href"] for item in soup.select(".listings-cards__list-item:has(a)") ] for link in links: soup = BeautifulSoup(requests.get(link).content, "html.parser") data = soup.select_one(".btn-whatsapp")["href"] data = urllib.parse.unquote(data) phone = re.search(r"phone=(.*?)&", data).group(1) print("{:<20} {}".format(phone, link))
Prints:
+254723099904 https://www.pigiame.co.ke/listings/nissan-note-5253578 +254722935411 https://www.pigiame.co.ke/listings/honda-freed-7-seater-2015-5291221 +254722763845 https://www.pigiame.co.ke/listings/2006-bmw-x3-2500cc-petrol-155000kms-5241375 +254722710833 https://www.pigiame.co.ke/listings/mazda-mpv-2006-5273382 +254713193417 https://www.pigiame.co.ke/listings/landrover-109-very-clean-accident-free-5282118 +254708467397 https://www.pigiame.co.ke/listings/landcrusser-prado-tx-fully-loaded-with-sunroof-2016-model-5304294 +254708467397 https://www.pigiame.co.ke/listings/landcrusser-prado-tx-fully-loaded-with-sunroof-5304293 +254708467397 https://www.pigiame.co.ke/listings/mistubishi-canter-2016-model-3tones-5304291 +254708467397 https://www.pigiame.co.ke/listings/hillux-revolution-2016-model-fully-loaded-5304288 +254708467397 https://www.pigiame.co.ke/listings/toyota-prado-2016-model-fully-loaded-with-sunroof-5304285 +254708467397 https://www.pigiame.co.ke/listings/toyota-prado-txl-diesel-fuel-2017-model-5304283 +254769333436 https://www.pigiame.co.ke/listings/toyota-fielder2015kdkbelow-50000km-5304279 +254708467397 https://www.pigiame.co.ke/listings/subaru-forester-sti-turbo-2016-model-5304280 +254769333436 https://www.pigiame.co.ke/listings/honda-fit2015kdkbelow-60000km-5304276 +254769333436 https://www.pigiame.co.ke/listings/honda-vezel2015kdklow-milleage-5304273