How to get the a href link from under the div class? using beautiful soup

I am trying to scrape the href attribute from links from a page, but I end up with [] as the output

The HTML code is

My desired output is: https://www.pigiame.co.ke/listings/nissan-latio-2016-36000-kms-5300124

Answer

You can try:

import re
import requests
import urllib.parse
from bs4 import BeautifulSoup

url = "https://www.pigiame.co.ke/cars"

headers = {
    "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:107.0) Gecko/20100101 Firefox/107.0",
}

soup = BeautifulSoup(requests.get(url, headers=headers).content, "html.parser")

links = [
    item.a["href"] for item in soup.select(".listings-cards__list-item:has(a)")
]

for link in links:
    soup = BeautifulSoup(requests.get(link).content, "html.parser")
    data = soup.select_one(".btn-whatsapp")["href"]
    data = urllib.parse.unquote(data)

    phone = re.search(r"phone=(.*?)&", data).group(1)
    print("{:<20} {}".format(phone, link))

Prints:

[phone]        https://www.pigiame.co.ke/listings/nissan-note-5253578
[phone]        https://www.pigiame.co.ke/listings/honda-freed-7-seater-2015-5291221
[phone]        https://www.pigiame.co.ke/listings/2006-bmw-x3-2500cc-petrol-155000kms-5241375
[phone]        https://www.pigiame.co.ke/listings/mazda-mpv-2006-5273382
[phone]        https://www.pigiame.co.ke/listings/landrover-109-very-clean-accident-free-5282118
[phone]        https://www.pigiame.co.ke/listings/landcrusser-prado-tx-fully-loaded-with-sunroof-2016-model-5304294
[phone]        https://www.pigiame.co.ke/listings/landcrusser-prado-tx-fully-loaded-with-sunroof-5304293
[phone]        https://www.pigiame.co.ke/listings/mistubishi-canter-2016-model-3tones-5304291
[phone]        https://www.pigiame.co.ke/listings/hillux-revolution-2016-model-fully-loaded-5304288
[phone]        https://www.pigiame.co.ke/listings/toyota-prado-2016-model-fully-loaded-with-sunroof-5304285
[phone]        https://www.pigiame.co.ke/listings/toyota-prado-txl-diesel-fuel-2017-model-5304283
[phone]        https://www.pigiame.co.ke/listings/toyota-fielder2015kdkbelow-50000km-5304279
[phone]        https://www.pigiame.co.ke/listings/subaru-forester-sti-turbo-2016-model-5304280
[phone]        https://www.pigiame.co.ke/listings/honda-fit2015kdkbelow-60000km-5304276
[phone]        https://www.pigiame.co.ke/listings/honda-vezel2015kdklow-milleage-5304273