I have an issue with .select which always returns an empty list while practicing webscraping. I am working on the following page: https://presse.ania.net/news/?page=1 using BeautifulSoup.
I am getting and parsing HTML as following:
url = f"https://presse.ania.net/news/?page=1"
headers = {'User-Agent':'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.64 Safari/537.36'
mr = requests.get(url, headers = headers)
soupmp = bs(mr.content, "lxml")
I try to retrieve the urls of each articles displayed on the page, under class “title row-space-1” (I use developer tools of chrome to find class, disabled JavaScript like suggested in other posts), and put them in a list called “news”
news = []
for link in soupmp.select("a.title.row-space-1[href]"):
news.append(link.get('href'))
However I keep having an empty list when I print ‘news’
[]
Searching on Stackoverflow I tried:
- Disabling JavaScript on website
- Adding a time sleep to let the page download
- Using .find_all, . find and .select, tried with CSS selectors first then kwargs (all return empty list or NoneType object).
None of these worked and I am stuck with my mistake. I think there is something specific in my way of understanding this HTML and selecting class with CSS but I can’t find what (partly because I successfully used this code for other websites earlier.).
Could you please educate me on what I am missing?
I appreciate your help!
Advertisement
Answer
Try this:
import requests
from bs4 import BeautifulSoup
css = ".card.item.thumbnail.card-topic .title a"
url = "https://presse.ania.net/news/?page=1"
headers = { 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.64 Safari/537.36'}
soup = [
f'https://presse.ania.net{a["href"]}' for a in
BeautifulSoup(requests.get(url, headers=headers).content, "lxml").select(css)
]
print("n".join(soup))
Output:
https://presse.ania.net/actualites/cp-ania-reaction-suite-a-lannonce-de-la-composition-du-nouveau-gouvernement-delisabeth-borne-lania-salue-la-double-reconnaissance-de-la-souverainete-industrielle-et-alimentaire-au-coeur-du-gouvernement-5c05-53c7f.html https://presse.ania.net/actualites/cp-ania-nomination-de-marie-buisson-au-poste-de-directrice-juridique-7a6b-53c7f.html https://presse.ania.net/actualites/cp-ania-lca-renegociations-commerciales-la-filiere-agroalimentaire-inquiete-bfe7-53c7f.html https://presse.ania.net/actualites/cp-ania-lca-guerre-en-ukraine-et-derogations-detiquetage-eviter-les-ruptures-tout-en-garantissant-la-securite-sanitaire-et-la-transparence-de-linformation-aux-consommateurs-9824-53c7f.html https://presse.ania.net/actualites/cp-ania-il-est-urgent-de-re-ouvrir-les-negociations-commerciales-c4d4-53c7f.html https://presse.ania.net/actualites/cp-ania-lca-reunion-interministerielle-sur-la-guerre-en-ukraine-appel-du-secteur-alimentaire-pour-des-mesures-durgence-abb3-53c7f.html https://presse.ania.net/actualites/cp-ania-lca-signature-de-lavenant-au-contrat-strategique-de-la-filiere-agroalimentaire-2022-2023-138f-53c7f.html https://presse.ania.net/actualites/ania-note-de-conjoncture-economique-une-rentree-2022-sous-tensions-216b-53c7f.html https://presse.ania.net/actualites/cp-ania-presidentielles-2022-le-grand-oral-de-lagroalimentaire-lalimentation-est-laffaire-de-tous-les-francais-lindustrie-alimentaire-a-la-rencontre-des-candidats-a-la-presidentielle-fe01-53c7f.html https://presse.ania.net/actualites/cp-ania-negociations-commerciales-avec-la-grande-distribution-j-5-le-compte-ny-est-toujours-pas-au-risque-de-lasphyxie-collective-de-toute-la-filiere-830b-53c7f.html https://presse.ania.net/actualites/cp-ania-lania-engagee-pour-leducation-et-la-promotion-des-comportements-favorables-a-la-sante-et-a-la-planete-fa16-53c7f.html https://presse.ania.net/actualites/cp-ania-varenne-de-leau-lania-salue-les-engagements-pris-par-letat-et-rappelle-la-mobilisation-des-entreprises-5ae0-53c7f.html