I have an issue with .select which always returns an empty list while practicing webscraping. I am working on the following page: https://presse.ania.net/news/?page=1 using BeautifulSoup. I am getting and parsing HTML as following: I try to retrieve the urls of each articles displayed on the page, under class…
Tag: web-scraping
How can I sift through various ‘a’ tags when scraping a website?
I’m trying to scrape athletic.net, a site that stores track and field times, to get a list for a given athlete of each season, each event that they ran, and every time they got for each event. So far I have printed the season title and the name of each event. I’m now trying to sift through a sea o…
web scraping returns an empty list
The problem here that printing job_titles returns an empty list instead of the job titles in the web site please help me fix this problem and any help would be appreciated Answer When I first went to the URL you’re requesting, I was shown a search page with no jobs listed. It was only after I submitted …
Python – How to print a whole url in a for loop not just characters
I have a variable called all_urls- it contains a list of URLs, for example: I want to print each URL using a for loop. but what I end up getting is: Does anyone know how to fix it so the program returns each URL? Thanks. Answer You can try with this: What I’m thinking is that you have a string
Why can’t I extract the other pages of the same website using beautifulsoup
I wrote this code to extract multiple pages of data from this site (base URL – “https://www.goodreads.com/shelf/show/fiction”). But it’s only showing the first 50 books’ data. How can I extract all fiction books’ names extracting all pages using beautifulsoup? Answer You ca…
How to scrape Trusted Shops?
I would appreciate your help on this scraping problem. I would like to scrape this site: https://www.trustedshops.de/bewertung/info_XDAD39B874C275A0751198C2510C3C670.html Although my code does not give me an error message, I do not see any output. I believe the problem is the bs.find_all statement; basically,…
Error handling in variables when using Requests
I have the below code that works fully up until I set x=37. At this point, I receive the error TypeError: ‘NoneType’ object is not subscriptable on the variable t[“vintage”][“wine”][“region”][“country”][“name”]. I have added another varia…
itterate through multiple URLs with BS4 – and store results into a csv-format
hi there i am currently working on a little tiny sraper – and i am putting some pieces together i have an URL which holds record of so called digital hubs: see https://s3platform-legacy.jrc.ec.europa.eu/digital-innovation-hubs-tool/-/dih/1096/view i want to export the 700 regords in (to) a csv-format: t…
Capture python output to variable
I’m trying to automate pybaseball from player lookup to data extraction and can’t seem to get past the first step of setting the player id as a variable. I’m utilizing the following code to pull player id: The output from the above lines are (exact copy/paste): Is there a method for setting …
Multiple errors when scraping premier league tables
I am learning web-scraping. I succeeded scraping top youtubers ranking with this as reference. I am using the same logic to scrape the PL ranking, but having two issues: it is only collecting up to 5th place. it is getting only the first place for the result and then, getting attribute error: Answer The issue…