Hello Stack Overflow contributors! I want to scrape multiple pages of a news website; it shows an error message during this step The error message is The lines of code are More specifically, this page and pages next to it are what I want to scrape: https://nypost.com/search/China+COVID-19/page/1/?orderby=rele…
Tag: web-scraping
XPath Selector to get IMDB release Date
I am practicing using Xpath selectors, and it seems to be very difficult to extract the release date from this website. I can get to the div class=’txt-block’, but not past that. I am trying to the get the date underneath it. E.g. “18 July 2008 (USA)” https://www.imdb.com/title/tt04685…
Get Bing search results in Python
I am trying to make a chatbot that can get Bing search results using Python. I’ve tried many websites, but they all use old Python 2 code or Google. I am currently in China and cannot access YouTube, Google, or anything else related to Google (Can’t use Azure and Microsoft Docs either). I want the…
Web page is loaded in selenium and reaches to end but does not contain all the elements inside the div
This is the site. https://www.talabat.com/uae/top-selling. There are somewhat 100 products and only 30 gets loaded. I was trying to fetch all the links and page reaches to end but only display 30 products and when clicked somewhere in the webdriver then loads the rest of the products. How can I print the link…
get a list from html source using python3
I am trying to get Cases list of COVID-19 positive cases from https://www.worldometers.info/, e.g. this The sample looks like(~line no: 700) : I am using bs4 as: which does scrape the file, but after that I am clueless how to get the data. The list I am looking for is commented with ## I NEED THIS LIST. Kindl…
How do I run through a list of links one by one and then scrape data using selenium(driver.get)?
I’m trying to loop through 2 sets of links. Starting with https://cuetracker.net/seasons > click through each season link (Last 5 seasons) and then click through each tournament link within each season link and scrape the match data from each tournament. Using the below code I have managed to get a l…
Extracting reviews of Android App from Google Play store using Web Scraping method (Python BS4) – index out of range
The issue with the below code is “list index out of range error”. Answer Try: It worked for me!
How can I scrape all 1000 novels? ,my code scrapes only 691
url to scrape Genre List for Genre in the Website List for scraped Novels For Loop to loop through navlist to append genre into the genre_list For Loop to loop through the genre_list for loop to loop each novel to get its title, cost, ratings Loop to find novels of a particular rating Writing these novels int…
How to scrape dataframe after select options from dropdown list?
I want to scrape dataframe from dropdow value with BeautifulSoup. I select the value in both dropdown I submit my selection I get a data table I would like to catch this dataframe with BS. any idea of the process to achieve this? example site: https://coinarbitragebot.com/arbitrage.php thanks Answer You can i…
Taking multiple prices on single page BS4
I’m creating an to help me learn but is also useful to me. I want to be able to parse multiple prices from (https://www.watchfinder.co.uk/search?q=114060&orderby=AgeNewToOld) one page, convert them to numbers and average them. The page will change so it could have 3 prices one day and 20 the next. T…