I have been trying to scrape this site and sending in the search phrase cause error. I have searched StackOverflow and the error usually is resolved when you redefine the web element again which I did but still getting the same error. I do not want to directly go to the search link like https://www.couriermai…
Tag: web-scraping
Selenium: Iterate through child elements with different class names
I want to iterate through child elements of parent . Access the child element, extract price information and move on to next child element. Here is my code Output: Concern/What I want help with: Price output should change in each iteration as element changes. But I am only getting same price as first element …
Incomplete scrapped data from shopee.co.id using BeautifulSoup and Selenium
I am trying to scrape shopee.co.id using beautifulsoup and selenium. There are 60 product in a single search results page. At the end of the code, I checked the extracted data using len() and it shows that I only extracted 42 of them. How should I fix the code to obtain all the search results? Here is the cod…
xpath inside founded elements using selenium
I found 10 elements using: Then i’m trying get information from collected divs by xpath in cycle: But always i get first element Question: Is it possible to get information from each block in cycle using relative xpath with already founded element by selenium? One of divs HTML: Answer Yes, you can easil…
page scraping using beautiful soup, without links
I am using the following code to extract text from a web page: The problem is, when I open text, I get all the links from the bottoms that exist at the top of the page, which I don’t want. How can i modify the above code to do so? I also gets the footnotes, which i may want, but
How to scrape billboard using find_all?
I have a pretty specific question if anyone could help that would be awesome. I am trying to scrape the songs from https://www.billboard.com/charts/hot-100/ and I am stuck on this code. What should I put for tag = soup.find_all(‘???’) to get the title ‘About Damn Time. Answer Try to select t…
python beautifulsoup duplicating results
I’m trying to learn beatifulsoup (and python as a whole, pretty much still a beginner) and playing around with how to use it properly. I notice that when I scrape the website I’m testing for data from the search results, it lists it 3 times. Specifically, I’m trying to output the title, link…
Scrapy extracting entire HTML element instead of following link
I’m trying to access or follow every link that appears for commercial contractors from this website: https://lslbc.louisiana.gov/contractor-search/search-type-contractor/ then extract the emails from the sites that each link leads to but when I run this script, scrapy follows the base url with the entir…
BeautifulSoup – can’t find attribute
I’m trying to scrape this link. I want to get to this part here: I can see where this part of the website is when I inspect the page: But I can’t get to it from BeautifulSoup. Here is the code that I’m using and all the ways I’ve tried to access it: Also, when I just look at the
nonetype object has no attribute text error while scraping data
when i try to scrap data from this amazon link. I got AttributeError: ‘NoneType’ object has no attribute ‘text’ My Code: I think that the laptop_rating variable store the content in string format even if we not include .text. I’m thinking that might be the reason for getting None…