After opening the enter link description here I can see an API gets called which returns some data in JSON format. Attached you may see as well. I am trying to call this API using my python script as follows. but unfortunately, it returns me this message instead of the real data. on the other hand, when I sen…
Tag: web-scraping
Need help scraping WSJ Markets Data
I am relatively new and trying to use Python to scrape data. Here is my code: I have already added headers, but the output shows no value. Any help would be greatly appreciated, thanks! Answer The data you’re looking for is loaded from external source via Ajax. You can use next example how to load it wi…
Converting elements list into single dictionary while scraping
I am scraping Glassdoor companies. There is a benefits section which will be different for every company, so I am grabbing all the elements in the div. That returns a list. Is there anyway to convert these into key:value pairs while scraping. They need to be written to a csv inside a loop. As of now, I am wri…
How to access a specific p tag while using BeautifulSoup
Hello everyone I’m having trouble while using BeautifulSoup , indeed i don’t succeed to access the information that I want, here is my code : The output of this code is that : and what I want is the second ‘p’ with the information : ’10. March 2021′ however I don’t kn…
How to parse HTML table that is inside div and not table in Python
I am trying to parse the table from this website. I started with just the Username column and with the help I got on stackoverflow, I was able to get the content of Username with the following code: which gives me My ultimate goal is to populate the entire table with [Rank, Grade, Username, Uploads, Followers…
Scraping search results off of Sportchek with Beautiful Soup 4 to find prices
So I’m trying to web scrape search results from Sportchek with BS4, specifically this link “https://www.sportchek.ca/categories/men/footwear/basketball-shoes.html?page=1”. I want to get the prices off of the shoes here and put them all into a system to sort it, however, to do this I need to …
Why do I run into trouble webscraping this website in Python?
I am new to Python and I am trying to webscrape this website. What I am trying to do is to get just dates and articles’ titles from this website. I follow a procedure I found on SO which is as follows: I got .title a , .date using SelectorGadget in the url I shared. However, print(movies) is empty. What
Can’t stratify output based on different headings and their corresponding paragraphs
I’m trying to fetch each heading and their corresponding paragraphs from the html elements below. The results should be stored within a dictionary. Whatever I’ve tried so far produces ludicrously haphazard output. I intentionally did not paste the current output only because of brevity of space. I…
Google Earth scraping using python selenium
I want to create a web scraper for earth.google.com/web. Whenever the user clicks while holding shift button, the script will print the coordinates which are displayed at the bottom right corner of the google earth web page. I am using selenium with chromedriver but it cannot find the coordinates web element.…
Scraping tables from a JavaScript webpage using Selenium, BeautifulSoup, and Panda
to begin with I am a beginner and trying to achieve something which is currently out of my league. However, I hope you guys can help me out. Much appreciated. I am trying to scrape the table from spaclens.com. I already tried using the out-of-the-box solution from Google sheets however the site is Java Script…