Skip to content
Advertisement

Attempting to web scrape. Downloaded html code is slightly different from code on live site

I’m new to web scraping and I’m trying to build a very basic stock tracker for the site pokemoncenter.com. When visiting the product pages of items on the live site, the add to cart button displays as:

JavaScript

When the item is out of stock the button is:

JavaScript

But whenever I try to scrape the site, regardless of whether the item is in stock or not, the button is:

JavaScript

So essentially it always displays as out of stock when I download the html code with requests.get().

JavaScript

In stock example: https://www.pokemoncenter.com/product/701-00364/primal-groudon-poke-plush-17-3-4-in
Out of stock example: https://www.pokemoncenter.com/product/701-06558/gigantamax-pikachu-poke-plush-17-in

Advertisement

Answer

As goalie1998 mentioned, the site could be using javascript to only load necessary images first to reduce initial load time. You could probably still use Selenium to scrape that website since it can imitate browser behavior.

User contributions licensed under: CC BY-SA
8 People found this is helpful
Advertisement