Web Scraping – URL extraction from Lazada ecommerce platform

I am currently trying to scrape the products URLs from Lazada ecommerce platform, however i am getting random links from the website rather than the products links.

https://www.lazada.com.my/oldtown-white-coffee/?langFlag=en&q=All-Products&from=wangpu&pageTypeId=2

My code below:

from bs4 import BeautifulSoup, SoupStrainer
import requests

url = "https://www.lazada.com.my/oldtown-white-coffee/?langFlag=en&q=All- 
       Products&from=wangpu&pageTypeId=2"

page = requests.get(url)    
data = page.text
soup = BeautifulSoup(data)


links = soup.find_all('div', {'class': 'c16H9d'})

for link in soup.find_all("a"):
    print(link.get("href"))

The result I am getting out of this code(which is not what i want) :

This is the section of the links that I need, i wanted to list down all the products URLs from the products page.

I hope you guys can help me on this, I know it is simple it just doesn’t seems to work, have been looking at this since yesterday.

Answer

The page is dynamic. Within the html source code is the script that generates a json format of the products. You can pull that, then parse the json object to print off the urls:

from bs4 import BeautifulSoup, SoupStrainer
import requests
import json

url = "https://www.lazada.com.my/oldtown-white-coffee/?langFlag=en&q=All-Products&from=wangpu&pageTypeId=2"

page = requests.get(url)    
data = page.text
soup = BeautifulSoup(data)


scripts = soup.find_all('script')

jsonObj = None
for script in scripts:
    if 'window.pageData=' in script.text:
        jsonStr = script.text

        jsonStr = jsonStr.split("window.pageData=")[1]
        jsonObj = json.loads(jsonStr)


products = jsonObj['mods']['listItems']

for item in products:
    print (item['productUrl'])

Output:

//www.lazada.com.my/products/bundle-of-6-oldtown-white-coffee-3-in-1-classic-15s-x6-packs-free-oldtown-4-sticks-random-flavor-i434054933-s631848411.html?mp=1
//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-hazelnut-15s-x6-packs-i436317818-s639072921.html?mp=1
//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-classic-15s-x4-packs-i436315786-s639051449.html?mp=1
//www.lazada.com.my/products/bundle-of-6-oldtown-white-coffee-3-in-1-extra-rich-15s-x6-packs-free-oldtown-brown-mug-x1-4-sticks-random-flavor-i434056795-s631848533.html?mp=1
//www.lazada.com.my/products/bundle-of-6-oldtown-white-coffee-2-in-1-coffee-creamer-15s-x6-packs-free-oldtown-brown-mug-x1-4-sticks-random-flavor-i434061102-s631849047.html?mp=1
//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-hazelnut-15s-x2-packs-free-2-sticks-random-flavor-i434043717-s631808896.html?mp=1
//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-hazelnut-15s-x4-packs-i436302855-s639061262.html?mp=1
//www.lazada.com.my/products/bundle-of-6-oldtown-white-coffee-3-in-1-less-sugar-15s-x6-packs-free-oldtown-brown-mug-x1-4-sticks-random-flavor-i434054922-s631844581.html?mp=1
//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-extra-rich-15s-x4-packs-i436302850-s639053701.html?mp=1
//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-less-sugar-15s-i428864462-s623547178.html?mp=1
//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-extra-rich-15s-x2-packs-free-2-sticks-random-flavor-i434051011-s631820678.html?mp=1
//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-classic-15s-i429053474-s623935676.html?mp=1
//www.lazada.com.my/products/oldtown-3-in-1-white-milk-tea-13s-i429057450-s623942383.html?mp=1
//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-natural-cane-sugar-15s-x6-packs-i436315838-s639079875.html?mp=1
//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-mocha-15s-i429056707-s623944523.html?mp=1
//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-hazelnut-15s-i446475373-s668555737.html?mp=1
//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-less-sugar-15s-x2-packs-free-2-sticks-random-flavor-i434038995-s631814134.html?mp=1
//www.lazada.com.my/products/bundle-of-6-oldtown-white-coffee-3-in-1-hazelnut-15s-x6-packs-free-oldtown-4-sticks-random-flavor-i434059429-s631843615.html?mp=1
//www.lazada.com.my/products/oldtown-3-in-1-white-milk-tea-13s-x2-packs-free-2-sticks-random-flavor-i434048248-s631818679.html?mp=1
//www.lazada.com.my/products/oldtown-white-coffee-2-in-1-coffee-creamer-15s-x2-packs-free-2-sticks-random-flavor-i434045440-s631815896.html?mp=1
//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-natural-cane-sugar-15s-x4-packs-i436347739-s639057542.html?mp=1
//www.lazada.com.my/products/oldtown-black-series-enrich-freeze-dried-instant-coffee-100g-i434617928-s633601781.html?mp=1
//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-less-sugar-15s-x4-packs-i436353732-s639059385.html?mp=1
//www.lazada.com.my/products/oldtown-white-coffee-2-in-1-coffee-creamer-15s-i429056294-s623942054.html?mp=1
//www.lazada.com.my/products/bundle-of-4-oldtown-white-coffee-2-in-1-coffee-creamer-15s-x4-packs-free-4-sticks-random-flavor-i434053005-s631824941.html?mp=1
//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-classic-15s-x2-packs-free-2-sticks-random-flavor-i434041889-s631816196.html?mp=1
//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-extra-rich-15s-i429055995-s623946380.html?mp=1
//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-less-sugar-15s-x6-packs-i436317821-s639075740.html?mp=1
//www.lazada.com.my/products/oldtown-black-series-enrich-freeze-dried-instant-coffee-100g-2-glass-i435467037-s636334815.html?mp=1
//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-natural-cane-sugar-15s-x2-packs-free-2-sticks-random-flavor-i434043759-s631818279.html?mp=1
//www.lazada.com.my/products/oldtown-white-coffee-2-in-1-coffee-creamer-15s-x2-packs-i436315754-s639016791.html?mp=1
//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-classic-15s-x6-packs-i436353749-s639073757.html?mp=1
//www.lazada.com.my/products/bundle-of-6-oldtown-3-in-1-white-milk-tea-13s-x6-packs-free-oldtown-4-sticks-random-flavor-i434058314-s631843262.html?mp=1
//www.lazada.com.my/products/bundle-of-4-oldtown-white-coffee-3-in-1-hazelnut-15s-x4-packs-free-4-sticks-random-flavor-i446532232-s668570216.html?mp=1
//www.lazada.com.my/products/oldtown-3-in-1-white-milk-tea-13s-x6-packs-i436303866-s639071227.html?mp=1
//www.lazada.com.my/products/oldtown-black-series-enrich-freeze-dried-instant-coffee-100g-2-glass-foc-glass-mug-i442451196-s654637474.html?mp=1
//www.lazada.com.my/products/bundle-of-4-oldtown-3-in-1-white-milk-tea-13s-x4-packs-free-4-sticks-random-flavor-i434046913-s631825874.html?mp=1
//www.lazada.com.my/products/bundle-of-4-oldtown-white-coffee-3-in-1-classic-15s-x4-packs-free-4-sticks-random-flavor-i434052193-s631832149.html?mp=1
//www.lazada.com.my/products/oldtown-3-in-1-white-milk-tea-13s-x2-packs-i436282923-s639018195.html?mp=1
//www.lazada.com.my/products/bundle-of-6-oldtown-white-coffee-3-in-1-natural-cane-sugar-15s-x6-packs-free-oldtown-4-sticks-random-flavor-i434062035-s631849174.html?mp=1