Web page is loaded in selenium and reaches to end but does not contain all the elements inside the div

This is the site. https://www.talabat.com/uae/top-selling. There are somewhat 100 products and only 30 gets loaded. I was trying to fetch all the links and page reaches to end but only display 30 products and when clicked somewhere in the webdriver then loads the rest of the products. How can I print the links of all the products? Thanks in advance!!

from selenium import webdriver
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time

from bs4 import BeautifulSoup

HOME_PAGE_URL = "https://www.talabat.com/uae/top-selling"
PATIENCE_TIME = 60
LOAD_MORE_XPATH = '//*[@id="comment-ajx"]/div'

driver = webdriver.Chrome(executable_path='C:\Users\Mansi Dhingra\Downloads\chromedriver.exe')
driver.get(HOME_PAGE_URL)
soup=BeautifulSoup(driver.page_source)

driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
# sleep for 30s
res=[]

results = driver.find_elements_by_xpath("/html/body/div[3]/ui-view/section/div/div[2]/div/div[2]/div/div[2]")

html_code = driver.find_element_by_tag_name("section").text
print(html_code)
for res in results:
    link=res.find_elements_by_tag_name('a')
    for x in link:
        product_link = x.get_attribute("href")

        print(product_link)
print(results)

Answer

The main point is that selenium reads the page before the page has loaded all the items, you need a wait.

Just read the docs: https://selenium-python.readthedocs.io/waits.html

Choose the best condition for you case and go for it.

Advertisement

Answer