How to scroll at the end of a page with finite number of load ? Selenium – Python

Tags: , ,

I would like to scroll until the end of a page like :

The fact is using this :

# # Get scroll height after first time page load
 last_height = driver.execute_script("return document.body.scrollHeight")
 while True:
     # Scroll down to bottom
     driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
     # Wait to load page
     # Calculate new scroll height and compare with last scroll height
     new_height = driver.execute_script("return document.body.scrollHeight")
     if new_height == last_height:
     last_height = new_height

does not work. yes it should work for pages with infinite loads but doesn’t work for yahoo finance, which has a finite number of loads but the condition should break when it reachs the end. So I’m quite confuse at the moment.

We could also use :

while driver.find_element_by_tag_name('tfoot'):
    # Scroll down three times to load the table
    for i in range(0, 3):
        driver.execute_script("window.scrollBy(0, 5000)")

but it sometimes blocks at a certain loads.

What would be the best way to do this ?


Requires pip install undetected-chromedriver, but will get the job done. It’s just my webdriver of choice, you can also do the exact same with normal selenium.

from time import sleep as s
from selenium.webdriver.common.keys import Keys
from import By
from import expected_conditions as EC
from import WebDriverWait

import undetected_chromedriver as uc
options = uc.ChromeOptions()
options.headless = False
driver = uc.Chrome(options=options)


WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CSS_SELECTOR, '#consent-page > div > div > div > div.wizard-body > div.actions.couple > form > button'))).click() #clicks the cookie warning or whatever

while True:
    WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CSS_SELECTOR, 'body'))).send_keys(Keys.DOWN)
    current_scroll_pos=str(driver.execute_script('return window.pageYOffset;'))
    if current_scroll_pos == last_scroll_pos:
        print('scrolling is finished')

Source: stackoverflow