Python Selenium click() only works once, then returns StaleElementReferenceException error [closed]

I am gathering a list of news headlines from Reuters.

from selenium import webdriver
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome('C:chromedriver.exe')
driver.get('https://uk.reuters.com/news/archive/domesticnews?view=page&page=1&pageSize=10')
driver.maximize_window()

AgreeButton = driver.find_element_by_id("_evidon-banner-acceptbutton")
AgreeButton.click()

I need to click on the EARLIER button to load previous headlines. I can successfully get the list of headlines from the first page and the Selenium click() works for the first EARLIER button. Then it starts to return StaleElementReferenceException error. I have checked that the all elements’ class_name remain the same across all pages. So I assume the issue is not caused by the change of DOM.

list = []
headlines = driver.find_elements_by_class_name("story-title")

for i in range(5):
    try:
        for x in headlines:
            list.append(x.text)
            print(x.text) 

        WebDriverWait(driver,10).until(EC.element_to_be_clickable((By.CLASS_NAME, "control-nav-next"))).click()
        
    except Exception as e:
        print(e)

Answer

The click is impacting the DOM, which is rendering all previous webElements to become stale. So even if the xpath remains the same, the webElement object will become stale:

Find some more information here: www.softwaretestingmaterial.com/stale-element-reference-exception-selenium-webdriver/

A solution would be: loop and save the urls, not the webelements OR Open links in new windows and use switchTo