Hello so i'm trying to scrape next page on google shopping website. But i get an error whenever my scraper clicks the next button. It stops working when it loads the next page. Here's my codeblock next_page = driver.find_element(By.XPATH, '//*[@id="pnnext"]') next_page.click() HERE'S THE ERROR I GET Traceback (most recent call last): File "c:UsersLPDocumentspythonwedgwoodwedgwood.py", line 50, in name = card.find_element(By.CLASS_NAME, 'OSrXXb').text.strip()

How do I stop my scraper from hitting an error whenever it clicks the next button?

Hello so i’m trying to scrape next page on google shopping website. But i get an error whenever my scraper clicks the next button. It stops working when it loads the next page.

Here’s my codeblock

from selenium import webdriver
import time
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
import pandas as pd




 url = 'https://www.google.com.ng/search?q=list+of+all+uk+e-commerce+stores+for+buying+prada+products&hl=en&biw=946&bih=625&tbm=lcl&sxsrf=ALiCzsaIKyYpvCJVWZx_fYTwSQerSvzC6g%3A1667482905673&ei=GcVjY4fUKJeG9u8PgvGwoAE&ved=0ahUKEwjHxIvykZL7AhUXg_0HHYI4DBQQ4dUDCAk&uact=5&oq=list+of+all+uk+e-commerce+stores+for+buying+prada+products&gs_lp=Eg1nd3Mtd2l6LWxvY2FsuAED-AEBMgUQABiiBDIHEAAYHhiiBDIFEAAYogQyBRAAGKIEwgIEECMYJ0iSHFDlBliOFHAAeADIAQCQAQCYAYYDoAHxDqoBBTItMS41iAYB&sclient=gws-wiz-local#rlfi=hd:;si:;mv:[[56.121909699999996,0.16756959999999999],[51.208233299999996,-4.5053765]]'

 service = Service(executable_path="C:/driver/chromedriver_win32/chromedriver.exe")

 driver = webdriver.Chrome(service=service)

 driver.get(url)

 driver.maximize_window()

 time.sleep(8)



for i in range(7):    

   site_cards = driver.find_elements(By.CLASS_NAME, 'uMdZh')
   time.sleep(4)

   site_list = []

   for card in site_cards:
      name = card.find_element(By.CLASS_NAME, 'OSrXXb').text.strip()
      submit = card.find_element(By.CLASS_NAME, 'OSrXXb')
      submit.click()
      time.sleep(4)
      try:
        more = driver.find_element(By.CLASS_NAME, 'Yy0acb').text.strip()
      except:
        print('none')
      try:
        more = driver.find_element(By.CLASS_NAME, 'mPcsfb').text.strip()
      except:
        print('none')
      time.sleep(2)
      try:
        more = driver.find_element(By.CLASS_NAME, 'YhemCb').text.strip()
      except:
        print('none')
      time.sleep(2)
      try:
        more = driver.find_element(By.CLASS_NAME, 'PQbOE').text.strip()
      except:
        print('none')
      try:
        more = driver.find_element(By.CLASS_NAME, 'Yy0acb').text.strip()
      except:
        print('none')
      try:
        more = driver.find_element(By.NAME, 'EvNWZc').text.strip()
      except:
        print('none')
      time.sleep(4)


      if ModuleNotFoundError:
        pass

      site_info = (name, more)
      site_list.append(site_info)

      col = ['Site Name', 'Site Link']
      df = pd.DataFrame([site_info], columns=col)
      df.to_csv("C:\UsersLPDocumentspythonwedgwoodprada2.csv", index=False, encoding='utf-8', mode='a+')

next_page = driver.find_element(By.XPATH, ‘//*[@id=”pnnext”]’) next_page.click()

HERE’S THE ERROR I GET

Traceback (most recent call last): File “c:UsersLPDocumentspythonwedgwoodwedgwood.py”, line 50, in name = card.find_element(By.CLASS_NAME, ‘OSrXXb’).text.strip()

selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document

Answer

That error means when it’s looking for that element ‘OSrXXb’ it can’t find it. You have a bunch of other elements wrapped in try blocks but it seems this one you expect to be there and it isn’t.

Advertisement

Answer