How do I capture hidden data from a table with Selenium and Python?

my problem is the following:

The web page indicated in the script has a button (“Ver todo 50”) that displays the rest of the records of a table. I effectively click on the button with Selenium, but I can’t get the 50 records of the table but only the first 15 that the page initially displays.

Does anyone have an idea to collect the full logs (50)?

The script is:

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By


chrome_path= r'C:UsersdddddddDesktopdddddchromedriver.exe'
driver= webdriver.Chrome(chrome_path)
driver.get('https://getdaytrends.com/es/venezuela/')

new= WebDriverWait(driver, 30).until(EC.element_to_be_clickable((By.XPATH, '//*[@id="trends"]/div/a'))).click()

#to determine the max rows to iterate
rows= len(driver.find_elements_by_xpath('//*[@id="trends"]/table[1]/tbody/tr'))

trends= []
for n in range(1, rows+1):
    dato= driver.find_element_by_xpath('//*[@id="trends"]/table[1]/tbody/tr['+str(n)+']/td[1]').text
    trends.append(dato)

print(len(trends)) #I need 50 records

Answer

I tried the below code, and the output was those 50 options.

    element = driver.find_elements_by_xpath("//div[@id='trends']/table//a")
    print(len(element))
    for ele in element:
        print(ele.get_attribute('text'))

Advertisement

Answer