I’m having a really hard time locating elements on this website. My ultimate aim here is scrape the score card data for each country. The way I’ve envisioned doing that is to have selenium click list view (as opposed to the default country view) and loop going in and out of all the country profiles, scraping the data for each country along the way. But methods to locate elements on the page that worked for me in the past, have been fruitless with this site.
Here’s some sample code outlining my issue.
from selenium import webdriver
driver = webdriver.Chrome(executable_path= "C:/work/chromedriver.exe")
driver.get('https://www.weforum.org/reports/global-gender-gap-report-2021/in-full/economy-profiles#economy-profiles')
# click the `list` view option
driver.find_element_by_xpath('//*[@id="root"]/div/div[1]/div[2]/div[2]/svg[2]')
As you can see, I’ve only gotten as far as step 1 of my plan. I’ve tried what other questions have suggested as far as adding waits, but to no avail. I see the site fully loaded in my DOM, but no xpaths are working for any element on there I can find. I apologize if this question is posted too often, but do know that any and all help is immensely appreciated. Thank you!
Advertisement
Answer
The element is inside an iframe you need to switch it to access the element.
Use WebDriverWait() wait for frame_to_be_available_and_switch_to_it()
driver.get("https://www.weforum.org/reports/global-gender-gap-report-2021/in-full/economy-profiles#economy-profiles")
wait=WebDriverWait(driver,10)
wait.until(EC.frame_to_be_available_and_switch_to_it((By.ID, "iFrameResizer0")))
wait.until(EC.element_to_be_clickable((By.XPATH,"//*[name()='svg' and @class='sc-gxMtzJ bRxjeC']"))).click()
You need following imports.
from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC from selenium.webdriver.common.by import By
Another update:
driver.get("https://www.weforum.org/reports/global-gender-gap-report-2021/in-full/economy-profiles#economy-profiles")
wait=WebDriverWait(driver,10)
wait.until(EC.frame_to_be_available_and_switch_to_it((By.ID, "iFrameResizer0")))
driver.find_element_by_xpath("//*[name()='svg' and @class='sc-gxMtzJ bRxjeC']").click()