Skip to content
Advertisement

Scraping dynamic dropdown with Selenium in Python

I woud like a bit of help with the following. I am trying to scrape the elements of the tickers’ dropdown on this website: https://live.hxro.io/tixwix

My code is as follow using selenium

from selenium import webdriver
from selenium.webdriver.support.select import Select
from selenium.webdriver.chrome.options import Options

url = "https://live.hxro.io/tixwix"
chrome_options = Options()
chrome_options.add_argument("--headless")
driver = webdriver.Chrome(executable_path = r'C:geckodriverchromedriver.exe',options = chrome_options)
driver.get(url)

tickers = driver.find_elements_by_class_name('lastprice-toggle')

tickers[0].text

This will only return

'BTCnLast Pricen$39,255.07'

As it is an Ajax call I am not sure how to retrieve the other tickers in an efficient way. I thought the function find_element’s’ will return all the elements into a list but I only get the first one tickers[1] is out of bound.

Screenshot of the page source: enter image description here

Thanks

Advertisement

Answer

The below code works fine on my local :

Explanation :

You need to click on accept cookies button and shall try with ExplicitWait plus you would need to click on a svg icon which is there in your drop down (XPATH : //span[text()='Last Price']/../following-sibling::*).

In the end span.moon contains all the elements that you are looking for.

Code :

executablePath = r'C:geckodriver.exe'
options = webdriver.FirefoxOptions()
options.add_argument("--headless")
driver = webdriver.Firefox(executable_path = executablePath, options=options)
driver.maximize_window()
driver.get("https://live.hxro.io/tixwix")
wait = WebDriverWait(driver, 10)
wait.until(EC.element_to_be_clickable((By.ID, "onetrust-accept-btn-handler"))).click()
wait.until(EC.element_to_be_clickable((By.XPATH, "//span[text()='Last Price']/../following-sibling::*"))).click()
sleep(5)
for values in driver.find_elements(By.CSS_SELECTOR, "span.moon"):
    print(values.get_attribute('innerHTML'))

output :

$39,321.02
$39,321.02
$8,728.5
$0.31063
$2,435.337
$24.232
$40.99
$8.9730
$22.685
$37,234.01
 HIGHER
<span class="tooltip-text">CLOSE ABOVE</span>
(CLOSE ABOVE)
(TOUCH)
 $100,000
14.20X
 $100,000
1000.05X
 $68,000
152.69X
 $66,000
127.38X
 $64,000
117.34X
 $63,000
112.25X
 $62,000
105.95X
 $61,000
4.490X
 $60,000
92.80X
 $44,000
4.790X

Process finished with exit code 0
User contributions licensed under: CC BY-SA
1 People found this is helpful
Advertisement