Skip to content
Advertisement

Extracting texts from Selenium WebElement

I’m trying to get all the names of the perfumes on this page

introducir la descripción de la imagen aquí

But when I try to get the text of the web element attribute I get nothing. In fact, I tried en colaboratory:

!pip install selenium
!apt-get update # to update ubuntu to correctly run apt install
!apt install chromium-chromedriver
!cp /usr/lib/chromium-browser/chromedriver /usr/bin
import sys
sys.path.insert(0,'/usr/lib/chromium-browser/chromedriver')
from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')
wd = webdriver.Chrome('chromedriver',chrome_options=chrome_options)
wd.get("https://www.beaute-test.com/parfums_femme.php")
media_content = wd.find_elements_by_class_name("bt__media__content")

En media_content we have several elements:

[<selenium.webdriver.remote.webelement.WebElement (session="18f3eed8343ccf0c22fc5c7bae48cff7", element="3123e486-a978-4113-b359-c3e9a89c7e95")>,
 <selenium.webdriver.remote.webelement.WebElement (session="18f3eed8343ccf0c22fc5c7bae48cff7", element="4a7b2d4e-0179-4957-ab32-8f37470bc694")>,
 <selenium.webdriver.remote.webelement.WebElement (session="18f3eed8343ccf0c22fc5c7bae48cff7", element="a7fb478d-aa55-46b8-8953-a7b4dd84220e")>,...
]

But this one does not return anything:

>>>media_content[1].get_attribute('text')
''
>>>media_content[1].text
''

I also tried using the xpath directly on one name I wass looking for but it returns nothing as well.

So how do I get the text of a WebElement ?

Advertisement

Answer

To display all 25 of the names do the following.

media_content=wd.find_elements_by_css_selector("div[class='bt__media__content']>p>a>b")
for media in media_content:
    print (media.text)

I also used to click on the agree popup.

WebDriverWait(wd, 10).until(EC.element_to_be_clickable((By.ID, "didomi-notice-agree-button"))).click()

Outputs

Libre
L'Interdit
Madame Glamour Suddenly
Gabrielle
My Way
Eau de Parfum - Alien
Idôle
Good Girl
Coco Noir
Scandal
Mon Guerlain
Mon Rouge!
Black Opium Eau de Parfum
Mauboussin à La Folie
Black Orchid
Voce Viva
La Nuit Trésor
Eau de Parfum - La Vie est Belle
Chloé Eau de Parfum
Eau de Parfum - La petite Robe noire
Suddenly Diamonds
Mon Paris Eau de Parfum
La Belle
Hypnotic Poison Eau de Toilette
Allure
Advertisement