I have been trying to scrape this site and sending in the search phrase cause error.
AttributeError: 'NoneType' object has no attribute 'send_keys'
I have searched StackOverflow and the error usually is resolved when you redefine the web element again which I did but still getting the same error.
I do not want to directly go to the search link like https://www.couriermail.com.au/search-results?q=Anthony+Albanese instead I want to click on the search icon and after the input field is open send in the search phrase.
Here is what I have tried
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "/html/body/nav/form/button"))).click()
search = driver.find_element(by=By.XPATH,value='/html/body/nav/form/button').click()
search.send_keys("Anthony Albanese")
time.sleep(10)
search.send_keys(Keys.ENTER)
I have tried simulating the typing-like behavior but it results in other errors stackoverflow help
This is the HTML of the page that I am referring to The HTML of the search icon so my question is what I am doing wrong? I have tried redefining the web element or imitating the typing-like behavior but I am not successful in getting the URLs of articles
Advertisement
Answer
Yet another answer, but with relative xpaths instead of absolute. Also, instead of simulating ENTER key, I have used the clicking of search button.
driver.get("https://www.couriermail.com.au/")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//*[@class='navigation']//button[@name='Search']"))).click()
input_text = driver.find_element(by=By.XPATH, value="//form[contains(@class, 'navigation_search')]//input")
input_text.send_keys("Anthony Albanese")
print(input_text.get_attribute('value'))
time.sleep(2)
driver.find_element(By.XPATH, "//*[@class='navigation']//button[@name='Search']").click()
time.sleep(20)
# WebDriverWait(driver, 30).until(EC.visibility_of_all_elements_located((By.XPATH, "//*[@class='storyblock_title']")))
all_articles = driver.find_elements(By.XPATH, "//*[@class='storyblock_title']")
article_headers = [article.text for article in all_articles]
print(article_headers)
print(driver.current_url)
assert driver.current_url == "https://www.couriermail.com.au/search-results?q=Anthony+Albanese"
driver.quit()
Output:
Anthony Albanese ['Albo saves Cutters’ PNG fixture after boycott threat', 'Sport diplomacy: Why a QRL game is so important to the PM', 'Albo’s cheeky dig at ScoMo during Midnight Oil event', 'How Albo amassed huge property portfolio', '‘I can’t solve everything’: PM’s big admission', 'PM admits: ‘Can’t solve everything’', "Anthony Albanese 'emulated Bob Hawke'", "People 'cheered like idiots' at Anthony Albanese skolling beer", "'Hold my beer': Anthony Albanese 'trying to be as popular as Hawke'", 'I want to look at ‘how issues could be improved in the future’: Anthony Albanese', 'PM Anthony Albanese Signals Review Of Scott Morrison’s Conduct Is On The Horizon 23/08/2022', 'Standing ovation as Albo necks beer', 'Albo chugs beer at Gang of Youths concert', 'Aussies being left behind by the PM', 'Prime Minister’s big jobs promise', 'Huge push for foreign workers in Australia', 'Can we afford another mouth to feed?', 'SA projects under review after former PM’s power snatch', 'Albanese hints at ScoMo inquiry', '‘Never again’: Albo flags huge action'] https://www.couriermail.com.au/search-results?q=Anthony+Albanese Process finished with exit code 0
P.S. print and sleep statements are for easy visual. They are optional and could be omitted. If needed by explicit_wait like webdriverwait could be used.