I am trying to scrape the data from naukri.com, here I am trying to scrape the location details for each recruiter visible on the page.
The code I am writing is :
def sol7(url): driver = webdriver.Chrome(r'C:UsersRahulDownloadschromedriver_win32chromedriver.exe') driver.maximize_window() driver.get(url) #getting link of recruiters recruiters_tag = driver.find_element_by_xpath("//ul[@class='midSec menu']/li[2]/a") driver.get(recruiters_tag.get_attribute('href')) #search and click for data scientist driver.find_element_by_xpath("//input[@class='sugInp']").send_keys('Data science') driver.find_element_by_xpath("//button[@id='qsbFormBtn']").click() highlight_table_tag = driver.find_elements_by_xpath("//p[@class='highlightable']") print(len(highlight_table_tag)) for i in highlight_table_tag: try: print((i.find_element_by_xpath("//small[@class='ellipsis']")).text) except NoSuchElementException: print('-')
1st I have extracted all recruiters details in list highlight_table_tag. highlight_table_tag includes all the elements on the page however the loop only takes the 0th element of my list. I want to scrape the location in such a way, if the location tag is not there in each element of highlight_table_tag then print ‘-‘ instead.
Please help!!!!
Advertisement
Answer
While xpathing child elements use . in front of the xpath otherwise you will be getting it from root node. Which ends up with 1 element everytime.
print((i.find_element_by_xpath(".//small[@class='ellipsis']")).text)