Why selenium is not starting next url after quiting the previous url

Actully I’m working on some URLs. URLs are store in my database while picking one by one and looking for it resources and storing that resources in a database. if I do this with out using driver.quit() somehow for everyurl it store information from the beginning so I decide to use driver.quit now it work only for the first URL and does not work on second URL and so on. But currently I’m just printing the data to see it works according to me. already check How to loop through a list of urls using Selenium and Python

below is the part of code::

def select_url_test():
    http = 0
    https = 0

    driver = webdriver.Chrome()

    db_conn1 = Foreign_Key_table.database_conn()
    db_conn1.execute("SELECT url,id FROM SELECTED_URLS WHERE url_status = 'VALID' AND captcha_status = 'NO' LIMIT 2")
    urls = db_conn1.fetchall()

    for url_aa in urls:
        full_url = 'https://' + url_aa[0]  # url is in form of ('google.com' ,)
        # print(url[0])
        # print(full_url)
        driver.get(full_url)
        time.sleep(2)
        for request in driver.requests:    
            if request.url.startswith('https'):
                https += 1            
                print(request.url)
        driver.quit()

Answer

You need to put the line driver = webdriver.Chrome() into the loop. That’s the line that actually opens chrome. So, at the end of the loop you do driver.quit(). That closes the browser. And at the beginning of each loop you need to open it again. So place driver = webdriver.Chrome() before driver.get(full_url) inside the loop.

Advertisement

Answer