I am new and am trying to get BeautifulSoup to work. I have Html problems with recovering classes and tags. I get closer, but there is something I’m wrong. I insert wrong tags and classes to scrape the title, time, link, and text of a news item. I would like to scrape all those titles in the vertical li…
Tag: web-scraping
How to extract a table from website without specifying the web browser in python
I’m trying to automate data extraction from ASX (https://www.asxenergy.com.au/futures_nz) website into my database by writing a web scraping python script and deploying it in Azure Databrick. Currently, the script I have is working in Visual Studio Code, but when I try to run it in databrick, it crashes…
Python / Selenium only print if class has a certain value + only print certain elements in html
I am coding a script that is going to test all free proxies available on: https://free-proxy-list.net/ On this site there is a list with all available proxies, and I managed to make my script print them all but, I only want to print the proxy value if https is enabled. This is how the Html looks when https is…
How to get all links from a webpage using selenium?
I am trying to webscrape a site using Python, Selenium, Beautifulsoup. When I tried to get all the links ,It’ returning an invalid string. This is what I have tried Can someone help me please? Answer It is your selection with xpath, you select the <div> that do not have an href attribute. Select a…
Python helium get contents of table after click
I am using helium to scrape a webpage. After the click action i am presented with a table and i need to scrape the contents of the table but How do i select the table after the click ? Answer You would need to use find_elements_… to get all <table>, and use for-loop to work with every table separa…
Selenium element is not attached to the page document
I am trying to scrape this particular site with Python: https://www.milanofinanza.it/quotazioni/ricerca/listino-completo-2ae?refresh_cens. I need to get all the isin codes and the names. My idea was to get them all in 2 separated lists, to do that I try to get the entire column (by changing the Xpath to tr ra…
Trying to split text from title
I want to remove these from my output: I want these only Wave Coffee Collection This is my code : Answer If this is your resulting output: Then you can easily achieve your desired output like this:
scrape next page by changing the number of page in URL
I have trouble scraping information on the next pages. I also have a problem when some tags change like when the website developer changes an “a href” with “h2 class” when I reach the appart_response= requests.get(link) Please can you check with me the following code: Answer There is s…
scraping multiple tags at once
I’m trying to scrape the imdb top 250 movies and I want to get all the links of those movies from this page https://www.imdb.com/chart/top/ I tried but I’m only getting the first link only, so my question is how to scale this code to include the whole list of 250 movies? Answer bs.find(‘td&#…
Scraping all entries of lazyloading page using python
See this page with ECB press releases. These go back to 1997, so it would be nice to automate getting all the links going back in time. I found the tag that harbours the links (‘//*[@id=”lazyload-container”]’), but it only gets the most recent links. How to get the rest? Answer The dat…