Im learning beautifulsoup and I came a cross one problem. Thats scraping dd tags in html. Check out the picture below, I want to get the parameters that are in the red color zone. The problem is I do not know how to access them. I have tried this: But the problem is that sometimes different pages have differe…
Tag: web-scraping
Web scraping problem during passing fuction as paramater in function
Hello I’ve created two functions that work well well called alone. But when I try to use a for loop with these functions I got a problem with my parameter. First function to search and get link to pass to the second one. Second function to scrape a link. All these function worked when I tested them on a…
Accessing the contents on links provided on a webpage while webscraping
This is a followup question of my previous question. I am trying to access the contents of a webpage. I could search for contents on the webpage. However, I am not sure how to access the contents in links given on the webpage. For instance, the first line of the search result for id 1.1.1.1 is 36EUL/ADL_7 1.1…
Trying to get only the text between two strong tags
I am currently trying to get only the HTML text (a list of names) that is between the first two occurrences of the strong tag. Here is a short example of the HTML I scraped Hers is some quick code that I wrote with the basic logic of counting the number of strong tags occurring. I know after the second
Creating custom web scraping tool to count unique words in python
I’m trying to create a function that has 2 arguments, a web URL, and a search word. The function should print out the number of times the word is seen on the page. I am currently unsure of what I’m doing wrong, as my output isn’t giving me neither an error nor an output… So if a user t…
How to remove target tr block using beautifulsoup
I want to remove target tr block with text, when i run it i got perfect output but there is a problem i have seen that it scraping <tr><td>Domain</td><td>Last Resolved Date</td></tr> actually i don’t want this line in my output so how can i remove it.Code bellow Got f…
How can I scrape all the images from a website?
I have a website where I’d like to get all the images from the website. The website is kind of a dynamic in nature, I tried using google’s Agenty Chrome extension and followed the steps: I Choose one image that I want to extract using CSS selector, this will make the extension select the same othe…
Python & BS4 – Strange behaviour, scraper freezes/stops working after a while without an error
I’m trying to scrape eastbay.com for Jordans. I have set up my scraper using BS4 and it works, but never finishes or reports an error, just freezes at some point. The strange thing is that it stops at some point and pressing CTRL+C in the Python console (where it’s outputting the prints as it̵…
Can this beautifulsoup script be simplified with Regex?
I wrote some beautifulsoup scripts, and one part seems really redundant, I am thinking if it can be simplified with Regex. All posts from this forum are marked with different colors, what I did is to search each color with one line. For six colors I did six lines with only one words difference. I am not sure …
pandas read_html – no tables found
I am attempting to see if I can read a table of data from WU.com, but I am getting a type error for no tables found. (first timer on web scraping too here) There is also another person with a very similar stackoverflow question here with WU table of data, but the solution is a little bit complicated to me.