I am using beautiful soup and requests to print full text of the article of this wedsite
https://www.vanityfair.com/style/society/2014/06/monica-lewinsky-humiliation-culture
This is my code:
import requests from bs4 import BeautifulSoup url = requests.get("https://www.vanityfair.com/style/society/2014/06/monica-lewinsky-humiliation-culture") html = url.text page = BeautifulSoup(html, 'html.parser') match = page.find_all('div', 'parbase cn_text') page_list = [[k.get_text() for k in i.find_all('p')] for i in match] for i in page_list[:-2]: for k in i: print(k + 'n')
My code run without any error but it does’t show any text in output plz help me find my error
Advertisement
Answer
What happens?
You try to find_all()
div tags with two classes that do not exists, so match
is empty.
How to fix that?
Use the correct pattern, I took the css selectors to avoid an additional loop:
select('article.article.main-content p')
List comprehension then looks like:
[p.get_text() for p in page.select('article.article.main-content p')]
Working example
import requests from bs4 import BeautifulSoup url = requests.get("https://www.vanityfair.com/style/society/2014/06/monica-lewinsky-humiliation-culture") html = url.text page = BeautifulSoup(html, 'html.parser') print(*[p.get_text() for p in page.select('article.article.main-content p')], sep='n')