I am using beautiful soup and requests to print full text of the article of this wedsite
https://www.vanityfair.com/style/society/2014/06/monica-lewinsky-humiliation-culture
This is my code:
import requests
from bs4 import BeautifulSoup
url = requests.get("https://www.vanityfair.com/style/society/2014/06/monica-lewinsky-humiliation-culture")
html = url.text
page = BeautifulSoup(html, 'html.parser')
match = page.find_all('div', 'parbase cn_text')
page_list = [[k.get_text() for k in i.find_all('p')] for i in match]
for i in page_list[:-2]:
for k in i:
print(k + 'n')
My code run without any error but it does’t show any text in output plz help me find my error
Advertisement
Answer
What happens?
You try to find_all() div tags with two classes that do not exists, so match is empty.
How to fix that?
Use the correct pattern, I took the css selectors to avoid an additional loop:
select('article.article.main-content p')
List comprehension then looks like:
[p.get_text() for p in page.select('article.article.main-content p')]
Working example
import requests
from bs4 import BeautifulSoup
url = requests.get("https://www.vanityfair.com/style/society/2014/06/monica-lewinsky-humiliation-culture")
html = url.text
page = BeautifulSoup(html, 'html.parser')
print(*[p.get_text() for p in page.select('article.article.main-content p')], sep='n')