Skip to content
Advertisement

Beautiful Soup and requests problem it doesn’t show any text output

I am using beautiful soup and requests to print full text of the article of this wedsite

https://www.vanityfair.com/style/society/2014/06/monica-lewinsky-humiliation-culture

This is my code:

import requests
from bs4 import BeautifulSoup

url = requests.get("https://www.vanityfair.com/style/society/2014/06/monica-lewinsky-humiliation-culture")
html = url.text
page = BeautifulSoup(html, 'html.parser')
match = page.find_all('div', 'parbase cn_text')
page_list = [[k.get_text() for k in i.find_all('p')] for i in match]

for i in page_list[:-2]:
   for k in i:
     print(k + 'n')

My code run without any error but it does’t show any text in output plz help me find my error

Advertisement

Answer

What happens?

You try to find_all() div tags with two classes that do not exists, so match is empty.

How to fix that?

Use the correct pattern, I took the css selectors to avoid an additional loop:

select('article.article.main-content p')

List comprehension then looks like:

[p.get_text() for p in page.select('article.article.main-content p')]

Working example

import requests
from bs4 import BeautifulSoup

url = requests.get("https://www.vanityfair.com/style/society/2014/06/monica-lewinsky-humiliation-culture")
html = url.text
page = BeautifulSoup(html, 'html.parser')


print(*[p.get_text() for p in page.select('article.article.main-content p')], sep='n')
User contributions licensed under: CC BY-SA
9 People found this is helpful
Advertisement