I’m trying to get some data from dividendinvestor.com. But the content results did not have any information such as “Consecutive Dividend Increases”.
Does anyone has a work around for this?
import requests
from bs4 import BeautifulSoup
url = 'https://www.dividendinvestor.com/dividend-quote/intc/'
headers={
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:83.0) Gecko/20100101 Firefox/83.0',
'Connection': 'keep-alive',
'Referer': f'{url}',
'Cookie': 'seenslideup=1; seenpopup=1',
'TE': 'Trailers'
}
print(str(BeautifulSoup(r_get.content, 'html.parser'))
Advertisement
Answer
As you can get ajax request URL from Network tab which returns
jsondata you can parse it tobs4and it returns HTML so you can extract what so ever data is needed!
import requests
from bs4 import BeautifulSoup
headers={"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36"}
res=requests.get("https://www.dividendinvestor.com/ajax/?action=quote_ajax&type=POST&nonce=ea2c75061e&symbol=INTC&user_type=visitor",headers=headers)
import json
data=json.loads(res.text)['html']
soup=BeautifulSoup(data,"lxml")
Output:
<html><body><h1>INTC:Intel Corp - Stock Price Quote and Dividend Data</h1> <div id="overview"> <div id="overview-close"> <h2>INTC STOCK PRICE LATEST CLOSE</h2> ....