I’m trying to scrape Historical Bitcoin Data from coinmarketcap.com in order to get close, volume, date, high and low values since the beginning of the year until Sep 30, 2021. After going through threads and videos for hours, and I’m new to scraping with Python, I don’t know what my mistake is (or is there something with the website I don’t detect?). The following is my code:
JavaScript
x
33
33
1
from bs4 import BeautifulSoup2
import requests3
import pandas as pd4
5
6
closeList = []7
volumeList = []8
dateList = []9
highList = []10
lowList = []11
12
website = 'https://coinmarketcap.com/currencies/bitcoin/historical-data/'13
14
r = requests.get(website)15
16
r = requests.get(website)17
soup = BeautifulSoup(r.text, 'lxml')18
19
tr = soup.find_all('tr')20
FullData = []21
for item in tr:22
closeList.append(item.find_all('td')[4].text)23
volumeList.append(item.find_all('td')[5].text)24
dateList.append(item.find('td',{'style':'text-align: left;'}).text)25
highList.append(item.find_all('td')[2].text)26
lowList.append(item.find_all('td')[3].text)27
FullData.append([closeList,volumeList,dateList,highList,lowList])28
29
df_columns = ["close", "volume", "date", "high", "low"]30
31
df = pd.DataFrame(FullData, columns = df_columns)32
print(df)33
As a result I only get:
JavaScript
1
4
1
Empty DataFrame2
Columns: [close, volume, date, high, low]3
Index: []4
The task obliges me to scrape with BeautifulSoup and then export to csv (which obviously then is simply df.to_csv – can somebody help me out? That would be highly appreciated.
Advertisement
Answer
Actually, data is loaded dynamically by javascript from api calls json response. So you can grab data easily as follows:
Code:
JavaScript
1
21
21
1
import requests2
import json3
import pandas as pd4
api_url= 'https://api.coinmarketcap.com/data-api/v3/cryptocurrency/historical?id=1&convertId=2781&timeStart=1632441600&timeEnd=1637712000'5
r = requests.get(api_url)6
data = []7
for item in r.json()['data']['quotes']:8
close = item['quote']['close']9
volume =item['quote']['volume']10
date=item['quote']['timestamp']11
high=item['quote']['high']12
low=item['quote']['low']13
data.append([close,volume,date,high,low])14
15
16
cols = ["close", "volume","date","high","low"]17
18
df = pd.DataFrame(data, columns= cols)19
print(df)20
#df.to_csv('info.csv',index = False)21
Output:
JavaScript
1
15
15
1
close volume date high low2
0 42839.751696 4.283935e+10 2021-09-24T23:59:59.999Z 45080.491063 40936.5571693
1 42716.593147 3.160472e+10 2021-09-25T23:59:59.999Z 42996.259704 41759.9204254
2 43208.539105 3.066122e+10 2021-09-26T23:59:59.999Z 43919.300970 40848.4616605
3 42235.731847 3.098003e+10 2021-09-27T23:59:59.999Z 44313.245882 42190.6325766
4 41034.544665 3.021494e+10 2021-09-28T23:59:59.999Z 42775.146142 40931.6625007
.. 8
56 58119.576194 3.870241e+10 2021-11-19T23:59:59.999Z 58351.113266 55705.1806859
57 59697.197134 3.062426e+10 2021-11-20T23:59:59.999Z 59859.880442 57469.72566110
58 58730.476639 2.612345e+10 2021-11-21T23:59:59.999Z 60004.426383 58618.93143211
59 56289.287323 3.503612e+10 2021-11-22T23:59:59.999Z 59266.358468 55679.84040412
60 57569.074876 3.748580e+10 2021-11-23T23:59:59.999Z 57875.516397 55632.75991213
14
[61 rows x 5 columns]15