I am attempting to see if I can read a table of data from WU.com, but I am getting a type error for no tables found. (first timer on web scraping too here) There is also another person with a very similar stackoverflow question here with WU table of data, but the solution is a little bit complicated to me.
JavaScript
x
6
1
import pandas as pd2
3
df_list = pd.read_html('https://www.wunderground.com/history/daily/us/wi/milwaukee/KMKE/date/2013-6-26')4
5
print(df_list)6
On the webpage of historical data for Milwaukee, this is the table of data (daily observations) that I am attempting to retrieve into Pandas:
Any tips help, thank you.
Advertisement
Answer
the page is dynamic which means you’ll need to to render the page first. So you would need to use something like Selenium to render the page, then you can pull the table using pandas .read_html():
JavaScript
1
14
14
1
from selenium import webdriver2
import pandas as pd3
4
5
driver = webdriver.Chrome('C:/chromedriver_win32/chromedriver.exe')6
driver.get("https://www.wunderground.com/history/daily/us/wi/milwaukee/KMKE/date/2013-6-26")7
8
html = driver.page_source9
10
tables = pd.read_html(html)11
data = tables[1]12
13
driver.close()14
Output:
JavaScript
1
35
35
1
print (data)2
Time Temperature Precip Accum Condition3
0 6:52 PM 68 F 0.0 in Mostly Cloudy4
1 7:52 PM 69 F 0.0 in Mostly Cloudy5
2 8:52 PM 70 F 0.0 in Mostly Cloudy6
3 9:52 PM 67 F 0.0 in Cloudy7
4 10:52 PM 65 F 0.0 in Partly Cloudy8
5 11:42 PM 66 F 0.0 in Mostly Cloudy9
6 11:52 PM 68 F 0.0 in Mostly Cloudy10
7 12:08 AM 68 F 0.0 in Cloudy11
8 12:52 AM 68 F 0.0 in Mostly Cloudy12
9 1:52 AM 70 F 0.0 in Cloudy13
10 2:13 AM 70 F 0.0 in Cloudy14
11 2:52 AM 71 F 0.0 in Cloudy15
12 3:52 AM 70 F 0.0 in Mostly Cloudy16
13 4:19 AM 70 F 0.0 in Cloudy17
14 4:29 AM 70 F 0.0 in Cloudy18
15 4:52 AM 70 F 0.0 in Cloudy19
16 5:25 AM 70 F 0.0 in Mostly Cloudy20
17 5:52 AM 71 F 0.0 in Cloudy21
18 6:52 AM 73 F 0.0 in Cloudy22
19 7:52 AM 74 F 0.0 in Cloudy23
20 8:52 AM 73 F 0.0 in Cloudy24
21 9:52 AM 71 F 0.0 in Cloudy25
22 10:52 AM 71 F 0.0 in Cloudy26
23 11:52 AM 70 F 0.0 in Cloudy27
24 12:52 PM 72 F 0.0 in Mostly Cloudy28
25 1:52 PM 70 F 0.0 in Mostly Cloudy29
26 2:52 PM 71 F 0.0 in Mostly Cloudy30
27 3:52 PM 71 F 0.0 in Partly Cloudy31
28 4:52 PM 68 F 0.0 in Mostly Cloudy32
29 5:52 PM 66 F 0.0 in Mostly Cloudy33
34
[30 rows x 11 columns]35