I am trying to see if I can use, and only use, Pandas’ read_html function to scrape HTML tables from the following website: https://www.baseball-reference.com/teams/ATL/2021.shtml
I can fulfil my needs using selenium/bs but want to see if I can scrape this site’s tables with just pd.read_html alone.
Currently, pd.read_html returns the first two tables, but is not able to access tables past the second table.
Here is an example of a table ‘id’ that I am trying to access: ‘the40man’
And my code, which returns ‘ValueError: No tables found’:
pd.read_html("https://www.baseball-reference.com/teams/ATL/2021.shtml", attrs = {'id': 'the40man'})The following code returns the first two tables, {‘id’: [‘team_batting’, ‘team_pitching’]}, but nothing more:
pd.read_html("https://www.baseball-reference.com/teams/ATL/2021.shtml")I am asking this question out of curiosity in case I’m missing something on my end. If not, this issue is likely due to pd.read_html’s limitations.
Thank you in advance for any input/pd.read_html tips!
Advertisement
Answer
The reference.com sites have some of those tables within the comments of the html. To pull those table out, you need to first pull out the comments. Then you can iterate through those to get the table you want:
import requestsfrom bs4 import BeautifulSoup, Commentimport pandas as pdurl = 'https://www.baseball-reference.com/teams/ATL/2021.shtml'result = requests.get(url).textdata = BeautifulSoup(result, 'html.parser')comments = data.find_all(string=lambda text: isinstance(text, Comment))tables = []for each in comments: if 'table' in str(each): try: tables.append(pd.read_html(str(each), attrs = {'id': 'the40man'})[0]) break except: continueOutput:
print(tables[0]) Rk Uni Name Unnamed: 3 Ht Wt DoB 1stYr0 1 30 Kyle Wright us US 6' 4" 215 Oct 2, 1995 20151 2 0 William Woods us US 6' 3" 190 Dec 29, 1998 20182 3 51 Will Smith us US 6' 5" 255 Jul 10, 1989 20083 4 68 Tyler Matzek us US 6' 3" 230 Oct 19, 1990 20104 5 64 Tucker Davidson us US 6' 2" 215 Mar 25, 1996 20165 6 62 Touki Toussaint us US 6' 3" 215 Jun 20, 1996 20146 7 65 Spencer Strider us US 6' 0" 195 Oct 28, 1998 20187 8 15 Sean Newcomb us US 6' 5" 255 Jun 12, 1993 20128 9 40 Mike Soroka ca CA 6' 5" 225 Aug 4, 1997 20159 10 54 Max Fried us US 6' 4" 190 Jan 18, 1994 201210 11 77 Luke Jackson us US 6' 2" 210 Aug 24, 1991 201111 12 33 A.J. Minter us US 6' 0" 215 Sep 2, 1993 201312 13 0 Kirby Yates us US 5' 10" 205 Mar 25, 1987 200913 14 0 Jay Jackson us US 6' 1" 195 Oct 27, 1987 200814 15 71 Jacob Webb us US 6' 2" 210 Aug 15, 1993 201415 16 19 Huascar Ynoa do DO 6' 2" 220 May 28, 1998 201516 17 36 Ian Anderson us US 6' 3" 170 May 2, 1998 201617 18 0 Freddy Tarnok us US 6' 3" 185 Nov 24, 1998 201718 19 74 Dylan Lee us US 6' 3" 214 Aug 1, 1994 201519 20 0 Alan Rangel mx MX 6' 2" 170 Aug 21, 1997 201520 21 0 Brooks Wilson us US 6' 2" 205 Mar 15, 1996 201521 22 50 Charlie Morton us US 6' 5" 215 Nov 12, 1983 200222 23 14 Adam Duvall us US 6' 1" 215 Sep 4, 1988 201023 24 24 William Contreras ve VE 6' 0" 180 Dec 24, 1997 201524 25 27 Austin Riley us US 6' 3" 240 Apr 2, 1997 201525 26 16 Travis d'Arnaud us US ... 6' 2" 210 Feb 10, 1989 200726 27 0 Travis Demeritte us US 6' 0" 180 Sep 30, 1994 201327 28 0 Chadwick Tromp aw AW 5' 8" 221 Mar 21, 1995 201328 29 25 Cristian Pache do DO 6' 2" 215 Nov 19, 1998 201629 30 13 Ronald Acuna Jr. ve VE 6' 0" 205 Dec 18, 1997 201530 31 1 Ozzie Albies cw CW 5' 8" 165 Jan 7, 1997 201431 32 9 Orlando Arcia ve VE 6' 0" 187 Aug 4, 1994 201132 33 7 Dansby Swanson us US 6' 1" 190 Feb 11, 1994 201333 34 0 Drew Waters us US 6' 2" 185 Dec 30, 1998 201734 35 20 Marcell Ozuna do DO 6' 1" 225 Nov 12, 1990 200835 36 0 Manny Pina ve VE 6' 0" 222 Jun 5, 1987 200536 37 38 Guillermo Heredia cu CU 5' 10" 195 Jan 31, 1991 200937 38 66 Kyle Muller us US 6' 7" 250 Oct 7, 1997 201638 Rk Uni Name NaN Ht Wt DoB 1stYr[39 rows x 14 columns]