How to scrape a website table that you cannot select the text using python? [closed]

Hello all I am looking to scrape the leaderboard table similar to the one @chitown88 did in Having Trouble Clicking in Date Field with Selenium

for the date 02/13/2022.

Answer

It’s just a matter of finding the right url and then parsing it:

import requests
import pandas as pd

url = 'https://service.fantasylabs.com/live-contests/?sport=NFL&contest_group_id=71194'
headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36'}
jsonData = requests.get(url, headers=headers).json()

contestIds = {}
for each in jsonData['live_contests']:
    contestIds[each['contest_name']] = each['contest_id']

rows = []
for contestName, contestId in contestIds.items():
    url = f'https://dh5nxc6yx3kwy.cloudfront.net/contests/nfl/20220213/{contestId}/lineups/'
    jsonData = requests.get(url, headers=headers).json()

    lineups = jsonData['lineups']
    for k, v in lineups.items():
        v.update({'contestName':contestName})
        rows.append(v)
    print(contestName)
        
df = pd.DataFrame(rows)

Output:

print(df.head().to_string())
                         lineupHash  lineupCt  lineupUserCt                                                                                 lineupPlayers  points  totalSalary  totalOwn  minOwn  maxOwn  avgOwn  lineupRank  isCashing  favoriteCt  underdogCt  homeCt  visitorCt  payout  lineupPercentile  correlatedPlayers                                                                          teamStacks                                            gameStacks                                                                                                                                                                                                                                                                                                                                lineupTrends                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           entryNameList                                                            contestName
0   6957:33786:1343:8491:8469:47456         1             1   {'CPT1': 6957, 'FLEX1': 33786, 'FLEX2': 1343, 'FLEX3': 8491, 'FLEX4': 8469, 'FLEX5': 47456}   97.98        44400    172.73     0.0   74.33   28.79      199517      False           5           1       0          0     0.0             42.51                  6                      {'2203': ['6957:1', '33786:0', '1343:0', '8491:0', '47456:0']}   {'5229230': [6957, 33786, 1343, 8491, 8469, 47456]}   {'qbPairedWithPassCatcher': True, 'qbStackPairedWithOpponent': False, 'rbPairedWithDefense': True, 'noPlayersAgainstDEF': False, 'maxOneRBPerGame': True, 'minOnePlayerWithLowOwnership': True, 'cptQBPairedWithMultiplePassCatchers': True, 'cptPassCatcherPairedWithQB': False, 'maxOneDEFPerGame': False, 'maxOneKickerPerGame': True}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              [Wendy318]  NFL Showdown $6M Super Bowl LVI Millionaire [$1M to 1st] (Super Bowl)
1  8491:33786:6957:64586:1343:47456       143           140  {'CPT1': 8491, 'FLEX1': 33786, 'FLEX2': 6957, 'FLEX3': 64586, 'FLEX4': 1343, 'FLEX5': 47456}   99.32        47900    230.82     0.0   74.33   38.47      185392      False           5           1       0          0     0.0             39.50                  6                      {'2203': ['8491:1', '33786:0', '6957:0', '1343:0', '47456:0']}  {'5229230': [8491, 33786, 6957, 64586, 1343, 47456]}    {'qbPairedWithPassCatcher': True, 'qbStackPairedWithOpponent': True, 'rbPairedWithDefense': True, 'noPlayersAgainstDEF': False, 'maxOneRBPerGame': True, 'minOnePlayerWithLowOwnership': True, 'cptQBPairedWithMultiplePassCatchers': False, 'cptPassCatcherPairedWithQB': False, 'maxOneDEFPerGame': True, 'maxOneKickerPerGame': True}                                                                            [dman16384, weezywe34, tradinfool, Ellio31, AshleeLing, thop78, yabo22, bmo115, tadpole453, jeffthemoron, parlaygods, deathgrip, KingDeer, vassassin, miamimariok, CoolTreez, MrWentz, tkeating11, DUB5588, bkat26, tankitytank, TheRooneyRule, BenSeligman, viktor122000, astros210, BATMAN1973, bastavros, tj620, dbuermann, briand83, magic253792, bobboman11, arbitrage88, tdvrbas, p0l4r21, Kbosh1989, dfecko12, beavea, invsbltorsoman, Ponta01, Francis1959, joseph3322, teeeeej, Eduardo17Gonzalez20, mme9482, 12strikesguy, Sturdy36, jhender, Mogag, MDTigers8, BobbyJ33, gabsdad2003, funfootball, mattblack70, jberry0461, ajhall03, plumbnasty361, bfahey1989, Benriggins86, cjcook017, losski1976, spider710, boosted23, ladodger13, zdub2344, furey17, TC2480, elctrichomes87, EuBaAsta, JeffHampe, lucascw10, hoffy555, jkaelin, williec7, Latrinadraft, MTtires, UncleBanjo, joejoe09, grrrrmonsta, jddez64, Tee247, tjw2004, yun.seungdo, KartoshkaIstanbul, exmo, insane1ac, bruins06, Gatorsjj40, JRCLCC, cmitc053, dmask9, Butts11, brielly1972, csh9022, lewishans1213, Troyalty14, Jaystar4lyfe, GigelB05, clayton_521, Jjluker, ...]  NFL Showdown $6M Super Bowl LVI Millionaire [$1M to 1st] (Super Bowl)
2   8491:33786:6957:64586:1343:8469       494           465   {'CPT1': 8491, 'FLEX1': 33786, 'FLEX2': 6957, 'FLEX3': 64586, 'FLEX4': 1343, 'FLEX5': 8469}   97.32        49500    226.04     0.0   74.33   37.67      208970      False           4           2       0          0     0.0             44.52                  6  {'2203': ['8491:1', '33786:0', '6957:0', '1343:0'], '2141': ['64586:0', '8469:0']}   {'5229230': [8491, 33786, 6957, 64586, 1343, 8469]}  {'qbPairedWithPassCatcher': True, 'qbStackPairedWithOpponent': True, 'rbPairedWithDefense': False, 'noPlayersAgainstDEF': False, 'maxOneRBPerGame': True, 'minOnePlayerWithLowOwnership': True, 'cptQBPairedWithMultiplePassCatchers': False, 'cptPassCatcherPairedWithQB': False, 'maxOneDEFPerGame': False, 'maxOneKickerPerGame': True}  [cinodolla5, drew_streckenbach, jfin711, HappyHollow74, EuBaAsta, Gconstance88, kinvest0314, Coltron, bigbuck1995, PaulusCarolus, dredre69, jsmooth816, andrewkole, Mandawg7676, RedskinsUNCfan, IcemanBK, moeryp, benchristine1, tenthletta, gtully404, lpmurphy, TxFantasyStud, FOMOSummer2, big_ugh, jdmcconnell75, AlmightyZoe, medlet01, pjs1974, jrl2002, kdevon09, Tuckerjr007, Wentz2Ertz, CarterBoys2006, Chunkytwilight, heyo212, FalseProphetxX, brianr87, Charms03, ramster_329, milkchoc021, supe302, DraftSmasher15, jluvv30, lhutzler, gman240, KynRi, mcork1223, jval71, Pigbear, jack.ryan369, saffronsnake, lenster1988, pipes22, lboogystl, DimDarkness10, 2hot4u2, robfree28, Bigkat777, dabears2424, xGhostMachinex, Raekwon19, rickychurch, jmorreale3, Roll...Tide, warrior75, 1PerfectLineup, bronsonnemelka, 05Doran.0505, Score112, TimWinner, oakraidrluva26, TOMMYBBQ1, BigPawka, jeverh11, DTopps, BradChavez, jbean850, b.nason1224, miller1116, beekiewah18, vimvicious, mattyyoung1911, hauk1973, bradleycounts8, Suchislife, FdThree, MrFrodo444, leek44, hbfrank949, hammertime77, mon122690, dmilner100, mrmpls701, yellowpup60, MarcPerch, spop17, 7danan, Gman1972, fperanio, 00BallsDeep00, ...]  NFL Showdown $6M Super Bowl LVI Millionaire [$1M to 1st] (Super Bowl)
3  1343:33786:6957:64586:8491:54513        85            82  {'CPT1': 1343, 'FLEX1': 33786, 'FLEX2': 6957, 'FLEX3': 64586, 'FLEX4': 8491, 'FLEX5': 54513}   97.12        49400    205.11     0.0   74.33   34.18      211429      False           5           1       0          0     0.0             45.05                  6                      {'2203': ['1343:1', '33786:0', '6957:0', '8491:0', '54513:0']}  {'5229230': [1343, 33786, 6957, 64586, 8491, 54513]}    {'qbPairedWithPassCatcher': True, 'qbStackPairedWithOpponent': True, 'rbPairedWithDefense': False, 'noPlayersAgainstDEF': False, 'maxOneRBPerGame': True, 'minOnePlayerWithLowOwnership': True, 'cptQBPairedWithMultiplePassCatchers': False, 'cptPassCatcherPairedWithQB': True, 'maxOneDEFPerGame': True, 'maxOneKickerPerGame': True}                                                                                                                                                                                                                                                               [iriestylee, millameterpeter, ihoagied, randallcopp, wager_32, msam, Gthitter, lpmurphy, retselwolsac, unclezeke, kokomo2433, Skibasher, babytempr17, philney, RoarinBoren, michaelchite, acetx555, robertnyg, cutemike37, malip1133, IamApollo68, Sparkle01, jreth24, OperationWalrus, mtw13002, Fedgret9199, andrewtbell, S.Cornett, farmboyz, markwillig, rhouston13, danieltmv, TreyDay, bkenard, mmalczyk, MVanSlooten, cxmele, jujack, tnguyen123, rkess100, aanderson61, brknets23, ahitspat, BryxAlv, BowieOrMaze, MasterDrafter007, MooreP7, DynamicGOB, rabblerouser, missbeccax3, baseballfan37, LlamaSan, cevansb0, anddgrah, mcgintymc, Twebba71, piledriver40, AnointedMatt, JJHawk30, RickyRococo, Ddohnalek, sport81, losingplace, kencamnat, Bashman1234, bigmac34786, KingOzzie, ctompkin, puckhead18, prettyboi90, kccputech, loegs14, KPerry6, Jordan993, clibass1, clivebixby, steve4booze, rjbiz123, ztrivett21, dnice39, GoGators1509, sevendrums]  NFL Showdown $6M Super Bowl LVI Millionaire [$1M to 1st] (Super Bowl)
4  47456:33786:6957:64586:1343:8491        11            11  {'CPT1': 47456, 'FLEX1': 33786, 'FLEX2': 6957, 'FLEX3': 64586, 'FLEX4': 1343, 'FLEX5': 8491}   99.32        47000    243.65     0.0   74.33   40.61      185535      False           5           1       0          0     0.0             39.53                  6                      {'2203': ['47456:1', '33786:0', '6957:0', '1343:0', '8491:0']}  {'5229230': [47456, 33786, 6957, 64586, 1343, 8491]}    {'qbPairedWithPassCatcher': True, 'qbStackPairedWithOpponent': True, 'rbPairedWithDefense': True, 'noPlayersAgainstDEF': False, 'maxOneRBPerGame': True, 'minOnePlayerWithLowOwnership': True, 'cptQBPairedWithMultiplePassCatchers': False, 'cptPassCatcherPairedWithQB': False, 'maxOneDEFPerGame': True, 'maxOneKickerPerGame': True}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             [Hardfour, p0l4r21, twonine29, jaks13, boltscrazy, SkillGame84, Omriking, sapplesauce55, gleetghost77, sspencer06, jozzy13]  NFL Showdown $6M Super Bowl LVI Millionaire [$1M to 1st] (Super Bowl)
....

[87031 rows x 24 columns]