How to scrape dataframe after select options from dropdown list?

I want to scrape dataframe from dropdow value with BeautifulSoup.

1. I select the value in both dropdown
2. I submit my selection
3. I get a data table

I would like to catch this dataframe with BS. any idea of the process to achieve this?

example site: https://coinarbitragebot.com/arbitrage.php

thanks

Answer

You can issue simple POST requests with custom parameters (the parameters you will see in Firefox/Chrome network tab when click Submit button). Then you can use pandas.read_html() function to get your DataFrame.

For example:

data = {'bibox':   1, 'biki':   1, 'binance':1, 'bit-z':  1, 'bitbns': 1, 'bitfinex':   1, 'bitforex':   1, 'bithumb':1,
'bitkub': 1, 'bitmart':1, 'bitmax': 1, 'bitrue': 1, 'bitso':  1, 'bitstamp':  1, 'bittrex':1, 'bleutrade':  1, 'btcturk':1,
'bw_com':1, 'catex':  1, 'cex_io': 1, 'coinall':1, 'coinbase':   1, 'coinbene':  1, 'coincheck':  1, 'coindeal':   1, 'coineal':1,
'coinsbit':   1, 'cointiger':  1, 'crex24': 1, 'dcoin':  1, 'digifinex':  1, 'exmo':   1, 'exx_com':1, 'fatbtc': 1, 'finexbox':   1,
'gate_io':1, 'graviex':1, 'hitbtc': 1, 'hotbit':1, 'huobi':  1, 'indodax':1, 'koineks':1, 'kraken': 1, 'kucoin': 1, 'latoken':1,
'lbank':  1, 'liquid': 1, 'livecoin':   1, 'mercatox':   1, 'mxc':1, 'okcoin': 1, 'okex':   1, 'p2pb2b': 1, 'poloniex':   1, 'simex':  1,
'sistemkoin': 1, 'stex':   1, 'tokok':  1, 'tradeogre':  1, 'tradesatoshi':   1, 'upbit':  1, 'yobit':  1, 'zb_com': 1, 'zbg':1,
'bcurr': 'usd', 'arb_margin':  25, 'sbmfrm':  1}

import requests
import pandas as pd
from bs4 import BeautifulSoup

url = 'https://coinarbitragebot.com/arbitrage.php'

data['bcurr'] = 'usd'       # <-- set to 'usd', 'btc' or 'all'
data['arb_margin'] = 5      # <-- set to your value

soup = BeautifulSoup( requests.post(url, data=data).text, 'html.parser' )

df = pd.read_html(str(soup.select_one('#tbl1')))[0]
df.columns = df.loc[0]
df = df.iloc[1:].set_index('Coin', drop=True)

print(df)

Prints:

0            bibox      biki  binance     bit-z  bitbns  bitfinex    bitforex    bithumb      bitkub  bitmart  ... sistemkoin stex  tokok tradeogre tradesatoshi upbit yobit     zb.com        zbg      %
Coin                                                                                                           ...                                                                                       
DOGE/USD  0.002102  0.002102   0.0021  0.002097       0         0  0.00209838          0  0.00205862        0  ...   0.002178    0      0         0            0     0     0  0.0021027  0.0021013  29.34
TRX/USD          0  0.014055  0.01408   0.01409       0  0.013905      0.0141  0.0137128           0  0.01408  ...   0.014512    0  0.014         0            0     0     0    0.01406     0.0145   7.63
XLM/USD          0         0  0.04733     0.047  0.0472   0.04724      0.0472  0.0460763   0.0471012  0.04733  ...   0.047811    0      0         0            0     0     0     0.0473     0.0475   5.08
BSV/USD          0   113.299        0         0       0    113.27     113.457    110.545     108.698  113.638  ...     113.69    0      0         0            0     0     0    112.172     113.48   5.89
NEO/USD      9.484      9.45    9.483    9.4823   9.386    9.4783        9.49          0           0    9.483  ...       9.91    0  9.483         0            0     0     0     9.4925          0   5.51
...            ...       ...      ...       ...     ...       ...         ...        ...         ...      ...  ...        ...  ...    ...       ...          ...   ...   ...        ...        ...    ...
PCX/USD          0         0        0         0       0         0           0          0           0        0  ...          0    0      0         0            0     0     0          0          0   6.86
QCX/USD          0         0        0         0       0         0           0          0           0        0  ...          0    0      0         0            0     0     0          0          0  24.54
XDCE/USD         0         0        0         0       0         0           0          0           0        0  ...          0    0      0         0            0     0     0          0          0   6.84
YAS/USD          0         0        0         0       0         0           0          0           0        0  ...          0    0      0         0            0     0     0          0          0  14.72
ZEL/USD          0         0        0         0       0         0           0          0           0        0  ...          0    0      0         0            0     0     0          0          0   9.93

[73 rows x 65 columns]

EDIT:

To select only binance, bitfinex and bittrex, you can set data like this:

data = {'binance':1, 'bitfinex':   1, 'bittrex':1, 'bcurr': 'all', 'arb_margin': 5, 'sbmfrm':  1}

This will print:

0          binance bitfinex     bittrex     %
Coin                                         
SC/BTC  0.00000018        0  0.00000017  5.56

If no arbitrage opportunity is found, no table is found (you will need to handle this case too probably).