I am trying to do some scraping from websites using GET and POST methods, but now I am facing a new challenge.
I am trying to get data from a credit simulator, I found this portuguese site (https://www.bancomontepio.pt/particulares/credito/pessoal/credito-pessoal-online).
As far as I know, I need to use POST method, but I have to specify the data (the Amount value, the Term…). I usually do it by creating a dictionary structure but that is not working.
I’m kinda lost to be fair, maybe the problem is on the header…
Here is my code:
JavaScript
x
17
17
1
import requests2
3
import warnings4
warnings.filterwarnings("ignore")5
6
term=247
amount=50008
9
url = 'https://simuladores.bancomontepio.pt/ITSCredit.External/Calculator/ITSCredit.Calculator.UI.External/gateway/Calculator/api/Calculator/Calculate?hash=-1359629931'10
11
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.122 Safari/537.36',12
'Accept-Language': 'pt-PT,pt;q=0.9,en-US;q=0.8,en;q=0.7'}13
14
payload = {'Amount': amount,'Term': term,'ProductCode':"26B1129900X"}15
16
response = requests.post(url, headers=headers, data=payload, verify=False).json()17
If i take off the .json(), I get the error Response [410].
The goal is to get the TAN or TAEG that change when term (“Prazo”) or amount (“Montante”) values change.
Any ideias?
[EDIT]
JavaScript
1
35
35
1
headers = {'Accept': 'application/json, text/plain, */*' ,2
'Accept-Encoding': 'gzip, deflate, br',3
'Accept-Language': 'pt-PT,pt;q=0.9,en-US;q=0.8,en;q=0.7',4
'Connection': 'keep-alive',5
'Content-Length': '957',6
'content-type': 'text/plain',7
'Cookie': '_gcl_au=1.1.911606195.1646064658; OptanonAlertBoxClosed=2022-02-28T16:45:11.586Z; _ga=GA1.2.1147601977.1646064657; _ga_8WVEJF7X11=GS1.1.1646305654.3.1.1646309750.0; _ga_63QCVBV1V3=GS1.1.1646305679.1.1.1646309750.0; ASP.NET_SessionId=wlfbf2dx4oatlio0vl1ftinq; _gid=GA1.2.449121330.1646650093; calc-cookie=; OptanonConsent=isGpcEnabled=0&datestamp=Mon+Mar+07+2022+11%3A38%3A48+GMT%2B0000+(Hora+padr%C3%A3o+da+Europa+Ocidental)&version=6.30.0&isIABGlobal=false&consentId=6caccc97-6af1-4b55-9049-5694835d9f7a&interactionCount=2&landingPath=NotLandingPage&groups=C0001%3A1%2CC0002%3A1%2CC0003%3A1%2CC0004%3A1&hosts=H10%3A1%2CH20%3A1%2CH7%3A1%2CH8%3A1%2CH23%3A1%2CH11%3A1%2CH24%3A1%2CH13%3A1%2CH25%3A1&genVendors=&geolocation=ES%3B&AwaitingReconsent=false; _gali=slider-container; _gat_UA-186811106-6=1',8
'Host': 'simuladores.bancomontepio.pt',9
'Origin': 'https://simuladores.bancomontepio.pt',10
'Referer': 'https://simuladores.bancomontepio.pt/',11
'sec-ch-ua': '" Not A;Brand";v="99", "Chromium";v="90", "Google Chrome";v="90"',12
'sec-ch-ua-mobile': '?0',13
'Sec-Fetch-Dest': 'empty',14
'Sec-Fetch-Mode': 'cors',15
'Sec-Fetch-Site': 'same-origin',16
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36'}headers = {'Accept': 'application/json, text/plain, */*' ,17
'Accept-Encoding': 'gzip, deflate, br',18
'Accept-Language': 'pt-PT,pt;q=0.9,en-US;q=0.8,en;q=0.7',19
'Connection': 'keep-alive',20
'Content-Length': '957',21
'content-type': 'text/plain',22
'Cookie': '_gcl_au=1.1.911606195.1646064658; OptanonAlertBoxClosed=2022-02-28T16:45:11.586Z; _ga=GA1.2.1147601977.1646064657; _ga_8WVEJF7X11=GS1.1.1646305654.3.1.1646309750.0; _ga_63QCVBV1V3=GS1.1.1646305679.1.1.1646309750.0; ASP.NET_SessionId=wlfbf2dx4oatlio0vl1ftinq; _gid=GA1.2.449121330.1646650093; calc-cookie=; OptanonConsent=isGpcEnabled=0&datestamp=Mon+Mar+07+2022+11%3A38%3A48+GMT%2B0000+(Hora+padr%C3%A3o+da+Europa+Ocidental)&version=6.30.0&isIABGlobal=false&consentId=6caccc97-6af1-4b55-9049-5694835d9f7a&interactionCount=2&landingPath=NotLandingPage&groups=C0001%3A1%2CC0002%3A1%2CC0003%3A1%2CC0004%3A1&hosts=H10%3A1%2CH20%3A1%2CH7%3A1%2CH8%3A1%2CH23%3A1%2CH11%3A1%2CH24%3A1%2CH13%3A1%2CH25%3A1&genVendors=&geolocation=ES%3B&AwaitingReconsent=false; _gali=slider-container; _gat_UA-186811106-6=1',23
'Host': 'simuladores.bancomontepio.pt',24
'Origin': 'https://simuladores.bancomontepio.pt',25
'Referer': 'https://simuladores.bancomontepio.pt/',26
'sec-ch-ua': '" Not A;Brand";v="99", "Chromium";v="90", "Google Chrome";v="90"',27
'sec-ch-ua-mobile': '?0',28
'Sec-Fetch-Dest': 'empty',29
'Sec-Fetch-Mode': 'cors',30
'Sec-Fetch-Site': 'same-origin',31
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36'}32
33
payload = {'CCRDCalculateInput':{},'MultifunctionsCalculateInput':{},'Device':{'Browser':'chrome','BrowserVersion':'90.0.4430.212','Device':'Desktop','Os':'windows','OsVersion':'windows-10','UserAgent':'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36'},'IsCustomer':'true','Amount':7500,'Term':60,'ConditionCode':'26B1129900X-01-I-129900-F','CreditDestinationCode':'129900','ProductCode':'26B1129900X','FinancedExpenses':'false','FrequencyTrancheCode':'null','GoalCode':'C006','GoalDescription':'PROJETOS PESSOAIS','FrequencyTypeCode':'M','FamilyCode':'CP','Proponents':[{'Position':1,34
'Birthday':'1992-03-07T13:03:30.000Z','State':'true','EntityType':{'ID':1,'CompanyID':1,'Code':'P','Description':'Proponente','Value':'null','ValueString':'null','State':'true','Imported':'null'},'ExpenseCodes':['009']}],'Counterparts':0,'OptionalExpenses':[{'Code':'009','Factor':1}],'ResidualValue':0,'InterestOnly':0,'Deferment':0}35
Now I’m getting a empty json()… Response 200 but I got this structure:
JavaScript
1
4
1
{'Status': 'Unknown',2
'Error': {'VisibleToHuman': False, 'Code': '0', 'Message': ''},3
'Result': None}4
As far as I know, the status should be “OK” to get some info on the Result.
Cheers
Advertisement
Answer
Looks like you need to expand the payload to include more (all) of the parameters (including the cookies, specifically the ASP.NET_SessionId).
JavaScript
1
71
71
1
import requests2
3
import warnings4
warnings.filterwarnings("ignore")5
6
term=247
amount=50008
9
url = 'https://simuladores.bancomontepio.pt/ITSCredit.External/Calculator/ITSCredit.Calculator.UI.External/gateway/Calculator/api/Calculator/Calculate?hash=-1359629931'10
11
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.122 Safari/537.36',12
'Accept-Language': 'pt-PT,pt;q=0.9,en-US;q=0.8,en;q=0.7',13
'Cookie':'ASP.NET_SessionId=fhkyn1vn5knlw3uhdnh50nii;'}14
15
16
payload = {17
"CCRDCalculateInput":{},18
"MultifunctionsCalculateInput":{},19
"Device":{20
"Browser":"chrome",21
"BrowserVersion":"96.0.4664.110",22
"Device":"Desktop",23
"Os":"windows",24
"OsVersion":"windows-10",25
"UserAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"},26
"IsCustomer":'true',27
"Amount":amount,28
"Term":term,29
"ConditionCode":"26B1129900X-01-I-129900-F",30
"CreditDestinationCode":"129900",31
"ProductCode":"26B1129900X",32
"FinancedExpenses":'false',33
"FrequencyTrancheCode":'null',34
"GoalCode":"C006",35
"GoalDescription":"PROJETOS PESSOAIS",36
"FrequencyTypeCode":"M",37
"FamilyCode":"CP",38
"Proponents":[{39
"Position":'1',40
"Birthday":"1992-03-10T13:03:24.000Z",41
"State":'true',42
"EntityType":{43
"ID":'1',44
"CompanyID":'1',45
"Code":"P",46
"Description":"Proponente",47
"Value":'null',48
"ValueString":'null',49
"State":'true',50
"Imported":'null'},51
"ExpenseCodes":[52
"009"]}],53
"Counterparts":'0',54
"OptionalExpenses":[{55
"Code":"009",56
"Factor":'1'}],57
"ResidualValue":'0',58
"InterestOnly":'0',59
"Deferment":'0'}60
61
jsonData = requests.post(url, headers=headers, json=payload, verify=False).json()62
results = jsonData['Result']63
64
mtic = results['MTIC']65
installment = results['PeriodInstallment'][0]['Installment']66
taeg = results['TAEG']67
tan = results['PeriodInstallment'][0]['TAN']68
69
70
print(f'Installment: {installment}nTAEG: {taeg}nTAN: {tan}nMTIC: {mtic}')71
Output:
JavaScript
1
5
1
Installment: 224.52
TAEG: 14.83
TAN: 7.04
MTIC: 5708.25