I am using Python-requests to pull data from a website. I am doing this currently :
params = {'A':'something', 'B':'something'}
response = requests.get(url, params = params)
which gives me: https://someurl?query=&A=something&B=something
This is all perfectly fine and great.
However, the website doesn’t accept my API call. After some meddling around, I discovered that my target url is actually this:
https://someurl?query=%26A=something%26B=something
Hence my question : Is there a workaround for this problem? I have combed through requests’ documentation and found nothing. I really don’t feel like working with the url directly because I really like Python-requests.
Advertisement
Answer
The URL https://someurl?query=&A=something&B=something is very different than the URL https://someurl?query=%26A=something%26B=something.
URL1 https://someurl?query=&A=something&B=something is interpreted by the HTTP server as a request with 3 parameters: {
“query”: “”,
“A”: “something”,
“B”: “something”
}
URL2 https://someurl?query=%26A=something%26B=something is interpreted by the HTTP server as a request with 1 parameter: {
“query”: “&A=something%26B=something”
}
where “%26” is decoded as “&” character so value is decoded as &A=something&B=something.
A HTTP query with a single parameter “query” with value of &A=something&B=something needs to be properly encoded otherwise it will be converted into the wrong value. If using params options in requests API then the encoding is automatically done for you.
url = "http://localhost:8000"
params = {'query': '&A=something&B=something'}
response = requests.get(url, params=params)
print(response.status_code)
If you want to debug requests under the covers then add this before calling requests.get().
import requests
import logging
# You must initialize logging, otherwise you'll not see debug output.
logging.basicConfig()
logging.getLogger().setLevel(logging.DEBUG)
requests_log = logging.getLogger("requests.packages.urllib3")
requests_log.setLevel(logging.DEBUG)
url = "http://localhost:8000"
params = {'query': '&A=something&B=something'}
response = requests.get(url, params=params)
Output:
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): localhost:8000 DEBUG:urllib3.connectionpool:http://localhost:8000 "GET /?query=%26A%3Dsomething%26B%3Dsomething HTTP/1.1" 200 5
Notice the “=” in the URL is also enocded to avoid any confusion since “&” and “=” are special characters in URL string.