AIOHTTP replacing %3A with :

import yarl
async with cs.get(yarl.URL(f"https://ipqualityscore.com/api/json/url/{self.token}/{url}",encoded=True)) as r:

Hello, i’m having this issue where AIOHTTP is converting characters like %3A to the original :. i need to use the %3A version in the API req, if not, it raises 404

My code:

for link in results:
    url = urllib.parse.quote(link, safe = '')
    print(url)
    ## ^^ 1st ^^

    async with aiohttp.ClientSession() as cs:

        print(f"https://ipqualityscore.com/api/json/url/{self.token}/{url}")
        ## ^^ 2nd ^^

        async with cs.get(f"https://ipqualityscore.com/api/json/url/{self.token}/{url}") as r:
            text = await r.json()
            print(text)

URL it should’ve used:

https://ipqualityscore.com/api/json/url/PRIVATE_TOKEN/https%3A%2F%2Fstreancommunuty.ru%2Ftradoffer%2Fnew%2F%3Fpartner%3D1284276379%26token%3DiMDdLkoe

error raised (and url used):

aiohttp.client_exceptions.ContentTypeError: 0, message='Attempt to decode JSON with unexpected mimetype: text/html; charset=utf-8', url=URL('https://ipqualityscore.com/api/json/url/PRIVATE_TOKEN/https:%2F%2Fstreancommunuty.ru%2Ftradoffer%2Fnew%2F%3Fpartner=1284276379&token=iMDdLkoe')

Answer

First of all, are you sure this is what you want to do? I ask because while : is a reserved character in URLs, it is not used as a delimiter in the path component of a URL, and so whether or not it is percent-encoded it should mean the exact same thing to the web server. Are you certain that whether the : is percent-encoded is the only thing causing your problem? That said, it’s possible this particular web server isn’t following the RFC properly, in which case it might be you need to work around it.

If it is what you want to do, I think you need to prevent aiohttp from normalizing the URL. From the answer to that question, it sounds like you could do something like this:

import yarl

...

ipqs_url = yarl.URL(
    f"https://ipqualityscore.com/api/json/url/{self.token}/{url}",
    encoded=True)
await ctx.send(ipqs_url)

Similarly, you can pass a yarl.URL object to cs.get.