How can I read exactly one response chunk with python’s http.client?

Using http.client in Python 3.3+ (or any other builtin python HTTP client library), how can I read a chunked HTTP response exactly one HTTP chunk at a time?

I’m extending an existing test fixture (written in python using http.client) for a server which writes its response using HTTP’s chunked transfer encoding. For the sake of simplicity, let’s say that I’d like to be able to print a message whenever an HTTP chunk is received by the client.

My code follows a fairly standard pattern for reading a large response:

conn = http.client.HTTPConnection(...)
conn.request(...)
response = conn.getresponse()

resbody = []

while True:
    chunk = response.read(1024)
    if len(chunk):
        resbody.append(chunk)
    else:
        break

conn.close();

But this reads 1024 byte chunks regardless of whether or not the server is sending 10 byte chunks or 10MiB chunks.

What I’m looking for would be something like the following:

while True:
    chunk = response.readchunk()
    if len(chunk):
        resbody.append(chunk)
    else
        break

If this is not possible with http.client, is it possible with another builtin http client library? If it’s not possible with a builtin client lib, is it possible with pip installable module?

Answer

I found it easier to use the requests library like so

r = requests.post(url, data=foo, headers=bar, stream=True)

for chunk in (r.raw.read_chunked()):
    print(chunk)

Advertisement

Answer