I have code like this:
import requests import multiprocessing as mp import json import time BASE_URL = 'http://127.0.0.1:3000/employees' with open('data.json', 'r') as f: array = json.load(f) def internet_resource_getter(post_data): stuff_got = [] response = requests.post(BASE_URL, json=post_data) stuff_got.append(response.json()) print(stuff_got) time.sleep(1) return stuff_got if __name__ == '__main__': # freeze_support() here if program needs to be frozen start=time.time() with mp.Pool(mp.cpu_count()) as pool: pool.map(internet_resource_getter, array) elapsed = (time.time() - start) print("n","time elapsed is :", elapsed)
in file data.json contains 500 records, for example:
[{"first_name":"John","last_name":"Swen"},{"first_name":"Ricard","last_name":"Candra"}]
in BASE_URL there is data like this:
[ { "id": 1, "first_name": "Sebastian", "last_name": "Eschweiler" }, { "id": 2, "first_name": "Steve", "last_name": "Palmer" }, { "id": 3, "first_name": "Ann", "last_name": "Smith" } ]
expected output after POST API:
[ { "id": 1, "first_name": "Sebastian", "last_name": "Eschweiler" }, { "id": 2, "first_name": "Steve", "last_name": "Palmer" }, { "id": 3, "first_name": "Ann", "last_name": "Smith" }, { "id": 4, "first_name": "John", "last_name": "Swen" }, { "id": 5, "first_name": "Ricard", "last_name": "Candra" }, ]
with my code above, the data that enters the url is only 420 records, even though my data.json is 500 records. how do I solve this so that I post 500 records to url. I don’t know why only 400 data are processed. I have an error like this:
multiprocessing.pool.RemoteTraceback: """ Traceback (most recent call last): File "C:UsersadminAppDataLocalProgramsPythonPython38libsite-packagesurllib3connectionpool.py", line 699, in urlopen httplib_response = self._make_request( File "C:UsersadminAppDataLocalProgramsPythonPython38libsite-packagesurllib3connectionpool.py", line 445, in _make_request six.raise_from(e, None) File "<string>", line 3, in raise_from File "C:UsersadminAppDataLocalProgramsPythonPython38libsite-packagesurllib3connectionpool.py", line 440, in _make_request httplib_response = conn.getresponse() File "C:UsersadminAppDataLocalProgramsPythonPython38libhttpclient.py", line 1347, in getresponse response.begin() File "C:UsersadminAppDataLocalProgramsPythonPython38libhttpclient.py", line 307, in begin version, status, reason = self._read_status() File "C:UsersadminAppDataLocalProgramsPythonPython38libhttpclient.py", line 276, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response During handling of the above exception, another exception occurred: Traceback (most recent call last): File "C:UsersadminAppDataLocalProgramsPythonPython38libsite-packagesrequestsadapters.py", line 439, in send resp = conn.urlopen( File "C:UsersadminAppDataLocalProgramsPythonPython38libsite-packagesurllib3connectionpool.py", line 755, in urlopen retries = retries.increment( File "C:UsersadminAppDataLocalProgramsPythonPython38libsite-packagesurllib3utilretry.py", line 531, in increment raise six.reraise(type(error), error, _stacktrace) File "C:UsersadminAppDataLocalProgramsPythonPython38libsite-packagesurllib3packagessix.py", line 734, in reraise raise value.with_traceback(tb) File "C:UsersadminAppDataLocalProgramsPythonPython38libsite-packagesurllib3connectionpool.py", line 699, in urlopen httplib_response = self._make_request( File "C:UsersadminAppDataLocalProgramsPythonPython38libsite-packagesurllib3connectionpool.py", line 445, in _make_request six.raise_from(e, None) File "<string>", line 3, in raise_from File "C:UsersadminAppDataLocalProgramsPythonPython38libsite-packagesurllib3connectionpool.py", line 440, in _make_request httplib_response = conn.getresponse() File "C:UsersadminAppDataLocalProgramsPythonPython38libhttpclient.py", line 1347, in getresponse response.begin() File "C:UsersadminAppDataLocalProgramsPythonPython38libhttpclient.py", line 307, in begin version, status, reason = self._read_status() File "C:UsersadminAppDataLocalProgramsPythonPython38libhttpclient.py", line 276, in _read_status raise RemoteDisconnected("Remote end closed connection without" urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) The above exception was the direct cause of the following exception: Traceback (most recent call last): File "paralel_pro.py", line 28, in <module> pool.map(internet_resource_getter, array) File "C:UsersadminAppDataLocalProgramsPythonPython38libmultiprocessingpool.py", line 364, in map return self._map_async(func, iterable, mapstar, chunksize).get() File "C:UsersadminAppDataLocalProgramsPythonPython38libmultiprocessingpool.py", line 771, in get raise self._value requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
Advertisement
Answer
To get your answer, check your server logs. I suspect bad data and/or server not handling unexpected data gracefully.
Client-Side Check
See if the 80 records are always the same 80 records.
Server-Side Check
Can you find the logs for your server that is running on http://127.0.0.1:3000/employees
? It looks like it is running on your same machine. The server’s application or service logs should have the smoking gun.