My goal is to extract all of the urls and add a get request to each ndjson file; however, this can be complicated when there are more 10 urls. Is there a better way to do this or do I need to put multiple GET requests in and then join the ndjson files and then parse the data.
JavaScript
x
2
1
print(response.text)
2
Output:
JavaScript
1
17
17
1
{"transactionTime":"2022-03-27T08:51:32.174-04:00","request":"https://api.site/data/5555/$export","requiresAccessToken":true,"output": [
2
{
3
"type":"robot",
4
"url":"https://api.site/data/5555/838916.ndjson"
5
},
6
{
7
"type":"robot",
8
"url":"https://api.site/data/5555/838917.ndjson"
9
},
10
{
11
"type":"robot",
12
"url":"https://api.site/data/5555/838918.ndjson"
13
}
14
]
15
"error":[],"JobID":12443}
16
17
JavaScript
1
2
1
list(response.text.values())
2
Output:
JavaScript
1
19
19
1
[
2
"1990-01-28T08:51:32.174-04:00",
3
"https://api.site/data/5555/$export",
4
true,
5
[
6
{
7
"type":"robot",
8
"url":"https://api.site/data/5555/838916.ndjson"
9
},
10
{
11
"type":"robot",
12
"url":"https://api.site/data/5555/838917.ndjson"
13
},
14
{
15
"type":"robot",
16
"url":"https://api.site/data/5555/838918.ndjson"
17
}
18
]
19
I currently add multiple GET requests here:
JavaScript
1
4
1
response1 = requests.get("https://api.site/data/5555/838916.ndjson",headers=headers)
2
response2 = requests.get("https://api.site/data/5555/838917.ndjson",headers=headers)
3
response3 = requests.get("https://api.site/data/5555/838918.ndjson",headers=headers)
4
Advertisement
Answer
If I understood your question correctly, you send some request which returns you provided JSON object. You need to send requests to every url from this object and merge data into a single container (e.g. dict
).
JavaScript
1
12
12
1
from requests import Session
2
3
headers = { } # some headers
4
5
sess = Session()
6
sess.headers.update(headers)
7
8
resp = sess.get("https://api.site/data/5555/$export")
9
for item in resp.json()["output"]:
10
ndjson = sess.get(item["url"])
11
# here some code to process ndjson.text
12
Normally ndjson is a list of JSON objects separated by newline char, so without actual data it’s not possible to help with code which will store this data in proper (for future parsing) format.