I am using Django server to call a Korean government’s weather API to retrieve weather data for around 1800 locations.
However, this weather API results in time out most of the time. I tried giving resting periods. For example, for every request, it will sleep for 0.8 seconds and after every 30 requests, it will sleep for 30 seconds. But this does not really work well for 1800 requests. At one point, it was successful. All other times, it ended up with fail.
In this kind of situation, what else should I do to make sure that 1800 requests are completed within one hour? Is there any way for this to restart the request sequence from the exact point it failed previously?
For someone who is curious about what this code looks like, I am posting it below:
import requests import pytz import json import time from datetime import date, datetime from django.http import JsonResponse from django.views import View from api.models import Weather, Location, Address from api.constants import (WEATHER_API_ENDPOINT, SERVICE_KEY, PAGE_NO, NUM_OF_ROWS, BASE_DATE_EMPTY, BASE_TIME_EMPTY, NX_EMPTY, NY_EMPTY, DATA_TYPE_JSON ) class WeatherCreate(View): def get(self, request): today = date.today().strftime("%Y%m%d") today = str(today) current_time = datetime.now(pytz.timezone('Asia/Seoul')).strftime('%H') current_time = str(current_time) + "00" current_time = "1100" print(current_time) if current_time not in ("0200", "0500", "0800", "1100", "1400", "1700", "2000", "2300"): return JsonResponse(data={"message": "DATA IS NOT AVAILABLE AT THIS TIME. TRY AGAIN LATER."}, status=406) # Call every single distinct address addresses = Address.objects.all() locations = [address.location for address in addresses] counter = 0 for location in locations: if counter >= 30 and counter % 30 == 0: print({"MSG": "30 locations requested. Sleeping for 30 seconds...", "counter": counter}) time.sleep(30) nx = str(location.grid_x) ny = str(location.grid_y) url = WEATHER_API_ENDPOINT + SERVICE_KEY + PAGE_NO + NUM_OF_ROWS + BASE_DATE_EMPTY + today + BASE_TIME_EMPTY + current_time + NX_EMPTY + nx + NY_EMPTY + ny + DATA_TYPE_JSON try: response = requests.get(url) result = response.json()["response"] except Exception as e: ############### I don't know how to save the failed location weather request for retries later ################ continue # return JsonResponse(data={"MSG": "Session failed", "counter": counter}, status=401) try: data = result['body'] items = data["items"]["item"] sky_value = None am_temp = None pm_temp = None for item in items: for key, value in item.items(): if value == "SKY" and sky_value == None: sky_value = item["fcstValue"] if value == "PTY" and sky_value == None: sky_value = item["fcstValue"] elif value == "TMN" and am_temp == None: am_temp = item["fcstValue"] elif value == "TMX" and pm_temp == None: pm_temp = item["fcstValue"] sky_value = int(float(sky_value)) if sky_value != None else None am_temp = int(float(am_temp)) if am_temp != None else None pm_temp = int(float(pm_temp)) if pm_temp != None else None weather = Weather.objects.create( location=location, state=sky_value, am_temp=am_temp, pm_temp=pm_temp ) counter += 1 print({"message": "WEATHER GENERATED!", "counter": counter}) time.sleep(0.3) except Exception as e: data = {"message": "ERROR: No response body"} print(data) continue return JsonResponse({"MSG": "FINISHED!"}, status=200)
Advertisement
Answer
Try using a while
loop on a single request,
while True: try: response = requests.get(url) result = response.json()["response"] break except Exception as e: print(str(e))
This way, whenever a request failed, it will just retry that request. But you have to make sure that the request will eventually be successful or else it will loop itself endlessly. You may add a trial
counter to count the number of trials and break if the trials exceed, for say like 5 times, to prevent infinite loop.