Calling an External API That Fails Randomly

I am using Django server to call a Korean government’s weather API to retrieve weather data for around 1800 locations.

However, this weather API results in time out most of the time. I tried giving resting periods. For example, for every request, it will sleep for 0.8 seconds and after every 30 requests, it will sleep for 30 seconds. But this does not really work well for 1800 requests. At one point, it was successful. All other times, it ended up with fail.

In this kind of situation, what else should I do to make sure that 1800 requests are completed within one hour? Is there any way for this to restart the request sequence from the exact point it failed previously?

For someone who is curious about what this code looks like, I am posting it below:

import requests
import pytz
import json
import time

from datetime import date, datetime

from django.http import JsonResponse
from django.views import View
from api.models import Weather, Location, Address
from api.constants import (WEATHER_API_ENDPOINT, SERVICE_KEY,
                           PAGE_NO, NUM_OF_ROWS, BASE_DATE_EMPTY,
                           BASE_TIME_EMPTY, NX_EMPTY, NY_EMPTY,
                           DATA_TYPE_JSON
                           )


class WeatherCreate(View):

    def get(self, request):
        today = date.today().strftime("%Y%m%d")
        today = str(today)
        current_time = datetime.now(pytz.timezone('Asia/Seoul')).strftime('%H')
        current_time = str(current_time) + "00"
        current_time = "1100"
        print(current_time)
        if current_time not in ("0200", "0500", "0800", "1100", "1400", "1700", "2000", "2300"):
            return JsonResponse(data={"message": "DATA IS NOT AVAILABLE AT THIS TIME. TRY AGAIN LATER."}, status=406)

        # Call every single distinct address
        addresses = Address.objects.all()
        locations = [address.location for address in addresses]
        counter = 0

        for location in locations:
            if counter >= 30 and counter % 30 == 0:
                print({"MSG": "30 locations requested. Sleeping for 30 seconds...", "counter": counter})
                time.sleep(30)
            nx = str(location.grid_x)
            ny = str(location.grid_y)
            url = WEATHER_API_ENDPOINT + SERVICE_KEY + PAGE_NO +
                  NUM_OF_ROWS + BASE_DATE_EMPTY + today +
                  BASE_TIME_EMPTY + current_time + NX_EMPTY + nx +
                  NY_EMPTY + ny + DATA_TYPE_JSON

            try:
                response = requests.get(url)
                result = response.json()["response"]

            except Exception as e:
                ############### I don't know how to save the failed location weather request for retries later ################
                
                continue
                
                # return JsonResponse(data={"MSG": "Session failed", "counter": counter}, status=401)
            try:
                data = result['body']
                items = data["items"]["item"]
                sky_value = None
                am_temp = None
                pm_temp = None

                for item in items:
                    for key, value in item.items():
                        if value == "SKY" and sky_value == None:
                            sky_value = item["fcstValue"]
                        if value == "PTY" and sky_value == None:
                            sky_value = item["fcstValue"]
                        elif value == "TMN" and am_temp == None:
                            am_temp = item["fcstValue"]
                        elif value == "TMX" and pm_temp == None:
                            pm_temp = item["fcstValue"]
                sky_value = int(float(sky_value)) if sky_value != None else None
                am_temp = int(float(am_temp)) if am_temp != None else None
                pm_temp = int(float(pm_temp)) if pm_temp != None else None

                weather = Weather.objects.create(
                    location=location,
                    state=sky_value,
                    am_temp=am_temp,
                    pm_temp=pm_temp
                )
                counter += 1
                print({"message": "WEATHER GENERATED!", "counter": counter})
                time.sleep(0.3)

            except Exception as e:
                data = {"message": "ERROR: No response body"}
                print(data)
                continue

        return JsonResponse({"MSG": "FINISHED!"}, status=200)

Answer

Try using a while loop on a single request,

while True:
    try:
        response = requests.get(url)
        result = response.json()["response"]
        break
    except Exception as e:
        print(str(e))

This way, whenever a request failed, it will just retry that request. But you have to make sure that the request will eventually be successful or else it will loop itself endlessly. You may add a trial counter to count the number of trials and break if the trials exceed, for say like 5 times, to prevent infinite loop.