updating API Prometheus metrics on an Http server error

I work on a docker project with several containers and I wanna use python Prometheus library to monitor some metrics in the containers, expose each container’s metrics on a local port inside the docker-network and collect them in another container called Prometheus_exporter.

For this purpose, I’ve defined several Prometheus metrics on my FastAPI and I wanna expose them on an http server with Prometheus library’s “start_http_server” method. here’s my API code and how I use it:

api.py: in this file I’ve defined my API and its endpoints

from prometheus_client import Counter, Histogram, start_http_server
app = FastAPI()

num_q = Counter('api_num_queries','counts number of requests sent to API', ['endpoint'])
num_err = Counter('api_num_errors','counts number of errors occurred')
latency = Histogram('api_latency', 'latency calculator')

@app.get('/userdata/{uid}')
@latency.time()
@num_err.count_exceptions()
def get_user_data(uid):
    udata = redis.get(uid)

    return udata
    

@app.get('/bookdata/{bid}')
@latency.time()
@num_err.count_exceptions()
def get_book_data(bid):
    bdata = redis.get(bid)
    
    return bdata

main.py: in this file I publish my API

import uvicorn
from api import app
from prometheus_client import start_http_server

if __name__ == '__main__':
    uvicorn.run("main:app", host="0.0.0.0", port=8080, workers=10)

The problem is when I place start_http_server(8000) in main.py file, like this:

import uvicorn
from api import app
from prometheus_client import start_http_server

if __name__ == '__main__':
    start_http_server(8000)
    uvicorn.run("main:app", host="0.0.0.0", port=8080, workers=10)

the metrics are exposed but they don’t update and stay at their initial zero value while the metrics are changing in api.py.

the other way that I tried was using start_httpserver(8000) like this in api.py:

from prometheus_client import Counter, Histogram, start_http_server
app = FastAPI()

num_q = Counter('api_num_queries','counts number of requests sent to API', ['endpoint'])
num_err = Counter('api_num_errors','counts number of errors occurred')
latency = Histogram('api_latency', 'latency calculator')

@app.get('/userdata/{uid}')
@latency.time()
@num_err.count_exceptions()
def get_user_data(uid):
    udata = redis.get(uid)
    
    start_http_server(8000)
    return udata
    

@app.get('/bookdata/{bid}')
@latency.time()
@num_err.count_exceptions()
def get_book_data(bid):
    bdata = redis.get(bid)
    
    start_http_server(8000)
    return bdata

this works fine at the first time and metrics are exposed with their latest value, but when I send a new request i get “Port Already in use” error.

what should I do and how I can expose my metrics correctly?

Answer

Ok, I figured something out for this. It’s not really a solution but It works! The reason for this problem is that the http_start_server() command doesn’t reset or update the info on the port automatically and It’s the prometheus_client library’s problem.

To solve this, you can define a new metrics endpoint in your API like below and return your metrics every time a request is sent:

from prometheus_client import generate_latest, CollectorRegistry
metrics_reg = CollectorRegistry()
#define your metrics and specify all the metrics registrys to be metrics_reg

@app.get('/metrics')
def get_metrics():
    return generate_latest(metrics_reg)

This solution works only on API modules.

Advertisement

Answer