I work on a docker project with several containers and I wanna use python Prometheus library to monitor some metrics in the containers, expose each container’s metrics on a local port inside the docker-network and collect them in another container called Prometheus_exporter.
For this purpose, I’ve defined several Prometheus metrics on my FastAPI and I wanna expose them on an http server with Prometheus library’s “start_http_server” method. here’s my API code and how I use it:
api.py: in this file I’ve defined my API and its endpoints
from prometheus_client import Counter, Histogram, start_http_server app = FastAPI() num_q = Counter('api_num_queries','counts number of requests sent to API', ['endpoint']) num_err = Counter('api_num_errors','counts number of errors occurred') latency = Histogram('api_latency', 'latency calculator') @app.get('/userdata/{uid}') @latency.time() @num_err.count_exceptions() def get_user_data(uid): udata = redis.get(uid) return udata @app.get('/bookdata/{bid}') @latency.time() @num_err.count_exceptions() def get_book_data(bid): bdata = redis.get(bid) return bdata
main.py: in this file I publish my API
import uvicorn from api import app from prometheus_client import start_http_server if __name__ == '__main__': uvicorn.run("main:app", host="0.0.0.0", port=8080, workers=10)
The problem is when I place start_http_server(8000)
in main.py file, like this:
import uvicorn from api import app from prometheus_client import start_http_server if __name__ == '__main__': start_http_server(8000) uvicorn.run("main:app", host="0.0.0.0", port=8080, workers=10)
the metrics are exposed but they don’t update and stay at their initial zero value while the metrics are changing in api.py.
the other way that I tried was using start_http
server(8000)
like this in api.py:
from prometheus_client import Counter, Histogram, start_http_server app = FastAPI() num_q = Counter('api_num_queries','counts number of requests sent to API', ['endpoint']) num_err = Counter('api_num_errors','counts number of errors occurred') latency = Histogram('api_latency', 'latency calculator') @app.get('/userdata/{uid}') @latency.time() @num_err.count_exceptions() def get_user_data(uid): udata = redis.get(uid) start_http_server(8000) return udata @app.get('/bookdata/{bid}') @latency.time() @num_err.count_exceptions() def get_book_data(bid): bdata = redis.get(bid) start_http_server(8000) return bdata
this works fine at the first time and metrics are exposed with their latest value, but when I send a new request i get “Port Already in use” error.
what should I do and how I can expose my metrics correctly?
Advertisement
Answer
Ok, I figured something out for this. It’s not really a solution but It works!
The reason for this problem is that the http_start_server()
command doesn’t reset or update the info on the port automatically and It’s the prometheus_client
library’s problem.
To solve this, you can define a new metrics
endpoint in your API like below and return your metrics every time a request is sent:
from prometheus_client import generate_latest, CollectorRegistry metrics_reg = CollectorRegistry() #define your metrics and specify all the metrics registrys to be metrics_reg @app.get('/metrics') def get_metrics(): return generate_latest(metrics_reg)
This solution works only on API modules.