Monitoring & Performance
A Feldera instance and its pipelines can be monitored using various tools. This tutorial will guide you through setting up monitoring for your Feldera instance.
Metrics with Grafana and Prometheus
Metrics are helpful to check the health of your Feldera instance and to identify resource bottlenecks. Feldera exposes a metrics endpoint that can be scraped by Prometheus. Grafana is then used to visualize these metrics.
Setup
- Prometheus: You must have Prometheus installed.
- Connect Prometheus to Feldera:
- Add the following to your
prometheus.yml
configuration file, usually located in/etc/prometheus/prometheus.yml
:
- job_name: 'feldera'
scrape_interval: 1s
static_configs:
- targets: ['localhost:8081']- Replace
localhost:8081
with the address of your Feldera instance. Note that8081
is the default port for metrics. - Restart Prometheus.
- Add the following to your
- Grafana: You must have Grafana installed.
- Add Prometheus To Grafana:
- If you are using a local prometheus instance, the URL for the Prometheus data source will
be
http://localhost:9090
. - Follow the steps in the Grafana documentation to add Prometheus as a data source.
- If you are using a local prometheus instance, the URL for the Prometheus data source will
be
Set up the monitoring Dashboard
- Copy the JSON of the Feldera template dashboard
- Import the dashboard into Grafana
- Under Dashboards, click the "New" icon, then click "Import Dashboard".
- Paste the JSON copied from the template in the text-box and click "Load".
A Feldera Instance Monitoring dashboard will be created in Grafana.
The dashboard is a template and may need to be adjusted to fit your specific needs.
Look for the feldera_*
metrics in Grafana to add more metrics to the dashboard.
Tracing with Jaeger
A Feldera pipeline can be traced using Jaeger. Tracing is useful if you need to analyze throughput or latency bottlenecks of your pipeline, as it instruments every step of the pipeline execution and provides detailed information about the execution time of each step.
Setup
- Jaeger: You must have Jaeger installed.
- Start Jaeger using the
jaeger-all-in-one
script:./jaeger-all-in-one
- Tracing a Feldera pipeline
- Enable/disable tracing and specify the Jaeger endpoint in the pipeline
runtime_config
:
curl -i -X PATCH http://localhost:8080/v0/pipelines/tracing-pipeline \
-H 'Content-Type: application/json' \
-d '{
"runtime_config": {"tracing": true, "tracing_endpoint_jaeger": "host.docker.internal:6831", <other config settings> }
}'- Use
host.docker.internal:6831
if you are running Feldera in docker or127.0.0.1:6831
if you run Feldera directly. - Make sure to specify other settings you changed from non-default values in the
runtime_config
as well.runtime_config
can be retrieved with:
curl -X 'GET' \
'http://127.0.0.1:8080/v0/pipelines/tracing-pipeline' \
-H 'accept: application/json' - Enable/disable tracing and specify the Jaeger endpoint in the pipeline
DBSP Profiles
A DBSP profile is a graph of the pipeline's circuit where each node represents an operator and each edge represents a data flow between operators. The profile includes information about how much CPU time or memory each operator consumes.
The API endpoint /v0/<pipeline_name>/dump_profile
can be used to download the DBSP
profile of a running pipeline. It returns a zip file containing multiple profiles (one
for each worker) as .dot
files, and a Makefile
to transform the .dot
files into
.pdf
files.