Sometimes you may want to delete some metrics from Prometheus if those metrics are unwanted or you just need to free up some disk space. Time series in Prometheus can be deleted by the administrative HTTP API only (disabled by default).
To enabled it, pass the flag “–web.enable-admin-api” to Prometheus, for example through a start-up script or a docker-compose file, depending on your installation method.
Delete metrics for a label
Use the following syntax to delete all time series metrics that match some label:
$> curl -X POST \
-g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={foo="bar"}'
Delete metrics for a job or instance
To delete time series metrics that match some job or instance, run:
$> curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={job="node_exporter_customer1"}'
$> curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={instance="CustomerXY_NodeZ"}'
Delete all
To delete all data from Prometheus, run:
$> curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={__name__=~".+"}'
Notes:
-
Changes can be shown immediately if you update your Grafana dashboard or execute a proQL query from Prometheus.
-
The above API calls don’t delete data immediately. The data remains on disk and will be cleaned up in future compaction.
To determine when to remove old data, use the option “–storage.tsdb.retention”, e.g.:
–storage.tsdb.retention=‘365d’
By default, Prometheus keeps data for 15 days.
To completely remove the data deleted by delete_series send clean_tombstones API call:
$> curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/clean_tombstones'
The successful exit status for the both delete_series and clean_tombstones is 204.