Prometheus Remote Storage

Remote read

When configured, Prometheus storage queries (e.g. via the HTTP API) are sent to both local and remote storage, and results are merged.

Note that to maintain reliability in the face of remote storage issues, alerting and recording rule evaluation use only the local TSDB.

Configuration

You configure the remote storage read path in the remote_read section of the Prometheus configuration file.

At its simplest, you will just specify the read endpoint URL for your remote storage, plus an authentication method. You can use either HTTP basic or bearer token authentication.

You might want to use the read_recent flag: when set to true, all queries will be answered from remote as well as local storage. When false (the default), any queries that can be answered completely from local storage will not be sent to the remote endpoint.

You can specify a set of required_matchers (label, value pairs) to restrict remote reads to some subset of queries. This is useful if e.g. you write only a subset of your metrics to remote storage (see below).

For more complex configurations, there are also options for request timeouts, TLS configuration, and proxy setup.

You can read from multiple remote endpoints by having one remote_read section for each.

Remote write

When configured, Prometheus forwards its scraped samples to one or more remote stores.

Remote writes work by "tailing" time series samples written to local storage, and queuing them up for write to remote storage.

The queue is actually a dynamically-managed set of "shards": all of the samples for any particular time series (i.e. unique metric) will end up on the same shard.

The queue automatically scales up or down the number of shards writing to remote storage to keep up with the rate of incoming data.

This allows Prometheus to manage remote storage while using only the resources necessary to do so, and with minimal configuration.

Configuration

You configure the remote storage write path in the remote_write section of the Prometheus configuration file.

Like for remote_read,  the simplest configuration is just a remote storage write URL, plus an authentication method. You can use either HTTP basic or bearer token authentication.

You can use write_relabel_configs to relabel or restrict the metrics you write to remote storage. For example, a common use is to drop some subset of metrics:

1. writeRelabelConfigs:
2.  # drop all metrics of this name across all jobs
3.  - sourceLabels: ["__name__"]
4.    regex: some_metric_prefix_to_drop_.*
5.    action: drop

The queue_config section gives you some control over the dynamic queue described above. Usually, you won't need to make changes here and can rely on Prometheus' defaults.

• capacity: each shard is itself a queue, and this is the number of samples queued before the shard "blocks" further additions;
  
• min_shards, max_shards: the minimum & maximum shards the dynamic queue will use;
  
• max_samples_per_send, batch_send_deadline: each shard batches samples up into blocks of max_samples_per_send, or if it can't make a batch of that size before batch_send_deadline, sends anyway; this latter will rarely happen on a busy Prometheus instance;
• min_backoff, max_backoff, max_retries: retry management; note max_retries is not used in the current implementation - each shard will just block and retry sends "forever".

Like for remote_read, you can also configure options for request timeouts, TLS configuration, and proxy setup.
‍
You can write to multiple remote endpoints by having one remote_write section for each.

Log messages

You may see some messages from the remote storage subsystem in your logs:

• dropped sample for series that was not explicitly dropped via relabelling
  Because of relabelling or for some other reason, we've ended up with a series with no labels in the remote write path; we drop it.
  
• Remote storage resharding from N to M
  The dynamic queue size is changing the number of shards - either growing to keep up with the number of incoming samples vs. outgoing remote storage write rate, or shrinking because we have more shards than are necessary.
• Currently resharding, skipping
  The dynamic queue wants to change to a new number of shards, but a reshard is already in progress.
  
• Failed to flush all samples on shutdown
  While shutting down a dynamic queue, Prometheus was unable to flush all samples to remote storage - it's probable there was a problem with the remote storage endpoint.
  

Metrics

The remote storage subsystem exports lots of metrics, prefixed with prometheus_remote_storage_ or prometheus_wal_watcher_Here's a selection you might find interesting:

• prometheus_remote_storage_samples_in_total: Samples in to remote storage, compare to samples out for queue managers (counter);
  
• prometheus_remote_storage_succeeded_samples_total: Total number of samples successfully sent to remote storage (counter);
  
• prometheus_remote_storage_pending_samples: The number of samples pending in the queue's shards to be sent to the remote storage (gauge);
  
• prometheus_remote_storage_shards: The number of shards used for parallel sending to the remote storage (gauge);
  
• prometheus_remote_storage_sent_batch_duration_seconds: Duration of sample batch send calls to the remote storage (histogram).
  
  

References

• https://prometheus.io/docs/prometheus/latest/configuration/configuration/
  The Prometheus configuration file reference, including details notes on remote_read and remote_write;
  
• https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage
  A list of services & software that offer remote storage endpoints;
  
• Integrating Long-Term Storage with Prometheus - Julius Volz
  A detailed presentation on how remote storage works in Prometheus, from CloudNativeCon Europe 2017;
  
• Things you wish you never knew about the Prometheus Remote Write API - Tom Wilkie
  Speaker notes on the details of remote_write, from PromCon 2017.