Statistics

This page is about the statistics support for Dovecot:

New in version v2.3.

For v2.1 and v2.2 see Old Statistics.

See Events for list of all events that can be used in statistics.

Dovecot supports gathering statistics from “Events Design”. Eventually all of the log messages should be events, so it will be possible to configure Dovecot to get statistics for anything that is logged. For debug messages it’s possible to get statistics even if the message itself isn’t logged.

Currently there are no statistics logged by default (but this might change). You’ll need to add some metrics:

metric name {
  # Individual events can be identified either by their name or source file:line location.
  # The source location of course can change between Dovecot versions, so it should be
  # avoided.
  event_name = example_event_name
  #source_location = example.c:123

  # Space-separated list of categories that must match the event (e.g. "mail" or "storage")
  #categories =

  # List of fields in event parameters that are included in the metrics.
  # All events have a default "duration" field that doesn't need to be listed here.
  #fields =

  # List of key=value pairs that must match the event. The value can contain '?' and '*' wildcards.
  #filter {
  #  field_key = wildcard
  #}

  # v2.3.10+
  # List of fields to group events by into automatically generated
  # metrics.
  #group_by = field another-field
}

Group by

The group_by metric setting allows dynamic hierarchical metric generation based on event fields’ values. Each field listed in the group_by generates one level of “sub-metrics”. These automatically generated metrics are indistinguishable from those statically defined in the config file.

Dovecot supports a number of aggregate functions that can be used to quantize a field’s value before it is used to generate a metric.

The format is always the same: the field name, a colon, the aggregation function name, and optionally a colon followed by colon delimited parameters to the aggregation function.

discrete

The simplest aggregation function is to use the value as is. Because this is a very common use case, not specifying an aggregation function is treated as an alias for discrete aggregation. In other words, field and field:discrete produce the same behavior.

Example:

metric imap_command {
  event_name = imap_command_finished
  group_by = cmd_name tagged_reply_state
}

This example configuration will generate statistics for each IMAP command. The first “sub-metric” level is based on the IMAP command name, and the second (and in this example final) level is based on the tagged reply. For example, a SELECT IMAP command that succeeded (in other words, it had an OK reply) will generate the metric imap_command_select_ok.

In addition to the final level metric, all intermediate level metrics are generated as well. For example, the same SELECT IMAP command will generate all of the following metrics:

  • imap_command

  • imap_command_select

  • imap_command_select_ok

Note: While the top level metrics (e.g., imap_command above) are generated at start up, all group_by metrics are generated dynamically when first observed.

exponential

The field’s integer value is quantized into exponentially sized ranges.

The exponential aggregation function takes three colon delimited integer arguments that define the set of ranges used: the minimum magnitude, the maximum magnitude, and the base. The exact configuration syntax is: field:exponential:min:max:base

Note: Currently, only base 2 and base 10 are supported.

The first range starts at negative infinity and ends at pow(base, min). The second range begins at pow(base, min) + 1 and ends at pow(base, min + 1), the next covers pow(base, min + 1) + 1 to pow(base, min + 2), and so on. The last range covers pow(base, max) + 1 to positive infinity.

For example, given the specification duration:exponential:1:5:10, the ranges would be:

  • (-inf, 10]

  • [11, 100]

  • [101, 1000]

  • [1001, 10000]

  • [10001, 100000]

  • [100001, +inf)

Much like the metric names generated with the discrete aggregation function, the ones generated by the exponential function include information about the value of the field. However, in this case it is the range the value belongs to.

Specifically, it is the name of the field being quantized, and the lower and upper bounds for the range.

Example:

metric imap_command {
  event_name = imap_command_finished
  group_by = cmd_name duration:exponential:1:5:10
}

This will generate metric names of the format imap_command_{cmd}_duration_{min}_{max} where {cmd} is the IMAP command name, and {min} and {max} are the range bounds. Therefore, for a SELECT IMAP command, the possible generated metric names are:

  • imap_command_select_ninf_10

  • imap_command_select_11_100

  • imap_command_select_101_1000

  • imap_command_select_1001_10000

  • imap_command_select_10001_100000

  • imap_command_select_100001_inf

Note: Since the metric names cannot contain -, the string ninf is used to denote negative infinity.

Note: Much like in the discrete case, the metrics are allocated only when first observed.

Finally, because all intermediate level metrics are generated as well. The above example, will also generate all of the following metrics:

  • imap_command

  • imap_command_select

linear

The field’s integer value is quantized into linearly sized ranges.

The linear aggregation function takes three colon delimited integer arguments that define the set of ranges used: the minimum value, the maximum value, and the range step size. The exact configuration syntax is: field:linear:min:max:step

The first range starts at negative infinity and ends at min. The second range begins at min + 1 and ends at min + step, the next covers min + step + 1 to min + (2 * step), and so on. The last range covers max + 1 to positive infinity.

For example, given the specification bytes_out:linear:0:5000:1000, the ranges would be:

  • (-inf, 0]

  • [1, 1000]

  • [1001, 2000]

  • [2001, 3000]

  • [3001, 4000]

  • [4001, 5000]

  • [5001, +inf)

See the description of the exponential aggregation function for how metric names are formed from these ranges.

Listing Statistics

The gathered statistics are available by running:

doveadm stats dump

Each event has a duration field, which tracks in microseconds how long the event existed. For example with imap_command_finished field it could be:

metric_name          field      count  sum       min  max     avg      median stddev %95
imap_commands        duration   35     1190122   162  340477  34003    244    31215  188637

The above means:

count

There have been 35 IMAP commands

sum

The IMAP commands were running in total for 1190122 microseconds (= 1.1 seconds)

min

The fastest IMAP command took 162 microseconds

max

The slowest IMAP command took 340477 microseconds

avg

The average time spent on an IMAP commands was 34003 microseconds

median

The median time spent on an IMAP command was 244 microseconds

stddev

Standard deviation for the time spent on IMAP commands

%95

95% of the IMAP commands took 188637 microseconds or less

The other fields (than duration) track whatever that field represents. For example with imap_command_finished’s bytes_in field could be tracking how many bytes were being used by the IMAP commands. Non-numeric fields can also be tracked, although only the count is relevant to those.

The list of fields can be specified with the -f parameter. The default is:

doveadm stats dump -f 'count sum min max avg median stddev %95'

It’s also possible to specify other percentiles than just 95%, for example:

doveadm stats dump -f 'count sum min max avg median stddev %95 %99 %99.9 %99.99'

The stats counters are reset whenever the stats process is started, which also means a dovecot reload will reset statistics. Using doveadm stats -r parameter will also reset the statistics atomically after they’re dumped.

Examples:

IMAP command statistics

metric imap_select_no {
  event_name = imap_command_finished
  filter {
    name = SELECT
    tagged_reply_state = NO
  }
}

metric imap_select_no_notfound {
  event_name = imap_command_finished
  filter {
    name = SELECT
    tagged_reply = NO*Mailbox doesn't exist:*
  }
}

metric storage_http_gets {
  event_name = http_request_finished
  categories = storage
  filter {
    method = get
  }
}

# generate per-command metrics on successful commands
metric imap_command {
  event_name = imap_command_finished
  filter {
    tagged_reply_state = OK
  }
  group_by = cmd_name
}

Push notifications

metric push_notifications {
  event_name = push_notification_finished
}

# for OX driver
metric push_notification_http_finished {
  event_name = http_request_finished
  categories = push_notification
}