Upgrading Dovecot CE from 2.4 to 2.4.x

Dovecot CE

v2.4.0 to v2.4.1

• Added MALLOC_MMAP_THRESHOLD_=131072 to import_environment when building with glibc. Note that dovecot_config_version value does not affect change - it's always added.

v2.4.2 to v2.4.3

• Using Settings Variables with LDAP settings now requires using | safe filter to allow passing through LDAP characters that are otherwise escaped. The safe filter must always be the last filter in the pipeline. Note that this should be used only for variables coming from trusted sources, not for e.g. username variable.

Changed Setting Defaults

These changes don't take effect until dovecot_config_version is changed from 2.4.0 to 2.4.1.

Setting	Old Default Value	New Default Value
mail_cache_fields	flags	flags hdr.date hdr.subject hdr.from hdr.sender hdr.reply-to hdr.to hdr.cc hdr.bcc hdr.in-reply-to hdr.message-id date.received size.virtual imap.bodystructure mime.parts hdr.references hdr.importance hdr.x-priority hdr.x-open-xchange-share-url pop3.uidl pop3.order
service/auth/unix_listener/auth-userdb/group	(empty = root)	$SET:default_internal_group
service/lmtp/restart_request_count	unlimited	1
lmtp_user_concurrency_limit	unlimited	10

v2.4.1 to v2.4.2

Changed Setting Defaults

These changes don't take effect until dovecot_config_version is changed to 2.4.2.

Setting	Old Default Value	New Default Value
@metric_defaults/proxy/metric		Adds proxy_session_established
service/anvil/unix_listener/anvil/mode	0600	0660
service/anvil/unix_listener/anvil/group	(empty = root)	$SET:default_internal_group
mailbox_directory_name_legacy	yes	no
mail_attachment_detection_options		add-flags content-type=!application/signature

v2.4.2 to v2.4.3

Changed Setting Defaults

These changes don't take effect until dovecot_config_version is changed to 2.4.3.

Setting	Old Default Value	New Default Value
fts_message_max_size		0 is not allowed anymore to mean unlimited use unlimited instead
last_valid_gid		0 is not allowed anymore to mean unlimited use unlimited instead
last_valid_uid		0 is not allowed anymore to mean unlimited use unlimited instead
lazy_expunge_only_last_instance	no	yes
mail_access_groups		default_internal_group
mail_cache_max_header_name_length		0 is not allowed anymore to mean unlimited use unlimited instead
mail_cache_max_headers_count		0 is not allowed anymore to mean unlimited use unlimited instead
mail_sort_max_read_count		0 is not allowed anymore to mean unlimited use unlimited instead
mail_vsize_bg_after_count		0 is not allowed anymore to mean unlimited use unlimited instead
passdb_default_password_scheme	PLAIN	CRYPT
sieve_quota_script_count		0 is not allowed anymore to mean unlimited use unlimited instead
sieve_quota_storage_size		0 is not allowed anymore to mean unlimited use unlimited instead
submission_max_recipients		0 is not allowed anymore to mean unlimited use unlimited instead
maildir/mail_attachment_detection_options	add-flags content-type=!application/signature
passdb_passwd_file/passdb_default_password_scheme	PLAIN	CRYPT
passdb_sql/passdb_default_password_scheme	PLAIN	CRYPT
service/doveadm/service_extra_groups	default_internal_group
service/imap/service_extra_groups	default_internal_group
service/imap-urlauth-worker/service_extra_groups	default_internal_group
service/indexer-worker/service_extra_groups	default_internal_group
service/lmtp/service_extra_groups	default_internal_group
service/pop3/service_extra_groups	default_internal_group
service/submission/service_extra_groups	default_internal_group
service/managesieve/service_extra_groups	default_internal_group
@mailbox_defaults/english/namespace/inbox/mailbox/Drafts/mailbox_auto	no	subscribe
@mailbox_defaults/english/namespace/inbox/mailbox/Junk/mailbox_auto	no	subscribe
@mailbox_defaults/english/namespace/inbox/mailbox/Sent/mailbox_auto	no	subscribe
@mailbox_defaults/english/namespace/inbox/mailbox/Trash/mailbox_auto	no	subscribe
service/anvil/unix_listener/anvil-auth-penalty/mode	0600	0660
service/anvil/unix_listener/anvil-auth-penalty/group		default_internal_group
service/imap/unix_listener/imap-master/user		default_internal_user

Removed Features

Feature	Notes
acl_global_path setting	See ACL Settings Migration.

v2.4.3 to v2.4.4

Forcibly Changed Setting Defaults

Unlike other default setting changes, this change takes effect regardless of dovecot_config_version. This is because indexer-worker now drops root privileges, and the old default could make the service non-working.

Setting	Old Default Value	New Default Value
service/indexer-worker/service_restart_request_count	service_restart_request_count	1

Changed Setting Defaults

These changes don't take effect until dovecot_config_version is changed to 2.4.4.

Setting	Old Default Value	New Default Value
push_notification_ox/http_client_request_max_attempts	1	2
push_notification_ox/http_client_request_timeout	1min	2s
service/lmtp/service_extra_groups		default_internal_group

v2.4.4 to v2.4.5

• String List and Boolean List setting keys are now expanded with Settings variables the same way their values already were. Keys defined in the configuration are always expanded. For settings overrides only default overrides expand the keys; keys coming from -o command line parameters and from userdb are kept literal.
• The per-field time and date Settings Variables providers are now deprecated: time:hour, time:min, time:minute, time:sec, time:second, time:usec and date:year, date:month, date:day. Use the new time:unix provider together with the date filter instead. Note that the old providers always used the local timezone, whereas the date filter defaults to UTC, so to keep the old behavior pass tz='local': use %{time:unix | date('%H', 'local')} in place of %{time:hour}. The new approach additionally supports UTC, timestamps from any other variable, and ISO 8601 / RFC 3339 output via the iso8601 filter. The deprecated providers keep working for now.

Changed Setting Defaults

These changes don't take effect until dovecot_config_version is changed to 2.4.5.

Setting	Old Default Value	New Default Value
sieve_max_cpu_time	0 (unlimited)	30s
imap_compress_on_proxy	no	yes

Changed Storage Formats

These changes don't take effect until dovecot_storage_version is changed to 2.4.5.

The dovecot.index.thread index, used by the IMAP THREAD command, has a new v2 format. The Message-ID hashes stored in it are now calculated with a keyed hash using a random per-file key instead of CRC32. This prevents users from crafting Message-IDs whose hashes deliberately collide, which would otherwise slow down threading.

Existing v1 index files are deleted and rebuilt in the v2 format when the mailbox is next opened for threading, so the first THREAD command for a large mailbox can be slower than usual. Dovecot versions older than v2.4.5 don't understand the v2 format and rebuild the index in the v1 format again, so this setting should be increased only once a rollback to an older version is no longer possible.

New Features

Feature	Notes
`escape` filter	Explicitly apply the configured escape function within a variable expansion pipeline. Useful for partial escaping when combined with concat and safe.
`time:unix` provider	Returns the current time as a <seconds>.<nanoseconds> UNIX timestamp. Recommended replacement for the deprecated per-field time:/date: providers.
`epoch`	Convert and format UNIX timestamps (unit scaling, calendar formatting via strftime, and ISO 8601 / RFC 3339 output).

Removed Features

Feature	Notes
auth: SIGHUP, SIGUSR2	Use doveadm auth cache flush to flush the cache and doveadm auth cache status to inspect cache statistics instead.

Sieve

Cumulative CPU usage tracking for sieve_max_cpu_time is no longer kept inside each compiled Sieve binary (.svbin). It is now stored in a single file sieve-rusage at the root of the user's INBOX namespace.

v2.4.5 to v2.4.6

• A passdb lookup returning an empty password now fails authentication instead of allowing login with an empty password. Passdb entries intended to have no fixed password should return the nopasswordextra field (e.g. user:::::::nopassword=yes in a passwd-file) instead of an empty password field; any non-empty client password is then accepted.

• An empty client-supplied password is now always rejected before the passdb lookup, for all passdb drivers. Previously only LDAP auth binds rejected it. nopassword does not allow an empty client password either; only mechanisms that authenticate without a password (EXTERNAL via client certificate, OAuth2) are unaffected.

• The EXTERNAL SASL mechanism now treats the client certificate as the authentication and no longer verifies any passdb password. Existing client-certificate setups (e.g. passwd-file entries with an empty password field) keep working. Passdbs that verify the password themselves no longer reject EXTERNAL logins either: e.g. an LDAP auth bind passdb attempts no bind - the lookup succeeds without verifying anything and only provides the extra fields, since the certificate already authenticated the user. For EXTERNAL loginspassdb { skip = authenticated } passdbs are now skipped and skip = unauthenticated passdbs run; if all passdbs are skipped, the login fails.

• Boolean settings no longer accept y or 1 as a value in the configuration. They were never documented, and only their true side ever worked - the matching n and 0 have always been errors. Use yes and no instead. Values that don't come from the configuration still accept them, so e.g. a userdb lookup can keep returning mail_debug=y.

• New file path and directory path setting types: a leading ~/ in the value is expanded to the user's home directory, and directory paths also drop a trailing /. The mail location *_path settings and the rawlog directory settings use these types now. The following settings previously used a ~/ prefix literally; it is now expanded (or fails, if no user home is available):

  • rawlog_dir (imap, pop3, submission and managesieve)
  • http_client_rawlog_dir
  • http_server_rawlog_dir

• doveconf output no longer drops a trailing / from mail_path and the other directory path settings; the value is shown as written in the configuration.

• IMAP search queries (SEARCH, UID SEARCH, SORT, THREAD and the search arguments of STORE and FETCH) and doveadm search queries are now rejected with Too much nesting in search query when the search keys are nested deeper than the process stack allows. The limit is derived from RLIMIT_STACK and is 1024 levels with the common 8 MB default stack, so normal clients are unaffected. See Search Query Nesting Limit for how to raise it.

New Cassandra Settings

New settings were added to configure the Cassandra driver. The defaults are the same as the driver's previous behavior, except that the client now sends cassandra_application_name and cassandra_application_version to the server.

If the Cassandra cluster has multiple datacenters, it's recommended to set cassandra_local_datacenter. Otherwise the local datacenter is the datacenter of whichever cassandra_hosts host answers first.

Setting	Notes
cassandra_local_datacenter	Setting was added.
cassandra_connections_per_host	Setting was added.
cassandra_reconnect_policy	Setting was added.
cassandra_reconnect_base_delay	Setting was added.
cassandra_reconnect_max_delay	Setting was added.
cassandra_tcp_keepalive	Setting was added.
cassandra_token_aware_routing	Setting was added.
cassandra_token_aware_shuffle_replicas	Setting was added.
cassandra_latency_aware_exclusion_threshold	Setting was added.
cassandra_latency_aware_scale	Setting was added.
cassandra_latency_aware_retry_period	Setting was added.
cassandra_latency_aware_update_rate	Setting was added.
cassandra_latency_aware_min_measured	Setting was added.
cassandra_request_queue_size	Setting was added.
cassandra_application_name	Setting was added.
cassandra_application_version	Setting was added.
cassandra_client_id	Setting was added.
cassandra_source_ip	Setting was added.

FTS Direct Autoindexing

fts_autoindex was changed from a boolean to no, yes or direct. The existing yes and no values work as before. The new direct value indexes the newly added mails directly in the same process, instead of asynchronously via the indexer service. This is mainly useful when migrating mails with doveadm sync or doveadm backup.

Setting	Notes
fts_autoindex	Setting value direct was added.

• Auth SQL queries (passdb_sql_query, userdb_sql_query and userdb_sql_iterate_query) now pass %{variable} values to the database as bind parameters instead of expanding them into the query text. Queries that relied on the old text expansion fail at lookup time with an error:
  • A %{variable} must produce a whole value on its own. Wrapping a single variable in single quotes still works ('%{user}'), but combining it with other text does not: '%{user | username}@example.com' and '%{user}%' must be rewritten with the var-expand concat filter, e.g. %{user | username | concat('@example.com')} and %{user | concat('%')}.
  • A backslash is no longer allowed inside a single-quoted string. Escape a quote as '', not \'.
  • SQL comments (--, /*, // and #) are no longer allowed outside a quoted string, and neither is PostgreSQL dollar quoting ($$...$$) or a positional parameter ($1, $2, ...).
  • A literal ? outside a quoted string can no longer appear in the query. This affects PostgreSQL's jsonb ?, ?| and ?& operators.
• Logged queries no longer show the %{password} value unless auth_debug_passwords = yes is set.

See Variables in Queries for details.

• shutdown_clients was replaced by service_shutdown_clients_timeout, which also allows everything in between its two endpoints: shutdown_clients=yes becomes service_shutdown_clients_timeout=0 and shutdown_clients=no becomes service_shutdown_clients_timeout=infinite. The conversion is done automatically as long as dovecot_config_version is older than this release. A reload keeps the existing client sessions running for that long, e.g. to take new SSL certificates into use without disconnecting anyone.

• When the timeout expires - or right away, with the default 0 - the processes disconnect all their remaining clients, also the ones that are in the middle of a command, and the login processes abort the logins that are still in progress. Previously shutdown_clients=yes kept an IMAP session running until the client had been idle for 10 seconds, or for 30 seconds if it never was, and the other services waited for up to 30 seconds for the clients to disconnect by themselves.

• Only the processes of service_type = client and service_type = login services keep serving their clients across a reload. The internal services are always replaced by it, so their old processes are stopped regardless of the timeout - otherwise they would pile up with every reload.

• The services whose processes serve externally visible client connections have service_type client now: imap, pop3, submission, imap-hibernate, imap-urlauth, imap-urlauth-worker, managesieve and doveadm. lmtp is left out on purpose, because an MTA can keep an idling LMTP connection open for a long time, which would then keep the old generation's processes running. The previous default isn't preserved for older dovecot_config_version values, because the type only tells the master process how the service's processes are to be treated.

• A custom service { .. } block that relied on shutdown_clients=no needs type = client added to it. Without it the reload stops the service's old processes in spite of the converted timeout.

• Some settings can now be used only where they have an effect. Using one inside a filter that doesn't use it is a configuration error instead of being silently ignored. For example quota_mail_size applies to the user rather than to a quota root, so it can't be used inside quota { .. }.

• The global default_* settings and protocols can be used only globally. Setting a default_* inside service { .. } used to change that service's derived settings, because their defaults expand $SET:default_internal_user and similar within the filter. Use the per-service setting instead, e.g. service_user rather than service foo { default_internal_user = .. }. Referring to the global settings with $SET: inside a filter keeps working.

• The listen setting is now called inet_listener_listen. The old name still works as an alias, so configurations don't need changes, but doveconf writes the new name outside of inet_listener { .. } blocks.

• source_location in log_debug and log_core_filter is now matched against each log line's own source location. Previously info, warning and error lines of events with a raised minimum log level (e.g. auth lines with auth_verbose = no) were matched against an internal source location, and the result matched for one log line was reused for the event's later log lines. Because the filter results can't be cached anymore when source_location is used, every debug log call becomes several times slower. Use source_location only temporarily while debugging. See Performance With source_location.

Unauthenticated Client Limit

A new login_unauthenticated_client_limit setting limits the number of unauthenticated client connections in each login process, separately from service_client_limit. When the limit is reached, the oldest unauthenticated connection is disconnected. This is mainly useful with high performance login mode. The default is unlimited, so the behavior doesn't change unless the setting is configured.

Setting	Notes
login_unauthenticated_client_limit	Setting was added.

• A local_name filter nested inside another local_name filter must now be a hostname that matches the outer filter's name, the same way as a nested local or remote network must be inside the outer network. For example local_name *.example.com { local_name imap.example.com { .. } } is allowed, but local_name *.example.com { local_name imap.example.org { .. } } or an inner name with a wildcard now fails the config parsing. Previously such blocks were accepted, but their settings applied only to connections matching both names, which was usually nothing. See Connection Filters.

Doveadm Logging Changes

doveadm-server no longer writes the Apache-style access log line for each Doveadm HTTP API request. The same information is now available from events: doveadm_command_finished for the command, the user and the exit code, and http_server_request_finished for the HTTP status and the transferred byte counts. Both are debug level events, so use Event Export to record them. Note that the format differs from the old log line.

doveadm_command_finished is emitted for command line and doveadm server protocol invocations as well, not just for the HTTP API.

Some Doveadm HTTP API log lines also changed level:

Log line	Old Level	New Level
Executing command, Executing command as '<user>'	info	debug
Error writing output in command <cmd>: <error>	info	error
read(<stream>) failed: <error>	info	error
error writing output: <error>	info	error

New Features

Feature	Notes
service_shutdown_clients_timeout	Keep the existing sessions running for a while after doveadm reload, e.g. to take new SSL certificates into use without disconnecting anyone. See reloading the configuration.
doveadm reload --kick-timeout	Override service_shutdown_clients_timeout for a single reload.
doveadm process status and doveadm service status generation and kill_time columns	Tell which configuration generation a process belongs to and when the master process is going to signal it next. doveadm service status -a lists also the older generations.

Removed Features

Feature	Notes
doveadm_allowed_commands setting	The setting only matched the command name, while the commands it was typically used to allow already gave full access to any user's mails. Dovecot fails to start if this setting is present in the configuration. Give doveadm server access only to trusted clients. The Doveadm HTTP API no longer returns the 403 HTTP response code, which was used only for commands rejected by this setting.
shutdown_clients setting	Replaced by service_shutdown_clients_timeout.