Data Access Patterns

Below is a list of operations that dictmap does for accessing data.

RED = Cassandra operations

BLUE = Object storage operations

  • Refreshing user root index

    • Frequency: very often – nearly every time a user logs in or a mail is delivered

    • Lookup object names from user_index_objects for the given username.

    • Lookup object names from user_index_diff_objects for the given username.

    • Download any missing index bundle objects

  • Refreshing folder index

    • Frequency: somewhat often

      • This lookup is usually done only when saving the index to local cache for the first time. Afterwards it’s not done unless the user’s cache becomes invalidated (user modified by another backend or the folder cache deleted).

    • Lookup object names from user_mailbox_index_objects for the given (username, folder GUID).

    • Lookup object names from user_mailbox_index_diff_objects for the given (username, folder GUID).

    • Lookup max_bucket from user_mailbox_buckets for the given (username, folder GUID)

    • Lookup object names from user_mailbox_objects for the given (username, folder GUID, 0..max_bucket). This is necessary for finding any newly delivered emails since the last folder index upload.

    • Download any missing index bundle objects

  • Writing user root self/diff index

    • Frequency: not often

      • The first time this backend modifies the user’s mailbox in any way.

      • Whenever user’s folders are created/renamed/deleted.

    • Insert to user_index_diff_objects (which overwrites the existing row)

    • Upload new index bundle object

    • Delete old index bundle objects

  • Writing user root base index

    • Frequency: rarely

      • Normally self/diff index is updated; only after there have been a lot of changes a new base index is created.

    • Insert to user_index_objects

    • Delete from user_index_objects by (user, object name)

    • Upload new index bundle object

    • Delete old index bundle objects

  • Writing folder diff/self index

    • Frequency: often

      • Every 10th mail delivery

      • Every 5 minutes after IMAP client has changed the folder (flag changes, deletions)

    • Insert to user_mailbox_index_diff_objects (which overwrites the existing row)

    • Upload new index bundle object

    • Delete old index bundle objects

  • Writing folder base index

    • Frequency: not very often

      • Normally self/diff index is updated; only after there have been a lot of changes a new base index is created.

    • Insert to user_mailbox_index_objects

    • Delete from user_mailbox_index_objects by (user, folder GUID, object name)

    • Upload new index bundle object

    • Delete old index bundle objects

  • Delivering a new email via LMTP, or saving a new email via IMAP APPEND

    • Frequency: Write folder index (as described above) for every 10th mail delivery (default)

    • Insert to user_mailbox_objects

    • Upload new email object

  • Reading email

    • Usually no lookup, because object ID is stored also in Dovecot indexes

    • Download email object (unless it’s already in fscache)

  • Deleting email

    • Lookup object ID from user_mailbox_objects_reverse to get list of (user, folder, object name). Dovecot only cares if the result is empty or non-empty.

    • Delete from user_mailbox_objects (user, folder, object name) and user_mailbox_objects_reverse (object ID)

    • Folder index is written lazily within the next 5 minutes

    • Delete email object

  • Copying email

    • Lookup object ID from user_mailbox_objects_reverse to get list of (user, folder, object name). Dovecot only cares if the result is empty or non-empty.

    • Write folder index (as described above) for every 10th mail delivery (default)

  • Moving email

    • This is identical to a combination of copying and then deleting the email.

  • Running “doveadm force-resync”

    • Frequency: Rarely, and always a manual operation

    • Refresh user & folder indexes as described above.

    • Lookup folder GUIDs from user_mailbox_index_diff_objects for the specified user to find any missing folders. With Cassandra this returns several duplicates (one per each index object in folder), which are de-duplicated internally.