IMAP Extension for object identifiersFastMailLevel 2, 114 William StMelbourneVIC 3000Australiabrong@fastmailteam.comhttps://www.fastmail.com
Applications
EXTRAIMAPemailThis document updates RFC3501 (IMAP4rev1) with persistent identifiers on
mailboxes and messages to allow clients to more efficiently re-use cached
data when resources have changed location on the server.
IMAP stores are often used by many clients. Each client may cache data from
the server so that they don't need to re-download information.
defines that a mailbox can be uniquely referenced by
its name and UIDVALIDITY, and a message within that mailbox can be uniquely
referenced by its mailbox (name + UIDVALIDITY) and UID. The triple of
mailbox name, UIDVALIDITY and UID is guaranteed to be immutable.
defines a COPYUID response which allows a client which
copies messages to know the mapping between the UIDs in the
source and destination mailboxes, and hence update its local cache.
If a mailbox is successfully renamed by a client, that client will know
that the same messages exist in the destination mailbox name as previously
existed in the source mailbox name.
The result is that the client which copies (or moves) messages or
renames a mailbox can update its local cache, but any other client connected
to the same store can not know with certainty that the messages are identical,
and so will re-download everything.
This extension adds new properties to a message (EMAILID) and mailbox (MAILBOXID)
which allow a client to quickly identify messages or mailboxes which have been
renamed by another client.
This extension also adds an optional thread identifier (THREADID) to messages,
which can be used by the server to indicate messages which it has identified
to be related. A server that does not implement threading will return NIL
to all requests for THREADID.
In examples, "C:" indicates lines sent by a client that is connected
to a server. "S:" indicates lines sent by the server to the client.
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL",
"SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED",
"MAY", and "OPTIONAL" in this document are to be interpreted as
described in when they appear in ALL CAPS. These words may
also appear in this document in lower case as plain English words,
absent their normative meanings.
IMAP servers that support this extension MUST include "OBJECTID" in
the response list to the CAPABILITY command.
The MAILBOXID is a server-allocated unique identifer for each mailbox.
The server MUST return the same MAILBOXID for a mailbox with the same
name and UIDVALIDITY.
The server MUST NOT report the same MAILBOXID for two mailboxes at the
same time.
The server MUST NOT reuse the same MAILBOXID for a mailbox which does not
obey all the invarients that defines for a mailbox which does
not change name or UIDVALIDITY.
The server SHOULD NOT change MAILBOXID when renaming a folder, as this
loses the main benefit of having a unique identifier.
This document extends the CREATE command to have the response code MAILBOXID
on successful mailbox creation.
A server advertising the OBJECTID capability MUST include the MAILBOXID
response code in the tagged OK response to all successful CREATE commands.
Syntax: "MAILBOXID" SP "(" <objectid> ")"
Example:
This document adds a new untagged response code to the SELECT and
EXAMINE commands.
A server advertising the OBJECTID capability MUST return an untagged
OK response with the MAILBOXID response code on all successful SELECT
and EXAMINE commands.
Syntax: "OK" SP "[" "MAILBOXID" SP "(" <objectid> ")" "]" text
Example:
This document adds the MAILBOXID attribute to the STATUS command using
the extended syntax defined in .
A server that advertises the OBJECTID capability MUST support the
MAILBOXID status attribute.
Syntax: "MAILBOXID"
Syntax: "MAILBOXID" SP "(" <objectid> ")"
Example:
When the LIST-STATUS IMAP capability defined in is also
available, the STATUS command can be combined with the LIST command.
Example:
The EMAILID data item is an objectid which uniquely identifies the
content of a single message. Anything which must remain
immutable on a {name, uidvalidity, uid} triple must also be the
same between messages with the same EMAILID.
The server MUST return the same EMAILID for the same triple,
hence EMAILID is immutable.
The server MUST return the same EMAILID as the source message for
the matching destination message in the COPYUID pairing after a COPY
or MOVE command.
The server MAY assign the same EMAILID as an existing message upon
APPEND (e.g. if it detects that the new message has exactly identical
content to that of an existing message)
NOTE: EMAILID only identifies the immutable content of the message. In
particular, it is possible for different messages with the same EMAILID
to have different keywords. This document does not specify a way to
STORE by EMAILID.
The THREADID data item is an objectid which uniquely identifies
a set of messages which the server believes should be grouped
together when presented.
THREADID calculation is generally based on some combination of References,
In-Reply-To and Subject, but the exact logic is left up to the server
implementation. describes some algorithms that could be used,
however this specfication does not mandate any particular strategy.
The server MUST return the same THREADID for all messages with the
same EMAILID.
The server SHOULD return the same THREADID for related messages even
if they are in different mailboxes.
The server MUST NOT change the THREADID of a message once reported.
THREADID is optional, if the server doesn't support THREADID or
is unable to calculate relationships between messages, it MUST return
NIL to all FETCH responses for the THREADID data item, and a SEARCH for
THREADID MUST NOT match any messages.
The server MUST NOT use the same objectid value for both EMAILIDs
and THREADIDs. If they are stored with the same value internally, the
server can generate prefixed values (as shown in the examples below
with M and T prefixes) to avoid clashes.
This document defines two FETCH items:
Syntax: EMAILID
Syntax: THREADID
And the following responses:
Syntax: EMAILID ( <objectid> )
Syntax: THREADID ( <objectid> )
Syntax: THREADID NIL
Example:
Example: (no THREADID support)
This document defines filters EMAILID and THREADID on the SEARCH command.
EMAILID <objectid>
THREADID <objectid>
Example: (as if run before the MOVE above when the mailbox had 3 messages)
The following syntax specification uses the Augmented Backus-Naur
Form (ABNF) notation. Elements not defined here can be found
in the formal syntax of the ABNF , IMAP , and IMAP
ABNF extensions specifications.
Except as noted otherwise, all alphabetic characters are case-
insensitive. The use of upper- or lowercase characters to define
token strings is for editorial clarity only. Implementations MUST
accept these strings in a case-insensitive fashion.
capability =/ "OBJECTID"
fetch-att =/ "EMAILID" / "THREADID"
fetch-emailid-resp = "EMAILID" SP "(" objectid ")"
; follows tagged-ext production from fetch-threadid-resp = "THREADID" SP "(" objectid ")" /
"THREADID" NIL
; follows tagged-ext production from objectid = 1*255(ALPHA / DIGIT / "_" / "-")
; characters in object identifiers are case
; significant
resp-text-code =/ "MAILBOXID" SP "(" objectid ")"
; incorporated before the expansion rule of
; atom [SP 1*<any TEXT-CHAR except "]">]
; that appears in search-key =/ "EMAILID" SP objectid / "THREADID" SP objectid
status-att =/ "MAILBOXID"
status-att-value =/ "MAILBOXID" SP "(" objectid ")"
; follows tagged-ext production from All objectid values are allocated by the server.
In the interests of reducing the possibilities of encoding mistakes, objectids
are restricted to a safe subset of possible byte values, and in order to allow
clients to allocate storage, they are restricted in length.
An objectid is a string of 1 to 255 characters from the following set of
64 codepoints. a-z, A-Z, 0-9, '_', '-'. These characters are safe to use
in almost any context (e.g. filesystems, URIs, IMAP atoms).
For maximum safety, servers should also follow defensive allocation strategies
to avoid creating risks where glob completion or data type detection may be
present (e.g. on filesystems or in spreadsheets). In particular it is wise to
avoid:
ids starting with -ids starting with digitsids which contain only digitsids which differ only by ASCII case (A vs a)the specific sequence of 3 characters NILA good solution to these issues is to prefix every ID with a single
alphabetical character.
The case of RENAME INBOX may need special handling, as it has special
behaviour as defined in section 6.3.5.
It is advisable (though not required) to have MAILBOXID be globally
unique, but it is only required to be unique within messages offered
to a single client login to a single server hostname. For example,
a proxy which aggregates multiple independent servers MUST NOT
advertise the OBJECTID capability unless it can guarantee that
different objects will never use the same identifiers, even if
backend object collide.
Servers that implement both RFC 6154 and this specification should
optimize their execution of command like UID SEARCH OR EMAILID 1234 EMAILID
4321.
Clients can assume that searching the all-mail mailbox using
OR/EMAILID or OR/THREADID is a fast way to find messages again if some
other client has moved them out of the mailbox where they were previously
seen.
Clients that cache data offline should fetch the EMAILID of all new
messages to avoid re-downloading already cached message details.
Clients should fetch the MAILBOXID for any new mailboxes before
discarding cache data for any mailbox which is no longer present
on the server, so that they can detect renames and avoid re-downloading
data.
This extension is intentionally defined to be compatible with the data
model in .
A future extension could be proposed to give a way to SELECT a mailbox
by MAILBOXID rather than name.
A future extension to could allow fileinto by MAILBOXID
rather than name.
An extension to allow fetching message content directly via EMAILID and
message listings by THREADID could be proposed.
IANA is requested to add "OBJECTID" to the "IMAP Capabilities"
registry located at .
IANA is requested to add "MAILBOXID" to the "IMAP Response Codes"
registry located at
with a Reference of [[THIS RFC]].
It is strongly advised that servers generate OBJECTIDs which are safe to
use as filesystem names, and unlikely to be auto-detected as numbers. See
implementation considerations.
If a digest is used for ID generation, it must have a collision resistent
property, so server implementations are advised to monitor current security
research and choose secure digests. As the IDs are generated by the server,
it will be possible to migrate to a new hash by just creating new IDs with
the new algorithm. This is particularly true if a prefix is used on each ID,
which can be changed when the algorithm changes.
The use of a digest for ID generation may be used as proof that a particular
sequence of bytes was seen by the server, however this is only a risk if IDs
are leaked to clients who don't have permission to fetch the data directly.
Servers that are expected to handle highly sensitive data should consider
using a ID generation mechanism which doesn't derive from a digest.
See also the security considerations in section 11.
To be removed by the editor before publication
changed some SHOULD to lower case in advice sections (genart review)clarified that THREADID MUST NOT changedescribed NIL THREADID in more detail (ad review)made RFC5256 a normative reference (ad review)fixed ABNF missing quote (ad review)documented hash upgrade process (ad review)referenced RFC3501 for INBOX rename (ad review)referenced RFC3501 security considerations (secdir review)turned mealy-mouthed "SHOULDs" in to "MUSTs" on immutability (genart review)remove suggested algorithms which are no longer legitimate (genart review)updated proxy advice to suggest rewriting ids (genart review)fixed minor gramatical issues (genart review)required that EMAILID and THREADID are not identical (own decision)added RFC3501 to Abstractupdated [[THIS RFC]] to not fail idnitschanged jmap-mail to be informative rather than normativeshortened IDs to stop wrapping and outdents in IMAP examplesadded "Client usage" sectionadded "updates" for RFC3501fixed domains in thread exampledescribed threading in more detailadded IANA request for Response Codeclarified RFC2119 referencessimplified some waffle in wordingadded security consideration to choose good digestadded MAILBOXID-UID suggestion for EMAILID generationupdated ABNF normative reference to RFC5234renamed draft to be objectid rather than uniqueidrenamed UNIQUEID (capability) to OBJECTIDrestricted objectid to 64 safe charactersadded security considerations and advice about choosing objectidwrapped all responses in () for RFC4466 compatibilitysignifiant rewrite of all sectionsrenamed draft to be an EXTRA documentadded example for LIST RETURN STATUSstarted work on ABNFattempted to add response codes for EMAILID and THREADIDrenamed UNIQUEID (status item) to MAILBOXIDrenamed MSGID to EMAILIDrenamed THRID to THREADIDadded TODO sectioninitial upload with names UNIQUEID/MSGID/THRIDThe EXTRA working group at IETF. In particular feedback from
Arnt Gulbrandsen, Brandon Long, Chris Newman and Josef Sipek.
The Gmail X-GM-THRID and X-GM-MSGID implementation as currently defined
at .
Dovecot X-GUID implementation.
Ideas for calculating MAILBOXID:
UUIDServer assigned sequence number (guaranteed not to be reused)Ideas for implementing EMAILID:
Digest of message content (RFC822 bytes) - expensive unless cached UUIDServer assigned sequence number (guaranteed not to be reused)Ideas for implementing THREADID:
Derive from EMAILID of first seen message in the thread. UUIDServer assigned sequence number (guaranteed not to be reused)There is a need to index and look up reference/in-reply-to data
at message creation to efficiently find matching messages
for threading. Threading may be either across folders, or within
each folder only. The server has significant leeway here.