On storing CBOR encoded items on stable storage
Sandelman Software Works
mcr+ietf@sandelman.ca
Universität Bremen TZI
Germany
cabo@tzi.org
Internet
CBOR Working Group
Internet-Draft
This document proposes an on-disk format for CBOR objects that
is friendly to common on-disk recognition systems like the Unix file(1) command.
This document is being discussed at: https://github.com/cbor-wg/cbor-magic-number
Introduction
Since very early in computing, operating systems have sought ways to mark which files could be processed by which programs.
For instance, the Unix file(1) command, which has existed since 1973 , has been able to identify many file formats for decades.
Many systems (Linux, MacOS, Windows) will select the correct application based upon the file contents, if the system can not determine it by other means: for instance, the classic MacOS maintained a resource fork that includes media type ("MIME type") information and therefore ideally never needs to know anything about the file.
Other systems do this by file extensions.
While having a media type associated with the file is a better solution in general, when files become disconnected from their type information, such as when attempting to do forensics on a damaged system,
then being able to identify a file type can become very important.
It is noted that in the media type registration, that a magic number is asked for, if available, as is a file extension.
A challenge for the file(1) program is often that it can be confused by the encoding vs the content.
For instance, an Android "apk" used to transfer and store an application may be identified as a ZIP file.
Both OpenOffice or MSOffice files are ZIP files of XML files.
(Unless OpenOffice files are flat (fodp) files, in which case they may appear to be generic XML files.)
As CBOR becomes a more and more common encoding for a wide variety of artifacts, identifying them as just "CBOR" is probably not sufficient.
This document provides a way to encode a magic number into the beginning of a CBOR format file.
Two options are presented: typically a CBOR Protocol author will specify one.
A CBOR Protocol is a specification which uses CBOR as its encoding.
Examples of CBOR Protocols currently under development include CoSWID , and EAT .
COSE itself is considered infrastructure, however the encoding of public keys in CBOR as described in would be an identified CBOR Protocol.
A major inspiration for this document is observing the mess in ASN.1 based systems where most files are PEM encoded, identified by the extension "pem", confusing public keys, private keys, certificate requests and S/MIME content.
These proposals are invasive to how CBOR protocols are written to disk, but in both cases, the
proposed envelope does not require that the tag be transferred on the wire.
In addition to the on-disk identification aspects, there are some protocols which may benefit from having such a magic number on the wire if they presently using a different (legacy) encoding scheme.
The presence of the identifiable magic sequence signals that CBOR is being used or a legacy scheme.
Requirements for a Magic Number
A magic number is ideally a unique fingerprint, present in the first 4 or 8 bytes of the file,
which does not change when the contents change, and does not depend upon the length of the file.
Less ideal solutions have a pattern that needs to be matched, but in which some bytes need to be ignored.
While the Unix file(1) command can be told to ignore bytes, this can lead to ambiguities.
Protocol
There are two variations of this practice. Both use CBOR Tags in a way that results in a deterministic first 8 to 12 bytes.
The CBOR Protocol Specific Tag
CBOR Protocol designers should obtain a tag for each major type of object that they might store on disk.
As there are more than 4 billion available 4-byte tags, there should be little issue in allocating a few to each available CBOR Protocol.
The policy is First Come First Served, so all that is required is an email to IANA, having filled in the small template provided in .
This tag should be allocated by the author of the CBOR Protocol, and to be in the four-byte range, it should be at least 0x01000000 (decimal 16777216) in value.
The use of a sequence of four US-ASCII codes which are mnemonic to the protocol is encouraged, but not required.
For CBOR byte strings that happen to contain a representation that is
described by a CoAP Content-Format Number (,
Subregistry
of ),
a tag number has been allocated in (see
for details and examples).
CBOR Tag Wrapped
This proposal starts with the Self-described CBOR tag, 55799, as
described in .
A second CBOR Tag is then allocated to describe the specific Protocol involved, as described above.
This proposal wraps the CBOR value as tags usually do.
Applications that need to send the CBOR value across a constrained link may wish to remove the two tags if the use is implicitly understood.
This is a decision of the CBOR Protocol specification.
CBOR Tag Sequence
This proposal makes use of CBOR Sequences as described in .
This proposal consists of two tags and a constant string for a total of 12 bytes.
- The file shall start with the Self-described CBOR Sequence tag, 55800.
- The file shall continue with a CBOR tag, from the First Come First Served space, which uniquely identifies the CBOR Protocol. The use of a four-byte tag is encouraged.
- The three byte CBOR byte string containing 0x42_4F_52. When encoded it shows up as "CBOR"
The first part identifies the file as being CBOR, and does so with all
the desirable properties explained in .
Specifically, it does not seem to conflict with any known file types, and it is not valid Unicode in any Unicode encoding.
The second part identifies which CBOR Protocol is used, as described above.
The third part is a constant value 0x43_42_4f_52, "CBOR".
That is, the CBOR encoded data item for the three byte sequence 0x42_4f_52 ("BOR").
This is the data item that is tagged.
The actual CBOR Protocol value then follows as the next data item(s) in the CBOR sequence, without a need for any further specific tag.
The use of a CBOR Sequence allows the application to trivially remove the first item with the two tags.
This means that should a file be reviewed by a human (directly in an editor, or in a hexdump display), it will include the string "CBOR" prominently.
This value is also included simply because the two tags need to tag something.
Advice to Protocol Developers
This document introduces a choice between a CBOR Sequence and a wrapped CBOR Tag.
Which should a protocol designer use?
In this discussion, one assumes that there is an object stored in a file, perhaps specified by a system operator in a configuration file.
For example: a private key used in COSE operations, a public key/certificate in C509 or CBOR format, a recorded sensor reading stored for later transmission, or a COVID vaccination certificate that needs to be displayed in QRcode form.
Both the CBOR Tag Sequence and the wrapped tag can be trivially removed by an application before sending the CBOR content out on the wire.
The CBOR Tag Sequence is a little bit easier to remove as in most cases, CBOR parsers will return it as a unit, and then return the actual CBOR item, which could be anything at all, and could include CBOR tags that do need to be sent on wire.
On the other hand, having the CBOR Tag Sequence in the file requires that all programs that expect to examine that file are able to skip what appears to be an empty CBOR item.
Programs which might not expect the CBOR Tag Sequence, but which would operate without a problem would include any program that expects to process CBOR Sequences from the file.
As an example of where there was a problem with previous security systems, "PEM" format certificate files grew to be able to contain multiple certificates by simple concatenation.
The PKCS1 format could also contain a private key object followed by a one or more certificate objects: but only when in PEM format.
But, when in binary DER format, concatenation of certificates was not compatible with most programs.
The use of CBOR Tag Wrapped format is easier to retrofit to an existing format with existing and unchangeable on-disk format.
This new sequence of tags are expected to be trivially ignored by an existing program when reading CBOR from disk.
But, a naive program might also then transmit them across the network.
Removing the CBOR Tag Wrapped format requires knowledge of the two tags involved.
Other tags present might be needed.
For a representation matching a specific media-type that is carried in
a CBOR byte string, the byte string head will already have to be
removed for use as such a representation, so it should be easy to
remove the enclosing tag heads as well.
This is of particular interest with the pre-defined tags provided by
for media-types with CoAP Content-Format numbers.
Here are some considerations:
Is the on-wire format new?
If the on-wire format is new, then it could be specified with the CBOR Tag Wrapped format if the extra eight bytes are not a problem.
The disk format is then identical to the on-wire format.
If the eight bytes are a problem (and they usually are if CBOR is being considered), then the CBOR Tag Sequence format should be adopted for on-disk storage.
Can many items be trivially concatenated?
If the programs that read the contents of the file already expect to process all of the items in the file (not just the first), then the CBOR Tag Sequence format may be easily retrofitted.
The program involved may throw errors or warnings on the CBOR Tag Sequence if they have not yet been updated, but this may not be a problem.
If it is, then consideration should be given to CBOR Tag Wrapped.
If only one item is ever expected in the file, the the use of CBOR Tag Sequence may present an implementation hurdle to programs that previously just read a single value and used it.
Are there tags at the start?
If the Protocol expects to use other tags values at the top-level, then it may be easier to explain if the CBOR Tag Sequence format is used.
Security Considerations
This document provides a way to identify CBOR Protocol objects.
Clearly identifying CBOR contents on disk may have a variety of impacts.
The most obvious is that it may allow malware to identify interesting objects on disk, and then corrupt them.
IANA Considerations
documents the allocation that was done for a
CBOR tag to be used in a CBOR sequence to identify the sequence.
allocates a CBOR tag for each actual or
potential CoAP Content-Format number.
CBOR Sequence Tag
IANA has allocated tag 55800 as the CBOR Sequence tag.
This tag is from the First Come/First Served area.
The value has been picked to have properties similar to the 55799 tag ().
The hexadecimal representation is: 0xd9_d9_f8.
This is not valid UTF-8: the first 0xd9 puts the value into the three-byte value of UTF-8, but the 0xd9 as the second value is not a valid second byte for UTF-8.
This is not valid UTF-16: the byte sequence 0xd9d9 (in either endian order) puts this value into the UTF-16 high-half zone, which would signal that this a 32-bit Unicode value. However, the following 16-bit big-endian value 0xf8.. is not a valid second sequence according to .
On a little-endian system, it would be necessary to examine the fourth byte to determine if it is valid. That next byte is determined by the subsequent encoding, and has already determined that no valid CBOR encodings result in a valid UTF-16.
-
Data Item:
-
byte string
-
Semantics:
-
indicates that the file contains CBOR Sequences
CBOR Tags for CoAP Content-Format Numbers
IANA is requested to allocate the
tag numbers 1668546560 (0x63740000) to
1668612095 (0x6374FFFF) as follows:
-
Data Item:
-
byte string
-
Semantics:
-
for each tag number NNNN, the representation of content-format (RFC7252) NNNN-1668546560
-
Reference:
-
RFCthis
(More details in .)
References
Normative References
Concise Binary Object Representation (CBOR)
The Concise Binary Object Representation (CBOR) is a data format whose design goals include the possibility of extremely small code size, fairly small message size, and extensibility without the need for version negotiation. These design goals make it different from earlier binary serializations such as ASN.1 and MessagePack.
This document obsoletes RFC 7049, providing editorial improvements, new details, and errata fixes while keeping full compatibility with the interchange format of RFC 7049. It does not create a new version of the format.
Concise Binary Object Representation (CBOR) Sequences
This document describes the Concise Binary Object Representation (CBOR) Sequence format and associated media type "application/cbor-seq". A CBOR Sequence consists of any number of encoded CBOR data items, simply concatenated in sequence.
Structured syntax suffixes for media types allow other media types to build on them and make it explicit that they are built on an existing media type as their foundation. This specification defines and registers "+cbor-seq" as a structured syntax suffix for CBOR Sequences.
Informative References
file (command)
Wikipedia
The Constrained Application Protocol (CoAP)
The Constrained Application Protocol (CoAP) is a specialized web transfer protocol for use with constrained nodes and constrained (e.g., low-power, lossy) networks. The nodes often have 8-bit microcontrollers with small amounts of ROM and RAM, while constrained networks such as IPv6 over Low-Power Wireless Personal Area Networks (6LoWPANs) often have high packet error rates and a typical throughput of 10s of kbit/s. The protocol is designed for machine- to-machine (M2M) applications such as smart energy and building automation.
CoAP provides a request/response interaction model between application endpoints, supports built-in discovery of services and resources, and includes key concepts of the Web such as URIs and Internet media types. CoAP is designed to easily interface with HTTP for integration with the Web while meeting specialized requirements such as multicast support, very low overhead, and simplicity for constrained environments.
Constrained RESTful Environments (CoRE) Parameters
IANA
Concise Software Identification Tags
Fraunhofer SIT
National Security Agency
The MITRE Corporation
National Institute of Standards and Technology
ISO/IEC 19770-2:2015 Software Identification (SWID) tags provide an
extensible XML-based structure to identify and describe individual
software components, patches, and installation bundles. SWID tag
representations can be too large for devices with network and storage
constraints. This document defines a concise representation of SWID
tags: Concise SWID (CoSWID) tags. CoSWID supports a similar set of
semantics and features as SWID tags, as well as new semantics that
allow CoSWIDs to describe additional types of information, all in a
more memory efficient format.
The Entity Attestation Token (EAT)
Qualcomm Technologies Inc.
Security Theory LLC
Qualcomm Technologies Inc.
Qualcomm Technologies Inc.
An Entity Attestation Token (EAT) provides a signed (attested) set of
claims that describe state and characteristics of an entity,
typically a device like a phone or an IoT device. These claims are
used by a relying party to determine how much it wishes to trust the
entity.
An EAT is either a CWT or JWT with some attestation-oriented claims.
To a large degree, all this document does is extend CWT and JWT.
Contributing
TBD
CBOR Object Signing and Encryption (COSE)
Concise Binary Object Representation (CBOR) is a data format designed for small code size and small message size. There is a need for the ability to have basic security services defined for this data format. This document defines the CBOR Object Signing and Encryption (COSE) protocol. This specification describes how to create and process signatures, message authentication codes, and encryption using CBOR for serialization. This specification additionally describes how to represent cryptographic keys using CBOR.
CBOR Encoded X.509 Certificates (C509 Certificates)
RISE AB
RISE AB
Ericsson AB
Ericsson AB
Nexus Group
This document specifies a CBOR encoding of X.509 certificates. The
resulting certificates are called C509 Certificates. The CBOR
encoding supports a large subset of RFC 5280 and all certificates
compatible with the RFC 7925, IEEE 802.1AR (DevID), CNSA, and CA/
Browser Forum Baseline Requirements profiles. When used to re-encode
DER encoded X.509 certificates, the CBOR encoding can in many cases
reduce the size of RFC 7925 profiled certificates with over 50%. The
CBOR encoded structure can alternatively be signed directly
("natively signed"), which does not require re-encoding for the
signature to be verified. The document also specifies COSE headers
as well as a TLS certificate type for C509 certificates.
UTF-16, an encoding of ISO 10646
This document describes the UTF-16 encoding of Unicode/ISO-10646, addresses the issues of serializing UTF-16 as an octet stream for transmission over the Internet, discusses MIME charset naming as described in [CHARSET-REG], and contains the registration for three MIME charset parameter values: UTF-16BE (big-endian), UTF-16LE (little- endian), and UTF-16. This memo provides information for the Internet community.
CBOR Tags for CoAP Content Formats
Often, there is a need to identify a media type (or content type,
i.e., media type optionally used with parameters) that describes a
byte string in a CBOR data item.
defines the concept of a Content-Format,
which is a short 16-bit unsigned integer that identifies a specific
content type (media type plus optionally parameters), optionally
together with a content encoding.
This specification allocates CBOR tag numbers 1668546560 (0x63740000) to
1668612095 (0x6374FFFF) for the tagging of representations of specific
content formats.
The tag content tagged with tag number NNNN (in above range) is a byte
string that is to be interpreted as a representation of the content
format NNNN-1668546560.
Content-Format Tag Examples
Subregistry of defines content formats that can be used as examples:
-
Content-Format 432 stands for media type application/td+json (no
parameters).
The corresponding tag number is 1668546992 (i.e., 1668546560+432).
So the following CDDL snippet can be used to identify
application/td+json representations:
td-json = #6.1668546992(bstr)
Note that a byte string is used as the type of the tag content, because a
media type representation in general can be any byte string.
-
Content-Format 11050 stands for media type application/json in
deflate encoding.
The corresponding tag number is 1668557610 (i.e., 1668546560+11050).
So the following CDDL snippet can be used to identify
application/json representations compressed in deflate encoding:
json-deflate = #6.1668557610(bstr)
The byte string is appropriate here as the type for the tag content,
because the compressed form is an instance of a general byte string.
Example from Openswan
The Openswan IPsec project has a daemon ("pluto"), and two control programs ("addconn", and "whack").
They communicate via a Unix-domain socket, over which a C-structure containing pointers to strings is serialized using a bespoke mechanism.
This is normally not a problem as the structure is compiled by the same compiler; but when there are upgrades it is possible for the daemon and the control programs to get out of sync by the bespoke serialization.
As a result, there are extra compensations to deal with shutting the daemon down.
During testing it is sometimes the case that upgrades are backed out.
In addition, when doing unit testing, the easiest way to load policy is to use the
normal policy reading process, but that is not normally loaded in the daemon.
Instead the IPC that is normally sent across the wire is compiled/serialized and placed in a file.
The above magic number is included in the file, and also on the IPC in order to distinguish the "shutdown" command CBOR operation.
In order to reduce the problems due to serialization, the serialization is being changed to CBOR.
Additionally, this change allows the IPC to be described by CDDL, and for any language that encode to CBOR can be used.
IANA has allocated the tag 1330664270, or 0x4f_50_53_4e for this purpose.
As a result, each file and each IPC is prefixed with:
In diagnostic notation:
Or in hex:
Acknowledgements
The CBOR WG brainstormed this protocol on January 20, 2021.
Contributors
jeffpc@josefsipek.net