Network Working Group P. Hallam-Baker
Internet-Draft Comodo Group Inc.
Intended status: Informational August 27, 2018
Expires: February 28, 2019

Data At Rest Encryption Part 2: DARE Container
draft-hallambaker-dare-container-02

Abstract

This document describes DARE Container, a message and file syntax that allows an append-only sequence of data frames to be represented with cryptographic integrity, signature and encryption enhancements. The format supports data integrity checks using digest chains and Merkle trees. The simplest supports efficient write operations and efficient read operations in either the forward or reverse direction. Support for efficient random-access reads may be provided through the use of binary trees or index records appended to the end of the file.

This document is also available online at http://mathmesh.com/Documents/draft-hallambaker-dare-container.html .

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."

This Internet-Draft will expire on February 28, 2019.

Copyright Notice

Copyright (c) 2018 IETF Trust and the persons identified as the document authors. All rights reserved.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.


Table of Contents

1. Introduction

DARE Container is a message and file syntax that allows a sequence of data frames to be represented with cryptographic integrity, signature, and encryption enhancements to be constructed in an append only format. DARE Container was developed in response to needs that arose out of the design of the Mathematical Mesh [draft-hallambaker-mesh-architecture] . It is built on the binary encodings of JSON data objects, JSON-B and JSON-C [draft-hallambaker-jsonbcd] and the DARE Message format [draft-hallambaker-dare-message] .

The format is designed to meet the requirements of a wide range of use cases including:

1.1. Container Format

A DARE Container consists of a sequence of variable length frames. Each frame consists of a forward length indicator, the framed data and a reverse length indicator. The reverse length indicator is written out backwards allowing the length and thus the frame to be read in the reverse direction:

[[This figure is not viewable in this format. The figure is available at http://mathmesh.com/Documents/draft-hallambaker-dare-container.html.]]


JBCD Bidirectional Frame

Each frame contains a single DARE Message consisting of a Header, Payload and Trailer (if required). The first frame in a container describes the container format options and defaults. These include the range of encoding options for frame metadata supported and the container profiles to which the container conforms.

All internal data formats are 64 bit clean allowing for containers of up to 18 exabytes to be written.

Five container types are currently specified but this may be reduced if the Digest and Tree types are withdrawn. These are:

Simple
The container does not provide any index or content integrity checks.
Tree
Frame headers contain entries that specify the start position of previous frames at the apex of the immediately enclosing binary tree. This enables efficient random access to any frame in the file.
Digest
Each frame trailer contains a PayloadDigest field. Modification of the payload will cause verification of the PayloadDigest value to fail on that frame.
Chain
Each frame trailer contains PayloadDigest and ChainDigest fields allowing modifications to the payload data to be detected. Modification of the payload will cause verification of the PayloadDigest value to fail on that frame and verification of the ChainDigest value to fail on all subsequent frames.
Merkle Tree
Frame headers contain entries that specify the start position of previous frames at the apex of the immediately enclosing binary tree. Frame Trailers contain TreeDigestPartial and TreeDigestFinal entries forming a Merkle digest tree.

1.2. Write

In normal circumstances, DARE Containers are written as an append only log. As with DARE Messages, integrity information (payload digest, signatures) is written to the message trailer. Thus, large payloads may be written without the need to buffer the payload data provided that the content length is known in advance.

1.3. Read Access

The use of reverse length indicators allows DARE containers to support efficient sequential access in either the forward or reverse directions.

Random access to any part of a file MAY be supported by means of a binary tree index and/or an index record providing direct access to any part of the file.

1.4. Encryption and Authentication

Frame payloads and associated attributes MAY be encrypted and/or authenticated using the approach described in [draft-hallambaker-dare-message] .

Incremental encryption is supported allowing encryption parameters from a single public key exchange operation to be applied to encrypt multiple frames. The public key exchange information is specified in the first encrypted frame and subsequent frames encrypted under those parameters specify the location at which the key exchange information is to be found by means of the ExchangePosition field.

The only restriction on the use of incremental encryption is that the frame containing the key exchange information MUST precede the frames that reference the exchange parameters.

To avoid cryptographic vulnerabilities resulting from key re-use, the DARE key exchange requires that each encrypted sequence use an encryption key and initialization vector derived from the master key established in the public key exchange by means of a unique salt.

Each DARE Message and by extension, each DARE Container frame MUST specify a unique salt value of at least 128 bits. Since the encryption key is derived from the salt value by means of a Key Derivation Function, erasure of the salt MAY be used as a means of rendering the payload plaintext value inaccessible without changing the payload value.

1.5. Integrity and Signature

Signatures MAY be applied to a payload digest, the final digest in a chain or tree. The chain and tree digest modes allow a single signature to be used to authenticate all frame payloads in a container.

The tree signature mode is particularly suited to applications such as file archives as it allows files to be verified individually without requiring the signer to sign each individually. Furthermore, in applications such as code signing, it allows a single signature to be used to verify both the integrity of the code and its membership of the distribution.

As with DARE Message, the signature mechanism does not specify the interpretation of the signature semantics. The presence of a signature demonstrates that the holder of the private key applied it to the specified digest value but not their motive for doing so. Describing such semantics is beyond the scope of this document and is deferred to future work.

1.6. Redaction

The chief disadvantage of using an append-only format is that containers only increase in size. In many applications, much of the data in the container becomes redundant or obsolete and a process analogous to garbage collection is required. This process is called redaction.

The simplest method of redaction is to create a new container and sequentially copy each entry from the old container to the new, discarding redundant frames and obsolete header information.

For example, partial index records may be consolidated into a single index record placed in the last frame of the container. Unnecessary signature and integrity data may be discarded and so on.

It is also possible but not necessarily advisable to perform such a redaction in-place provided that the redaction process does not increase the length of any individual frame and that appropriate provision is made for file locking to prevent conflicts and to provide for safe resumption should an interruption occur during the process.

1.7. Alternative approaches

Many file proprietary formats are in use that support some or all of these capabilities but only a handful have public, let alone open, standards. DARE Container is designed to provide a superset of the capabilities of existing message and file syntaxes, including:

Attempting to make use of these specifications in a layered fashion would require at least three separate encoders and introduce unnecessary complexity. Furthermore, there is considerable overlap between the specifications providing multiple means of achieving the same ends, all of which must be supported if decoders are to work reliably.

1.8. Efficiency

Every data format represents a compromise between different concerns, in particular:

Compactness
The space required to record data in the encoding.
Memory Overhead
The additional volatile storage (RAM) required to maintain indexes etc. to support efficient retrieval operations.
Number of Operations
The number of operations required to retrieve data from or append data to an existing encoded sequence.
Number of Disk Seek Operations
Optimizing the response time of magnetic storage media to random access read requests has traditionally been one of the central concerns of database design. The DARE Container format is designed to the assumption that this will cease to be a concern as solid state media replaces magnetic.

While the cost of storage of all types has declined rapidly over the past decades, so has the amount of data to be stored. DARE Container represents a pragmatic balance of these considerations for current technology. In particular, since payload volumes are likely to be very large, memory and operational efficiency are considered higher priorities than compactness.

2. Definitions

2.1. Related Specifications

DARE Container makes use of the following related standards and specifications.

Encoding
Content frame headers are encoded using JavaScript Object Notation (JSON) [RFC7159] , JSON-B or JSON-C [draft-hallambaker-jsonbcd] .
Cryptography
The encryption and signature schemes used are based on JSON Web Signature [RFC7515] and JSON Web Encryption [RFC7516] as applied in DARE Message [draft-hallambaker-dare-message] .

2.2. Requirements Language

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119] .

3. Container Navigation

Three means of locating frames in a container are supported:

Sequential
Access frames sequentially starting from the start or the end of the container.
Binary search
Access any container frame by frame number in O(log2(n)) time by means of a binary tree constructed while the container is written.
Index
Access and container frame by frame number or by key by means of an index record.

All DARE Containers support sequential access. Only tree and Merkle tree containers support binary search access. An index frame MAY be written appended to any container and provides O(1) access to any frame listed in the index.

Two modes of compilation are considered:

Monolithic
Frames are added to the container in a single operation, e.g. file archives,
Incremental
Additional frames are written to the container at various intervals after it was originally created, e.g. server logs, message spools.

In the monolithic mode, navigation requirements are best met by writing an index frame to the end of the container when it is complete. It is not necessary to construct a binary search tree unless a Merkle tree integrity check is required.

In the incremental mode, Binary search provides an efficient means of locating frames by frame number but not by key. Writing a complete index to the container every m write operations provides O(m) search access but requires O(n2) storage.

Use of partial indexes provides a better compromise between speed and efficiency. A partial index is written out every m frames where m is a power of two. A complete index is written every time a binary tree apex record is written. This approach provides for O(log2(n)) search with incremental compilation with approximately double the overhead of the monolithic case.

3.1. Tree

Binary search is supported by means of the TreePosition parameter specified in the FrameHeader. This parameter specifies the value of the immediately preceding apex.

Calculation of the immediately preceding apex is most easily described by representing the array index in binary with base of 1 (rather than 0). An array index that is a power of 2 (2, 4, 8, 16, etc.) will be the apex of a complete tree. Every other array index has the value of the sum of a set of powers of 2 and the immediately preceding apex will be the value of the next smallest power of 2 in the sum.

For example, to find the immediately preceding apex for frame 5, we add 1 to get 6. 6 = 4 + 2, so we ignore the 2 and the preceding frame is 4.

The values of Tree Position are shown for the first 8 frames in figure xx below:

[[This figure is not viewable in this format. The figure is available at http://mathmesh.com/Documents/draft-hallambaker-dare-container.html.]]


Merkle Tree Integrity check

An algorithm for efficiently calculating the immediately preceding apex is provided in Appendix C.

3.2. Position Index

Contains a table of frame number, position pairs pointing to prior locations in the file.

3.3. Metadata Index

Contains a list of IndexMeta entries. Each entry contains a metadata description and a list of frame indexes (not positions) of frames that match the description.

4. Integrity Mechanisms

Frame sequences in a DARE container MAY be protected against a frame insertion attack by means of a digest chain, a binary Merkle tree or both.

4.1. Digest Chain calculation

A digest chain is simple to implement but can only be verified if the full chain of values is known. Appending a frame to the chain has O(1) complexity but verification has O(n) complexity:

[[This figure is not viewable in this format. The figure is available at http://mathmesh.com/Documents/draft-hallambaker-dare-container.html.]]


Hash chain integrity check

The value of the chain digest for the first frame (frame 0) is H(H(null)+H(Payload0)), where null is a zero length octet sequence and payloadn is the sequence of payload data bytes for frame n

The value of the chain digest for frame n is H(H(Payloadn-1 +H(Payloadn)), where A+B stands for concatenation of the byte sequences A and B.

4.2. Binary Merkle tree calculation

The tree index mechanism describe earlier may be used to implement a binary Merkle tree. The value TreeDigest specifies the apex value of the tree for that node.

Appending a frame to the chain has O(log2n) complexity provided that the container format supports at least the binary tree index. Verifying a chain has O(log2 n) complexity, provided that the set of necessary digest inputs is known.

To calculate the value of the tree digest for a node, we first calculate the values of all the sub trees that have their apex at that node and then calculate the digest of that value and the immediately preceding local apex.

5. Reference

TBS stuff

5.1. Container Headers

TBS stuff

5.1.1. Structure: ContainerHeaderFirst

Inherits: ContainerHeader
Inherits: ContainerHeader
DataEncoding: String (Optional)
Specifies the data encoding for the header section of for the following frames. This value is ONLY valid in Frame 0 which MUST have a header encoded in JSON.

5.1.2. Structure: ContainerHeader

Inherits: DAREHeader

Describes a container header. A container header MAY contain any DARE Message header.

Index: Integer (Optional)
The record index within the file. This MUST be unique and satisfy any additional requirements determined by the ContainerType.
ContainerType: String (Optional)
Specifies the container type for the following records.
IsMeta: Boolean (Optional)
If true, the current frame is a meta frame and does not contain a payload.
Note: Meta frames MAY be present in any container. Applications MUST accept containers that contain meta frames at any position in the file. Applications MUST NOT interpret a meta frame as a data frame with an enpty payload.
UniqueID: String (Optional)
Unique object identifier
ContentMeta: ContentMeta (Optional)
Content meta data.
TreePosition: Integer (Optional)
Position of the frame containing the apex of the preceding sub-tree.
IndexPosition: Integer (Optional)
Specifies the position in the file at which the last index entry is to be found
ExchangePosition: Integer (Optional)
Specifies the position in the file at which the key exchange data is to be found
ContainerIndex: ContainerIndex (Optional)
An index of records in the current container up to but not including this one.
PayloadDigest: Binary (Optional)
If present, contains the digest of the Payload.
ChainDigest: Binary (Optional)
If present, contains the digest of the PayloadDigest values of this frame and the frame immediately preceding.
TreeDigest: Binary (Optional)
If present, contains the Binary Merkle Tree digest value.
Event: String (Optional)
Unique object identifier
Labels: String [0..Many]
List of labels that are applied to the payload of the frame.
KeyValues: KeyValue [0..Many]
List of key/value pairs describing the payload of the frame.
First: Integer (Optional)
Frame number of the first object instance value.
Previous: Integer (Optional)
Frame number of the immediately prior object instance value

5.2. Content Metadata Structure

TBS stuff

5.2.1. Structure: ContentMeta

Information describing the object instance

ContentType: String (Optional)
The content type field as specified in JWE
Paths: String [0..Many]
List of filename paths for the payload of the frame.
UniqueID: String (Optional)
Unique object identifier
Created: DateTime (Optional)
Initial creation date.
Modified: DateTime (Optional)
Date of last modification.

5.3. Index Structures

TBS stuff

5.3.1. Structure: ContainerIndex

A container index

Full: Boolean (Optional)
If true, the index is complete and contains position entries for all the frames in the file. If absent or false, the index is incremental and only contains position entries for records added since the last frame containing a ContainerIndex.
Positions: IndexPosition [0..Many]
List of container position entries
Metas: IndexMeta [0..Many]
List of container position entries

5.3.2. Structure: IndexPosition

Specifies the position in a file at which a specified record index is found

Index: Integer (Optional)
The record index within the file.
Position: Integer (Optional)
The record position within the file relative to the index base.

5.3.3. Structure: KeyValue

Specifies a key/value entry

Key: String (Optional)
The key
Value: String (Optional)
The value corresponding to the key

5.3.4. Structure: IndexMeta

Specifies the list of index entries at which a record with the specified metadata occurrs.

Index: Integer [0..Many]
List of record indicies within the file where frames matching the specified criteria are found.
ContentType: String (Optional)
Content type parameter
Paths: String [0..Many]
List of filename paths for the current frame.
Labels: String [0..Many]
List of labels that are applied to the current frame.

5.4. Signature

Payload data MAY be signed using a JWS [RFC7515] as applied in the DARE Message format [draft-hallambaker-dare-message] .

Signatures are specified by the Signatures parameter in the content header. The data that the signature is calculated over is defined by the typ parameter of the Signature as follows.

Payload
The value of the PayloadDigest parameter
Chain
The value of the ChainDigest parameter
Tree
The value of the TreeDigestFinal parameter

If the typ parameter is absent, the value Payload is implied.

A frame MAY contain multiple signatures created with the same signing key and different typ values.

The use of signatures over chain and tree digest values permit multiple frames to be validated using a single signature verification operation.

6. Security Considerations

7. IANA Considerations

8. Acknowledgements

9. Appendix A: Examples and Test Vectors

The data payloads in all the following examples are identical, only the authentication and/or encryption is different.

For conciseness, the raw data format is omitted for examples after the first, except where the data payload has been transformed, (i.e. encrypted).

9.1. Simple container

the following example shows a simple container with first frame and a single data frame:

f4 5d 
f0 59 
f0 00 
5d f4 
f5 01 40 
f0 0f 
f1 01 2c 
40 01 f5 

Figure 1

Since there is no integrity check, there is no need for trailer entries. The header values are:

Frame 0

{
  "Index":0,
  "ContainerType":"List",
  "ContentMeta":{},
  "DataEncoding":"JSON"}

[Empty trailer]

Figure 2

Frame 1

{
  "Index":1}

[Empty trailer]

Figure 3

9.2. Payload and chain digests

The following example shows a chain container with a first frame and three data frames. The headers of these frames is the same as before but the frames now have trailers specifying the PayloadDigest and ChainDigest values:

Frame 0

{
  "Index":0,
  "ContainerType":"Chain",
  "ContentMeta":{},
  "DataEncoding":"JSON"}

[Empty trailer]

Figure 4

Frame 1

{
  "Index":1}

{
  "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZDlZe
  aWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw",
  "ChainDigest":"T7S1FcrgY3AaWD4L-t5W1K-3XYkPTcOdGEGyjglTD6yMYVRVz9
  tn_KQc6GdA-P4VSRigBygV65OEd2Vv3YDhww"}

Figure 5

Frame 2

{
  "Index":2}

{
  "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZDlZe
  aWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw",
  "ChainDigest":"T7S1FcrgY3AaWD4L-t5W1K-3XYkPTcOdGEGyjglTD6yMYVRVz9
  tn_KQc6GdA-P4VSRigBygV65OEd2Vv3YDhww"}

Figure 6

Frame 3

{
  "Index":3}

{
  "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZDlZe
  aWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw",
  "ChainDigest":"T7S1FcrgY3AaWD4L-t5W1K-3XYkPTcOdGEGyjglTD6yMYVRVz9
  tn_KQc6GdA-P4VSRigBygV65OEd2Vv3YDhww"}

Figure 7

9.3. Merkle Tree

The following example shows a chain container with a first frame and six data frames. The trailers now contain the TreePosition and TreeDigest values:

Frame 0

{
  "Index":0,
  "ContainerType":"Merkle",
  "ContentMeta":{},
  "DataEncoding":"JSON"}

[Empty trailer]

Figure 8

Frame 1

{
  "Index":1,
  "TreePosition":0}

{
  "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZDlZe
  aWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw",
  "TreeDigest":"T7S1FcrgY3AaWD4L-t5W1K-3XYkPTcOdGEGyjglTD6yMYVRVz9t
  n_KQc6GdA-P4VSRigBygV65OEd2Vv3YDhww"}

Figure 9

Frame 2

{
  "Index":2,
  "TreePosition":319}

{
  "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZDlZe
  aWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw",
  "TreeDigest":"7fHmkEIsPkN6sDYAOLvpIJn5Dg3PxDDAaq-ll2kh8722kokkFnZ
  QcYtjuVC71aHNXI18q-lPnfRkmwryG-bhqQ"}

Figure 10

Frame 3

{
  "Index":3,
  "TreePosition":319}

{
  "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZDlZe
  aWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw",
  "TreeDigest":"T7S1FcrgY3AaWD4L-t5W1K-3XYkPTcOdGEGyjglTD6yMYVRVz9t
  n_KQc6GdA-P4VSRigBygV65OEd2Vv3YDhww"}

Figure 11

Frame 4

{
  "Index":4,
  "TreePosition":1451}

{
  "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZDlZe
  aWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw",
  "TreeDigest":"vJ6ngNATvZcXSMALi5IUqzl1GBxBnTNVcC87VL_BhMRCbAvKSj8
  gs0VFgxxLkZ2myrtaDIwhHoswiTiBMLNWug"}

Figure 12

Frame 5

{
  "Index":5,
  "TreePosition":1451}

{
  "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZDlZe
  aWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw",
  "TreeDigest":"T7S1FcrgY3AaWD4L-t5W1K-3XYkPTcOdGEGyjglTD6yMYVRVz9t
  n_KQc6GdA-P4VSRigBygV65OEd2Vv3YDhww"}

Figure 13

Frame 6

{
  "Index":6,
  "TreePosition":2586}

{
  "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZDlZe
  aWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw",
  "TreeDigest":"WgHlz3EHczVPqgtpc39Arv7CFIsCbFVsk8wg0j2qLlEfur9SZ0m
  dr65Ka-HF0Qx8gg_DAoiJwUrwADDXyxVJOg"}

Figure 14

9.4. Signed container

The following example shows a tree container with a signature in the final record. The signing key parameters are:

{
  "PrivateKeyRSA":{
    "kid":"MBWNO-2J43U-ESWKW-XQWL6-6YGEW-UOPWU-A",
    "n":"1NzbmakMalVH1mRv7TEDEhwXDNojn5wZbq1tv1gp5PgZwzX-klYXuFhj0-
  MpO0zcwptsUaYJhwdvvgW_1udUpISQYluXOB3UMj2e_0yl64MvnqTL47SZQuAN3QQ
  9cuCw_-_Eyj_jerspauqa6RpNzGcabZrtRl7J7DPVZ3SNlw-H_Wxd4HkrFVW_Yqup
  htNL1JciQJYm2DSu9dbetqPZ80x6IBargY850mBYOzvNNE5S-dRJQHoJY4SG-ESYF
  BuAHtBlOMgbI0XNiq96EegA-vPW9XRF-SHdX5mkafefDGK4rT_RoE4gRwhDM3jbZ8
  1-F2ZA_GpNVEvB-25_vF96lQ",
    "e":"AQAB",
    "d":"k_v_h7Jo-TvUt44X6jSax-pTdBHrljk1zSYxGEe4yIBbmMVe-Gl2ECkTLe
  nNbnafO4RGJ_Vgxkk7PEZO-p7Uz5OBtX-rf83tCgihEyg8aaFIZ-h1_xY9Pqr5uGA
  MQGNJaoVMsLb99QNNZhE4JTquP56mVvDQaI3Zn6bhhA0ZqpxS_x6iRUV5KnHCRd47
  DKGHcAsQR_caxGec7M_XNPqpl0tcoGQz46-I0SVVcqtjb_YysEVez4eJhb_ZTU4C7
  pz4wXDj6B0ppFJVZEMaIhKo8FCnoQdXEJl4vSiBzUFrSlPc6gjQk2CBhxc_kb782w
  VwW5wtgOVxhV1k2piQ2NrlOQ",
    "p":"6osXfrJLiPfK0ky17vqzRs-M5mOxZU2LEFGyHTXxx6EYTWixEsx2Sdf6kx
  UD5K_QdYnqqd3yobbGdDpMwwEuwgogWVCz90nc9QdCUy4MCpz8lpOQd1i_tXMmtOg
  GBu9mapMTEOwc7HlLlSDRezv_TzTAH4izv-CUEZ_M5EcwyFM",
    "q":"6FYD6NV4rKU1ACtGQyIvwGrkrS_F6phB-whx0nSVFkbkwppJiPnC6XqjZ3
  OYPCZylxaTHFnxCs0nntrraeNEcWfNPrpTN5XIZjbOiIaKA2iCQkWlDoi8oduEtTK
  oIcuy32oBz6MpUeCWzl0ZQ4EeTF3RyCc_jpf8oRvZ0e3ItHc",
    "dp":"hfjbe9BmWx-HqCaPSanEW-9UQYmym_X2OGUiA5N7vxci5ZymgOFvs_B9v
  iQj7C4NOgaEl3EjFgJsS5m9nSoAxm-4WKxDkD6NyxzRYugLkshnc69otvNn1kKnWn
  CqeK2o57mJC4KDZwRGCzIK1oTH6jtsfta8Lh8fFQ4doEuV7uc",
    "dq":"r6R_ViE0Foja1aLhflU09mmZMViBbkXm86nBqtHZ97pmrLvJRdVTxgCh0
  c6w0yBZ1uEJHBDeykSoZE6qVCWtE3Le1kI0MTx6ANQENXBInCUA_Kr8Ck3TFSYIYJ
  fIRaxiMMZKUjfOQAji2WXGeKL_TcpLkt4hDWLXaNDOTgdOiSc",
    "qi":"DfHtLB1Ox1Kgp3E4jqy5Qxeb7-v7_uv8n_5-E1OQ3NLSRV2m_auojkR19
  nY3gokHKNSXM41qKlJLU00lROjOO2KUq57s8GZkheVfbJLNCJ6KAw_aRT2IgyJm2b
  e2v5OCHSkm88tgJWbtKj-OPKTFV5gOMVdeCzGX286ErjDHGCM"}}

Figure 15

The container headers and trailers are:

Frame 0

{
  "Index":0,
  "ContainerType":"Merkle",
  "ContentMeta":{},
  "DataEncoding":"JSON"}

[Empty trailer]

Figure 16

Frame 1

{
  "Index":1,
  "TreePosition":0}

{
  "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZDlZe
  aWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw",
  "TreeDigest":"T7S1FcrgY3AaWD4L-t5W1K-3XYkPTcOdGEGyjglTD6yMYVRVz9t
  n_KQc6GdA-P4VSRigBygV65OEd2Vv3YDhww"}

Figure 17

Frame 2

{
  "dig":"S512",
  "Index":2,
  "TreePosition":319}

{
  "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZDlZe
  aWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw",
  "TreeDigest":"7fHmkEIsPkN6sDYAOLvpIJn5Dg3PxDDAaq-ll2kh8722kokkFnZ
  QcYtjuVC71aHNXI18q-lPnfRkmwryG-bhqQ"}

Figure 18

9.5. Encrypted container

The following example shows a container in which all the frame payloads are encrypted under the same master secret established in a key agreement specified in the first frame.

Frame 0

{
  "enc":"A256CBC",
  "Salt":"Th7V_b2--KdV5rETQZbHsw",
  "recipients":[{
      "kid":"MDAD3-E4BYE-MK6CH-QA2HD-TKRS2-KIX5Y-A",
      "epk":{
        "PublicKeyDH":{
          "kid":"MANS7-VSJ2X-75NSK-4SEOM-FRITG-6Q7EO-A",
          "Domain":"YE6bnq1MlX5ojaJto6PLP_PEwA",
          "Public":"MYw__h5Ii4honclOGFKY42KdvJed5T47CZI3QTRSBXUPiOD
  0D0W49YIyf-m30K-BvMWVzNsh9qAQFpK53A5UJO_pQmfbmrZLHq3AjdYoRlc44YOn
  usUokZN43xtGly4m7Rcf_42M3cofa80UWejL3mz3zQKTojxwXz3cykLA2tsTjHdKE
  O0Q6jWef952YkPQNKJmR-42aX7Pfz5kZt1QtgfUV4QYk_b0NFfHSNk9CAs6Klav5K
  qGjxL4mfIxvmrksyIR5solCo2BMbh_AOdyZNuEa4nURvS9YtWrXZRjQtMZPKB8MXG
  _mSAxUco3U1em2hmdkgIkTveF7-da6otILw"}},
      "wmk":"0HfYk15gAvxngunLoHymHSunrb0KlGjGuTERlpSxB5JLpFJj6KJZXA"}
    ],
  "Index":0,
  "ContainerType":"List",
  "ContentMeta":{},
  "DataEncoding":"JSON"}

[Empty trailer]

Figure 19

Frame 1

{
  "enc":"A256CBC",
  "Salt":"BmY8ZYeD8Kg1snIrBIAUyA",
  "Index":1}

[Empty trailer]

Figure 20

Frame 2

{
  "enc":"A256CBC",
  "Salt":"pA-kigxfhhsGC5lM9xrAZQ",
  "Index":2}

[Empty trailer]

Figure 21

Here are the container bytes. Note that the content is now encrypted and has expanded by 25 bytes. These are the salt (16 bytes), the AES padding (4 bytes) and the JSON-B framing (5 bytes).

f5 03 3c 
f1 03 27 
f0 10 
3c 03 f5 
f5 01 7c 
f0 47 
f1 01 30 
7c 01 f5 
f5 01 7c 
f0 47 
f1 01 30 
7c 01 f5 

Figure 22

The following example shows a container in which all the frame payloads are encrypted under separate key agreements specified in the payload frames.

Frame 0

{
  "Index":0,
  "ContainerType":"List",
  "ContentMeta":{},
  "DataEncoding":"JSON"}

[Empty trailer]

Figure 23

Frame 1

{
  "enc":"A256CBC",
  "Salt":"LUAkC3V-ztXJfF7OS-RNwQ",
  "recipients":[{
      "kid":"MDAD3-E4BYE-MK6CH-QA2HD-TKRS2-KIX5Y-A",
      "epk":{
        "PublicKeyDH":{
          "kid":"MCYIZ-PH7YZ-XAY6Q-2EMQR-GMRXG-66YPK-A",
          "Domain":"YE6bnq1MlX5ojaJto6PLP_PEwA",
          "Public":"4cDrUAlsZpyGZGhgVbLlunEyl7O9a8ku-6IGVZaSqyOmsb5
  rCt_pjnN49fjy7hqsS364S4eUDRj7e3vfIBEav-zeynaiJB6lY6eJBMmlaEuomyyy
  01Srq5eaX6i0A8CRWuuRK8IJAHQ31Q89Bah5XayMxc3bqPxciOL1gqU75w65fKmY9
  4mRdAIw2Yk-PTcQVztVDk2ooGJuvLENuIk29g-leFbMmCAXND9hhDErVupBND3Ljh
  BXylTvfIx-mrw0SMPO_IsVLbhrXQ6lTMygucttT3GPf25lgWBaEdiHVuPqmTBtagB
  Vgq9-gHySYtuCNrK7RfB5rtz6CUmENEwx1QA"}},
      "wmk":"25yqAeqIG1S6BrCSQxU_M8WKB4t8J_lHWI-68DNYORTUoCpaitosig"}
    ],
  "Index":1}

[Empty trailer]

Figure 24

Frame 2

{
  "enc":"A256CBC",
  "Salt":"EVrEz8ai_v9s9hZ0ZXMB1Q",
  "recipients":[{
      "kid":"MDAD3-E4BYE-MK6CH-QA2HD-TKRS2-KIX5Y-A",
      "epk":{
        "PublicKeyDH":{
          "kid":"MB43P-C7BVX-SROQ6-Z5LW5-5DLRO-PS675-A",
          "Domain":"YE6bnq1MlX5ojaJto6PLP_PEwA",
          "Public":"mJRyK45fAf2ksxW7M8W961hCKI1g7dQqn9rA2K6vdSh1o4v
  gLqjHDzLYqr9yQKteGUfl0PLIuHAOFq3p_oJDhaioJKX6Akv-y-dcfDdi2WgOOr_k
  R9NPtl-nmsbJ8fx6v1QdXLzdoGZjsj_t6wyIo5w287p7DylFdYkhriqVv7tgBELZI
  53Pur4F5QbK-qJYeSwY3fAksQYrviMZbFVwic8v5QQNoAGcoGguBOOz-aljFuo1g9
  0o7pNT-Soj3DAPfmXS1hmPR-hpqyzELAat-q_O22P4d_NtUOmaLwZXSZVYUBLE8Fj
  1okt4_8zGfCsGpsHl9rwqlyoVIZ4VadldWw"}},
      "wmk":"P6UjESsvq5bATPHc5vPfEcs5u1pCkLVBCv5_aYdHdwQG3rpPihHj2g"}
    ],
  "Index":2}

[Empty trailer]

Figure 25

10. Appendix B

public long PreviousFrame (long Frame) {
    long x2 = Frame + 1; 
    long d = 1; 

    while (x2 > 0) {
        if ((x2 & 1) == 1) {
            return x2 == 1 ? (d / 2) - 1 : Frame - d; 
            }
        d = d * 2; 
        x2 = x2 / 2; 
        }
    return 0; 
    }

Figure 26

11. References

11.1. Normative References

[draft-hallambaker-dare-message] Hallam-Baker, P., "Data At Rest Encryption Part 1: DARE Message", Internet-Draft draft-hallambaker-dare-message-01, July 2018.
[draft-hallambaker-jsonbcd] Hallam-Baker, P., "Binary Encodings for JavaScript Object Notation: JSON-B, JSON-C, JSON-D", Internet-Draft draft-hallambaker-jsonbcd-13, July 2018.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997.
[RFC7159] Bray, T., "The JavaScript Object Notation (JSON) Data Interchange Format", RFC 7159, DOI 10.17487/RFC7159, March 2014.
[RFC7515] Jones, M., Bradley, J. and N. Sakimura, "JSON Web Signature (JWS)", RFC 7515, DOI 10.17487/RFC7515, May 2015.
[RFC7516] Jones, M. and J. Hildebrand, "JSON Web Encryption (JWE)", RFC 7516, DOI 10.17487/RFC7516, May 2015.

11.2. Informative References

[BLOCKCHAIN] Chain.com, "Blockchain Specification"
[draft-hallambaker-mesh-architecture] Hallam-Baker, P., "Mathematical Mesh: Architecture", Internet-Draft draft-hallambaker-mesh-architecture-04, September 2017.
[RFC5652] Housley, R., "Cryptographic Message Syntax (CMS)", STD 70, RFC 5652, DOI 10.17487/RFC5652, September 2009.
[ZIPFILE] PKWARE Inc, "APPNOTE.TXT - .ZIP File Format Specification", October 2014.

Author's Address

Phillip Hallam-Baker Comodo Group Inc. EMail: philliph@comodo.com