Network Working Group P. Hallam-Baker
Internet-Draft Comodo Group Inc.
Intended status: Informational May 24, 2018
Expires: November 25, 2018

Data At Rest Encryption: DARE Container
draft-hallambaker-dare-container-00

Abstract

This document describes DARE Container, a message and file syntax that allows a sequence of data frames to be represented with cryptographic integrity, signature and encryption enhancements to be constructed in an append only format. The format supports data integrity checks using digest chains and Merkle trees. The simplest supports efficient append only write operations and efficient read operations in either the forward or reverse direction. Support for efficient random-access reads may be provided through the use of binary trees or index records appended to the end of the file.

This document is also available online at http://mathmesh.com/Documents/draft-hallambaker-dare-container.html .

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."

This Internet-Draft will expire on November 25, 2018.

Copyright Notice

Copyright (c) 2018 IETF Trust and the persons identified as the document authors. All rights reserved.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.


Table of Contents

1. Introduction

DARE Container is a message and file syntax that allows a sequence of data frames to be represented with cryptographic integrity, signature, and encryption enhancements to be constructed in an append only format. DARE Container was developed in response to needs that arose out of the design of the Mathematical Mesh [draft-hallambaker-mesh-architecture] . It is built on the binary encodings of JSON data objects, JSON-B and JSON-C [draft-hallambaker-jsonbcd] and the DARE Message format [draft-hallambaker-dare-message] .

The high level requirements supported include:

The features supported by DARE Container include:

Many file proprietary formats are in use that support some or all of these capabilities but only a handful have public, let alone open, standards. DARE Container is designed to provide a superset of the capabilities of existing message and file syntaxes, including:

Attempting to make use of these specifications in a layered fashion would require at least three separate encoders and introduce unnecessary complexity.

Every data format represents a compromise between different concerns, in particular:

Data Storage
The space required to record data in the encoding.
Memory Overhead
The additional volatile storage (RAM) required to maintain indexes etc. to support efficient retrieval operations.
Number of Operations
The number of operations required to retrieve data from or append data to an existing encoded sequence.
Number of Disk Seek Operations
Optimizing the response time of magnetic storage media to random access read requests has traditionally been one of the central concerns of database design. The DARE Container format is designed to the assumption that this will cease to be a concern as solid state media replaces magnetic.

While the cost of storage of all types has declined rapidly over the past decades, so has the amount of data to be stored. DARE Container represents a pragmatic balance of these considerations for current technology. In particular, since payload volumes are likely to be very large, memory and operational efficiency are considered higher priorities than data volume.

2. Definitions

2.1. Related Specifications

DARE Container makes use of the following related standards and specifications.

Encoding
Content frame headers are encoded using JavaScript Object Notation (JSON) [RFC7159] , JSON-B or JSON-C [draft-hallambaker-jsonbcd] .
Cryptography
The encryption and signature schemes used are based on JSON Web Signature [RFC7515] and JSON Web Encryption [RFC7516] .

2.2. Requirements Language

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119] .

3. Container Format

A DARE Container consists of a sequence of JBCD Frames containing up to three ordered JBCD records as follows:

Header Record [required]
Metadata of any type including container metadata, payload metadata and DARE message headers.
Payload Record [optional]
The frame data. Payload records are either complete or incremental. A complete payload contains a complete unit of whatever data is being written to the log. An incremental payload contains a part of a unit that has been split across multiple records.
Guard Record [optional]
An opaque data record with a known value written to the end of a frame to allow a writer to avoid corruption of the container framing data by detecting an incomplete append state.

A DARE frame consists of a forward length indicator, the framed data and a reverse length indicator. The reverse length indicator is written out backwards to allow the frame to be read in the reverse direction:

[[This figure is not viewable in this format. The figure is available at http://mathmesh.com/Documents/draft-hallambaker-dare-container.html.]]


JBCD Bidirectional Frame

When first reading an existing file, an application will typically read the first frame and the last frame (if the container has more than one frame). This allows the reader to quickly determine the format(s) used by the container, the number of frames in the container and the location of any index frames (if present).

The container format is designed to support creation of write-once and append-only file formats. Each frame SHOULD be written as an atomic operation.

The first frame in a container and the first record in a frame have special roles that are described in this document.

3.1. Container Profile

A key objective of the DARE Container format is that the simplest possible reader be capable of reading any container file albeit with possibly reduced performance.

A Container MAY conform to one or more profiles. Conforming to a profile typically requires a writer to provide additional information when writing a file but does not require a reader to interpret it unless use of a feature (e.g. authentication) that depends on the additional information is required.

3.1.1. Index Profiles

The following profiles are currently defined:

Tree
Frame headers contain IndexPosition entries that specify the start position of previous frames. This enables efficient random access to any frame in the file.
Digest
Frame headers contain PayloadDigest entries that specify the digest value of the corresponding payload data in that frame.
Chain
Frame headers contain ChainDigest entries that link each frame to the preceding frame.
Merkle
Frame headers contain TreeDigestPartial and TreeDigestFinal entries linking all the frames in the container in a binary Merkle Tree.

The use of Chain and Merkle Trees for integrity checks is described below.

The use of Tree and Index frames is described below.

3.1.2. Content Profiles

The following profiles are currently defined:

Singleton
A container with exactly one content frame. A container declared as a singleton frame cannot have additional content frames appended (but metadata frames may be)
Multi
A container whose payload data is limited to content frames. A container declared as a multi container may contain 0, 1 or more content frames.
Archive
A multi-container whose payload data is limited to content frames whose last frame contains a metadata index for the content frames in the container.
Unitary
A multi-container in which each frame represents exactly one payload object.
Serial
A multi-container in which payload objects MAY be split across multiple consecutive frames.
Interleaved
A multi-container in which payload objects MAY be split across multiple frames which may in turn be interleaved with frames containing other payload objects in complete or partial form.

3.2. Payload Encryption

DARE container payloads MAY be encrypted as DARE Message Enhanced Data Sequences [draft-hallambaker-dare-message] . This specification builds on JSON Web Encryption (JWE) [RFC7516] to provide a flexible framework allowing a single key exchange to be applied to encrypt multiple data sequences.

The DARE Container and DARE Message format are designed to compliment each other:

4. Index Mechanisms

An index may be appended to an existing file at any time. Since the use of bidirectional frames makes reading the last record is as efficient as reading the first, the last record in an indexed file is usually either the index itself or a pointer to the last index.

An index frame consists of a frame header

Use of index frames provides read access to any record in the file in O(1) operations but attempting to compiling a complete index with every write incurs an O(n) penalty on write for both operations and storage. Accordingly, random read access to a file while it is being written is better supported using an index tree.

4.1. Tree

Binary search is supported by means of the TreePosition parameter specified in the FrameHeader. This parameter specifies the value of the immediately preceding apex.

Calculation of the immediately preceding apex is most easily described by representing the array index in binary with base of 1 (rather than 0). An array index that is a power of 2 (2, 4, 8, 16, etc.) will be the apex of a complete tree. Every other array index has the value of the sum of a set of powers of 2 and the immediately preceding apex will be the value of the next smallest power of 2 in the sum.

For example, to find the immediately preceding apex for frame 5, we add 1 to get 6. 6 = 4 + 2, so we ignore the 2 and the preceding frame is 4.

The values of Tree Position are shown for the first 8 frames in figure xx below:

[[This figure is not viewable in this format. The figure is available at http://mathmesh.com/Documents/draft-hallambaker-dare-container.html.]]


Merkle Tree Integrity check

An algorithm for efficiently calculating the immediately preceding apex is provided in Appendix C.

4.2. Position Index

Contains a table of index, position pairs pointing to prior locations in the file.

4.3. Metadata Index

Contains a list of IndexMeta entries. Each entry contains a metadata description and a list of frame indexes (not positions) of frames that match the description.

5. Integrity Mechanisms

Frame sequences in a DARE container MAY be protected against a frame insertion attack by means of a digest chain, a binary Merkle tree or both.

5.1. Digest Chain calculation

A digest chain is simple to implement but can only be verified if the full chain of values is known. Appending a frame to the chain has O(1) complexity but verification has O(n) complexity:

[[This figure is not viewable in this format. The figure is available at http://mathmesh.com/Documents/draft-hallambaker-dare-container.html.]]


Hash chain integrity check

The value of the chain digest for the the first frame (frame 0) is H(IV+H(Payload0)), where IV is an initialization vector consisting of a string of zero bytes and payloadn is the sequence of payload data bytes for frame n

The value of the chain digest for frame n is H(H(Payloadn-1 +H(Payloadn)), where A+B stands for concatenation of the byte sequences A and B.

5.2. Binary Merkle tree calculation

The tree index mechanism describe earlier may be used to implement a binary Merkle tree. The value TreeDigest specifies the apex value of the tree for that node.

Appending a frame to the chain has O(log2n) complexity provided that the container format supports at least the binary tree index. Verifying a chain has O(log2 n) complexity, provided that the set of necessary digest inputs is known.

To calculate the value of the tree digest for a node, we first calculate the values of all the sub trees that have their apex at that node and then calculate the digest of that value and the immediately preceding local apex.

6. Reference

TBS stuff

6.1. Container Headers

TBS stuff

6.1.1. Structure: ContainerHeaderFirst

Inherits: ContainerHeader
Inherits: ContainerHeader
DataEncoding: String (Optional)
Specifies the data encoding for the header section of for the following frames. This value is ONLY valid in Frame 0 which MUST have a header encoded in JSON.

6.1.2. Structure: ContainerHeader

Inherits: DAREHeader

Describes a container header. A container header MAY contain any DARE Message header.

Index: Integer (Optional)
The record index within the file. This MUST be unique and satisfy any additional requirements determined by the ContainerType.
ContainerType: String (Optional)
Specifies the container type for the following records.
IsMeta: Boolean (Optional)
If true, the current frame is a meta frame and does not contain a payload.
Note: Meta frames MAY be present in any container. Applications MUST accept containers that contain meta frames at any position in the file. Applications MUST NOT interpret a meta frame as a data frame with an enpty payload.
UniqueID: String (Optional)
Unique object identifier
ContentMeta: ContentMeta (Optional)
Content meta data.
TreePosition: Integer (Optional)
Position of the frame containing the apex of the preceding sub-tree.
IndexPosition: Integer (Optional)
Specifies the position in the file at which the last index entry is to be found
ExchangePosition: Integer (Optional)
Specifies the position in the file at which the key exchange data is to be found
ContainerIndex: ContainerIndex (Optional)
An index of records in the current container up to but not including this one.
PayloadDigest: Binary (Optional)
If present, contains the digest of the Payload.
ChainDigest: Binary (Optional)
If present, contains the digest of the PayloadDigest values of this frame and the frame immediately preceding.
TreeDigest: Binary (Optional)
If present, contains the Binary Merkle Tree digest value.
Event: String (Optional)
Unique object identifier
Labels: String [0..Many]
List of labels that are applied to the payload of the frame.
KeyValues: KeyValue [0..Many]
List of key/value pairs describing the payload of the frame.
First: Integer (Optional)
Frame number of the first object instance value.
Previous: Integer (Optional)
Frame number of the immediately prior object instance value

6.2. Content Metadata Structure

TBS stuff

6.2.1. Structure: ContentMeta

Information describing the object instance

ContentType: String (Optional)
The content type field as specified in JWE
Paths: String [0..Many]
List of filename paths for the payload of the frame.
UniqueID: String (Optional)
Unique object identifier
Created: DateTime (Optional)
Initial creation date.
Modified: DateTime (Optional)
Date of last modification.

6.3. Index Structures

TBS stuff

6.3.1. Structure: ContainerIndex

A container index

Full: Boolean (Optional)
If true, the index is complete and contains position entries for all the frames in the file. If absent or false, the index is incremental and only contains position entries for records added since the last frame containing a ContainerIndex.
Positions: IndexPosition [0..Many]
List of container position entries
Metas: IndexMeta [0..Many]
List of container position entries

6.3.2. Structure: IndexPosition

Specifies the position in a file at which a specified record index is found

Index: Integer (Optional)
The record index within the file.
Position: Integer (Optional)
The record position within the file relative to the index base.

6.3.3. Structure: KeyValue

Specifies a key/value entry

Key: String (Optional)
The key
Value: String (Optional)
The value corresponding to the key

6.3.4. Structure: IndexMeta

Specifies the list of index entries at which a record with the specified metadata occurrs.

Index: Integer [0..Many]
List of record indicies within the file where frames matching the specified criteria are found.
ContentType: String (Optional)
Content type parameter
Paths: String [0..Many]
List of filename paths for the current frame.
Labels: String [0..Many]
List of labels that are applied to the current frame.

6.4. Signature

Payload data MAY be signed using a JWS [RFC7515] as applied in the DARE Message format [draft-hallambaker-dare-message] .

Signatures are specified by the Signatures parameter in the content header. The data that the signature is calculated over is defined by the typ parameter of the Signature as follows.

Payload
The frame payload data.
PayloadDigest
The value of the PayloadDigest parameter
ChainDigest
The value of the ChainDigest parameter
TreeDigestFinal
The value of the TreeDigestFinal parameter

If the typ parameter is absent, the value Payload is implied.

A frame MAY contain multiple signatures created with the same signing key and different typ values.

The use of signatures over chain and tree digest values permit multiple frames to be validated using a single signature verification operation.

7. Further Work

The container format is intended to be the basis of future work to support:

7.1. Fast open with random access

The container format is designed to be capable of supporting efficient random access to frames in containers considerably larger than the processing memory of the host computer without the need to pre-load indexes.

A combination of the following strategies is being considered:

7.2. Partitioning of very large data sets across hosts

While storage devices capable of storing tends of Tb of data with RAID redundancy are commonplace, it is generally desirable that there be at least as many CPU cores as disks. Thus, partitioning of data sets across multiple hosts becomes desirable for throughput even if a single host could handle the storage requirement.

7.3. Filtering and redaction

In the types of applications envisaged in the Mesh, almost every data set may be reduced to collections that are bound to a single account. While it is obviously desirable that a user's mail messages (for example) be replicated across multiple machines to provide fault tolerance, fragmenting the copies of this data set across multiple machines should be avoided unless the data volumes are so large as to require it.

7.4. Encryption of large data blocks

The encoding scheme is 64-bit clean throughout and thus supports containers and frames as large as 18 petabytes. Larger data volumes could be supported through use of 128-bit integer pointers but even if the technology to support such data volumes were developed, it is highly unlikely anyone would want to represent data sets anywhere near this size in a serial format.

Due to limitations in the design of the encryption schemes that may be used (e.g. AES-GCM), the maximum encrypted frame size is 64GB. While this is not currently a major concern for encryption of individual data files, it is easy to see situations in which an archive of encrypted files could exceed that amount. One possibility would be to define a modification to AES -GCM which caused the encryption key to be incremented by a fixed amount after encrypting a certain amount of data, though this might well present implementation challenges unless the maximum data block size was chosen to be deliberately small so as to force code paths to be exercised. Another possibility would be to limit the size of encrypted data frames by requiring the frame pointer to be no larger than 32 bits and require larger data items to be represented as a sequence of frames.

7.5. Concurrent Writes

The container format deliberately avoids support for concurrent write operations. Should this be desirable, some mechanism must be provided to cache write fragments to an intermediate file and then consolidate them for writing to the master log.

8. Security Considerations

9. IANA Considerations

10. Acknowledgements

11. Appendix A: Examples and Test Vectors

The data payloads in all the following examples are identical, only the authentication and/or encryption is different.

For conciseness, the wire format is omitted for examples after the first, except where the data payload has been transformed, (i.e. encrypted).

11.1. Simple container

Here the simple container:

f4 5b 
f0 59 
  7b 0a 20 20 22 49 6e 64   65 78 22 3a 20 30 2c 0a 
  20 20 22 43 6f 6e 74 61   69 6e 65 72 
  ...
  7d 2c 0a 20 20 22 44 61   74 61 45 6e 63 6f 64 69 
  6e 67 22 3a 20 22 4a 53   4f 4e 22 7d 
5b f4 
f5 01 40 
f0 0f 
  7b 0a 20 20 22 49 6e 64   65 78 22 3a 20 31 7d 
f1 01 2c 
  00 01 02 03 04 05 06 07   08 09 0a 0b 0c 0d 0e 0f 
  10 11 12 13 14 15 16 17   18 19 1a 1b 1c 1d 1e 1f 
  20 21 22 23 24 25 26 27   28 29 2a 2b 2c 2d 2e 2f 
  30 31 32 33 34 35 36 37   38 39 3a 3b 3c 3d 3e 3f 
  40 41 42 43 44 45 46 47   48 49 4a 4b 4c 4d 4e 4f 
  50 51 52 53 54 55 56 57   58 59 5a 5b 5c 5d 5e 5f 
  60 61 62 63 64 65 66 67   68 69 6a 6b 6c 6d 6e 6f 
  70 71 72 73 74 75 76 77   78 79 7a 7b 7c 7d 7e 7f 
  80 81 82 83 84 85 86 87   88 89 8a 8b 8c 8d 8e 8f 
  90 91 92 93 94 95 96 97   98 99 9a 9b 9c 9d 9e 9f 
  a0 a1 a2 a3 a4 a5 a6 a7   a8 a9 aa ab ac ad ae af 
  b0 b1 b2 b3 b4 b5 b6 b7   b8 b9 ba bb bc bd be bf 
  c0 c1 c2 c3 c4 c5 c6 c7   c8 c9 ca cb cc cd ce cf 
  d0 d1 d2 d3 d4 d5 d6 d7   d8 d9 da db dc dd de df 
  e0 e1 e2 e3 e4 e5 e6 e7   e8 e9 ea eb ec ed ee ef 
  f0 f1 f2 f3 f4 f5 f6 f7   f8 f9 fa fb fc fd fe ff 
  00 01 02 03 04 05 06 07   08 09 0a 0b 0c 0d 0e 0f 
  10 11 12 13 14 15 16 17   18 19 1a 1b 1c 1d 1e 1f 
  20 21 22 23 24 25 26 27   28 29 2a 2b 
40 01 f5 

Figure 1

The header values are:

Frame 0

{
  "Index": 0,
  "ContainerType": "List",
  "ContentMeta": {},
  "DataEncoding": "JSON"}

Figure 2

Frame 1

{
  "Index": 1}

Figure 3

11.2. Payload and chain digests

Frame 0

{
  "Index": 0,
  "ContainerType": "Chain",
  "ContentMeta": {},
  "PayloadDigest": "z4PhNX7vuL3xVChQ1m2AB9Yg5AULVxXcg_SpIdNs6c5H0
  NE8XYXysP-DGNKHfuwvY7kxvUdBeoGlODJ6-SfaPg",
  "ChainDigest": "FEHy24Y6cLModDXWH31kVc2a3TdhjXPooKHpLAb2JbsO1YQ
  nJolmowXAYHhkOGY0kg3jrKNTjds0myf4Dw1sdg",
  "DataEncoding": "JSON"}

Figure 4

Frame 1

{
  "Index": 1,
  "PayloadDigest": "8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZD
  lZeaWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw",
  "ChainDigest": "7JaijhBvQUOjBiO1_Zt6NtJil8iB0rW9HeM_4iYooc_AaAf
  utlF0LLVY6PO7INB-eztypyEqVzgMil9JkjtRGQ"}

Figure 5

Frame 2

{
  "Index": 2,
  "PayloadDigest": "8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZD
  lZeaWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw",
  "ChainDigest": "wJZFYd61nntCJ0Bv80l6-Cn-sR2u3iD0zCRjOLxje8dsKIu
  UnP4X1mgeNenNDBdXysrFs3vVAqkC-hfSAPF0Aw"}

Figure 6

Frame 3

{
  "Index": 3,
  "PayloadDigest": "8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZD
  lZeaWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw",
  "ChainDigest": "RORNZxIcM23cZtXPh9vuHhkgiGa_O4a0ZiU0ku2OK4dB974
  clvh5F0VZsX7IwVBayAG2nDTdqhyZ-qOnTRiumA"}

Figure 7

11.3. Merkle Tree

Frame 0

{
  "Index": 0,
  "ContainerType": "Merkle",
  "ContentMeta": {},
  "PayloadDigest": "z4PhNX7vuL3xVChQ1m2AB9Yg5AULVxXcg_SpIdNs6c5H0
  NE8XYXysP-DGNKHfuwvY7kxvUdBeoGlODJ6-SfaPg",
  "TreeDigest": "FEHy24Y6cLModDXWH31kVc2a3TdhjXPooKHpLAb2JbsO1YQn
  JolmowXAYHhkOGY0kg3jrKNTjds0myf4Dw1sdg",
  "DataEncoding": "JSON"}

Figure 8

Frame 1

{
  "Index": 1,
  "TreePosition": 0,
  "PayloadDigest": "8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZD
  lZeaWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw",
  "TreeDigest": "fPTYagAvSDP_755jpFUs-Wq6cgvtr5vrFwW-E12vsrbq1ReN
  sGzp-V2XqzFPiWaUckACPjegD7ioe1bGzxoWQQ"}

Figure 9

Frame 2

{
  "Index": 2,
  "TreePosition": 315,
  "PayloadDigest": "8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZD
  lZeaWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw",
  "TreeDigest": "7fyKKQNLGEeHX1oCsV8NtOdPm615SkDnM1vkcexx2tOuVd5k
  kZIdLdsWRCLic9luTSsUN6D6_-c-8ftbhL9dJg"}

Figure 10

Frame 3

{
  "Index": 3,
  "TreePosition": 315,
  "PayloadDigest": "8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZD
  lZeaWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw",
  "TreeDigest": "b9ca9Pv-6fxUg-V3ulOhhRngxebkZCxyDmWhQUYeADmSvvPb
  jMcNTUJxdDpKlMPrDBInSWMChinsc5s9Tv4byw"}

Figure 11

Frame 4

{
  "Index": 4,
  "TreePosition": 1441,
  "PayloadDigest": "8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZD
  lZeaWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw",
  "TreeDigest": "g1hQeWJgDlNoTSGfMb6NhQk5-p6iaAI2_GiAhBM-F2Cp3UvJ
  7AR_bC2Drp5YElGXAzC2K5qZ30l7j2D-jqykFw"}

Figure 12

Frame 5

{
  "Index": 5,
  "TreePosition": 1441,
  "PayloadDigest": "8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZD
  lZeaWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw",
  "TreeDigest": "p89BhjJAgMMoSrOmot6oaBGa6Dgz-zogZjZ9mm1Iz4yLHxm9
  7nWAIBaZFiC1XkuCoP-tr3tag_rHoZhgQV8_PQ"}

Figure 13

Frame 6

{
  "Index": 6,
  "TreePosition": 2570,
  "PayloadDigest": "8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZD
  lZeaWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw",
  "TreeDigest": "HEA7EeUGfSjZqjmN3PDp0FVbnixBBXfSQAYm_rNPHVWJVMDu
  3SfmxKvN_yBTtMXk-Jad9cyXDKsecLNHLyoQWg"}

Figure 14

11.4. Signed container

11.5. Encrypted container

The following example shows a container in which all the frame payloads are encrypted under the same master secret established in a key agreement specified in the first frame.

Frame 0

{
  "enc": "A256CBC",
  "recipients": [{
      "kid": "MCRW5-YRWQK-M4HE3-D4JE6-IYQLM-22ITA-A",
      "epk": {
        "PublicKeyDH": {
          "kid": "MBSNR-T4GZ6-LBZTD-G2CP3-MGT4K-J4VGF-A",
          "Domain": "YE6bnq1MlX5ojaJto6PLP_PEwA",
          "Public": "Cmh_MVvRHNDOH6GB2Gjke8n8z3UvqNXmNcYQRKPMqEvSsa_zagAw
  ChVd3EX1KZ7D8mQYiCbh-F5n2vvydzJ6UOqa4Ur-VzrGCaorB_zdSWoy8DxlHIcSL
  iCoE24lf_hHdv25lCu9-wPRXAlr2f3aAPMd9Mq_iqpkSoonXjMdN16uhUxkC-dX70
  2YVq2z9S5_Q_sWeYAgdg-9nVVdMQcY3I9mWu5HvbIo_z-lnTUP5mmRD4Otii67EpG
  K02QooxPJb0u8UEKFp665ZMu0Snd6nCfBwlxeuVoUgoXlB6vPAc_65c7AdyY3obnF
  ai6qZ8wiwV8NFyyyRL3jUuT8mGba1AA"}},
      "wmk": "PkGqGm7GBjO1sTgQbBdpccPNqoVq76b0OVxLweTerdzYyx5iRdIXXA"}],
  "Index": 0,
  "ContainerType": "List",
  "ContentMeta": {},
  "DataEncoding": "JSON"}

Figure 15

Frame 1

{
  "Index": 1,
  "ExchangePosition": 0}

Figure 16

Frame 2

{
  "Index": 2,
  "ExchangePosition": 0}

Figure 17

Here are the container bytes. Note that the content is now encrypted and has expanded by 25 bytes. These are the salt (16 bytes), the AES padding (4 bytes) and the JSON-B framing (5 bytes).

f5 03 0c 
f1 03 04 
  7b 0a 20 20 22 65 6e 63   22 3a 20 22 41 32 35 36 
  43 42 43 22 2c 0a 20 20   22 72 65 63 
  ...
  7d 2c 0a 20 20 22 44 61   74 61 45 6e 63 6f 64 69 
  6e 67 22 3a 20 22 4a 53   4f 4e 22 7d 
f0 03 
  88 01 00 
0c 03 f5 
f5 01 72 
f0 28 
  7b 0a 20 20 22 49 6e 64   65 78 22 3a 20 31 2c 0a 
  20 20 22 45 78 63 68 61   6e 67 65 50 6f 73 69 74 
  69 6f 6e 22 3a 20 30 7d 
f1 01 45 
  88 10 a8 9e 8c f3 f7 d6   fa c1 b7 e7 a0 30 71 bc 
  ae 7a 89 01 30 04 90 ce   1d 1a 79 f3 92 31 ae 2b 
  a5 b3 13 29 75 9c 48 07   b1 9b f1 46 d3 95 d2 81 
  3e 60 4f 76 16 fe 6d 2f   23 99 40 13 84 91 ca 1f 
  53 30 ec fe 66 b4 e1 38   95 6c 64 9f f7 59 30 5b 
  b7 d0 b5 1d 7d 29 77 61   21 e2 2e 29 30 66 ed 7a 
  92 4f 63 d4 7f 9d 3c 7f   12 59 8e 3f d9 f2 5a 63 
  8e ab 0a 3b 95 fa fc 16   1f b7 43 86 c5 f1 ef 6b 
  d8 fc 8d 43 44 2f 55 4c   06 e4 19 08 2e ba 28 21 
  af 04 88 e8 0e 5c 5f 25   7e 5a af 7a d5 44 aa de 
  2e ed e2 cd 9c df 58 95   b2 d2 05 d8 64 90 0c 65 
  b6 0b be 7b 86 99 90 e3   4d 73 f9 91 74 bf 07 7f 
  29 0b 0e 9d 5f bf ce 60   90 4a fb ef 37 9c 62 85 
  e8 01 2d 87 c8 a4 5c 07   ea dc c0 ec e4 10 d7 48 
  ee 57 e2 35 aa c0 e7 65   5f 5a 16 b3 cb e0 96 6b 
  ee 85 86 ff 71 d1 37 7d   65 81 6d 67 88 97 81 58 
  5d 30 1d 6c 8d 1a 2f 08   b4 d5 d4 80 17 fe 22 54 
  cd 4c 02 34 fa 4a e4 9c   c0 c9 2f a2 e0 3e 58 f6 
  14 ef a8 53 9b f2 69 f1   d0 10 26 93 45 13 d8 b5 
  86 57 2e 6c b5 fb 74 eb   ee b7 b5 a5 87 f6 ec b8 
  2d 55 48 31 ac 
72 01 f5 
f5 01 72 
f0 28 
  7b 0a 20 20 22 49 6e 64   65 78 22 3a 20 32 2c 0a 
  20 20 22 45 78 63 68 61   6e 67 65 50 6f 73 69 74 
  69 6f 6e 22 3a 20 30 7d 
f1 01 45 
  88 10 2d 8f 69 b8 89 72   a5 f0 65 76 9d b6 7f 52 
  bd b2 89 01 30 11 1d 1b   a4 ee 8d bc d7 2a 02 08 
  82 d2 3a 96 a3 36 b3 4a   ad 67 d5 31 4c 5e 12 b0 
  ac 16 d3 ee 9d 34 d2 0a   f7 70 6b e7 63 d2 63 21 
  8d 65 4f 60 13 78 51 11   e9 53 f6 e2 70 79 5d 8f 
  a9 66 8c 7f 0c 55 d1 8f   9a 4b e7 2c dc b5 85 9e 
  9c aa a8 2f ba 79 1f b6   c5 3b e8 2d 72 04 74 d8 
  f6 a6 ca 8e 67 68 d4 64   59 1c be 65 a2 51 02 22 
  af 94 bd fe ac 01 bc 65   23 e8 fb 4f bb 62 21 f7 
  77 7d e2 8a 4a a7 86 09   9e b0 b3 3c 44 fa 91 d0 
  95 22 e0 78 46 ee ab 8b   7c 84 83 c9 e2 67 ae 2e 
  f6 24 42 2b 42 3e 70 c6   e1 91 ec 12 83 c2 95 ff 
  a5 ea 09 70 8d 93 35 e8   1d 11 d6 90 6c d8 be 9e 
  bc d7 98 ad 7f c7 58 cc   e3 0d ce 51 d7 5d 05 5b 
  23 e8 42 ba ea 79 81 98   2a 9b a9 5d e8 2e c9 7e 
  14 d4 7a 7f 0f f8 aa 5c   2c b2 8c 0a f4 c6 b3 89 
  52 e4 c8 16 69 aa c2 a3   f8 6e 05 ff c4 8d 04 92 
  9e 83 c5 b2 8f 8a 1f 19   72 e2 f4 bc 8c 48 4f fc 
  75 a2 17 73 d5 8a 6e 22   9a b7 f8 a9 86 98 33 a4 
  4b 57 9a 60 59 a2 19 7f   e2 35 f1 e0 63 5d bf 25 
  c5 8f c6 90 f6 
72 01 f5 

Figure 18

The following example shows a container in which all the frame payloads are encrypted under separate key agreements specified in the payload frames.

Frame 0

{
  "Index": 0,
  "ContainerType": "List",
  "ContentMeta": {},
  "DataEncoding": "JSON"}

Figure 19

Frame 1

{
  "enc": "A256CBC",
  "recipients": [{
      "kid": "MCRW5-YRWQK-M4HE3-D4JE6-IYQLM-22ITA-A",
      "epk": {
        "PublicKeyDH": {
          "kid": "MCHK5-FYGDU-J3LE2-H427C-OPD6G-HD4QY-A",
          "Domain": "YE6bnq1MlX5ojaJto6PLP_PEwA",
          "Public": "kRBM44hYJidVQYULAtk8SHrYUMVjmBxLnPVwEwmi7tVK4xBipCAH
  rx06dlbO-gtd9f6V984yEtgXs1GJibo4pCvwXyWq70CoPosc9Rymingo2c6jjLFQP
  NvPks0e9kYSVLVTSM34XaTzQKjX4S9AkG4zI4IIrENDPQN6gfRe0Px-NCB-tR3pCW
  xJPPxpK2cYopHJX1jYamN69aF7r4ALx2jQSTiwv1558bU-atJNRqYv6dXVseiLpTo
  TUVpwG838VGPtjP1DeL_Tuk-di7uxj_n-EUVARr1N0HJwVcjJUHtrOt37jKtM5-RQ
  5XKZ3WRvX23N0ae50WjZV-aHmWjvIw"}},
      "wmk": "iU--JG-ERFbIiIlpm9WYlOCg9K2yUEefcTFXWnzgkIqsEt84dqP_FA"}],
  "Index": 1}

Figure 20

Frame 2

{
  "enc": "A256CBC",
  "recipients": [{
      "kid": "MCRW5-YRWQK-M4HE3-D4JE6-IYQLM-22ITA-A",
      "epk": {
        "PublicKeyDH": {
          "kid": "MAK46-P3CXG-TWHTU-KX7AV-EKBKZ-JIZW4-A",
          "Domain": "YE6bnq1MlX5ojaJto6PLP_PEwA",
          "Public": "HvdTPFfrFM9x6lzaXC6VqCX_QKxg1KRCirshm4lYs9dmZgvTJTep
  QLejdDzuT4IUpCcdDq_lZloCbQVPtl-lHxm60UWYzeJeoBHcxpffXhJVEjFU7Ieet
  vBAZP4IHnT_uhyxHomf6oHAo-VvtUa_nPRUpA4PyKApEL6bZPwwwU0WjoCfX25ezs
  M8ODZDA2BF4-TQlVHmfzpHIIEZ0oscpcHEcjYBAOjoYrYxdyORUa56E0aXifW5Hb1
  R00ZqXPB9dg3j-TEG6Bt1lMk1GSPe3zw339j-9bFl9VLTAlhX5WHvZfDSHQdJev1T
  bFV14t_tCaIg0jfKgcZSLzVNN1GwhQA"}},
      "wmk": "G69T62RWbpRGzv1IDAKSAB_ETERF3ZGHNx5Jqv-zj-3EPpgJsJZizA"}],
  "Index": 2}

Figure 21

12. Appendix B

public long PreviousFrame (long Frame) {
    long x2 = Frame + 1; 
    long d = 1; 

    while (x2 > 0) {
        if ((x2 & 1) == 1) {
            return x2 == 1 ? (d / 2) - 1 : Frame - d; 
            }
        d = d * 2; 
        x2 = x2 / 2; 
        }
    return 0; 
    }

Figure 22

13. References

13.1. Normative References

[draft-hallambaker-dare-message] , "[Reference Not Found!]"
[draft-hallambaker-jsonbcd] Hallam-Baker, P., "Binary Encodings for JavaScript Object Notation: JSON-B, JSON-C, JSON-D", Internet-Draft draft-hallambaker-jsonbcd-12, May 2018.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997.
[RFC7159] Bray, T., "The JavaScript Object Notation (JSON) Data Interchange Format", RFC 7159, DOI 10.17487/RFC7159, March 2014.
[RFC7515] Jones, M., Bradley, J. and N. Sakimura, "JSON Web Signature (JWS)", RFC 7515, DOI 10.17487/RFC7515, May 2015.
[RFC7516] Jones, M. and J. Hildebrand, "JSON Web Encryption (JWE)", RFC 7516, DOI 10.17487/RFC7516, May 2015.

13.2. Informative References

[BLOCKCHAIN] Chain.com, "Blockchain Specification"
[draft-hallambaker-mesh-architecture] Hallam-Baker, P., "Mathematical Mesh: Architecture", Internet-Draft draft-hallambaker-mesh-architecture-04, September 2017.
[RFC5652] Housley, R., "Cryptographic Message Syntax (CMS)", STD 70, RFC 5652, DOI 10.17487/RFC5652, September 2009.
[ZIPFILE] PKWARE Inc, "APPNOTE.TXT - .ZIP File Format Specification", October 2014.

Author's Address

Phillip Hallam-Baker Comodo Group Inc. EMail: philliph@comodo.com