Binary Encodings for JavaScript Object Notation: JSON-B, JSON-C, JSON-DComodo Group Inc.philliph@comodo.com
Three binary encodings for JavaScript Object Notation (JSON) are presented. JSON-B (Binary) is a strict superset of the JSON encoding that permits efficient binary encoding of intrinsic JavaScript data types. JSON-C (Compact) is a strict superset of JSON-B that supports compact representation of repeated data strings with short numeric codes. JSON-D (Data) supports additional binary data types for integer and floating-point representations for use in scientific applications where conversion between binary and decimal representations would cause a loss of precision.
This document is also available online at
http://prismproof.org/Documents/draft-hallambaker-json-bcd.html
.
JavaScript Object Notation (JSON) is a simple text encoding for the JavaScript Data model that has found wide application beyond its original field of use. In particular JSON has rapidly become a preferred encoding for Web Services.
JSON encoding supports just four fundamental data types (integer, floating point, string and boolean), arrays and objects which consist of a list of tag-value pairs.
Although the JSON encoding is sufficient for many purposes it is not always efficient. In particular there is no efficient representation for blocks of binary data. Use of base64 encoding increases data volume by 33%. This overhead increases exponentially in applications where nested binary encodings are required making use of JSON encoding unsatisfactory in cryptographic applications where nested binary structures are frequently required.
Another source of inefficiency in JSON encoding is the repeated occurrence of object tags. A JSON encoding containing an array of a hundred objects such as {"first":1,"second":2} will contain a hundred occurrences of the string "first" (seven bytes) and a hundred occurrences of the string "second" (eight bytes). Using two byte code sequences in place of strings allows a saving of 11 bytes per object without loss of information, a saving of 50%.
A third objection to the use of JSON encoding is that floating point numbers can only be represented in decimal form and this necessarily involves a loss of precision when converting between binary and decimal representations. While such issues are rarely important in network applications they can be critical in scientific applications. It is not acceptable for saving and restoring a data set to change the result of a calculation.
The following were identified as core objectives for a binary JSON encoding:
Low overhead encoding and decoding
Easy to convert existing encoders and decoders to add binary support
Efficient encoding of binary data
Ability to convert from JSON to binary encoding in a streaming mode (i.e. without reading the entire binary data block before beginning encoding.
Lossless encoding of JavaScript data types
The ability to support JSON tag compression and extended data types are considered desirable but not essential for typical network applications.
Three binary encodings are defined:
Encodes JSON data in binary. Only the JavaScript data model is supported (i.e. atomic types are integers, double or string). Integers may be 8, 16, 32 or 64 bits either signed or unsigned. Floating points are IEEE 754 binary64 format
[IEEE754]
. Supports chunked encoding for binary and UTF-8 string types.
As JSON-B but with support for representing JSON tags in numeric code form (16 bit code space). This is done for both compact encoding and to allow simplification of encoders/decoders in constrained environments. Codes may be defined inline or by reference to a known dictionary of codes referenced via a digest value.
As JSON-C but with support for representing additional data types without loss of precision. In particular other IEEE 754 floating point formats, both binary and decimal and Intel's 80 bit floating point, plus 128 bit integers and bignum integers.
Each encoding is a proper superset of JSON, JSON-C is a proper superset of JSON-B and JSON-D is a proper superset of JSON-C. Thus a single decoder MAY be used for all three new encodings and for JSON. Figure 1 shows these relationships graphically:
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in
[RFC2119]
.
The JSON-B, JSON-C and JSON-D encodings are all based on the JSON grammar
[RFC7159]
using the same syntactic structure but different lexical encodings.
JSON-B0 and JSON-C0 replace the JSON lexical encodings for strings and numbers with binary encodings. JSON-B1 and JSON-C1 allow either lexical encoding to be used. Thus any valid JSON encoding is a valid JSON-B1 or JSON-C1 encoding.
The grammar of JSON-B, JSON-C and JSON-D is a superset of the JSON grammar. The following productions are added to the grammar:
Binary encodings for data values. As the binary value encodings are all self delimiting
An object member where the value is specified as an X-value and thus does not require a value-separator.
Binary data encodings defined in JSON-B.
Defined length string encoding defined in JSON-B.
Tag code definition defined in JSON-C. These may only appear before the beginning of an Object or Array and before any preceding white space.
Tag code value defined in JSON-C.
Additional binary data encodings defined in JSON-D for use in scientific data applications.
The JSON grammar is modified to permit the use of x-value productions in place of ( value value-separator ) :
The productions number and string are defined as before:
The JSON-B encoding defines the b-value and b-string productions:
The lexical encodings of the productions are defined in the following tables where the column 'tag' specifies the byte code that begins the production, 'Fixed' specifies the number of data bytes that follow and 'Length' specifies the number of bytes used to define the length of a variable length field following the data bytes:ProductionTagFixedLengthData Descriptionstring-termx80-1Terminal String 8 bit lengthstring-termx81-2Terminal String 16 bit lengthstring-termx82-4Terminal String 32 bit lengthstring-termx83-8Terminal String 64 bit lengthstring-chunkx84-1Terminal String 8 bit lengthstring-chunkx85-2Terminal String 16 bit lengthstring-chunkx86-4Terminal String 32 bit lengthstring-chunkx87-8Terminal String 64 bit lengthdata-termx88-1Terminal String 8 bit lengthdata-termx89-2Terminal String 16 bit lengthdata-termx8A-4Terminal String 32 bit lengthdata-termx8B-8Terminal String 64 bit lengthdata-termX8C-1Terminal String 8 bit lengthdata-termx8D-2Terminal String 16 bit lengthdata-termx8E-4Terminal String 32 bit lengthdata-termx8F-8Terminal String 64 bit length
Table 1: Codes for String and Data itemsProductionTagFixedLengthData Descriptionp-int8xA01-Positive 8 bit Integerp-int16Xa12-Positive 16 bit Integerp-int32Xa24-Positive 32 bit Integerp-int64Xa38-Positive 64 bit Integerp-bignum16Xa5-2Positive Bignumn-int8xA81-Negative 8 bit Integern-int16xA92-Negative 16 bit Integern-int32xAA4-Negative 32 bit Integern-int64xAB8-Negative 64 bit Integern-bignum16xAD-2Negative Bignumbinary64x928-IEEE 754 Floating Point Binary 64 bitb-valuexB0--Trueb-valuexB1--Falseb-valuexB2--Null
Table 2: Codes for Integers, 64 Bit Floating Point, Boolean and Null items.
A data type commonly used in networking that is not defined in this scheme is a datetime representation. To define such a data type, a string containing a date-time value in Internet type format is typically used.
The following examples show examples of using JSON-B encoding:
JSON-C (Compressed) permits numeric code values to be substituted for strings and binary data. Tag codes MAY be 8, 16 or 32 bits long encoded in network byte order.
Tag codes MUST be defined before they are referenced. A Tag code MAY be defined before the corresponding data or string value is used or at the same time that it is used.
A dictionary is a list of tag code definitions. An encoding MAY incorporate definitions from a dictionary using the dict-hash production. The dict hash production specifies a (positive) offset value to be added to the entries in the dictionary followed by the UDF fingerprint
[draft-hallambaker-udf]
of the dictionary to be used.ProductionTagFixedLengthData Descriptionc-tagxC01-8 bit tag codec-tagxC12-16 bit tag codec-tagxC24-32 bit tag codec-defxC41-8 bit tag definitionc-defxC52-16 bit tag definitionc-defxC64-32 bit tag definitionc-tagxC81-8 bit tag code and definitionc-tagxC92-16 bit tag code and definitionc-tagxCA4-32 bit tag code and definitionc-defxCC1-8 bit tag dictionary definitionc-defxCD2-16 bit tag dictionary definitionc-defxCE4-32 bit tag dictionary definitiondict-hashxD041UDF fingerprint of dictionary
Table 3: Codes Used for Compression
All integer values are encoded in Network Byte Order (most significant byte first).
The following examples show examples of using JSON-C encoding:
JSON-B and JSON-C only support the two numeric types defined in the JavaScript data model: Integers and 64 bit floating point values. JSON-D (Data) defines binary encodings for additional data types that are commonly used in scientific applications. These comprise positive and negative 128 bit integers, six additional floating point representations defined by IEEE 754
[IEEE754]
and the Intel extended precision 80 bit floating point representation
[INTEL]
.
Should the need arise, even bigger bignums could be defined with the length specified as a 32 bit value permitting bignums of up to 2^35 bits to be represented.
The codes for these values are as follows:ProductionTagFixedLengthData Descriptionp-int128xA416-Positive 128 bit Integern-int128xAC16-Negative 128 bit Integerbinary16x902-IEEE 754 Floating Point Binary 16 bitbinary32x914-IEEE 754 Floating Point Binary 32 bitbinary128x9416-IEEE 754 Floating Point Binary 64 bitIntel80x9510-Intel extended Floating Point 80 bitdecimal32x964-IEEE 754 Floating Point Decimal 32Decimal64x978-IEEE 754 Floating Point Decimal 64Decimal128x9816-IEEE 754 Floating Point Decimal 128
Table 4: Additional Codes for Scientific Data
This work was assisted by conversations with Nico Williams and other participants on the applications area mailing list.
A correctly implemented data encoding mechanism should not introduce new security vulnerabilities. However, experience demonstrates that some data encoding approaches are more prone to introduce vulnerabilities when incorrectly implemented than others.
In particular, whenever variable length data formats are used, the possibility of a buffer overrun vulnerability is introduced. While best practice suggests that a coding language with native mechanisms for bounds checking is the best protection against such errors, such approaches are not always followed. While such vulnerabilities are most commonly seen in the design of decoders, it is possible for the same vulnerabilities to be exploited in encoders.
A common source of such errors is the case where nested length encodings are used. For example, a decoder relies on an outermost length encoding that specifies a length on 50 bytes to allocate memory for the entire result and then attempts to copy a string with a declared length of 1000 bytes within the sequence.
The extensions to the JSON encoding described in this document are designed to avoid such errors. Length encodings are only used to define the length of x-value constructions which are always terminal and cannot have nested data entries.
[TBS list out all the code points that require an IANA registration]IEEE Standard for Floating-Point ArithmeticIEEE Computer SocietyUnknownIntel Corp.Key words for use in RFCs to Indicate Requirement LevelsThe JavaScript Object Notation (JSON) Data Interchange FormatUniform Data Fingerprint (UDF)