Extensible Binary Meta Language
draft-ietf-cellar-ebml-00

Abstract

This document defines the Extensible Binary Meta Language (EBML) format as a genearlized file format for any type of data in a hierarchical form. EBML is designed as a binary equivalent to XML and utilizes a storage-efficient approach to building nested Elements with identifiers, lengths, and values. Similar to how an XML Schema defines the structure and semantics of an XML Document, this document defines an EBML Schema to convey the semantics of an EBML Document.

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at http://datatracker.ietf.org/drafts/current/.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."

This Internet-Draft will expire on March 27, 2017.

Copyright Notice

Copyright (c) 2016 IETF Trust and the persons identified as the document authors. All rights reserved.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.

Table of Contents

• 1. EBML specifications
• 1.1. Introduction
• 1.2. Notation and Conventions
• 1.3. Security Considerations
• 1.4. Structure
• 1.5. Variable Size Integer
• 1.5.1. VINT_WIDTH
• 1.5.2. VINT_MARKER
• 1.5.3. VINT_DATA
• 1.5.4. VINT Examples
• 1.6. Element ID
• 1.7. Element Data Size
• 1.8. EBML Element Types
• 1.9. EBML Document
• 1.9.1. EBML Header
• 1.9.2. EBML Body
• 1.10. EBML Stream
• 1.11. Elements semantic
• 1.11.1. EBML Schema
• 1.11.2. EBML Header Elements
• 1.11.3. Global elements (used everywhere in the format)
• Authors' Addresses

1. EBML specifications

1.1. Introduction

EBML, short for Extensible Binary Meta Language, specifies a binary and octet (byte) aligned format inspired by the principle of XML.

The goal of the EBML Specification is to define a generic, binary, space-efficient format that may be utilized to define more complex formats (such as containers for multimedia content) using an EBML Schema. The definition of the EBML format recognizes the idea behind HTML and XML as a good one: separate structure and semantics allowing the same structural layer to be used with multiple, possibly widely differing semantic layers. Except for the EBML Header and a few global elements this specification does not define particular EBML format semantics; however this specification is intended to define how other EBML-based formats may be defined.

EBML uses a simple approach of building Elements upon three pieces of data (tag, length, and value) as this approach is well known, easy to parse, and allows selective data parsing. The EBML structure additionally allows for hierarchical arrangement to support complex structural formats in an efficient manner.

1.2. Notation and Conventions

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.

1.3. Security Considerations

EBML itself does not offer any kind of security. It has nothing to do with authentication, it does not provide confidentiality, only marginally useful and effective data integrity options (CRC elements).

EBML does not provide any kind of authorization.

Even if the semantic layer offers any kind of encryption, EBML itself may leak information at both the semantic layer (as declared via the DocType element) and within the EBML structure (you can derive the presence of EBML elements even with an unknown semantic layer with a heuristic approach; not without errors, of course, but with a certain degree of confidence).

Attacks on an EBML reader may include: - Invalid Element IDs that are longer than the limit stated in the EBMLMaxIDLength Element of the EBML Header. - Invalid Element IDs that are not encoded in the shortest-possible way. - Invalid Element IDs comprised of reserved values. - Invalid Element Data Size values that are longer than the limit stated in the EBMLMaxSizeLength Element of the EBML Header. - Invalid Element Data Size values (e.g. extending the length of the Element beyond the scope of the Parent Element; possibly triggering access-out-of-bounds issues). - Very high lengths in order to force out-of-memory situations resulting in a denial of service, access-out-of-bounds issues etc. - Missing Elements that are mandatory and have no declared default value. - Usage of 0x00 octets in EBML Elements with a string type. - Usage of invalid UTF-8 encoding in EBML Elements of UTF-8 type (e.g. in order to trigger access-out-of-bounds or buffer overflow issues). - Usage of invalid data in EBML Elements with a date type.

1.4. Structure

EBML uses a system of Elements to compose an EBML Document. Elements incorporate three parts: an Element ID, an Element Data Size, and Element Data. The Element Data, which is described by the Element ID, may include either binary data or one or many other Elements.

1.5. Variable Size Integer

The Element ID and Element Data Size are both encoded as a Variable Size Integer, developed according to a UTF-8 like system. The Variable Size Integer is composed of a VINT_WIDTH, VINT_MARKER, and VINT_DATA, in that order. Variable Size Integers shall be referred to as VINT for shorthand.

1.5.1. VINT_WIDTH

Each Variable Size Integer begins with a VINT_WIDTH which consists of zero or many zero-value bits. The count of consecutive zero-values of the VINT_WIDTH plus one equals the length in octets of the Variable Size Integer. For example, a Variable Size Integer that starts with a VINT_WIDTH which contains zero consecutive zero-value bits is one octet in length and a Variable Size Integer that starts with one consecutive zero-value bit is two octets in length. The VINT_WIDTH MUST only contain zero-value bits or be empty.

1.5.2. VINT_MARKER

The VINT_MARKER serves as a separator between the VINT_WIDTH and VINT_DATA. Each Variable Size Integer MUST contain exactly one VINT_MARKER. The VINT_MARKER MUST be one bit in length and contain a bit with a value of one. The first bit with a value of one within the Variable Size Integer is the VINT_MARKER.

1.5.3. VINT_DATA

The VINT_DATA portion of the Variable Size Integer includes all data that follows (but not including) the VINT_MARKER until end of the Variable Size Integer whose length is derived from the VINT_WIDTH. The bits required for the VINT_WIDTH and the VINT_MARKER combined use one bit per octet of the total length of the Variable Size Integer. Thus a Variable Size Integer of 1 octet length supplies 7 bits for VINT_DATA, a 2 octet length supplies 14 bits for VINT_DATA, and a 3 octet length supplies 21 bits for VINT_DATA. If the number of bits required for VINT_DATA are less than the bit size of VINT_DATA, then VINT_DATA may be zero-padded to the left to a size that fits. The VINT_DATA value MUST be expressed as a big-endian unsigned integer.

1.5.4. VINT Examples

This table shows examples of Variable Size Integers at widths of 1 to 5 octets. The Representation column depicts a binary expression of Variable Size Integers where VINT_WIDTH is depicted by '0', the VINT_MARKER as '1', and the VINT_DATA as 'x'.

Note that data encoded as a Variable Size Integer may be rendered at octet widths larger than needed to store the data. In this table a binary value of 0b10 is shown encoded as different Variable Size Integers with widths from one octet to four octet. All four encoded examples have identical semantic meaning though the VINT_WIDTH and the padding of the VINT_DATA vary.

1.6. Element ID

The Element ID MUST be encoded as a Variable Size Integer. By default, EBML Element IDs may be encoded in lengths from one octet to four octets, although Element IDs of greater lengths may be used if the octet length of the EBML Document's longest Element ID is declared in the EBMLMaxIDLength Element of the EBML Header. The VINT_DATA component of the Element ID MUST NOT be set to either all zero values or all one values. The VINT_DATA component of the Element ID MUST be encoded at the shortest valid length. For example, an Element ID with binary encoding of 1011 1111 is valid, whereas an Element ID with binary encoding of 0100 0000 0011 1111 stores a semantically equal VINT_DATA but is invalid because a shorter VINT encoding is possible. The following table details this specific example further:

The octet length of an Element ID determines its EBML Class.

1.7. Element Data Size

The Element Data Size expresses the length in octets of Element Data. The Element Data Size itself MUST be encoded as a Variable Size Integer. By default, EBML Element Data Sizes can be encoded in lengths from one octet to eight octets, although Element Data Sizes of greater lengths MAY be used if the octet length of the EBML Document's longest Element Data Size is declared in the EBMLMaxSizeLength Element of the EBML Header. Unlike the VINT_DATA of the Element ID, the VINT_DATA component of the Element Data Size is not required to be encoded at the shortest valid length. For example, an Element Data Size with binary encoding of 1011 1111 or a binary encoding of 0100 0000 0011 1111 are both valid Element Data Sizes and both store a semantically equal value.

Although an Element ID with all VINT_DATA bits set to zero is invalid, an Element Data Size with all VINT_DATA bits set to zero is allowed for EBML Data Types which do not mandate a non-zero length. An Element Data Size with all VINT_DATA bits set to zero indicates that the Element Data of the Element is zero octets in length. Such an Element is referred to as an Empty Element. If an Empty Element has a default value declared then that default value MUST be interpreted as the value of the Empty Element. If an Empty Element has no default value declared then the semantic meaning of Empty Element is defined as part of the definition of the EBML Element Types.

An Element Data Size with all VINT_DATA bits set to one is reserved as an indicator that the size of the Element is unknown. The only reserved value for the VINT_DATA of Element Data Size is all bits set to one. This rule allows for an Element to be written and read before the size of the Element is known; however unknown Element Data Size values SHOULD NOT be used unnecessarily. An Element with an unknown Element Data Size MUST be a Master-element in that it contains other EBML Elements as sub-elements. Master-elements MAY only use an unknown size if the unknownsizeallowed attribute of the EBML Schema is set to true. The end of a Master-element with unknown size is determined by the beginning of the next element that is not a valid sub-element of that Master-element. An Element with an unknown Element Data Size is referred to as an Unknown-Sized Element.

For Element Data Sizes encoded at octet lengths from one to eight, this table depicts the range of possible values that can be encoded as an Element Data Size. An Element Data Size with an octet length of 8 is able to express a size of 2^56-2 or 72,057,594,037,927,934 octets (or about 72 petabytes). The maximum possible value that can be stored as Element Data Size is referred to as VINTMAX.

If the length of Element Data equals 2^(n*7)-1 then the octet length of the Element Data Size MUST be at least n+1. This rule prevents an Element Data Size from being expressed as a reserved value. For example, an Element with an octet length of 127 MUST NOT be encoded in an Element Data Size encoding with a one octet length. The following table clarifies this rule by showing a valid and invalid expression of an Element Data Size with a VINT_DATA of 127 (which is equal to 2^(1*7)-1).

1.8. EBML Element Types

Each defined EBML Element MUST have a declared Element Type. The Element Type defines a concept for storing data that may be constrained by length, endianness, and purpose.

1.9. EBML Document

An EBML Document is comprised of only two components, an EBML Header and an EBML Body. An EBML Document MUST start with an EBML Header which declares significant characteristics of the entire EBML Body. An EBML Document MAY only consist of EBML Elements and MUST NOT contain any data that is not part of an EBML Element. The initial EBML Element of an EBML Document and the Elements that follow it are considered Level 0 Elements. If an EBML Master-element is considered to be at level N and it contains one or many other EBML Elements then the contained Elements shall be considered at Level N+1. Thus a Level 2 Element would have to be contained by a Master-element (at Level 1) that is contained by another Master-element (at Level 0).

1.9.1. EBML Header

The EBML Header is a declaration that provides processing instructions and identification of the EBML Body. The EBML Header may be considered as analogous to an XML Declaration. All EBML Documents MUST begin with a valid EBML Header.

The EBML Header documents the EBML Schema (also known as the EBML DocType) that may be used to semantically interpret the structure and meaning of the EBML Document. Additionally the EBML Header documents the versions of both EBML and the EBML Schema that were used to write the EBML Document and the versions required to read the EBML Document.

The EBML Header consists of a single Master-element with an Element ID of 'EBML'. The EBML Header MUST ONLY contain EBML Elements that are defined as part of the EBML Specification.

All EBML Elements within the EBML Header MUST NOT utilize any Element ID with a length greater than 4 octets. All EBML Elements within the EBML Header MUST NOT utilize any Element Data Size with a length greater than 4 octets.

1.9.2. EBML Body

All data of an EBML Document following the EBML Header may be considered the EBML Body. The end of the EBML Body, as well as the end of the EBML Document that contains the EBML Body, is considered as whichever comes first: the beginning of a new level 0 EBML Header or the end of the file. The EBML Body MAY only consist of EBML Elements and MUST NOT contain any data that is not part of an EBML Element. Although the EBML specification itself defines precisely what EBML Elements are to be used within the EBML Header, the EBML specification does not name or define what EBML Elements are to be used within the EBML Body. The definition of what EBML Elements are to be used within the EBML Body is defined by an EBML Schema.

1.10. EBML Stream

An EBML Stream is a file that consists of one or many EBML Documents that are concatenated together. An occurrence of a Level 0 EBML Header marks the beginning of an EBML Document.

1.11. Elements semantic

1.11.1. EBML Schema

An EBML Schema is an XML Document that defines the properties, arrangement, and usage of EBML Elements that compose a specific EBML Document Type. The relationship of an EBML Schema to an EBML Document may be considered analogous to the relationship of an XML Schema to an XML Document. An EBML Schema MUST be clearly associated with one or many EBML Document Types. An EBML Schema must be expressed as well-formed XML. An EBML Document Type is identified by a unique string stored within the EBML Header element called DocType; for example matroska or webm.

As an XML Document, the EBML Schema MUST use <EBMLSchema> as the top level element. The <EBMLSchema> element MAY contain <element> sub-elements. Each <element> defines one EBML Element through the use of several attributes which are defined in the section on Section 1.11.1.1. EBML Schemas MAY contain additional attributes to extend the semantics but MUST NOT conflict is the definitions of the <element> attributes defined within this specification.

Within the EBML Schema each EBML Element is defined to occur at a specific level. For any specificied EBML Element that is not at level 0, the Parent EBML Element refers to the EBML Master-element that that EBML Element is contained within. For any specifiied EBML Master-element the Child EBML Element refers to the EBML Elements that may be immediately contained within that Master-element. For any EBML Element that is not defined at level 0, the Parent EBML Element may be identified by the preceding <element> node which has a lower value as the defined level attribute. The only exception for this rule are Global EBML Elements which may occur within any Parent EBML Element within the restriction of the Global EBML Element's range declaration.

An EBML Schema MUST declare exactly one Element at Level 0 (referred to as the Root Element) that MUST occur exactly once within an EBML Document. The Root Element MUST be mandatory (with minOccurs set to 1) and MUST be defined to occur exactly once (maxOccurs set to 1). Note that the EBML and Void Elements may also occur at Level 0 but are not considered to be Root Elements.

Elements defined to only occur at Level 1 are known as Top-Level Elements.

The EBML Schema does not itself document the EBML Header, but documents all data of the EBML Document that follows the EBML Header. The EBML Header itself is documented by this specification in the Section 1.11.2 section. The EBML Schema also does not document Global Elements that are defined by the EBML Specification (namely Void and CRC-32).

1.11.1.1. EBML Schema Element Attributes

Within an EBML Schema the <EBMLSchema> uses the following attributes to define the EBML Schema:

Within an EBML Schema the <element> uses the following attributes to define an EBML Element.

The <element> nodes shall contain a description of the meaning and use of the EBML Element stored within one or many <documentation> sub-elements. The <documentation> sub-element may use a lang attribute which may be set to the RFC 5646 value of the language of the element's documentation. The <documentation> sub-element may use a type attribute to distinguish the meaning of the documentation. Recommended values for the <documentation> sub-element's type attribute include: definition, rationale, usage notes, and references.

The <element> nodes MUST be arranged hierarchically according to the permitted structure of the EBML Document Type. An <element> node that defines an EBML Element which is a Child Element of another Parent Element MUST be stored as an immediate sub-element of the <element> node that defines the Parent Element. <element> nodes that define Level 0 Elements and Global Elements should be sub-elements of <EBMLSchema>.

1.11.1.2. EBML Schema Example

<?xml version="1.0" encoding="utf-8"?>
<EBMLSchema docType="files-in-ebml-demo" version="1">
 <!-- Root Element-->
 <element name="Files" level="0" id="0x1946696C" type="master">
  <documentation lang="en" type="definition">Container of data and
  attributes representing one or many files.</documentation>
  <element name="File" level="1" id="0x6146" type="master" minOccurs="1"
  maxOccurs="unbounded">
   <documentation lang="en" type="definition">An attached file.
   </documentation>
   <element name="FileName" level="2" id="0x614E" type="utf-8"
   minOccurs="1">
    <documentation lang="en" type="definition">Filename of the attached
    file.</documentation>
   </element>
   <element name="MimeType" level="2" id="0x464D" type="string"
     minOccurs="1">
    <documentation lang="en" type="definition">MIME type of the
    file.</documentation>
   </element>
   <element name="ModificationTimestamp" level="2" id="0x4654"
     type="date" minOccurs="1">
    <documentation lang="en" type="definition">Modification timestamp of
    the file.</documentation>
   </element>
   <element name="Data" level="2" id="0x4664" type="binary"
     minOccurs="1">
    <documentation lang="en" type="definition">The data of the
    file.</documentation>
   </element>
  </element>
 </element>
</EBMLSchema>

1.11.1.3. Identically Recurring Elements

An Identically Recurring Element is an Element that may occur within its Parent Element more than once but that each recurrence within that Parent Element MUST be identical both in storage and semantics. Identically Recurring Elements are permitted to be stored multiple times within the same Parent Element in order to increase data resilience and optimize the use of EBML in transmission. Identically Recurring Elements SHOULD include a CRC-32 Element as a Child Element; this is especially recommended when EBML is used for long-term storage or transmission. If a Parent Element contains more than one copy of an Identically Recurring Element which includes a CRC-32 Child Element then the first instance of the Identically Recurring Element with a valid CRC-32 value should be used for interpretation. If a Parent Element contains more than one copy of an Identically Recurring Element which does not contain a CRC-32 Child Element or if CRC-32 Child Elements are present but none are valid then the first instance of the Identically Recurring Element should be used for interpretation.

1.11.1.4. Expression of range

The range attribute MUST only be used with EBML Elements that are either signed integer, unsigned integer, or float. The range attribute does not support date EBML Elements. The range expression may contain whitespace for readability but whitespace within a range expression MUST NOT convey meaning. The expression of the range MUST adhere to one of the following forms:

• x-y where x and y are integers or floats and y must be greater than x, meaning that the value must be greater than or equal to x and less than or equal to y.
• >x where x is an integer or float, meaning that the value MUST be greater than x.
• x where x is an integer or float, meaning that the value MUST be equal x.

The range may use the prefix not to indicate that the expressed range is negated. Please also see the section on Section 1.11.1.5.

1.11.1.5. Textual expression of Floats

When a float value is represented textually in an EBML Schema, such as within a default or range value, the float values MUST be expressed as a Hexadecimal Floating-Point Constants as defined in the C11 standard ISO/IEC 9899:2011 (see section 6.4.4.2 on Floating Constants). The following table provides examples of expressions of float ranges.

Within an expression of a float range, as in an integer range, the - (hyphen) character is the separator between the minimal and maximum value permitted by the range. Note that Hexadecimal Floating-Point Constants also use a - (hyphen) when indicating a negative binary power. Within a float range, when a - (hyphen) is immediately preceded by a letter p, then the - (hyphen) is a part of the Hexadecimal Floating-Point Constant which notes negative binary power. Within a float range, when a - (hyphen) is not immediately preceded by a letter p, then the - (hyphen) represents the separator between the minimal and maximum value permitted by the range.

1.11.1.6. Note on the Use of default attributes to define Mandatory EBML Elements

If a Mandatory EBML Element has a default value declared by an EBML Schema and the EBML Element's value is equal to the declared default value then that Element is not required to be present within the EBML Document if its Parent EBML Element is present. In this case, the default value of the Mandatory EBML Element may be assumed although the EBML Element is not present within its Parent EBML Element. Also in this case the parser of the EBML Document MUST insert the defined default value of the Element.

If a Mandatory EBML Element has no default value declared by an EBML Schema and its Parent EBML Element is present than the EBML Element must be present as well. If a Mandatory EBML Element has a default value declared by an EBML Schema and its Parent EBML Element is present and the EBML Element's value is NOT equal to the declared default value then the EBML Element MUST be used.

This table clarifies if a Mandatory EBML Element MUST be written, according to if the default value is declared, if the value of the EBML Element is equal to the declared default value, and if the Parent EBML Element is used.

1.11.1.7. Note on the Use of default attributes to define non-Mandatory EBML Elements

If an EBML Element is not Mandatory, has a defined default value, and is an Empty EBML Element then the EBML Element MUST be interpreted as expressing the default value.

1.11.2. EBML Header Elements

This specification here contains definitions of all EBML Elements of the EBML Header.

1.11.3. Global elements (used everywhere in the format)

Authors' Addresses