INTERNET-DRAFT R. Bullen Document: Wild Packets Category: Informational February 2001 Open Capture Format (OCF) Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC 2026 [1]. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. Abstract Many protocol analyzer users and vendors have identified the need for an open, standards-based trace file format that allows free and easy exchange of captured data, whether it originates from a data network or otherwise. This document outlines the requirements of such a file format and a proposal that satisfies them--the Open Capture Format (OCF). Table of Contents 1. Introduction 1.1 Problem Statement 1.2 Mission Statement 2. Conventions and Definitions 2.1 Requirement Word Usage 2.2 Definitions 2.3 C-Style Data Types 3. OCF File Requirements 4. OCF File Structure 4.1 OCFAttribute Record 4.1.1 The Tag Field Bullen Informational - Expiration [Page 1] draft-bullen-ocf-00.txt November 2000 Open Capture Format 4.1.2 The Value Field 5. Special OCFAttribute Records 5.1 OCFAttributeBlank 5.2 OCFAttributeCount 5.3 OCFAttributeDataLength 5.4 OCFAttributeValueExtension 6 OCF File Contents and Sections 6.1 Preamble 6.1.1 Signature and Version Number 6.1.2 Byte Order Indicator 6.1.3 File Cyclic Redundancy Check (CRC) 6.1.4 Compression Indicator 6.1.5 HeaderPosition 6.2 OCF File Sections 6.2.1 Header Section 6.2.2 Session Section 6.2.3 Interface Section 6.2.4 ProtocolStack Section 6.2.5 ProtocolStackIndex Section 6.2.6 Address Section 6.2.7 Record Section 6.2.8 RecordIndex Section 7. OCF Implementation Details and Elaboration 7.1 Cyclic Redundancy Check (CRC) 8. Security Considerations 9. Appendicies 9.1 Tag Allocations 10. References 11. Acknowledgments 12. Author's Address 13. Full Copyright Statement 1. Introduction 1.1 Problem Statement Currently, there are many different popular protocol analyzers in the network analysis industry. One benefit of this fact is that each analyzer has its own feature set and interpretation of network activity and protocol decodes. It is often to the advantage of the network analyst to leverage multiple analyzers when looking at trace files for a "second opinion." In fact, it is common for companies to have at least two different protocol analyzers. However, a major stumbling block in working with multiple protocol analyzers is that they do not exchange information well because each has its own proprietary file format. 1.2 Mission Statement The goal of the Open Capture Format (OCF) is to develop a standards- based file format that all network traffic capturing devices will be able to use to store data. This format will allow the easy exchange Bullen Informational - Expiration [Page 2] draft-bullen-ocf-00.txt November 2000 Open Capture Format of traces between protocol analyzers and individuals within an organization or around the world. 2. Conventions and Definitions 2.1 Requirement Word Usage The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [2]. 2.2 Definitions 2.2.1 Capture Device The term "capture device" is used in reference to any device that has the ability to locally capture and store data traffic, whether it originates from a computer network (LAN, WAN, etc.) or otherwise. A protocol analyzer is considered a capture device. 2.2.2 OCF Reader The term "OCF reader" is used to specify a hardware or software device capable of reading and interpreting an OCF file. 2.2.3 OCF Writer The term "OCF writer" is used to specify a hardware or software device capable of composing or modifying an OCF file. 2.2.4 OCF Device The term "OCF device" connotes any device that is an OCF reader, OCF writer, or both. In this context it should be taken as any device that has OCF capabilities. 2.3 C-Style Data Types There will be C-style code type declarations and snippets used in this document to illustrate data storage and algorithms. Described here are some data types that should be understood before implementing an OCF device. It is left to OCF device implementers to correctly declare these types in terms of native types for their particular compiler/platform. 2.3.1 Integers Interestingly, the C language does not define fixed sizes for its integer data types but rather size relations between them. For example, a long integer must be greater than or equal to an integer, which must be greater than or equal to a short integer. Therefore Bullen Informational - Expiration [Page 3] draft-bullen-ocf-00.txt November 2000 Open Capture Format integer data type sizes vary by compiler/platform implementation. In order to avoid ambiguity, compiler/platform prejudice, and because it is expected that this file format will be supported on a wide range of hardware, C integer data types (such as unsigned short and unsigned long) will not be used. Instead, integer data types will be declared in the following format (stated here in ABNF as specified in RFC 2234 [3]): integer-data-type = integer-sign-prefix "Int" integer-size-suffix integer-sign-prefix = "S" / "U" integer-size-suffix = "8" / "16" / "32" / "64" The prefixes S and U stand for signed and unsigned, respectively, and the suffix number represents the number of bits the integer occupies in memory. Hence, SInt8 is a signed, 8-bit integer and UInt32 is an unsigned, 32-bit integer. 2.3.2 Characters and Strings We are at a point in computer history where we are transitioning from legacy single byte character sets to multi-byte character sets in order to support the many glyphs required by different cultures and languages around the world. The OCF reflects this fact by containing both the ASCII (the predominant standard of the past) and UNICODE (the standard to which the world is converging) character set standards. The presence of ASCII is limited to a single place in the file: the File Signature and Version Number (see 6.1.1) at the very beginning. All other strings in the file will be in UNICODE format. Therefore, two character types must be declared: ASCIIChar - single-byte character type (most likely declared as a char) UNICODEChar - double-byte character type (most likely declared as a wchar) 3 OCF File Requirements Several key requirements were recognized as being necessary in the OCF file format: - Use open, standards-based technology whenever possible. - Allow files to be interchanged in a manner independent of platform, vendor, and capture device. - Be highly extensible, allowing for vendor-specific data to be stored in private areas. Bullen Informational - Expiration [Page 4] draft-bullen-ocf-00.txt November 2000 Open Capture Format - Allow files to carry forward previous vendor-specific, private, or unrecognized information when they are rewritten. - Be capable of storing large amounts of data (e.g. greater than the 32-bit/4 gigabyte boundary). - Support multiple capture sessions within a single file (e.g. on a scheduled basis). - Support multiple capture devices per session (e.g. multiple interfaces for full-duplex simulated capture). - Contain other files as attachments (e.g. text, images, audio, video, etc.). - Allow files to be natively compressed using a standard compression algorithm because of the ever-increasing speed of networks and, consequently, the ever-increasing size of trace files. - Allow for the possibility of real-time logging or streaming to disk captures. It should be noted that throughout the file format, trade-offs between extensibility/interchangeability and space/performance arise. In almost all cases the OCF has elected in favor of the former as it reflects the OCF file format's top priority. 4 OCF File Structure The OCF file format is a tag-based schema, similar to the popular tagged image file format (TIFF), where a unique tag is used for each element or attribute in the file. This section of the draft describes the structure and interpretation of the OCF tagging system. 4.1 OCFAttribute Record The OCF file structure employs a tag-value attribute pair, known as the OCFAttribute record, to identify, store, and locate data in the OCF file. The OCFAttribute record is 6 octets in size and is composed of a 2-octet Tag field followed by a 4-octet Value field. Below is a C-style example declaration of the OCFAttribute record. typedef struct { UInt16 Tag; UInt32 Value; } OCFAttribute; 4.1.1 The Tag Field The Tag field of an OCFAttribute record will always be interpreted as a 16-bit unsigned integer (which can contain values 0-65,535) and Bullen Informational - Expiration [Page 5] draft-bullen-ocf-00.txt November 2000 Open Capture Format will always be an indicator of how to interpret the Value field. Tag ranges are allocated as follows: Decimal Hex Description ----- ----- ----- 0- 0000- Special OCFAttribute records. 127 007F 128- 0080- Reserved for future use. 255 00FF 256- 0100- Globally defined, well-known OCFAttributes 16,383 03FF (currently grouped into blocks of 32 per section with gaps of 32 in between for expansion). 16,384- 0400- Reserved for future use. 32,767 07FF 32,768- 0800- Reserved for private use by individual vendors 65,279 FEFF for implementation-specific features. However, tags in this range cannot be used until they are officially allocated to vendors upon request to and approval by the governing OCF standards body. This tag space will be allocated to vendors in blocks of 16. 65,280- FF00- Reserved for public use by any party that 65,534 FFFE wishes to implement site-specific or experimental features. The OCF standards body will not monitor these tags. Parties should use them at their own risk and only if the OCF files will not propagate out of the site. 65,535 FFFF Reserved for future use. 4.1.2 The Value Field The meaning of the Value field of an OCFAttribute record will be dependent upon the Tag and doesnĘt necessarily have to represent an unsigned integer or even an ordinal at all, but it will always be 32 bits in size. Some commonly used interpretations of the Value field are defined below: - UInt32: A 32-bit unsigned integer that stores a numerical value or bit-field. - Offset: A 32-bit unsigned integer that stores the position of data within the file relative to the beginning (0 = nil/null/empty). In this document, the word "pointer" will be used loosely with regard to this data type and should be interpreted to mean, "offset into the file" or "position in the file." Bullen Informational - Expiration [Page 6] draft-bullen-ocf-00.txt November 2000 Open Capture Format The OCF shall attempt to preserve the meaning of the Value field set to all 0s and all 1s as "ignored" and "unknown", respectively, where applicable. 5 Special OCFAttribute Records The OCF has recognized the need for several special OCFAttribute records. These attributes help reduce the requirements for an OCF device to know of a large number of predefined tags and to allow for changes in a tag to be accommodated without necessarily requiring changes in existing code. Tags in the reserved range 0-127 will identify these special OCFAttribute records, as specified in 4.1.1. 5.1 OCFAttributeBlank The OCFAttributeBlank attribute has been reserved for use as a simple blank space or as a temporary placeholder. As a blank space, it can be used anytime a pad of zeros in the OCF file is desired. As a placeholder, this attribute can be used when writing to an OCF file is required but certain data that must exist in the file but is not available until a later time. When the data finally is known, the OCF writer can go back and replace the placeholder attribute by the real attribute. Whenever an OCF reader encounters an OCFAttributeBlank record, the attribute should be ignored. Tag: 0 Value: UInt32 = 0 (should be ignored but must always be set to zero) 5.2 OCFAttributeCount The OCFAttributeCount attribute specifies the number of OCFAttribute records immediately following it. This must be the first attribute at the beginning of a section, or series of attribute records, so that the OCF reader knows how many attributes to read in before moving on to a new section. This also allows the reader to read in a series of attributes at one time, possibly making the reader more efficient. Tag: 1 Value: UInt32 (indicates the number of immediately following OCFAttribute records) 5.3 OCFAttributeDataLength The OCFAttributeDataLength is used in conjunction with variable length data such as frames, character strings, or attached/embedded files. Any time an OCF reader jumps from an Offset which pointed to a variable length data item, this must be the first record encountered. There are many benefits to reserving a single tag for length. One is that when storing variable sized items, such as user comments, file attachments, frame data, and vendor specific data Bullen Informational - Expiration [Page 7] draft-bullen-ocf-00.txt November 2000 Open Capture Format areas, there need not be two predefined tags for each--the first identifying the position and the second identifying the length. Another is that this allows chained data, such as frames, to be placed sequentially in the file and easily traversed. By looking at the OCFAttributeDataLength record and seeking forward by the amount specified in the Value field, the next read operation would be positioned at the next OCFAttribute structure. Of course, an OCFAttributeDataLength with Value set to 0 will be an ending delimiter. Third, vendor specific information can be carried along with the file, even if another vendor reads and rewrites the file. Tag: 2 Value: UInt32 (indicates the number of octets of data immediately following the OCFAttributeDataLength record) 5.4 OCFAttributeValueExtension The OCFAttributeValueExtension immediately precedes any OCFAttribute whose Value field needs to be extended in size to a multiple of 32- bits. By positioning it in front of (rather than behind) the value it modifies and making it the upper 32 bits, it won't be necessary to emember the last attribute read and modify it when encountering the CFAttributeExtension. Instead, when encountering the OCAttributeExtension, just shift its Value field left by 32 bits and perform a bitwise OR with the next attribute to obtain a 64-bit value. The actual OCFAttribute's Value field that is being modified does not change and remains the low order 32 bits of the concatenated bit string. This operation is chainable. That is, if two OCFAttributeExtensions are found in succession, then 64 bits will be added to the next 32-bit value, yielding a 96-bit Offset or unsigned integer, etc. Of course, it shouldn't be used in place of a pointer to a large chunk of data. By using the OCFAttributeValueExtension with offset data types, the problem of the 4-gigabyte file limitation (due to the Offset data types being 32 bits) is solved. Another example of this attribute's usefulness is not having to define two attributes for one piece of information. Imagine that there is an OCFAttribute identified by Tag = 100. The Value portion is a bit-field consisting of 32 flags. Several years after this attribute has been standardized, a 33rd bit flag is required. Rather than defining a new attribute with a new Tag (probably not even close to 100, so it isn't even logically grouped) and Value, one can use the OCFAttributeValueExtension attribute and the new flag can be viewed as the high-order 33rd bit. Tag: 3 Value: (interpretation dependent upon immediately succeeding attribute) 6 OCF File Contents and Sections Bullen Informational - Expiration [Page 8] draft-bullen-ocf-00.txt November 2000 Open Capture Format This section of the OCF Draft describes the different sections present in an OCF file and how they are organized. Some special notes about the contents of the file are: - All strings in the file, such as comments, must be in UNICODE format, with the exception of the File Signature and Version Number, which is located in the OCF File Preamble at the beginning of the file and will be a US-ASCII string. UNICODE is a fully compatible implementation of the ISO/IEC 10646-1:2000 standard. 6.1 Preamble The OCF Preamble occupies the first 256 bytes of the file. Its purpose is to provide vital information needed to proceed reading the file. It contains the following fields: - Signature and Version Number: Identifies the file as OCF and in which version of the OCF standard the file is structured. - Byte Order Indicator: Indicates the byte ordering of the file. - File Cyclic Redundancy Check (CRC) - Provides a means for validating the contents of the file. - Compression Indicator: Indicates whether the remainder of the file is compressed. - HeaderPosition: An OCFAttribute structure that specifies the position of the Header section, which is the starting point and table of contents of the file. The preamble is the only portion of the file that does not completely conform to the OCFAttribute tagged-value format. Instead it can be easily accessed through the following C-style structure: typedef struct { ASCIIChar[208] SignatureAndVersionNumber; UInt8 ByteOrderIndicator; UInt32 FileCRC; UInt8 CompressionIndicator; } OCFPreamble; The ordering of fields within the OCFPreamble structure reflects the way in which an OCF device would parse an OCF file. First, the OCF device confirms that the file is truly in OCF format by checking the Signature and Version Number. Next, the OCF device must know which byte ordering the file is in and so it consults the ByteOrderIndicator. Once the byte ordering is known, the FileCRC can be reordered if necessary and used to validate the contents of the file. Finally, before accessing the actual data content of the file, the OCF device must know whether the file is in compressed form. Bullen Informational - Expiration [Page 9] draft-bullen-ocf-00.txt November 2000 Open Capture Format Notice that the HeaderPosition, although considered part of the preamble, is not included in the OCFPreamble structure's declaration. This is due to the fact that the HeaderPosition is an OCFAttribute, which may have one or more OCFAttributeValueExtension attributes preceding it. See the HeaderPosition explanation for more information. Each of the preamble fields is described in more detail below. 6.1.1 Signature and Version Number The OCF Signature and Version Number is the first field in the OCF File Preamble and therefore the first data encountered in the file. It occupies the first 208 bytes (not all of them used at this point) and consists of a human readable US-ASCII character string containing the file signature and version number. This is to uniquely identify and verify OCF documents without regard to file name/extension or state of the OCF file layout. The OCF file signature is "Open Capture Format (OCF) Document", followed immediately by a carriage-return line-feed character sequence (0x0D0A in hexadecimal). The file version string has the format "Version: x.y", where x and y are whole numbers greater than or equal to zero, and which is also followed by the carriage-return line-feed character sequence. Once the standard is in working practice, the version number will commence at 1.0. For testing purposes and implementations used before then, it is recommended that OCF writers use version numbers less than that. The two strings are concatenated together and terminated with the null character (0x00) followed by the SUB or Ctrl-Z character (more commonly known as the "end of file" character) or (0x1A). The remaining unused portion (the 156 bytes following the currently 56-byte long signature and version) of the 208 bytes must be set to all zeros or null characters. Hence, the very beginning of all official OCF files after this format has been put into working practice shall be as follows (where hexadecimal values between angle brackets are interpreted as the literal value of the associated characters and with no line breaks): Open Capture Format (OCF) Document<0x0D0A>Version: 1.0<0x0D0A><0x00><0x1A><0x00><0x00>...<0x00>...<0x00><0x00> Or as it would appear outputted to a terminal: Open Capture Format (OCF) Document Version: 1.0 ^Z The contents of the file signature and version number string may change over time and so, too, will the length of the string. Therefore, it is recommended that OCF implementers scan for the "Version: " sub-string to find the version number rather than Bullen Informational - Expiration [Page 10] draft-bullen-ocf-00.txt November 2000 Open Capture Format seeking to absolute positions in the file and assuming that the file contents will properly align with hard-coded offsets. Interpretation and incrementing of the Version number shall be as follows: additions impose minor version increments; changes impose major version increments. Any small additions to the standard will cause a minor version increment where the y is incremented in the x.y version format. A small addition is defined as any feature that can be safely introduced to the standard without requiring modifications to the way an OCF device reads, writes, or interprets the file structure. In other words, if a new feature is added and an older OCF device reads a file that contains the new feature, no compatibility issues will arise, assuming that the OCF device ignores the new feature as it should. Therefore, all OCF readers implementing version 1.0 of the standard should theoretically be able to read 1.9 versions of the standard. Hence, backward compatibility is the primary concern when implementing new additions. Major version increments (x is incremented in the x.y version format) will occur when significant changes are made to the standard such that there may be compatibility issues when older OCF devices try to read new versions of the file. For example, if the OCF standards body runs out of tag-space and decides to make tags 32 bits instead of 16-bits, then this is a major version increment because, obviously, that change would break all existing code. 6.1.2 Byte Order Indicator The second field of the OCF Preamble is the Byte Order Indicator, which is one octet long. This field is necessary to fulfill the platform independent goal of this file format. Rather than having a single, mandatory byte ordering, the OCF allows for two byte orderings: big endian and little endian. This is advantageous for sites that use nearly all one platform and/or specialized capture hardware. By allowing sites to use native byte ordering while reading and writing to a file, the performance penalty of rearranging bytes to conform to the mandatory ordering is not incurred every time a file is read or written. Instead, the penalty is only encountered when exchanging an OCF file between platforms. In addition, specialized hardware or software that writes to disk as it captures doesn't have to deal with the overhead of byte reversing or swapping. The possible values that the Byte Order Indicator can hold and their meanings are: - 0x00: Little endian. - 0x01: Big endian. - 0x02 - 0xFF: Reserved for future use. Byte ordering pertains to all data structures in the file except for the following items: Bullen Informational - Expiration [Page 11] draft-bullen-ocf-00.txt November 2000 Open Capture Format - Frames and cells will maintain network byte ordering. In other words, frames and cells will be stored in exactly the same byte order as they appeared on the network. - MAC addresses in the Address section will be stored in canonical format (or Ethernet format), which is the way the IEEE chooses to represent and allocate Organizationally Unique Identifiers (OUIs). - All other (non-MAC) addresses will be stored in network order. - UNICODE character strings are exempt from byte ordering when it comes to storage because the UNICODE standard specifies that they are a sequence of two-byte characters with the most significant byte positioned in front of the least significant byte (canonical form). 6.1.3 File Cyclic Redundancy Check (CRC) The File Cyclic Redundancy Check (CRC) is a 32-bit value in the third field of the OCF Preamble and is used to verify that the contents of the OCF file have not been corrupted. The CRC is calculated using the standard CCITT crc32 algorithm with the divisor polynomial 0x04C11DB7 over the entire length of the final file (after compression is applied if it is used) with the CRC field set to zero. The calculation of this field is optional and must be set to zero if the CRC calculation is not performed. If the file CRC has been calculated (non-zero), then an OCF reader should verify that the file is valid by computing the CRC on its own and comparing it to the one present in the file. For more information on the CRC and the algorithm used to implement it, see the OCF Implementation Details and Elaboration section. 6.1.4 Compression Indicator The Compression Indicator is a one-octet field that occupies the forth position in the OCF Preamble. Its purpose is to indicate whether compression was used for the remainder of the file beginning immediately after the OCF Preamble (after the HeaderPosition). The possible values that the Compression Indicator can hold and their meanings are: - 0x00: Compression is not used. - 0x01: DEFLATE compression is used. - 0x02 - 0xFF: Reserved for future use. For information on how compression is used in the OCF format, see the OCF Implementation Details and Elaboration and the DEFLATE reference []. 6.1.5 HeaderPosition Bullen Informational - Expiration [Page 12] draft-bullen-ocf-00.txt November 2000 Open Capture Format The HeaderPosition is the fifth and final field in the OCF Preamble. It is the first OCFAttribute found in the file and is simply a position specifier for the Header section, which is the starting place for obtaining all other information in the file. This attribute is subject to an OCFAttributeValueExtension attribute preceding it in the event that the Header section is located beyond 4 GB into the file. If compression is used in the file, then it will commence immediately after the HeaderPosition attribute. 6.2 OCF File Sections This portion of the OCF Draft outlines the logical grouping of related data, or rather OCFAttributes describing and pointing to data, into sections. Each section has its own range of attribute tags, so if you know the value of an attribute's Tag field, you can determine the section in which it belongs. Each section has an OCFAttribute that points to it from other sections. This attribute will be called the section's Position attribute and will have a name of the format Position. For example HeaderPosition is the position attribute for the Header section. The position attribute will own the lowest possible tag number in a section's tag-space. For example, the Header section, which identifies the creator of an OCF file and/or capture session, occupies the tag-space 512-639. Hence, the HeaderPosition attribute has a Tag value of 512. In general, there are no specific ordering requirements for attributes within a section. One exception is that the OCFAttributeCount must be the first attribute found in every section, so that the reader knows how many OCFAttribute records are present in the section. Another exception, which is common sense but is stated for completeness, is that any Index section that acts as an ordered lookup table must have all of its position attributes in order. For example, the RecordIndex section, which is an ordered table of all stored frames and/or cells captured in a session, will have the OCFAttributeCount as its first attribute as all sections do. The OCFAttributeCount attribute must be followed by a RecordPosition pointer as its first attribute, RecordPosition1 as its second attribute,..., RecordPositionN as its final attribute. Succinctly, no ordering of attributes is assumed in a section unless otherwise stated. All of the sections described here have attribute tags ordered numerically for the sake of convenience only. Keep in mind that when an attribute has a description saying something to the effect of "pointer to character string", "pointer to private data", or pointer to anything that is of variable length, that it is actually a pointer to an OCFAttributeDataLength record that specifies the length of the immediately following data. Bullen Informational - Expiration [Page 13] draft-bullen-ocf-00.txt November 2000 Open Capture Format There is a table in each section below that presents all of the possible attributes that can be found in it. Here is an example of one of these tables and an explanation of what each column means. Attribute Tag Tag Value Count Description Name Dec Hex Type ----- --- --- ----- ----- ----- OCFAttributeCount 1 0001 UInt32 1 Specifies the number of immediately following attributes in this section. Must be the first attribute in the section. The Attribute Name column specifies a name for the attribute that is used in this text and can be used as a constant identifier in code. The Tag Dec and Tag Hex columns give the 16-bit value of of the attribute's tag. The Value Type column displays the way the attribute's 32-bit Value field should be interpreted. The Description column provides a brief summary of what the purpose of attribute is and any other additional comments that are relevant. The Count column is used to indicate how many of the attribute can appear in a section. For example, a 0/1 means that the attribute is optional but if it is present then there can be only one. 0+ in the count column means zero or more. A 1 means there must be one and only one of the attribute in the section. A 1+ means at least one of the attribute must be present but there could be more. 6.2.1 Header Section The Header section acts as a table of contents. Its position in an OCF file is not restricted to the beginning and in fact can be Bullen Informational - Expiration [Page 14] draft-bullen-ocf-00.txt November 2000 Open Capture Format positioned anywhere in the file. The Header section is located by following the HeaderPosition located in the OCF File Preamble. The reason for allowing the Header section to be anywhere in the file is for flexibility when rewriting an OCF file. For example, if a new capture session is to be appended to an existing file, then the header will need to be changed and expanded by a few bytes. In this case, rather than completely rewriting the file, it is much less resource intensive to position the updated header at the new end of the file and update the HeaderPosition attribute at the beginning of the file. The Header section is referenced from the OCF File Preamble's HeaderPosition field. Header section tag space: 256-287 (0100-011F) HeaderPosition.Tag = 256 Attribute Tag Tag Value Count Description Name Dec Hex Type ----- --- --- ----- ----- ----- OCFAttributeCount 1 0001 UInt32 1 Specifies the number of immediately following attributes in this section. Must be the first attribute in the section. HeaderComment 257 0101 Offset 0+ Pointer to a comment string describing the contents of the file. HeaderAttachment 258 0102 Offset 0+ Pointer to an Attached file. SessionPosition 320 0140 Offset 1+ Pointer to a Session section describing a single capture session. There must be one for each capture session contained in the file. 6.2.2 Session Section Bullen Informational - Expiration [Page 15] draft-bullen-ocf-00.txt November 2000 Open Capture Format The Session section contains all data relating to a single capture session in the OCF file. An OCF file can contain any number of capture sessions from different times and capture devices. The Session section is referenced from the Header section. Session section tag space: 320-351 (0140-15F) SessionPosition.Tag = 320 Attribute Tag Tag Value Count Description Name Dec Hex Type ----- --- --- ----- ----- ----- OCFAttributeCount 1 0001 UInt32 1 Specifies the number of immediately following attributes in this section. Must be the first attribute in the section. SessionComment 321 0141 Offset 0+ Pointer to a comment describing the contents of this capture session. SessionID 322 0142 UInt32 1 Zero-based index identifying the session within the file (0 = 1st session, 1 = 2nd session, etc.). SessionStartTime 323 0143 UInt32 1 Value representing the capture start time for this session in seconds since since 00:00:00 GMT, January 1, 1970 (see the C standard library time() function for more Bullen Informational - Expiration [Page 16] draft-bullen-ocf-00.txt November 2000 Open Capture Format information). SessionStopTime 324 0144 UInt32 0/1 Value representing the capture stop time for this session in seconds since 00:00:00 GMT, January 1, 1970 (see the C standard library time() function for more information). SessionVendorDescription 325 0145 Offset 0/1 Pointer to a string describing the vendor company and capture device from which the session was created. InterfacePosition 384 0180 Offset 1+ Pointer to an Interface section describing the interface used for capturing. There must be one for each interface used during this capture session. AddressPosition 576 0240 Offset 0/1 Pointer to the first Address section for this session from which all other addresses can be found in a linked list fashion. RecordPosition 640 0280 Offset 1 Pointer to the first Record section for this session from which all Bullen Informational - Expiration [Page 17] draft-bullen-ocf-00.txt November 2000 Open Capture Format other records can be found in a linked list fashion. RecordIndexPosition 704 02C0 Offset 0/1 Pointer to a RecordIndex section. Present only if the optional RecordIndex section has been generated. Note: Although it appears that the SessionStartTime and SessionStopTime attributes do not provide ample storage for the number of seconds since midnight Jan. 1, 1970, it is believed that the C standard library function time() will be expanded or a new version of the time() function will be created to return a 64-bit value. At that time using the OCFAttributeValueExtension in conjunction with the SessionStartTime and SessionStopTime will accommodate the 64-bit value returned by that function. 6.2.3 Interface Section The Interface section describes a single interface used during a capture session. There may be multiple interfaces used per session. The Interface section is referenced from the Session section and Record section. Interface section tag space: 384-415 (0180-019F) InterfacePosition.Tag = 384 Attribute Tag Tag Value Count Description Name Dec Hex Type ----- --- --- ----- ----- ----- OCFAttributeCount 1 0001 UInt32 1 Specifies the number of immediately following attributes in this section. Must be the first attribute in the section. InterfaceComment 385 0181 Offset 0+ Pointer to a comment Bullen Informational - Expiration [Page 18] draft-bullen-ocf-00.txt November 2000 Open Capture Format describing the configuration of this interface. InterfaceID 386 0182 UInt32 1 Zero-based index identifying the interface within the session (0 = 1st interface, 1 = 2nd interface, etc.). InterfaceTopology 387 0183 UInt32 1 Constant indicating the topology this Interface connects to. InterfaceBandwidth 388 0184 UInt32 1 Integer representing the data rate of this interface in bits per second. InterfaceChannelCount 389 0185 UInt32 1 Value indicating the number of communication channels this interface possesses. As an example, half-duplex and full- duplex interfaces have one and two channels, respectively. ProtocolStackPosition 448 01C0 Offset 1 Pointer to a record that begins a new protocol stack. Used for decoding purposes. AddressPosition 576 0240 Offset 0/1 Pointer to an Address section having Bullen Informational - Expiration [Page 19] draft-bullen-ocf-00.txt November 2000 Open Capture Format both the hardware address and name of this interface. Note: At first glance, the InterfaceBandwidth attribute seems to have the limitation that it can only support bandwidths up to 4 GB. However, the OCFAttributeValueExtension can be used to expand the 32-bit value to any bandwidth. 6.2.4 ProtocolStack Section The ProtocolStack section is designed to assist OCF devices with protocol decoding by providing a protocol encapsulation list. Currently, this section is undefined, but is intended to be an ordered series of protocol identifiers (constants) that represent the encapsulation of one or more frames. For example, an HTTP packet running over an Ethernet using 802.3 encapsulation would be represented by a series of protocol identifiers in the following order: 802.3, IP, TCP, HTTP. Obviously the most important piece of information is the first layer of encapsulation because all other layers can usually be derived from the previous layer. But, the ProtocolStack section can also be useful in implementing protocol forcing, where protocol decodes can be imposed upon the interpreting OCF device. The ProtocolStack section is referenced from the ProtocolStackIndex section. ProtocolStack section tag space: 448-479 (01C0-01DF) ProtocolStackPosition.Tag = 448 Attribute Tag Tag Value Count Description Name Dec Hex Type ----- --- --- ----- ----- ----- OCFAttributeCount 1 0001 UInt32 1 Specifies the number of immediately following attributes in this section. Must be the first attribute in the section. ProtocolStackComment 449 01C1 Offset 0+ Pointer to a comment describing this protocol stack. ProtocolStackID 450 01C2 UInt32 1 Zero-based index identifying the protocol stack within the Bullen Informational - Expiration [Page 20] draft-bullen-ocf-00.txt November 2000 Open Capture Format ProtocolStackIndex section (0 = 1st protocol stack, 1 = 2nd protocol stack, etc.). 6.2.5 ProtocolStackIndex Section The ProtocolStackIndex section functions as an array of pointers to ProtocolStack sections. The ProtocolStackIndex section is referenced from the Interface section. ProtocolStackIndex tag space: 512-543 (0200-021F) ProtocolStackIndex.Tag = 512 Attribute Tag Tag Value Count Description Name Dec Hex Type ----- --- --- ----- ----- ----- OCFAttributeCount 1 0001 UInt32 1 Specifies the number of immediately following attributes in this section. Must be the first attribute in the section. ProtocolStackPosition 448 01C0 Offset 1+ Pointer to a ProtocolStack section. There is a ProtocolStackPosition attribute present for each ProtocolStack section used by the interface from which this section was referenced. 6.2.6 Address Section The Address section describes a single address, whether it be a physical or network address, and any symbolic name(s) associated with that address. The Address section is referenced from the Session section, Interface section, and other Address sections. Address section tag space: 576-607 (0240-025F) AddressPosition.Tag = 576 Bullen Informational - Expiration [Page 21] draft-bullen-ocf-00.txt November 2000 Open Capture Format Attribute Tag Tag Value Count Description Name Dec Hex Type ----- --- --- ----- ----- ----- OCFAttributeCount 1 0001 UInt32 1 Specifies the number of immediately following attributes in this section. Must be the first attribute in the section. AddressComment 577 0241 Offset 0+ Pointer to a character string that explains or expands upon the AddressName. AddressID 578 0242 UInt32 1 Zero-based index identifying the address within the address linked list. AddressNextPosition 579 0243 Offset 1 Pointer to the next Address section in the list of addresses. AddressParentID 580 0244 UInt32 0+ Index of a lower layer Address section that is used on the same hardware device or station. For example, an IP address may have an AddressParentID that is the index of the MAC Address section for the same interface. AddressType 581 0245 UInt32 1 Constant indicating the type of address. These constants have yet to be defined. AddressData 582 0246 Offset 1 Pointer to the actual address. For a MAC address, this will point to a 6-byte long structure (preceded by an OCFAttributeDataLength record). All MAC addresses will be stored in canonical form, all other addresses will be stored in network order. AddressName 583 0247 Offset 0+ Pointer to a character Bullen Informational - Expiration [Page 22] draft-bullen-ocf-00.txt November 2000 Open Capture Format string representing the symbolic name associated with this address. There should be one for each name. 6.2.7 Record Section The Record section is where an individual captured frame or cell is stored. In addition, the Record section can be used to store inter- frame events or information, such as collisions, dropped frames, line state changes, etc. Different attributes will be required depending upon whether the Record section describes a frame or an event, but there will be some attributes that are common to both. Thus, there are three tables below: attributes common to both frames and events, attributes only for frames, and attributes only for events. Although the time stamps are common to both frames and event, instead of being in the first table, they are found in both the event table and the frame table. The time stamp attribute is the same in both cases but its presence has different requirements depending on whether it is stamping a frame or an event. The Record section is referenced from the RecordIndex section, Session section, and other Record sections. Record section tag space: 640-671 (0280-029F) RecordPosition.Tag = 640 Attributes common to both frames and events: Attribute Tag Tag Value Count Description Name Dec Hex Type ----- --- --- ----- ----- ----- OCFAttributeCount 1 0001 UInt32 1 Specifies the number of immediately following attributes in this section. Must be the first attribute in the section. RecordComment 641 0281 Offset 0+ Pointer to a character string that explains or expands upon this frame or event. RecordID 642 0282 UInt32 0/1 A zero-based index of the frame or event's position within the capture Bullen Informational - Expiration [Page 23] draft-bullen-ocf-00.txt November 2000 Open Capture Format session. This attribute is optional but its presence is either ubiquitous or nonexistent. That is, if one record has a RecordID, all records must have a RecordID; if one record does not have a RecordID, all records must not have a RecordID. RecordInterfaceID 643 0283 UInt32 0/1 Index of the Interface section describing the interface on which this event occurred or from which this frame was captured. Present when there are multiple Interface sections in the Session section, meaning multiple interfaces were used for the capture. RecordInterfaceChannel 644 0284 UInt32 0/1 A zero-based index that indicates the channel of communication from which the frame or event was captured when using a multi- channel interface. A half-duplex interface is a single channel medium and thus this attribute will always have a value of 0 and is optional. A full- duplex interface will posses two channels (channel 0 and channel 1) and therefore this Bullen Informational - Expiration [Page 24] draft-bullen-ocf-00.txt November 2000 Open Capture Format attribute will be required. RecordNextPosition 645 0285 Offset 1 Pointer to the next frame in the capture session. This value will be null if this is the last frame. RecordEventType 646 0286 UInt32 0/1 Indicates the type of event this section is. This attribute is optional if this section describes a frame (Value = 0) but is mandatory if this section describes an event. RecordData 647 0287 Offset 0/1 Pointer to the location of the actual captured frame data or additional event description information. RecordStatusFlags 648 0288 UInt32 0/1 Bit mask indicating all error and special flags. Present only when status flags are requires. Event-specific attributes RecordTimeStamp_s 649 0289 UInt32 0/1 Number of seconds since the start of capture specified by the Session section's SessionStartTime. RecordTimeStamp_ns 650 028A UInt32 0/1 Number of nanoseconds in addition to FrameTimeStamp_s. RecordTimeStamp_as 651 028B UInt32 0/1 Number of attoseconds (10-18) in addition to FrameTimeStamp_s and FrameTimeStamp_ns. This field is for future use and will Bullen Informational - Expiration [Page 25] draft-bullen-ocf-00.txt November 2000 Open Capture Format only be present if a highly precise capture device is used. RecordEventRepeatCount 652 028C UInt32 0/1 Number of frames with the same RecordType that occurred after this frame but are not stored. For example, the next 10 frames are not present because they are all dropped like this one. Present only when an event is repeated. Note: The time stamp is optional for an event because the exact time an event occurred is not always known until the next frame is captured. In this case it is assumed that the event took place sometime between the previous and the next time stamps. Frame-specific attributes RecordTimeStamp_s 649 0289 UInt32 1 Number of seconds since the start of capture specified by the Session section's SessionStartTime. RecordTimeStamp_ns 650 028A UInt32 1 Number of nanoseconds in addition to FrameTimeStamp_s. RecordTimeStamp_as 651 028B UInt32 0/1 Number of attoseconds (10- 18) in addition to FrameTimeStamp_s and FrameTimeStamp_ns. This field is for future use and will only be present if a highly precise capture device is used. RecordFrameOriginalLength 653 028C UInt32 0/1 Specifies the size of the sliced frame before it Bullen Informational - Expiration [Page 26] draft-bullen-ocf-00.txt November 2000 Open Capture Format was sliced and is present only when a frame has been sliced. RecordFrameProtocolStackID 654 028D UInt32 0/1 Index of the ProtocoStack section within the ProtocolStackIndex section for the Interface section specified by the RecordInterfaceID. This will aid OCF readers in decoding this frame. Note: Whereas the event time stamps are optional, frame time stamps are mandatory. Note: You may ask yourself why, if there is a RecordFrameOriginalLength, is there no RecordFrameSliceLength that specifies the length of the sliced frame? Remember that all variable sized data in an OCF file (e.g. a sliced frame) is immediately preceded by an OCFAttributeDataLength, which specifies the length of the sliced frame. Note: The three RecordTimeStamp attributes may need a little explanation. The RecordTimeStamp_s field is simply the number of seconds that have elapsed since the start of capture. This is an unsigned 32-bit integer (having values 0-4,294,967,295) yielding a capture that could roughly span about 136 years. The RecordTimeStamp_ns attribute specifies the number of nanoseconds since the value specified by RecordTimeStamp_s. The same holds true for RecordTimeStamp_as with respect to RecordTimeStamp_ns. For example, assume that a trace began at 12:00:00 am and that he first frame didnĘt arrive until 1:00:00.043 821 am. The number of seconds elapsed would be 3600 (number of seconds in 1 hour), which would be the value stored in RecordTimeStamp_s. The frame arrived 43.821 milliseconds after 1:00 am, which is 43,831,000 nanoseconds after the value specified by RecordTimeStamp_s. Therefore value held by RecordTimeStamp_ns is 43,831,000. The FrameTimeStamp_as attribute allows for resolutions down to 10^(-18) seconds (attoseconds) and is for high precision. The greatest value both RecordTimeStamp_ns and RecordTimeStamp_as should hold is 999,999,999. This is because a value 1 greater than that should be stored in the next lower resolution container. For example, a value of 1,000,000,000 stored in RecordTimeStamp_ns represents 1,000,000,000 nanoseconds, or 1 second. Therefore, the value of RecordTimeStamp_s should be incremented by 1 and RecordTimeStamp_ns should hold the value 0. 6.2.8 RecordIndex Section Bullen Informational - Expiration [Page 27] draft-bullen-ocf-00.txt November 2000 Open Capture Format The RecordIndex section is an optional section intended to give quick and easy access to any record contained in the file. It is basically an array of RecordPosition pointers to Record sections. The RecordIndex section is referenced from the Session section. RecordIndex section tag space: 704-735 (02C0-02DF) RecordIndexPosition.Tag = 704 Attribute Tag Tag Value Count Description Name Dec Hex Type ----- --- --- ----- ----- ----- OCFAttributeCount 1 0001 UInt32 1 Specifies the number of immediately following attributes in this section. Must be the first attribute in the section. RecordPosition 640 0280 Offset 1+ Pointer to a Record section. There is a RecordPosition attribute present for each record in the capture session. 7. OCF Implementation Details and Elaboration 7.1 Cyclic Redundancy Check (CRC) To do: Elaborate on the CRC and find a reference for the standard (IEEE?). The CRC computed in the OCF format is generated by the CCITT CRC-32 algorithm. 7.2 Compression To do: Elaborate on compression, specifically the DEFLATE method (RFC 1950?). The compression scheme used in the OCF format will be block based. The entire file is not compressed as one long stream but, instead, a compressed OCF file consists of a series of compression blocks placed back-to-back in the file. This allows OCF devices to compress and uncompress only portions of the file. This feature is beneficial when an OCF reader wants access to specific data in the file and Bullen Informational - Expiration [Page 28] draft-bullen-ocf-00.txt November 2000 Open Capture Format doesn't have to uncompress the entire file. Also, if an OCF writer finds that an undesirably low compression ratio is achieved on a particular block, it can write the uncompressed data because it is not justifiable to spend time decompressing that particular block when reading and writing the file. A compression block is defined as an OCFCompressionBlockHeader structure (illustrated in a C-style declaration below) followed immediately by a block of data in compressed or uncompressed form. typedef struct { UInt32 CompressedSize; UInt32 UncompressedSize; } OCFCompressionBlockHeader; The OCFCompressionBlock header contains the following two members: CompressedSize - Specifies the length of data immediately following the structure. UncompressedSize - Specifies the length of the original uncompressed data. As mentioned previously, a block does not necessarily have to be compressed and this is exhibited when the CompressedSize member is equal to the UncompressedSize member. Therefore, if an OCF writer is compressing a file and finds that a 1:1 compression ratio is achieved (no compression is yielded) then the writer must write the original data in uncompressed form to the file. 7.3 Attachments To do: Elaborate on MIME and how a file is embedded, see RFCs 2045- 49 (include in references). Attachments can be embedded within an OCF file. An example of this would be embedding a network diagram image that specifies the physical connectivity of the network from which a capture was taken. Attachments will rely on the MIME standard for identifying the type of file and its contents. 8. Security Considerations The OCF format allows easy viewing of trace files. It is recommended that users sending OCF files outside of their organizations be mindful of the possibility of sensitive data within the trace file falling into the wrong hands. Built-in encryption of OCF files was a consideration of the OCF SIG, but was dismissed because it adds complexity to the file format and Bullen Informational - Expiration [Page 29] draft-bullen-ocf-00.txt November 2000 Open Capture Format the desire to use ever more powerful or possibly proprietary encryption mechanisms external to the OCF. If an OCF device chooses to ignore the file CRC, then a corruption in the file will go undetected. The corruption may be harmless or catastrophic, depending on where in the file it occurs and how extensive its changes are. If a single byte in a frame's padding is corrupted, it will be an error that is very hard to detect. If, however, the file is compressed and the corruption causes the decompression routine to misconstrue the file, what may remain is gibberish. 9. Appendicies 9.1 Tag Allocations Decimal Hex Description ----- ----- ----- 0- 0000- Special OCFAttribute records. 127 007F 0 0000 OCFAttributeBlank 1 0001 OCFAttributeCount 2 0002 OCFAttributeDataLength 3 0003 OCFAttributeValueExtension ----- ----- ----- 128- 0080- Reserved for future use. 255 00FF ----- ----- ----- 256- 0100- Header section attributes. 287 011F 256 0100 HeaderPosition 257 0101 HeaderComment 258 0102 HeaderAttachment ----- ----- ----- 320- 0140- Session section attributes. 351 015F 320 0140 SessionPosition 321 0141 SessionComment 322 0142 SessionID 323 0143 SessionStartTime 324 0144 SessionStopTime 325 0145 SessionVendorDescription ----- ----- ----- Bullen Informational - Expiration [Page 30] draft-bullen-ocf-00.txt November 2000 Open Capture Format 384- 0180- Interface section attributes. 415 019F 384 0180 InterfacePosition 385 0181 InterfaceComment 386 0182 InterfaceID 387 0183 InterfaceTopology 388 0184 InterfaceBandwidth 389 0185 InterfaceChannelCount ----- ----- ----- 448- 01C0- ProtocolStack section attributes. 479 01DF 448 01C0 ProtocolStackPosition 449 01C1 ProtocolStackComment 450 01C2 ProtocolStackID ----- ----- ----- 512- 0200- ProtocolStackIndex section attributes. 543 021F 512 0200 ProtocolStackIndexPosition ----- ----- ----- 576- 0240- Address section attributes. 607 025F 576 0240 AddressPosition 577 0241 AddressComment 578 0242 AddressID 579 0243 AddressNextPosition 580 0244 AddressParentID 581 0245 AddressType 582 0246 AddressData 583 0247 AddressName ----- ----- ----- 640- 0280- Record section attributes. 671 029F 640 0280 RecordPosition 641 0281 RecordComment 642 0282 RecordID 643 0283 RecordInterfaceID 644 0284 RecordInterfaceChannel 645 0285 RecordNextPosition 646 0286 RecordEventType 647 0287 RecordData Bullen Informational - Expiration [Page 31] draft-bullen-ocf-00.txt November 2000 Open Capture Format 648 0288 RecordStatusFlags 649 0289 RecordTimeStamp_s ----- ----- ----- 704- 02C0- RecordIndex section attributes 735 02DF ----- ----- ----- 16,384- 0400- Reserved for future use. 32,767 07FF ----- ----- ----- 32,768- 0800- Reserved for private use by individual vendors 65,279 FEFF for implementation-specific features. However, tags in this range cannot be used until they are officially allocated to vendors upon request to and approval by the governing OCF standards body. This tag space will be allocated to vendors in blocks of 16. ----- ----- ----- 65,280- FF00- Reserved for public use by any party that 65,534 FFFE wishes to implement site-specific or experimental features. The OCF standards body will not monitor these tags. Parties should use them at their own risk and only if the OCF files will not propagate out of the site. ----- ----- ----- 65,535 FFFF Reserved for future use. 10. Reference 1 Bradner, S., "The Internet Standards Process -- Revision 3", BCP 9, RFC 2026, October 1996. 2 Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997 3 Crocker, D. and Overell, P.(Editors), "Augmented BNF for Syntax Specifications: ABNF", RFC 2234, Internet Mail Consortium and Demon Internet Ltd., November 1997 4 Deutsch, P. and Gailly, J-L, "ZLIB Compressed Data Format Specification version 3.3", RFC 1950, May 1996 5 Unicode, Inc., "Unicode Home Page", http://www.unicode.org, 2000 Bullen Informational - Expiration [Page 32] draft-bullen-ocf-00.txt November 2000 Open Capture Format 11. Acknowledgments The OCF came about due to the efforts of the following individuals and organizations: Robert Bullen WildPackets, Inc. Eugene Cookmeyer Acterna Corp. Tom Ferguson Agilent Technologies Scott Haugdahl WildPackets, Inc. Kevin Hay Network Associates, Inc. Bill Ives Agilent Technologies Kevin Matsumoto Spirent Communications Roman Oliynyk Network Instruments, LLC David Runyon Acterna Corp. Ben Yoshino Spirent Communications 12. Author's Address Robert Bullen Wild Packets, Inc. 4555 Erin Dr Suite 240 Email: robert@wildpackets.com Bullen Informational - Expiration [Page 33] draft-bullen-ocf-00.txt November 2000 Open Capture Format 13. Full Copyright Statement "Copyright (C) The Internet Society (2001). All Rights Reserved. This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implmentation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English. The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns. This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE." Bullen Informational - Expiration [Page 34]