Biflow implementation support in IPFIX Internet-Draft E. Boschi Document: draft-boschi-ipfix-biflow-00.txt Hitachi Europe Expires: April 2005 B. Trammell CERT/NetSA October 2005 Biflow implementation support in IPFIX draft-boschi-ipfix-biflow-00.txt Status of this Memo By submitting this Internet-Draft, each author represents that any applicable patent or other IPR claims of which he or she is aware have been or will be disclosed, and any of which he or she becomes aware will be disclosed, in accordance with Section 6 of BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. This Internet-Draft will expire on April 20, 2005. Copyright Notice Copyright (C) The Internet Society (2005). Boschi, Trammell Expires April 2006 [Page 1] Biflow implementation support in IPFIX Abstract This document describes how bidirectional flows (biflows) can be implemented in the IP Flow Information Export (IPFIX) protocol. We propose an extension to the Information Model that allows specifying biflow semantic and a set of counters needed for biflow processing. Conventions used in this document The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119. Table of Contents 1. Introduction············································2 2. Terminology·············································3 3. Biflow Implementation Strategies························3 3.1 Biflow Semantics········································3 3.2 Flow Association using flowId IE························4 3.2.1 Example................................................4 3.3 Flow Aggregation using flowId IE························4 3.3.1 Example................................................5 3.4 Multiple Counters using flowID scope, and direction IE··5 3.5 Single Record biflows using directionDomain IE··········6 3.5.1 reverseOctetDeltaCount.................................6 3.5.2 reversePostOctetDeltaCount.............................6 3.5.3 reverseOctetDeltaSumOfSquares..........................6 3.5.4 reverseOctetTotalCount.................................6 3.5.5 reversePostOctetTotalCount.............................7 3.5.6 reverseOctetTotalSumOfSquares..........................7 3.5.7 reversePacketDeltaCount................................7 3.5.8 reversePostPacketDeltaCount............................7 3.5.9 reversePacketTotalCount................................8 3.5.10 reversePostPacketTotalCount...........................8 3.5.11 reverseDroppedOctetDeltaCount.........................8 3.5.12 reverseDroppedPacketDeltaCount........................8 3.5.13 reverseDroppedOctetTotalCount.........................8 3.5.14 reverseDroppedPacketTotalCount........................9 3.5.15 directionDomain.......................................9 4. IANA Considerations·····································9 5. Security Considerations································10 6. References·············································10 6.1 Normative References···································10 6.2 Informative References·································10 7. Author's Addresses·····································10 8. Intellectual Property Statement························11 9. Copyright Statement····································11 10. Disclaimer·············································11 1. Introduction The export of some metrics and network behaviors that relate to network flows could benefit from a bi-directional flow data model (e.g. RTT, TCP packet retransmission, existence of routing failures in an OSPF routed network). Also some existing commercial network flow data would be better supported with a bi-directional model. We propose a method to export bidirectional flow (biflow) information using IPFIX. The method requires an extension to the Boschi, Trammell Expires April 2006 [Page 2] Biflow implementation support in IPFIX IPFIX Information Model to include some biflow-related Information Elements (IEs). 2. Terminology Uniflow (unidirectional flow) A uniflow is a set of IP packets passing an Observation Point in the network during a certain time interval. All packets belonging to a particular Uniflow have a set of common properties. Each property is defined as the result of applying a function to the values of: 1. One or more packet header fields, transport header fields, or application header fields. 2. One or more characteristics of the packet itself. 3. One or more fields derived from packet treatment. A packet is said to belong to a Uniflow if it completely satisfies all the defined properties of the Uniflow. Biflow (bidirectional flow) A biflow is the product of matching the two uniflow sides of a bidirectional communication session (e.g., TCP session, UDP DNS question and answer) into a single entity. The semantics of "source" and "destination" information elements within the context of a given biflow are variable. 3. Biflow Implementation Strategies This section introduces the problems associated with representing biflows in IPFIX, and presents a variety of strategies for implementing biflow support. 3.1 Biflow Semantics When handling uniflows, the semantics of "source" and "destination" information elements are clearly defined by the semantics of the underlying packet header data. When grouping biflows into single IPFIX data records, the definitions of "source" and "destination" become less clear. The most basic method for classifying the two addresses in a biflow is to define the source address of the flow as the source address of the first packet seen in that flow by the metering process. Some metering technologies may attempt to improve upon this method using some knowledge of the transport or application protocols (e.g., TCP flags, DNS question/answer counts) in order to define the source address of the flow as the connection or transaction initiator. In either case, the design is the same: one of the underlying uniflows is assumed to be in the "forward" direction, and one in the "reverse" direction; the "forward" uniflow is selected based upon some characteristic of the connection itself. Another way to classify biflow addresses is by perimeter; in Boschi, Trammell Expires April 2006 [Page 3] Biflow implementation support in IPFIX this method, a metering process discriminates between "inside" and "outside" the network, and defines the source address as the address on one side of this perimeter (generally the "outside" address; defining source loosely as "attacker"). Perimeter biflows may require additional information elements to identify the flow initiator, if such functionality is supported by the metering process. This revision of this draft does not address perimeter biflows further. These two are by no means an exhaustive list of biflow semantics. However, IPFIX should aim to provide better support for the simpler, more common semantics in preference to more exotic schemes. 3.2 Flow Association using flowId IE The simplest way to implement biflow support using IPFIX is to sidestep the single-record unification entirely, and annotate two uniflow records as being part of the same biflow by using the existing flowId information element. If an exporting process supports biflow export via this method, it MUST ensure that the two flow data records comprising the biflow appear sequentially, within the same data set, in order of start time. This ensures that the collecting process does not need to cache more than a single flow to support biflow reassembly on its end. This method has the advantage of simplicity and minimal impact on the protocol. However, as two uniflows in a single biflow share all flow key data, it is not the most efficient method for transporting biflow data. 3.2.1 Example The format of the two template records exporting uniflow information has the following structure: +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Set ID = 2 | Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Template ID = 256 | Field Count | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |0| flow key fields | Field Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |0| FlowId = 148 | Field Length = 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |0| IE to be exported | Field Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ The fields have been simply distinguished in FlowID (that identifies the two uniflows as part of the same biflow), flow key fields and IE to be exported 3.3 Flow Aggregation using flowId IE The main shortcoming of the previous method is the replication of information exported, since two uniflows in a biflow share all flow key data. This second method aggregates the flow fields common to both uniflows sending them in an Option record with FlowID as scope. The FlowID IE represents the identifier of the biflow. All subsequent data of the biflow is sent in uniflow data records using standard counters IEs plus the flowID IE plus one IE indicating the direction of the flow (but not the common flow key fields). The direction of the flow can be specified using source address or the directionDomain IE (see section 3.5.15). Boschi, Trammell Expires April 2006 [Page 4] Biflow implementation support in IPFIX Using the source address to indicate the direction doesn't require any extension to the Information Model, but exporting an IP address (especially an IPv6 address) is more costly than the one octet IE directionDomain. If an exporting process supports biflow export via this method, it MUST ensure that the option record is sent first. The exporter sends the flow key fields just once and subsequently refers to them using the flowID. This method makes sense when the records are exported at regular intervals but doesn't introduce an improvement if the data is exported just at the end of the flow. This solution has the advantage that we need just one new IE. The method to send the flow data as an option has been also used in [BoMa05]. 3.3.1 Example Let's suppose the biflow we want to export is an IPv4 flow, it is defined by the two fields flow key 1 and flow key 2 and we use the IPv4 source address to distinguish the uniflows. The option record will look like: +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Set ID = 3 | Length = x octets | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Template ID = 256 | Field Count = 1 + 2 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Scope Field Count = 1 |0| FlowID = 148 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | scope Field Length = 4 |0| flow key 1 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Field Length |0| flow key 2 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Field Length | Padding (optional) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Each of the uniflow data records will look like: +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Set ID = 2 | Length = h octets | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Template ID > 256 | Field Count = k | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |0| sourceIPv4Address = 8 | Field Length = 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |0| IE to be exported | Field Length = n | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 3.4 Multiple Counters using flowID scope, and direction IE [TODO] Boschi, Trammell Expires April 2006 [Page 5] Biflow implementation support in IPFIX 3.5 Single Record biflows using directionDomain IE Biflows can be represented in IPFIX using a single data record per flow by extending the IPFIX information model. First, a set of "reverse" counters are defined to count packets sent by the biflow destination, and the current "forward" counters are defined to count packets send by the biflow source. Second, an additional information element, directionDomain, is defined to specify the semantics of biflow source and destination. Note that no reverse post-multicast counters are defined, because multicast flows are inherently unidirectional. 3.5.1 reverseOctetDeltaCount Description: The number of octets since the previous report (if any) in packets sent by the biflow destination for this biflow. The number of octets includes IP header(s)and IP payload. Abstract Data Type: unsigned64 Data Type Semantics: deltaCounter ElementId: 216 Status: proposed Units: octets 3.5.2 reversePostOctetDeltaCount Description: The definition of this Information Element is identical to the definition of Information Element 'reverseOctetDeltaCount', except that it reports a potentially modified value caused by a middlebox function after the packet passed the observation point. Abstract Data Type: unsigned64 Data Type Semantics: deltaCounter ElementId: 217 Status: proposed Units: octets 3.5.3 reverseOctetDeltaSumOfSquares Description: The sum of the squared numbers of octets since the previous report (if any) in packets sent by the biflow destination for this biflow. The number of octets include IP header(s) and IP payload. Abstract Data Type: unsigned64 ElementId: 218 Status: proposed Units: octets 3.5.4 reverseOctetTotalCount Description: The total number of octets in packets sent by the biflow destination since the Metering Process (re-)initialization for this Observation Point. The number of octets includes IP header(s) and IP payload. Abstract Data Type: unsigned64 Boschi, Trammell Expires April 2006 [Page 6] Biflow implementation support in IPFIX Data Type Semantics: totalCounter ElementId: 219 Status: proposed Units: octets 3.5.5 reversePostOctetTotalCount Description: The definition of this Information Element is identical to the definition of Information Element 'reverseOctetTotalCount', except that it reports a potentially modified value caused by a middlebox function after the packet passed the observation point. Abstract Data Type: unsigned64 Data Type Semantics: totalCounter ElementId: 220 Status: proposed Units: octets 3.5.6 reverseOctetTotalSumOfSquares Description: The total sum of the squared numbers of octets in packets sent by the biflow destination for this biflow since the Metering Process (re-)initialization for this Observation Point. The number of octets include IP header(s) and IP payload. Abstract Data Type: unsigned64 ElementId: 221 Status: proposed Units: octets 3.5.7 reversePacketDeltaCount Description: The number of packets since the previous report (if any) for this biflow sent by the biflow destination. Abstract Data Type: unsigned64 Data Type Semantics: deltaCounter ElementId: 222 Status: proposed Units: packets 3.5.8 reversePostPacketDeltaCount Description: The definition of this Information Element is identical to the definition of Information Element 'reversePacketDeltaCount', except that it reports a potentially modified value caused by a middlebox function after the packet passed the observation point. Abstract Data Type: unsigned64 Data Type Semantics: deltaCounter ElementId: 223 Status: proposed Units: packets Boschi, Trammell Expires April 2006 [Page 7] Biflow implementation support in IPFIX 3.5.9 reversePacketTotalCount Description: The total number of packets sent by the biflow destination for this biflow at the since the Metering Process (re-) initialization for this Observation Point. Abstract Data Type: unsigned64 Data Type Semantics: totalCounter ElementId: 224 Status: proposed Units: packets 3.5.10 reversePostPacketTotalCount Description: The definition of this Information Element is identical to the definition of Information Element 'reversePacketTotalCount', except that it reports a potentially modified value caused by a middlebox function after the packet passed the observation point. Abstract Data Type: unsigned64 Data Type Semantics: totalCounter ElementId: 225 Status: proposed Units: packets 3.5.11 reverseDroppedOctetDeltaCount Description: The number of octets since the previous report (if any) in packets sent by the biflow destination for this biflow dropped by packet treatment. The number of octets include IP header(s) and IP payload. Abstract Data Type: unsigned64 Data Type Semantics: deltaCounter ElementId: 226 Status: proposed Units: octets 3.5.12 reverseDroppedPacketDeltaCount Description: The number of packets since the previous report (if any) sent by the biflow destination for this biflow dropped by packet treatment. Abstract Data Type: unsigned64 Data Type Semantics: deltaCounter ElementId: 227 Status: proposed Units: packets 3.5.13 reverseDroppedOctetTotalCount Description: The total number of octets in packets sent by the biflow destination for this biflow dropped by packet treatment since the Metering Process (re-)initialization for this Observation Point. The number of octets include IP header(s) and IP payload. Abstract Data Type: unsigned64 Boschi, Trammell Expires April 2006 [Page 8] Biflow implementation support in IPFIX Data Type Semantics: totalCounter ElementId: 228 Status: proposed Units: octets 3.5.14 reverseDroppedPacketTotalCount Description: The total number of packets sent by the biflow destination for this biflow dropped by packet treatment since the Metering Process (re-)initialization for this Observation Point. Abstract Data Type: unsigned64 Data Type Semantics: totalCounter ElementId: 229 Status: proposed Units: packets 3.5.15 directionDomain Description: Defines the semantics of source and destination information elements, and of reverse counters. This information element is intended to be bound to a sourceID or a templateID in an options data record. Defined values include: 0x00: Uniflow. Source and destination are defined as in the packet headers from which the flows were generated. Reverse counters have no defined meaning. This is the present default IPFIX behavior. 0x01: First Packet Biflow. The source of the biflow is defined as the source of the first packet seen within the flow by the Metering Process, and the destination of the biflow is defined as the destination of that first packet. Counters count packets sent by the first packet source, and reverse counters count packets sent by the first packet destination. 0x02: Initiator Biflow. The source of the biflow is defined as the source of the packet initiating the connection as determined by the Metering Process, and the destination of the biflow is defined as the destination of the packet initiating the connection. This is similar to First Packet Biflow, except it gives the Metering Process some latitude to attempt to determine the connection initiator when the initiator is not necessarily the sender of the first packet seen by the Metering Process. 0x03 - 0x7F: Reserved for future assignment 0x80 - 0xFF: Reserved for private or experimental use. Abstract Data Type: octet Data Type Semantics: identifier ElementId: 215 Status: proposed 4. IANA Considerations Boschi, Trammell Expires April 2006 [Page 9] Biflow implementation support in IPFIX This documents defines a set of new IPFIX Information Elements that extend those already defined in [IPFIX-INFO]. As specified in [IPFIX-INFO] IANA needs to create a new registry for IPFIX Information Element identifiers. New assignments for IPFIX Information Elements will be administered by IANA, on a First Come First Served basis [RFC2434], subject to Expert Review [RFC2434], i.e. review by one of a group of expert designated by an IETF Operations and Management Area Director. The group of experts must double check the Information Elements definitions with already defined Information Elements for completeness, accuracy, redundancy, and correct naming following the naming conventions in [IPFIX-INFO]. Those experts will initially be drawn from the Working Group Chairs and document editors of the IPFIX and PSAMP Working Groups [IPFIX-INFO]. 5. Security Considerations The same security considerations as for the IPFIX protocol [IPFIX-PROTO] apply. 6. References 6.1 Normative References [IPFIX-PROTO] B. Claise et Al.: IPFIX Protocol Specification, Internet-draft work in progress , September 2005 [IPFIX-INFO] J. Quittek, S.Bryant, B.Claise, J. Meyer: Information Model for IP Flow Information Export Internet-draft work in progress , September 2005 6.2 Informative References [IPFIX-REQ] J. Quittek, et Al.: Requirements for IP Flow Information Export, RFC 3917, October 2004 [IPFIX-AS] T. Zseby, E. Boschi, N. Brownlee, B. Claise: IPFIX Applicability, Internet Draft , June 2005 [RFC2434] Narten, T. and H. Alvestrand: Guidelines for Writing an IANA Considerations Section in RFCs, BCP 26, RFC 2434, October 1998. [BoMa05] E. Boschi, L. Mark: Use of IPFIX for Export of Per- Packet Information, Internet-draft work in progress , June 2005 7. Author's Addresses Elisa Boschi Hitachi Europe SAS Immeuble Le Theleme 1503 Route des Dolines 06560 Valbonne, France Phone: +33 4 89874180 Email: elisa.boschi@hitachi-eu.com Boschi, Trammell Expires April 2006 [Page 10] Biflow implementation support in IPFIX Brian H. Trammell CERT Network Situational Awareness Software Engineering Institute 4500 Fifth Avenue Pittsburgh, PA 15213 US Phone: +1 412 268 9748 Email: bht@cert.org 8. Intellectual Property Statement The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79. Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr. The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf-ipr@ietf.org. 9. Copyright Statement Copyright (C) The Internet Society (2005). This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights. 10. Disclaimer This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Boschi, Trammell Expires April 2006 [Page 11]