Internet Engineering Task Force L. Peluso Internet-Draft University of Napoli Intended status: Standards Track T. Zseby Expires: July 11, 2011 Fraunhofer Institute FOKUS S. D'Antonio CINI Consortium/University of Napoli "Parthenope" C. Henke Fraunhofer Institute FOKUS January 07, 2011 Flow Selection Techniques draft-ietf-ipfix-flow-selection-tech-04.txt Abstract Flow selection is the process of selecting a subset of flows from all flows observed at an observation point. The objective of flow selection is to reduce the effort of post-processing flow data and transferring flow records. The flow selection process can be enabled at different stages of the measurement process. It can be applied either directly after classification or at recording/exporting time by limiting the number of flows to be stored and/or exported to the collecting process. This document describes motivations for flow selection and presents flow selection techniques. It furthermore provides an information model for configuring flow selection techniques and discusses what information about a flow selection process is worth exporting through a suitable information model. Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119]. Status of this Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at http://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any Peluso, et al. Expires July 11, 2011 [Page 1] Internet-Draft Flow Selection Techniques January 2011 time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on July 11, 2011. Copyright Notice Copyright (c) 2011 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Peluso, et al. Expires July 11, 2011 [Page 2] Internet-Draft Flow Selection Techniques January 2011 Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 5 2. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 3. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 6 4. Flow selection as a function of the IPFIX Exporter . . . . . . 7 4.1. Flow selection in the metering process . . . . . . . . . . 9 4.2. Flow selection in the flow recording process . . . . . . . 9 4.3. Flow selection during the exporting process . . . . . . . 11 5. Flow selection as a function of the IPFIX Mediator . . . . . . 12 6. Flow selection techniques . . . . . . . . . . . . . . . . . . 14 6.1. Flow selection based on flow record content . . . . . . . 14 6.2. Flow selection based on flow record arrival time or sequence . . . . . . . . . . . . . . . . . . . . . . . . . 14 6.3. Flow selection on external events . . . . . . . . . . . . 14 7. Information model for flow selection information exporting . . 14 7.1. Meter process related (TBD1-TBD3) . . . . . . . . . . . . 16 7.1.1. fsMeterUnmeasPacketCountTsFirst . . . . . . . . . . . 17 7.1.2. fsMeterUnmeasPacketCountTsLast . . . . . . . . . . . . 17 7.1.3. fsMeterUnmeasBytesCount . . . . . . . . . . . . . . . 18 7.2. Flow recording process related (TBD4-TBD11) . . . . . . . 18 7.2.1. fsFrecPacketInDroppedRecsCountTsFirst . . . . . . . . 18 7.2.2. fsFrecPacketInDroppedRecsCountTsLast . . . . . . . . . 19 7.2.3. fsFrecByteInDroppedRecsCount . . . . . . . . . . . . . 19 7.2.4. fsFrecFrecDroppedCountTsFirst . . . . . . . . . . . . 20 7.2.5. fsFrecFrecDroppedCountTsLast . . . . . . . . . . . . . 20 7.2.6. fsFrecUnexportedFrecCount . . . . . . . . . . . . . . 20 7.2.7. fsFrecUnexportedPacketInFrecCount . . . . . . . . . . 21 7.2.8. fsFrecUnexportedBytesInFrecCount . . . . . . . . . . . 21 7.3. Flow exporting process related (TBD12-TBD19) . . . . . . . 21 7.3.1. fsExpPacketInDroppedRecsCountTsFirst . . . . . . . . . 22 7.3.2. fsExpPacketInDroppedRecsCountTsLast . . . . . . . . . 22 7.3.3. fsExpByteInDroppedRecsCount . . . . . . . . . . . . . 23 7.3.4. fsExpFrecDroppedCountTsFirst . . . . . . . . . . . . . 23 7.3.5. fsExpFrecDroppedCountTsLast . . . . . . . . . . . . . 24 7.3.6. fsExpUnexportedCount . . . . . . . . . . . . . . . . . 24 7.3.7. fsExpUnexportedPacketCount . . . . . . . . . . . . . . 24 7.3.8. fsExpUnexportedByteInExpCount . . . . . . . . . . . . 25 8. Implementation requirements . . . . . . . . . . . . . . . . . 25 9. Information Model for Configuration of Flow Selection Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . 25 9.1. selectorMethod . . . . . . . . . . . . . . . . . . . . . . 26 9.2. flowMaxAdmitFlowRecords . . . . . . . . . . . . . . . . . 27 9.3. flowRecordBytesSize . . . . . . . . . . . . . . . . . . . 27 9.4. flowRecordPacketsSize . . . . . . . . . . . . . . . . . . 28 9.5. flowInactivityTime . . . . . . . . . . . . . . . . . . . . 28 9.6. flowIPAddressVersion . . . . . . . . . . . . . . . . . . . 29 9.7. flowIPv4SourceAddress . . . . . . . . . . . . . . . . . . 29 Peluso, et al. Expires July 11, 2011 [Page 3] Internet-Draft Flow Selection Techniques January 2011 9.8. flowIPv4DestinationAddress . . . . . . . . . . . . . . . . 30 9.9. flowIPv6SourceAddress . . . . . . . . . . . . . . . . . . 30 9.10. flowIPv6DestinationAddress . . . . . . . . . . . . . . . . 30 10. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 31 11. Security Considerations . . . . . . . . . . . . . . . . . . . 31 12. References . . . . . . . . . . . . . . . . . . . . . . . . . . 31 12.1. Normative References . . . . . . . . . . . . . . . . . . . 31 12.2. Informative References . . . . . . . . . . . . . . . . . . 32 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 33 Peluso, et al. Expires July 11, 2011 [Page 4] Internet-Draft Flow Selection Techniques January 2011 1. Introduction This document describes flow selection techniques for traffic measurements. As stated in [RFC5475], packet selection is the process of selecting a subset of packets among those collected at an observation point. The element on which this selection mechanism is performed is a single packet and the selection decision is random, deterministic or based on packet properties. In contrast to this, flow selection techniques consider flows as the basic elements on which a selection process is performed. A flow is defined as a set of packets with common properties [IPFIX-REQ]. Flow selection reduces the resource demands for capturing, storing, exporting and post-processing flow-based measurement results. Maintaining and exporting all flow records to the collecting process can exhaust available resources with the result that measurement data is discarded which reduces the utility of the measurement data and makes accurate estimations impossible. Flow Selection can be applied to only select flows that are of interest for a certain application or to select a representative subset of the flows. Selecting representative subsets of flows allows for flow characteristic estimation while reducing the measurement data. Example applications in which flow selections can be applied are accounting, attack and intrusion detection, traffic engineering, traffic classification, etc.. In many networks few large flows contribute to the majority of the overall traffic volume [DuLT01a], [DuLT01b], which is also referred to as "Quasi-Zipf-Law" [KuXW04]. This "elephant and mice" phenomenon plays an important role in flow based measurements as it poses challenges on the flow selection process depending on the application and the available resources. For example in accounting purposes it is useful to concentrate on the so-called "heavy hitter" flows to cope with a limited flow cache size or limited transmission capacity in times when resources are scarce. 2. Scope This document describes flow selection techniques and their parameters. It addresses the configuration of flow selection techniques and defines which information should be reported by devices that perform flow selection. It only describes processes directly acting on traffic flows during the metering phase and/or the exporting phase. Therefore it is assumed that flow selection is performed after packets are classified into flows. This document does not address the flow selection effects that might arise from sampling or filtering packets in the metering process before the classification process is performed. Such packet selection techniques are described in [RFC5475] and, therefore, are out of the scope of this document. Peluso, et al. Expires July 11, 2011 [Page 5] Internet-Draft Flow Selection Techniques January 2011 3. Terminology This document is consistent with the terminology introduced in [RFC5470], [RFC5475] and [IPFIX-REQ]. Further some additional terms are presented which extend the terminology. * Flow A Flow is defined as a set of packets with common properties passing an Observation Point in the network during a certain time interval. * Flow Record A Flow Record contains information about a specific flow that was metered at an observation point. A Flow Record contains measured properties of the flow (e.g., the total number of bytes of all packets of the flow) and usually characteristic properties of the flow (e.g., source IP address). * Classification Is a filtering process in which packets are mapped to specific flow records based on the packet properties. These properties make up the flow key (e.g. header information, packet content, AS number). In case a flow record for this specific flow key already exist the flow record is updated, otherwise a new flow record is created. * Flow Selection Process A Flow Selection Process takes a set of Flow Records as its input and selects a subset of that set as its output. * Flow Selection State A Flow Selection Process may maintain state information for use by the Flow Selection Process. At a given time, the Flow Selection State may depend on flows observed at and before that time, as well as other variables. Examples include: (i) number of accounted flow records; (ii) memory space available for flow recording; Peluso, et al. Expires July 11, 2011 [Page 6] Internet-Draft Flow Selection Techniques January 2011 (iii) state of the pseudorandom number generators; (iv) hash values calculated during the selection. (v) timestamps of flow observation (e.g. first or last packet of flow) and flow duration * Flow Selector A Flow Selector defines the action of a Flow Selection Process on a single flow of its input. The Flow Selector can make use of the following information in order to establish whether or not a flow has to be selected or not: (i) the content of the flow record; (ii) any information state related to the flow recording process; (iii) any flow selection state that may be maintained by the Flow Selection Process. * IPFIX Exporter The IPFIX exporter is a software that receives measurement data from e.g. the Metering Process and exports this data to a collector. * IPFIX Mediator The IPFIX Mediator receives flow records from IPFIX Exporters and, after some post-processing (e.g. more fine-grained selection), sends them to one or multiple Collectors. This selection can be driven based on the information requested by the individual Collector. 4. Flow selection as a function of the IPFIX Exporter Figure 1 shows the IPFIX reference model as defined in [RFC5470], and extends it by introducing the functional components where flow selection can take place. Peluso, et al. Expires July 11, 2011 [Page 7] Internet-Draft Flow Selection Techniques January 2011 Packet(s) coming in to Observation Point(s) | | v v +----------------+---------------------------+ +-----+-------+ | Metering Process on an | | | | Observation Point | | | | | | | | packet header capturing | | | | | |...| Metering | | timestamping | | Process N | | | | | | | packet selection | | | | | | | | | classification | | | | | | | | | flow state dependent packet sampling (*) | | | | | | | | | aggregation | | | | | | | | | flow recording (*) | | | | | | | | | | Timing out Flows | | | | | Handle resource overloads | | | +--------|-----------------------------------+ +-----|-------+ | | Flow Records (selected by Observation Domain) Flow Records | | +----------------------+----------------------+ | +----------------------|---------------+ | Exporting Process | | | +---------------+-----------+ | | | flow exporting (*) | | | +---------------+-----------+ | | | | +----------------------+---------------+ | v Export packet to Collector (*) indicates where flow selection can take place. Figure 1: Flow selection as a function of the IPFIX Exporter In contrast to packet selection, flow selection is always applied after the packets are classified into flows. Flows can be selected at different stages of the measurement chain: Peluso, et al. Expires July 11, 2011 [Page 8] Internet-Draft Flow Selection Techniques January 2011 1. during metering [RFC5475]; 2. during flow recording; 3. during flow exporting. 4.1. Flow selection in the metering process The main reason for applying flow state dependent sampling during the metering process is that the flow recording process may not have, at a certain point in time, enough memory positions to record all observable flows. Another reason may be that there might not be enough processing resources to create and manage a new flow record. To overcome these limitations, a number of possible policies can be applied, the simplest one being to discard new packets which cannot be assigned to existing flow records (i.e. that would require the creation of a new flow record). More complex policies are applicable, mainly aimed at detecting the so called elephant flows, so as to prioritize flows carrying a higher traffic volume in the flow recording process. For instance, [EsVa01] proposes criteria to define a packet as eligible to create a new flow record (sample and hold, multistage filters). Regardless of specific algorithms, we are concerned about identifying what information on the flow state dependent packet sampling is worth keeping and making available to applications (by exporting it out of an IPFIX device). An option could be to keep a cumulative counter of the total number of packets and bytes that were not taken into account for measurement purposes because of flow state dependent sampling. Furthermore, it is possible to keep a timestamp for the first and last of these discarded packets. In practice, this implies aggregating all these packets in a single macro flow, and keeping track of its volume and duration. Storing more detailed information about packets which have not been measured because of flow state dependent sampling would contradict the fact that the sampling is done because of lack of memory and/or processing resources. 4.2. Flow selection in the flow recording process As described in the previous section, because of lack of memory positions in the flow recording process some incoming packets might be discarded if they lead to the creation of a new flow record. However, under certain circumstances, it may be advantageous to discard an existing flow record during the flow recording process in order to make room for a new one which has been created upon arrival of a new packet. For example, an algorithm to make the decision whether to discard a new incoming packet or an existing flow record Peluso, et al. Expires July 11, 2011 [Page 9] Internet-Draft Flow Selection Techniques January 2011 is described in [Moli03]. In this section we focus on the selection of the information to be stored regarding the record removal rather than on the details of the decision making algorithm. For the reasons we mentioned above, it does not make sense to store separate information for each discarded flow record, as it would contradict the motivation why discarding is done (i.e. lack of memory resources). The information that can be kept with a limited overhead is the cumulative counter of the total number of not yet exported packets and bytes belonging to flow records that were removed during the flow recording process. Ideally, we would like to keep also a timestamp for the first (T_fd) and last (T_ld) not yet exported packets belonging to every discarded flow record. This would mean aggregating all these packets in a macro flow, and keeping track of its volume and duration. To do so, we would need to maintain a timestamp of the first and last non- exported packets in each flow record. The values of such timestamps would be checked whenever a record is discarded in order to verify whether they are smaller or larger than T_fd and T_ld, respectively. If so, the timestamps would be updated. Another information that can be easily maintained is the number of discarding actions, along with the timestamps of the first and last action. This information SHOULD not be used by applications to re- normalize the received per flow statistics (because a flow may be discarded and created multiple times), but rather to monitor and control the performance of the implemented policy. Note that we take into account a discarding event only when the discarded flow record contains data about traffic which has not been exported. Differently, the removal of a record whose traffic was exported (either after a timeout or after the arrival of specific packets, e.g. TCP FIN or RST) is part of the normal operation of an IPFIX flow metering system. Note also that we consider only the case when the elimination of a flow record during the flow recording process leads to the complete loss of all the information contained in the flow record itself. The implementation of a different policy, such as immediate exporting of the flow record before elimination, or freezing of the flow record and moving it to another area of memory for later exporting, is not considered as an elimination and therefore it is out of the scope of this document. Along with the information about the number of discarded flow records and associated packets and bytes, it is useful to keep cumulative information about the number of flow records containing not yet exported traffic and being currently handled by the flow recording Peluso, et al. Expires July 11, 2011 [Page 10] Internet-Draft Flow Selection Techniques January 2011 process. Another information worth keeping is the cumulative number of not exported packets and bytes contained in them. 4.3. Flow selection during the exporting process The exporting process may implement policies for exporting only a subset of the flow records which have been stored in the system memory. The decision to export only a subset of the flow records can be motivated by the existence of an explicit policy which filters out the flow records to be exported. An example of such a policy could be to export only the flow records associated to flows whose accounted traffic is below a certain threshold, or to implement a more complex mechanism such as the one described in [DuLT01a] or [DuLT01b]. Another motivation which might bring to the exporting of a subset of stored flow records is resource limitation. For example, the exporting process has been assigned a limited time slot to operate or it exports only a predefined number of packets. Hybrid cases can happen where the exporting of a subset of the flow records is motivated by the co-existence of resource limitations and ad-hoc policies which are applied in order to optimize the exporting process. For example, given that the exporting process applies to a subset of the flow records, such subset is selected so that the overall number of exported packets and bytes belonging to the subset is maximized. Selecting flow records during the exporting process raises the issue of identifying the information which is worth keeping about the flow selection process. Two different scenarios cab be envisaged. If a flow record is not exported and then it does not feed the flow recording process, the scenario is the same as when the deletion of the flow record is caused by the need to make room to another record. The metrics to be kept are cumulative packets and bytes associated with non-exported flow records, timestamps of the first and last packets belonging to non-exported flow records, counter of dropping events and timestamps of first and last dropping event. If a record eligible for exporting is not exported and it enters the flow recording process it has a chance of being exported in the future. It would be beneficial for an application to get information, in terms of number of generated packets and bytes, about the flow records which are not being exported due to the existence of exporting policies and/or resource limitations. This is intended to make it possible to detect potential pathologic conditions, like the missed exporting of a large number of flow records and/or associated traffic, or the growing number of flow records being involved in the flow recording process. Peluso, et al. Expires July 11, 2011 [Page 11] Internet-Draft Flow Selection Techniques January 2011 The selection of the flow records to be exported implies performing a complete scanning of the memory area where flow information is stored, thus jeopardizing the efficiency of the overall exporting process. For this reason, flow exporting protocol specification does not include flow selection during the recording process as a mandatory function even if the information model has been designed to enable such function. 5. Flow selection as a function of the IPFIX Mediator As shown in Figure 2, flow selection can be performed as an intermediate process within an IPFIX Mediator. This process selects the flow records from a sequence which meets pre-defined criteria and exports them to an IPFIX Collector. This selection function can be seen as a more fine-grained process with respect to the selection performed by an IPFIX Exporter. The criteria used to drive the selection process at Mediator's level might be applied to the set of flow records coming from the IPFIX Exporter, thus triggering a further flow selection process. Peluso, et al. Expires July 11, 2011 [Page 12] Internet-Draft Flow Selection Techniques January 2011 Packet(s) coming in to Observation Point(s) | IPFIX Original | Exporter v +------------------+-------------------+ | | | Metering Process on an | | Observation Point | | | | | Flow metering and selection | | | | | Flow recording and selection | | | | | Flow exporting and selection | | | | +------------------+-------------------+ | v Flow Records (selected by Observation Domain) | IPFIX Mediator v +------------------+-------------------+ | | | Collecting process | | | | | Flow selection (*) | | | | | Exporting process | | | | +------------------+-------------------+ | v Flow Records Figure 2: Flow selection as a function of the IPFIX Mediator As an example, if an IPFIX Mediator interacts with a set of IPFIX Collectors, flow records arriving at the IPFIX Mediator might be selected based on the IPFIX Collector requesting flow information. As described in previous sections, flow selection can take place during metering, recording, and exporting processes of an IPFIX exporter depending on the policies which are implemented to meet application requirements. In case flow selection is performed at Mediator's level, we envisage the use of flow selection techniques as a step of the exporting process aimed to identify the flow records to be exported among those stored in the system's memory. This is because the lighter is the intermediate selection process, the better Peluso, et al. Expires July 11, 2011 [Page 13] Internet-Draft Flow Selection Techniques January 2011 is the performance of the mediation framework. 6. Flow selection techniques We can distinguish the following selection techniques: 1. based on flow record content (i.e. all reported flow characteristics); 2. based on flow record arrival time; 3. based on external events like the exhaustion of local resources. 6.1. Flow selection based on flow record content Flow selection can be done based on fields in an IPFIX flow record. This can be done similarly to field match filtering for packet selection described in [RFC5475]. The difference is that instead of packet fields flow record fields are here used to drive the selection decision. An example would be to select flow records based on parameters like the flow size in bytes or the number of packets. Another application would be to select flow records based on either flow start time or flow keys (IP addresses, ports) of the stored flow record. 6.2. Flow selection based on flow record arrival time or sequence Flow records can be selected based on their arrival time at the exporting process. An example would be to select a number of flow records during certain time intervals. Another option is to select flow records based on the order they arrive at the exporting process. In such case, one can either periodically select the k-th records or select randomly a set of flow records. 6.3. Flow selection on external events The selection of flow records can be also triggered by external events. An example would be to activate flow selection based on the value of a router state parameter like the number of entries in the flow cache. 7. Information model for flow selection information exporting In this section we define the elements devoted to containing the information described in the previous sections. Some elements have been associated with a pair of timestamps, which are referred to as Peluso, et al. Expires July 11, 2011 [Page 14] Internet-Draft Flow Selection Techniques January 2011 element_nameTsFirst and element_nameTsLast. We would like to point out that only packet or flow related counts have associated timestamps, while bytes related counts do not. The reason is that element_nameTsFirst and element_nameTsLast are referred to the timestamp of the first and last received packets belonging to not exported flows, while timestamp is meanliness with regard to bytes information. Note that all the following information elements are aimed at describing macro flows parameters (e.g. the total number of packets and bytes contained in all dropped or not created flow records). Some of these macro flows parameters are additive, in the sense that it is only possible to add contributions to them, but never subtract some amount. For example, the macro flow of the packets contained in flow records that are discarded from the flow reporting process receives an addition when a flow record is discarded, and this addition can never be subtracted. On the other side, some macro flow parameters can dynamically receive and loose additions. For example, the macro flows of packets not yet exported receives an addition when a new packet arrives, and looses addition as an exporting event takes place. Associating a timestamp to the oldest and most recent additions in case of additive flows is an easy task, while it is complicated for the others (it would require to maintain full state information) and that is why we did not define timestamps for these information elements. The information elements herein introduced are defined in accordance with the IPFIX information model [RFC5102]. Furthermore, the data types used to represent the Flow Selection-related information elements are those defined in section 3.1 of the IPFIX information model [RFC5102]. List of additional Flow Selection information elements: Peluso, et al. Expires July 11, 2011 [Page 15] Internet-Draft Flow Selection Techniques January 2011 +-------+---------------------------------------+ | ID | Name | +-------+---------------------------------------+ | TBD1 | fsMeterUnmeasPacketCountTsFirst | +-------+---------------------------------------+ | TBD2 | fsMeterUnmeasPacketCountTsLast | +-------+---------------------------------------+ | TBD3 | fsMeterUnmeasBytesCount | +-------+---------------------------------------+ | TBD4 | fsFrecPacketInDroppedRecsCountTsFirst | +-------+---------------------------------------+ | TBD5 | fsFrecPacketInDroppedRecsCountTsFirst | +-------+---------------------------------------+ | TBD6 | fsFrecByteInDroppedRecsCount | +-------+---------------------------------------+ | TBD7 | fsFrecFrecDroppedCountTsFirst | +-------+---------------------------------------+ | TBD8 | fsFrecFrecDroppedCountTsLast | +-------+---------------------------------------+ | TBD9 | fsFrecUnexportedFrecCount | +-------+---------------------------------------+ | TBD10 | fsFrecUnexportedPacketInFrecCount | +-------+---------------------------------------+ | TBD11 | fsFrecUnexportedBytesInFrecCount | +-------+---------------------------------------+ | TBD12 | fsExpPacketInDroppedRecsCountTsFirst | +-------+---------------------------------------+ | TBD13 | fsExpPacketInDroppedRecsCountTsLast | +-------+---------------------------------------+ | TBD14 | fsExpBytesInDroppedRecsCount | +-------+---------------------------------------+ | TBD15 | fsExpFrecDroppedCountTsFirst | +-------+---------------------------------------+ | TBD16 | fsExpFrecDroppedCountTsLast | +-------+---------------------------------------+ | TBD17 | fsExpUnexportedCount | +-------+---------------------------------------+ | TBD18 | fsExpUnexportedPacketCount | +-------+---------------------------------------+ | TBD19 | fsExpUnexportedByteInExpCount | +-------+---------------------------------------+ 7.1. Meter process related (TBD1-TBD3) Information Elements presented in this section are related to Flow Selection performed during the Metering Process. Peluso, et al. Expires July 11, 2011 [Page 16] Internet-Draft Flow Selection Techniques January 2011 +------+---------------------------------+ | ID | Name | +------+---------------------------------+ | TBD1 | fsMeterUnmeasPacketCountTsFirst | +------+---------------------------------+ | TBD2 | fsMeterUnmeasPacketCountTsLast | +------+---------------------------------+ | TBD3 | fsMeterUnmeasBytesCount | +------+---------------------------------+ 7.1.1. fsMeterUnmeasPacketCountTsFirst Description: Specifies the timestamp of the first packet not measured because of the use of the flow state dependent sampling. Together with the IE fsMeterUnmeasPacketCountTsLast it allows to evaluate the count of packets that were not measured because of the use of the flow sampling. Abstract Data Type: dateTimeSeconds ElementId: TBD2 Status: Proposed Units: seconds 7.1.2. fsMeterUnmeasPacketCountTsLast Description: Specifies the timestamp of the last packet not measured because of the use of the flow state dependent sampling. Together with the IE fsMeterUnmeasPacketCountTsFirst it allows to evaluate the count of packets that were not measured because of the use of the flow sampling. Abstract Data Type: dateTimeSeconds ElementId: TBD2 Status: Proposed Units: seconds Peluso, et al. Expires July 11, 2011 [Page 17] Internet-Draft Flow Selection Techniques January 2011 7.1.3. fsMeterUnmeasBytesCount Description: This Information Element specifies the count of bytes that were not measured because of the use of the flow state dependent sampling. Abstract Data Type: unsigned64 Data Type Semantics: deltaCounter ElementId: TBD3 Status: Proposed Units: bytes 7.2. Flow recording process related (TBD4-TBD11) Information Elements presented in this section are related to Flow Selection performed during the Flow Recording Process. +-------+---------------------------------------+ | ID | Name | +-------+---------------------------------------+ | TBD4 | fsFrecPacketInDroppedRecsCountTsFirst | +-------+---------------------------------------+ | TBD5 | fsFrecPacketInDroppedRecsCountTsLast | +-------+---------------------------------------+ | TBD6 | fsFrecByteInDroppedRecsCount | +-------+---------------------------------------+ | TBD7 | fsFrecFrecDroppedCountTsFirst | +-------+---------------------------------------+ | TBD8 | fsFrecFrecDroppedCountTsLast | +-------+---------------------------------------+ | TBD9 | fsFrecUnexportedFrecCount | +-------+---------------------------------------+ | TBD10 | fsFrecUnexportedPacketInFrecCount | +-------+---------------------------------------+ | TBD11 | fsFrecUnexportedBytesInFrecCount | +-------+---------------------------------------+ 7.2.1. fsFrecPacketInDroppedRecsCountTsFirst Description: Peluso, et al. Expires July 11, 2011 [Page 18] Internet-Draft Flow Selection Techniques January 2011 Specifies the timestamp of the first non-exported packet that was contained in the flow records eliminated from the flow recording process because of resource limitations/policies in the flow recording process. Abstract Data Type: dateTimeSeconds ElementId: TBD4 Status: Proposed Units: seconds 7.2.2. fsFrecPacketInDroppedRecsCountTsLast Description: Specifies the timestamp of the last non-exported packet that was contained in the flow records eliminated from the flow recording process because of resource limitations/policies in the flow recording process. Abstract Data Type: dateTimeSeconds ElementId: TBD5 Status: Proposed Units: seconds 7.2.3. fsFrecByteInDroppedRecsCount Description: This Information Element specifies the count of the non-exported bytes that were contained in the flow records eliminated from the flow recording process because of resource limitations/policies in the flow recording process. Abstract Data Type: unsigned64 Data Type Semantics: deltaCounter ElementId: TBD6 Status: Proposed Units: bytes Peluso, et al. Expires July 11, 2011 [Page 19] Internet-Draft Flow Selection Techniques January 2011 7.2.4. fsFrecFrecDroppedCountTsFirst Description: Specifies the timestamp of the first flow record elimination event occurring during the flow recording process. Together with the IE fsFrecFrecDroppedCountTsLast it allows to estimate the count of flow records containing non-exported packets eliminated from the flow recording process because of resources limitations/policies in the flow recording process. Abstract Data Type: dateTimeSeconds ElementId: TBD7 Status: Proposed Units: seconds 7.2.5. fsFrecFrecDroppedCountTsLast Description: Specifies the timestamp of the last flow record elimination event occurring during the flow recording process. Together with the IE fsFrecFrecDroppedCountTsFirst it allows to estimate the count of flow records containing non-exported packets eliminated from the flow recording process because of resources limitations/policies in the flow recording process. Abstract Data Type: dateTimeSeconds ElementId: TBD8 Status: Proposed Units: seconds 7.2.6. fsFrecUnexportedFrecCount Description: This Information Element specifies the count of the flow records currently existing in the flow recording process containing at least one non-exported packet. Abstract Data Type: unsigned32 Peluso, et al. Expires July 11, 2011 [Page 20] Internet-Draft Flow Selection Techniques January 2011 Data Type Semantics: deltaCounter ElementId: TBD9 Status: Proposed Units: flow records 7.2.7. fsFrecUnexportedPacketInFrecCount Description: This Information Element specifies the count of non-exported packets contained in flow records of the flow recording process. Abstract Data Type: unsigned32 Data Type Semantics: deltaCounter ElementId: TBD10 Status: Proposed Units: packets 7.2.8. fsFrecUnexportedBytesInFrecCount Description: This Information Element specifies the count of non-exported bytes contained in flow records of the flow recording process. Abstract Data Type: unsigned64 Data Type Semantics: deltaCounter ElementId: TBD11 Status: Proposed Units: bytes 7.3. Flow exporting process related (TBD12-TBD19) Information Elements presented in this section are related to Flow Selection performed during the Flow Exporting Process. Peluso, et al. Expires July 11, 2011 [Page 21] Internet-Draft Flow Selection Techniques January 2011 +-------+--------------------------------------+ | ID | Name | +-------+--------------------------------------+ | TBD12 | fsExpPacketInDroppedRecsCountTsFirst | +-------+--------------------------------------+ | TBD13 | fsExpPacketInDroppedRecsCountTsLast | +-------+--------------------------------------+ | TBD14 | fsExpByteInDroppedRecsCount | +-------+--------------------------------------+ | TBD15 | fsExpFrecDroppedCountTsFirst | +-------+--------------------------------------+ | TBD16 | fsExpFrecDroppedCountTsLast | +-------+--------------------------------------+ | TBD17 | fsExpUnexportedCount | +-------+--------------------------------------+ | TBD18 | fsExpUnexportedPacketCount | +-------+--------------------------------------+ | TBD19 | fsExpUnexportedByteInExpCount | +-------+--------------------------------------+ 7.3.1. fsExpPacketInDroppedRecsCountTsFirst Description: Specifies the timestamp of the first non-exported packet belonging to an eliminated flow record. Together with the IE fsExpPacketInDroppedRecsCountTsLast it allows to estimate the count of non-exported packets that were contained in the flow records eliminated from the flow recording process because of resource limitations/policies in the exporting process. Abstract Data Type: dateTimeSeconds ElementId: TBD12 Status: Proposed Units: seconds 7.3.2. fsExpPacketInDroppedRecsCountTsLast Description: Specifies the timestamp of the last non-exported packet belonging to an eliminated flow record. Together with the IE fsExpPacketInDroppedRecsCountTsFirst it allows to estimate the count of non-exported packets that were contained in the flow records eliminated from the flow recording process because of Peluso, et al. Expires July 11, 2011 [Page 22] Internet-Draft Flow Selection Techniques January 2011 resource limitations/policies in the exporting process. Abstract Data Type: dateTimeSeconds ElementId: TBD13 Status: Proposed Units: seconds 7.3.3. fsExpByteInDroppedRecsCount Description: This Information Element specifies the count of non-exported bytes that were contained in the flow records eliminated from the flow recording process because of resource limitations/policies in the exporting process. Abstract Data Type: unsigned64 Data Type Semantics: deltaCounter ElementId: TBD14 Status: Proposed Units: bytes 7.3.4. fsExpFrecDroppedCountTsFirst Description: Specifies the timestamp of the first flow record elimination event occurring during the flow recording process. Together with the IE fsExpFrecDroppedCountTsLast it allows to estimate the count of flow records containing non-exported packets eliminated from the flow recording process because of resource limitations/policies in the exporting process. Abstract Data Type: dateTimeSeconds ElementId: TBD15 Status: Proposed Units: seconds Peluso, et al. Expires July 11, 2011 [Page 23] Internet-Draft Flow Selection Techniques January 2011 7.3.5. fsExpFrecDroppedCountTsLast Description: Specifies the timestamp of the last flow record elimination event occurring during the flow recording process. Together with the IE fsExpFrecDroppedCountTsFirst it allows to estimate the count of flow records containing non-exported packets eliminated from the flow recording process because of resource limitations/policies in the exporting process. Abstract Data Type: dateTimeSeconds ElementId: TBD16 Status: Proposed Units: seconds 7.3.6. fsExpUnexportedCount Description: This Information Element specifies the count of the flow records currently existing in the flow recording process, containing non- exported traffic and not being exported because of exporting process resource limitations/policies. Abstract Data Type: unsigned32 Data Type Semantics: deltaCounter ElementId: TBD17 Status: Proposed Units: flow records 7.3.7. fsExpUnexportedPacketCount Description: This Information Element specifies the count of non-exported packets contained in the flow records of the flow recording process not being exported because of exporting process resource limitations/policies. Abstract Data Type: unsigned32 Peluso, et al. Expires July 11, 2011 [Page 24] Internet-Draft Flow Selection Techniques January 2011 Data Type Semantics: deltaCounter ElementId: TBD18 Status: Proposed Units: packets 7.3.8. fsExpUnexportedByteInExpCount Description: This Information Element specifies the count of non-exported bytes contained in the flow records of the flow recording process not being exported because of exporting process resource limitations/ policies. Abstract Data Type: unsigned64 Data Type Semantics: deltaCounter ElementId: TBD19 Status: Proposed Units: bytes 8. Implementation requirements In order to implement the described information model counters for non-exported packets and bytes have to be inserted in the flow records as flow metrics. Sometimes these counters are referred to as delta counts. An implementation may also keep absolute counts for purposes not being specified in this information model (both delta and absolute counters can be exported in the IPFIX information model, see [RFC5102]). In addition, to fully support this information model, it would be REQUIRED to keep in a flow record the timestamps of the first and last non-exported packets. An implementation may need to keep timestamps of the first and last exported packets for other purposes than those of this information model, or to join the two timers for the last exported and first exported packets, or even to approximate them with the time of the exporting event. 9. Information Model for Configuration of Flow Selection Techniques This section aims at describing the representative parameters of the Peluso, et al. Expires July 11, 2011 [Page 25] Internet-Draft Flow Selection Techniques January 2011 flow selection techniques presented above. To this regard, it provides the basis of an information model to be adopted in order to configure the flow selection process within an IPFIX device. The information elements herein introduced are defined in accordance with the IPFIX information model [RFC5102]. Furthermore, the data types used to represent the Flow Selection-related information elements are those defined in section 3.1 of the IPFIX information model [RFC5102]. List of additional Flow Selection information elements: +-------+-------------------------+-------+-----------------------+ | ID | Name | ID | Name | +-------+-------------------------+-------+-----------------------+ | TBD20 | selectorMethod | TBD21 | flowRecordPacketsSize | +-------+-------------------------+-------+-----------------------+ | TBD22 | flowMaxAdmitFlowRecords | TBD23 | flowInactivityTime | +-------+-------------------------+-------+-----------------------+ | TBD24 | flowRecordBytesSize | | | +-------+-------------------------+-------+-----------------------+ 9.1. selectorMethod Description: This Information Element identifies the flow selection method that is applied by the Flow Selection process, in accordance with what is described in section 5 of this document. Some of these methods MAY have parameters in order to fully support the selected technique. For that reason, further Information Elements are defined in the following subsections. Flow selection methods identifiers are herein defined: +----+-----------------+--------------------------------------------+ | ID | Method | Parameters | +----+-----------------+--------------------------------------------+ | 1 | Selection based | flowMaxAdmitFlowRecords | | | on flow size | flowRecordBytesSize flowRecordPacketsSize | | | count | | +----+-----------------+--------------------------------------------+ | 2 | Selection based | flowMaxAdmitFlowRecords | | | on flow content | flowIPAddressVersion flowSourceIPv4Address | | | property match | flowDestinationIPv4Address | | | | flowSourceIPv6Address | | | | flowDestinationIPv6Address | | | | flowDestinationPort | Peluso, et al. Expires July 11, 2011 [Page 26] Internet-Draft Flow Selection Techniques January 2011 | 3 | Selection based | flowMaxAdmitFlowRecords flowInactivityTime | | | on flow record | | | | arrival time or | | | | sequence | | +----+-----------------+--------------------------------------------+ | 4 | Selection based | flowMaxAdmitFlowRecords | | | on external | | | | events | | +----+-----------------+--------------------------------------------+ Abstract Data Type: unsigned16 Data Type Semantics: identifier ElementId: TBD20 Status: Proposed 9.2. flowMaxAdmitFlowRecords Description: This Information Element specifies the maximum number of eligible flow records which might be created in the flow cache. It is used by the Selector Process in order to identify the time when flow selection should be triggered. A value of 0 means that the Flow Selection State related to the memory space available for flow recording MUST be used to estimate the max flow cache size. For example, this Information Element MAY be used to set the configuration of a flow size count Flow Selector. Abstract Data Type: unsigned32 Data Type Semantics: quantity ElementId: TBD21 Status: Proposed Units: flow records 9.3. flowRecordBytesSize Description: This Information Element specifies the minimum number of bytes contained in a flow record to be considered as not eligible for Peluso, et al. Expires July 11, 2011 [Page 27] Internet-Draft Flow Selection Techniques January 2011 removal. It MAY be used for elephant flows identification. For example, this Information Element MAY be used to describe the configuration of a flow size count Flow Selector. Abstract Data Type: unsigned64 Data Type Semantics: quantity ElementId: TBD22 Status: Proposed Units: bytes 9.4. flowRecordPacketsSize Description: This Information Element specifies the minimum number of packets contained in a flow record to be considered as not eligible for removal. It MAY be used for elephant flows identification. For example, this Information Element MAY be used to describe the configuration of a flow size count Flow Selector. Abstract Data Type: unsigned32 Data Type Semantics: quantity ElementId: TBD23 Status: Proposed Units: packets 9.5. flowInactivityTime Description: This Information Element specifies the time interval in microseconds during which the corresponding flow record may be considered as still active. It is used by the metering process and/or the flow recording process in order to make the decision on whether to discard an existing flow to make room for a new one. For example, this Information Element may be used to describe the configuration of a flow arrival time Flow Selector. Peluso, et al. Expires July 11, 2011 [Page 28] Internet-Draft Flow Selection Techniques January 2011 Abstract Data Type: dateTimeMicroseconds Data Type Semantics: quantity ElementId: TBD24 Status: Proposed Units: microseconds 9.6. flowIPAddressVersion Description: This Information Element specifies the IP version field of the packets belonging to flow records which SHOULD be considered as not eligible for removal. Abstract Data Type: unsigned8 Data Type Semantics: identifier ElementId: 60 Status: current Reference: see [RFC5102] for the definition of the ipVersion information element and its Id. 9.7. flowIPv4SourceAddress Description: This Information Element specifies the IPv4 source address which is the parameter to be used as flow key to access the flow record not eligible for removal. Abstract Data Type: ipv4Address Data Type Semantics: identifier ElementId:8 Status: current Reference: see [RFC5102] for the definition of the sourceIPv4Address information element and its Id. Peluso, et al. Expires July 11, 2011 [Page 29] Internet-Draft Flow Selection Techniques January 2011 9.8. flowIPv4DestinationAddress Description: This Information Element specifies the IPv4 destination address which is the parameter to be used as flow key to access the flow record not eligible for removal. Abstract Data Type: ipv4Address Data Type Semantics: identifier ElementId:12 Status: current Reference: see [RFC5102] for the definition of the destinationIPv4Address information element and its Id. 9.9. flowIPv6SourceAddress Description: This Information Element specifies the IPv6 source address which is the parameter to be used as flow key to access the flow record not eligible for removal. Abstract Data Type: ipv6Address Data Type Semantics: identifier ElementId:27 Status: current Reference: see [RFC5102] for the definition of the sourceIPv6Address information element and its Id. 9.10. flowIPv6DestinationAddress Description: This Information Element specifies the IPv6 destination address which is the parameter to be used as flow key to access the flow record not eligible for removal. Abstract Data Type: ipv6Address Peluso, et al. Expires July 11, 2011 [Page 30] Internet-Draft Flow Selection Techniques January 2011 Data Type Semantics: identifier ElementId:28 Status: current Reference: see [RFC5102] for the definition of the destinationIPv6Address information element and its Id. 10. IANA Considerations This document introduces several new information elements as an extension to the IPFIX information model. Values TBD1-TBD19 in this document should be replaced with the assigned numbers by IANA. 11. Security Considerations In this section security issues concerning an IPFIX device performing flow selection are pointed out. In case the flow selection function is activated an IPFIX device might be exposed to security threats. Since flow selection implies analysing flow packets, associating them to a specific traffic flow and selecting flow records, a malicious user who was able to gain control of an IPFIX device might access both packet and flow data, thus violating their confidentiality. Furthermore, the intruder might be attracted by the possibility of altering the flow selection process by modifying the criteria used to select flow records. In this case, the IPFIX device would export flow data which are different from the ones that the Collector expects to receive. It is apparent that these security threats can be mitigated by authenticating entities that interact with the IPFIX device and keeping information for flow selection configuration confidential. 12. References 12.1. Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. Peluso, et al. Expires July 11, 2011 [Page 31] Internet-Draft Flow Selection Techniques January 2011 12.2. Informative References [DuLT01a] Duffield, N., Lund, C., and M. Thorup, "Charging from Sampled Network Usage", ACM Internet Measurement Workshop IMW 2001, San Francisco, USA, November 2001. [DuLT01b] Duffield, N., Lund, C., and M. Thorup, "Properties and Prediction of Flow Statistics from Sampled Packet Streams", ACM SIGCOMM Internet Measurement Workshop 2002, November 2002. [EsVa01] Estan, C. and G,. Varghese, "New Directions in Traffic Measurement and Accounting: Focusing on the Elephants, Ignoring the Mice", ACM SIGCOMM Internet Measurement Workshop 2001, San Francisco (CA), November 2001. [IPFIX-REQ] Quittek, J., Zseby, T., Claise, B., and S. Zander, "Requirements for IP Flow Information Export", RFC3917 Requirements for IP Flow Information Export (IPFIX), July 2008. [KuXW04] Kumar, K., Xu, J., Wang, J., Spatschek, O., and L. Li, "Space-code bloom filter for efficient per-flow traffic measurement", INFOCOM 2004 Twenty-third AnnualJoint Conference of the IEEE Computer and Communications Societies, March 2004. [Moli03] Molina, M., "A scalable and efficient methodology for flow monitoring in the Internet", International Teletraffic Congress (ITC-18), Berlin, September 2003. [RFC5102] Quittek, J., Bryant, S., Claise, B., Aitken, P., and J. Meyer, "Information Model for IP Flow Information Export", RFC 5102, January 2008. [RFC5470] Sadasivan, G., Bownlee, N., Claise, B., and J. Quittek, "Architecture for IP Flow Information Export", RFC5470 Architecture for IP Flow Information Export, September 2006. [RFC5475] Zseby, T., Molina, M., Raspall, F., Duffield, N., and S. Niccolini, "Sampling and Filtering techniques for IP Packet Selection", RFC5475 Sampling and Filtering techniques for IP Packet Selection, July 2008. Peluso, et al. Expires July 11, 2011 [Page 32] Internet-Draft Flow Selection Techniques January 2011 Authors' Addresses Lorenzo Peluso University of Napoli Via Claudio 21 Napoli 80125 Italy Phone: +39 081 7683821 Email: lorenzo.peluso@unina.it Tanja Zseby Fraunhofer Institute FOKUS Kaiserin-Augusta-Allee 31 Berlin 10589 Germany Phone: +49 30 3463 7153 Email: tanja.zseby@fokus.fraunhofer.de Salvatore D'Antonio CINI Consortium/University of Napoli "Parthenope" Monte S.Angelo, Via Cinthia Napoli 80126 Italy Phone: +39 081 679944 Email: saldanto@unina.it Christian Henke Fraunhofer Institute FOKUS Kaiserin-Augusta-Allee 31 Berlin 10589 Germany Phone: +49 30 3463 7366 Email: christian.henke@fokus.fraunhofer.de Peluso, et al. Expires July 11, 2011 [Page 33]