Exporting Aggregated Flow Data using the IP Flow Information Export (IPFIX) Protocol
draft-trammell-ipfix-a8n-00.txt

Abstract

This document describes the export of aggregated Flow information using IPFIX. An aggregated Flow is essentially an IPFIX Flow with an externally imposed time interval, generally representing packets from one or more original Flows. The document describes aggregated Flow export within the framework of IPFIX Mediators, defines an interoperable, implementation-independent method for aggregated Flow export, covering time interval export, counter distribution and export, and counting of original Flows.

Status of this Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at http://datatracker.ietf.org/drafts/current/.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as “work in progress.”

This Internet-Draft will expire on January 6, 2011.

Copyright Notice

Copyright (c) 2010 IETF Trust and the persons identified as the document authors. All rights reserved.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.

Table of Contents

1.  Introduction
2.  Terminology
3.  Requirements for Aggregation Support in IPFIX
4.  Aggregation of IP Flows
    4.1.  A general model for IP Flow Aggregation
    4.2.  Aggregating and Distributing Counters
    4.3.  Counting Original Flows
5.  Aggregation in the IPFIX Architecture
6.  Export of Aggregated IP Flows using IPFIX
    6.1.  Flow Count Export
        6.1.1.  originalFlowsPresent Information Element
        6.1.2.  originalFlowsInitiated InformationElement
        6.1.3.  originalFlowsCompleted InformationElement
        6.1.4.  originalFlows InformationElement
    6.2.  Aggregate Counter Distibution Export
    6.3.  Time Interval Export
7.  Examples
8.  Security Considerations
9.  IANA Considerations
10.  References
    10.1.  Normative References
    10.2.  Informative References
§  Authors' Addresses

1.  Introduction

The aggregation of packet data into flows serves a variety of different purposes, as noted in [RFC3917] (Quittek, J., Zseby, T., Claise, B., and S. Zander, “Requirements for IP Flow Information Export (IPFIX),” October 2004.) and [RFC5472] (Zseby, T., Boschi, E., Brownlee, N., and B. Claise, “IP Flow Information Export (IPFIX) Applicability,” March 2009.). Aggregation beyond the flow level, into records representing multiple flows, is a common analysis and data reduction technique as well. Often these aggregation operations result in a time series of keys and counters, which is useful for gaining insight into trends and anomalies. Aggregation may also has added benefits for privacy: when data about specific transactions and hosts are aggregated together, it has an anonymising effect on the data.

Aggregation can be applied in parallel with storage of original packet and flow data. In certain situations, aggregation can be performed at mediator, with aggregated data can be used for initial analysis (e.g., to provide a reduced data stream for analysis at a central point, with original flow data stored in a higher-access-latency manner (e.g. tape storage at the mediator at a remote site). Other applications may benefit from a reverse application: online analysis of original flows, with long-term storage of aggregated data. Aggregation can also be used to provide a lower-volume, lower-privacy-risk source of data for reporting or data exchange across organizational boundaries.

While IPFIX can be used to export [RFC5101] (Claise, B., “Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of IP Traffic Flow Information,” January 2008.) and store [RFC5655] (Trammell, B., Boschi, E., Mark, L., Zseby, T., and A. Wagner, “Specification of the IP Flow Information Export (IPFIX) File Format,” October 2009.) aggregated data, there exists as yet no common terminology or aggregation metadata representation for this purpose, no set of requirements to define what is meant by aggregation, and no facility for counting original Flows from which Aggregated Flows are derived. This document seeks to remedy this situation, defining a common basis for the application of IPFIX to the handling of aggregated data, with applicability to large-scale network data processing, archiving, and inter-organization exchange.

2.  Terminology

Terms used in this document that are defined in the Terminology section of the IPFIX Protocol (Claise, B., “Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of IP Traffic Flow Information,” January 2008.) [RFC5101] document are to be interpreted as defined there.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 (Bradner, S., “Key words for use in RFCs to Indicate Requirement Levels,” March 1997.) [RFC2119].

In addition, this document defines the following terms

Aggregated Flow:

A Flow, as defined by [RFC5101] (Claise, B., “Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of IP Traffic Flow Information,” January 2008.), derived from a set of zero or more original Flows within a defined time interval. The two primary differences between a Flow and an Aggregated Flow are (1) that the time interval of a Flow is generally derived from information about the timing of the packets comprising the Flow, while the time interval of an Aggregated Flow are generally externally imposed; and (2) that an Aggregated Flow may represent zero packets (i.e., an assertion that no packets were seen for a given Flow Key in a given time interval).

(Intermediate) Aggregation Function:

A mapping from a set of zero or more original Flows into a set of aggregated Flow, that separates the original Flows into a set of one or more given time intervals.

(Intermediate) Aggregation Process:

An Intermediate Process, as in [I‑D.ietf‑ipfix‑mediators‑framework] (Kobayashi, A., Claise, B., Muenz, G., and K. Ishibashi, “IPFIX Mediation: Framework,” April 2010.), hosting an Intermediate Aggregation Function.

[TODO: may want to define a more precise terminology for aggregated flow time interval, "contribution"/"presence" of an original Flow within an aggregated Flow...]

3.  Requirements for Aggregation Support in IPFIX

In defining a terminology, additional information elements, and metadata export methods for Aggregated Flow export using IPFIX, we have sought to meet the following requirements.

First, a specification of Aggregated Flow export must seek to be as interoperable as possible. Export of Aggregated Flows using the techniques described in this document will result in Flow data which can be collected by Collecting Processes and read by File Readers which do not provide any special support for Aggregated Flow export.

Second, a specification of Aggregated Flow export must seek to be as implementation-independent as the IPFIX protocol itself. In Section 5 (Aggregation in the IPFIX Architecture), we specify the flow aggregation process as an intermediate process within the IPFIX Mediator framework (Kobayashi, A., Claise, B., Muenz, G., and K. Ishibashi, “IPFIX Mediation: Framework,” April 2010.) [I‑D.ietf‑ipfix‑mediators‑framework], and specify a variety of different architectural arrangements for flow aggregation; these are meant to be descriptive as opposed to proscriptive. In metadata export, we seek to define properties of the set of exported Aggregated Flows, as opposed to the properties of the specific algorithms used to aggregate these Flows. Specifically out of scope for this effort are any definition of a language for proscribing aggregation records, or the configuration parameters of Aggregation Processes.

From the definition of presented in Section 2 (Terminology), an Aggregated Flow is a Flow as in [RFC5101] (Claise, B., “Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of IP Traffic Flow Information,” January 2008.), with a restricted defintion as to the packets making up the Flow. Practically speaking, Aggregated Flows are derived from original Flows, as opposed to a raw packet stream. Key to this definition of Aggregated Flow is how timing affects the process of aggregation, as for the most part flow aggregation takes place within some set of (usually regular) time intervals. Any specification for Aggregated Flow export must account for the special role time intervals play in aggregation, and the many-to-many relationship between Aggregated Flows and original Flows which this implies.

4.  Aggregation of IP Flows

[TODO: frontmatter: restate definition: a flow is just a flow. Here we talk about Aggregation Process actions.]

4.1.  A general model for IP Flow Aggregation

[TODO: introduce as little of an Aggregation Process as necessary to be able to talk about things practically. Define in simple terms the role of externally imposed time intervals on aggregation. ]

4.2.  Aggregating and Distributing Counters

In general, counters in Aggregated Flows are treated the same as in any Flow: on a per-Information Element basis, the counters are calculated as if they were derived from the set of packets in the original flow. For the most part, when aggregating original Flows into Aggregated Flows, this is simply done by summation.

However, this raises a complication when aggregating original Flows for which original packet data is not available: often, when imposing an external time interval on an original Flow, the original Flow will incompletely cover one or more time intervals, and apply to one or more Aggregated Flows; in this case, the Aggregation Process must distribute the counters in the original Flows across the multiple Aggregated Flows. There are several methods for doing this, listed here in increasing order of complexity and accuracy:

End Interval:

The counters for an original Flow are added to the counters of the appropriate Aggregated Flow containing the end time of the original Flow.

Start Interval:

The counters for an original Flow are added to the counters of the appropriate Aggregated Flow containing the start time of the original Flow.

Mid Interval:

The counters for an original Flow are added to the counters of a single appropriate Aggregated Flow containing some timestamp between start and end time of the original Flow.

Simple Uniform Distribution:

Each counter for an original Flow is divided by the number of time intervals the original Flow covers (i.e., of appropriate Aggregated Flows sharing the same Flow Key), and this number is added to each corresponding counter in each Aggregated Flow.

Proportional Uniform Distribution:

Each counter for an original Flow is divided by the number of time _units_ the original Flow covers, to derive a mean count rate. This mean count rate is then multiplied by the number of time units in the intersection of the duration of the original Flow and the time interval of each Aggregated Flow. This is like simple uniform distribtion, but accounts for the fractional portions of a time interval covered by an original Flow in the first and last time interval.

Simulated Process:

Each counter of the original Flow is distributed among the intervals of the Aggregated Flows according to some function the Aggregation Process uses based upon properties of Flows presumed to be like the original Flow. For example, bulk transfer flows might follow a more or less proportional uniform distribtion, while interactive processes are far more bursty.

Direct:

The Aggregating Process has access to the original packet timings from the packets making up the original Flow, and uses these to distribute or recalculate the counters.

A method for exporting the distribution of counters across multiple Aggregated Flows is detailed in Section 6.2 (Aggregate Counter Distibution Export). In any case, counters MUST be distributed across the multiple Aggregated Flows in such a way that the total count is preserved; this property allows data to be aggregated and re-aggregated without any loss of original count information. To avoid confusion in interpretation of the aggregated data, all the counters for a set of given original Flows SHOULD be distributed via the same method.

4.3.  Counting Original Flows

When aggregating multiple original Flows into an Aggregated Flow, it is often useful to know how many original Flows are present in the Aggregated Flow. This document introduces four new information elements in Section 6.1 (Flow Count Export) to export these counters.

There are two possible ways to count original Flows, which we call here conservative and non-conservative. Conservative flow counting has the property that each original Flow contributes exactly one to the total flow count within a set of aggregated Flows. In other words, conservative flow counters are distributed just as any other counter, except each original Flow is assumed to have a flow count of one. When a count for an original Flow must be distributed across a set of Aggregated Flows, and a distribution method is used which does not account for that original Flow completely within a single Aggregated Flow, conservative flow counting requires a fractional representation.

By contrast, non-conservative flow counting is used to count how many flows are represented in an Aggregated Flow. Flow counters are not distributed in this case. An original Flow which is present within N Aggregated Flows would add N to the total non-conservative flow count, one to each Aggregated Flow.

5.  Aggregation in the IPFIX Architecture

[TODO: describe these diagrams]

+==========================================+
| Exporting Process                        |
+==========================================+
  |                                      |
  |             (Aggregated Flow Export) |
  V                                      |
+=============================+          |
| Mediator                    |          |
+=============================+          |
  |                                      |
  | (Aggregating Mediator)               |
  V                                      V
+==========================================+
| Collecting Process                       |
+==========================================+
        |
        | (Aggregation for Storage)
        V
+--------------------+
| IPFIX File Storage |
+--------------------+

packets --+                     +- IPFIX Messages -+
          |                     |                  |
          V                     V                  V
+==================+ +====================+ +=============+
| Metering Process | | Collecting Process | | File Reader |
+==================+ +====================+ +=============+
          |            original | Flows            |
          V                     V                  V
+=========================================================+
|           Intermediate Aggregation Process (IAP)        |
+=========================================================+
          | Aggregated                  Aggregated |
          | Flows                            Flows |
          V                                        V
+===================+                       +=============+
| Exporting Process |                       | File Writer |
+===================+                       +=============+
          |                                        |
          +------------> IPFIX Messages <----------+

         +----+  +-----+  +----+
 pkts -> | MP |->| IAP |->| EP |-> Export of Aggregated Flows
         +----+  +-----+  +----+
         +----+  +-----+  +----+
IPFIX -> | CP |->| IAP |->| EP |-> Aggregating Mediator
         +----+  +-----+  +----+
         +----+  +-----+  +----+
IPFIX -> | CP |->| IAP |->| FW |-> Aggregation for storage in IPFIX Files
         +----+  +-----+  +----+
         +----+  +-----+  +----+
IPFIX -> | FR |->| IAP |->| FW |-> Aggregation for analysis
 File    +----+  +-----+  +----+   (aggregating File manipulator)

6.  Export of Aggregated IP Flows using IPFIX

[TODO: frontmatter: here we talk about export specifics.]

6.1.  Flow Count Export

[TODO: describe these four IEs. for now, see Section 4.3 (Counting Original Flows)]

6.1.1.  originalFlowsPresent Information Element

Description:

The non-conservative count of original Flows containing packets from which a this Aggregated Flow was aggregated.

Abstract Data Type:

unsigned64

ElementId:

TBD1

Status:

Proposed

6.1.2.  originalFlowsInitiated InformationElement

Description:

The conservative count of original Flows whose first packet is represented within this Aggregated Flow.

Abstract Data Type:

unsigned64

ElementId:

TBD2

Status:

Proposed

6.1.3.  originalFlowsCompleted InformationElement

Description:

The conservative count of original Flows whose last packet is represented within this Aggregated Flow.

Abstract Data Type:

unsigned64

ElementId:

TBD3

Status:

Proposed

6.1.4.  originalFlows InformationElement

Description:

The conservative count of original Flows represented within this Aggregated Flow; may be distributed via any of the methods described in Section 4.2 (Aggregating and Distributing Counters)

Abstract Data Type:

float64

ElementId:

TBD4

Status:

Proposed

6.2.  Aggregate Counter Distibution Export

[TODO: options template and IE for representing the distribution methods in Section 4.2 (Aggregating and Distributing Counters)]

6.3.  Time Interval Export

Since an Aggregated Flow is simply a Flow, the existing timestamp Information Elements in the IPFIX Information Model (e.g., flowStartMilliseconds, flowEndNanoseconds) are sufficient to specify the time interval for aggregation. Therefore, this document specifies no new aggregation-specific Information Elements for exporting time interval information.

Each Aggregated Flow SHOULD contain both an interval start and interval end timestamp. If an exporter of Aggregated Flows omits the interval end timestamp from each Aggregated Flow, the time interval for Aggregated Flows within an Observation Domain and Transport Session MUST be regular and constant, and SHOULD be exported using the Options Record described in this section. However, note that this approach might lead to interoperability problems when exporting Aggregated Flows to non-aggregation-aware Collecting Processes and downstream analysis tasks; therefore, an Exporting Process capable of exporting only interval start timestamps MUST provide a configuration option to export interval end timestamps as well.

[TODO: options template and IE for binding a time interval to a session.]

7.  Examples

[TODO]

8.  Security Considerations

[TODO]

9.  IANA Considerations

[TODO: add all IEs defined in Section 6.]

10.  References

10.1. Normative References

10.2. Informative References

Authors' Addresses