VNF Benchmarking MethodologyUniversity of CampinasAv. Albert Einstein 300CampinasSao Paulo13083-852Brazilraphaelvrosa@gmail.comhttp://www.intrig.dca.fee.unicamp.br/Ericsson Research, HungaryIrinyi Jozsef u. 4-20Budapest1117Hungaryrobert.szabo@ericsson.comhttp://www.ericsson.com/
IETF
BMWGInternet-DraftThis document describes VNF benchmarking methodologies.New paradigms of network services envisioned by NFV bring VNFs as
software based entities, which can be deployed in virtualized
environments . In order to be
managed/orchestrated or compared with physical network functions,
VNF Descriptors can specify performance profiles containing
metrics (e.g., throughput) associated with allocated resources
(e.g., vCPU). This document describes benchmarking methodologies
to obtain VNF profiles (resource - performance figures).The reader is assumed to be familiar with the terminology as
defined in the European Telecommunications Standards Institute
(ETSI) NFV document . Some of these terms,
and others commonly used in this document, are defined below.Network Function Virtualization - The
principle of separating network functions from the
hardware they run on by using virtual hardware
abstraction.NFV Infrastructure Point of
Presence - Any combination of virtualized compute,
storage and network resources.NFV Infrastructure - Collection of
NFVI PoPs under one orchestrator.Virtualized Infrastructure Manager - functional
block that is responsible for controlling and managing the
NFVI compute, storage and network resources, usually within
one operator's Infrastructure Domain (e.g. NFVI-PoP).NFV Orchestrator - functional
block that manages the Network Service (NS) life-cycle
and coordinates the management of NS life-cycle,
VNF life-cycle (supported by the VNFM) and NFVI
resources (supported by the VIM) to ensure an optimized
allocation of the necessary resources and connectivity Virtualized Network Function - a
software-based network function.Virtualised Network Function Descriptor -
configuration template that describes a VNF in terms of its
deployment and operational behaviour, and is used in the
process of VNF on-boarding and managing the life cycle of a
VNF instance.Virtualized Network Function
Forwarding Graph - an ordered list of VNFs creating a
service chain.Management and Orchestration - In the
ETSI NFV framework , this is the
global entity responsible for management and
orchestration of NFV life-cycle. composition of Network Functions and defined
by its functional and behavioural specification.Additional terminology not defined by ETSI NFV ISG.VNF Benchmarking Profile - the specification
how to measure a VNF Profile. VNF-BP may be specific to a VNF or
applicable to several VNF types. The specification includes
structural and functional instructions, and variable parameters
(metrics) at different abstractions (e.g., vCPU, memory, throughput,
latency; session, transaction, tenants, etc.).is a mapping between virtualized
resources (e.g., vCPU, memory) and VNF performance (e.g.,
throughput, latency between in/out ports) at a given NFVI PoP. An
orchestration function can use the VNF Profile to select a host
(NFVI PoP) for a VNF and to allocate necessary resources to deliver
the required performance characteristics. A user/subscriber/consumer of ETSI's
Network Service. Network Functions performing benchmarking
tasks (e.g., synthetic traffic sources and sinks; measurement and
observation functions, etc.). System Under Test comprises the VNF under
test.This document assumes VNFs as black boxes when defining VNF
performance benchmarking methodologies. White box benchmarking of
VNFs are left for further studies and may be added later.We assume a VNF benchmarking set-up as shown in . Customers can request Network Services (NS) from
an NFVO with associated service level specifications (e.g.,
throughput and delay). The NFVO, in turn, must select hosts and
software resource allocations for the VNFs and build the necessary
network overlay to meet the requirements. Therefore, the NFVO must
know VNF Profiles per target hosts to perform location and resource
assignments.In a highly dynamic environment, where both the VNF instances
(e.g., revised VM image) and the NFVI resources (e.g., hw upgrades)
are changing, the NFVO should be able to create VNF Profiles
on-demand.We assume, that based on VNF Benchmarking Profile definitions
NFVOs can run benchmarking evaluations to learn VNF Profiles per
target hosts. In a virtualization environment, however, not only the SUT but
all the other benchmarking agents may be software defined (physical
or virtualized network functions). shows an example, where the NFVO can use
PoPa and PoPb to set-up benchmarking functions to test VNFs hosted
in PoP 1, 2, 3 domains corresponding to VIM 1, 2 and 3. The NFVO
uses the VNF Benchmarking Profiles to deploy agents according to the
SUT VNF. The VNF Benchmarking Profile is defined by the VNF
Developer. The results of the VNF benchmarking is stored in a VNF
Profile.VNF benchmarking considerations are defined in . Additionally, VNF pre-deployment testing
considerations are well explored in .This document list further considerations: In virtualization environments neither the VNF instance nor the underlying virtualization environment nor the agents specifics may be known by the entity managing abstract resources. This implies black box testing with black box functional components, which are configured by opaque configuration parameters defined by the VNF developers or alike for the benchmarking entity (e.g., NFVO). Following the ETSI's model (), we distinguish three methods for VNF
evaluation:Where resource {cpu, memory, storage}
parameters are provided and the corresponding {latency, throughput}
performance parameters are obtained. Note, such request might
create multiple reports, for example, with minimal latency or
maximum throughput results.Both resources {cpu, memory, storage}
and performance {latency, throughput} parameters are provided and
agents verifies if the given association is correct or not.Where performance parameters {latency,
throughput} are provided and the corresponding {cpu, memory,
storage} resource parameters obtained. Note, multiple deployment
interactions may be required, or if possible, underlying allocated
resources need to be dynamically altered.Note: Verification and Dimensioning can be reduced to
Benchmarking. Therefore, we detail Benchmarking in what follows.All benchmarking methodologies described in this section
consider the definition of VNF-BPs for each testing procedure.
Information about Benchmarking Methodology for Network
Interconnect Devices, defined in , is
considered in all subsections below. Besides, the tests are
defined based on notions introduced and discussed in the IP
Performance Metrics (IPPM) Framework document .
Provide, for a particular set of
resources allocated, the throughput among two or more VNF ports,
expressed in VNF-BP. VNF (SUT) must be deployed and
stable and its allocated resources collected.
VNF must be reachable by agents. The frame size to
be used for agents must be defined in the VNF-BP.Establish connectivity between agents and VNF ports.Agents initiate source of traffic, specifically designed
for VNF test, increasing rate periodically.Throughput is measured when traffic rate is achieved
without frame losses. report must contain VNF allocated
resources and throughput measured (aka
throughput in ). Provide, for a particular set of
resources allocated, the latency among two or more VNF ports,
expressed in VNF-BP. VNF (SUT) must be deployed and
stable and its allocated resources collected.
VNF must be reachable by agents. The frame size and
respective throughput to be used for agents must be defined in
the VNF-BP.Establish connectivity between agents and VNF ports.Agents initiate source of traffic, throughput and frame size specifically designed for VNF test.Latency is measured when throughput is achieved for the period of time specified in VNF-BP. report must contain VNF allocated
resources, throughput used for
stimulus and latency measurement (aka latency in ). Provide, for a particular set of
resources allocated, the frame loss rate among two or more VNF
ports, expressed in VNF-BP. VNF (SUT) must be deployed and
stable, its allocated resources collected specifying any
particular feature of the underlying VNF virtualized
environment, provided by NFVO/VIM or independently
extracted. VNF must be reachable by agents. Rate of source
traffic and frame type used for agents stimulus must be defined
in VNF-BP.Establish connectivity between agents and VNF ports.Agents initiate source of traffic, specifically designed for VNF test,
achieving rate of source traffic defined in VNF-BP.Frame loss rate is measured when pre-defined traffic rate
is achieved for period of time established in
VNF-BP. report must contain VNF allocated
resources, rate of source traffic used
as stimulus and frame loss rate measurement (aka frame loss rate
in ).This document describes black-box benchmarking methodologies for black-box VNFs in virtualization environments (e.g., ETSI NFV framework) to create VNF Profiles containing the association of resources and performance metrics of a given VNF at a given host (e.g., NFVI PoP).The authors see the following next steps: Two scaling options: single instance with more resources or multiple instances. Questions: What is the maximum performance of a single instance VNF at a given host with increasing resources? How many independent VNF instances (or components) can be run with maximum performance at a given host? On the other hand, what is the performance of the smallest resource footprint VNF allocation? this metric concerns at least three components: VNF bootstraping (SUT), execution environment and the orchestration process. This memo includes no request to IANA.TBDThe authors would like to thank the support of Ericsson Research,
Brazil.This work is partially supported by FP7 UNIFY, a research project
partially funded by the European Community under the Seventh
Framework Program (grant agreement no. 619609). The views
expressed here are those of the authors only. The European
Commission is not liable for any use that may be made of the
information in this document.Architectural Framework - ETSI GS NFV 002 V1.2.1 ETSITerminology for Main Concepts in NFV - ETSI GS NFV 003 V1.2.1 ETSINFV Pre-deployment Testing - ETSI GS NFV TST001 V0.0.15 ETSIConsiderations for Benchmarking Virtual Network Functions and Their InfrastructureA. MortonBenchmarking Methodology for Network Interconnect DevicesS. Bradner and J. McQuaidFramework for IP Performance MetricsV. Paxson, G. Almes, J. Mahdavi, M. Mathis