| Internet-Draft | VNFBench | October 2020 |
| Rosa, et al. | Expires 23 April 2021 | [Page] |
This document describes a common methodology for the automated benchmarking of Virtualized Network Functions (VNFs) executed on general-purpose hardware. Specific cases of automated benchmarking methodologies for particular VNFs can be derived from this document. An open source reference implementation is reported as running code embodiment of the proposed, automated benchmarking methodology.¶
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶
This Internet-Draft will expire on 23 April 2021.¶
Copyright (c) 2020 IETF Trust and the persons identified as the document authors. All rights reserved.¶
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.¶
In [RFC8172] the Benchmarking Methodology Working Group (BMWG) presented considerations for benchmarking of VNFs and their infrastructure, similar to the motivation given, the following aspects reinforce and justify the need for VNF benchmarking: (i) pre-deployment infrastructure dimensioning to realize associated VNF performance profiles; (ii) comparison factor with physical network functions; (iii) and output results for analytical VNF development.¶
Even if many methodologies the BMWG already describes, e.g., self- contained black-box benchmarking, can be applied to VNF benchmarking scenarios, further considerations have to be made. This is because VNFs, which are software components, might not have strict and clear execution boundaries and depend on underlying virtualization environment parameters as well as management and orchestration decisions [ETS14a].¶
Different enabling technologies advent of Software Defined Networking (SDN) and Network Functions Virtualization (NFV) have propitiated the disaggregation of VNFs and benchmarking tools, turning their Application Programming Interfaces (APIs) open and programmable. This process have occurred mostly by: (i) the decoupling of network function's control and data planes; (ii) the development of VNFs as multi-layer and distributed software components; (iii) and the existence of multiple underlying hardware abstractions to be utilized by VNFs.¶
Utilizing SDN and NFV enabling technologies, a diversity of benchmarking tools have been created to facilitate the active stimulus and the passive monitoring of a VNF via diverse software abstraction layers, propitiating a wide variety of abstractions for benchmarking mechanisms in the formulation of a VNF benchmarking methodology. In this manner of establishing the disaggregation of a VNF benchmarking setup, the abstracted VNF benchmarking mechanisms can be programmable, enabling the execution of their underlying technologies by the means of well defined parameters and producing a report with standardized metrics.¶
Turning programmable the execution of a VNF benchmarking methodology enables a richer apparatus for the benchmarking of a VNF and consequently facilitates the high-fidelity assessment of a VNF behaviour. Estimating the behaviour of a VNF depends on three correlated factors:¶
The role of a VNF benchmarking methodology consists in defining how to tackle the diversity of settings imposed by the above enlisted factors in order to extract performance metrics associated with particular VNF packet processing behaviors. The sample space of testing such diversity of settings can be extensively large, turning manual benchmarking experiments prohibitively expensive. Indeed, portability as an intrinsic characteristic of VNFs allows them to be deployed in multiple execution environments, enabling benchmarking setups in a myriad of settings. Thus, the establishment of a methodology for VNF benchmarking automation detains utter importance.¶
Accordingly, can and should the flexible, software-based nature of VNFs be exploited to fully automate the entire benchmarking methodology end-to-end. This is an inherent need to align VNF benchmarking with the agile methods enabled by the concept of Network Functions Virtualization (NFV) [ETS14e]. More specifically it allows: (i) the development of agile performance-focused DevOps methodologies for Continuous Integration and Delivery (CI/CD) of VNFs; (ii) the creation of on-demand VNF test descriptors for upcoming execution environments; (iii) the path for precise-analytics of automated catalogues of VNF performance profiles; (iv) and run-time mechanisms to assist VNF lifecycle orchestration/management workflows, e.g., automated resource dimensioning based on benchmarking insights.¶
Common benchmarking terminology contained in this document is derived from [RFC1242]. The reader is assumed to be familiar with the terminology as defined in the European Telecommunications Standards Institute (ETSI) NFV document [ETS14b]. Some of these terms, and others commonly used in this document, are defined below.¶
This document assumes VNFs as black boxes when defining their benchmarking methodologies. White box approaches are assumed and analysed as a particular case under the proper considerations of internal VNF instrumentation, later discussed in this document.¶
This document outlines a methodology for VNF benchmarking, specifically addressing its automation, without limiting the automated process to a specific benchmarking case or infrastructure. The document addresses state-of-the-art work on VNF benchmarking from scientific publications and current developments in other standardization bodies (e.g., [ETS14c], [ETS19f] and [RFC8204]) wherever possible.¶
Whenever utilizing the specifications of this document, a particular automated VNF benchmarking methodology must be described in a clear and objective manner following four basic principles:¶
VNF benchmarking considerations are defined in [RFC8172]. Additionally, VNF pre-deployment testing considerations are well explored in [ETS14c]. Further, ETSI provides test specifications for networking benchmarks and measurement methods for NFV infrastructure in [ETS19f], which complements the presented work on VNF benchmarking methodologies.¶
Following ETSI's model in [ETS14c], we distinguish three methods for a VNF evaluation:¶
Note: Verification and Dimensioning can be reduced to Benchmarking.¶
The realization of an automated benchmarking methodology can be divided into three stages:¶
A VNF benchmarking architectural framework, shown in Figure 1, establishes the disposal of essential components and control interfaces, explained below, that realize the automation of a VNF benchmarking methodology.¶
+---------------+
| Manager |
Control | (Coordinator) |
Interfaces +---+-------+---+
+---------+-----------+ +-------------------+
| | |
| | +--------------------+ |
| | | System Under Test | |
| | | | |
| | | +-----------------+| |
| +--+--------+ | | VNF || |
| | | | | || |
| | | | | +----+ +----+ || |
| | <===> |VNFC|...|VNFC| || |
| | | | | +----+ +----+ || |
| | Monitor(s)| | +----.---------.--+| |
+-----+---+ |{listeners}| | : : | +-----+----+
| Agent(s)| | | | +----^---------V--+| | Agent(s)|
|(Sender) | | <===> Execution || |(Receiver)|
| | | | | | Environment || | |
|{Probers}| +-----------+ | | || |{Probers} |
+-----.---+ | +----.---------.--+| +-----.----+
: +------^---------V---+ :
V : : :
:.................>.........: :........>..:
Stimulus Traffic Flow
A scenario, as well referred as a benchmarking setup, consists of the actual instantiation of physical and/or virtual components of a "VNF Benchmarking Architectural Framework" needed to habilitate the execution of an automated VNF benchmarking methodology. The following considerations hold for a scenario:¶
In general, an automated benchmarking methodology must execute Tests repeatedly so it must capture the relevant causes of the performance variability of a VNF. To dissect a VNF benchmarking Test, in the sections that follow a set of benchmarking phases are categorized defining generic operations that may be automated. When executing an automated VNF benchmarking methodology, all the influencing aspects on the performance of a VNF must be carefully analyzed and comprehensively reported in each automated phase of a benchmarking Test.¶
The placement (i.e., assignment and allocation of resources) and the interconnection, physical and/or virtual, of network function(s) and benchmarking components can be realized by orchestration platforms (e.g., OpenStack, Kubernetes, Open Source MANO). In automated manners, the realization of a benchmarking scenario through those means usually rely on network service templates (e.g., TOSCA, YANG, Heat, and Helm Charts). Such descriptors have to capture all relevant details of the execution environment to allow the benchmarking framework to correctly instantiate the SUT as well as helper functions required for a Test.¶
The configuration of benchmarking components and VNFs (e.g., populate routing table, load PCAP source files in source of traffic stimulus) to execute the Test settings can be realized by programming interfaces in an automated way. In the scope of NFV, there might exist management interfaces to control a VNF during a benchmarking Test. Likewise, infrastructure or orchestration components can establish the proper configuration of an execution environment to realize all the capabilities enabling the description of the benchmarking Test. Each configuration registry, its deployment timestamp and target, must all be contained in the report of a VNF benchmarking Test.¶
In the execution of a benchmarking Test, the VNF configuration can be programmed to be changed by itself or by a VNF management platform. It means that during a Trial execution, particular behaviors of a VNF can be automatically triggered, e.g., auto-scaling of its internal components. Those must be captured in the detailed procedures of the VNF execution and its performance report. I.e., the execution of a Trial can determine arrangements of internal states inside a VNF, which can interfere in observed benchmarking metrics. For instance, in a particular benchmarking case where the monitoring measurements of the VNF and/or execution environment are available for extraction, comparison Tests must be run to verify if the monitoring of the VNF and/or execution environment can impact the VNF performance metrics.¶
The result of a VNF benchmarking Test might contain generic metrics (e.g., CPU and memory consumption) and VNF-specific traffic processing metrics (e.g., transactions or throughput), which can be stored and processed in generic or specific ways (e.g., by statistics or machine learning algorithms). More details about possible metrics and the corresponding capturing methods can be found in [ETS19g]. If automated procedures are applied over the generation of a benchmarking Test result, those must be explained in the result itself, jointly with their input raw measurements and output processed data. For instance, any algorithm used in the generation of processed metrics must be disclosed in the Test result.¶
The execution of an automated benchmarking methodology consists in elaborating a VNF Benchmarking Report, its inputs and outputs. The inputs part of a VNF-BR must be written by a VNF benchmarking tester. When the VNF-BR, with its inputs fulfilled, is requested from the Manager component of a implementation of the "VNF Benchmarking Architectural Framework", the Manager must utilize the inputs part to obtain the outputs part of the VNF-BR, addressing the execution of the automated benchmarking methodology as defined in Section 5.4.¶
The flow of information in the execution of an automated benchmarking methodology can be represented by the YANG modules defined by this document. The sections that follow present an overview of such modules.¶
VNF Benchmarking Descriptor (VNF-BD) -- an artifact that specifies how to realize the Test(s) and Trial(s) of an automated VNF benchmarking methodology in order to obtain a VNF Performance Profile. The specification includes structural and functional instructions and variable parameters at different abstraction levels, such as the topology of the benchmarking scenario, and the execution parameters of prober(s)/listener(s) in the required Agent(s)/Monitor(s). A VNF-BD may be specific to a VNF or applicable to several VNF types.¶
More specifically, a VNF-BD is defined by a scenario and its proceedings. The scenario defines nodes (i.e., benchmarking components) and links interconnecting them, a topology that must be instantiated in order to execute the VNF-BD proceedings. The proceedings contain the specification of the required Agent(s) and Monitor(s) needed in the scenario nodes. Detailed in each Agent/Monitor follows the specification of the Prober(s)/Listener(s) required for the execution of the Tests, and in the details of each Prober/Listener follows the specification of its execution parameters. In the header of a VNF-BD is specified the number of Tests and Trials that a Manager must run them. Each Test realizes a unique instantiation of the scenario, while each Trial realizes a unique execution of the proceedings in the instantiated scenario of a Test. The VNF-BD YANG module is presented in Section 10.1.¶
VNF Performance Profile (VNF-PP) -- an output artifact of a VNF-BD execution performed by a Manager component. It contains all the metrics from Monitor(s) and/or Agent(s) components after realizing the execution of the Prober(s) and/or the Listener(s) proceedings, specified in its corresponding VNF-BD. Metrics are logically grouped according to the execution of the Trial(s) and Test(s) defined by a VNF-BD. A VNF-PP is specifically associated with a unique VNF-BD.¶
More specifically, a VNF-PP is defined by a structure that allows benchmarking results to be presented in a logical and unified format. A VNF-PP report is the result of an unique Test, while its content, the so called snapshot(s), each containing the results of the execution of a single Trial. Each snapshot is built by a single Agent or Monitor. A snapshot contains evaluation(s), each one being the output of the execution of a single Prober or Listener. An evaluation contains one or more metrics. In summary, a VNF-PP aggregates the results from reports (i.e., the Test(s)); a report aggregates Agent(s) and Monitor(s) results (i.e., the Trial(s)); a snapshot aggregates Prober(s) or Listener(s) results; and an evaluation aggregates metrics. The VNF-PP YANG module is presented in Section 10.2.¶
VNF Benchmarking Report (VNF-BR) -- the core artifact of an automated VNF benchmarking methodology consisted of three parts: a header, inputs and output. The header refers to the VNF-BR description items (e.g., author, version, name), the description of the target SUT (e.g., the VNF version, release, name), and the environment settings specifying the parameters needed to instantiate the benchmarking scenario via an orchestration platform. The inputs contain the definitions needed to execute the automated benchmarking methodology of the target SUT, a VNF-BD and its variables settings. The outputs contain the results of the execution of the inputs, a list of entries, each one containing a VNF-BD filled with one of the combinations of the input variables settings, and the obtained VNF-PP reported after the execution of the Test(s) and Trial(s) of the parsed VNF-BD. The process of utilizing the VNF-BR inputs to generate its outputs concerns the realization of an automated VNF benchmarking methodology, explained in details in Section 5.4.2. The VNF-BR YANG module is presented in Section 10.3.¶
In details, each one of the variables in the inputs part of a VNF-BR is defined by: a name (the actual name of the variable); a path (the YANG path of the variable in the input VNF-BD); a type (the type of the values, such as string, int, float, etc); class (one of: stimulus, resource, configuration); and values (a list of the variable actual values). The values of all the variables must be combined all-by-all, generating a list containing the whole sample space of variables settings that must be used to create the VNF-BD instances. A VNF-BD instance is defined as the result of the parsing of one of those combinations of input variables into the VNF-BD of the VNF-BR inputs. The parsing takes place when the variable path is utilized to set its value in the VNF-BD. Interatively, all the VNF-BD instances must have its Test(s) and Trial(s) executed to generate its corresponding VNF-PP. After all the VNF-BD instances had their VNF-PP accomplished, the realization of the whole automated VNF benchmarking methodology is complete, fulfilling the outputs part of the VNF-BR as shown in Figure 2.¶
+------------+ +-----------------+ +------------+
| | | | | |
| VNF-BR | | Execution of | | VNF-BR |
| (Inputs) +----------->+ the Automated +--------->+ (Inputs) |
| | | Benchmarking | | (Outputs) |
+------------+ | Methodology | | |
| | +------------+
+-----------------+
The methodology for VNF benchmarking automation encompasses the process defined in Figure 2, i.e., the procedures that utilize the inputs part to obtain the outputs part of a VNF-BR. This section details the procedures that realize such process.¶
The plan of an automated VNF benchmarking methodology consists in the definition of all the header and the inputs part of a VNF-BR, the artifacts to be utilized by the realization of the methodology, and the establishment of the execution environment where the methodology takes place. The topics below contain the details of such planning.¶
Accomplished all the planning procedures, the process of the realization of the automated benchmarking methodology must be realized as the following topics describe.¶
After the realization of an automated benchmarking methodology, some automated procedures can be performed to improve the quality and the utility of the obtained VNF-BR, as described in the following topics.¶
As described in [RFC8172], VNF benchmarking might require to change and adapt existing benchmarking methodologies. More specifically, the following cases need to be considered.¶
VNFs are usually deployed inside containers or VMs to build an abstraction layer between physical resources and the resources available to the VNF. According to [RFC8172], it may be more representative to design experiments in a way that the VMs hosting the VNFs are operating at maximum of 50% utilization and split the workload among several VMs, to mitigateside effects of overloaded VMs. Those cases are supported by the presented automation methodologies through VNF-BDs that enable direct control over the resource assignments and topology layouts used for a benchmarking experiment.¶
As a VNF might be composed of multiple components (VNFCs), there exist different schemas of redundancy where particular VNFCs would be in active or standby mode. For such cases, particular monitoring endpoints should be specified in VNF-BD so listeners can capture the relevant aspects of benchmarking when VNFCs would be in active/standby modes. In this particular case, capturing the relevant aspects of internal functionalities of a VNF and its internal components provides important measurements to characterize the dynamics of a VNF, those must be reflected in its VNF-PP.¶
One of the main challenges of NFV is to create isolation between VNFs. Benchmarking the quality of this isolation behavior can be achieved by Agents that take the role of a noisy neighbor, generating a particular workload in synchrony with a benchmarking procedure over a VNF. Adjustments of the Agent's noisy workload, frequency, virtualization level, among others, must be detailed in the VNF- BD.¶
Hardware and software components will fail or have errors and thus trigger healing actions of the benchmarked VNFs (self-healing). Benchmarking procedures must also capture the dynamics of this VNF behavior, e.g., if a container or VM restarts because the VNF software crashed. This results in offline periods that must be captured in the benchmarking reports, introducing additional metrics, e.g., max. time-to-heal. The presented concept, with a flexible VNF-PP structure to record arbitrary metrics, enables automation of this case.¶
Having software based network functions and the possibility of a VNF to be composed by multiple components (VNFCs), internal events of the VNF might trigger changes in VNF behavior, e.g.,activating functionalities associated with elasticity such as automated scaling. These state changes and triggers (e.g. the VNF's scaling state) must be captured in the benchmarking results (VNF-PP) to provide a detailed characterization of the VNF's performance behavior in different states.¶
As described in [RFC8172], does the sheer number of test conditions and configuration combinations create a challenge for VNF benchmarking. As suggested, machine readable output formats, as they are presented in this document, will allow automated benchmarking procedures to optimize the tested configurations. Approaches for this are, e.g., machine learning-based configuration space sub-sampling methods, such as [Peu-c].¶
A benchmarking setup must be able to define scenarios with and without monitoring components inside the VNFs and/or the hosting container or VM. If no monitoring solution is available from within the VNFs, the benchmark is following the black-box concept. If, in contrast, those additional sources of information from within the VNF are available, VNF-PPs must be able to handle these additional VNF performance metrics.¶
Currently, technical motivating factors in favor of the automation of VNF benchmarking methodologies comprise: (i) the facility to run high-fidelity and commodity traffic generators by software; (ii) the existent means to construct synthetic traffic workloads purely by software (e.g., handcrafted pcap files); (iii) the increasing availability of datasets containing actual sources of production traffic able to be reproduced in benchmarking tests; (iv) the existence of a myriad of automating tools and open interfaces to programmatically manage VNFs; (v) the varied set of orchestration platforms enabling the allocation of resources and instantition of VNFs through automated machineries based on well-defined templates; (vi) the ability to utilize a large tool set of software components to compose pipelines that mathematically analyze benchmarking metrics in automated ways.¶
In simple terms, the enlisted factors above justify that network softwarization enables the automation of VNF benchmarking methodologies. There exists an open source reference implementation that is built to demonstrate the concepts and methodology of this document in order to automate the benchmarking of Virtualized Network Functions.¶
The software, named Gym, is a framework for automated benchmarking of Virtualized Network Functions (VNFs). It was coded following the initial ideas presented in a 2015 scientific paper entitled "VBaaS: VNF Benchmark-as-a-Service" [Rosa-a]. Later, the evolved design and prototyping ideas were presented at IETF/IRTF meetings seeking impact into NFVRG and BMWG.¶
Gym was built to receive high-level test descriptors and execute them to extract VNFs profiles, containing measurements of performance metrics - especially to associate resources allocation (e.g., vCPU) with packet processing metrics (e.g., throughput) of VNFs. From the original research ideas [Rosa-a], such output profiles might be used by orchestrator functions to perform VNF lifecycle tasks (e.g., deployment, maintenance, tear-down).¶
In [Rosa-b] Gym was utilized to benchmark a decomposed IP Multimedia Subsystem VNF. And in [Rosa-c], a virtual switch (Open vSwitch - OVS) was the target VNF of Gym for the analysis of VNF benchmarking automation. Such articles validated Gym as a prominent open source reference implementation for VNF benchmarking tests. Such articles set important contributions as discussion of the lessons learned and the overall NFV performance testing landscape, included automation.¶
Gym stands as one open source reference implementation that realizes the VNF benchmarking methodologies presented in this document. Gym is released as open source tool under Apache 2.0 license [gym].¶
Another software that focuses on implementing a framework to benchmark VNFs is the "5GTANGO VNF/NS Benchmarking Framework" also called "tng-bench" (previously "son-profile") and was developed as part of the two European Union H2020 projects SONATA NFV and 5GTANGO [tango]. Its initial ideas were presented in [Peu-a] and the system design of the end-to-end prototype was presented in [Peu-b].¶
Tng-bench aims to be a framework for the end-to-end automation of VNF benchmarking processes. Its goal is to automate the benchmarking process in such a way that VNF-PPs can be generated without further human interaction. This enables the integration of VNF benchmarking into continuous integration and continuous delivery (CI/CD) pipelines so that new VNF-PPs are generated on-the-fly for every new software version of a VNF. Those automatically generated VNF-PPs can then be bundled with the VNFs and serve as inputs for orchestration systems, fitting to the original research ideas presented in [Rosa-a] and [Peu-a].¶
Following the same high-level VNF testing purposes as Gym, namely: Comparability, repeatability, configurability, and interoperability, tng- bench specifically aims to explore description approaches for VNF benchmarking experiments. In [Peu-b] a prototype specification for VNF-BDs is presented which not only allows to specify generic, abstract VNF benchmarking experiments, it also allows to describe sets of parameter configurations to be tested during the benchmarking process, allowing the system to automatically execute complex parameter studies on the SUT, e.g., testing a VNF's performance under different CPU, memory, or software configurations.¶
Tng-bench was used to perform a set of initial benchmarking experiments using different VNFs, like a Squid proxy, an Nginx load balancer, and a Socat TCP relay in [Peu-b]. Those VNFs have not only been benchmarked in isolation, but also in combined setups in which up to three VNFs were chained one after each other. These experiments were used to test tng-bench for scenarios in which composed VNFs, consisting of multiple VNF components (VNFCs), have to be benchmarked. The presented results highlight the need to benchmark composed VNFs in end-to-end scenarios rather than only benchmark each individual component in isolation, to produce meaningful VNF- PPs for the complete VNF.¶
Tng-bench is actively developed and released as open source tool under Apache 2.0 license [tng-bench]. A larger set of example benchmarking results of various VNFs is available in [Peu-d].¶
Benchmarking tests described in this document are limited to the performance characterization of VNFs in a lab environment with isolated network.¶
The benchmarking network topology will be an independent test setup and MUST NOT be connected to devices that may forward the test traffic into a production network, or misroute traffic to the test management network.¶
Special capabilities SHOULD NOT exist in the VNF benchmarking deployment scenario specifically for benchmarking purposes. Any implications for network security arising from the VNF benchmarking deployment scenario SHOULD be identical in the lab and in production networks.¶
This document registers one URI in the "ns" subregistry of the IETF XML Registry [RFC3688]. Following the format in [RFC3688], the following registrations are requested:¶
URI: urn:ietf:params:xml:ns:yang:ietf-vnf-bd
Registrant Contact: The BMWG of the IETF.
XML: N/A, the requested URI is an XML namespace.
URI: urn:ietf:params:xml:ns:yang:ietf-vnf-pp
Registrant Contact: The BMWG of the IETF.
XML: N/A, the requested URI is an XML namespace.
URI: urn:ietf:params:xml:ns:yang:ietf-vnf-br
Registrant Contact: The BMWG of the IETF.
XML: N/A, the requested URI is an XML namespace.
This document registers three YANG modules in the YANG Module Names registry [RFC6020]. Following the format in [RFC6020], the following registration is requested:¶
name: ietf-vnf-bd
namespace: urn:ietf:params:xml:ns:yang:ietf-vnf-bd
prefix: vnf-bd
reference: RFC CCCC
name: ietf-vnf-pp
namespace: urn:ietf:params:xml:ns:yang:ietf-vnf-pp
prefix: vnf-pp
reference: RFC CCCC
name: ietf-vnf-br
namespace: urn:ietf:params:xml:ns:yang:ietf-vnf-br
prefix: vnf-br
reference: RFC CCCC
The following sections contain the YANG modules defined by this document.¶
module vnf-bd {
namespace "urn:ietf:params:xml:ns:yang:vnf-bd";
prefix "vnf-bd";
organization "IETF/BMWG";
contact "Raphael Vicente Rosa <raphaelvrosa@gmail.com>,
Manuel Peuster <peuster@mail.uni-paderborn.de>";
description "Yang module for a VNF Benchmarking
Descriptor (VNF-BD).";
revision "2019-08-13" {
description "V0.3: Reviewed proceedings,
tool - not VNF specific";
reference "";
}
revision "2019-03-13" {
description "V0.2: Reviewed role, policies, connection-points,
lifecycle workflows, resources";
reference "";
}
revision "2019-02-28" {
description "V0.1: First release";
reference "";
}
typedef workflows {
type enumeration {
enum create {
description "When calling the create workflow.";
}
enum configure {
description "When calling the configure workflow.";
}
enum start {
description "When calling the start workflow.";
}
enum stop {
description "When calling the stop workflow.";
}
enum delete {
description "When calling the delete workflow.";
}
enum custom {
description "When calling a custom workflow.";
}
}
description "Defines basic life cycle workflows for a
node in a scenario.";
}
grouping node_requirements {
container resources {
container cpu {
leaf vcpus {
type uint32;
description "The number of cores to be allocated
for a node.";
}
leaf cpu_bw {
type string;
description "The CPU bandwidth (CFS limit in 0.01-1.0)";
}
leaf pinning {
type string;
description "The list of CPU cores, separated by comma,
that a node must be pinned to.";
}
description "The node CPU resources that must
be allocated for a benchmarking Test.";
}
container memory {
leaf size {
type uint32;
description "The memory allocation size.";
}
leaf unit {
type string;
description "The memory unit.";
}
description "The node memory resources
that must be allocated for a benchmarking
Test.";
}
container storage {
leaf size {
type uint32;
description "The storage allocation size.";
}
leaf unit {
type string;
description "The storage unit.";
}
leaf volumes {
type string;
description "Volumes to be allocated by
a node storage.
A volume defines a mapping of an outside storage
partition inside the node storage system.
Volumes must be separated by comma and be defined
using a colon to separate the node internal and external
references of storage system paths.";
}
description "The node storage resources
that must be allocated for a benchmarking Test.";
}
description "The set of resources that must be allocated
for a node in a benchmarking Test.";
}
description "'The grouping determining the
resource requirements for a node in a scenario.";
}
grouping connection_points {
leaf id {
type string;
description "The connection-point
unique identifier";
}
leaf interface {
type string;
description "The name of the node interface
associated with the connection-point.";
}
leaf type {
type string;
description "The type of the network the
connection-point interface is attached to.";
}
leaf address {
type string;
description "The Network address of the
connection-point. It can be specified as a
Ethernet MAC address, a IPv4 address or an IPv6 address.";
}
description "A connections-point of a node.";
}
grouping nodes {
leaf id {
type string;
description "The unique identifier of a node
in a scenario.";
}
leaf type {
type string;
description "The type of a node.";
}
leaf image {
type string;
description "The name of the image to be used to instantiate
a node.";
}
leaf format {
type string;
description "The node format (e.g., container, process, VM).";
}
leaf role {
type string;
description "The role of the node in the Test scenario.
The role must be one of: manager, agent, monitor, sut.";
}
uses node_requirements;
list connection_points {
key "id";
uses connection_points;
description "The list of connection points of a node.";
}
list relationships {
key "name";
leaf name {
type string;
description "Name of the relationship.";
}
leaf type {
type string;
description "Type of the relationship.";
}
leaf target {
type string;
description "Target of the relationship.";
}
description "Relationship of a node with the other
scenario components.";
}
list lifecycle {
key "workflow";
leaf workflow {
type workflows;
description "The type of the Workflow.";
}
leaf name {
type string;
description "The workflow name.";
}
list parameters {
key "input";
leaf input {
type string;
description "The name of the parameter.";
}
leaf value {
type string;
description "The value of the parameter";
}
description "The list of parameters to be
applied to the node workflow.";
}
leaf-list implementation {
type string;
description "The workflow implementation.";
}
description "The life cycle workflows to be
applied to this node.";
}
description "The specification of a node to be used
in a scenario for a benchmarking Test.";
}
grouping link {
leaf id {
type string;
description "The link unique identifier.";
}
leaf name {
type string;
description "The name of the link.";
}
leaf type {
type string;
description "The type of the link.";
}
leaf network {
type string;
description "The network the link belongs to.";
}
leaf-list connection_points {
type leafref {
path "../../nodes/connection_points/id";
}
description "Reference to the connection points of nodes
the link is adjacent.";
}
description "A link between nodes in a scenario.";
}
grouping scenario {
list nodes {
key "id";
uses nodes;
description "The list of nodes that must be
instantiated in a scenario in order to enable
a benchmarking Test.";
}
list links {
key "id";
uses link;
description "The list of links among nodes that must be
instantiated in a scenario in order to enable
a benchmarking Test.";
}
list policies {
key "name";
leaf name {
type string;
description "The name of the policy.";
}
leaf type {
type string;
description "The type of the policy";
}
leaf targets {
type string;
description "The targets of the policy.
Uuid of nodes and/or links separated by comma.";
}
leaf action {
type string;
description "The action of the policy";
}
description "Definition of policies to be
utilized on the instantiation of the scenario.
A policy is defined by a name, it type,
the targets (nodes and/or links) to which it must
be applied to, and the proper action that
realizes the policy.";
}
description "Describes the deployment of all
involved functional components mandatory for
the execution of a benchmarking Test.";
}
grouping tool {
leaf id {
type uint32;
description "The unique identifier of a tool.
This information specifies how a tool can be
identified in a list of probers/listeners of an
Agent/Monitor.";
}
leaf instances {
type uint32;
description "The number of the tool instances that
must be executed in parallel.";
}
leaf name {
type string;
description "The name of a tool.";
}
list parameters {
key "input";
leaf input {
type string;
description "The input key of a parameter";
}
leaf value {
type string;
description "The value of a parameter";
}
description "List of parameters for the execution
of the tool. Each tool detains the proper set of running
parameters that must be utilized to realize a benchmarking
test.";
}
container sched {
leaf from {
type uint32;
default 0;
description "The initial time (in seconds)
of the execution of the tool.";
}
leaf until {
type uint32;
description "The final/maximum time (in seconds)
of the execution of the tool summed all its instances
repeat, duration and interval parameters.";
}
leaf duration {
type uint32;
description "The total duration (in seconds) of the execution
of each instance of the tool.";
}
leaf interval {
type uint32;
description "The interval (in seconds) to be awaited
among each one of the instances of the
execution of the tool.";
}
leaf repeat {
type uint32;
description "The number of times the tool must be executed.";
}
description "The scheduling parameters of a tool.
Each Agent/Monitor must utilize the scheduling parameters
to perform the execution of its tools (probers/listeners)
accordingly.";
}
description "A tool to be used in a benchmarking test.
A tool can be inferred as a prober or a listener.";
}
grouping component {
leaf uuid {
type string;
description "A unique identifier";
}
leaf name {
type string;
description "The name of component";
}
description "A generic component.";
}
grouping agent {
uses component;
list probers {
key "id";
uses tool;
description "Defines a list of the Prober(s)
that must be used in a benchmarking test.";
}
description "An Agent defined by its uuid,
name and the mandatory list of probers to be used
by a benchmarking test.";
}
grouping monitor {
uses component;
list listeners {
key "id";
uses tool;
description "Defines a list of the Listeners(s)
that must used in a benchmarking test.";
}
description "A Monitor defined by its uuid,
name and the mandatory list of probers to be used
by a benchmarking test.";
}
grouping proceedings {
list agents {
key "uuid";
uses agent;
description "Defines a list containing the
Agent(s) needed for a VNF-BD test.";
}
list monitors {
key "uuid";
uses monitor;
description "Defines a list containing the
Monitor(s) needed for a VNF-BD test.";
}
description "Information utilized by a Manager
component to execute a benchmarking test.";
}
grouping vnf-bd {
container experiments {
leaf trials {
type uint32;
default 1;
description "Number of trials.
A trial is a single process or iteration
to obtain VNF performance metrics from
benchmarking the VNF-BD proceedings.";
}
leaf tests {
type uint32;
default 1;
description "Number of tests.
Each test defines unique structural
and functional parameters (e.g., configurations,
resource assignment) for benchmarked components
to perform one or multiple Trials.
Each Test must be executed following a
particular scenario.";
}
description "Defines the number of trials and tests
the VNF-BD must execute.";
}
container scenario {
uses scenario;
description "Scenarios defined by this VNF-BD.
A scenario contains all information needed to describe
the deployment of all involved functional components
mandatory for the execution of a benchmarking Test.";
}
container proceedings {
uses proceedings;
description "Proceedings of VNF-BD.
The proceedings are utilized by the Manager component
to execute a benchmarking Test. It consists of
agent(s)/monitor(s) settings, detailing their
prober(s)/listener(s) specification and
running parameters.";
}
description "A single VNF-BD.
A VNF-BD contains all required definitions and
requirements to deploy, configure, execute, and
reproduce VNF benchmarking tests.";
}
uses vnf-bd;
}
module vnf-pp {
namespace "urn:ietf:params:xml:ns:yang:vnf-pp";
prefix "vnf-pp";
organization "IETF/BMWG";
contact "Raphael Vicente Rosa <raphaelvrosa@gmail.com>,
Manuel Peuster <peuster@mail.uni-paderborn.de>";
description "Yang module for a VNF Performance Profile (VNF-PP).";
revision "2019-10-15" {
description "Reviewed VNF-PP structure -
defines reports, snapshots, evaluations";
reference "";
}
revision "2019-08-13" {
description "V0.1: First release";
reference "";
}
grouping tuple {
description "A tuple used as key-value.";
leaf key {
type string;
description "Tuple key.";
}
leaf value {
type string;
description "Tuple value.";
}
}
grouping metric {
leaf name {
type string;
description "The metric name";
}
leaf unit {
type string;
description "The unit of the metric value(s).";
}
leaf type {
type string;
mandatory true;
description "The data type encoded in the value.
It must refer to a known variable type, i.e.,
string, float, uint, etc.";
}
choice value {
case scalar {
leaf scalar {
type string;
mandatory true;
description "A single scalar value.";
}
}
case vector {
leaf-list vector {
type string;
min-elements 1;
description "A list of scalar values";
}
}
case series {
list series {
key "key";
uses tuple;
description "A list of key/values,
e.g., a timeseries.";
}
}
mandatory true;
description "Value choice: scalar, vector, series.
A metric can only contain a value with one of them.";
}
description "A metric that holds the recorded benchmarking
results, can be a single value (scalar), a list of values
(vector), or a list of key/value
data (series), e.g., for timeseries.";
}
grouping evaluation {
leaf id {
type string;
description "The evaluation
unique identifier.";
}
leaf instance {
type uint32;
description "The unique identifier of the
parallel instance of the prober/listener that
was executed and created the evaluation.";
}
leaf repeat {
type uint32;
description "The unique identifier of the
prober/listener repeatition instance
was executed and created the evaluation.";
}
container source {
leaf id {
type string;
description "The unique identifier of the source
of the evaluation,
i.e., the prober/listener unique identifier.";
}
leaf name {
type string;
description "The name of the source of the evaluation,
i.e., the prober/listener name.";
}
leaf type {
type string;
description "The type of the source of the evaluation,
i.e., one of prober or listener, that was used to obtain
it.";
}
leaf version {
type string;
description "The version of the tool interfacing
the prober/listener that was used to obtain
the evaluation.";
}
leaf call {
type string;
description "The full call of the tool realized by
the source of the evaluation that performed
the acquisiton of the metrics.";
}
description "The details regarding the
source of the evaluation.";
}
container timestamp {
leaf start {
type string;
description "Time (date, hour, minute, second)
when the evaluation started";
}
leaf stop {
type string;
description "Time (date, hour, minute, second)
when the evaluation stopped";
}
description "Timestamps of the procedures
that realized the extraction of the evaluation.";
}
list metrics {
key "name";
uses metric;
description "List of metrics obtained
from a single evaluation.";
}
leaf error {
type string;
description "Error, if existent,
when obtaining evaluation.";
}
description "The set of metrics and their source
associated with a single Trial.";
}
grouping snapshot {
leaf id {
type string;
description "The snapshot
unique identifier.";
}
leaf trial {
type uint32;
description "The identifier of the trial
when the snapshot was obtained.";
}
container origin {
leaf id {
type string;
description "The unique identifier of the
component of the origin of the snapshot,
i.e., the agent or monitor unique identifier.";
}
leaf role {
type string;
description "The role of the component,
origin of the snapshop, i.e.,
one of agent or monitor.";
}
leaf host {
type string;
description "The hostname where the
source of the snapshot was placed.";
}
description "The detailed origin of
the snapshot.";
}
list evaluations {
key "id";
uses evaluation;
description "The list of evaluations
contained in a single snapshot Test.";
}
leaf timestamp {
type string;
description "Time (date, hour, minute, second)
when the snapshot was created.";
}
leaf error {
type string;
description "Error, if existent,
when obtaining the snapshot.";
}
description "The set of evaluations and their origin
output of the execution of a single trial.";
}
grouping report {
leaf id {
type string;
description "The report unique identifier.";
}
leaf test {
type uint32;
description "The identifier of the Test
when the snapshots were obtained.";
}
list snapshots {
key "id";
uses snapshot;
description "List of snapshots contained
in a single report.";
}
leaf timestamp {
type string;
description "Time (date, hour, minute, second)
when the report was created.";
}
leaf error {
type string;
description "Error, if existent,
when obtaining the report.";
}
description "The set of snapshots output
of a single Test.";
}
grouping header {
leaf id {
type string;
description "Unique identifier of the VNF-PP.";
}
leaf name {
type string;
description "Name of the VNF-PP.";
}
leaf version {
type string;
description "Version of the VNF-PP.";
}
leaf description {
type string;
description "Description of the VNF-PP";
}
leaf timestamp {
type string;
description "Time (date, hour, minute, second)
when the VNF-PP was created.";
}
description "The header content of a VNF-PP.";
}
grouping vnf-pp {
uses header;
list reports {
key "id";
uses report;
description "List of the reports of a VNF-PP.";
}
description "A single VNF-PP.";
}
uses vnf-pp;
}
module vnf-br {
namespace "urn:ietf:params:xml:ns:yang:vnf-br";
prefix "vnf-br";
import vnf-bd {
prefix "vnfbd";
revision-date 2020-10-08;
}
import vnf-pp {
prefix "vnfpp";
revision-date 2020-10-08;
}
organization "IETF/BMWG";
contact "Raphael Vicente Rosa <raphaelvrosa@gmail.com>,
Manuel Peuster <peuster@mail.uni-paderborn.de>";
description "Yang model for a VNF Benchmark Report (VNF-BR).";
revision "2020-09-09" {
description "V0.2: Review the structure
and the grouping/leaf descriptions.";
reference "";
}
revision "2020-09-09" {
description "V0.1: First release";
reference "";
}
grouping variable {
leaf name {
type string;
description "The name of the variable.";
}
leaf path {
type string;
description "The VNF-BD YANG path of the
variable.";
}
leaf type {
type string;
description "The type of the
variable values.";
}
leaf class {
type string;
description "The class of the
variable (one of resource, stimulus,
configuration).";
}
leaf-list values {
type string;
description "The list of values
of the variable.";
}
}
grouping output {
leaf id {
type string;
description "The output unique identifier.";
}
list variables {
key "name";
leaf name { type string; }
leaf value { type string; }
description "The list of instance of varibles
from VNF-BR:inputs utilized by a VNF-BD to
generate a VNF-PP.";
}
container vnfbd {
uses vnfbd:vnf-bd;
description "The VNF-BD that was executed
to generate a output.";
}
container vnfpp {
uses vnfpp:vnf-pp;
description "The output VNF-PP of the
execution of a VNF-BD.";
}
}
grouping vnf {
leaf id {
type string;
description "The VNF unique identifier.";
}
leaf name {
type string;
description "The VNF name.";
}
leaf version {
type string;
description "The VNF version.";
}
leaf author {
type string;
description "The author of the VNF.";
}
leaf description {
type string;
description "The description of the VNF.";
}
description "The details of the VNF SUT.";
}
grouping header {
leaf id {
type string;
description "The unique identifier of the VNF-BR ";
}
leaf name {
type string;
description "The name of the VNF-BR.";
}
leaf version {
type string;
description "The VNF-BR version.";
}
leaf author {
type string;
description "The VNF-BR author.";
}
leaf description {
type string;
description "The description of the VNF-BR.";
}
container vnf {
uses vnf;
description "The VNF-BR target SUT VNF.";
}
container environment {
leaf name {
type string;
description "The evironment name";
}
leaf description {
type string;
description "A description
of the environment";
}
leaf deploy {
type boolean;
description "Defines if (True) the environment enables
the automated deployment by an orchestrator platform.";
}
container orchestrator {
leaf name {
type string;
description "Name of the orchestrator
platform.";
}
leaf type {
type string;
description "The type of the orchestrator
platform.";
}
leaf description {
type string;
description "The description of the
orchestrator platform.";
}
list parameters {
key "input";
leaf input {
type string;
description "The name of the parameter";
}
leaf value {
type string;
description "The value of the parameter";
}
description "List of orchestrator
input parameters.";
}
description "The specification of the orchestration platform
settings of a VNF-BR.";
}
description "The environment settings of a VNF-BR.";
}
description "Defines the content of a VNF-BR header.";
}
grouping vnf-br {
description "Grouping for a single vnf-br.";
uses header;
container inputs {
list variables {
key "name";
uses variable;
description "The list of
input variables.";
}
container vnfbd {
uses vnfbd:vnf-bd;
description "The input VNF-BD.";
}
description "The inputs needed to
realize a VNF-BR.";
}
list outputs {
key "id";
uses output;
description "The list of outputs
of a VNF-BR.";
}
container timestamp {
leaf start {
type string;
description "Time (date, hour, minute, second)
of when the VNF-BR realization started";
}
leaf stop {
type string;
description "Time (date, hour, minute, second)
of when the VNF-BR realization stopped";
}
description "Timestamps of the procedures that
realized the realization of a VNF-BR.";
}
leaf error {
type string;
description "The VNF-BR error,
if ocurred during its realization.";
}
}
uses vnf-br;
}
The authors would like to thank the support of Ericsson Research, Brazil. Parts of this work have received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No. H2020-ICT-2016-2 761493 (5GTANGO: https://5gtango.eu).¶