nfvrg R. Szabo, Ed.
Internet-Draft Ericsson
Intended status: Informational S. Lee, Ed.
Expires: May 3, 2017 ETRI
N. Figueira
Brocade
October 30, 2016

Policy-Based Resource Management
draft-irtf-nfvrg-policy-based-resource-management-02

Abstract

abstract to be defined

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at http://datatracker.ietf.org/drafts/current/.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."

This Internet-Draft will expire on May 3, 2017.

Copyright Notice

Copyright (c) 2016 IETF Trust and the persons identified as the document authors. All rights reserved.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.


Table of Contents

1. Introduction

NFV "Point of Presence" (PoP) will be likely constrained in compute and storage capacity. Since practically all NFV PoPs are foreseen to be distributed, inter-datacenter network capacity is also a constraint. Additionally, energy is also a constraint, both as a general concern for NFV operators, and in particular for specific-purpose NFV PoPs such as those in mobile base stations. This draft focuses on the optimized resource management and workload distribution based on policy to address such contraints.

1.1. Scope

For the first version of the draft, only the research group currently adopted drafts (i.e., [I-D.norival-nfvrg-nfv-policy-arch], [I-D.irtf-nfvrg-resource-management-service-chain], and [I-D.unify-nfvrg-recursive-programming]) are considered as inputs to this document. The initial goal is to summarize these inputs and to assess gaps and open questions.

2. Terminology

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119].

3. Definitions

This document uses the terms of [ETSI-NFV-TERM]:

Additionally, we use the following terms:

4. Requirements

tbd

5. Architecture Considerations

5.1. MANO Architecture

According to the ETSI MANO framework [ETSI-NFV-MANO], an NFVO is split into two functions (see Figure 1):

Similarly, a VIM is split into two functions (see Figure 1):

 
                       +-------------------+
                       |NVFO               |
                       |  +--------------+ |
                       |  |NFVO:         | |
                       |  |Service       | |
                       |  |Lifecycle     | |
                       |  |Management    | |
                       |  +------+-------+ |
                       |         |         |
                       |  +------+-------+ |
                       |  |NFVO:         | |
                       |  |Resource      | |
                       |  |Orchestration | |
                       |  +--+---+----+--+ |
                       +-----|---|----|----+
                            /    |     \
                 /---------/     |      \------------\
                /                |                    \
 +-------------|-----+  +--------|----------+  +------|------------+
 |VIM          |     |  |VIM     |          |  |VIM   |            |
 |  +----------+---+ |  |  +-----+--------+ |  |  +---+----------+ |
 |  |VIM:          | |  |  |VIM:          | |  |  |VIM:          | |
 |  |Orchestration | |  |  |Orchestration | |  |  |Orchestration | |
 |  |&             | |  |  |&             | |  |  |&             | |
 |  |Optimization  | |  |  |Optimization  | |  |  |Optimization  | |
 |  +------+-------+ |  |  +------+-------+ |  |  +------+-------+ |
 |         |         |  |         |         |  |         |         |
 |  +------+-------+ |  |  +------+-------+ |  |  +------+-------+ |
 |  |VIM:          | |  |  |VIM:          | |  |  |VIM:          | |
 |  |Virtualized 2 | |  |  |Virtualized 2 | |  |  |Virtualized 2 | |
 |  |Pys mapping   | |  |  |Pys mapping   | |  |  |Pys mapping   | |
 |  +--------------+ |  |  +--------------+ |  |  +--------------+ |
 +-------------------+  +-------------------+  +-------------------+

Figure 1: Functional decomposition of the NFVO and the VIM according to the ETSI MANO

In Figure 2 we show various policies mapped to the MANO architecture (see Section 5.2 for more dicussions on policies in the MANO architeture):

 
                         T1 T2...Tn
                          |  |   |
                    +-----|--|---|------+
                    |NVFO |  |   |      |            Tenant
                    |  +--+--+---+----+ |         <- Policies
                    |  |NFVO:         | |
     Operational    |  |Service       | |
     Policies->     |  |Lifecycle     | |
                    |  |Management    | |
                    |  +------+-------+ |
                    |         |         |
                    |  +------+-------+ |
                    |  |NFVO:         | |
     Operational    |  |Resource      | |
     Policies->     |  |Orchestration | |       ^
                    |  +--+---+----+--+ |       |Import
 to                 +-----|---|---------+       |Policies
 other NFVO              /    \
     \          +-------+      \
      \        /                \      to NFVO  ^
+------\------|-----+            \       /      |Export
|VIM    \     |     |             \     /       |Policies
|  +-----+----+---+ |     +--------|----|-----+
|  |VIM:          | |     |VIM     |    |     |     Tenant
|  |Orchestration | |     |  +-----+----+---+ |  <- Policies
|  |&             | |     |  |VIM:          | |
|  |Optimization  | | .   |  |Orchestration | |
|  +------+-------+ |  .  |  |&             | |  <- Operational
|         |         |     |  |Optimization  | |     Policies
|  +------+-------+ |     |  +------+-------+ |
|  |VIM:          | |     |         |         |
|  |Virtualized 2 | |     |  +------+-------+ |
|  |Pys mapping   | |     |  |VIM:          | |  <- Operational
|  +--------------+ |     |  |Virtualized 2 | |     Policies
+-------------------+     |  |Pys mapping   | |
                          |  +--------------+ |
                          +-------------------+

Figure 2: Policies within the MANO framework

5.2. Policies in the MANO Architecture

The current industry work in the area of policy for NFV is mostly considered in the framework of general cloud services, and typically focused on individual subsystems and addressing very specific use cases or environments. For example, [ETSI-NFV-WHITE-PAPER] addresses network subsystem policy for network virtualization, [ODL-GB-POLICY] and [ODL-NIC-PROJECT] are open source projects in the area of network policy as part of the OpenDaylight [ODL-SDN-CONTROLLER] software defined networking (SDN) controller framework, [RFC3060] specifies an information model for network policy, [VM-HOSTING-NET-CLUSTER] focuses on placement and migration policies for distributed virtual computing, [OPENSTACK-CONGRESS] is an open source project proposal in OpenStack [OPENSTACK] to address policy for general cloud environments.

A policy framework applicable to the MANO architure must consider NFV services from the perspective of overall orchestration requirements for services involving multiple subsystems (e.g., Figure 1 and Figure 2).

While this document discusses policy atributes as applicable to the MANO architecture, the general topic of policy has already been intensively studied and documented on numerous publications over the past 10 to 15 years (see [RFC3060], [POLICY-FRAMEWORK-WG], [RFC3670], [RFC3198], and [CERI-DATALOG] to name a few). This document's purpose is to discuss and document a policy framework applicable to the MANO architecture using known policy concepts and theories to address the unique requirements of NFV services including multiple PoPs and networks forming hierarchical domain architectures [SDN-MULTI-DOMAIN].

With the above goals, this document analyses "global versus local policies" (Section 5.3), a "hierarchical policy framework" (Section 5.4) to address the demanding and growing requirements of NFV environments, a "policy pub/sub bus in the hierarchical framework" (Section 5.5), "policy intent versus subsystem actions" (Section 5.6), "static versus dynamic versus autonomic policies" (Section 5.7), "policy conflict detection and resolution" (Section 5.8), and "soft versus hard policy constraints" (Section 5.9), which can be relevant to resource management in service chains [RESOURCE-MGMT-SERVICE-CHAIN].

5.3. Global vs Local Policies

Some policies may be subsystem specific in scope, while others may have broader scope and interact with multiple subsystems. For example, a policy constraining certain customer types (or specific customers) to only use certain server types for VNF or Virtual Machine (VM) deployment would be within the scope of the compute subsystem. A policy dictating that a given customer type (or specific customers) must be given "platinum treatment" could have different implications on different subsystems. As shown in Figure 8, that "platinum treatment" could be translated to servers of a given performance specification in a compute subsystem and storage of a given performance specification in a storage subsystem.

Policies with broader scope, or global policies, would be defined outside affected subsystems and enforced by a global policy engine (Figure 3), while subsystem-specific policies or local policies, would be defined and enforced at the local policy engines of the respective subsystems.

Examples of sub-system policies can include thresholds for utilization of sub-system resources, affinity/anti-affinity constraints with regard to utilization or mapping of sub-system resources for specific tasks, network services, or workloads, or monitoring constraints regarding under-utilization or over- utilization of sub-system resources.

+----------------------------------------------------------------+
|        +----------------------------------------------+        |
|        |             Global Policy Engine             |        |
|        +----------------------------------------------+        |
|                                                                |
|        +----------------------------------------------+        |
|        |                Global Policies               |        |
|        +----------------------------------------------+        |
+----------------------------------------------------------------+
       ^                ^                ^                ^
       |                |                |                |
       V                V                V                V
+-------------+  +-------------+  +-------------+  +-------------+
|Compute      |  |Network      |  |Storage      |  |Whatever     |
|Subsystem    |  |Subsystem    |  |Subsystem    |  |Subsystem    |
|             |  |             |  |             |  |             |
|Local Policy |  |Local Policy |  |Local Policy |  |Local Policy |
|Engine       |  |Engine       |  |Engine       |  |Engine       |
|             |  |             |  |             |  |             |
|Local        |  |Local        |  |Local        |  |Local        |
|Policies:    |  |Policies     |  |Policies     |  |Policies     |
| P0, P1,     |  | P0, P1,     |  | P0, P1,     |  | P0, P1,     |
|             |  |             |  |             |  |             |
+-------------+  +-------------+  +-------------+  +-------------+

Figure 3: Global versus Local Policy Engines

5.4. Hierarchical Policy Framework

So far, we have referenced compute, network, and storage as subsystems examples. However, the following subsystems may also support policy engines and subsystem specific policies:

  • SDN Controllers, e.g., OpenDaylight [ODL-SDN-CONTROLLER].
  • OpenStack [OPENSTACK] components such as, Neutron, Cinder, Nova, and etc.
  • Directories, e.g., LDAP, ActiveDirectory, and etc.
  • Applications in general, e.g., standalone or on top of OpenDaylight or OpenStack.
  • Physical and virtual network elements, e.g., routers, firewalls, application delivery controllers (ADCs), and etc.
  • Energy subsystems, e.g., OpenStack Neat [OPENSTACK-NEAT].

Therefore, a policy framework may involve a multitude of subsystems. Subsystems may include other lower level subsystems, e.g., Neutron [OPENSTACK-NEUTRON] would be a lower level subsystem in the OpenStack subsystem. In other words, the policy framework is hierarchical in nature, where the policy engine of a subsystem may be viewed as a higher level policy engine by lower level subsystems. In fact, the global policy engine in Figure 3 could be the policy engine of a Data Center subsystem and multiple Data Center subsystems could be grouped in a region containing a region global policy engine. In addition, one could define regions inside regions, hierarchically, as shown in Figure 4.

Metro and wide-area network (WAN) used to interconnect data centers would also be independent subsystems with their own policy engines.

   To higher level domain
            ^
Region 1    |
Domain      V
+-------------------+        +-------------------+
| +---------------+ |        | +---------------+ |
| |Region 1 Global| |<------>| |WAN 1 Global   | |
| |Policy Engine  | |        | |Policy Engine  | |
| +---------------+ |        | +---------------+ |
|                   |        |                   |
| +---------------+ |        | +---------------+ |
| |Whatever       | |        | |Whatever       | |
| |Subsystems     | |        | |Subsystems     | |
| |               | |        | |               | |
| |Local Policy   | |        | |Local Policy   | |
| |Engines        | |        | |Engines        | |
| +---------------+ |        | +---------------+ |
+-------------------+        +-------------------+
              ^   ^
              |   |
              |   +-------------------------+
              |                             |
DC 1 Domain   V              DC N Domain    V
+-------------------+        +-------------------+
| +---------------+ |        | +---------------+ |
| |DC 1 Global    | |        | |DC N Global    | |
| |Policy Engine  | |        | |Policy Engine  | |
| +---------------+ |        | +---------------+ |
|                   |        |                   |
| +---------------+ |        | +---------------+ |
| |Whatever       | |        | |Whatever       | |
| |Subsystems     | |        | |Subsystems     | |
| |               | |        | |               | |
| |Local Policy   | |        | |Local Policy   | |
| |Engines        | |        | |Engines        | |
| +---------------+ |        | +---------------+ |
+-------------------+        +-------------------+

Figure 4: A Hierarchical Policy Framework

5.4.1. Mapping to Hierarchical Resource Orchestration

If the MANO framework is extended to multi layer hierarchies [I-D.unify-nfvrg-recursive-programming], then a potential mapping of the hierarchical policies to the MANO architecture is shown in Figure 5

                                          T1 T2...Tn
                                        +--|--|----|---+
                                        |              |
                                        |Service       |
                              Domain 4  |  Orchestrator|
************************************    +--+-----------+
                         T1 T2...Tn *******|******************
 Tenant Policies ->    +--|--|----|---+    |
                       | NFVO:        |    |
                       | Service      |    |    <-Domain Policies
                       | Lifecycle    |    |
                       | Orchestrator |    |
                       +-------+------+   /
Domain 3                       |         /
********************         +-+---------+--+   <-Tenant Policies
                    *        |NFVO:         |
******************   *       |Rersource     |   <-Domain Policies
  T1 T2...Tn      *   *      |Orchestration |
+--|--|----|---+   *   *     +--+---+-------+
| NFVO:        |    *   ********|***|************************
| Service      |     *         /    |
| Lifecycle    |    /---------/     |           <-Domain Policies
| Orchestrator |   /   *            |
+---------+----+  |    *            |
          |       |    *            |
       +--+-------+---+*            |           <-Tenant Policies
       |              |*            |
       |NFVO:         |*            |           <-Domain Policies
       |Rersource     |*            |
       |Orchestration |*            |
       +------+-------+*            |
             /|\       *   *********|**********
       +------+-------+*   * +------+-------+ * <-Tenant Policies
    +--|a/VIM:        |*   * |VIM:          | *
 +--|b |Resource      |*   * |Resource      | * <-Domain Policies
 |c |  |Orchestrationn|*   * |Orchestration | *
 |  |  +--------------+*   * +--------------+ *
Domain 1               *   * Domain 2         *
************************   *                  *

Figure 5: Policies in a Hierarchical Orchestration View

5.5. Policy Pub/Sub Bus

In [I-D.irtf-nfvrg-nfv-policy-arch] the authors argued for the need of policy subsystems to subscribe to policy updates at a higher policy level. A policy publication/subscription (pub/sub) bus would be required as shown in Figure 6.

+----------------------------------------------------------------+
|        +----------------------------------------------+        |
|        |             Global Policy Engine             |        |
|        +----------------------------------------------+        |
|                                                                |
|        +----------------------------------------------+        |
|        |                Global Policies               |        |
|        +----------------------------------------------+        |
+----------------------------------------------------------------+
                                 ^
                                 |
                                 |
Policy Pub/Sub Bus               V
  --------------------------------------------------------------
       ^                ^                ^                ^
       |                |                |                |
       |                |                |                |
       V                V                V                V
+-------------+  +-------------+  +-------------+  +-------------+
|Compute      |  |Network      |  |Storage      |  |Whatever     |
|Subsystem    |  |Subsystem    |  |Subsystem    |  |Subsystem    |
|             |  |             |  |             |  |             |
|Local Policy |  |Local Policy |  |Local Policy |  |Local Policy |
|Engine       |  |Engine       |  |Engine       |  |Engine       |
|             |  |             |  |             |  |             |
|Local        |  |Local        |  |Local        |  |Local        |
|Policies:    |  |Policies     |  |Policies     |  |Policies     |
| P0, P1,     |  | P0, P1,     |  | P0, P1,     |  | P0, P1,     |
|             |  |             |  |             |  |             |
+-------------+  +-------------+  +-------------+  +-------------+

Figure 6: A Policy Pub/Sub Bus

A higher tier policy engine would communicate policies to lower tier policy engines using a policy pub/sub bus. Conversely, lower tier policy engines would communicate their configured policies and services to the higher tier policy engine using the same policy pub/sub bus. Such communications require each policy pub/sub bus to have a pre-defined/pre-configured policy "name space". For example, a pub/sub bus could define services using the name space "Platinum", "Gold", and "Silver". A policy could then be communicated over that pub/sub bus specifying a Silver service requirement.

In a hierarchical policy framework, a policy engine may use more than one policy pub/sub bus, e.g., a policy pub/sub bus named "H" to communicate with a higher tier policy engine and a policy pub/sub bus named "L" to communicate with lower tier policy engines. As the name spaces of policy pub/sub buses H and L may be different, the policy engine would translate policies defined using the policy pub/sub bus H name space to policies defined using the policy pub/sub bus L name space, and vice-versa.

5.5.1. Pub/sub bus in the hierarchical framework

Figure 7 shows the Pub/sub bus in the hierarchical MANO framework. Policy communications would employ a policy pub/sub bus between the subsystems' policy engines in the policy hierarchy (see Section 5.4). The global NFVO subsystem should have visibility into the policies defined locally at each PoP to be able to detect any potential global policy conflicts, e.g., a local PoP administrator could add a local policy that violates or conflicts with a global policy. In addition, the global NFVO subsystem would benefit from being able to import the currently configured services at each PoP. The global NFVO would use such information to monitor global policy conformance and also to facilitate detection of policy violations when new global policies are created, e.g., a global level administrator is about to add a new global policy that, if committed, would make certain already configured services a violation of the policy. The publication of subsystem service tables for consumption by a global policy engine is a concept used in the Congress [OPENSTACK-CONGRESS] OpenStack [OPENSTACK] project.

          Customers
              |
              V
       +--------------+
       |  Web Portal  |
       +--------------+
              ^
              |
              V
     +-----------------+        +-----------------+
     | OSS/BSS         |        | Global NFVO     |
     | +-------------+ |        | +-------------+ |
     | |OSS/BSS      | | Policy | |NFVO         | |
     | |Policy Engine|<---------->|Policy Engine| |
     | +-------------+ |        | +-------------+ |
     |                 |        |        ^        |
     |         ...     |        |        | ...    |
     +-----------------+        +--------|--------+
                                         |
       Policy (Pub/Sub Bus)              V
      -------------------------------------------
        ^                  ^                  ^
        |                  |                  |
+-------|-------+  +-------|-------+  +-------|-------+
| PoP A |       |  | PoP B |       |  | PoP C |       |
|       V       |  |       V       |  |       V       |
| +-----------+ |  | +-----------+ |  | +-----------+ |
| |Local NFVO | |  | |Local NFVO | |  | |Local NFVO | |
| | +-------+ | |  | | +-------+ | |  | | +-------+ | |
| | |Policy | | |  | | |Policy | | |  | | |Policy | | |
| | |Engine | | |  | | |Engine | | |  | | |Engine | | |
| | +-------+ | |  | | +-------+ | |  | | +-------+ | |
| +-----------+ |  | +-----------+ |  | +-----------+ |
|       ^       |  |       ^       |  |       ^       |
|       |       |  |       |       |  |       |       |
|       V       |  |       V       |  |       V       |
| +-----------+ |  | +-----------+ |  | +-----------+ |
| |VIM        | |  | |VIM        | |  | |VIM        | |
| | +-------+ | |  | | +-------+ | |  | | +-------+ | |
| | |Policy | | |  | | |Policy | | |  | | |Policy | | |
| | |Engine | | |  | | |Engine | | |  | | |Engine | | |
| | +-------+ | |  | | +-------+ | |  | | +-------+ | |
| +-----------+ |  | +-----------+ |  | +-----------+ |
|         ...   |  |        ...    |  |         ...   |
+---------------+  +---------------+  +---------------+

Figure 7: Pub/sub bus in the hierarchical MANO framework

5.6. Policy Intent Statement versus Subsystem Actions and Configurations

Content to be merged

+----------------------------------------------------------------+
|   Policy: "a given customer must be given Platinum treatment"  |
+----------------------------------------------------------------+
       ^                ^                ^                ^
       |                |                |                |
       V                V                V                V
+-------------+  +-------------+  +-------------+  +-------------+
|Compute      |  |Network      |  |Storage      |  |Whatever     |
|Subsystem    |  |Subsystem    |  |Subsystem    |  |Subsystem    |
|             |  |             |  |             |  |             |
|Policy       |  |Policy       |  |Policy       |  |Policy       |
|translation: |  |translation: |  |translation: |  |translation: |
|             |  |             |  |             |  |             |
|Install      |  |Give customer|  |Give customer|  | ...         |
|customer VMs |  |the best QoS,|  |the fastest  |  |             |
|on servers   |  |which        |  |SSD storage. |  |             |
|with 3GHz    |  |translates   |  |             |  |             |
|16-core Xeon |  |here to set  |  |             |  |             |
|processors,  |  |DHCP to xx,  |  |             |  |             |
|and etc.     |  |and etc.     |  |             |  |             |
+-------------+  +-------------+  +-------------+  +-------------+

Figure 8: Example of Subsystem Translations of Policy Actions

5.7. Static vs Dynamic vs Autonomic Policies

Content to be merged

5.8. Policy Conflicts and Resolution

Content to be merged

5.9. Soft vs Hard Policy Constraints

Content to be merged

5.10. Operational Policies for Resource management

 
              +-------------------+
              |NFVO               |  <- AAA Policy
              |  +--------------+ |
              |  |NFVO:         | |
              |  |Service       | |  <- RD Policy
              |  |Lifecycle     | |
              |  |Management    | |
              |  +------+-------+ |	
              |         |         |
              |  +------+-------+ |
              |  |NFVO:         | |
              |  |Resource      | |  <- RS Policy
              |  |Orchestration | |
              |  +---+--+----+--+ |		
              +------|--|----|----+	  	
                    /    \    \
                   /      \    \   +-------+	
                  /        \    ---+  VNFM |
                 /          \      +---+---+
                /            \         |
               /              \        |
              /                \       |                
   +---------+---------+  +-----+------+------+  
   |WIM                |  |VIM                |            
   |  +--------------+ |  |  +--------------+ |  
   |  |WIM:          | |  |  |VIM:          | |  
   |  |Orchestration | |  |  |Orchestration | |  <- RA Policy
   |  |&             | |  |  |&             | |  
   |  |Optimization  | |  |  |Optimization  | |  
   |  +------+-------+ |  |  +------+-------+ |  
   |         |         |  |         |         | 
   |  +------+-------+ |  |  +------+-------+ | 
   |  |WIM:          | |  |  |VIM:          | | 
   |  |Virtualized 2 | |  |  |Virtualized 2 | |  <- RE Policy
   |  |Pys mapping   | |  |  |Pys mapping   | | 
   |  +--------------+ |  |  +--------------+ |  
   +-------------------+  +-------------------+  

Figure 9: Operational policies for resource management

The use of NFVI resources for multiple network services can be optimized in various objectives as defined in the operational policies (as described in Section 5.2).

The operational policies can be split to different layers of NFVO and VIM/WIM and they include 1) resource scheduling (RS) policy, resource adaptation (RD) policy and authentication, authorization, accounting (AAA) policy at NFVO, and 2) resource allocation (RA) policy and resource embedding (RE) policy at VIM/WIM. They can be mapped to the MANO architecture as shown in Figure 9.

5.10.1. Operational Policies at NFVO

During NS/VNF lifecycles, states of NFVI/WAN resources or the performance of VNF and VL instances may vary in time (e.g., the performance degradation due to incorrect placement or incorrect forwarding action). Another concern for such dynamic changes is fail-over as a fundamental consideration, i.e., physical resources or virtualized resources in NFVI may fail during network services. These dynamic changes significantly could affect the overall performance for NS. Therefore, such dynamic changes triggered during NS/VNF lifecycles should be coped with for guaranteeing the NS performance and the optimized resource usage. Figure 9 shows that NFVO needs to enforce resource adaptation (RD) policy as an operational policy at NFVO. RD policy supports how NFVO adapts the allocated NFVI/WAN resources (e.g., VM migration, scaling) by dealing with triggered variations. RD policy engine can detect the changes from measurement and diagnosis from VNFM and/or VIM/WIM.

Figure 9 also shows that NFVO needs to enforce resource scheduling (RS) policy. RS policy determines the locations of VNF and VL instances that constitute NS across multiple PoPs and WANs while optimally allocating NFVI and WAN resources to the instances.

In particular, RD and RA policies would consider a business model from OSS/BSS which specifies operational (or business) objectives (e.g., overall energy consumption and NFVI resource utilization) within its domain and with taking account of (on-boarded) network service descriptor (NSD) as an NS policy including the virtualization aspects of application feature, QoS parameters, affinity, anti-affinity rules, and so on.

On the one hand, for the user authorization, authentication, authorization, accounting (AAA) policy may be needed. Authentication policy provides a way of identifying a user while the authorization policy determines whether the user has the authority for virtualized resources (i.e., NFVI/WAN resources) to receive the network service or not. Accounting policy measures the resources the user consumes during the network service. This can include the amount of system time/data, and so on.

5.10.2. Operational Policies at VIM/WIM

As shown in Figure 9, RA policy supports how each subsystem (e.g., compute, storage subsystem) in NFVI is allocated depending on the placement information from NFVO to further optimize the resource usage. Moreover, the assigned NFVI resources are embedded (or allocated) to physical resources in VIM/WIM depending on states and usage of resources by means of resource embedding (RE) policy as shown in Figure 9. In other words, RE policy determines and coordinates how the allocated virtual resources are mapped to physical resources. For example, RE policy may be updated when some physical resources are failed or a virtualization technique is changed.

6. Policy-Based Resource Management Examples

6.1. Policy-Based Multipoint Ethernet Service

Content to be merged

6.2. Policy-Based NFV Placement

Content to be merged

6.3. Policy-Based VNF-FG Management

   
                  +-------------------+
                  | NFVO              |
                  | +---------------+ |
                  | |NFVO           | |
                  | |RS Policy      | |
                  | |Engine         | |
                  | +---------------+ |
                  |         ^         |
                  |         |    ...  |
                  +---------|---------+ 
                            |               
                            V   Allocation information            
     -----------------------------------------------
        ^                   ^                   ^
        |                   |                   |
+-------|---------+ +-------|---------+ +-------|---------+
|vPoP A |         | |WAN    |         | |vPoP B |         |
|       V         | |       V         | |       V         |
| +-------------+ | | +-------------+ | | +-------------+ |	
| |VIM A        | | | |WIM          | | | |VIM B	| | 
| | +---------+ | | | | +---------+ | | | | +---------+ | |
| | |RA Policy| | | | | |RA Policy| | | | | |RA Policy| | |
| | |Engine   | | | | | |Engine   | | | | | |Engine   | | |  
| | +---------+ | | | | +---------+ | | | | +---------+ | | 
| +-------------+ | | +-------------+ | | +-------------+ |		
|          ...    | |          ...    | |          ... 	  |
|                 | |                 | |                 |
| +-----+         | |                 | |                 |
| |VNF-A|         | |                 | |                 |
| +-----+         | |                 | |                 | 
|    #	  +-----+ | |                 | |     +-----+     |
|    #####|VNF-B|#############################|VNF-C| 	  |
|   VL1	  +-----+ | | 	     VL2      | |     +-----+     |
|                 | |                 | |                 |
+-----------------+ +-----------------+ +-----------------+

### NFP


Figure 10: Policy-based VNF-FG Management

Another subsystem example for the policy framework is VNF-FG. When VNF-FGs of end-to-end network services are realized, NFVI resources across multiple NFVI-PoPs and WAN resources that connect among them should be allocated to the VNF-FGs. It depends on the target KPIs of individual VNF and VL instances that constitute VNF-FGs. In particular, in case of VNF-FG, chained performances and capabilities of VNF and VL instances need to be considered together with on VL instances the inter-connectivity between different NFVI-PoPs. For example, if one of the VNF instances or VL instances along the VNF-FG gets overloaded, the end-to-end network service may also get affected. Therefore, while features of such VNF-FG are carefully considered, proper operational policies for resource management (see Section 5.10) are required.

As shown in Figure 10, consider a scenario where a user requests a VNF-FG composed of "VNF A-VL 1-VNF B-VL 2-VNF C". For the VNF-FG, an RA policy is enforced in which it is designed to avoid over-utilization of PoP A and to reduce latency on VL 1. Therefore, NFVO places VNF A, VNF B, and VL 1 on PoP A by consuming its computing and network resources to achieve low latency. On the other hand, VL 2 and VNF C is allocated to the resources of WAN and PoP B, respectively to avoid over-utilization of PoP A.

On the one hand, dynamic changes such as a VNF failure significantly affect on the overall performance of VNF-FG since VNF-FG is a chain of VNF and VL instances. Thus, such dynamic changes should be coped with by RD policy for guaranteeing the VNF-FG performance and the optimized resource usage. A fault management for VNF-FG based on policy example is shown in Section 6.4.

6.4. Policy-Based Fault Management

   
                  +-------------------+
                  | NFVO              |
                  | +---------------+ |
                  | |NFVO           | |
                  | |RS Policy      | |
                  | |Engine         | |
                  | +---------------+ |
                  |         ^         | 
                  |         |    ...  | 
                  +---------|---------+ 
                            
                            V	
      ----------------------------------------------
        ^                   ^                   ^
        |                   |                   |
+-------|---------+ +-------|---------+ +-------|---------+
|vPoP A |         | |vPoP B |         | |vPoP C |         |
|       V         | |       V         | |       V         |
| +-------------+ | | +-------------+ | | +-------------+ |	
| |VIM A        | | | |VIM B        | | | |VIM C	| | 
| | +---------+ | | | | +---------+ | | | | +---------+ | |
| | |RA Policy| | | | | |RA Policy| | | | | |RA Policy| | |
| | |Engine   | | | | | |Engine   | | | | | |Engine   | | |  
| | +---------+ | | | | +---------+ | | | | +---------+ | | 
| +-------------+ | | +-------------+ | | +-------------+ |		
|          ...    | |          ...    | |          ... 	  |
|                 | |                 | |                 |
|    +-----+      | |  	  +-----+     | |     +-----+     |
|    |VNF-A|##############|VNF-B|#############|VNF-C|     |
|    +-----+ VL-1 | |     +-----+ VL-2| |     +-----+     | 
|                 | |        ^        | |                 |
|                 | |        |	      | |                 |
|                 | |    (failure)    | |                 |
|                 | |                 | |                 |
+-----------------+ +-----------------+ +-----------------+

### NFP


Figure 11: Failure Scenario for VNF-FG


 +------------------------------------+
 |NFVO                                |
 | +--------------------------------+ |
 | |       RD policy engine         | |
 | |Adapts resources to the failed  | |
 | |elements while guaranteeing     | |
 | |performance                     | |
 | +--------------------------------+ |
 +------------------+-----------------+
                    |
                    |
+-------------------+---------------------+
|VNFM/VIM/WIM                             |
| +-------------------------------------+ |
| |       Diagnosis / Measurement       | |
| |A failure event                      | |
| |Throughputs of VNF and VL instances  | |
| |Topological location...              | |
| +-------------------------------------+ |
+-----------------------------------------+

Figure 12: Architecture for policy-based fault management

As shown in Figure 11, consider a scenario that a VM related to VNF-B (i.e., a VNF-B instance) is failed in the given VNF-FG composed VNF-A, VNF-B, VNF-C in order. Note that the NFVI and WAN resources are already allocated to the instances by RS policy. For service continuity, failure of the VNF-B instance needs to be detected based on diagnosis function in VIM/VNFM and the failed one needs to be replaced with a new instance or to be assigned to the existing instance which is available. The diagnosis and measurement function may collect current performance measures and location for instances as well as such a failure event.

   
                  +-------------------+
                  | NFVO              |
                  | +---------------+ |
                  | |NFVO           | |
                  | |RD Policy      | |
                  | |Engine         | |
                  | +---------------+ |
                  |         ^         | 
                  |         |    ...  | 
                  +---------|---------+ 
                            |               
                            V	           
      ----------------------------------------------
        ^                   ^                   ^
        |                   |                   |
+-------|---------+ +-------|---------+ +-------|---------+
|vPoP A |         | |vPoP B |         | |vPoP C |         |
|       V         | |       V         | |       V         |
| +-------------+ | | +-------------+ | | +-------------+ |	
| |VIM A        | | | |VIM B        | | | |VIM C	| | 
| | +---------+ | | | | +---------+ | | | | +---------+ | |
| | |RA Policy| | | | | |RA Policy| | | | | |RA Policy| | |
| | |Engine   | | | | | |Engine   | | | | | |Engine   | | |  
| | +---------+ | | | | +---------+ | | | | +---------+ | | 
| +-------------+ | | +-------------+ | | +-------------+ |		
|          ...    | |          ...    | |          ... 	  |
|                 | |                 | |                 |
|    +-----+      | |  	  +-----+     | |     +-----+     |
|    |VNF-A|      | |     |VNF-B|     |	|     |VNF-C|     |
|    +-----+  	  | |     +-----+     | |     +-----+     | 
|       #         | |        ^        | |        #        |
|       # New     | |        |	      | |        #        |
|       # VL-1    | |    (failure)    |          #        |
|    +-------+    | |                 | |        #        |
|    |New    |####################################        |
|    |VNF-B  |	  | |                 | |     New VL-2    |
|    +-------+	  | |                 | |                 |
|                 | |                 | |                 |
+-----------------+ +-----------------+ +-----------------+

### NFP


Figure 13: Re-instantiation for VNF-FG

In the first case where a VNF instantiation is needed, a new VNF instantiation is determined by the RD policy engine in NFVO. For example, NFVO may avoid replacement of VNF B on NFVI-PoP B owing to high possibility of failure. Therefore, NFVO could instantiate VNF B on NFVI-PoP A or NFVI-PoP C with the setup of new connection points (CPs) while guaranteeing performance as shown in Figure 13.

   
                  +-------------------+
                  | NFVO              |
                  | +---------------+ |
                  | |NFVO           | |
                  | |RD Policy      | |
                  | |Engine         | |
                  | +---------------+ |
                  |         ^         | 
                  |         |    ...  | 
                  +---------|---------+ 
                            |
                            V	           
      ----------------------------------------------
        ^                   ^                   ^
        |                   |                   |
+-------|---------+ +-------|---------+ +-------|---------+
|vPoP A |         | |vPoP B |         | |vPoP C |         |
|       V         | |       V         | |       V         |
| +-------------+ | | +-------------+ | | +-------------+ |	
| |VIM A        | | | |VIM B        | | | |VIM C	| | 
| | +---------+ | | | | +---------+ | | | | +---------+ | |
| | |RA Policy| | | | | |RA Policy| | | | | |RA Policy| | |
| | |Engine   | | | | | |Engine   | | | | | |Engine   | | |  
| | +---------+ | | | | +---------+ | | | | +---------+ | | 
| +-------------+ | | +-------------+ | | +-------------+ |		
|          ...    | |          ...    | |          ... 	  |
|                 | |                 | |                 |
|    +-----+      | |  	  +-----+     | |     +-----+     |
|    |VNF-A|      | |     |VNF-B|     | |     |VNF-C|     |
|    +-----+      | |     +-----+     | |     +-----+     | 
|       #         | |        ^        | |        #        |
|       #         | |        |	      | |        #        |
|       # New     | |    (failure)    | |        #        |
|       # VL-1	  | |                 | |        # New	  |
|       #         | |     +-------+   | |        # VL-2   |
|     	##################| VNF-B1|###############        |              
|                 | |     +-------+   | |                 |
+-----------------+ +-----------------+ +-----------------+

### NFP


Figure 14: No Re-instantiation for VNF-FG

In the second case where no VNF instantiation is needed since a redundant VNF exists, the available VNF-B instance can used by the VNF-FG. For example, a redundant VNF B instance exists in NFVI-PoP B. Therefore, NFVO selects the instance and re-constructs two VLs as shown in Figure 14, and the corresponding NS can be continued without re-instantiation.

7. Implementation Examples

tbd

8. Gaps and Open Questions

tbd

9. Conclusions

tbd

9.1. Relation to other IETF/IRTF activities

tbd

10. Acknowledgements

The research leading to some of the results described in this document has received funding from the European Union Seventh Framework Programme (FP7/2007-2013) under grant agreement no. 619609 - the UNIFY project. The views expressed here are those of the authors only. The European Commission is not liable for any use that may be made of the information in this document.

11. Contributors

This document is the result of merging multiple drafts. This section acknowledges those who provided important ideas and text into this document.

12. IANA Considerations

tbd

13. Security Considerations

tbd

14. References

14.1. Normative References

[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997.
[RFC3060] Moore, B., Ellesson, E., Strassner, J. and A. Westerinen, "Policy Core Information Model -- Version 1 Specification", RFC 3060, DOI 10.17487/RFC3060, February 2001.
[RFC3198] Westerinen, A., Schnizlein, J., Strassner, J., Scherling, M., Quinn, B., Herzog, S., Huynh, A., Carlson, M., Perry, J. and S. Waldbusser, "Terminology for Policy-Based Management", RFC 3198, DOI 10.17487/RFC3198, November 2001.
[RFC3670] Moore, B., Durham, D., Strassner, J., Westerinen, A. and W. Weiss, "Information Model for Describing Network Device QoS Datapath Mechanisms", RFC 3670, DOI 10.17487/RFC3670, January 2004.

14.2. Informative References

, ", ", ", ", ", ", ", "
[CERI-DATALOG] Ceri, S. and others, "What you always wanted to know about Datalog (and never dared to ask", IEEE Transactions on Knowledge and Data Engineering, (Volume: 1, Issue: 1), August 2002.
[ETSI-NFV-MANO] ETSI, "Network Function Virtualization (NFV) Management and Orchestration V0.6.3", October 2014.
[ETSI-NFV-PER001] ETSI, "Network Function Virtualization: Performance and Portability Best Practices v1.1.1", June 2014.
[ETSI-NFV-TERM] ETSI, "NFV Terminology for Main Concepts in NFV", October 2013.
[ETSI-NFV-WHITE-PAPER] ETSI NFV White Paper, "Network Functions Virtualisation, An Introduction, Benefits, Enablers, Challenges, & Call for Action"
[I-D.ietf-bmwg-virtual-net] Morton, A., "Considerations for Benchmarking Virtual Network Functions and Their Infrastructure", Internet-Draft draft-ietf-bmwg-virtual-net-04, August 2016.
[I-D.irtf-nfvrg-nfv-policy-arch] Figueira, N., Krishnan, R., Lopez, D., Wright, S. and D. Krishnaswamy, "Policy Architecture and Framework for NFV Infrastructures", Internet-Draft draft-irtf-nfvrg-nfv-policy-arch-04, September 2016.
[I-D.irtf-nfvrg-resource-management-service-chain] Lee, S., Pack, S., Shin, M., Paik, E. and R. Browne, "Resource Management in Service Chaining", Internet-Draft draft-irtf-nfvrg-resource-management-service-chain-03, March 2016.
[I-D.liu-bmwg-virtual-network-benchmark] Liu, V., Liu, D., Mandeville, B., Hickman, B. and G. Zhang, "Benchmarking Methodology for Virtualization Network Performance", Internet-Draft draft-liu-bmwg-virtual-network-benchmark-00, July 2014.
[I-D.norival-nfvrg-nfv-policy-arch] Figueira, N., Krishnan, R., Lopez, D. and S. Wright, "Policy Architecture and Framework for NFV Infrastructures", Internet-Draft draft-norival-nfvrg-nfv-policy-arch-04, June 2015.
[I-D.unify-nfvrg-recursive-programming] Szabo, R., Qiang, Z. and M. Kind, "Towards recursive virtualization and programming for network and cloud resources", Internet-Draft draft-unify-nfvrg-recursive-programming-02, October 2015.
[ODL-GB-POLICY]OpenDaylight Group Based Policy"
[ODL-NIC-PROJECT]OpenDaylight Network Intent Composition Project"
[ODL-SDN-CONTROLLER]OpenDaylight SDN Controller"
[OPENSTACK]OpenStack"
[OPENSTACK-CONGRESS]OpenStack Congress"
[OPENSTACK-NEAT]OpenStack Neat"
[OPENSTACK-NEUTRON]OpenStack Neutron"
[POLICY-FRAMEWORK-WG]Policy Framework Working Group (IETF)"
[RESOURCE-MGMT-SERVICE-CHAIN] Lee, S. and others, Resource Management in Service Chaining"
[SDN-MULTI-DOMAIN] Figueira, N. and R. Krishnan, "SDN Multi-Domain Orchestration and Control: Challenges and Innovative Future Directions", IEEE International Conference on Computing (ICNC), February 2015.
[VM-HOSTING-NET-CLUSTER] Grit, L. and others, "Virtual Machine Hosting for Networked Clusters: Building the Foundations for "Autonomic" Orchestration", Virtualization Technology in Distributed Computing (VTDC), 2006.

Authors' Addresses

Robert Szabo (editor) Ericsson Konyves Kaman krt. 11 Budapest, EMEA 1097 Hungary Phone: +36703135738 EMail: robert.szabo@ericsson.com
Seungik Lee (editor) ETRI 218 Gajeong-ro Yuseung-Gu Daejeon, 305-700 Korea Phone: +82 42 860 1483 EMail: seungiklee@etri.re.kr
Norival Figueira Brocade EMail: nfigueir@Brocade.com