+--------+   +--------+
        |  ALTO  |   |  ALTO  |
        | Client |   | Client |
        +--------+   +--------+
               /\     /\
               ||     || ALTO protocol
               ||     ||
               \/     \/
              +---------+
              |  ALTO   |
              | Server  |
              +---------+
               :   :   :    
               :   :   :
      +........+   :   +........+ Provisioning
      :            :            : protocol
      :            :            :
 +---------+  +---------+  +---------+
 |   BGP   |  |   I2RS  |  |   NMS   | Potential
 | Speaker |  |  Client |  |   OSS   | data sources
 +---------+  +---------+  +---------+
      ^            ^            ^
      |            |            |
 Link-State      I2RS      SNMP/NETCONF,
  NLRI for       data      traffic statistics,
  IGP/BGP                  IPFIX, etc.

Figure 8: Potential data sources for ALTO

As illustrated in Figure 8, the topology data used by an ALTO server can originate from different data sources:

• Relevant information sources are interior gateway protocols (IGPs) or the Border Gateway Protocol (BGP). An ALTO server could get network routing information by listening to IGPs and/or peering with BGP speakers. For data collection, link-state protocols are more suitable since every router propagates its information throughout the whole network. Hence, it is possible to obtain information about all routers and their neighbors from one single router in the network. In contrast, distance-vector protocols are less suitable since routing information is only shared among neighbors. To obtain the whole topology with distance-vector routing protocols it is necessary to retrieve routing information from every router in the network.
• The document [I-D.ietf-idr-ls-distribution] describes a mechanism by which links state and traffic engineering information can be collected from networks and shared with external components using the BGP routing protocol. This is achieved using a new BGP Network Layer Reachability Information (NLRI) encoding format. The mechanism is applicable to physical and virtual IGP links and can also include Traffic Engineering (TE) data. For instance, prefix data can be carried and originated in BGP, while TE data is originated and carried in an IGP. The mechanism described is subject to policy control.
• The Interface to the Routing System (I2RS) is a solution for state transfer in and out of the Internet's routing system [I-D.ietf-i2rs-architecture]. An ALTO server could use an I2RS client to observe routing-related information. With the rise of Software-Defined Networking (SDN) and a decoupling of network data and control plane, topology information could also be fetched from an SDN controller. This scenario is not further discussed in the remainder of this document.
• An ALTO server can also leverage a Network Management System (NMS) or an Operations Support System (OSS) as data sources. NMS or OSS solutions are used to control, operate, and manage a network, e.g., using the Simple Network Management Protocol (SNMP) or NETCONF. As explained for instance in [RFC7491], the NMS and OSS can be consumers of network events reported and can act on these reports as well as displaying them to users and raising alarms. The NMS and OSS can also access the Traffic Engineering Database (TED) and Label Switched Path Database (LSP-DB) to show the users the current state of the network. In addition, NMS and OSS systems may have access to IGP/BGP routing information, network inventory data (e.g., links, nodes, or link properties not visible to routing protocols, such as Shared Risk Link Groups), statistics collection system that provides traffic information, such as traffic demands or link utilization obtained from IP Flow Information Export (IPFIX), as well as other Operations, Administration, and Maintenance (OAM) information (e.g., syslog). NMS or OSS systems also may have functions to correlate and orchestrate information originating from other data sources. For instance, it could be required to correlate IP prefixes with routers (Provider, Provider Edge, Customer Edge, etc.), IGP areas, VLAN IDs, or policies.

The data sources mentioned so far are only a subset of potential topology sources and depending on the network type, (e.g. mobile, satellite network) different hardware and protocols are in operation to form and maintain the network.

In general it is challenging to gather detailed information about the whole Internet, since the network consists of multiple domains and in many cases it is not possible to collect information across network borders. Hence, potential information sources may be limited to a certain domain.

3.2.3. Partitioning and Grouping of IP Address Ranges

ALTO introduces provider-defined network location identifiers called Provider-defined Identifiers (PIDs) to aggregate network endpoints in the Map Services. Endpoints within one PID may be treated as single entity, assuming proximity based on network topology or other similarity. A key use case of PIDs is to specify network preferences (costs) between PIDs instead of individual endpoints. It is up to the operator of the ALTO server how to group endpoints and how to assign PIDs. For example, a PID may denote a subnet, a set of subnets, a metropolitan area, a POP, an autonomous system, or a set of autonomous systems.

This document only considers deployment scenarios in which PIDs expand to a set of IP address ranges (CIDR). A PID is characterized by a string identifier and its associated set of endpoint addresses [RFC7285]. If an ALTO server offers the Map Service, corresponding identifiers have to be configured.

An automated ALTO implementation may use dynamic algorithms to aggregate network topology. However, it is often desirable to have a mechanism through which the network operator can control the level and details of network aggregation based on a set of requirements and constraints. This will typically be governed by policies that enforce a certain level of abstraction and prevent leakage of sensitive operational data.

For instance, an ALTO server may leverage BGP information that is available in a networks service provider network layer and compute the group of prefix. An example are BGP communities, which are used in MPLS/IP networks as a common mechanism to aggregate and group prefixes. A BGP community is an attribute used to tag a prefix to group prefixes based on mostly any criteria (as an example, most ISP networks originate BGP prefixes with communities identifying the Point of Presence (PoP) where the prefix has been originated). These BGP communities could be used to map IP address ranges to PIDs. By an additional policy, the ALTO server operator may decide an arbitrary cost defined between groups. Alternatively, there are algorithms that allow the dynamic computation of costs between groups. The ALTO protocol itself is independent of such algorithms and policies.

3.2.4. Rating Criteria and/or Cost Calculation

An ALTO server indicates preferences amongst network locations in the form of path costs. Path costs are generic costs and can be internally computed by the operator of the ALTO server according to its own policy. For a given ALTO network map, an ALTO cost map defines directional path costs pairwise amongst the set of source and destination network locations defined by the PIDs.

The ALTO protocol permits the use of different cost types. An ALTO cost type is defined by the combination of a cost metric and a cost mode. The cost metric identifies what the costs represent. The cost mode identifies how the costs should be interpreted, e.g., whether returned costs should be interpreted as numerical values or ordinal rankings. The ALTO protocol also allows the definition of additional constraints defining which elements of a cost map shall be returned.

The ALTO protocol specification [RFC7285] defines the "routingcost" cost metric as basic set of rating criteria, which has to be supported by all implementations. This cost metric conveys a generic measure for the cost of routing traffic from a source to a destination. A lower value indicates a higher preference for traffic to be sent from a source to a destination. It is up to the ALTO server how that metric is calculated.

It is possible to calculate the "routingcost" cost metric based on actual routing protocol information. Typically, Interior Gateway Protocols (IGP) provide details about endpoints and links within the own network, while the Bordger Gateway Protocol (BGPs) is used to provide details about links to endpoints in other networks. Besides topology and routing information networks have a multitude of other attributes about its state, condition, and operation. That comprises but is not limited to attributes like link utilization, bandwidth and delay, ingress/egress points of data flows from/towards endpoints outside of the network up to the location of nodes and endpoints.

In order to enable use of extended information, there is an extension procedure for adding new ALTO cost types. The following list gives an overview on further rating criteria that have been proposed or which are in use by ALTO-related prototype implementations. This list is not intended as normative text; a definition of further metrics can be found for instance in [I-D.wu-alto-te-metrics]. Instead, the only purpose of the following list is to document and discuss rating criteria that have been proposed so far. It can also depend on the use case of ALTO whether such rating criteria are useful, and whether the corresponding information would indeed be made available by ISPs.

Distance-related rating criteria:

• Relative topological distance: The term relative means that a larger numerical value means greater distance, but it is up to the ALTO service how to compute the values, and the ALTO client will not be informed about the nature of the information. One way of generating this kind of information may be counting AS hops, but when querying this parameter, the ALTO client must not assume that the numbers actually are AS hops. In addition to the AS path, a relative cost value could also be calculated taking into account other routing protocol parameters, such as BGP local preference or multi-exit discriminator (MED) attributes.
• Absolute topological distance, expressed in the number of traversed autonomous systems (AS).
• Absolute topological distance, expressed in the number of router hops (i.e., how much the TTL value of an IP packet will be decreased during transit).
• Absolute physical distance, based on knowledge of the approximate geo-location (e.g., continent, country) of an IP address.

Performance-related rating criteria:

• The minimum achievable throughput between the resource consumer and the candidate resource provider, which is considered useful by the application (only in ALTO queries).
• An arbitrary upper bound for the throughput from/to the candidate resource provider (only in ALTO responses). This may be, but is not necessarily the provisioned access bandwidth of the candidate resource provider.
• The maximum round-trip time (RTT) between resource consumer and the candidate resource provider, which is acceptable for the application for useful communication with the candidate resource provider (only in ALTO queries).
• An arbitrary lower bound for the RTT between resource consumer and the candidate resource provider (only in ALTO responses). This may be, for example, based on measurements of the propagation delay in a completely unloaded network.

Charging-related rating criteria:

• Traffic volume caps, in case the Internet access of the resource consumer is not charged by "flat rate". For each candidate resource provider, the ALTO service could indicate the amount of data that may be transferred from/to this resource provider until a given point in time, and how much of this amount has already been consumed. Furthermore, it would have to be indicated how excess traffic would be handled (e.g., blocked, throttled, or charged separately at an indicated price). The interaction of several applications running on a host, out of which some use this criterion while others don't, as well as the evaluation of this criterion in resource directories, which issue ALTO queries on behalf of other endpoints, are for further study.
• Other metrics representing an abstract cost, e.g., determined by policies that distinguish "cheap" from "expensive" IP subnet ranges without detailing the cost function.

These rating criteria are subject to the remarks below:

The ALTO client must be aware that with high probability the actual performance values will differ from whatever an ALTO server exposes. In particular, an ALTO client must not consider a throughput parameter as a permission to send data at the indicated rate without using congestion control mechanisms.

The discrepancies are due to various reasons, including, but not limited to the facts that

• the ALTO service is not an admission control system
• the ALTO service may not know the instantaneous congestion status of the network
• the ALTO service may not know all link bandwidths, i.e., where the bottleneck really is, and there may be shared bottlenecks
• the ALTO service may not have all information about the actual routing
• the ALTO service may not know whether the candidate endpoint itself is overloaded
• the ALTO service may not know whether the candidate endpoint throttles the bandwidth it devotes for the considered application
• the ALTO service may not know whether the candidate endpoint will throttle the data it sends to the client (e.g., because of some fairness algorithm, such as tit-for-tat).

Because of these inaccuracies and the lack of complete, instantaneous state information, which are inherent to the ALTO service, the application must use other mechanisms (such as passive measurements on actual data transmissions) to assess the currently achievable throughput, and it must use appropriate congestion control mechanisms in order to avoid a congestion collapse. Nevertheless, the rating criteria may provide a useful shortcut for quickly excluding candidate resource providers from such probing, if it is known in advance that connectivity is in any case worse than what is considered the minimum useful value by the respective application.

Rating criteria that should not be defined for and used by the ALTO service include:

• Performance metrics that are closely related to the instantaneous congestion status. The definition of alternate approaches for congestion control is explicitly out of the scope of ALTO. Instead, other appropriate means, such as using TCP based transport, have to be used to avoid congestion.
• Performance metrics that raise privacy concerns. For instance, it has been questioned whether an ALTO service could publicly expose the provisioned access bandwidth, e.g. of cable / DSL customers, because this could enables identification of "premium" customers.

3.3. Known Limitations of ALTO

3.3.1. General Limitations

ALTO is designed as a protocol between clients integrated in applications and servers that provide network information and guidance (e.g., basic network location structure and preferences of network paths). The objective is to modify network resource consumption patterns at application level while maintaining or improving application performance. This design focus results in a number of characteristics of ALTO:

• Endpoint focus: In typical ALTO use cases, neither the consumer of the topology information (i.e., the ALTO client) nor the considered resources (e.g., files at endpoints) are part of the network. The ALTO server presents an abstract network topology containing only information relevant to an application overlay for better-than-random resource provider selection among its endpoints. The ALTO protocol specification [RFC7285] is not designed to expose network internals such as routing tables or configuration data that are not relevant for application-level resource provider selection decisions in network endpoints.
• Abstraction: The ALTO services such as the Network/Cost Map Service or the ECS provide an abstract view of the network only. The operator of the ALTO server has full control over the granularity (e.g., by defining policies how to aggregate subnets into PIDs) and the level-of-detail of the abstract network representation (e.g., by deciding what cost types to support).
• Multiple administrative domains: The ALTO protocol is designed for use cases where the ALTO server and client can be located in different organizations or trust domains. ALTO assumes a loose coupling between server and client. In addition, ALTO does not assume that an ALTO client has any a priori knowledge about the ALTO server and its supported features. An ALTO server can be discovered automatically.
• Read-only: ALTO is a query/response protocol to retrieve guidance information. Neither network/cost map queries nor queries to the endpoint cost service are designed to affect state in the network.

If ALTO shall be deployed for use cases violating these assumptions, the protocol design may result in limitations.

For instance, in an Application-Based Network Operation (ABNO) environment the application could issue explicit service request to the network [RFC7491]. In this case, the application would require detailed knowledge about the internal network topology and the actual state. A network configuration would also require a corresponding security solution for authentication and authorization. ALTO is not designed for operations to control, operate, and manage a network.

Such deployments could be addressed by network management solutions, e.g., based on SNMP [RFC3411] or NETCONF [RFC6241] and YANG [RFC6020] that are typically designed to manipulate configuration state. Reference [RFC7491] contains a more detailed discussion of interfaces between components such as Element Management System (EMS), Network Management System (NMS), Operations Support System (OSS), Traffic Engineering Database (TED), Label Switched Path Database (LSP-DB), Path Computation Element (PCE), and other Operations, Administration, and Maintenance (OAM) components.

3.3.2. Limitations of Map-based Approaches

The specification of the Map Service in the ALTO protocol [RFC7285] is based on the concept of network maps. A network map partitions the network into Provider-defined Identifiers (PIDs) that group one or more endpoints (e.g., subnetworks) to a single aggregate. The "costs" between the various PIDs are stored in a cost map. Map-based approaches lower the signaling load on the server as maps have to be retrieved only if they change.

One main assumption for map-based approaches is that the information provided in these maps is static for a long period of time. This assumption is fine as long as the network operator does not change any parameter, e.g., routing within the network and to the upstream peers, IP address assignment stays stable (and thus the mapping to the partitions). However, there are several cases where this assumption is not valid:

1. ISPs reallocate IP subnets from time to time;
2. ISPs reallocate IP subnets on short notice;
3. IP prefix blocks may be assigned to a router that serves a variety of access networks;
4. Network costs between IP prefixes may change depending on the ISP's routing and traffic engineering.

These effects can be explained as follows:

Case 1: ISPs may reallocate IP subnets within their infrastructure from time to time, partly to ensure the efficient usage of IPv4 addresses (a scarce resource), and partly to enable efficient route tables within their network routers. The frequency of these "renumbering events" depend on the growth in number of subscribers and the availability of address space within the ISP. As a result, a subscriber's household device could retain an IP address for as short as a few minutes, or for months at a time or even longer.

It has been suggested that ISPs providing ALTO services could sub-divide their subscribers' devices into different IP subnets (or certain IP address ranges) based on the purchased service tier, as well as based on the location in the network topology. The problem is that this sub-allocation of IP subnets tends to decrease the efficiency of IP address allocation, in particular for IPv4. A growing ISP that needs to maintain high efficiency of IP address utilization may be reluctant to jeopardize their future acquisition of IP address space.

However, this is not an issue for map-based approaches if changes are applied in the order of days.

Case 2: ISPs can use techniques that allow the reallocation of IP prefixes on very short notice, i.e., within minutes. An IP prefix that has no IP address assignment to a host anymore can be reallocated to areas where there is currently a high demand for IP addresses.

Case 3: In residential access networks (e.g., DSL, cable), IP prefixes are assigned to broadband gateways, which are the first IP-hop in the access-network between the Customer Premises Equipment (CPE) and the Internet. The access-network between CPE and broadband gateway (called aggregation network) can have varying characteristics (and thus associated costs), but still using the same IP prefix. For instance one IP address IP1 out of a given CIDR prefix can be assigned to a VDSL access line (e.g., 2 MBit/s uplink) while another IP address IP2 within the same given CIDR prefix is assigned to a slow ADSL line (e.g., 128 kbit/s uplink). These IP addresses may be assigned on a first come first served basis, i.e., a single IP address out of the same CIDR prefix can change its associated costs quite fast. This may not be an issue with respect to the used upstream provider (thus the cross ISP traffic) but depending on the capacity of the aggregation-network this may raise to an issue.

Case 4: The routing and traffic engineering inside an ISP network, as well as the peering with other autonomous systems, can change dynamically and affect the information exposed by an ALTO server. As a result, cost maps and possibly also network maps can change.

3.3.3. Limitations of Non-Map-based Approaches

The specification of the ALTO protocol [RFC7285] also includes the Endpoint Cost Service (ECS) mechanism. ALTO clients can ask the ALTO server for guidance for specific IP addresses, thereby avoiding the need of processing maps. This can mitigate some of the problems mentioned in the previous section.

However frequent requests, particularly with long lists of IP addresses, may overload the ALTO server. The server has to rank each received IP address, which causes load at the server. This may be amplified when not only a single ALTO client is asking for guidance, but a larger number of them. The results of the ECS are also more difficult to cache than ALTO maps. Therefore, the ALTO client may have to await the server response before starting a communication, which results in an additional delay.

Caching of IP addresses at the ALTO client or the usage of the H12 approach [I-D.kiesel-alto-h12] in conjunction with caching may lower the query load on the ALTO server.

When ALTO server receives an ECS request, it may not have the most appropriate topology information in order to accurately determine the ranking. [RFC7285] generally assumes that a server can always offer some guidance. In such a case the ALTO server could adopt one of the following strategies:

• Reply with available information (best effort).
• Query another ALTO server presumed to have better topology information and return that response (cascaded servers).
• Redirect the request to another ALTO server presumed to have better topology information (redirection).

The protocol mechanisms and decision processes that would be used to determine if redirection is necessary and which mode to use is out of the scope of this document, since protocol extensions could be required.

3.4. Monitoring ALTO

3.4.1. Impact and Observation on Network Operation

ALTO presents a new opportunity for managing network traffic by providing additional information to clients. In particular, the deployment of an ALTO server may shift network traffic patterns, and the potential impact to network operation can be large. An ISP providing ALTO may want to assess the benefits of ALTO as part of the management and operations (cf. [RFC7285]). For instance, the ISP might be interested in understanding whether the provided ALTO maps are effective, and in order to decide whether an adjustment of the ALTO configuration would be useful. Such insight can be obtained from a monitoring infrastructure. An ISP offering ALTO could consider the impact on (or integration with) traffic engineering and the deployment of a monitoring service to observe the effects of ALTO operations. The measurement of impacts can be challenging because ALTO-enabled applications may not provide related information back to the ALTO service provider.

To construct an effective monitoring infrastructure, the ALTO service provider should decide how to monitor the performance of ALTO and identify and deploy data sources to collect data to compute the performance metrics. In certain trusted deployment environments, it may be possible to collect information directly from ALTO clients. It may also be possible to vary or selectively disable ALTO guidance for a portion of ALTO clients either by time, geographical region, or some other criteria to compare the network traffic characteristics with and without ALTO. Monitoring an ALTO service could also be realized by third parties. In this case, insight into ALTO data may require a trust relationship between the monitoring system operator and the network service provider offering an ALTO service.

The required monitoring depends on the network infrastructure and the use of ALTO, and an exhaustive description is outside the scope of this document.

3.4.2. Measurement of the Impact

ALTO realizes an interface between the network and applications. This implies that an effective monitoring infrastructure may have to deal with both network and application performance metrics. This document does not comprehensively list all performance metrics that could be relevant, nor does it formally specify metrics.

The impact of ALTO can be classified regarding a number of different criteria:

• Total amount and distribution of traffic: ALTO enables ISPs to influence and localize traffic of applications that use the ALTO service. An ISP may therefore be interested in analyzing the impact on the traffic, i.e., whether network traffic patterns are shifted. For instance, if ALTO shall be used to reduce the inter-domain P2P traffic, it makes sense to evaluate the total amount of inter-domain traffic of an ISP. Then, one possibility is to study how the introduction of ALTO reduces the total inter-domain traffic (inbound and/our outbound). If the ISPs intention is to localize the traffic inside his network, the network-internal traffic distribution will be of interest. Effectiveness of localization can be quantified in different ways, e.g., by the load on core routers and backbone links, or by considering more advanced effects, such as the average number of hops that traffic traverses inside a domain.
• Application performance: The objective of ALTO is improve application performance. ALTO can be used by very different types applications, with different communication characteristics and requirements. For instance, if ALTO guidance achieves traffic localization, one would expect that applications achieve a higher throughput and/or smaller delays to retrieve data. If application-specific performance characteristics (e.g., video or audio quality) can be monitored, such metrics related to user experience could also help to analyze the benefit of an ALTO deployment. If available, selected statistics from the TCP/IP stack in hosts could be leveraged, too.

Of potential interest can also be the share of applications or customers that actually use an offered ALTO service, i.e., the adoption of the service.

Monitoring statistics can be aggregated, averaged, and normalized in different ways. This document does not mandate specific ways how to calculate metrics.

3.4.3. System and Service Performance

A number of interesting parameters can be measured at the ALTO server. [RFC7285] suggests certain ALTO-specific metrics to be monitored:

• Requests and responses for each service listed in a Information Directory (total counts and size in bytes).
• CPU and memory utilization
• ALTO map updates
• Number of PIDs
• ALTO map sizes (in-memory size, encoded size, number of entries)

This data characterizes the workload, the system performance as well as the map data. Obviously, such data will depend on the implementation and the actual deployment of the ALTO service. Logging is also recommended in [RFC7285].

3.4.4. Monitoring Infrastructures

Understanding the impact of ALTO may require interaction between different systems, operating at different layers. Some information discussed in the preceding sections is only visible to an ISP, while application-level performance can hardly be measured inside the network. It is possible that not all information of potential interest can directly be measured, either because no corresponding monitoring infrastructure or measurement method exists, or because it is not easily accessible.

One way to quantify the benefit of deploying ALTO is to measure before and after enabling the ALTO service. In addition to passive monitoring, some data could also be obtained by active measurements, but due to the resulting overhead, the latter should be used with care. Yet, in all monitoring activities an ALTO service provider has to take into account that ALTO clients are not bound to ALTO server guidance as ALTO is only one source of information, and any measurement result may thus be biased.

Potential sources for monitoring the use of ALTO include:

• Network Operations, Administration, and Maintenance (OAM) systems: Many ISPs deploy OAM systems to monitor the network traffic, which may have insight into traffic volumes, network topology, and bandwidth information inside the management area. Data can be obtained by SNMP, NETCONF, IP Flow Information Export (IPFIX), syslog, etc.
• Applications/clients: Relevant data could be obtained by instrumentation of applications.
• ALTO server: If available, log files or other statistics data could be analyzed.
• Other application entities: In several use cases, there are other application entities that could provide data as well. For instance, there may be centralized log servers that collect data.

In many ALTO use cases some data sources are located within an ISP network while some other data is gathered at application level. Correlation of data could require a collaboration agreement between the ISP and an application owner, including agreements of data interchange formats, methods of delivery, etc. In practice, such a collaboration may not be possible in all use cases of ALTO, because the monitoring data can be sensitive, and because the interacting entities may have different priorities. Details of how to build an over-arching monitoring system for evaluating the benefits of ALTO are outside the scope of this memo.

3.5. Abstract Map Examples for Different Types of ISPs

3.5.1. Small ISP with Single Internet Uplink

The ALTO protocol does not mandate how to determine costs between endpoints and/or determine map data. In complex usage scenarios this can be a non-trivial problem. In order to show the basic principle, this and the following sections explain for different deployment scenarios how ALTO maps could be structured.

For a small ISP, the inter-domain traffic optimizing problem is how to decrease the traffic exchanged with other ISPs, because of high settlement costs. By using the ALTO service to optimize traffic, a small ISP can define two "optimization areas": one is its own network; the other one consists of all other network destinations. The cost map can be defined as follows: the cost of a link between clients of the inner ISP's network is lower than between clients of the outer ISP's network and clients of inner ISP's network. As a result, a host with an ALTO client inside the network of this ISP will prefer retrieving data from hosts connected to the same ISP.

An example is given in Figure 9. It is assumed that ISP A is a small ISP only having one access network. As operator of the ALTO service, ISP A can define its network to be one optimization area, named as PID1, and define other networks to be the other optimization area, named as PID2. C1 is denoted as the cost inside the network of ISP A. C2 is denoted as the cost from PID2 to PID1, and C3 from PID1 to PID2. For the sake of simplicity, in the following C2=C3 is assumed. In order to keep traffic local inside ISP A, it makes sense to define: C1<C2

           -----------
       ////           \\\\
     //                   \\
   //                       \\                  /-----------\
  | +---------+               |             ////             \\\\
  | | ALTO    |  ISP A        |    C2      |    Other Networks   |
 |  | Service |  PID 1         <-----------     PID 2
  | +---------+  C1           |----------->|                     |
  |                           |  C3 (=C2)   \\\\             ////
   \\                       //                  \-----------/
     \\                   //
       \\\\           ////
           -----------

   HTTP/1.1 200 OK
   ...
   Content-Type: application/alto-networkmap+json

   {
    ...
     "network-map" : {
       "PID1" : {
         "ipv4" : [
           "192.0.2.0/24",
           "198.51.100.0/25"
         ]
       },
       "PID2" : {
         "ipv4" : [
           "0.0.0.0/0"
         ],
         "ipv6" : [
           "::/0"
         ]
       }
     }
   }

   HTTP/1.1 200 OK
   ...
   Content-Type: application/alto-costmap+json

   {
       ...
       "cost-type" : {"cost-mode"  : "numerical",
                      "cost-metric": "routingcost"
       }
     },
     "cost-map" : {
       "PID1": { "PID1": C1,  "PID2": C2 },
       "PID2": { "PID1": C2,  "PID2": 0 },
     }
   }

 +------------------------------------+         +----------------+
 | ISP A   +---------------+          |         |                |
 |         |    Backbone   |          |   C7    |      ISP B     |
 |     +---+    Network    +----+     |<--------+      PID 4     |
 |     |   +-------+-------+    |     |         |                |
 |     |           |            |     |         |                |
 |     |           |            |     |         +----------------+
 | +---+--+     +--+---+     +--+---+ |
 | |AN A  |  C4 |AN B  |  C5 |AN C  | |
 | |PID 1 +<--->|PID 2 |<--->+PID 3 | |
 | |C1    |     |C2    |     |C3    | |         +----------------+
 | +---+--+     +------+     +--+---+ |         |                |
 |     ^                        ^     |   C8    | Other Networks |
 |     |                        |     |<--------+ PID 5          |
 |     +------------------------+     |         |                |
 |                  C6                |         |                |
 +------------------------------------+         +----------------+

 +-----------------------------------------------------------------+
 |                                                                 |
 |  ISP A                 +-------------+                          |
 |               +--------+   ALTO      +---------+                |
 |               |        |   Service   |         |                |
 |               |        +------+------+         |                |
 |               |               |                |                |
 |               |               |                |                |
 |               |               |                |                |
 |       +-------+-------+       | C6    +--------+------+         |
 |       |     AN A      |<--------------|      AN B     |         |
 |       |     PID 2     |   C7  |       |      PID 3    |         |
 |       |     C2        |-------------->|      C3       |         |
 |       +---------------+       |       +---------------+         |
 |             ^    |            |              |     ^            |
 |             |    |            |              |     |            |
 |             |    |C4          |              |     |            |
 |          C5 |    |            |              |     |            |
 |             |    |   +--------+---------+    |     |            |
 |             |    +-->|  Mobile Network  |<---+     |            |
 |             |        |  PID 1           |          |            |
 |             +------- |  C1              |----------+            |
 |                      +------------------+                       |
 +-----------------------------------------------------------------+

+--------------+      +--------+      +--------+     +--------------+
|100.1.102.0/24+------+   R6   |      |   R7   +-----+100.1.103.0/24|
+--------------+      +----+---+      +----+---+     +--------------+
                           |               |
+--------------+           |               |
|     AS 2     |           |               |
| 100.2.0.0/16 |           |               |
+-------+------+           |               |
        |                  |               |
        |                  |               |
    +---+----+        +----+---+      +----+---+     +--------------+
    |   R1   +--------+   R3   +------+   R5   |-----+100.1.104.0/24|
    +---+----+        +----+---+      +----+---+     +--------------+
        |     \      /     |               |
        |      \    /      |               |
        |       \  /       |               |         +--------------+
        |        \/        |               |         |     AS 4     |
        |        /\        |               |         | 100.4.0.0/16 |
        |       /  \       |               |         +------+-------+
        |      /    \      |               |                |
        |     /      \     |               |                |
    +---+----+        +----+---+      +----+---+     +------+-------+
    |   R2   |        |   R4   |      |   R8   +-----+     AS 3     |
    +---+----+        +----+---+      +----+---+     | 100.3.0.0/16 |
        |                  |               |         +--------------+
        |                  |               |
        |                  |               |
+-------+------+           |          +----+---+     +--------------+
|100.1.101.0/24|           +----------+   R9   +-----+100.1.105.0/24|
+--------------+                      +--------+     +--------------+

           R1   R2   R3   R4   R5   R6   R7   R8   R9
       R1   0   15   15   20    -    -    -    -    -
       R2  15    0   20    -    -    -    -    -    -
       R3  15   20    0    5    5   10    -    -    -
       R4  20    -    5    0    5    -    -    -   20
       R5   -    -    5    5    0    -   10   10    -
       R6   -    -   10    -    -    0    -    -    -
       R7   -    -    -    -   10    -    0    -    -
       R8   -    -    -    -   10    -    -    0   10
       R9   -    -    -   20    -    -    -   10    0

   ID:  2

   Neighbor IDs:  1,3 (R1, R3)

   Endpoints:

      Endpoint:  100.1.101.0/24

      Metric:  10 (default client subnet metric)

      ASNumber:  1 (our own AS)

      ASDistance:  0

   Host Name:  R2

   ID:  8

   Neighbor IDs:  5,9 (R5, R9)

   Endpoints:

      Endpoint:  100.3.0.0/16

      Metric:  100

      ASNumber:  3

      ASDistance:  1

      Endpoint:  100.4.0.0/16

      Metric:  200

      ASNumber:  4

      ASDistance:  2

   Host Name:  R8

   R1->R2:

   Source ID:  1

   Destination ID:  2

   Weight:  15

   Bandwidth:  10Gbit/s

   Utilization:  0.1

   Latency:  2ms

   R2->R1:

   Source ID:  2

   Destination ID:  1

   Weight:  15

   Bandwidth:  10Gbit/s

   Utilization:  0.55

   Latency:  5ms

       PID  |  Endpoints
   ---------+-----------------------------------
        R1  |  100.2.0.0/16
        R2  |  100.1.101.0/24
        R5  |  100.1.104.0/24
        R6  |  100.1.102.0/24
        R7  |  100.1.103.0/24
        R8  |  100.3.0.0/16, 100.4.0.0/16
        R9  |  100.1.105.0/24
    default |  0.0.0.0/0, ::/0

      PID  | default | R1  | R2  | R5  | R6  | R7  | R8  | R9  |
   --------+---------+-----+-----+-----+-----+-----+-----+-----|
   default |    x    |  x  |  x  |  x  |  x  |  x  |  x  |  x  |
      R1   |    x    |  0  |  2  |  3  |  3  |  4  |  4  |  3  |
      R2   |    x    |  2  |  0  |  3  |  3  |  4  |  4  |  4  |
      R5   |    x    |  3  |  3  |  0  |  3  |  2  |  2  |  3  |
      R6   |    x    |  3  |  3  |  3  |  0  |  4  |  4  |  4  |
      R7   |    x    |  4  |  4  |  2  |  4  |  0  |  3  |  4  |
      R8   |    x    |  4  |  4  |  2  |  4  |  3  |  0  |  2  |
      R9   |    x    |  3  |  4  |  3  |  4  |  4  |  2  |  0  |

                         ,-------.         +-----------+
       ,---.          ,-'         ========>|   Peer 1  |********
    ,-'     `-.      /     ISP 1  V  \     |ALTO Client|       *
   /           \    / +-------------+ \    +-----------+       *
  /    ISP X    \   | + ALTO Server | |    +-----------+       *
 /               \  \ +-------------+<====>|   Peer 2  |       *
;   +---------+   :  \               /     |ALTO Client|****** *
|   | Global  |   |   `-.         ,-'      +-----------+     * *
|   | Tracker |   |      `-------'                           * *
|   +---------+   |      ,-------.         +-----------+     * *
:        *        ;   ,-'         ========>|   Peer 3  |     * *
 \       *       /   /     ISP 2  V  \     |ALTO Client|**** * *
  \      *      /   / +-------------+ \    +-----------+   * * *
   \     *     /    | | ALTO Server | |    +-----------+   * * *
    `-.  *  ,-'     \ +-------------+<====>|   Peer 4  |** * * *
       `-*-'         \               /     |ALTO Client| * * * *
         *            `-.         ,-'      +-----------+ * * * *
         *               `-------'                       * * * *
         *                                               * * * *
         *******************************************************
    Legend:
    === ALTO protocol
    *** Application protocol

                              ,-------.
       ,---.               ,-'         `-.   +-----------+
    ,-'     `-.           /     ISP 1     \  |   Peer 1  |********
   /           \         / +-------------+ \ |           |       *
  /    ISP X    \   ++====>| ALTO Server |  )+-----------+       *
 /               \  ||   \ +-------------+ / +-----------+       *
; +-----------+   : ||    \               /  |   Peer 2  |       *
| |  Tracker  |<====++     `-.         ,-'   |           |****** *
| |ALTO Client|   |           `-------'      +-----------+     * *
| +-----------+<====++        ,-------.                        * *
:        *        ; ||     ,-'         `-.   +-----------+     * *
 \       *       /  ||    /     ISP 2     \  |   Peer 3  |     * *
  \      *      /   ||   / +-------------+ \ |           |**** * *
   \     *     /    ++====>| ALTO Server |  )+-----------+   * * *
    `-.  *  ,-'          \ +-------------+ / +-----------+   * * *
       `-*-'              \               /  |   Peer 4  |** * * *
         *                 `-.         ,-'   |           | * * * *
         *                    `-------'      +-----------+ * * * *
         *                                                 * * * *
         *                                                 * * * *
         *********************************************************
    Legend:
    === ALTO protocol
    *** Application protocol

                   ISP 1  ,-------.         +-----------+
        ,---.          +-------------+******|   Peer 1  |
     ,-'     `-.      /|   Tracker   |\     |           |
    /           \    / +-------------+****  +-----------+
   /    ISP X    \   |       ===       | *  +-----------+
  /               \  \ +-------------+ / *  |   Peer 2  |
 ;   +---------+   :  \| AlTO Server |/  ***|           |
 |   | Global  |   |   +-------------+      +-----------+
 |   | Tracker |   |      `-------'
 |   +---------+   |                        +-----------+
 :        *        ;      ,-------.         |   Peer 3  |
  \       *       /    +-------------+  ****|           |
   \      *      /    /|   Tracker   |***   +-----------+
    \     *     /    / +-------------+ \    +-----------+
     `-.  *  ,-'     |       ===       |    |   Peer 4  |**
        `-*-'        \ +-------------+ /    |           | *
          *           \| ALTO Server |/     +-----------+ *
          *            +-------------+                    *
          *        ISP 2  `-------'                       *
          *************************************************
     Legend:
     === ALTO protocol
     *** Application protocol

Peer w. ALTO cli.            Tracker               ALTO Server 
--------+--------       --------+--------       --------+--------
        | F1 Tracker query      |                       |    
        |======================>|                       |
        | F2 Tracker reply      |                       |    
        |<======================|                       |    
        | F3 ALTO protocol query                        |
        |---------------------------------------------->|    
        | F4 ALTO protocol reply                        |
        |<----------------------------------------------|    
        |                       |                       |    

====  Application protocol (i.e., tracker-based P2P app protocol)
----  ALTO protocol

      Peer               Tracker w. RDAC           ALTO Server 
--------+--------       --------+--------       --------+--------
        | F1 Tracker query      |                       |    
        |======================>|                       |
        |                       | F2 ALTO cli. p. query |
        |                       |---------------------->|    
        |                       | F3 ALTO cli. p. reply |
        |                       |<----------------------|
        | F4 Tracker reply      |                       |    
        |<======================|                       |    
        |                       |                       |    

====  Application protocol (i.e., tracker-based P2P app protocol)
----  ALTO protocol

+--------------------+
| CDN Request Router |
|  with ALTO Client  |
+--------------------+
          /\
          || ALTO protocol
          ||
          \/
      +---------+
      |  ALTO   |
      | Server  |
      +---------+
           :
           : Provisioning protocol
           :
     ,-----------.
  ,-'  Source of  `-.
 (    topological    )
  `-. information ,-'
     `-----------'

User Agent                  Request Router                 Surrogate
     |                             |                           |
     |     F1 Initial Request      |                           |
     +---------------------------->|                           |
     |                             +--+                        |
     |                             |  | F2 Surrogate Selection |
     |                             |<-+       (using ALTO)     |
     |   F3 Redirection Response   |                           |
     |<----------------------------+                           |
     |                             |                           |
     |     F4 Content Request      |                           |
     +-------------------------------------------------------->|
     |                             |                           |
     |                             |          F5 Content       |
     |<--------------------------------------------------------+
     |                             |                           |

                    +---------------+
                    |  Customer's   |
                    |   management  |
                    |  application  |.
                    | (ALTO client) |  .
                    +---------------+    .  VPN provisioning
                           /\              . (out-of-scope)
                           || ALTO           .
                           \/                  .
                 +---------------------+       +----------------+
                 |     ALTO server     |       | VPN portal/OSS |
                 |   provided by NSP   |       | (out-of-scope) |
                 +---------------------+       +----------------+
                            : VPN network
                            : and cost maps
                            :
                  /---------:---------\ Network service provider
                  |         :         |
     +-------+   _______________________   +-------+
     | App a | ()_____. .________. .____() | App d |
     +-------+    |   | |        | |  |    +-------+
                  \---| |--------| |--/           
                      | |        | |
                      |^|        |^| Customer VPN
                       V          V
                   +-------+  +-------+
                   | App b |  | App c |
                   +-------+  +-------+

         +--------------+                +------+
         | ISP 1 network+----------------+Peer 1|
         +-----+--------+                +------+
         |
+--------+------------------------------------------------------+
|        |                                      ISP 2 network   |
|  +---------+                                                  |
|  |L1 Cache |                                                  |
|  +-----+---+                                                  |
|        +--------------------+----------------------+          |
|        |                    |                      |          |
| +------+------+      +------+-------+       +------+-------+  |
| | AN1         |      | AN2          |       | AN3          |  |
| | +---------+ |      | +----------+ |       |              |  |
| | |L2 Cache | |      | |L2 Cache  | |       |              |  |
| | +---------+ |      | +----------+ |       |              |  |
| +------+------+      +------+-------+       +------+-------+  |
|        |                                           |          |
|        +--------------------+                      |          |
|        |                    |                      |          |
| +------+------+      +------+-------+       +------+-------+  |
| | SUB-AN11    |      | SUB-AN12     |       | SUB-AN31     |  |
| | +---------+ |      |              |       |              |  |
| | |L3 Cache | |      |              |       |              |  |
| | +---------+ |      |              |       |              |  |
| +------+------+      +------+-------+       +------+-------+  |
|        |                    |                      |          |
+--------+--------------------+----------------------+----------+
         |                    |                      |
     +---+---+            +---+---+                  |
     |       |            |       |                  |
  +--+--+ +--+--+      +--+--+ +--+--+            +--+--+
  |Peer2| |Peer3|      |Peer4| |Peer5|            |Peer6|
  +-----+ +-----+      +-----+ +-----+            +-----+