Internet Architecture Board F. Bronzino Internet-Draft Universite Savoie Mont Blanc Intended status: Informational E. Culley Expires: 1 May 2021 Comcast N. Feamster S. Liu University of Chicago J. Livingood Comcast P. Schmitt Princeton University 28 October 2020 IAB COVID-19 Workshop: Interconnection Changes in the United States draft-feamster-livingood-iab-covid19-workshop-01 Abstract During the early weeks and months of the COVID-19 pandemic, significant changes to Internet usage occurred as a result of a sudden global shift to people working, studying and quarantining at home. One aspect that this affected was interconnection between networks, which this paper studies. This paper explores some of the effects of these changes on Internet interconnection points, in terms of utilization, traffic ratios, and other performance characteristics such as latency. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on 1 May 2021. Bronzino, et al. Expires 1 May 2021 [Page 1] Internet-Draft COVID-19 Interconnection Traffic Effects October 2020 Copyright Notice Copyright (c) 2020 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/ license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 2. Long-Term Interconnection Data . . . . . . . . . . . . . . . 3 3. Comcast's COVID-19-Related Experiences . . . . . . . . . . . 4 4. Detailed Statistical Observations . . . . . . . . . . . . . . 6 5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 6 6. Security Considerations . . . . . . . . . . . . . . . . . . . 6 7. Normative References . . . . . . . . . . . . . . . . . . . . 6 8. Informative References . . . . . . . . . . . . . . . . . . . 6 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 7 1. Introduction During the early weeks and months of the COVID-19 pandemic [WHO-Declaration], significant changes to Internet usage occurred as a result of a sudden global shift to people working, studying and quarantining at home. One aspect that this affected was interconnection between networks, which this paper studies. In 2016, the Interconnection Measurement Project [Tinker-Blog] was launched. The IMP platform initially collected interconnection- related data from seven U.S.-based cable-based Internet Service Providers (ISPs) and an [ArXiv-Paper] shared some of the details and findings. The initial focus of the project was to explore utilization characteristics of interconnect links during a period of time when video traffic was steadily increasing. The project concluded that there was ample aggregate capacity on interconnect links between ISPs and peers (including content providers), and that ISPs continually added capacity to their interconnects to keep pace with the growth in traffic. Bronzino, et al. Expires 1 May 2021 [Page 2] Internet-Draft COVID-19 Interconnection Traffic Effects October 2020 This IAB paper looks specifically at the long-term interconnection data from one of those ISPs, Comcast. We examine the long-term pre- COVID-19 trend as well as what occurred as COVID-19 impacted the Internet from February 2020 through August 2020. We also include observations from Comcast concerning interconnection changes during this timeframe. We hope that this information will be useful to the IAB workshop and the Internet community more broadly. It may serve as an interesting and useful historical reference in the future. 2. Long-Term Interconnection Data The IMP platform collected interconnection data starting in 2016, roughly four years prior to the COVID-19-driven shift in Internet usage. This provides an interesting capability for a before and after view of interconnection. A full explanation of the data can be found in Section 3 of the [ArXiv-Paper]. At a high level, the maindata collected encompasses: * Timestamp (representing a five-minute interval) * Region (representing an aggregated link group) * Anonymized partner network * Access ISP * Total ingress bytes * Total egress bytes * Capacity Utilization is captured based on sampled IPFIX records, with a packet sampling rate of 1/1,000. SNMP polling data yields information about the capacity on each link. The IMP platform does not have direct access to partner network identities---this dataset only includes an anonymous identifier corresponding to that particular partner. However, in collaboration with Comcast, IMP has worked to identify specific partner networks in the dataset for specific peers in the interest of detail COVID-19-related study. Focusing on Comcast within this dataset reveals several trends, in both aggregate capacity and utilization, as well as how utilization and capacity changed during the period of the COVID-19 pandemic. The data also reveals how traffic volumes changed for specific peers during this time period. Bronzino, et al. Expires 1 May 2021 [Page 3] Internet-Draft COVID-19 Interconnection Traffic Effects October 2020 The IMP data that tracks capacity shows Comcast steadily adding capacity from mid-2018 to present, with a significant increase in the rate of additional capacity from the beginning of 2020, increasing further in the second and third quarters of 2020. Specifically, we analyzed the rate at which capacity was added during these periods, on a month-by-month basis. We found that Comcast was adding aggregate capacity on its interconnects at *nearly twice* the rate as it was being added during 2019. Over a long timeframe, Comcast's daily peak hour interconnection port utilization on its busiest links has remained consistently around 90%. For our analysis, we consider the links that represent the 99th percentile in terms of utilization. During the period at the end of March and the beginning of April 2020, daily peak utilization briefly increased to about 97% but steadily returned to normal levels in a matter of weeks by the end of April 2020, as Comcast quickly increased the rate at which it added capacity to keep pace with growing traffic demands. We also explored traffic volumes associated with each Comcast peer, comparing the ranks and volumes of each individual peer as measured on September 1, 2020 as compared to January 1, 2020, in both the upstream and downstream directions. Doing so allowed us to understand both the magnitude of changes in traffic volumes and ratios, as well as how specific individual peers deviated from normal baseline behavior, in terms of both upstream and downstream traffic rates and ratios. The data suggests that both the upstream and downstream directions saw some changes, although the deviations in upstream traffic patterns were greater: the Pearson coefficient for downstream traffic was 0.977, whereas for upstream traffic the Pearson correlation coefficient was only 0.935, indicating a more significant shift in upstream traffic ratios during this timeframe. 3. Comcast's COVID-19-Related Experiences Comcast observed a wide range of significant changes in Internet usage as residential users remained at home and shifted to working and studying from home. Changes in usage patterns observed in the access network in sum led to changes in the traffic flowing to interconnected networks. As the pandemic developed, there was a wide variety of changes in traffic volumes. At some locations in the network little change was detected while other locations saw a huge growth in the volume of traffic. At the peak of the surge, in March and April 2020, the average amount of growth observed across locations and types was roughly 33%. Voice & video conferencing (conferencing hereafter) jumped as much as 285% and Wi-Fi use increased 36% among our Xfinity Mobile (MVNO) customers Bronzino, et al. Expires 1 May 2021 [Page 4] Internet-Draft COVID-19 Interconnection Traffic Effects October 2020 [June-Blog] [July-Blog]. As this continued into May 2020 we observed conferencing remained up roughly 210-285%, VPN traffic up 30-40% and gaming downloads up 20-80%, and web-based streaming video consumption up 20-40% [May-Blog]. In this several week period, traffic essentially grew at or more than it had in the prior year, which was significant growth in a short period of time. In the months following the onset of the pandemic Comcast observed: * Overall average downstream peak growth is up 13% (up as much as 20% at times). * Overall average upstream peak growth is up 36%. * In the access network, an average of 771 network augments per week were performed, peaking at over 1,800 in a single week and over with over 7 weeks with more than 1,000 per week. For comparison, the average earlier in the year was roughly 350 per week. * In the core network, over 500 augments were made in order to add 146 Tbps in capacity. * On a daily basis roughly 700,000 automated speed tests from customer homes were conducted in order to gauge the customer experience during this time. Average speeds to customers (both downstream and upstream) have generally remained at or above 105% of advertised speeds since March 1, 2020 in all regions. National average speeds have remained between 110% - 115% of advertised speeds over the same period. [NF-Paper-1] and [NF-Paper-2] * The share of streaming video as a percentage of total traffic is declined slightly from 67% to 63%. Despite strong growth, conferencing occupies a small share of total traffic and grew from 1% share to 4%. But gaming software released have driven significant download spikes since late April 2020. * For interconnection, peering coordinators across operators worked cooperatively and quickly to cut through any red tape and add new capacity as quickly as possible. * In 2019, settlement free interconnection capacity [SFI-Policy] - a subset of overall interconnect types - grew by 15%. Between January and August 2020, driven by COVID-19 changes, there was an overall 37% increase in capacity from that prior 2019 level. And between March and October 2020 one Settlement Free Peer alone increased 115%. Bronzino, et al. Expires 1 May 2021 [Page 5] Internet-Draft COVID-19 Interconnection Traffic Effects October 2020 * Between March and October 2020 from Comcast observed other notable per-peer traffic increases of 245% and 3,900%. 4. Detailed Statistical Observations As briefly mentioned in previous sections, downstream traffic rates from many partners remained stable---the Pearson correlation coefficient for peak download rates between January 1, 2020 and September 2, 2020 is 0.977, indicating that the peak download rates to most peers was similar between these two time periods. On the other hand, certain peers experienced either a significant increase or decrease in peak download rates---often by two or three orders of magnitude. Similarly, other peers experienced a decrease in peak downstream rates by several orders of magnitude. On the other hand, upstream traffic rates were far less stable: In contrast, the Pearson coefficient for upload rates between January 1, 2020 and September 2, 2020 was only 0.935, suggesting more a more significant deviation in peak upstream rates. As with peak download rates, some peers experienced significant decreases, as well: in one outlier case, peak rate decreased by almost five orders of magnitude. A small handful of peers saw similar decreases. Yet, a far greater number of peers saw increases in peak upload rates by two to three orders of magnitude. 5. IANA Considerations This document includes no request to IANA. 6. Security Considerations This document includes no security considerations. 7. Normative References 8. Informative References [ArXiv-Paper] Feamster, NF., "Revealing Utilization at Internet Interconnection Points", 5 September 2016, . [July-Blog] Nafshi, EN., "COVID-19 Network Report: How A Smart Network Delivered Speed and Stability When it Mattered", 13 July 2020, . Bronzino, et al. Expires 1 May 2021 [Page 6] Internet-Draft COVID-19 Interconnection Traffic Effects October 2020 [June-Blog] Werner, TW., "Cresting the Wave: The Factors that Powered our Network Through the COVID-19 Surge", 15 June 2020, . [May-Blog] Comcast, "COVID-19 Network Update", 20 May 2020, . [NF-Paper-1] Jones, AJ., Sevcik, PS., and AL. Lacy, "NetForecast Design Audit Report of Comcast's Network Performance Measurement System", April 2020, . [NF-Paper-2] Jones, AJ., Sevcik, PS., and AL. Lacy, "NetForecast's Report on Comcast's Network Performance Measurement System Results Data", May 2020, . [SFI-Policy] Comcast Cable Communications Management, LLC, "Comcast Settlement-Free Interconnection (SFI) Policy", October 2013, . [Tinker-Blog] Feamster, NF., "Interconnection Measurement Project Website", 9 May 2016, . [WHO-Declaration] Adhanom Ghebreyesus, TAG., "WHO Director-General's opening remarks at the media briefing on COVID-19 - 11 March 2020", 11 March 2020, . Authors' Addresses Bronzino, et al. Expires 1 May 2021 [Page 7] Internet-Draft COVID-19 Interconnection Traffic Effects October 2020 Francesco Bronzino Universite Savoie Mont Blanc Annecy-le-Vieux France Email: francesco.bronzino@univ-smb.fr Elizabeth Culley Comcast Mount Laurel, NJ United States of America Email: elizabeth_culley@comcast.com Nick Feamster University of Chicago Chicago, IL United States of America Email: feamster@uchicago.edu Shinan Liu University of Chicago Chicago, IL United States of America Email: shinanliu@uchicago.edu Jason Livingood Comcast Philadelphia, PA United States of America Email: jason_livingood@comcast.com Paul Schmitt Princeton University Princeton, NJ United States of America Email: pschmitt@cs.princeton.edu Bronzino, et al. Expires 1 May 2021 [Page 8]