Working for an Internet that is open, globally connected, and secure.
Category:Measuring the Internet
The more we understand the evolution of the Internet, the better we can address emerging issues. We’ll create a reliable place to help people find data and trends shaping the Internet’s infrastructure and assess the ‘health’ of the Internet model of networking.
Here at the Internet Society, we believe that the Internet is for everyone. Our work centers on increasing the Internet’s reach, reliability and resilience, as well as ensuring that the network of networks remains open, globally connected, secure, and trustworthy.
But how do we assess whether our efforts – and the efforts of the global ecosystem of organizations that facilitate the smooth functioning of the Internet – are working? How can we see where protocols, such as IPv6, are being deployed and at what rate so we can better understand where more education on the benefits of such technologies might be helpful? Where can policy makers find a comprehensive set of data from various sources to help show decision makers that Internet Showdowns damage local economies and potentially harm citizens?
A Single Platform
There are many people, projects and organizations that are collecting data on various facets of the Internet, but there’s no single site that provides a curated set of insights. So, to help everyone gain deeper, data-driven insight into the Internet, the Internet Society is building a tool that consolidates trusted third-party Internet measurement data from various sources into a single platform – insights.internetsociety.org.
We’ll use the data presented on the Insights platform to examine Internet trends, generate reports, and tell data-driven stories so that policymakers, researchers, journalists, network operators, civil society groups and others can better understand the health, availability and evolution of the Internet.
It’s important to note that the Internet Society is not collecting data or performing measurements itself. Instead, we’re collating and curating publicly available data and making it available in one place so that users do not have to go to many individual sites to get the multiple sources of information they need. Our data sources include Internet Society Organization Members Facebook, Google and Oracle, among others.
We’ve also put in place agreements and partnerships with several other organizations in the Internet measurement community and are continuing these outreach efforts to secure access to more data. These key strategic relationships will enable us to present and use Internet measurement data that is currently not publicly available and data that might have limited public availability. And, as the Insights platform develops, these relationships will also facilitate collaboration with industry leaders on other aspects of data collection and Internet measurements.
One such relationship is with Internet Society Organization Member, AFRINIC, the Regional Internet Registry (RIR) for Africa. Signed on 23 July 2020, the agreement builds upon a strategic, long-term partnership agreement held between both organizations aimed at strengthening collaboration throughout Africa. The primary goal of this partnership is to drive the development of the Internet in the region through projects and research related to Internet measurements, Internet resilience, routing security, open Internet standards, and Internet Exchange Points (IXPs).
An agreement has also been signed with the Center for Applied Internet Data Analysis (CAIDA). CAIDA’s long-standing work on monitoring the Internet in near-real time to identify macroscopic Internet outages will further strengthen the Insights platform’s focus on Internet Shutdowns and Internet Resilience.
We’ve been working on the Insights platform as part of our Measuring the Internet project since the beginning of 2020 and are making good progress. Development work is already under way and we plan to launch by the end of the 2020. As we want to make Insights available as soon as possible, we’ll initially launch with two focus areas, Internet Shutdowns and Enabling Technologies. Meanwhile, work will continue on finalizing metrics and data sources for our other two focus areas, Internet Resilience and the Internet Way of Networking (IWN), as well as on outreach and partnership development.
Keep an eye on the Measuring the Internet project homepage for updates about Insights.
Last week, we announced an expanded partnership with AFRINIC, the Regional Internet Registry for the African region. On Friday, 24 July, Eddy Kayihura, Chief Executive Officer at AFRINIC, and I, on behalf of the Internet Society, signed a Memorandum of Understanding (MoU) between the two organizations to work on several projects including Internet measurements, routing security, and infrastructure and community development.
Right after the virtual signing ceremony, we described the first collaborative activity under the new MoU – the Africa Internet Measurements. Part of our Measuring the Internet project, the effort aims to tackle the problem of Internet resilience and reliability in the continent.
Although Africa has significantly increased Internet penetration in the last decade, the continent must improve the resilience and the reliability of its Internet infrastructure to pave the way for future innovations and technological advancements as expressed in the African Union’s 2063 agenda. Without proper measurements and data, we don’t know where the problem is, what we need to improve, or if our solutions work. Much of the available Internet measurement data relating to Africa measures only specific types of Internet traffic and not overall Internet resilience, which is the ability of the network to provide and maintain an acceptable level of service in the face of faults and challenges to normal operation.
Work is already underway, with teams from AFRINIC and the Internet Society collaborating on establishing a framework of metrics that will help to determine Internet resilience throughout Africa. This framework will eventually inform the Internet Resilience focus area of the Measuring the Internet project’s Insights platform (currently under development), which will provide actionable data to governments and policymakers as well as provide historical data on the growth of, and improvements to, the continent’s Internet resilience.
Eddy Kayihura, Chief Executive Officer at AFRINIC, and Dawit Bekele, Regional Vice President – Africa at the Internet Society, just after signing the MoU
We’ve been working with AFRINIC for many years; this partnership agreement merely formalizes and extends a relationship that has already expanded Internet access across Africa through community networks, promoted open Internet standards, built Internet Exchange Points, and more. We’re also honored to have AFRINIC as an Internet Society Organization Member.
Watch the full signing ceremony and Measurements Q&A in the video below, and if you’re interested in getting involved with the Measuring the Internet project, contact us at firstname.lastname@example.org.
Although the COVID-19 pandemic is, unfortunately, far from over, the Internet continues to be resilient, supporting the additional demands that we have placed on it, including the rapid growth in online learning, work videoconferences, e-commerce, streaming video entertainment, and more.
Because the Internet exists as a network of networks, this resilience is largely due to the planning, actions, and cooperation of all of the interconnected participants. These participants include, but are certainly not limited to, network providers, Internet Exchange Points (IXPs), and Content Delivery Networks (CDNs).
We’ll hear from panel participants Scott Rasmussen of NYC Mesh, a community network provider, Jana Iyengar of Fastly, a CDN platform, and Jim Troutman of NNENIX, an IXP.
Register now to join us and learn more about the traffic shifts that they observed on their networks and platforms, associated challenges and how they handled them, and how they are ensuring they remain resilient in this new normal.
To help everyone gain deeper insight into the Internet, we’re building a tool that consolidates trusted third-party Internet measurement data from various sources into a single platform. We’ll use this data to examine trends, generate reports, and tell data-driven stories. Find out more about the Measuring the Internet project.
If your household is anything like mine, your Internet connection has experienced a significant increase in usage over these last several months. We’re streaming more and more media each day, and we’re on seemingly endless hours of videoconferences for work and for school. While all of that streaming media consumes downstream capacity, those videoconferences can generate a significant amount of upstream traffic. I’m fortunate enough to have fiber-based broadband connectivity that can easily handle this traffic, but I know others aren’t as lucky. They’re stuck with copper-based connections or satellite links that struggle to deliver streaming media or video calls with any sort of viewable quality.
Across this spectrum of “last mile” Internet connections, I looked at the impact from both a provider and user perspective. What kind of traffic growth have last mile network providers experienced? What steps have these providers taken to ensure they have sufficient capacity? And most importantly for end users, how has increased traffic impacted last mile connection speeds?
The network connections from customer- and subscriber-facing Internet service providers are often referred to as last mile networks. These are Internet services delivered over a notional distance – the “last mile” – to subscriber premises, such as homes and offices, and even devices, like cell phones.
Within the United States, Comcast, Spectrum, AT&T, and Verizon are among the largest providers of consumer last mile connectivity. Based on data available as of late May, in comparison to pre-COVID baseline traffic levels (generally late February/early March), these last mile providers saw growth of more than 30% in upstream traffic, with downstream traffic increasing on the order of 20%. As expected, application traffic volumes increased significantly as well.
Comcast reported experiencing a 33% increase in upstream traffic and a 13% increase in downstream traffic, but also noted that its network traffic is beginning to plateau in most markets, including those impacted early by COVID-19.
In mid-April, Spectrum reported an increase of nearly 20% for downstream traffic and 32% for upstream traffic.
As of the end of April, AT&T reported a 22% increase in core network traffic (which includes business, home broadband, and wireless usage) as compared to the end of February.
In mid-May, Verizon reported significant growth at an application level, highlighting an 81% increase in VPN usage, a 1,200% increase in collaboration tool usage, and video streaming 36% higher than typical pre-COVID days.
In Europe, the Body of European Regulators for Electronic Communications (BEREC) publishes summary reports based on data collected from National Regulatory Authorities (NRAs) regarding Internet traffic across network providers in EU member states. The report published on 8 May stated, “In general, traffic on fixed and mobile networks has increased during the Covid-19 crisis, but no major congestion issues have occurred. For more than six weeks, a growing number of NRAs have been reporting a stabilization in overall traffic.” Other NRAs reported they “have been detecting a decrease in overall Internet traffic from the peak that was reached after the beginning of the Covid-19 crisis,” while several others “confirmed that data traffic keeps decreasing, but still remains at a higher level compared to the period before the outbreak of Covid-19.” Similar observations were made in the 20 May report.
In Venezuela, Movistar’s radical growth in traffic placed significant strain on the network, since the network was already operating at maximum capacity, according to a letter from the company’s president. Because maintaining sufficient excess capacity within a network to be able to absorb unexpected (and planned) traffic spikes is considered an industry best practice, Movistar’s issues speak to the greater fragility of telecommunications infrastructure within Venezuela.
Other network providers are working to maintain resilience by accelerating planned investments to add additional capacity. One example is Vodacom South Africa, which announced on 15 April it would spend R500 million (approximately USD 27 million) within two months to add network capacity, especially as it sees peak traffic levels throughout the day.
However, the increased traffic appears to be less of an issue for providers in other regions. Executives from United States-based broadband provider Cox Communications noted its current network has enough capacity to handle the sudden shift in usage trends resulting from increases in online activity during the pandemic, and attributed its resilience to execution of long-term network upgrade plans. NTT Communications in Japanreassured customers “the company’s telecommunications facilities are built to handle traffic even if it doubles the current peak amount.” As of mid-April, traffic on NTT’s broadband service was approximately 28 percent higher than late February.
Within the United Kingdom, OpenReach noted its existing network is already built to handle peak capacity. BT’s Chief Technology Officer stated “We have more than enough capacity in our UK broadband network to handle mass-scale homeworking.” TalkTalk said it was confident it could manage an increase in traffic volumes as its services regularly experience peaks in demand.
Connection Speed Trends
Although you may be paying your Internet service provider for a certain level of connectivity, determining what speed you are actually getting can be challenging. Measurements can be impacted by local factors, including WiFi or cellular signal quality, and external factors, such as the location of the speed testing server. In addition, there are multiple ways calculating effective connection speeds, from active single- and multi-connection tests to passive analysis of data exhaust. No single measurement tells a complete story, so it is encouraging that a number of organizations have published insights into trends in both fixed and mobile connection speeds and how they have changed over the past several months.
Performance monitoring company Ookla (operators of speedtest.net) initially published “Tracking COVID-19’s Impact on Global Internet Performance” in mid-March. It has updated the data each week since, tracking changes to mobile and fixed connection speeds in more than 100 countries. Using the week of March 2nd as a baseline, they’ve observed a wide range of changes. As of the 27 May update, average fixed connection speeds have declined as much as 28% (in Ethiopia), and increased as much as 127% (in Lebanon). Average mobile connection speeds have dropped as much as 30% (in Sri Lanka) and have grown as much as 89% (in Trinidad & Tobago). Median declines were in a similar range, but increases were lower.
On April 8, CDN provider Fastly published “How COVID-19 is affecting internet performance,” based on the analysis of data exhaust from their platform. They looked at download speed changes for four European countries, Japan, and four locations in the United States. Although their findings were mixed, they did note that at the time of publication, “the majority of those who are able to stay home are already there, meaning that the most extreme cases of speed degradation have already occurred.”
On 14 April, performance testing company SamKnows published a blog post that looked at changes to fixed connection speeds in the United States, comparing performance from 12 March (before lockdown) to performance from 24 March (after most states went into lockdown). They found that almost all states had seen a nominal drop in download speeds, with most around 1% or less. However, they also noted that because these were averages, they may hide more significant drops in speed seen by some users.
Other surveys of connection speed trends during the pandemic have been published by OpenSignal (looking at 4G download speeds globally), BroadbandNow (tracking changes to upload and download speeds), Fing (focusing on European Internet performance), and CIRA (looking at speeds in the context of Canada’s digital divide).
NOTE: A globally exhaustive review of these changes would result in a blog post too long to read, so I looked at a limited number of countries within this post. However, based on my review of available information, shifts in traffic patterns and changes in traffic volumes and connection speeds have been within the same general ranges in other countries as well.
Last Mile Resilience
“Events” often strain the resilience of last mile networks, whether a sporting event driving millions of simultaneous streams or an eagerly anticipated software release resulting in millions of concurrent downloads. Although these events drive massive spikes in traffic, they are generally limited in duration, mitigating the long-term impact to network capacity and connection speeds.
Lockdown actions taken in response to the COVID-19 pandemic, however, have made event-level traffic the new normal across many of these networks, with traffic peaks now persisting throughout the day as many now work and learn from home.
When individual subscriber connections, whether fixed or mobile, are challenged to keep up with the increased demand, it results in users experiencing lower connection speeds.
However, last mile networks have remained fairly resilient overall, despite massive increases in both upstream and downstream traffic.
Localized community networks initiatives such as NYC Mesh are also bringing last mile resilience to areas where broadband connectivity is unavailable or unaffordable. It is now more important than ever that broadband connectivity, whether fixed or mobile, is available and affordable to all.
Earlier this year, as COVID-19 began to dominate our lives, the world turned to the Internet. This sudden shift to distance learning, working from home, and families sheltering in place drove up online streaming demand, placing additional load on Internet application platforms like Zoom, Netflix, and educational tools such as Kahoot. There was also a dramatic traffic increase across supporting network providers.
Faced with the specter of millions of daily Zoom calls and endless hours of Netflix binging, many wondered if the Internet could handle the strain of such rapid traffic growth and increased latency. Would it cause a catastrophic failure of the Internet? Our answer then: not likely.
But were we right? As the world is now more than a month into mandatory lockdowns and stay-at-home orders, with anticipated growth in application platform usage, media consumption, and overall Internet traffic, we can now state:
“No – increased Internet usage will not cause a catastrophic Internet failure.”
Each Zoom call, Netflix video, and Kahoot quiz relies on this architecture to work the way it should.
However, the past couple of months have made it clear that there has been a different catastrophic failure: the failure to make “last mile” broadband connectivity widely accessible and affordable.
This is to say nothing of the divide found in developing countries, where Internet access is even more limited or may be altogether absent. This last mile digital divide has led to Internet connections that struggle to support professional videoconferencing, media-heavy educational tools, or streaming video, especially when used concurrently. Students who have no usable Internet access at home can be found sitting outside schools and libraries, accessing the WiFi to complete their assignments.
In upcoming blog posts, we will review observations, measurements, and statistics from across the industry to examine the trends seen by the participants listed above, and look at how they are handling increased usage. We will also look at how countries around the world are recognizing the importance of available and affordable Internet connectivity, and the steps that they are taking to close the digital divide.
It’s being called the world’s biggest work-from-home experiment. With concerns growing over the spread of the COVID-19 Coronavirus, schools and businesses around the world are implementing contingency plans that encourage distance learning and work from home. Usage of e-learning, messaging, and videoconferencing tools is also growing rapidly, placing additional load on these Internet-based applications and platforms and generating additional traffic. And with more people staying at home, online media consumption is poised to increase as well.
Many are wondering if the Internet can handle the strain of rapid traffic growth and increased latency. Will it cause a catastrophic failure of the Internet? The answer: not likely.
Core Internet infrastructure providers should be able to easily absorb the increase in traffic and demand, especially if the growth is gradual over a period of days, weeks, or months. Cloud infrastructure providers should also have sufficient additional compute, storage, and bandwidth capacity to enable their customers, including the e-learning, messaging, and videoconferencing tool providers, to scale their systems as necessary. In order to keep traffic local, content delivery infrastructure from companies including Akamai, Cloudflare, Google, Netflix, and Apple is deployed in many last-mile networks.
The more likely place for the failures to occur is with the tools themselves, if they have not provisioned sufficient compute, storage, or bandwidth resources to deal with the increased traffic. Examples of such failures have already been observed in China, where servers supporting Baidu’s iQiyi streaming service, an educational application called Xuexitong, popular office applications including video conferencing applications DingTalk and WeChat Work, and online games from Tencent all reportedly crashed due to increased traffic. These failures are arguably similar to those seen on e-commerce or gaming platforms on high-traffic days, where backend systems are unable to process the higher than normal volume of orders or account activations.
Tracking the impact to the Internet of increased remote work
has been highlighted
as an area of interest to the Internet measurement community. As a co-leader of
the Internet Society’s “Measuring the
Internet” project (one of the Internet Society’s eight 2020
Action Plan projects), it’s of interest to me as well. Measuring the
Internet can help identify emerging issues – and help develop solutions.
Measurements done by RIPE Atlas and Oracle Internet
Intelligence can help gauge changes in latency, and Internet Exchange Point
(IXP) traffic data (for example, from Kenya’s IXP) can help illustrate
local changes in traffic levels. But attribution is the key challenge. We can’t
definitively state that observed latency or traffic changes are due to
increased use of remote work tools and platforms, or increased media
How Can You Help?
If you are an infrastructure provider, such as a content delivery network, share relevant traffic statistics, aggregated at a country and/or industry vertical level.
If you are an end-user, find and bookmark the status pages for the tools and platforms that you rely on for updated information about issues they may be experiencing. Downdetector is also useful for learning if other users are also experiencing similar issues.
And finally, if you are a journalist, you can be responsible in your reporting and accurate in your headlines. When a single platform or tool, no matter how important, is briefly slow or unavailable, report it as just that – not a broad Internet outage or failure.
If you’re interested in supporting or learning more about the “Measuring the Internet” project, let us know. We’re also happy to talk more about the spectrum of Internet disruptions, from inaccessible sites to complete national shutdowns.