Measuring Broadband Canada

Download this report in PDF.

Report, June 2020

View previous report, September 2016

Table of contents

• About the Project
• Executive Summary
• Measurement Results
  • Download Speed
  • Upload Speed
  • Latency
  • Webpage Loading Time
  • Packet Loss
• Conclusion
• Glossary
  • Measurements
  • Statistical Terms
• Methodology
  • Whiteboxes
  • Measurements
  • Test Destinations
  • Sample Plan
  • Data Processing
  • Code of Conduct

About the Project

The Canadian Radio-television and Telecommunications Commission (CRTC) has commissioned SamKnows to conduct a study of the performance of broadband services sold to Canadian consumers. SamKnows is a global leader in broadband measurement and has been working with governments, ISPs, content service providers, application developers, consumer groups, and academics to accurately measure Internet performance since 2008. In doing so, SamKnows has built a global Internet measurement platform, which now spans five continents and conducts many millions of measurements each day.

Data presented in this report was collected between the 1^st of October 2019 and the 31^st of October 2019.

3,266 Whiteboxes were deployed to Canadian volunteers as part of this study, out of which 3,024 were validated^{Footnote 1} (i.e., the volunteer’s subscription package and other metadata were confirmed) by the Internet Service Providers participating in the program. From this pool of validated Whiteboxes, a total of 2,035 were on internet packages that were included in the sample plan, with a sufficiently large data sample for inclusion in the report. Please note that no 1Gbps tiers are tested in the sample plan. While some providers do offer advertised speeds of 1Gbps or faster, services above 940Mbps cannot be measured with conventional speed tests. As such, speed tiers of 1Gbps or faster are excluded from the 2019 Measuring Broadband Canada project.

The ISPs participating in this project were Bell Canada, Bell Aliant, Bell MTS, Cogeco, Northwestel, Rogers, Shaw, TELUS, and Videotron. The respective internet packages included in this report are those packages with the highest subscriber counts, therefore representing a majority of Canadian fixed-line broadband users. Other packages are offered by ISPs, but are not included in this report as they either did not meet the sample plan conditions (see Section 6d), or, as mentioned above, the packages offered speeds in excess of 940 Mbps which cannot be measured with conventional speed tests, or the number of Whiteboxes did not meet the minimum required sample size. In addition, please note that all references to “as a percentage of advertised speed” in the report refer to “percentage of maximum advertised” [by the ISP]. The report does not compare results to advertised “minimum” or “most customers get” as advertised by certain ISPs.

Internet packages included in this report represent three different access technologies, with each ISP’s users testing across the following:

• Bell Canada: DSL/FTTH (Fiber to the Home)
• Bell MTS: DSL
• Bell Aliant: FTTH
• Cogeco: Cable/HFC (Hybrid fiber-coaxial)
• Northwestel: Cable/HFC
• Rogers: Cable/HFC
• Shaw: Cable/HFC
• TELUS: DSL; FTTH
• Videotron: Cable/HFC

SamKnows recommends each internet package in the sample plan have a minimum number (40) of Whiteboxes reporting and collecting data in order to provide statistical accuracy in the results. In addition each Whitebox must have provided a minimum of 5 days of valid data during the reporting period in order to be included.

Exceptions to the 40-Whitebox minimum have been made on a case-by-case basis. Northwestel’s packages have been included in order to demonstrate the performance of a remote broadband provider. Other internet packages have been included when the sample size is of at least 35 Whiteboxes, and the 95% confidence interval is within 5%, which indicate the vast majority of users can be expected to have a very similar performance if further samples were collected.

Regional performance is also reviewed in the report. Canada is split into the West & North, Central and Eastern regions. The Provinces and Territories which make up each region are listed here (Provinces and Territories not listed did not have Whiteboxes measuring performance in that area). A map of Canada showing the split between regions is included below:

• West & North: Yukon Territory, Northwest Territories, British Columbia, Alberta, Saskatchewan, Manitoba.
• East: New Brunswick, Nova Scotia, Newfoundland, Prince Edward Island.
• Central: Ontario, Quebec.

This report is designed to provide accurate data on the broadband performance experienced by the majority of Canadian fixed-line broadband users, and should not be used to make comparative marketing claims. The participating ISPs have agreed to a Code of Conduct that requires that they do not use this Phase II Report to make or support performance claims in the marketplace, comparing its results against those of any other participant, ISP or of different technologies.

As in the last Measuring Broadband Canada report the Whitebox will run tests regardless of whether IPTV traffic is present or not. Many of the represented ISPs have service offerings that include the potential for IPTV subscriptions to be combined and carried over the same medium as their internet packages. For most of the tested speed tiers in this report, it is not anticipated to materially affect results; however, some of the lower speed tiers, particularly carried over DSL, may see an impact during peak periods where users are most likely to be watching TV while internet tests are being carried out.

Executive Summary

This report presents the findings of the measurement study conducted by SamKnows on behalf of the CRTC during the month of October 2019, and investigates the following internet performance metrics: download and upload speeds, latency, packet loss and webpage loading time.

The study was conducted using data collected from 2,035 Whiteboxes that were deployed to Canadian homes. Volunteers were selected to take part in the study on the basis of their ISP and internet package. Each Whitebox conducts end-to-end tests 24 hours a day, 7 days a week to test servers located in major Canadian metropolitan areas and aims to provide a representative picture on the status of internet performance across the majority of Canadian internet users. Results herein are presented by ISP, internet package, and region.

Unless otherwise stated the results presented are taken from peak periods of user activity at the local time, which is defined as 7pm to 11pm Monday to Friday. Off-peak periods are defined as any hour and day exclusive of peak periods. A minimum sample size of 40 Whiteboxes has been used to determine the inclusion of individual internet packages, except for few cases evaluated individually on the basis of sample confidence intervals. Any case where the sample size is lower than 40 has been denoted by an asterisk (*). In order to achieve a like-to-like comparison across internet packages featuring different download and upload speeds, all test data pertaining download and upload in this report is expressed as a percentage of the maximum speeds advertised by ISPs across both metrics.

The test data for this report finds that all major Canadian ISPs are delivering users with average download speeds that exceed maximum advertised rates and that overprovisioning (providing users with additional throughput) is common. The only internet packages included in this report that did not meet the advertised maximum download speeds were Bell Canada’s DSL 100x10Mbps and Shaw’s Cable/HFC 300x15Mbps at a respective 93% and 99%.

Upload speeds were similarly high across the board, with most ISPs delivering their advertised speeds. The tiers tested for Bell Aliant and Bell Canada were lower than 100% with 94% and 87% during peak hours. Bell Canada’s average upload was noticeably lower across the DSL 15x10, DSL 25x10, and DSL 50x10Mbps tiers, which tested at a respective 71%, 77%, and 91% of advertised maximum speeds.

Download and upload speeds were not found to decrease significantly during peak hours, which is representative of a broadband network that is well provisioned to cope with periods of increased user activity.

Latency data for this report was focused on Whiteboxes located within a 150km radius of the test server locations in order to minimize the effect of distance on measurements. Average latency during peak hours was of 4 milliseconds for users on fiber connections, and a respective 12 and 14 milliseconds for users on DSL and Cable/HFC connections. Packet loss was also low, averaging 0.06% for users on fiber, 0.11% on DSL, and 0.13% for users on Cable/HFC connections. These are levels at which most users would experience no discernible issues arising from packet loss.

Average webpage loading times to a selection of websites popular amongst Canadian users was under 3 seconds, with the majority of users being able to fully load webpages between 1.3 and 2.4 seconds. Load times are found to be approximately 0.7 seconds faster for internet packages with download speeds of 25Mbps or higher, but see no discernible improvement as speeds increase to the 600Mbps range.

Measurement Results

Download Speed

Download speed measures the capacity of the user’s broadband connection for downloading content from the internet. Higher speeds are more desirable, as it allows users to retrieve content, such as web pages, videos, files, or music more quickly.

Internet packages are commonly advertised by service providers on the basis of download and upload speeds, which are expressed in megabits per second (abbreviated as “Mbps” or, less commonly, “Mbit/s”). For example, an ISP might advertise an internet package as “up to 50x10Mbps” or “up to 50/10Mbps”, signifying “the maximum speed expected is of 50 megabits per second for download and 10 megabits per second for upload”. In order for internet packages of different speeds to be comparable, download speeds are therefore presented in terms of the percentage of their advertised maximum speed. For example, if a user is subscribed to an internet package advertised as “up to 50Mbps download” and their download speeds averaged 45Mbps during the testing period, we would say that the user’s average percentage of maximum download speed was (45Mbps/50Mbps)*100% = 90%.

Figure 1 below shows the average download speed as a percentage of advertised maximum during peak and off- peak hours for each ISP and technology. The different columns represent the periods of highest and lowest network usage in Canada. Peak hours are defined as 7-11pm from Monday to Friday, whereas off-peak hours comprise all other hours of weekdays, as well as every hour of the weekend. The 95% confidence interval^{Footnote 2} is also displayed as thin lines above and below the average value.

Long Description

This bar chart illustrates a comparison of peak versus off-peak hours of download speed by technology and ISP, as a percentage of maximum advertised speeds as follows:

Technology-ISP	Peak Hour Download Speed	Off-Peak Hour Download Speed
DSL-Bell Canada	105%	108%
DSL-Bell MTS	106%	109%
DSL-TELUS	107%	110%
Cable/HFC-Cogeco	109%	110%
Cable/HFC-Northwestel	111%	112%
Cable/HFC-Rogers	129%	130%
Cable/HFC-Shaw	104%	107%
Cable/HFC-Videotron	110%	111%
FTTH-Bell Aliant	102%	103%
FTTH-Bell Canada	113%	113%
FTTH-TELUS	115%	115%

Average download speeds for users in Canada exceeded those advertised across all ISPs included in this report, ranging from 102% during peak hours for Bell Aliant FTTH to 129% for Rogers Cable/HFC. The effect of network congestion during peak hours is minimal, with most ISPs seeing an average decrease in download speed in the order of 1 to 2 percentage points. This decrease in performance is slightly higher for Bell MTS DSL and Shaw Cable/HFC, at 3 and 4 percentage points respectively, but the impact on performance would nevertheless be negligible to the vast majority of a user’s online activities.

Figures 2 to 4 below show the average peak-hour download speed as a percentage of maximum advertised rates for each internet package included in this report. For ease of presentation, the charts have been divided according to “speed bins”. Packages that did not meet the minimum recommended sample size but had at least 35 Whiteboxes reporting and a small enough standard deviation have been marked with an asterisk (*).

Long Description

This bar chart illustrates download speed by ISP and Package, for 10-25Mbps download tiers, as a percentage of maximum advertised speed as follows:

ISP-Download Speed Tier	Download Speed
Bell MTS-10x2	117%
Bell MTS-25x2	100%
Bell Canada-15x10	113%
Bell Canada-25x10	108%
Cogeco-15x2	113%
Northwestel-20x3	112%
Shaw-15x1.5	110%
TELUS-15x1	109%
TELUS-25x5	107%
TELUS-25x25	117%
Videotron-10x1.5	111%
Videotron-15x10	111%

Average performance was good across internet packages whose download speed was advertised between 10 and 25Mbps, as all services met their respective advertised speeds during peak hours. Most internet packages in the chart above show download performances that exceed maximum advertised rates by a significant margin, meaning that overprovisioning (ISPs providing users with higher speeds than advertised) is commonplace.

Long Description

This bar chart illustrates download speed by ISP and Package, for 30-75Mbps download tiers, as a percentage of maximum advertised speed as follows:

ISP-Download Speed Tier	Download Speed
Bell Canada-50x10	106%
Bell Canada-50x50	124%
Cogeco-40x10	107%
Cogeco-60x10	116%
Northwestel-50x4	108%
Rogers-60x10	112%
Rogers-75x10	117%
Shaw-75x7.5	108%
TELUS-50x10	113%
TELUS-75x15	101%
TELUS-75x75	119%
Videotron-30x10	109%
Videotron-60x10	111%

Overprovisioning also appears to be common amongst internet packages with maximum download speeds between 30 and 75Mbps, where average speeds also exceeded advertised rates by good margins. Bell Canada’s FTTH 50x50Mbps package was a particularly strong performer, delivering an average peak-hour performance of 124% during the testing period.

Long Description

This bar chart illustrates download speed by ISP and Package, for 100-600Mbps download tiers, as a percentage of maximum advertised speed as follows:

ISP-Download Speed Tier	Download Speed
Bell Aliant-500x500	102%
Bell Canada-100x10	93%
Bell Canada-150x150	120%
Bell Canada-300x300	106%
Cogeco-120x10	101%
Rogers-150x15	147%
Rogers-500x20	130%
Shaw-300x15	99%
Shaw-600x20	103%
TELUS-150x150	114%
TELUS-300x300	111%

Figure 4 above shows the highest-speed tiers included in this report, with speeds ranging from 100Mbps to 600Mbps. Most internet packages here were also overprovisioned to varying degrees, with Rogers’ Cable/HFC 150x15Mbps service showing very high speeds compared to those advertised by the ISP. Rogers’ Cable/HFC 500x20Mbps package was also a high performer that commonly exceeded advertised rates, as was Bell Canada’s FTTH 150x150Mbps. Bell Canada’s DSL 100x10Mbps plan did not meet, on average, the ISP’s advertised rates, delivering speeds of 93% during peak hours. Shaw’s 300x15Mbps package was also under 100%, at 99%.

Figure 5 shows the average download performance across Canada by service technology and region. Please note that no DSL products in the East region have been included in the final sample for this report.

Long Description

This bar chart illustrates a comparison of peak versus off-peak hours of download speed by technology and region, as a percentage of maximum advertised speeds as follows:

Technology-Region	Peak Hour Download Speed	Off-Peak Hour Download Speed
DSL-Central	105%	108%
DSL-West&North	107%	110%
Cable/HFC-Central	116%	117%
Cable/HFC-East	128%	130%
Cable/HFC-West&North	105%	108%
FTTH-Central	112%	113%
FTTH-East	102%	103%
FTTH-West&North	115%	115%

Canadian ISPs delivered users with average download speeds that exceeded the maximum advertised rates across all technologies and regions. DSL services had comparable performances of 105% and 107% across the Central and West & North regions, with similar decreases of 3 percentage points compared to off-peak hours. Cable/HFC services had their highest performance in the East at 128%, followed by the Central region (116%) and then West & North (105%), with limited decreases compared to off-peak hours. FTTH services instead saw their highest performance in the West & North regions at 115%, followed by the Central region (112%), with performance in the East lower at 102%. Service deterioration was lowest on FTTH services, with virtually no change (1 percentage point) compared to off-peak hours.

Table 1 below shows the advertised maximum download speed at peak hours for each ISP and internet package included in this report, expressed both in terms of megabits per second as well as a percentage of advertised, along with the sample count and standard deviation (in terms of percentage of advertised) for each.

Upload Speed

Upload speed is the measure of how fast data can be transmitted from the home to the internet. Higher speeds can allow for pictures, music, and documents to be uploaded and shared more quickly.

Results are presented as a ratio of the average measured speed to the advertised maximum by the ISP. For example, if a user on an internet package advertised as “up to 50x10Mbps” receives, on average, upload speeds of 12Mbps during tests, then we say that the user received (12Mbps / 10Mbps)*100% = 120% of maximum advertised speeds.

Figure 6 below shows the average upload speed as a percentage of maximum advertised during peak and off-peak hours for each ISP and technology. The different columns represent the periods of highest and lowest network usage in Canada. Peak hours are defined as 7-11pm from Monday to Friday, whereas off-peak hours comprise all other hours of weekdays, as well as every hour of the weekend. The 95% confidence interval is also displayed as thin lines above and below the average value.

Long Description

This bar chart illustrates a comparison of peak versus off-peak hours of upload speed by technology and ISP, as a percentage of maximum advertised speeds as follows:

Technology-ISP	Peak Hour Upload Speed	Off-Peak Hour Upload Speed
DSL-Bell Canada	87%	87%
DSL-Bell MTS	115%	115%
DSL-TELUS	164%	164%
Cable/HFC-Cogeco	109%	110%
Cable/HFC-Northwestel	119%	119%
Cable/HFC-Rogers	104%	106%
Cable/HFC-Shaw	105%	105%
Cable/HFC-Videotron	109%	109%
FTTH-Bell Aliant	94%	95%
FTTH-Bell Canada	109%	109%
FTTH-TELUS	113%	113%

Average upload speeds were largely above advertised rates during peak usage hours, with TELUS delivering the highest at 164% on DSL (113% on FTTH). It is worth noting, however, the very wide confidence interval which reflects that performance is likely to vary significantly from one user to the next. Northwestel Cable/HFC and Bell MTS DSL also delivered high upload performance to their users at 119% and 115%. These were followed in turn by Cogeco (109%), Videotron (109%), Shaw (105%) and Rogers (104%). On DSL Bell Canada delivered upload speeds that were below their advertised maximum rates, at 87%. Similarly Bell Aliant FTTH 500x500 fell under 100% of maximum advertised speed with 94% during peak hours.^{Footnote 3} As with download speeds, upload speeds were not particularly affected by increased network congestion during peak hours, thus showing a broadband network that is well-provisioned and able to cope with increased user activity during the evenings.

Figures 7 to 10 below show the average peak-hour upload speed as a percentage of advertised rates for each ISP’s package included in this report. As with download speeds, the different internet packages have been categorized into “speed bins” for ease of presentation and comparison.

Long Description

This bar chart illustrates upload speed by ISP and Package, for 1-7.5Mbps upload tiers, as a percentage of maximum advertised speed as follows:

ISP-Upload Speed Tier	Upload Speed
Bell MTS-10x2	114%
Bell MTS-25x2	115%
Cogeco-15x2	111%
Northwestel-20x3	122%
Northwestel-50x4	114%
Shaw-15x1.5	104%
Shaw-75x7.5	105%
TELUS-15x1	345%
TELUS-25x5	102%
Videotron-10x1.5	103%

Internet packages with upload speeds between 1 and 7.5Mbps all delivered upload performance that exceeded the maximum speeds advertised by ISPs during the busiest hours of the week. As with download speeds, upload speeds are also subject to overprovisioning in order to ensure optimal service during these busy hours, and the amount can vary from ISP to ISP and from tier to tier. For example, TELUS’ DSL 25x5Mbps package exceeded maximum advertised speeds by 2 percentage points, performing at 102%. By contrast, subscribers to the same ISP’s DSL 15x1Mbps received average upload speeds that were in excess of 3 to 4Mbps. Northwestel and Bell MTS can also be seen to overprovision certain tiers, such as the Cable/HFC 20x3Mbps and DSL 25x2Mbps respectively.

Long Description

This bar chart illustrates upload speed by ISP and Package, for 10Mbps upload tiers, as a percentage of maximum advertised speed as follows:

ISP-Upload Speed Tier	Upload Speed
Bell Canada-15x10	71%
Bell Canada-25x10	77%
Bell Canada-50x10	91%
Bell Canada-100x10	106%
Cogeco-40x10	107%
Cogeco-60x10	117%
Cogeco-120x10	104%
Rogers-60x10	103%
Rogers-75x10	105%
TELUS-50x10	108%
Videotron-15x10	110%
Videotron-30x10	110%
Videotron-60x10	111%

Subscribers to internet packages with maximum advertised upload speeds of 10Mbps generally received speeds that either met or exceeded their sold rates, although results varied. For example, TELUS’ DSL 50x10Mbps plan delivered an average of 108% in upload speed during busy hours, but with a large standard deviation (seen in the width of the confidence interval), meaning that the distribution was uneven. Bell Canada’s DSL 25x10 and DSL 15x10Mbps packages saw especially low speeds with upload performances of 77% and 71%. Both the 15x10Mbps and 25x10Mbps packages had fairly wide confidence intervals, meaning that test results varied significantly across users. This is not surprising however, given the effect of copper loop lengths (i.e. how far a customer is from the nearest central office or node) online performance.

Long Description

This bar chart illustrates upload speed by ISP and Package, for 15-75Mbps upload tiers, as a percentage of maximum advertised speed as follows:

ISP-Upload Speed Tier	Upload Speed
Bell Canada-50x50	116%
Rogers-150x15	103%
Rogers-500x20	104%
Shaw-300x15	105%
Shaw-600x20	105%
TELUS-75x15	137%
TELUS-25x25	116%
TELUS-75x75	120%

Internet packages with maximum advertised upload speeds between 15 and 75Mbps all delivered performances that exceeded advertised rates. As with other tiers, overprovisioning occurs across all ISPs to varying degrees. TELUS’ DSL 75x15Mbps and FTTH 75x75Mbps packages delivered upload speeds much higher than the maximum advertised (at 137% and 120% respectively), as did Bell Canada’s FTTH 50x50 package at 116%.

Long Description

This bar chart illustrates upload speed by ISP and Package, for 150-500Mbps upload tiers, as a percentage of maximum advertised speed as follows:

ISP-Upload Speed Tier	Upload Speed
Bell Aliant-500x500	94%
Bell Canada-150x150	114%
Bell Canada-300x300	105%
TELUS-150x150	111%
TELUS-300x300	107%

Maximum advertised upload speeds were also largely exceeded at the highest speed tiers, with Bell Canada and TELUS’ FTTH 150x150Mbps packages overperforming. Bell Aliant’s FTTH 500x500Mbps internet package averaged peak hour upload speeds just under the maximum advertised rates at 94% during the measurement month. It should be noted that Bell Aliant identified an issue with the software running on their optical line terminals, which resulted in their customers measuring lower upload speeds during the initial seconds of an upload test. The issue was resolved on October 17^th through a configuration change applied to the entire base of 500x500Mbps customers, slightly more than halfway through the testing month. In the latter half of the measurement month, after the configuration change was applied, Bell Aliant’s FTTH 500x500Mbps package average upload speeds were in excess of advertised speeds.

Figure 11 below shows average upload speed during peak and off-peak hours as a percentage of advertised, by region. As with the regional download charts, please note that no DSL services in the East have been included as part of this report.

Long Description

This bar chart illustrates a comparison of peak versus off-peak hours of upload speed by technology and region, as a percentage of maximum advertised speeds as follows:

Technology-Region	Peak Hour Upload Speed	Off-Peak Hour Upload Speed
DSL-Central	87%	87%
DSL-West&North	149%	149%
Cable/HFC-Central	107%	108%
Cable/HFC-East	106%	107%
Cable/HFC-West&North	107%	107%
FTTH-Central	109%	109%
FTTH-East	95%	95%
FTTH-West&North	113%	113%

DSL services had the highest upload performance in the West & North at 149% of advertised maximum rates, whereas the Central region showed a much lower performance of 87%. Upload performance of Cable/HFC services was instead nearly identical across the country, at 107% for the Central and West & North regions, and 106% in the East. FTTH also had its strongest performance in the West & North at 113%, followed by the Central region at 109%, while maximum advertised speeds were not met in the East, at 95%. Network congestion was minimal across all service technologies and regions.

Please note that Bell Aliant identified an issue with the software running on their optical line terminals, which resulted in their customers measuring lower upload speeds during the initial seconds of an upload test. The issue was resolved on October 17^th and after the configuration change was applied, Bell Aliant’s 500x500mbps package average upload speeds were in excess of advertised speeds. This may impact on regional performance, which was averaged over the entirety of the month.

Table 2 below shows the advertised upload speed at peak hours for each ISP and internet package included in this report, expressed both in terms of megabits per second as well as a percentage of maximum advertised, along with the technology, sample count and standard deviation (in term of percentage of advertised) for each.

Latency

Latency is a measure of how long it takes a packet to travel between point A and point B. It is a significant factor in internet performance, as latency is a fundamental property of the infrastructure upon which everything else must build. If you have a high-latency link, then it does not matter how fast your broadband connection is, as latency will always remain an impediment. Besides infrastructure itself, latency is also dependent on geography: a user located physically closer to a content server or exchange will have a lower latency than one located several hundred kilometers away.

While latency is unrelated to line speed itself, an increase in latency can have a detrimental effect on how long it takes to transfer files and other internet content. Moreover, an increase in latency during peak hours of usage is an early indicator of congestion in the network, as routers are taking longer to receive data packets and pass them on.

The results presented below show “round-trip latency” (i.e., how long it takes for a packet of data to travel from point A to point B, and back to point A). As latency is dependent on the distance between the user and the target – in this case, a test server – results below have been divided into two sets to account for test servers being located in the metropolitan areas of Halifax, Montreal, Toronto, and Vancouver.

The chart below shows peak and off-peak latency split by access technology, including only test results from volunteers located within a 150km radius of the cities listed above as, demographically, they provide the most representative sample of the majority of Canadian internet users. Whiteboxes included in the chart below represent a sample from homes connected to internet services by Bell Canada, Bell Aliant, Cogeco, Rogers, Shaw, TELUS, and Videotron. As latency is (among other factors) dependent on distance, users located farther away from test servers will experience higher latencies.

Long Description

This bar chart illustrates a comparison of peak versus off-peak hours of latency by technology as follows:

Technology	Peak Hour Latency	Off-Peak Hour Latency
Cable/HFC	14 msec	13 msec
DSL	12 msec	12 msec
FTTH	4 msec	4 msec

Fiber connections had the lowest average latency of 4 milliseconds, with no change shown between peak and off-peak hours. DSL users averaged latencies of 12 milliseconds during both peak and off-peak hours, with Cable/HFC connections showing very similar results of 13 milliseconds during off-peak hours, which increased by a negligible 1 millisecond during peak hours. In real-world terms, all of these results are very good as most users would see no practical effect from a 10ms difference except in cases of highly latency-sensitive applications such as in real-time multiplayer games.

As mentioned previously, latency results have been divided between those from users located in the vicinity of test servers, and those outside of the 150km radius. The chart below shows latency results for users of all ISPs in this report located outside of the metropolitan areas of Halifax, Montreal, Toronto, and Vancouver. It should be noted once again that latency is highly dependent on distance between the users and test servers, and therefore results may vary significantly across individual users.

Long Description

This bar chart illustrates a comparison of peak versus off-peak hours of latency by technology, for users located greater than 150 km from test server, as follows:

Technology	Peak Hour Latency	Off-Peak Hour Latency
Cable/HFC	27 msec	26 msec
DSL	25 msec	25 msec
FTTH	12 msec	11 msec

A similar scenario was found in latency tests for users located over 150km away from test servers, with fiber connections showing the lowest average latency of 12 milliseconds during peak times, followed by DSL at 25ms and Cable/HFC closely behind at 27ms during peak hours. As with the previous chart’s results, latency shows no real-world difference between peak and off-peak hours, and are indicative of an infrastructure that is well provisioned to handle increased user activity in the evenings.

Webpage Loading Time

The webpage loading time test captures how long it takes for all of the elements of a webpage to be received by the user. Unlike other tests in this report, which measure against dedicated test servers, webpage loading time tests are conducted against actual websites commonly visited by Canadian users.

Webpage loading time is heavily influenced by many factors, including the user’s download speed and latency (examined previously), along with the quality of the web server itself, distance between it and the user, as well as congestion in other networks that contribute content to the website’s delivery.

Figure 14 below shows average webpage loading times for each ISP^{Footnote 4} during peak and off-peak hours.

Long Description

This bar chart illustrates a comparison of peak versus off-peak hours of Webpage Loading Time by technology and ISP as follows:

Technology-ISP	Peak Hour Webpage Loading Time	Off-Peak Hour Webpage Loading Time
DSL-Bell Canada	1.6 sec	1.6 sec
DSL-Bell MTS	2.4 sec	2.3 sec
DSL-TELUS	1.6 sec	1.6 sec
Cable/HFC-Cogeco	1.7 sec	1.6 sec
Cable/HFC-Rogers	1.6 sec	1.5 sec
Cable/HFC-Shaw	2.0 sec	2.0 sec
Cable/HFC-Videotron	2.1 sec	2.0 sec
FTTH-Bell Aliant	1.6 sec	1.5 sec
FTTH-Bell Canada	1.3 sec	1.3 sec
FTTH-TELUS	1.4 sec	1.4 sec

Webpage loading times were well under 3 seconds across all ISPs, with negligible increases of 0.1 seconds or less during peak hours of user activity. Users on Bell Aliant FTTH, Bell Canada DSL/ FTTH, Cogeco Cable/HFC, Rogers Cable/HFC, and TELUS DSL/FTTH all recorded average loading times between 1.3 and 1.7 seconds with minor differences across service technologies, Shaw Cable/HFC and Videotron Cable/HFC followed at 2 and 2.1 seconds. Average times for Bell MTS were marginally higher at 2.3-2.4 seconds, with user distance to major content servers – which are typically located near major urban centers – contributing to the small increment.

Figures 15 to 17 below show the average webpage loading time at peak hours for users across separate ISPs and internet packages. As with previous sections, these have been split and categorized according to download speed.

Long Description

This bar chart illustrates Webpage Loading Time by ISP and Package, for 10-25Mbps download tiers as follows:

ISP-Download Speed Tier	Webpage Loading Time
Bell MTS-10x2	2.8 sec
Bell MTS-25x2	2.1 sec
Bell Canada-15x10	2.1 sec
Bell Canada-25x10	1.6 sec
Cogeco-15x2	2.1 sec
Shaw-15x1.5	2.3 sec
TELUS-15x1	2.2 sec
TELUS-25x5	1.6 sec
TELUS-25x25	1.4 sec
Videotron-10x1.5	3.1 sec
Videotron-15x10	2.3 sec

Internet packages with maximum advertised download speeds in the 10 to 25Mbps range largely recorded webpage loading times between 2 and 3 seconds. A small subset comprised of Bell Canada’s DSL 25x10Mbps, and TELUS’ DSL 25x5 and FTTH 25x25Mbps tiers had lower webpage loading times in the 1.4 to 1.6 second range, whilst Videotron’s Cable/HFC 10x1.5Mbps users saw a slightly higher average instead at just over 3 seconds.

Long Description

This bar chart illustrates Webpage Loading Time by ISP and Package, for 30-75Mbps download tiers as follows:

ISP-Download Speed Tier	Webpage Loading Time
Bell Canada-50x10	1.5 sec
Bell Canada-50x50	1.3 sec
Cogeco-40x10	1.5 sec
Cogeco-60x10	1.5 sec
Rogers-60x10	1.4 sec
Rogers-75x10	1.6 sec
Shaw-75x7.5	2.2 sec
TELUS-50x10	1.4 sec
TELUS-75x15	1.4 sec
TELUS-75x75	1.3 sec
Videotron-30x10	1.5 sec
Videotron-60x10	1.4 sec

Webpage loading times for packages in the 30 to 75Mbps download speed tiers were markedly faster, with the vast majority averaging between 1.3 and 1.6 seconds, and only Shaw’s Cable/HFC 75x7.5Mbps package showing a higher average of 2.2 seconds.

Long Description

This bar chart illustrates Webpage Loading Time by ISP and Package, for 100-600Mbps download tiers as follows:

ISP-Download Speed Tier	Webpage Loading Time
Bell Aliant-500x500	1.5 sec
Bell Canada-100x10	1.5 sec
Bell Canada-150x150	1.3 sec
Bell Canada-300x300	1.3 sec
Cogeco-120x10	1.6 sec
Rogers-150x15	1.5 sec
Rogers-500x20	1.6 sec
Shaw-300x15	2.0 sec
Shaw-600x20	1.6 sec
TELUS-150x150	1.4 sec
TELUS-300x300	1.4 sec

Average webpage loading time for higher-speed internet packages in the 100 to 600Mbps maximum advertised download range was similar to those in the 30-75Mbps range seen previously, with most averages between 1.3 and 1.6 seconds, except for Shaw’s Cable/HFC 300x15Mbps service, which saw an average of 2 seconds.

Effect of download speed on webpage loading time

Internet users frequently wonder how much the various broadband performance metrics may affect – positively or negatively – common tasks, such as browsing websites. Figure 20 below offers an additional view, with peak-hour webpage loading times shown as a function of the package’s download speed (i.e. “10” on the horizontal axis includes Bell MTS’ DSL 10x2Mbps and Videotron’s Cable/HFC 10x1.5Mbps packages).

Long Description

This bar chart illustrates Webpage Loading Time by Download Speed as follows:

Package Download Speed (Mbps)	Webpage Loading Time
10	3.0 sec
15	2.2 sec
20	2.5 sec
25	1.7 sec
30	1.5 sec
40	1.5 sec
50	1.5 sec
60	1.5 sec
75	1.6 sec
100	1.5 sec
120	1.6 sec
150	1.4 sec
300	1.6 sec
500	1.6 sec
600	1.6 sec

In the chart above, two distinct “groups” can be discerned: the leftmost side of the chart where average loading times are 2.2 seconds and higher and comprises speed tiers up to 20Mbps download, and the rightmost one comprising download speed tiers of 25Mbps and above, all of which average webpage loading times of 1.7 seconds or faster.

The above shows that download speeds are indeed important when it comes to webpage loading times, but only up to a certain point. A user on a 25Mbps internet package is unlikely to see noticeably better loading times if they move to a 100, 300, or 500Mbps download package. There are, however, other factors that can affect the speed of webpage loading, such as the location of hosting servers as well as Content Delivery Networks, as well as the quality of their infrastructure.

Packet Loss

The rate of packet loss describes how likely it is that a packet of data sent from point A will not reach point B. Packet loss is closely related to latency and is a fundamental metric in determining how applications perform on a broadband connection. It is generally expressed as a percentage of packets that failed to meet the target compared to total packets sent.

A high rate of packet loss will prevent many applications from working to a satisfactory level. As a rule of thumb, a 1% packet loss average tends to indicate connection issues and may translate into user experience problems such as webpages being unable to load, or the inability to successfully download or upload files. A small increase in packet loss during peak hours is to be expected, as networks are busier and congestion at even one point in a network path may lead to a packet being dropped.

Figure 19 below shows the average packet loss for each access technology. It should be noted that packet loss tests occur concurrently with latency tests but, as packet loss is not directly correlated to distance the way latency is, results are inclusive of all Whiteboxes regardless of distance from the test server. Results from one ISP’s users have however been excluded, as tests may have been impacted by a phenomenon known as “buffer bloat” that resulted in excessively high packet loss levels. Buffer bloat can affect the results of latency/loss tests because these tests are carried out by the Whiteboxes immediately after a speed test is concluded. If a modem has a large buffer, data from the previous speed test could still be queued when the latency/loss test begins. If this “queueing" lasts more than three seconds the latency/loss test times out, leading to reported packet loss.

Long Description

This bar chart illustrates a comparison of peak versus off-peak hours of Packet Loss by technology as follows:

Technology	Peak Hour Packet Loss	Off-Peak Hour Packet Loss
Cable/HFC	0.13%	0.14%
DSL	0.11%	0.10%
FTTH	0.06%	0.07%

Packet loss was low across all access technologies, averaging 0.06% for FTTH users during peak hours, 0.11% for DSL users, and 0.13% across Cable/HFC, with differences between peak and off-peak times being negligible. These averages are well below the 1% packet loss threshold where users would notice a degradation of services, such as slower webpage load times or interruptions of file transfers.

Conclusion

This report presents numerous findings about the current state of Canadian internet performance across some of the most popular internet packages offered by ISPs. Overall, Canadian ISPs have mostly met or exceeded maximum advertised download and upload speeds across tiers and regions. This quality of service is consistent across Canada. Average download performance is in excess of 100% of the maximum advertised speed across all regions and technologies.

Download speeds either met or exceeded their maximum advertised rates, with only Bell Canada’s DSL 100x10Mbps and Shaw’s Cable/HFC 300x15Mbps internet packages delivering speeds below their advertised maximum (93% and 99% respectively) during peak hours.

The majority of ISPs delivered upload speeds that, on average, met or exceeded their maximum advertised upload rates, with the exception of Bell Canada at 97%. Bell Canada’s average was particularly affected by underperformance from their DSL 15x10Mbps, DSL 25x10Mbps, and DSL 50x10Mbps internet packages.

Across all regions upload speeds performed very well with two exceptions; the DSL tiers tested in Central Canada fell just short of 100% maximum advertised upload speed as did the FTTH tiers in the East region. It should be noted here that Bell Aliant’s 500x500 FTTH tier experienced a software issue which was identified halfway through the reporting month, once this was addressed it tested in excess of advertised maximum speed.

Importantly, neither download nor upload performance were found to be affected by peak-hour network congestion to a degree that would be noticeable to users.

Other quality metrics such as latency and packet loss were similarly unaffected by peak-hour congestion and present a picture of a broadband network that is well provisioned to cope with the demands placed by increased user activity in the evenings.

The Measuring Broadband Canada project aims to continue providing Canadians with information on their broadband services, to help inform and educate consumers on what the best provider and package is for their needs.

As with any volunteer driven project, this would not be possible without the help of volunteers all over the country hosting a Whitebox in their home. The project always needs more people to take part and if you wish to do so then sign up here: Help improve Canada's broadband! Sign up .

Glossary

Measurements

Download Speed: The speed at which data can be transferred from the SamKnows test server to the user’s home, measured in megabits per second (Mbps). In this report, it is generally expressed in terms of “measured download speed as a percentage of advertised download speed” in order to enable a like- for-like comparison across different speed tiers.

Upload Speed: The speed at which information is transferred from the user’s home to the SamKnows test server. Similarly to download speed, it is measured in megabits per second (Mbps) and also expressed as a percentage of the upload speed advertised by the ISP.

Latency: the timespan it takes for a data packet to go from the user’s home to the SamKnows test server, and back to the home. It is commonly expressed in milliseconds (ms). A shorter latency is better.

Packet Loss: is the number of data packets that are sent over the network, but fail to reach their destination. It is measured as a percentage of packets lost to total packets sent.

Webpage Loading Time: also known as “webpage load time”. It is the time it takes for a website to fully load all of its elements, such as text, images, sounds, and miscellaneous scripts, and is commonly expressed in either seconds (s) or milliseconds (ms). It is determined by a myriad of factors, such as download speed, latency, and the quality of the network of servers providing the website’s contents. A shorter loading time is better.

Statistical Terms

Mean: the arithmetic mean, or average. It is calculated by summing all the samples in a set, and dividing by the number of samples. This is the method used in calculations for packet loss.

Trimmed mean: also known as the truncated mean. It is calculated by removing the top and bottom percentiles of a data set, and then evaluating the arithmetic mean of the remaining data. In download, upload, and latency measurements for this report, the top and bottom 1% of results for an individual Whitebox are “trimmed”, or excluded, prior to any other aggregation.

Standard Deviation: a measure of the variability of a data set. It is dependent on both the number of samples, and the distribution of measurements. An internet package with few Whiteboxes, but whose results are very similar will have a small standard deviation, whereas a package with a high number of Whiteboxes but an uneven distribution of results (i.e., very high download speeds and very low ones) may have a higher standard deviation.

Confidence Interval: when conducting a measurement experiment such as the one presented in this report, one is using a finite number of samples to approximate the real average across an entire population. The confidence interval is a range of values within which we are 95% certain that the “real” average to be. It is a function of the standard deviation and the sample size.

Methodology

Whiteboxes

SamKnows deployed 3,266 Whiteboxes to Canadian consumers for the purposes of this project. The Whitebox is a consumer-grade device that is installed in a user’s home network between their home modem/router and their devices. The Whitebox’s core function is to measure the quality of the user’s Internet connection. The Whitebox model sent out for this Phase of the project was the Whitebox 8.0. The specs of this are included below:

The current generation of Whitebox (8.0) is capable of measuring 1Gbps downstream and upstream over both TCP and UDP.

The specifications of the device are as follows:

• Dual 2.4 GHz and 5GHz Wi-Fi radios, supporting 802.11a/b/g/n/ac
• Dual-core 880MHz CPU
• 128MB RAM
• 16MB flash storage
• 4x 1Gbps LAN interfaces
• 1x 1Gbps WAN interfaces
• USB 2.0 port
• DC power (12V @ 2000mA)

The measurements are conducted autonomously by the Whitebox to a variety of destinations on the Internet. No user interaction is required to conduct measurements; they are executed automatically according to a test schedule.

End-user cross-traffic is monitored continuously by the Whitebox. If cross-traffic exceeds a certain threshold then measurements are not conducted until the cross-traffic subsides. This ensures that the Whitebox’s measurements are not distorted by end-user activity, and that the Whitebox’s measurement traffic does not interfere with a user’s experience of the Internet.

A full description of the Whitebox and its features can be found at SamKnows Whitebox .

Measurements

The Whiteboxes run a suite of active performance measurements according to a pre-defined test schedule. These include the following network measurements: download speed, upload speed, latency and packet loss, as well as application measurements such as web browsing performance.

A full description of the methodology underpinning each test can be found at SamKnows tests .

Test Destinations

The measurements carried out by the Whiteboxes are conducted against two different types of destination servers.

Firstly, there are dedicated measurement servers. These are installed at major peering and Internet exchanges at the following locations:

• Montreal
• Halifax
• Vancouver
• Toronto

Each server met minimum specifications set out by SamKnows. The dedicated measurement servers are utilized for the download speed, upload speed, latency and packet loss measurements.

Secondly, there are real applications / content providers that measurements are carried out against. For the purposes of the Measuring Broadband Canada project, this is limited to the web browsing measurements only. The websites tested against were as follows:

• ici.radio-canada.ca
• cbc.ca
• facebook.com/policies
• google.ca
• canada.ca
• ebay.com
• ca.yahoo.com

Sample Plan

SamKnows constructed a sampling plan to govern the distribution of Whiteboxes amongst Canadian Internet consumers. This sample plan was built using subscriber data provided by the ISPs, and took into consideration which internet packages met the minimum sample size requirements outlined below:

• The package had a minimum of 25,000 subscribers.
• Legacy packages (those that are no longer in market and sold to customers) to be included if they have over 10% of the ISP’s subscriber base on.
• Packages with an advertised download speed of 10Mbps or less are not included.

While some providers do offer advertised speeds of 1Gbps or faster, services above 940Mbps cannot be measured with a conventional speed test. As such, speed tiers of 1Gbps or faster are excluded from the 2019 Measuring Broadband Canada project. Consumers should visit each of the participating ISP’s websites for details about the availability of higher or lower speed tiers.

Internet packages marked with a single asterisk (*) did not meet the minimum required number of Whiteboxes (40) but have nevertheless been included due to having at least 35 Whiteboxes and a narrow (5%) confidence interval. Packages marked with two asterisks (**) were initially considered for inclusion in the report, but have been subsequently excluded due to low sample size. The parties acknowledge that a limited exception to the minimum number of subscribers per plan has been made to address inequity between ISPs serving the same area.

Data Processing

All data included in this report was obtained via tests conducted between the 1^st and 31^st of October 2019. For inclusion in the final data set, a Whitebox must have:

1. Been successfully validated by the ISP (i.e., the subscribed download and upload tiers confirmed, along with metadata such as province and postcode)
2. Have provided a minimum of 5 days of data during the reporting period. If the volunteer’s ISP or internet package changed during the measurement period, considerations were made based on the number of days of data obtained under each ISP/package, and sample plan requirements
3. For inclusion in peak and off-peak verticals, a minimum of 5 samples must have been provided by each Whitebox.

Data for each Whitebox was then aggregated via a trimmed mean for download, upload, latency, and webpage loading time metrics. This trimmed mean method excludes outlier results in the top and bottom 1% and helps ensure a cleaner dataset that is better representative of the typical performance. Packet loss data is instead aggregated via the arithmetic mean. Individual results from Whiteboxes are finally aggregated on the basis of ISP, internet package, and region for packages that meet the minimum sample size requirements.

Code of Conduct

Code of Conduct Measuring Broadband Canada

2019

The Canadian Radio-television and Telecommunications Commission (the "CRTC") has established a Canadian Broadband Measurement Project (the "Project") in collaboration with participating Canadian ISPs ("ISPs") and SamKnows Limited ("SamKnows").

To ensure the effective and unbiased use of the data, and the integrity and validity of study results, this Code of Conduct has been drawn up which each of the ISPs, SamKnows and the CRTC (each a "Participant") agrees to sign.

The undersigned, as a Participant in the Project, agree with the following principles:

1. At all times during the Project, the Participants agree to act in good faith.
2. The Participants agree not to tamper with the testing infrastructure or methodology, or take any other actions which influence the results of any test, through positive action or omission, for any individual panelist or participating ISP.
3. The Participants acknowledge that it will not be a violation of the principle set out in paragraph 2 above for ISPs or SamKnows to:
1. Operate and manage their business, which includes modifying or improving services delivered to any class of subscribers that may or may not include panelists among them, provided that such actions are consistent with normal business practices;
2. Address service issues for individual panelists at the request of the panelists or based on information not derived from the trial; and
3. Advise their customer service representatives of the identity of subscribers who are panelists in the trial so that those representatives may address service and billing questions.
4. The Participants acknowledge that it will not be a violation of the principle set out in paragraph 2 above to monitor the tests and components of the testing architecture provided that no impact to the CRTC data occurs. In particular, the Participants acknowledge that the ISPs may advise SamKnows wherever a technical concern is observed in respect of an individual panelist, so that SamKnows may contact said panelist to investigate and remedy the problem.
5. A Participant shall not use the Phase II Report to make or support performance claims in the marketplace (including claims employing superlatives such as “best” or “fastest”) comparing its results against those of any other Participant, ISP or of different technologies, except that a Participant may use the Phase II Report, however necessary, subject to compliance with Section 16 of the 2019 MOU and this Code of Conduct, to directly answer a question or correct any perceived misinterpretation of the Phase II Report when responding to press or social media inquiries or requests for comment.
6. For greater certainty, this Code of Conduct does not apply to any SamKnows measurement panel, or any data generated from such panel, that any ISP or SamKnows operates independently of the CRTC Broadband Canada Measurement Platform.
7. The Participants shall ensure that their employees, agents and representatives, as appropriate, act in accordance with this Code of Conduct.
8. The ISPs shall utilize customer recruitment methods that ensure a random selection of panelists for each speedtier and geographic area served pursuant to the Sample Plan.
9. Each press release and any other communication (including, without limitation, the Phase II Report and any announcement published on the Measuring Broadband Canada website, the CRTC’s, and SamKnows’ official website(s) and social media platform(s)) issued by or on behalf of the CRTC or SamKnows that: (i) describes the Project or its objectives, or (ii) describes the scope of Phase II testing or sampling methodologies, or (iii) references any Phase II test results, must, in each case, prominently include the following statement: “While some providers do offer advertised speeds of 1Gbps or faster, services above 940Mbps cannot be measured with a conventional speed test. As such, speed tiers of 1Gbps or faster are excluded from the 2020 Measuring Broadband Canada project. Consumers should visit each of the participating ISP’s websites for details about the availability of higher or lower speed tiers.”. Notwithstanding anything to the contrary in the Project documentation, the initial press release announcing Phase II of the Project and the press release announcing publication of the Phase II Report, including any subsequent changes to the foregoing, issued by the CRTC or SamKnows are subject to review and approval in accordance with Section 14 of this 2020 MOU.
10. Any breach of Section 9 of this Appendix A will be a material breach of the 2020 MOU and: (i) the breaching Party shall cure any material breach within forty-eight (48) hours of becoming aware thereof; and (ii) if SamKnows is the breaching Party, then the ISPs will be entitled to withhold any and all payments in connection with the Project, whether under the 2020 MOU or otherwise, until such material breach is cured.