Measuring Broadband Canada – Phase 3 – Fixed wireless Canada

Download this report in PDF (1.6 MB).

Report, May 2024

View previous reports:

• June 2020
• March and April 2016
• October and November 2015

The CRTC published two reports in the past as part of the Measuring Broadband Canada project which was a collaborative effort with several Canadian Internet service providers (ISPs) and SamKnows, a UK-based company specializing in broadband measurements. This project focused specifically on the performance of fixed wired broadband Internet services provided to Canadian homes.

Building on these studies, there is a need to assess the performance of ISP-reported speed data for download speed of 50 megabits per second (Mbps) and upload speed of 10 Mbps (50/10 Mbps) and above delivered via fixed wireless access (FWA) technology.

Table of contents

• About the Project
• Executive summary
• Quality of service
  • Speed tests
  • Latency
  • Outages
• Impacts of weater
  • Precipitation
• Survey data
  • Canadian Fixed Wireless survey
• Conclusion
• Methodology
  • Whiteboxes
  • Measurements
  • Test destinations
  • Data processing
• Appendix
  • Glossary
  • Supporting tables

About the Project

The Canadian Radio-television and Telecommunications Commission (CRTC) launched the Measuring Broadband Canada project to better understand how Canadians are experiencing Internet performance in their own homes.

The first phase was completed with the publication of the SamKnows’ phase 1 report in 2016. The project’s second phase was conducted in October 2019, followed by the publication of the SamKnows’ phase 2 report in September 2020. Both phases measured the fixed wired Internet service performance of Canadian Internet service providers (ISPs).

In 2022, the CRTC in partnership with Innovation, Science and Economic Development Canada (ISED) started conducting the third phase of the project. This phase is focused specifically on the performance of fixed wireless Internet services using any technology, with download speeds of 50 Mbps and upload speeds of 10 Mbps or faster provided to Canadian homes. The goal of the testing is the same as the previous phases, this time focused on fixed wireless technology and various external factors to better understand fixed wireless Internet performance in Canada.

The data presented in this report was collected between the 15th of January 2024 and the 13th of February 2024, a total of 30 days. In total, 42 units passed data validations and processing checks to make it into this report.^{Footnote 1} Due to this sample size, it was not possible to split the panel by package or ISP.

Executive summary

This report presents the findings of the measurement study conducted by SamKnows on behalf of the CRTC and ISED from January and February 2024, and investigates the following Internet performance metrics: download and upload speeds, latency, and external effects.

The study was conducted using data collected from 42 Whiteboxes that were deployed to Canadian homes. Volunteers were selected to take part in the study based on connection type and Internet package. Each Whitebox conducts end-to-end tests 24 hours a day, 7 days a week to test servers located in major Canadian metropolitan areas and aims to provide a representative picture of the status of Internet performance across Canadian Fixed Wireless 50/10 Internet users.

Unless otherwise stated the results presented are taken from peak periods of user activity at the local time, which is defined as 7 pm to 11 pm. Off-peak periods are defined as any hour and day exclusive of peak periods.

On average, both the advertised download and upload speeds were achieved across a 24-hour period. Tests were spread across a wide range of speeds, showing the variation in Fixed Wireless performance.

Outages were found to last less than 60 seconds in 92% of instances.

Local weather data was used to compare Fixed Wireless performance in varying conditions. On average, download speed was highest when there was no registered precipitation, peaking at 53 Mbps during all hours and 49 Mbps during peak hours. Rainfall and snow coincided with lower download and upload speeds during this measurement period.

Quality of service

Speed tests

Download refers to the process of receiving data from a remote server or another device on the Internet to a user’s device. It’s a fundamental aspect of online activities that involve accessing content such as web pages, streaming videos, downloading files, and more. The quality of the download connection and speed can have a significant impact on user experience across various online tasks. Faster download speeds lead to quicker loading times for web pages, seamless playback of high-definition videos on platforms like YouTube, Netflix, and streaming services, and faster downloading time for files.

Upload refers to the process of sending data from a user’s device to a remote server or another device on the Internet. It is the opposite of download. Upload is an essential aspect of online activities such as sending emails, sharing files, posting content on social media, video conferencing, online gaming, and more. A low upload speed can lead to, for example, larger files taking a long time to upload, leading to delays in sharing important content.

Peak hours between 7-11 pm. Error bars show 95% confidence of the mean.

Long Description

This bar chart illustrates the average speeds for households, both the download and upload speeds, during peak and all-time hours.

Period	Download (Mbps)	Upload (Mbps)
24/7	51	12
Peak	48	11

Key Observations

• On average, both the advertised download and upload speeds were achieved across a 24-hour period.
• During peak hours (7-11 pm), both speeds dropped slightly. While upload remained above the advertised speed of 10 Mbps on average, download speed was 2 Mbps lower than the 50 Mbps threshold. Figure 2 looks at this in more detail.
• With a limited number of units, the 95% confidence intervals are slightly larger, meaning 95% of the time download will fall between 45.2 Mbps and 57 Mbps during all hours. For upload, this ranges from 7.58 to 15.5 Mbps.

Average represented by mean

Long Description

This chart illustrates the average hourly download speed across a 24-hour period.

Hourly speed	Download speed (Mbps)
12 AM	59.1 Mbps
1 AM	59.1 Mbps
2 AM	59.1 Mbps
3 AM	59.1 Mbps
4 AM	59.1 Mbps
5 AM	59.1 Mbps
6 AM	53.0 Mbps
7 AM	53.0 Mbps
8 AM	53.0 Mbps
9 AM	53.0 Mbps
10 AM	53.0 Mbps
11 AM	53.0 Mbps
12 PM	48.0 Mbps
1 PM	48.0 Mbps
2 PM	48.0 Mbps
3 PM	48.0 Mbps
4 PM	48.0 Mbps
5 PM	48.0 Mbps
6 PM	38.9 Mbps
7 PM	38.9 Mbps
8 PM	47.0 Mbps
9 PM	47.0 Mbps
10 PM	57.0 Mbps
11 PM	57.0 Mbps

Key Observations

• Figure 2 shows the average hourly speeds across a 24-hour period. The highlighted blue section shows the hours defined in this report as peak hours.
• Download speed peaks during the hours of 12 am to 5 am, which is often when the network is used the least. During these hours, the average download speed is 59.1 Mbps across the panel.
• The slowest speeds are registered between 6 pm and 7 pm. During these hours, network congestion is highest and often leads to slower speeds for the user. This results in the lows of 38.9 Mbps across the panel.

Test level data

Long Description

This bar chart illustrates the download speed during peak and 24-hour periods.

Test period	24 - hour (Mbps) A	Peak hour (Mbps) B
Less than 10	4%	5%
10 to 20	6%	8%
20 to 30	10%	11%
30 to 40	13%	14%
40 to 50	21%	19%
50 to 60	10%	8%
60 to 70	12%	11%
70 to 80	12%	12%
80 to 90	7%	7%
90 to 100	3%	2%
More than 100	3%	2%

Key Observations

• Figure 3 shows the distribution of all tests across the reporting period.
• Across a 24-hour period, 54% of all tests fall below the advertised 50 Mbps threshold. There are also households that experienced speeds well above 50 Mbps, with 13% of test speeds measuring above 80 Mbps.
• During peak hours, 57% of all tests were below 50 Mbps. As shown previously, this is during a period when the network is most congested as users are more likely to be online. Across the measuring period, 43% of all households still managed speeds above this threshold.

Latency

Latency refers to the delay or time lag between sending a request or data and receiving a response. It’s the time it takes for data to travel from its source to its destination. Latency is measured in milliseconds (ms) and can have a significant impact on user experience in various online activities. Slow response times due to high latency can lead to slower webpage loading, longer waits for content to appear, and a generally less smooth browsing experience, while high latency can also cause buffering issues in video streaming services. Here we refer to latency as round-trip time, which includes both the time it takes for the packet to be sent and the acknowledgment it has been received.

A latency under load test is designed to assess how a network or system performs under heavy traffic loads while measuring the impact on latency. This type of test helps identify potential bottlenecks, performance degradation, and latency issues that can arise when a network or system is under stress. This test measures the latency whilst the broadband connection is heavily utilized (by way of a speed test run in parallel). This is more representative of user experience than idle latency as it shows the impact of downloading or uploading data to the Internet (e.g. gaming or streaming videos) on latency.

Some ISPs look to combat these issues by implementing Smart Queue Management (SQM) in their routers, while many use quality of service (QoS) settings, which will prioritize certain, user-defined traffic flows. Depending on the individual ISP, SQM can help to prioritize the traffic that needs to get sent through quickly to avoid any bottlenecks and prevent high latencies.

Peak hours between 7-11pm

Long Description

This bar chart illustrates the average idle latency and responsiveness for households.

Latency (ms)	24/7	Peak
Idle latency	37 ms	39 ms
Latency under download	163 ms	174 ms
Latency under upload	398 ms	415 ms

Key Observations

• Figure 4 shows the average idle latency and responsiveness under working conditions of the Internet connection (also known as "buffer bloat" or latency under load).
• With an idle connection, latency averages 37ms across the panel during all hours. This increases slightly to 39 ms during peak hours.
• During an upstream test, latency is highest with a roundtrip time of 415 ms during peak hours. This suggests that when uploading a large file, a user will experience the highest delay on their connection, while heavy download activities, such as downloading a large video file, will cause latencies of around 174 ms during peak hours.

Average represented by mean

Long Description

Average hourly idle latency across 24 hours.

Hour	Average latency (ms)
12 AM	37.0 ms
1 AM	37.0 ms
2 AM	35.5 ms
3 AM	35.5 ms
4 AM	36.0 ms
5 AM	38.0 ms
6 AM	36.0 ms
7 AM	36.0 ms
8 AM	37.5 ms
9 AM	37.6 ms
10 AM	36.0 ms
11 AM	36.7 ms
12 PM	37.0 ms
1 PM	37.3 ms
2 PM	37.4 ms
3 PM	38.0 ms
4 PM	38.5 ms
5 PM	39.0 ms
6 PM	40.0 ms
7 PM	42.0 ms
8 PM	39.0 ms
9 PM	39.0 ms
10 PM	38.0 ms
11 PM	38.0 ms

Key Observations

• Figure 5 shows the hourly average idle latency across a 24-hour period. The highlighted blue section shows the hours defined in this report as peak hours.
• Idle latency peaks at 7 pm, when network congestion is busiest.
• While there is a slight increase in peak hours, this is likely to be imperceptible to the average user in everyday Internet activities on an idle connection. For general Internet browsing, social media usage, and even video streaming, latency in the range of 35-42 ms or lower is generally considered desirable. Users typically start to notice latency issues more prominently when the values reach higher levels, such as over 100 ms.

Outages

The frequency of Internet disconnections experienced by users can vary widely based on several factors, including the quality of their Internet connection, the stability of their ISP’s network, the technology used for Internet access, and local environmental conditions.

This test records instances when two or more consecutive packets are lost to the same test server. Alongside each event, the timestamp, number of packets lost, and the duration of the event are recorded.

In this report, outages are recorded if they last more than 30 seconds. Furthermore, disconnections recorded during the hours of 1 am to 6 am are excluded as this is an off-peak period with many ISPs and could include occurrences of scheduled network maintenance.

Please note, in this metric, outages are not exclusively the fault of the connection and can be caused by unique conditions within a household.

Long Description

This bar chart illustrates the outage frequency for all units on the panel.

A– Outage frequency

Description	Percentage of households
No outages	3%
Outages at most every other day	32%
Outages on most days	24%
Outages at least once a day.	41%

This bar chart illustrates the outage duration for all units on the panel.

B– Outage duration

Description	Percentage of outages
30 - 60 seconds	92%
1 - 3 minutes	5%
3 - 10 minutes	1%
10 min or more	1%

Key Observations

• Figure 6 shows both the duration and frequency of outages for all units on the panel.
• All but one unit experienced outages every other day or more, with 41% experiencing at least one outage a day. However, these are very short outages as shown by Figure 6B, meaning the connection was re-established very quickly. Major outages often last well over 3 minutes and constituted only 2% of all outages in this measurement period.
• Outages while more frequent, were short, potentially caused by weather conditions or a brief loss of signal.

Impacts of weather

Precipitation

Fixed wireless Internet, which relies on radio signals transmitted between a fixed point (typically a base station or tower) and a subscriber’s antenna or receiver, can be influenced by various weather conditions. While fixed wireless technology is designed to handle certain environmental challenges, severe weather can still have noticeable effects on performance.

Heavy rain can absorb and weaken radio signals as they pass through the raindrops. This attenuation can lead to a decrease in signal strength and, in some cases, result in a temporary loss of connectivity.

To identify the potential effects of weather, we compared the hourly climate data for all units, using local weather stations for each individual unit. These captured the temperature, precipitation, and wind speeds across Canada.

It is worth noting that within our sample strong winds and temperature had very little effect on fixed wireless performance during the measurement period.

Calculated using hourly and climate data.

Long Description

This bar chart illustrates the average download speeds during periods of heavy rain and snow, during peak and all hours.

Download speed (Mbps)	24/7 (All-hours)	Peak hours
None	53 Mbps	49 Mbps
Snow	47 Mbps	40 Mbps
Rain	46 Mbps	41 Mbps

Key Observations

• Figure 7 shows download speeds during periods of heavy rain and snow.
• On average, download speed was highest when there was no registered precipitation, peaking at 53 Mbps during all hours and 49 Mbps during peak hours.
• During periods of rainfall, download speeds dropped to 46 Mbps during all hours and down to 41 Mbps during peak hours.
• Download speeds were lowest at peak times during snowfall, registering 40 Mbps.

Calculated using hourly speed and climate data.

Long Description

This bar chart illustrates the average upload speed during periods of heavy rain and snow, during peak and all-hours.

Upload speed (Mbps)	24/7 (All-hours)	Peak hours
None	11.8 Mbps	12.0 Mbps
Snow	8.6 Mbps	8.5 Mbps
Rain	7.6 Mbps	7.4 Mbps

Key Observations

• Figure 8 shows upload speeds during periods of heavy rain and snow.
• On average, upload speed was highest when there was no registered precipitation, peaking at 12 Mbps during peak hours and 11.8 Mbps during all hours. This slight increase during peak hours is likely due to sample size.
• During periods of rainfall, upload speeds dropped to 8.6 Mbps during all hours and down to 8.5 Mbps during peak hours.
• Upload speeds were lowest at peak times during snowfall, registering 7.4 Mbps, rising slightly for all hours to 7.6 Mbps.

Survey data

Canadian Fixed Wireless survey

The CRTC and ISED partnering with SamKnows conducted the Measuring Broadband Canada project, a survey campaign to investigate Canadians’ interest in the Fixed Wireless landscape. This specifically focused on 50/10 packages and above intending to find out the limitations of availability, price, and awareness of the technology.

To submit a valid response to this survey, a respondent was required to provide information on their current Internet situation. This includes their ISP, package, limiting reason behind wanting a Fixed Wireless 50/10 package, location, and speed data.

In total there were 1157 responses from all over Canada, allowing a wider picture of the Fixed Wireless landscape. The results are displayed below.

Data split by province.

Long Description

This Map illustrates Measuring Broadband Canada's responses to answering the question of why customers do not have 50x10 Mbps.

Province	I was not aware of these packages	I'm interested but it's not affordable	I'm interested but it's not available to my household	I'm not interested	Other
Total Respondents	28%	17%	32%	18%	5%
Alberta	32%	15%	29%	17%	7%
British Columbia	27%	18%	50%	5%	0%
Manitoba	23%	15%	36%	26%	0%
New Brunswick	33%	19%	29%	17%	1%
Newfoundland & Labrador	23%	21%	33%	23%	0%
Nova Scotia	29%	15%	32%	24%	0%
Prince Edward Island	13%	50%	25%	0%	13%
Ontario	27%	15%	34%	20%	3%
Quebec	26%	20%	29%	16%	10%
Saskatchewan	39%	22%	33%	0%	6%

Key Observations

• Figure 9 shows the results of the Measuring Broadband Canada survey. 32% of the total respondents were interested in Fixed Wireless, but it was not available to their household.
• The highest number of respondents (34%) were in Ontario where 150 people wanted Fixed Wireless but could not get it.
• Of all of Canada, only 18.2% of respondents were not interested in Fixed Wireless, with the highest number of them living in Ontario.
• In total, 384 respondents identified that 50/10 Fixed Wireless was not available to their household. Of these 57% of them were on speeds lower than 50/10, while only 21.6% were not on Fixed Wireless at all.
• There were no valid data submissions for Yukon, Northwest Territories, and Nunavut for this survey.

Conclusion

This report presents numerous findings from a limited review of Fixed Wireless in Canada. Choosing to focus on 50/10 or higher packages, this report found that, in this sample, speeds met these advertised numbers. For download speeds, across a 24-hour period, speeds remained above 50 Mbps, only dropping during peak periods. For upload speeds, even during peak periods, speeds never dropped below 10 Mbps on average.

This report also aimed to look at different latency measures to investigate how Fixed Wireless in Canada performs. Idle latency was found to range between 37-39 ms on average, while during heavy usage, maxed at 415 ms.

Influences of weather on performance were also considered for this report. Firstly, no decrease in performance was found in high winds or extreme temperatures, while both download and upload speeds dropped slightly during rain and snowfall.

The Measuring Broadband Canada project aims to provide Canadians with information on their broadband services, to help inform and educate consumers on the best provider and package for their needs. As with any volunteer-driven project, this would not be possible without the help of volunteers all over the country hosting a Whitebox in their homes.

Methodology

Whiteboxes

SamKnows deployed 231 Whiteboxes to Canadian Fixed Wireless consumers for the purposes of this project. The Whitebox is a consumer-grade device that is installed in a user’s home network between their home modem/router and their devices. The Whitebox’s core function is to measure the quality of the user’s Internet connection. The Whitebox model sent out for this Phase of the project was the Whitebox 8.0+.

The current generation of Whitebox (8.0+) is capable of measuring 1Gbps downstream and upstream over both TCP and UDP.

The specifications of the device are as follows:

• Dual 2.4 GHz and 5GHz WiFi radios, supporting 802.11a/b/g/n/ac
• Dual-core 880MHz CPU
• 128MB RAM
• 16MB flash storage
• 4x 1Gbps LAN interfaces
• 1x 1Gbps WAN interfaces
• USB 2.0 port
• DC power (12V @ 2000mA)

The measurements are conducted autonomously by the Whitebox to a variety of destinations on the Internet. No user interaction is required to conduct measurements; they are executed automatically according to a test schedule.

End-user cross-traffic is monitored continuously by the Whitebox. If cross-traffic exceeds a certain threshold then measurements are not conducted until the cross-traffic subsides. This ensures that the Whitebox’s measurements are not distorted by end-user activity and that the Whitebox’s measurement traffic does not interfere with a user’s experience of the Internet.

A full description of the Whitebox and its features can be found at https://samknows.com/technology/agents/samknows-whitebox

Measurements

The Whiteboxes run a suite of active performance measurements according to a pre-defined test schedule. These include the following network measurements: download speed, upload speed, latency, and packet loss, as well as application measurements such as web browsing performance.

A full description of the methodology underpinning each test can be found at https://samknows.com/tests

Test destinations

The measurements carried out by the Whiteboxes are conducted against four dedicated measurement servers.

These are installed at major peering and Internet exchanges at the following locations:

• Halifax
• Montréal
• Toronto
• Vancouver

Each server met the minimum specifications set out by SamKnows. The dedicated measurement servers are utilized for the download speed, upload speed, and latency measurements.

Data processing

All data included in this report was obtained via tests conducted between the 15th of January and the 13th of February 2024. For inclusion in the final data set, a Whitebox must have:

1. Been successfully identified as 50/10 or above Fixed Wireless based on volunteer information and test results (a combination of latency and speed measurement characteristics), with ISP and location confirmed by a combination of IP address and cell tower locations.
2. Provided a minimum of 5 days of data during the reporting period. If the volunteer’s ISP or Internet package changed during the measurement period, considerations were made based on the number of days of data obtained under each ISP/package, and sample plan requirements.
3. A minimum of 5 samples for inclusion in peak and off-peak verticals.

Data for each Whitebox was then aggregated via a trimmed mean for download, upload, latency, and webpage loading time metrics. This trimmed mean method excludes outlier results in the top and bottom 1% and helps ensure a cleaner dataset that is better representative of the typical performance. For combined chart data, these Whiteboxes are then combined to create an overall average, based on a minimum sample size of 40 Whiteboxes. Given the fact that not enough qualified participants were recruited, the sample size was not big enough to split the Whiteboxes by package, location, or ISP.

Appendix

Glossary

Supporting tables

Report submitted by: SamKnows

The Pavilion, 1 Newham's Row, London SE1 3UZ, United Kingdom