Search

Telecom - Commission Letter adressed to Ted Woodhead (Rogers Communications Canada Inc.)

Ottawa, 12 July 2022

Our reference: 8000-C12-201909780 and 8000-C12-202203868

BY EMAIL

Ted Woodhead
Senior Vice President, Regulatory Affairs
Rogers Communications Canada Inc.
333 Bloor Street East
10th Floor
Toronto, ON M4W 1G9
regulatory@rci.rogers.com

RE: Rogers Canada-wide service outage of July 2022

Dear Mr. Woodhead:

This letter is in relation to the national service outage experienced by Rogers Communications Canada Inc. (Rogers) that began on 8 July 2022. As widely reported throughout Canada and internationally, this event disrupted the lives of several millions of Canadians and prevented access to such services as 9-1-1 and emergency alerting in addition to other critical infrastructure and services. By Monday, 11 July, many Rogers customers were still reporting service interruptions despite messaging on the company’s website indicating that as of 9 July services had been restored and networks and systems were nearly fully operational.

The Commission has received numerous complaints in addition to requests for a public inquiry in relation to this event. In the first several hours of the outage, it became clear that Rogers was either unable to reassure, or ineffective in reassuring, its customers and providing critical information about what to expect. Few details regarding the outage were provided on either the company’s webpage or its social media accounts. For example, Rogers failed to provide Canadians with information about alternate means of reaching 9-1-1 in case of emergency. Individual Canadians also complained of being unable to work, loss of income, lack of access to banking and other essential services, and lack of a comprehensive explanation from Rogers. Many businesses also felt significant economic impacts resulting from this outage.

Rogers has publicly attributed the cause of this service outage to a maintenance upgrade in its core network. This is reminiscent of another significant network outage in April 2021 that Rogers similarly attributed to a software update. Events of this magnitude paralysing sectors of our country’s economy and jeopardizing the safety of millions of Canadians for several hours or days are simply unacceptable, and telecommunications service providers such as Rogers should have mitigation strategies in place to address such possibilities.

As an initial step, comprehensive information is required in order for the Commission and Canadians to understand what happened leading up to, during, and in the aftermath of this national service outage. Just as importantly, it is essential that we understand what is being done to prevent further outages. In exercising its powers and performing its duties under the Telecommunications Act, the Commission must implement the Canadian telecommunications policy objectives set out in section 7 of that Act, including to render reliable and affordable telecommunications services of high quality accessible to Canadians in both urban and rural areas in all regions of Canada.

Given the seriousness of this event, and while the Commission considers the complaints and calls for a public inquiry before it, Rogers is to provide, by 22 July 2022, comprehensive answers, including rationale and any supporting information, to the questions included in the attachment.

This letter and any subsequent correspondence will be placed on the public record. Should Rogers designate any information in its response as confidential pursuant to section 39 of the Telecom Act, an abridged version of the response must be provided for the public record. Note that, in accordance with its normal practices, the Commission may disclose or require the disclosure of information designated as confidential if its disclosure is in the public interest, i.e., where the specific direct harm does not outweigh the public interest in disclosure.

In light of the immense public interest in understanding what happened, Commission staff expects Rogers to disclose information on the public record to the maximum extent possible.

Sincerely,

Original signed by

Fiona Gilfillan
Executive Director
Telecommunications sector

cc.:Tony Staffieri, President and CEO, Rogers, anthony.staffieri@rci.rogers.com
Éric Dagenais, Senior Assistant Deputy Minister, ISED, eric.dagenais@ised-isde.gc.ca
Howard Slawner, Vice President, Regulatory Telecom, Rogers, regulatory@rci.rogers.com
Michel Murray, CRTC, michel.murray@crtc.gc.ca

Request for Information

Questions to RogersFootnote1 in relation to the nationwide service interruption of July 2022.

Current Outage

Provide a complete and detailed report on the service outage that began on 8 July 2022, including but not limited to:

  1. status to date and all relevant timelines (including details of actions taken, what unfolded and what were the results, factors that contributed to the outage and led it to become progressively worse, and why could steps not be taken to contain the outage before it impacted more services such as Interac and others);
  2. what was the root cause of the outage (including what processes, procedures or safeguards failed to prevent the outage, such as planned redundancy or patch upgrade validation procedures);
  3. which Rogers companies and services were impacted and how;
  4. which other telecommunications service providers (TSPs) were impacted and how;
  5. total number of customers impacted, broken down by province, by TSP (for Rogers and each of its affiliates, for each wholesale customer, others), and, where possible, by type of end-customer (residential or personal, small business, all other businesses/enterprises);
  6. impact on federal, provincial, territorial and municipal government services;
  7. a description of the extent of the Interac outage (e.g. only for businesses who had Rogers as their service provider or broader) and extent to which any other critical infrastructure sectors (e.g. health, transportation, energy, etc.) were affected;
  8. how did Rogers prioritize reinstating services and what repairs were required;
  9. what measures or steps were put in place in the aftermath of the earlier-mentioned April 2021 outage, and why they failed in preventing this new outage;
  10. what measures or steps Rogers has, or plans to, put in place to prevent issues such as those that led to this incident going forward, as well as the timelines for any future measures to be put in place;
  11. what is Rogers’ internal process for conducting major network upgrades including governance and accountability for major engineering decisions;  
  12. how did the outage impact Rogers’ own staff and their ability to determine the cause of the outage and restore services;
  13. what contingencies, if any, did Rogers have in place to ensure that its staff could communicate with each other particularly in the early hours of the outage;
  14. how does Rogers plan to improve its internal communications in light of this event;
  15. what information was used to confirm the 9 July message that services had been restored and networks and systems were close to fully operational, and indicate whether this information was accurate and reliable;
  16. provide a list, including timeline, medium and messaging of all communications efforts undertaken by Rogers to advise its customers of this service outage;
  17. how does Rogers plan to improve its communications to its customers and the general public in light of this event;
  18. actions taken by Rogers during this service interruption to mitigate the impact on Canadian institutions, infrastructure and customers, including in relation to emergency services;
  19. extent to which Rogers sought or received assistance from other TSPs in addressing the outage or situation arising from the service interruption;
  20. describe what more Rogers could have done to secure assistance from other TSPs to help address the outage;
  21. whether any service level agreements (SLAs) were breached between specific vendors and Rogers in relation to this outage; and,
  22. whether Rogers breached any SLAs between itself and its customers (e.g. Interac, others) in relation to this outage.

Impact on Emergency Services

Provide a complete and detailed report on the impact on emergency services of the outage that began on 8 July 2022, including but not limited to:

  1. specific impact on emergency services including wireless public alerting and 9-1-1 and details of when access to emergency services was fully restored;
  2. whether the outage specifically impacted the 9-1-1 networks or only the originating networks, and if the former, how was this possible in light of resiliency and redundancy obligations imposed by the Commission;
  3. whether the outage impacted broadcasting services and by extension the ability to issue public alerts via Rogers’ broadcasting operations;
  4. when and how were the operator of the National Alert Aggregation and Dissemination NAAD system, alert issuers and users notified that alerts could not be received on devices connected to the Rogers Network;
  5. whether customers were advised on how they could get alerts for their area during the outage;
  6. how were the Emergency alerts processed during the outage and were devices connected to Rogers’ network able to receive alerts from other providers during the outage;
  7. what measures could be put in place to maintain emergency alerting capabilities during a Rogers’ network outage;
  8. how Rogers prioritized the restoration of alerting capabilities on its network;
  9. number of 9-1-1 calls made that could not be completed as a result of the service interruption, broken down by province and platform;
  10. number of public alerts sent that did not reach Rogers’ customers, broken down by province;
  11. how were 9-1-1 calls processed during the outage and whether they were able to be processed by other wireless networks within the same coverage area;
  12. whether other measures could have been taken to re-establish 9-1-1 services sooner;
  13. what alternatives are available to Rogers’ customers to access 9-1-1 services during such outages;
  14. details of communications related to access to emergency services over the course of the outage and whether more could have been done to explain how customers could reach 9-1-1;
  15. details of communications to Public Safety Answering Points (PSAPs) and 9-1-1 governing authorities during the outage;
  16. Rogers’ plan to enhance communications to its customers, 9-1-1 network providers, emergency management officials, PSAPs, 9-1-1 governing authorities and first responders in relation to access to 9-1-1 during a network outage;
  17. extent to which the network outage affected Next Generation 9-1-1 (NG 9-1-1) networks or was in any way related to Canada’s transition to NG9-1-1; and,
  18. extent to which Rogers sought or received assistance from other TSPs in addressing the impact on emergency services arising from the service interruption.   

Past Outages

Provide a list of all service outages that affected the Rogers network since 1 January 2019, which lasted four or more hours and affected 100,000 subscribers or more at the peak. For each, indicate:

  1. the relevant timelines;
  2. the services impacted;
  3. the cause of the outage:
  4. the number of customers affected, broken out by province and by TSP (Rogers affiliates, wholesale customers and others) and by type of customers (residential, small business, all other businesses/enterprises);
  5. impact on federal, provincial, territorial and municipal government services;
  6. the extent to which any critical infrastructure sectors (e.g. financial, health, transportation, energy, etc.) were affected;
  7. the specific impact on emergency services including public alerting and 9-1-1;
  8. what safeguards were put in place, after each outage, to prevent future outages of that nature;
  9. the compensation provided to customers, distinguishing between residential and business customers;
  10. whether any service level agreements (SLAs) were breached between specific vendors and Rogers in relation to this outage or what caused this outage; and,
  11. whether Rogers breached any SLAs between itself and its customers in relation to this outage.

For each identified past outage, provide a copy of any post mortem report, including lessons learned and subsequent action plan.

Compensation for Customers

In a message from Tony Staffieri, President and CEO of Rogers, posted on the Rogers website as of 8 July 2022, Mr. Staffieri made the commitment to “make this right” for customers by proactively applying a credit to all customers impacted by the outage.

  1. Provide the details of how Rogers is planning to honor this commitment, focusing in particular on residential and small business customers but also including other parties impacted (e.g. wholesale customers and their end-customers, etc.).
  2. Explain how Rogers determined that this level of compensation is appropriate.
  3. Is a distinction being made between residential customers and small business customers when determining compensation?
Date modified: