Commission Letter addressed to Ted Woodhead (Rogers Communications Canada Inc.)
Ottawa, 5 August 2022
Our reference: 8000-C12-202203868
Senior Vice President, Regulatory Affairs
Rogers Communications Canada Inc.
333 Bloor Street East
Toronto, ON M4W 1G9
RE:Rogers Canada-wide service outage of July 2022
Dear Mr. Woodhead:
This letter is in relation to the national service outage experienced by Rogers Communications Canada Inc. (Rogers) that began on 8 July 2022 and is in reply to Rogers’ response dated 22 July 2022 to the Commission’s request for information (the Response). The Response provided valued information for the Commission to understand the cause of the outage, impact of the outage and immediate steps to mitigate the impact.
Since the Response, Rogers has made public statements about measures to reduce the likelihood of similar events as well as address consumers’ concerns. To better understand the impact of the event, mitigating measures as well as these recent announcements, further information is required for the Commission to assess this situation.
Given the seriousness of this event, and while the Commission considers the complaints and calls for a public inquiry before it, Rogers is to provide, by 15 August 2022, comprehensive answers, including rationale and any supporting information, to the questions included in the attachment.
This letter and any subsequent correspondence will be placed on the public record. Should Rogers designate any information in its response as confidential pursuant to section 39 of the Telecom Act, an abridged version of the response must be provided for the public record. Note that, in accordance with its normal practices, the Commission may disclose or require the disclosure of information designated as confidential if its disclosure is in the public interest, i.e., where any specific direct harm likely to result from disclosure does not outweigh the public interest in disclosure.
The Commission has yet to make a determination with respect to disclosure in response to PIAC’s request for disclosure of Rogers’ response to the initial request for information. Any disclosure findings made in that context will automatically be applied to Rogers’ response to this second request for information. Procedure for further requests for disclosure with respect to this second response will be addressed once the disclosure requests with respect to the first letter are applied. It is also noted that certain public statements by Rogers and others may have rendered some information designated confidential in the Response no longer confidential. In response to this second request for information, Commission staff reminds Rogers to disclose information on the public record to the maximum extent possible.
Original signed by
Ian Baggley for
cc.: Tony Staffieri, President and CEO, Rogers, email@example.com
Éric Dagenais, Senior Assistant Deputy Minister, ISED, firstname.lastname@example.org
Howard Slawner, Vice President, Regulatory Telecom, Rogers, email@example.com
Michel Murray, CRTC, firstname.lastname@example.org
Noah Moser, CRTC, email@example.com
Request for Information
Questions to Rogers) Footnote1 in relation to the nationwide service interruption of July 2022.
As requested by Mr. Han Dong during the Commission’s appearance before the Standing Committee on Industry and Technology on 25 July 2022, please quantify the direct economic losses of the outage. Further, please provide the methodology to calculate the impact as well as any assumptions.
During Rogers’ appearance before the Standing Committee on Industry and Technology on 25 July 2022, Mr. Tony Staffieri stated that Rogers was spending at least $250M to physically separate its Internet and wireless networks as well as investing $10B over the next three years to build out and strengthen its network:
- Provide a detailed cost breakdown of the $250M for physically separating the network;
- Provide a detailed description of implementation timelines;
- Provide a detailed explanation of how the physical separation of the networks would improve resiliency;
- Provide a technical description, associated cost breakdown and timelines related to the network upgrades/changes to be undertaken as part of the $10B investment;
- Provide a detailed explanation as to how the upgrades and changes described in response to question 2(d) will improve resiliency of Rogers’ network.
In response Rogers (CRTC) 11July 2022 – 4(iii), Rogers stated “All customers will be getting 5 days of compensation. There is no distinction between residential and small business customers.” Confirm that all residential and small business customers received the credit.
Network issues prior to outage
After the outage, there were media reports indicating that certain customers were observing issues with Rogers’ services in the weeks prior to the outage) Footnote2 :
- Was Rogers aware of these issues?
- Were any problems identified in the weeks leading up to the outage?
- What mechanisms did Rogers have in place to monitor recurrent issues in its network?
General Technical Information
In the response to Rogers (CRTC)11July 2022 - 1(i), Rogers noted that # # change that deleted a routing filter which led to the outage. What was the actual network policy change?
- What lines were removed and/or added that triggered the outage?
- Why were those configuration items removed and/or added?
- More specifically, what filter was removed, and why?
- What was the expected behavior of the routing filter removal on the # # routers?
- Was this removal part of the approved Method of Procedure?
- What is the default behavior of the # # routers?
Explain in more detail how the # # change led to the # # issue. # #
The timeline of events in Rogers (CRTC)11July 2022-1_i_Appendix refers to two changes: # #
In the response Rogers (CRTC)11July2022 - 1(iii), Rogers noted that some wireless customers "had intermittent service(s) throughout the day on our GSM and 3G networks".
- Why and how did GSM and 3G voice calls fail if the impact was to the Rogers core IP network?
- Which GSM and 3G wireless services were impacted and why?
- Is it typical for 3G and mobile services such as voice, text, 9-1-1 to be impacted by an IP core failure?
With respect to the process improvements implemented following the April 2021 outage and noted in Rogers (CRTC)11July 2022 - 1(ix), # #
With respect to the immediate actions Rogers is taking in examining its "change, planning and implementation" in response to the 8 July 2022 outage as detailed in the response to Rogers (CRTC)11July 2022 - 1(x):, # # How was this change classified? Was it high-risk?
Impact to 9-1-1 Services
In the response to Rogers (CRTC)11July 2022 -1(xviii), Rogers stated that its Network Operations Center (“NOC”) notified the ILECs (i.e. 9-1-1 Network Providers) at 8:39am EDT and that the ILECs then notified the Public Safety Answering Points (“PSAPs”). Given that the outage began at 4:45am, explain why it took nearly 4 hours to initiate the notification process to PSAPs.
In the response to Rogers (CRTC)11July2022 - 2(xi), Rogers explained that Bell and TELUS confirmed that some of Rogers’ customers were able to connect to Bell and TELUS’ respective wireless networks in order to place 9-1-1 calls. Explain why some calls were able to connect to other wireless networks to make 9-1-1 calls while others were not?
In the response to Rogers (CRTC)11July 2022 - 2(xi), Rogers stated that it was able to route thousands of 9-1-1 calls on 8 July 2022 and that its wireless network worked intermittently during that day as it was trying to restore the IP core network, varying region by region. Further, in the response to Rogers (CRTC)11July 2022 - 2(xii), Rogers’ noted that its RAN remained in service on 8 July 2022 and that many Rogers customers phones did not attempt to connect to another network.
- As ‘the RAN remained in service’, explain in what circumstances ‘the wireless network worked intermittently’? Was this in areas where the IP core network had been restored or where the IP core network worked intermittently?
- Describe why some wireless 9-1-1 calls were successful despite the catastrophic failure of the IP core?
In the response to Rogers (CRTC)11July 2022 - 2(xii), Rogers explained that generally, if a customer’s device cannot connect to their own carrier’s RAN, the device will automatically connect to the strongest signal available, even from another carrier, for the purpose of making a 9-1-1 call. However, since Rogers’ RAN remained in service on 8 July 2022, many Rogers customers phones did not attempt to connect to another network. In hindsight, and given the significant number of unsuccessful wireless 9-1-1 calls on 8-9 July 2022, what advice should Rogers have given their subscribers, including registered Text with 9-1-1 users, and PSAPs, about how to access 9-1-1 during the outage?
In the response to Rogers(CRTC)11July2022 -2(xvi), Rogers outlined its plan to enhance communications to its customers, 9-1-1 network providers, emergency management officials, PSAPs, 9-1-1 governing authorities and first responders in relation to access to 9-1-1 during a network outage, many of which include updated information on webpages, e.g. how to remove a SIM card and Wi-Fi calling. Given that Rogers’ subscribers were not able to access the Internet and therefore those webpages during the outage, explain whether other mediums should be considered, such as a paper insert/Tips card and/or instructions available directly on the wireless device.
Impact on Emergency Alerting
In the response to Rogers (CRTC)11July 2022 - 1A(iii), Rogers stated that a number of Rogers Media broadcasting services were unable to deliver emergency alerts during the outage. Are you aware of other broadcasting clients of Rogers who rely on Rogers’ services to establish a connection to the National Alert Aggregation and Dissemination (NAAD) System that were therefore unable to deliver emergency alerts during the outage?
In the response to Rogers (CRTC)11July 2022 - 2(iii) it is indicated that Rogers will be implementing a solution to maintain alerting capabilities on its radio and television stations. Will this solution be applicable to broadcasters that rely on Rogers’ services for internet to establish a connection to the NAAD to deliver emergency alerts to their audience? Further, are there any potential solutions available to broadcasters that rely on Rogers’ services for internet to establish a connection to the NAAD?
Impact to Broadcasting Services
Beyond Rogers’ own broadcasting services, is Rogers aware of the impact of the 8 July 2022 outage on other radio, television or broadcasting distribution services? If so, provide a complete and detailed report on the services impacted during the service outage that began on 8 July 2022, including but not limited to:
- which television, radio and broadcasting distribution services were impacted, how they were impacted and for how long?;
- any communication that was sent to the impacted broadcasting services?;
- the compensation that will be provided to the impacted broadcasting services, if any;
- the measures or steps, if any, that Rogers has, or plans to put in place to prevent issues such as those that led to the outage, as well as the timelines to put in place any future remedial measures.
- Date modified: