Testing the Response of Mobile Telephones to Wireless Public Alerts

Table of Contents

• Preface
• Executive Summary
• Background
• Results and Findings
  • Device-to-Device Variance in Baseline and Full Volume
  • Device-to-Device Variability in Alert Volume Settings
    • Distinct Public Alert Settings
  • Airplane Mode
  • Do Not Disturb
  • Bluetooth Headphone Behaviour
  • Receiving Calls
  • Mobile Devices Sourced from Canadian WSPs
• Conclusion
• Appendix 1: Test Descriptions
  • Methodology
    • Test #1: Device Verification
    • Test #2: Baseline Test
    • Test #3: Sound Off
    • Test #4: Airplane Mode
    • Test #5: Silent Mode
    • Test #6: Vibrate Only
    • Test #7: Do Not Disturb/Sleep Mode
    • Test #8: Baseline Test with Wireless Headphones
    • Test #9: Sound Off with Wireless Headphones
    • Test #10: Full Volume
    • Test #11: Response to Phone Calls

List of Figures

• Figure 1: Comparison of full volume and baseline alert volume
• Figure 2: Examples of Android device volume menus
• Figure 3: Broadcast settings for some devices can override all alarm volume settings for public alerts
• Figure 4: Alarm volume when device set to baseline volume and Do Not Disturb
• Figure 5: Do Not Disturb menu for the Huawei P20 Mate Pro (left) and the LG G8 ThinQ (right)

List of Tables

• Table 1: Summary of public alerts distributed in 2019
• Table 2: Common sounds and their associated decibel levels
• Table 3: Description of the default volume settings that control the volume of audible public alerts
• Table 4: Example of device where ring volume and alert volume are separate
• Table 5: Summary of the settings and whether they silenced public alert alarms

Preface

The Certification and Engineering Bureau (CEB), a partner in this testing endeavour, has reviewed this report and provided the following statement:

“The Certification & Engineering Bureau’s (CEB) role in this study was to provide technical support to the CRTC in the execution of a previously developed test plan”

- Stephane Proulx, Acting Director, Certification and Engineering Bureau

Executive Summary

This report summarizes the findings originating from mobile phone testing that occurred between the 20th and 24th of January, 2020, at the Certification and Engineering Bureau (CEB)^{Footnote 1} in Ottawa, ON. CRTC staff planned the testing in partnership with the Communications Research Centre (CRC) as a result of complaints that the CRTC Client Services received following the transmission of public alerts. The goal of the testing was to document the behaviours for a sampling of the mobile phones available in the Canadian market when subject to public alerts. During testing, CEB and CRTC staff generated wireless public alerts in a shielded environment and observed the resultant behaviour of 10 mobile telephones, available at the time of testing, from 5 major manufacturers.

The testing yielded notable findings with implications to Canadian mobile phone users. Firstly, there is an expected amount of variability in the volume at which mobile devices sound in response to public alerts. Additionally, each device has its own unique settings for controlling public alert volumes, sometimes including separate distinct menus that may not be intuitive to users and could result in alerts sounding at unintended volumes. Results demonstrated that “Do Not Disturb” does not silence audible public alerts on certain mobile phones, whereas, “Airplane” mode successfully blocks out cellular activity, including public alerts on all tested phones. Headphone testing identified a significant amount of variable behaviour; for this reason, no firm conclusions could be drawn. Finally, some mobile phones contained the characteristic Canadian alert tones, other mobile phones did not, and one mobile phone did not receive public alerts throughout testing. Behaviour in response to public alerts may differ based on the origin of the phone.

Background

All wireless service providers (WSPs) in Canada have been required to implement the wireless public alerting capability on their LTE networks since 2018 as part of their participation in the National Public Alerting System (NPAS). The individual Public Management Organizations (EMOs) and other organizations such as Environment and Climate Change Canada (ECCC) act as alert issuers. Consequently, WSPs distribute qualifying alerts through their mobile networks. In 2019, WSPs distributed 131 such alerts in Canada (see Table 1).

CRTC Client Services typically receives correspondence from the public following the distribution of these public alerts. The number of complaints received appears to vary depending, in part, on the time of day the alert is issued, the distribution area, and type of alert. CRTC staff reads, groups, and analyzes the complaints and subsequently identifies common themes to understand the Canadian public’s experiences with public alerts. Some complainants report that the audible public alert volume is not respecting the settings of their mobile phones. Despite their intention to silence or suppress the alarm volume, they perceive that their phones are sounding at high volumes and disregarding the settings.

Staff at the CRTC planned and executed, in collaboration with the CRC and the CEB, a series of tests on mobile telephones in response to complaints. Staff configured 10 mobile phones from 5 manufacturers to different modes and settings, triggered public alerts, and recorded the resultant behaviour. The goal of the testing was to document the behaviours for a sampling of the mobile phones available in the Canadian market when subject to public alerts.

Results and Findings

The following tests were conducted (see Appendix 1: Test Descriptions for a detailed methodology):

• Test #1: Device Verification – Ensured the devices are prepared for testing.
• Test #2: Baseline Test – Response to public alerts when the test item is set at mid-volume.
• Test #3: Sound Off – Response to public alerts when the test item’s volume is set to 0.
• Test #4: Airplane Mode – Response to public alerts when the test item is set to “Airplane” mode.
• Test #5: Silent Mode – Response to public alerts when the test item is set to “Silent” mode.
  • Note: Only performed when “Silent” mode and “Sound Off” settings were different
• Test #6: Vibrate Only – Response to public alerts when the test item is set to vibrate only.
• Test #7: Do Not Disturb/Sleep Mode – Response to public alerts when the test item is set to the “Do Not Disturb” setting.
• Test #8: Baseline Test with Wireless Headphones – Behaviour of public alerts through wireless Bluetooth earbuds when the test item is set at mid-volume.
• Test #9: Sound Off with Wireless Headphones – Behaviour of public alerts through wireless Bluetooth earbuds when the test item’s volume is set to 0.
• Test #10: Full Volume – Response to public alerts when the test item is set at full volume.
• Test #11: Response to Phone Calls – Test to determine if public alert and phone call volumes could be set independently

Device-to-Device Variance in Baseline and Full Volume

In Test #2 and Test #10, the mobile phones were set to middle (baseline) and maximum (full) volume, respectively. Staff broadcast the alerts to the mobile phones and recorded the resulting alert volumes. The results collected from testing demonstrated that there is device-to-device variability in audible public alert volumes with respect to their baseline and full volume alerts (see Figure 1)^{Footnote 3}.

Long description

Comparison full volume and baseline alert volume in decibels for the ten tested phones. The iPhone 7 produced a baseline volume between 95-100 dB and a full volume between 105-110 dB. The iPhone 8 produced a baseline volume between 90-95 dB and a full volume between 105-110 dB. The Samsung A70 did not produce any alerts. The Samsung A9 produced a baseline volume between 75-80 dB and a full volume between 95-100 dB. The Google Pixel 3a produced a baseline volume between 80-85 dB and a full volume between 100-105 dB. The Google Pixel 3 XL produced a baseline volume between 80-85 dB and a full volume between 100-105 dB. The LG G8 ThinQ produced a baseline volume between 70-75 dB and a full volume between 85-90 dB. The LG G7 produced a baseline volume between 90-95 dB and a full volume between 105-110 dB. The Huawei P30 produced a baseline volume between 80-85 dB and a full volume between 100-105 dB. The Huawei P20 Mate Pro produced a baseline volume between 80-85 dB and a full volume between 100-105 dB.

Some phones achieved higher volumes when alerting, which is not unexpected as speaker capability varies by device. The iPhone 7, iPhone 8, and the LG G7 produced peak volumes above 105 dB and baseline volumes above 90 dB. All other phones produced baseline volumes of approximately 85 dB and below.  In contrast, the LG G8 ThinQ produced a full volume alert at approximately 90 dB and a baseline volume of approximately 74 dB. For context, see Table 2 for common noises and their approximate magnitudes in decibels:

Table 2 illustrates the notable difference between decibel levels and shows how, given the volumes measured during testing, the experiences of the public with respect to alert volumes can vary.

Device-to-Device Variability in Alert Volume Settings

Repeated alerts were sent to each phone to learn how to control alert volumes. This activity and the tests described in this report revealed that different device manufacturers may use different volume controls for controlling public alert volume for each model of device; this was more evident with Android phones. Android phones generally have four or more volume controls for customizing the volume settings; it is not always apparent which one controls the volume of public alerts (see Figure 2).

Long description

Two examples of Android device volume menus. The first menu (left) has separate volume controls for 1) media, 2) ringtone, 3) alarms, and 4) calls. The second menu (right) contains volume controls for 1) ringtone, 2) notification sound, 3) touch feedback & system, and 4) music, video, games & other media.

Table 3 lists the default volume setting that controls the volume of audible public alerts for each tested device. CEB and CRTC staff learned how to control alert volume for each device by sending many alerts throughout testing. The average member of the public does not have the necessary equipment and facilities at their disposal to learn, through trial and error, exactly how to control the volume of public alerts. Therefore, there are many possible scenarios in which an individual may conclude that their phone is ignoring their settings when, in fact, they have configured the phone incorrectly.

Mobile phones have volume controls to customize the volumes for notifications, alarms, phone calls, music, media, etc. In many cases, the volume of a phone call ring can be modified separately from public alert volume. However, in some cases the alert sets the magnitude of the alert volume but the ring volume being set to “ON” or “OFF” dictates whether the public alert will sound or not. In these cases, setting the alert volume to the lowest level yields a very low volume alert, but setting the ring volume to the lowest level eliminates the sound of the alert (see Table 4).

The iPhone 7 and iPhone 8 have volume buttons on the sides of the phone that control either the media volume or the ring volume; by default, the buttons on the sides of the device control the ring volume. The user may modify the settings to allow these buttons to control the media volume; the media volume controls the volume of public alerts. Therefore, users may mistakenly use these buttons to lower the volume of all incoming communications but be unaware that the public alert volume is unaffected unless these buttons have been configured to modify media volume.

Distinct Public Alert Settings

Preliminary testing with the Huawei P30 demonstrated that the volume settings had no effect on the alert volume, and in every test case, the phone was emitting audible public alerts at full volume. However, the testers discovered that the device had a separate menu for public alerts with a toggle switch labelled “Use full volume” (see Figure 3). When this switch is activated, the phone ignores all volume settings and “Do Not Disturb” functionality and sounds an audible alarm at full volume.

Long description

Use full volume

Ignore other volume and Do not disturb preferences.

Members of the public may not be aware of this menu with their devices and could become frustrated with their inability to control the volume of audible public alerts.

Airplane Mode

CRTC staff sought to confirm whether or not “Airplane” mode always suppressed alarms, as expected.  Testing showed that if a user opts to place a device on “Airplane” mode, all cellular services are disabled, including the distribution of public alerts. Since “Airplane” mode ceases all cellular services, the device does not receive calls, text messages, or other forms of incoming data.

Do Not Disturb

Some members of the public may believe that activating “Do Not Disturb” mode on their device prevents alerts from alarming audibly. Consequently, CRTC staff planned and executed testing to document the behaviour of select mobile phones placed in this mode. Results demonstrated that this configuration mode did not suppress public alert alarms in many cases. The iPhones, the Samsung S9, as well as the Huawei P30 sounded an audible alarm despite being set to “Do Not Disturb” mode (see Figure 4).

Long description

The alarm volumes when devices are set to baseline and Do Not Disturb. The Samsung A70, Google Pixel 3a, Google Pixel 3 XL, LG ThinQ, LG G7, and Huawei P20 Mate Pro did not produce any sound when set to Do Not Disturb. The iPhone 7 produced an alarm between 95-100 dB, the iPhone 8 produced an alarm between 90-95 dB, the Samsung S9 produced an alarm between 70-75 dB, and the Huawei P30 produced an alarm between 80-85 dB.

The “Do Not Disturb” functionality on the Google phones set the alarm volume to ‘0’ and rendered it inaccessible in the drop-down menu; this effectively silenced the alarm. The LG G8 ThinQ device has both a “Sleep” mode and a “Do Not Disturb” mode and each mode allows the user to select “Priority only” or “Total Silence”. In both cases, only selecting “Total Silence” silenced public alerts. Finally, the Huawei P20 Mate Pro has three possible sub-selections under “Do Not Disturb”; these selections are “Allow Priority”, “Alarms only”, and “Don’t Interrupt”. The “Don’t Interrupt” selection was the only option that silenced alarms during testing.

Long description

The Do Not Disturb menu for the Huawei P20 Mate Pro (left) has individual options to 1) allow priority interruptions only, 2) allow alarms only, and 3) don’t interrupt. The Do Not Disturb menu for the LG G8 ThinQ (right) has individual options to 1) allow Priority only, which does not silence priorities and alarms, and 2) Total silence, which silences all notifications.

In summary, configuring devices to “Do Not Disturb” or “Sleep” mode yields notable device-to-device behavioural variability in response to public alerts. “Do Not Disturb” mode does not suppress public alerts in some cases; whereas, only selecting the correct sub-selection suppresses the alarm in other cases. It is not necessarily evident to the user which selection suppresses audible public alerts.

Bluetooth Headphone Behaviour

CRTC staff planned and executed exploratory testing using the mobile phones and two sets of wireless Bluetooth headphones to observe the behaviours of public alerts when the receiving phone is connected to Bluetooth headphones. The testing demonstrated that both the headphones and the mobile phone influence how the devices deliver audible public alert alarms to the user.

In certain cases, the alert caused an initial high volume blare through the headphones, followed by an alarm at a lower volume. In other cases, the alert caused an initial non-descript sound in the headphones followed by silence.  With some mobile phones, the tester could control the volume in which the audible alarm sounded in the headphones, whereas with other devices, the alarms always sounded at the same volume regardless of media settings. Finally, for some devices, the alarm sounded from the speakers on the mobile phone itself and not the headphones. In the cases above, the results depended both on the headphones used and the mobile phone used.

The subject testing was exploratory in nature and not intended to characterize the complete range of all possible behaviours. However, results showed that user experiences with wireless headphones may vary. CRTC staff has received some public complaints from members of the hearing impaired community. These individuals indicated that they require that their assistive hearing devices be connected to their mobile phones most of the time. Often independent testing is conducted to ensure devices are compatible with assistive hearing devices, and phones may have independent settings for these purposes. The subject testing was not comprehensive in this area.

Receiving Calls

Members of the public sometimes expressed their desire to receive calls while they slept, but suppress audible public alerts. The reasons given for this type of request included examples such as: 1) caregivers for aging parents who need to receive calls, and 2) those that work on an on-call basis. In response to these types of complaints, the testing sought to determine whether the subject mobile phones could have their public alerting alarm settings configured separately from their incoming phone call settings. Based on the results acquired during testing, the testers configured all mobile phones to the following settings:

• Vibrate on receipt of public alert
• Silent on receipt of public alert

As described earlier in this report, each mobile phone had its own settings for vibrating or being silent upon alert. Once it was verified that the correct settings were chosen for each mobile phone, the testers called these phones and documented their subsequent behaviour.

In the case of the iPhones, the Huawei P20 Mate Pro, and the LG G7 phones, if the devices are set to silent or vibrate on public alerts, they could not be set to ring audibly for phone calls.

The following devices could receive an audible ringtone for phone calls despite being set to silent or vibrate for public alerts:

• Samsung S9
• Google Pixel 3a
• Google Pixel 3 XL
• LG G8 ThinQ
• Huawei P30

The results demonstrated that in some cases, it was possible to set call settings separately from public alert settings; however, without the luxury of calibrating their phone settings through repetitive public alerts, members of the public may not be able to determine if this is possible with their device.

Mobile Devices Sourced from Canadian WSPs

Testing revealed that four of the mobile devices contained the characteristic Canadian public alert tone, comprised of alternating complex tones lasting for 8 seconds, and the remaining phones did not. This finding was not an intended outcome of the testing, but in hindsight it is important to consider that there may be a number of phones currently being used by Canadians that are not configured for public alerting in Canada.

There are two documents that define the set of functional and technical requirements for WPA in Canada, as per Telecom Regulatory Policy 2017-91, the NPAS Common Look and Feel (CLF)^{Footnote 5} guideline and the ATIS-0700021 standard. These documents also define the Canadian alert tone cadence. In this Regulatory Policy, the Commission directed “wireless service providers to implement wireless public alerting capability on their long-term evolution networks by 6 April 2018”; this directive included adherence to the aforementioned documents. However, only Canadian WSPs are required to follow this policy. Consequently, mobile phones that are not acquired directly from Canadian WSPs may not display the expected behaviour described in the ATIS standard or the CLF. This was evident during testing as the Samsung A70 phone did not receive any public alerts throughout the testing and five other mobile devices contained a non-Canadian alert tone and cadence. It stands to reason that some portion of the Canadian population may therefore experience public alerting in a manner that is different from what is prescribed in the applicable standards and guidance documents.

Conclusion

Mobile testing revealed a number of valuable observations and findings with respect to the behaviour of mobile phones and wireless Bluetooth headphones in response to public alerts. Table 5 demonstrates a summary of the settings applied in testing and whether or not each phone sounded an audible public alert.

Note:

• indicates that the alarm was effectively silenced.
• indicates the alarm sounded regardless of the settings.
• Device did not receive public alerts during testing
• “N/A” signifies that the test was not carried out because either it was captured in another test or the specific setting does not exist in that phone.

The following conclusions were drawn from the observations and findings collected during the mobile phone testing:

• There is a notable amount of device-to-device variability with respect to the settings that control public alert volume; this may include distinct public broadcast settings that can override device settings for public alerts. This finding means that phone users may have difficulty finding the correct settings to control the volume of alerts and may experience frustration when receiving an alert.
•  “Do Not Disturb” or “Silent” modes may or may not suppress public alerts, but often are not intended to. This may be counterintuitive to users who may place their devices in this mode to avoid receiving public alerts during certain times of the day. This finding highlights another potential source of frustration that the public could feel towards the NPAS system.
• “Airplane” mode suppresses the receipt of public alerts. NPAS partners and stakeholders may provide this as an option for silencing alarms to the general public with the caveat that this mode disables all other cellular communications. This may be a viable option for those that do not want to receive interruptions during particular periods of the day, such as during sleep.
• Canadian WSPs are responsible for ensuring that the devices they provide to Canadians have been configured to adhere to the alerting standards and guidelines, whereas manufacturers are not. Therefore, members of the public that purchase devices from sources other than Canadian WSPs may experience inconsistent phone behaviour with respect to public alert broadcasts.
• In some cases, incoming call settings can be configured independently from public alert settings; this is mobile phone dependent. This means that in some cases, consumers may be able to silence public alerts while sleeping whilst being able to receive incoming calls. This finding is relevant to those that work on-call or are required to be reachable during sleeping hours.

Appendix 1: Test Descriptions

Methodology

Test #1: Device Verification

Purpose: Ensure the devices are prepared for subsequent tests.

Steps:

• Ensure that the battery of the test item was at a sufficient level (>75%) for testing
• Record the make, model, and operating system of the test items
• Perform pre-verification testing
  • Identify which settings control alert volume in preparation for subsequent tests
  • Ensure that the device can receive public alerts

Test #2: Baseline Test

Purpose: Quantify the volume and duration of public alerts when the test item is set at mid-volume.

Steps:

• Set the alert volume to approximately half
• Initiate an public alert
• Record the sound output of the device in decibels
• Measure the approximate duration of the public alert
• Record any notable observations

Test #3: Sound Off

Purpose: Quantify and qualify the test item response to public alerts when the test item’s volume is set to 0.

Steps:

• Use the volume control settings to create a silent public alert
• Initiate an public alert
• Record the sound output of the device in decibels
• Measure the approximate duration of the public alert
• Record any notable observations

Test #4: Airplane Mode

Purpose: Quantify and qualify the test item response to public alerts when the test item is set to “Airplane” mode.

Steps:

• Place the test item in “Airplane” mode
• Initiate an public alert
• Record the sound output of the device in decibels
• Measure the approximate duration of the public alert
• Record any notable observations

Test #5: Silent Mode

Purpose: Quantify and qualify the test item response to public alerts when the test item is set to “Silent” mode.

Steps:

• Place the test item in “Silent” mode
• Initiate an public alert
• Record the sound output of the device in decibels
• Measure the approximate duration of the public alert
• Record any notable observations

Note: In certain devices there is a “Silent” mode activated by a hard or soft switch. In cases where “Silent” mode is identical to the conditions of Test #3, no testing was conducted.

Test #6: Vibrate Only

Purpose: Quantify and qualify the test item response to public alerts when the test item is set to vibrate only.

Steps:

• Place the test item in “Vibration” mode
• Initiate an public alert
• Record the sound output of the device in decibels
• Measure the approximate duration of the public alert
• Record any notable observations

Test #7: Do Not Disturb/Sleep Mode

Purpose: Quantify and qualify the test item response to public alerts when the test item is set to “Do Not Disturb” or “Sleep” mode.

Steps:

• Place the test item in “Do Not Disturb” or “Sleep” mode
• Initiate an public alert
• Record the sound output of the device in decibels
• Measure the approximate duration of the public alert
• Record any notable observations

Test #8: Baseline Test with Wireless Headphones

Purpose: Qualify the volume of public alerts through Bluetooth earbuds when the test item is set at mid-volume.

Steps:

• Set the alert volume to approximately half
• Initiate an public alert
• Make a qualitative observation of the sound being emitted from the wireless headphones
• Repeat for second set of headphones
• Record any notable observations

Test #9: Sound Off with Wireless Headphones

Purpose: Qualify the volume of public alerts through Bluetooth wireless earbuds when the test item’s volume is set to 0.

Steps:

• Set the alert volume to 0
• Initiate an public alert
• Make a qualitative observation of the sound being emitted from the wireless headphones
• Repeat for second set of headphones
• Record any notable observations

Test #10: Full Volume

Purpose: Quantify the volume and duration of public alerts when the test item is set at full volume.

Steps:

• Set the alert volume to full volume
• Initiate an public alert
• Record the sound output of the device in decibels
• Measure the approximate duration of the public alert
• Record any notable observations

Test #11: Response to Phone Calls

Purpose: This test was devised on the last day of testing. Some complainants have indicated that when sleeping, they prefer to receive phone calls in case of family emergencies, but not public alerts. The purpose of this test was to observe mobile device behaviour when the public alert alarms are set to silent or vibrate and the device receives a phone call.

Steps:

• Set the mobile device settings to receive a silent public alert
• Make a telephone call to the mobile device
• Observe and record whether the device rings, vibrates, is silent, etc.
• Set the mobile device settings to vibrate only for an public alert
• Repeat the telephone call and subsequent recording