# Broadcasting Regulatory Policy CRTC 2016-435

References: 2015-325, 2015-325-1, 2015-325-2 and 2015-325-3

Ottawa, 2 November 2016

## English-language closed captioning quality standard related to the accuracy rate for live programming

Canadians who use and rely on closed captioning to watch English-language television programming should have access to the highest possible quality of closed captioning in live programming.

Broadcasters are required to abide by established quality standards when providing closed captioned programming. One of these standards relates to the captioning accuracy rate for live programming.

In Broadcasting Notice of Consultation 2015-325 (the Notice), the Commission called for comments on the effectiveness, measurability and achievability of the English-language closed captioning quality standard related to the accuracy rate for live programming.

Following the call for comments, a group of broadcasters voluntarily formed a working group (the 2016 Working Group) with representatives from an advocacy group for captioning users, closed captioning providers and the Northern Alberta Institute of Technology to look into the issues raised in the Notice. The 2016 Working Group submitted a proposal for a trial to develop and test an alternative method to measure the captioning accuracy, with a view to improving the quality. The working group requested that the Commission temporarily suspend the requirement that broadcasters reach a given captioning accuracy rate for live programming during the trial.

As a result of this constructive and collaborative initiative, the Commission will suspend the requirement that broadcasters achieve a captioning accuracy rate of at least 95% for live programming until 31 August 2019 to allow the 2016 Working Group to conduct a trial of the alternative method of measuring captioning accuracy. During the suspension, all programming will continue to be captioned and the methods used to provide captioning will not change. All other quality standards continue to be in effect. Further, the Commission still expects broadcasters to meet the quality standard related to the accuracy rate for live English-language programming during the trial period.

The revised English-language closed captioning quality standards are set out in the appendix to this regulatory policy.

The Commission expects the 2016 Working Group to submit a final proposal for a new accuracy rate and measurement model for live programming by 2 November 2018. The proposal must include sufficient detail to readily operationalize the standard and must demonstrate how it meets the needs of users. Following the receipt of this proposal, the Commission will hold a public consultation on the accuracy rate for live English-language programming in light of the 2016 Working Group's proposed model.

The collaborative efforts of the 2016 Working Group will help further the objective of ensuring that all Canadians are able to equitably access live English-language television programming.

The French-language closed captioning framework is distinct and remains unchanged.

### Introduction

1. In Broadcasting Notice of Consultation 2015-325 (the Notice), the Commission called for comments on a targeted review of the English-language closed captioning quality standard related to the accuracy rate for live programming.
2. Closed captioningFootnote 1 enables persons who are deaf or hard of hearing to have full access to television. It is also useful for people learning to read or speak a second language, and can allow people to enjoy television programming in restaurants, gyms and other locations where the volume is often muted.
3. Broadcasters must abide by the English-language closed captioning quality standards, which are set out in the appendix to Broadcasting Regulatory Policy 2012-362. The quality standards set out, among other things, what accuracy rates are to be achieved for pre-recorded as well as live programming, how accuracy is to be monitored and reported on, and what format closed captioning is to take.
4. Quality standard #3 requires English-language broadcasters to reach a captioning accuracy rate of at least 95% averaged over a given live program using a verbatim calculation method which consists of counting word by word the number of deviations between the caption file and the original audio. The current calculation method is set out below:

Accuracy rate = (words – errors*) / words x 100

*For the purpose of this standard, an error includes word substitutions, word omission, and word insertion.

5. The quality standards are incorporated by reference into the standard conditions of licence for private broadcasters as well as into the conditions of licence for the Canadian Broadcasting Corporation (CBC). This is intended to further the policy objective set out in section 3(1)(p) of the Broadcasting Act (the Act), which states that programming accessible by disabled persons should be provided within the Canadian broadcasting system as resources become available for the purpose.
6. In October 2014, broadcasters filed, in accordance with quality standard #4, the first biennial reports describing their efforts to improve accuracy rates. One of the reports was filed by the English Broadcasters Group (EBG), an industry group consisting of the Canadian Association of Broadcasters' (CAB) English membership and the CBC (the CAB-CBC Report). In the report, the EBG stated that, despite the broadcasters' best efforts, the accuracy rate set out in quality standard #3 had proven to be unachievable. It also reported that many types of live English-language programming scored well under the required 95% verbatim accuracy rate. Similar concerns were raised by other broadcasters.

### Working group proposal

1. Shortly after the Notice was issued, the EBG voluntarily established a working group (the 2016 Working Group) to respond to the issues raised in the Notice. The 2016 Working Group consists of representatives from broadcasters as well as representatives from Media Access Canada, the Canadian Association of the Deaf, the Canadian Hard of Hearing Association, and the Canadian Hearing Society (collectively, the Captioning Consumer Advocacy Alliance [CCAA]); National Captioning Canada (NCC) and Closed Caption Services (CCS) (closed captioning providers); and the Northern Alberta Institute of TechnologyFootnote 2 (NAIT). A Commission staff member attended the meetings as an invited observer.
2. The 2016 Working Group developed a proposal that, in their view, would improve the quality of live, English-language captioning in Canada and submitted it as part of the record for this proceeding. The group proposed to develop a modified version of the NER ModelFootnote 3 to evaluate the accuracy of captioning.
3. Whereas the current model uses a verbatim measurement, the NER Model measures accuracy by evaluating whether the meaning or information was lost or received by the consumer, rather than by the number of words omitted, added or mistranslated in the captions. The NER Model allows for the use of paraphrasing in some instances and is based on the following formula:

$\frac{N – E – R}{N}=\mathrm{XX%}$

N: Total number of words, including punctuation.

E: 'Edition' errors, where a difference arises from a choice made by the captioner to paraphrase the verbatim speech.

R: 'Recognition' errors, where the captioner, or the software used by the captioner, delivers a caption word that is wrong, misspelled or garbled.

4. The 2016 Working Group acknowledged that, for this evaluation method to be consistent and effective, it will require expert, trained and unbiased evaluators who are able to categorize errors based on the severity of the loss of information to the viewer.
5. Specifically, the 2016 Working Group proposed the following:
1. The Commission should suspend the requirement set out in quality standard #3 for two years. During the two-year period the 95% verbatim accuracy rate for live programming would become an expectation.
2. Broadcasters would continue to monitor and report on their activities, as required by the current standards. This would include measuring captioning accuracy of ten-minute segments for two live programs per month from several program genres, using both the current method and the proposed NER Model. However, monitoring and reporting obligations would also become expectations.
3. Broadcasters would ensure the certification of all persons asked to perform the NER monitoring function.
4. Broadcasters would commit to a two-year trial of the NER Model for measuring captioning accuracy for English-language live programming.
5. The 2016 Working Group would continue to meet quarterly during the two-year trial with the first meeting taking place no later than 30 days following the Commission's decision.
6. Broadcasters would regularly publish the results (every six months) of the trial so that others may assess the results and provide feedback.
7. Broadcasters would develop a public awareness strategy that includes creating and disseminating materials to inform caption users of how captioning is performed, the various issues involved, and how captioners solve problems to help provide Canadians with assurances that their needs are being addressed, using video as appropriate.
8. The CCAA would undertake consumer research.

### Interventions

1. The Commission received 29 interventions in response to the call for comments. The public record for this proceeding can be found on the Commission's website at www.crtc.gc.ca.

1. The Commission must first consider whether the existing standard is measurable, effective and achievable, and if not, it must determine whether the proposal submitted by the 2016 Working Group represents an appropriate way forward.

#### Positions of parties

1. Several individuals submitted that closed captioning of high quality is very important to Canadians who are deaf or hard of hearing. One captioning user stated: "If I can make it through a full 30-minute newscast without thinking I missed a single point or purpose of any story, then that captioner has done their job correctly."
2. In the biennial reports as well as in their interventions for this proceeding, broadcasters submitted that they have been unable to consistently reach the accuracy rate set out in quality standard #3, despite best efforts and significant investments. In this respect, broadcasters submitted that considerable preparation goes into the captioning of any individual live program, including the provision of certain materials to captioners. For example, "station sheets," which inform the captioner about such items as the preferred placement of captions for each element of the program and the program structure, are provided to the captioner prior to the airing of the program.
3. Broadcasters submitted that the current standard is not effective and is not achievable. Further, no party argued that the current standard was effective or achievable.
4. The Public Interest Advocacy Centre (PIAC) stated that it generally supports the discussions of the 2016 Working Group and the proposed examination of the NER Model, but expressed an interest in obtaining more information as it becomes available.
5. Some captioning users and members of the closed captioning industry expressed their support for the proposal submitted by the 2016 Working Group.

#### Commission's analysis and decision

1. The Commission recognizes the importance of closed captioning to Canadians who are deaf or hard of hearing. For individuals who rely on captioning, poorly captioned programming is almost equivalent to uncaptioned programming: when captioning cannot be comprehended, the programming it accompanies becomes inaccessible. Recognition of the importance of closed captioning is what led to the adoption of quality standards. Based on the information provided in the biennial reports and in the interventions, the Commission is satisfied that broadcasters have made reasonable efforts to comply with quality standard #3 and considers that while the current standard is measurable, it is, in general, neither sufficiently effective nor achievable.
2. The 2016 Working Group submitted a proposal to the Commission to develop a modified method of calculating the accuracy of live English-language closed captioning which meets the needs that have been identified by the users of live English-language closed captioning. The Commission is satisfied that the CCAA, closed captioning providers and NAIT are equal partners in the 2016 Working Group and have been fully engaged.
3. The Commission considers that the 2016 Working Group has submitted a detailed explanation of the accuracy model that will be studied, trialed and modified for use in the Canadian broadcasting context, as well as detailed timelines and plans to release information regarding progress to the public.
4. In light of the above, the Commission considers that the proposal by the 2016 Working Group to conduct a trial to assess whether an accuracy measurement method based on a modified version of the NER Model represents an appropriate way forward. The Commission addresses the following issues regarding the implementation of the trial:
• Suspension of certain requirements during the trial
• Modified version of the NER Model
• 2016 Working Group meetings
• Public awareness strategy
• Progress reports and the 2016 Working Group final submission
• French-language closed captioning
##### Suspension of certain requirements during the trial
1. As part of its proposal, the 2016 Working Group called for the temporary suspension of the requirement that broadcasters adhere to the accuracy rate for live programming set out in quality standard #3 as well as the reporting and monitoring requirements set out in quality standard #4. It proposed that these requirements be converted into expectations during the two-year trial period.
###### Positions of parties
1. All 2016 Working Group members, including captioning users as represented by the CCAA, supported the suspension.
2. In its intervention, PIAC opposed the suspension of the accuracy rate requirement, but argued that, if it were to be suspended, the 2016 Working Group should propose an alternative standard for the duration of any trial to ensure that Canadians continue to be served by captions which can be measured to an existing standard.
###### Commission's analysis and decision
1. Suspending the requirement set out in quality standard #3 will allow the 2016 Working Group to direct its efforts to finding a permanent solution that is a viable alternative that serves the needs of captioning users rather than working out a temporary one. Furthermore, during the suspension, all programming will continue to be captioned and the methods used to provide captioning will not change.
2. The Commission does not consider it necessary to convert the monitoring and reporting requirements set out in quality standard #4 into expectations while the 2016 Working Group conducts the trial. By replacing the requirement in quality standard #3 with an expectation and maintaining the requirements set out in the other quality standards, including the monitoring and reporting requirement set out in quality standard #4, the 2016 Working Group will nonetheless be able to move forward with its proposal in a manner that is consistent with the policy objective set out in section 3(1)(p) of the Act.
3. Accordingly, the Commission temporarily suspends the requirement set out in quality standard #3 until 31 August 2019. During the suspension period, the Commission still expects all English-language broadcasters to continue to work towards achieving an accuracy rate of at least 95% averaged over a given live program, as measured by the calculation method set out in quality standard #3.
4. The quality standards, including an amended quality standard #3,Footnote 4 are set out in the appendix to this regulatory policy and replace the mandatory standards originally set out in Broadcasting Regulatory Policy 2012-362.
5. The Commission also expects all English-language broadcasters to maintain the same level of resources and support for the closed captioning of live programming that was provided prior to the suspension.
6. In setting a timeframe for the suspension period, the Commission took into consideration the complexity of the task. It is the Commission's view that the 2016 Working Group will need the requested two years to research, evaluate and modify the NER Model to reflect the Canadian context. As well, the timeframe for the suspension takes into account the additional time the Commission will need to initiate a public consultation on whether a new standard would be appropriate in light of the 2016 Working Group's findings.
##### Modified version of the NER model
1. The 2016 Working Group proposed to use a modified version of the NER Model to measure the accuracy of captions. The 2016 Working Group participants recognized that the NER Model in its current form may not be appropriate for the Canadian broadcasting system and proposed to develop a Canada-specific version of the model that would take into account at least the following:
• For English-language programming, live captioning is generally done by trained stenographers. This is in contrast to the voice-recognition technology used by other jurisdictions that have adopted the NER Model.
• There is a greater degree of subjectivity in the calculation used in the NER Model in contrast to the current verbatim model, given that in the NER Model, captioners are permitted to paraphrase in some circumstances and evaluators are required to categorize different types of errors and judge the effect the errors would have on a viewer's experience. The 2016 Working Group recognizes that using the NER Model will require careful collaboration from all interested groups to ensure that evaluation of accuracy truly reflects the Canadian viewer experience.
2. The 2016 Working Group acknowledged that using the NER Model system requires expert, trained and unbiased evaluators if it is to be consistent and effective. The broadcasters committed to ensuring the certification of all persons asked to perform the monitoring function.
3. The 2016 Working Group's proposal also committed to having broadcasters measure both NER and verbatim accuracy for ten-minute segments of two programs for several program genres.
###### Positions of parties
1. NCC and NAIT argued that word-per-minute calculations should be included in any new formula calculating captioning quality. NCC and NAIT submitted that stenographers are able to caption up to 280 words per minute verbatim, which is a higher word-per-minute rate than is generally achievable using voice-recognition technology.
2. SOVO Technologies, a French-language captioning company, was of the view that the NER Model may not be "discriminative enough" in its current form, making it very difficult to distinguish between great, good and bad captions.
3. Multiple interveners submitted that, generally, while verbatim captioning should always be the ultimate goal, a degree of paraphrasing that accurately conveys the meaning of the on-screen content could be appropriate.
###### Commission's analysis and decision
1. The Commission considers that the 2016 Working Group has the necessary participation, including that of NCC and NAIT, and understanding of the issues to ensure that a modified version of the NER Model will reflect the nature of English-language live closed captioning in Canada.
2. After the trial and once the 2016 Working Group has submitted a proposed model, the Commission will hold a public consultation. Interested persons will therefore have an opportunity to evaluate and comment on any new model to measure captioning accuracy.
3. In addition to the existing monitoring and reporting requirements set out in quality standard #4, the Commission expects Bell Media Inc. (Bell), Corus Entertainment Inc. (Corus), Rogers Media Inc. (Rogers), and the CBCFootnote 5 to measure both NER and verbatim accuracy for ten-minute segments of two programs from several program genres for the duration of the trial, which will provide essential data for the assessment.
4. The Commission expects Bell, Corus, Rogers and the CBC to ensure that all persons performing the NER measurements are properly trained and qualified.
##### 2016 Working Group meetings
1. The 2016 Working Group committed to meet quarterly over the two years of the trial with its first meeting occurring no later than 30 days following the publication of this regulatory policy. In light of the current Commission proceedings in which many parties may be involved, The Commission expects Bell, Corus, Rogers and the CBC to ensure that the 2016 Working Group meets quarterly during the two-year trial with its first meeting taking place no later than 31 January 2017.
2. The Commission requires Bell, Corus, Rogers and the CBC to continue participating in the 2016 Working Group, at least to the same extent and in the same manner as they have been throughout the proceeding leading to this regulatory policy. If the Commission becomes aware that any of these broadcasters has ceased collaborating with the 2016 Working Group as required, it will consider appropriate next steps, which could include revisiting the suspension of quality standard #3.
3. To ensure that the working group has the necessary resources during the trial period, the Commission expects Bell, Corus, Rogers and the CBC to continue supporting the 2016 Working Group in a manner and to a level similar to what they do currently.
4. The Commission encourages the CCAA, captioning providers and NAIT to continue to actively participate in the 2016 Working Group. It also encourages the CCAA to fulfill its commitment to conduct consumer research that will help the 2016 Working Group get a better understanding of captioning users' needs.
5. A Commission staff representative will continue to observe the work of the 2016 Working Group, on an as-needed basis.
##### Public awareness strategy
1. The 2016 Working Group's proposal included a commitment to create a public awareness strategy regarding how captioning is performed. The 2016 Working Group stated that the goal of this public awareness effort is to provide Canadians with assurances that their needs are being addressed.
2. The Commission therefore expects Bell, Corus, Rogers and the CBC to create a public awareness strategy using different types of media, such as video, where appropriate, and disseminate materials that will inform caption users of how captioning is performed, the various issues involved and how captioners solve problems.
3. The Commission expects Bell, Corus, Rogers and the CBC to consult CCAA and other relevant stakeholders (such as captioning providers and NAIT) during the development of communications materials.
##### Progress reports and 2016 Working Group final submission
1. The Commission expects Bell, Corus, Rogers and the CBC to make the results of the trial public every six months. At the end of the first year, in addition to reporting on the results, the report should include an accounting of the work that was achieved by the 2016 Working Group up to that point, as well as set out what it expects to accomplish in year two of the trial.
2. The Commission expects Bell, Corus, Rogers and the CBC to submit the final findings of the 2016 Working Group, including a proposal agreed upon by all participants defining a revised, detailed and operational accuracy measurement model that meets the needs of users, for the Commission's consideration, by no later than 2 November 2018.
3. As noted above, following the receipt of this proposal, the Commission will hold a public consultation on the appropriateness of quality standard #3 in light of the 2016 Working Group's findings.
##### French-language closed captioning
1. The Centre québecois pour la déficience auditive, a French-language closed captioning user group, provided examples of issues encountered with French-language closed captioning in its intervention. The Commission has also received several complaints from closed captioning users with respect to closed captioning by French-language broadcasters. The Commission considers that the oral public hearing announced in Broadcasting Notice of Consultation 2016-225 to renew the broadcasting licences held by the large English- and French-language ownership groups is the appropriate forum to discuss issues related to French-language closed captioning and to determine whether any further actions are necessary.

### Conclusion

1. The Commission recognizes that the work achieved to date is the result of the engagement and equal participation of Bell, Corus, Rogers, CBC, CCAA, NCC, CCS and NAIT in the 2016 Working Group.
2. The Commission commends the members of the 2016 Working Group for their collaborative work to improve the experience for Canadians who rely on closed captioning to access Canadian television programming.

Secretary General

### Related documents

• Notice of hearing – Renewal of television licences held by large English- and French-language ownership groups, Broadcasting Notice of Consultation CRTC 2016-225, 15 June 2016
• Various television services and stations – Corporate reorganization (transfer of shares), Broadcasting Decision 2016-110, 23 March 2016
• Call for comments on the English-language closed captioning quality standard related to the accuracy rate for live programming, Broadcasting Notice of Consultation CRTC 2015-325, 22 July 2015
• Quality standards for English-language closed captioning, Broadcasting Regulatory Policy CRTC 2012-362, 5 July 2012

## Appendix to Broadcasting Regulatory Policy 2016-435

### English-language closed captioning mandatory quality standards

The mandatory standards below replace those set out in Quality Standards for English-language closed captioning, Broadcasting Regulatory Policy CRTC 2012-362, 5 July 2012.

#### 1) Lag time

For live programming, the lag time between the audio and the captions must not exceed six seconds, averaged over the program.

#### 2) Accuracy rate for pre-recorded programming

Captioning for pre-recorded programs must target an accuracy rate of 100%, including spelling.

#### 3) Accuracy rate for live programming

Until 31 August 2019, broadcasters are expected to reach an accuracy rate of at least 95% averaged over the program, as measured by the following calculation method:

Accuracy rate = (words – errors*) / Words X 100

Starting on 1 September 2019, broadcasters must reach an accuracy rate of at least 95% averaged over the program, as measured by the following calculation method or by another method approved by the Commission:

Accuracy rate = (words – errors*) / Words X 100

*For the purpose of this standard, an error includes word substitutions, word omissions, and word insertions.

#### 4) Monitoring of accuracy rate

Every month each broadcaster must calculate the accuracy rate for two programs containing live content.

Every two years, broadcasters must provide the Commission with a report describing their efforts made in-house and requests to caption providers in order to improve the accuracy rate.

#### 5) Rebroadcast of programs initially aired live

When a broadcaster rebroadcasts a program that was initially aired live, it must correct errors in the captioning if:

• in the case of Category 1 (News) or Category 3 (Reporting and Actualities) programs, the time between the end of the original broadcast and the rebroadcast is equal to at least two times the total duration of the program; and
• in the case of all other live programs, the program is rebroadcast more than 24 hours after the end of the original broadcast.

#### 6) On-screen information

Positioning: For both live and pre-recorded captioning, captions must be positioned to avoid covering action, visual elements or any information required to understand the message.

Conflict between captions and on-screen information: If, despite the broadcaster's efforts, it is impossible to present captions without obstructing other graphic elements on screen (e.g. sports scores, weather data, breaking news), captions take precedence.

#### 7) Speed

Captions must be verbatim representations of the audio, regardless of the age of the target audience.

Speech must only be edited as a last resort, when technical limitations or time and space restrictions will not accommodate all of the spoken words at an appropriate presentation rate.

For the purpose of this standard, "appropriate presentation rate" is defined as 120-130 words per minute for children's programming.

#### 8) Captioning format for Canadian pre-recorded programming

Pop-on captions are to be used for all new Canadian pre-recorded programming. Pre-recorded programs are those that have been delivered in their entirety—lacking only the closed captioning information—96 hours before they are to be broadcast.

#### 9) Closed captioning of emergency alerts

With the exception of emergency alerts issued by the National Alert Aggregation and Dissemination System, in the event of a community emergency, broadcasters are required to provide all relevant information in both vocal and written formats and must insert the captions in their programming as soon as possible, using the method described in the CAB Closed Captioning Manual.

## Footnotes

Footnote 1

Closed captioning is the on-screen textual representation of the audio component of a program. For live programming, it is usually presented as a banner and provides a text rendition of the on-screen dialog and may include text renditions of sound effects and non-speech information such as the identity of speakers and their manner of speaking.

Footnote 2

NAIT offers a captioning program.

Footnote 3

The NER Model was created by a team from the University of Roehampton in the United Kingdom and was designed to measure the accuracy of live captioning using the re-speaking method. The re-speaking method uses a middle person to repeat what is said on the program so that the voice recognition software can translate speech into text. In Canada, live English-language closed captioning is generally provided by highly trained stenographers.

Footnote 4

Only quality standard #3 has been amended, all other quality standards remain the same.

Footnote 5

The broadcasters listed above are the largest members of the EBG and are members of the 2016 Working Group. Shaw Media Inc. is not listed in light of the corporate reorganization and transfer of shares to Corus, which was approved by the Commission in Broadcasting Decision 2016-110.

Date modified: