The Current Regulatory Framework for AI-driven Voice Analytics

More and more software uses machine learning and artificial intelligence (AI) to identify and measure emotional and mood elements, find keywords in audio files, and recognise expression-based sentiments and speech characteristics, such as speech rate, intonation and articulation. AI also uses speech-signal processing to detect waiting time, silence and talking in the sound file.

These solutions help companies to reach various goals, such as quality control and improving the standard of the company’s services, reducing the number of customer complaints, increasing customer satisfaction, customer retention and customer education. AI voice analytics may also enable more transparent and accurate employee performance appraisal and development.

The regulatory framework for the use of these solutions is still forming. The first draft of the EU’s Artificial Intelligence Regulation contained a specific definition for “‘emotion recognition systems” (“an AI system for the purpose of identifying or inferring emotions or intentions of natural persons on the basis of their biometric data”), but sets out only an information obligation for the users of such systems (“[u]sers of an emotion recognition system shall inform of the operation of the system the natural persons exposed thereto”). However, if the AI-driven voice analytics solution is intended to be used for making decisions on, for example, promotion of employees, or for monitoring and evaluating their performance, it is considered a "high-risk AI system" and falls under strict risk management obligations.

The EU’s Artificial Intelligence Regulation is still a draft. If companies want to verify the compliance aspects of such AI systems in more detail (particularly how to fulfil the principles of lawfulness, fairness and transparency), they must rely on the GDPR.

GDPR compliance of the voice analytics AI systems is of particular importance because Hungary’s data protection authority, Nemzeti Adatvédelmi és Információszabadság Hatóság (NAIH), recently levied a fine of HUF250 million against a bank for the shortcomings of its automatic AI analysis of recordings of customer service calls, which included assessing the emotional state of the speaker and other characteristics. This is the first time in the EU that a regulator has investigated an AI-driven voice analytics solution. Therefore, the takeaways would be useful for all companies who intend to develop and/or use similar solutions.

Facts of the Hungarian Case

Background

In the underlying case, the bank used the results to monitor the quality of calls, prevent complaints, rate the quality of work and increase the efficiency of its call-handling staff. Specifically, the AI-based speech-signal processing technology automatically analysed a list of keywords and the emotional state of the speaker. The results of the detected keywords and emotions were also stored along with the call, and the calls could be replayed within the voice-analytics software for up to 45 days. The software ranked the calls and provided recommendations on the priority of callers to be contacted. Call-backs were performed by random selection while using the system, but calls were ranked based on the characteristics established by the software.

The aim of the software was to make quality control more professional, and to make the work of nearly 180 call centre employees more efficient by improving the call-selection process. In addition, the voice-analytics software was designed to help the individual development of the bank's employees (notably, their professionalism and communication skills), to improve the efficiency of processes and to increase the quality of customer experience.

The NAIH’s findings

NAIH found the following shortcomings in this particular solution.

• The bank’s customer service privacy notice did not contain any substantive information on voice analysis. The privacy notice only mentioned quality assurance and complaint prevention as data processing purposes.
• Customers did not receive any information at the beginning of the telephone conversation about the voice analytics and the evaluation of their emotions, as well as the possibility of a call-back on this basis.
• The bank based data processing on its legitimate interest in retaining customers and in improving the efficiency of its internal operations. However, the different data processing operations related to these interests were not separated, either in the privacy notice or in the balancing of interest test (LIA).
• The bank's data protection impact assessment (DPIA) concluded that this processing is high risk for a number of reasons. However, the DPIA did not provide substantive solutions to address these risks.
• The bank did not actually examine the proportionality of the data processing and its effects on data subjects, and trivialised the significant risks to fundamental rights. It expressly failed to take into account the right of data subjects to adequate information and their right to object.

Takeaways for Other Companies

It is important to note that the NAIH did not rule the AI analysis of recorded customer service calls generally unlawful, only the way in which the relevant bank implemented it.

If a company provides more detailed information about the use of AI-driven voice analytics in the related privacy notice and LIA and properly documents such use, the privacy risks raised by the use of the AI system in the form of a data protection impact assessment would result in such a high risk that the underlying data processing would be nearly impossible. The NAIH decision explicitly identifies as a risk that AI “is generally difficult to understand and monitor due to its operating principles, so that new technologies present significant data protection risks". Therefore, companies must be extremely cautious during their assessment of the underlying risks and should consider the following potential measures to reduce their exposure.

Consultation with employees and customers

Consultation with the relevant stakeholders is important to address the challenges and concerns raised by the introduction of an AI system.

According to Article 35 (9) GDPR, the data protection impact assessment "shall, where appropriate, without prejudice to the protection of commercial or public interests or the security of data processing operations, seek the views of data subjects or their representatives on the intended processing". In practice, this is another rarely used provision of the GDPR, but given that the trust of both customers and employees is key to avoid possible regulatory investigations, companies using voice analytics may want to consider the extent to which consultation with employees is possible, and the extent to which it is possible to conduct a survey among customers to record their opinions on the proposed voice-analytics solution. A pilot period or using a focus group for voice analytics may be a good opportunity for this. We note that, in the absence of "sandbox" regulation, the company must also comply with the applicable laws in such a trial period.

Addressing the concerns surrounding “black box”

Similar to other regulators, such as the UK’s Information Commissioner's Office (ICO), the European Data Protection Board emphasised that AI systems are becoming more and more complex, which makes them less transparent and the inner workings of the algorithms more difficult to understand or explain (ie, “black box”). (The ICO defines a "black box" model as any AI system whose inner workings and rationale are opaque or inaccessible to human understanding.) This can lead to a lack of transparency towards the individuals whose data is processed by the algorithm and a loss of human autonomy for those working with algorithms.

Some findings in the resolution of the NAIH also suggest that the problem of “black box” may also affect voice-analytics solutions. For example, the NAIH states that the automated analysis in the underlying case considers speech speed, volume, pitch, and length of speech pauses. However, the detailed considerations are the trade secret of the developer.

The NAIH also notes that the calls were ranked based on the characteristics established by the software, but the bank does not know these features either. This missing information may cause trust issues on the side of the customer and employees. Therefore, companies developing or using AI-driven voice analytics should formulate their privacy notices and other communications in connection with the solution in such a way that individuals should be able to easily understand how the AI system is being used in their particular case.

Practical examples on the use of AI-driven voice analytics in the privacy notice may enhance transparency. Employees may want to see concrete examples on individual and group expectations determined on the basis of voice analytics, and how quality-assurance aspects are measured in practice. It would be preferable to show how performance appraisal relies on the results of voice analytics, and to identify the objective criteria that will make employee appraisal more predictable, comparable and comprehensive as the result of the use of voice analytics. It should also be shown how, in practice, the results of using voice analytics can improve the performance of each employee, the quality of the company’s service, and the comparisons and feedback based on AI-generated data.

Information on automated decision-making and profiling made on the basis of AI is essential, due to the potential lack of trust and information on the side of the individual and the nature of the new technology. Individuals must be informed on the type of AI, its self-learning abilities, the type of teaching databases used (eg, sample calls) and its preliminarily defined parameters.

The NAIH decision also provides for the necessity to identify the legal and ethical issues raised by profiling. The ethical principles for the development, deployment and use of AI, robotics and related technologies are of utmost importance – unfortunately, the NAIH does not provide any suggestions as to what ethical questions should be identified. Based on the existing ethical guidelines pertaining to the use of AI, the most important ethical prerequisite is that the AI system must be clearly free of any sort of profiling, bias and discrimination.

Information on the data used by AI

High data quality is essential for the performance of many AI systems, especially when techniques involving the training of models are used, with a view to ensure that the AI system performs as intended, operates safely and does not become the source of discrimination. The GDPR also requires companies to properly record the scope of the personal data they are processing.

In case of voice analytics solutions, the recorded audio material contains the voice of the customer and the operator in addition to a unique call identifier for the customer’s system, the calling or called telephone number, the direction of the call, the date, the name of the operator, and the operator's organisational unit.

However, AI also generates additional personal data as the result of voice analytics (eg, description and analysis of the call, evaluation results, expectations), which must be listed in the underlying privacy notice as well. The requirement stems from Article 14 (1) (d) of the GDPR – if personal data has not been obtained from the data subject, the data controller must provide the data subject with information on the categories of the personal data concerned. It is also recommended to explain in more detail what personal characteristics of the customer and employee concerned can be inferred from the data collected (eg, number of words or waiting time, speaking ratio between client and employee) and to what extent it is relevant to the purpose of the voice analytics.

Beyond the data-protection information, the concept of “explainable AI” (as emphasised by the UK’s ICO) may require providing information on the data collected and pre-processed for the purpose of training the AI, such as the source of the training data, how it was collected, assessments about its quality, and steps taken to address quality issues (eg, completing, augmenting, or removing data). Developers of voice analytics AI systems should also check the data used within their model to ensure it is sufficiently representative of those customers who companies are making decisions about, and that the modelling, testing and monitoring stages of the use of the AI system leads to accurate results.

Avoiding the use of health data within the AI system

The NAIH decision suggests that the data related to the voice of the customer, as well as certain data that form the result of the voice analysis (emotion, mental state, etc) may be considered health data. The processing of health data under Article 9 of the GDPR (Processing of Special Categories of Personal Data) requires the fulfilment of one of the additional conditions set out in Article 9 (2) of the GDPR. As this list does not contain any conditions that would govern the use of AI-driven voice analytics, companies should implement measures to avoid processing health data. For example, if the customer has a speech impediment and this is relevant to the evaluation of the conversation, the company should merely record that the customer’s speech was incomprehensible, but should not record the exact reasons.

Legitimate interest regarding the use of voice analytics

The NAIH hinted that only informed consent, freely and actively given, could be the basis for data processing operations in connection with voice analytics. However, this is only a hint – the legal basis of the underlying data processing must be examined on a case-by-case basis. If the company using voice analytics ensures that the rights of the individuals are adequately protected – most importantly, that they have advance knowledge of data processing and are able to object to it before the actual data processing operation – the company’s legitimate interest can also be an alternative legal basis.

Therefore, it is important to examine what technical settings can be made to ensure the rights of customers, particularly their right to adequate information and to have their objections addressed. For example, after receiving brief but substantial information about the use of voice analytics before speaking to the telephone operator, customers may be reminded of the availability of a detailed privacy notice. (If the operator reads the detailed privacy notice at the beginning of the call, this will extend the time for submitting the complaint or customer request by at least 10–15 minutes.) After that, customers may indicate to the operator if they do not want the use of the voice analytics in connection with the call.

Customers should also be informed if they have been called back based on data obtained through the use of AI. In this case, the operator will record the call and, based on this, the company will know that it cannot perform analysis on that particular call. Operators should be provided with appropriate, regular (eg, half-yearly) and documented (training material, attendance sheets, possibly exams, etc) training to answer customers' questions in relation to the operation of AI and the protection of their personal data.

This approach does not require any further action on the part of the customer as opposed to obtaining the customer's consent. In that case, the customer must give consent by means of an “action” under Article 4 (11) of the GDPR, which, in the event of a telephone call being recorded, must be accompanied by an additional option that allows the customer to express their consent. However, this can lead to deterioration in the customer experience and customer turnover.

Alternatives to AI voice analytics

The use of AI systems must be proportionate and necessary to achieve the underlying objectives.

Therefore, it is important to document the alternatives to using AI-driven voice analytics and their disadvantages. Companies using voice analytics must be able to demonstrate that these alternatives have actually been considered. The NAIH decision expressly states that exclusion from the use of telephone customer service cannot be a real option for individuals. Customers should therefore be able to continue to operate by telephone without using AI voice analytics.

In practice, the only alternative to using AI voice analytics is random inspection of calls. The disadvantages of this, both for the company and for its employees and customers, need to be quantified with concrete figures. A statistical indicator of the effectiveness of random checks is a good starting point here, since practical experience in call centres suggests that only 1–2% of calls can be verified without automation.

The company’s legitimate interest to use the AI-driven voice-analytics solution is supported by the fact that it is in the interest of both customers and employees that the company can analyse and draw conclusions from a larger dataset, but it is important to explain and emphasise this.

Weighing the benefits and the disadvantages of AI voice analytics

The benefits of AI voice analytics need to be weighed against the potential risks and disadvantages. The NAIH decision provided only one example in this respect, and found that the bank concerned could perform its duties by employing fewer customer service staff was "not in itself a proportionate and adequate reason to deprive those concerned of their fundamental rights". Given the NAIH's above approach, it would be essential to support the proportionality of the underlying data processing with several practical examples.

In relation to employees, companies should record how employees would be evaluated without AI voice analytics, and which elements of the evaluation will be improved by the new technology. For example, companies may emphasise how AI voice analytics can further improve an objective, transparent, comprehensive and well-supported employee assessment.

An important aspect to consider is that the NAIH decision explicitly emphasised that "the effectiveness of the system based on the recognition of emotions in the underlying case is highly questionable, as there was no recognisable emotion in 91.96% of cases". Therefore, the company should support the effectiveness of the emotion detection and analysis function with, among other things, appropriate reasoning and figures. According to the NAIH, this "reinforces that the sound analysis carried out in that form is suitable for achieving the stated objectives and that its use would be an inevitable and proportionate restriction of the rights of those concerned".

For example, companies should use specific statistics to support why a call can be verified more accurately, quickly, and efficiently using a transcript with AI analysis than it does when listening to an audio recording. Regarding employees, it is recommended to record the steps and interventions that can be taken in practice after measuring the productivity and assessing the emotional indicators of the employee. It is also recommended to provide concrete examples of how emotional indicators help to achieve the company’s goals. In regard to the specific figure (91.96%) cited by the NAIH, one may interpret the percentage more positively – the voice analytics solution can already provide a significant benefit if it can identify 10% of those angry or desperate customers who need immediate assistance.

Companies using voice analytics should also consider that employees are in a vulnerable position due to the employer-employee relationship, and that the company is in a position of trust and significant information dominance vis-à-vis the customers as well. According to the NAIH, the company must provide internal guarantees to handle all inequalities. These “guarantees” are mentioned in several places in the NAIH decision, but the NAIH does not make any specific proposals, only stating that "pseudonymisation is a useful but not a sufficient guarantee in itself".

Therefore, the company should provide unique identification of these data protection guarantees. In the context of the EU’s draft Artificial Intelligence Regulation, such guarantees may comprise data governance, documentation and recording-keeping, transparency and provision of information to users, human oversight, robustness, accuracy and security.

Avoiding errors and biases

Companies must carefully assess how individuals can request the correction of their AI-processed data and if there are any limitations. For example, can the affected customer or employee indicate that their speech rate is fast because they are nervous (and are speaking faster than average), or can the customer indicate that the waiting or silence period was long for unforeseen reasons (eg, the customer had was distracted by a visitor)? The obligations for ex ante testing, risk management and human oversight will also facilitate the respect of other fundamental rights by minimising the risk of erroneous or biased AI-assisted decisions.

Periodic review of risk assessments

Companies should review their risk assessments (including the LIA) periodically (eg, every six months), setting out the main aspects of this review, including in particular the number of objecting customers and employees, and the nature of any questions, comments or complaints regarding the processing of personal data. The review should include an examination of whether the actual operation of AI meets initial expectations and adequately protects the rights of those affected.

Closing Thoughts

Overall, the NAIH decision is a useful guide for organisations wishing to use voice analytics. The above ideas are intended as inspiration in case a company is considering implementing such solutions. However, it is also important to keep in mind that AI is still in its infancy and its use can be both a competitive advantage but also a regulatory risk. In any case, the EU AI Regulation is needed to create a predictable legal environment – we are looking forward to the final wording of the new framework, and the experience in connection with its practical application.