Intellectual Property Protection of Speaker Recognition ML Models

Problem Statement

In recent years, research has increased on model protection techniques, especially in the image domain. On the other hand, the audio domain, specifically speaker recognition (SR) models and ways to protect them from being stolen, is still rather unexplored. Speaker recognition models are designed to identify and verify a speaker’s identity based on an audio sample. Their primary task lies in determining the identity of a speaker from a list of known speakers. Speaker recognition models are widely used in multiple domains, including biometrics, security, healthcare, customer service, and entertainment. Since these models become more sophisticated and prevalent, protecting them from unauthorised usage is crucial.

Audio processing for Machine Learning

The way audio is processed for model training is highly dependent on the type of audio signal and the task that this model aims to solve. Existing speaker recognition models vary in their architectures and input structure, whether it is a technique based on (i) image processing that works with spectrograms or (ii) raw waveforms of audio signals

The original audio signal is an analog signal that needs to be converted into a digital signal, so thatit can be processed by a computer. This process is called Analog-Digital-Conversion (ADC).

Before further processing the digital signal to retrieve meaningful features, it is usually divided into smaller overlapping segments, also called frames. This step is performed because the signal is volatile and usually changes over time due to variations in pitch, intensity, background noise, or speaker articulation. To preserve the continuity of the signal, the frames need to overlap, to avoid losing phonetic details and ensure a smooth transition between frames. Subsequently, by applying Short-Time Fourier Transform (STFT), one can obtain a spectrogram (an image representaion of an audio signal).

Speaker Recognition Models Back-box Watermarking

Black-box machine learning model watermarking relies on memorisation of certain input-output patterns (trigger set), so that at the verification stage, the ownership of the model can be proven just via an inference process. This approach is especially relevant for real-world scenarios, as it provides greater flexibility in verifying ownership when stolen models are made available, e.g., only behind an API. Only a few recent studies [1,2,3] discuss model protection techniques for speaker recognition models that were created specifically for the audio domain. However, these works are fragmented and do not follow a unified methodology, and are thus not easily comparable.

Wang et al. [1] embeded a watermark into all segments of random samples via discrete cosine transform (DCT). Zhang et al. [2] also used a transformation-based technique. However, they incorporate multiple steps: STFT, magnitude spectrum, conversion into Mel-spectrogram, and DCT, after which the key embedding is done and the intermediate result is converted back to audio. The most recent work by Liao et al. [3] explored several approaches: adding Gaussian noise considering the noise level of an audio signal, adding Gaussian noise only in extreme frequencies of spectrogram representation, which makes the watermark less audible, and adding unrelated audio on top of the original sample.

Experiment Design

In order to verify usability and applicability of the state-of-the-art watermarking methods, we reproduce and evaluate them according to the watermarking requirements: generality, effectiveness, fidelity, imperceptibility, robustness, unforgeability. To assess the generality, we watermark 3 models (SincNet, MobileNet, AM-MobileNet) trained on TIMIT dataset[4] and 3 models (ResNet18, ResNet34, AutoSpeech) trained on VoxCeleb1 dataset[5]. The TIMIT dataset contains clean speech, while the VoxCeleb1 dataset comprises audio of varying quality, often accompanied by background noise of different origins. Considering datasets with different characteristics, we investigate the effect they have on the watermarking scheme.

We evaluate imperceptibility using signal-to-noise (SNR) ratio and log spectral distortion (LSD). While SNR reveals the overall distortion of the signal, LSD reflects human auditory perception and measures spectral differences. For LSD, the values in range 0 and 1 dB indicate the most imperceptible watermark, but the range between 1 to 2 dB is also acceptable. Conversely, for SNR, the higher the SNR, the less perceptible the watermark is, with the threshold being at 20 dB.

The robustness of the watermarks is assessed based on the success of the executed attacks, which we evaluate using both the Watermark Success Rate (WMSR), which measures whether the model still responds to trigger samples and the Attack Cost, defined as the impact on the original task performance, measured by sentence error rate (SER). We thus define a successful attack as one that reduces WMSR to less than 95% while having no significant impact on SER, for which we consider a difference of less than 0.05. We consider both input preprocessing (e.g. noise addition, pitch shifting, bandpass filtering, etc.) and model modification attacks (e.g. fine-tuning and weight pruning).

Results

The 3 watermarking methods proposed by Liao et al. [3] demonstrate applicability across the evaluated models and datasets, achieving high watermark effectiveness and maintaining model fidelity.

Imperceptibility

In terms of imperceptibility, the Unrelated Audio watermark performs best across both datasets, while Frequency Noise produces the most perceptible audio samples. Only the SNR of the Unrelated Audio watermark approaches the acceptable perceptibility threshold of greater than 20 dB. Therefore, we present investigated the trigger creation parameters of the watermarking methods aiming to improve the imperceptibility of the resulting trigger sets.

Robustness

Data preprocessing attacks have mixed effects on watermarking methods. Some increase SER too much (>0.05), e.g., all configurations of pitch shifting (PS), most configurations of maintaining a bandpass (B), and time stretching (TS). The latter two, as well as three Gaussian noise attacks (SNR, WN, PN) and compression (C), in fact do not succeed in removing the watermark without degrading model fidelity in any configuration, whereas keeping low (LT) and high frequencies (HT) succeed in some cases. HT with the cut-off at 40 Hz is effective against UAM applied on ResNets and AutoSpeech for the unrelated audio sample with a noise variance of 0.0000578. The same attack on VoxCeleb1 triggers created with the unrelated audio sample with a noise variance of 0.00043166 does not affect the watermarked part at all up to 150 Hz, where WMSR becomes 99.6%. Our experiments revealed that the choice of unrelated audio can significantly impact both the imperceptibility of the watermark and its robustness to perturbations.

The only successful fine-tuning attack applied to a model trained on the TIMIT dataset was the one conducted on the AM-MobileNet model watermarked with Gaussian Noise. Moreover, since in general the SER of AM-MobileNet models worsened during fine-tuning, the only point when the attack was successful was the first epochs of fine-tuning, while SER stayed within the defined bounds of 0.05 cost. For the models trained on the VoxCeleb1 dataset, we observe a different pattern of the fine-tuning attack. Since the SER on the subset of the test set improved during fine-tuning, which can be attributed to a three times higher number of classes and a much higher number of audio samples in the dataset compared to TIMIT, the attack showed successful results more often.

In Table ✗ denotes a successful attack configuration and ✓ a robust watermark under attack configuration. We define a successful attack as one that reduces WMSR to less than 95% while having no significant impact on SER, for which we consider a difference of less than 0.05.

Conclusion

The six considered models were successfully watermarked using the proposed watermarking methods that we managed to reproduce. Model fidelity varied across watermarking techniques, with the Unrelated Audio method performing the worst in this regard. With respect to imperceptibility requirement, the most notable difference lies in the level of distinction required in a watermarking pattern embedded in the trigger samples. The models trained on TIMIT are able to detect even subtle discrepancies without any particular loss in watermark effectiveness, although a gradual loss in fidelity can be observed. In contrast, the models trained on VoxCeleb1 require a watermark to be more pronounced for the model to reliably recognise a trigger sample. The results of robustness evaluation revealed that the imperceptibility of a watermarking pattern significantly affects its robustness. We see the trade-off in the watermark characteristics and the need to enhance the watermarking approaches for speaker recognition machine learning models protection.

References

[1] Yumin Wang and Hanzhou Wu. Protecting the Intellectual Property of Speaker Recognition Model by Black-Box Watermarking in the Frequency Domain. Symmetry, 14(3):619, March 2022.

[2] Jing Zhang, Long Dai, Liaoran Xu, Jixin Ma, and Xiaoyi Zhou. Black-Box Watermarking and Blockchain for IP Protection of Voiceprint Recognition Model. Electronics, 12(17):3697, January 2023.

[3] Junpei Liao, Liang Yi, Wenxin Shi, Wenyuan Yang, Yanmei Fang, and Xin Yang. Imperceptible backdoor watermarks for speech recognition model copyright protection. Visual Intelligence, 2(1):23, July 2024.

[4] I.J. Cox, J. Kilian, F.T. Leighton, and T. Shamoon. Secure spread spectrum watermarking for multimedia. IEEE Transactions on Image Processing, 6(12):1673–1687, December 1997.

[5] Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. VoxCeleb: a large-scale speaker identification dataset. In Interspeech 2017, pages 2616–2620, August 2017.