The sound patterns of Whisper : an informal workshop on audio LLM response to speech stimuli

PROVISIONAL PROGRAMME

9h Nicolas Ballier & Guillaume Wisniewski (UPCité) : introduction

We briefly present current research in the making on speech language models.

Session 1 : speech variability and LLM response : some experiments and questions

9h 15 Tori (Georgina) Fullerton (UPCité) : How Whisper models respond to speech variability?

[This talk compares Whisper model transcriptions to human perception of VOT variation
and showcases research in the making on the detection of compact compounds in English.]

9h 40 discussion

10h Richard Wright (university of Washington), Nicolas Ballier & Philippe Martin (UPCité): resynthesing speech and analysing response to segmental and suprasegmental features
[This talk reports research in the making on  vowel and pitch synthesis and how Whisper models respond]

10h15 discussion

10h 30 Behnoosh Namdarzadeh (UPCité) Whisper transcriptions for lesser-resourced languages : the case of Persian

[This talk reports our findings for the transcription task of a language trained with 24 hours of speech]

10h40 coffee break

Session 2 : Probing Speech Language Models : first experiments and results

11h00 Nicolas Ballier and Jean-Baptiste Yunès (UpCité)  : a customised C++ implementation of Whisper to probe representations
[This talk presents our reverse engineering method based on the internal representations of the Whisper models.]

11h10 Hosein Mohebbi (Tilburg) : Adapting context mixing methods to explore speech transformers
[This talk presents the workflow used to analyse acoustic and linguistic representations in the EMNLP2023 paper.]

Mohebbi, H., Chrupała, G., Zuidema, W., & Alishahi, A. (2023). Homophone Disambiguation Reveals Patterns of Context Mixing in Speech Transformers. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (pp. 8249-8260).

11h40 discussion

Session 3: So many parameters, so little time

12h00 round table

[We discuss some of the LLM parameters and methods to investigate audio LLMs. How to systematically investigate hallucinations and effects of language models?]

Antonio Balvet (Lille) : Prompt engineering for spoken transcriptions of French
Peter Uhrig (Erlangen) : Checking accuracy and investigating speed optimisation with Whisper

Alternates : Guillaume Wisniewski and the Deeptypo team (tbc) : playing with Wav2vec and raw signal