SANE 2026 - Speech and Audio in the Northeast

October 30, 2026

Aerial view of the East Campus of the Massachusetts Institute of Technology (MIT) along the Charles River, 2015, Nick Allen

SANE 2026, a one-day event gathering researchers and students in speech and audio from the Northeast of the American continent, will be held on Friday October 30, 2026 at MIT, in Cambridge, MA.

It is the 13th edition in the SANE series of workshops, which started in 2012 and is held every year alternately in Boston and New York. Since the first edition, the audience has steadily grown to about 200 participants and 50 posters each year, and SANE has established itself as a vibrant, must-attend event for the speech and audio community across the northeast and beyond.

SANE 2026 will feature invited talks by leading researchers from the Northeast as well as from the wider community. It will also feature a lively poster session, open to both students and researchers.

This year, SANE will take place right after the AIR and DCASE workshops, held October 27-29 in Cambridge as well.

Details

Date: Friday, October 30, 2026
Venue: Thomas Tull Concert Hall, W18 Building, MIT
Time: 8:20am - 6:15pm

Schedule

8:20-8:50	Breakfast
8:50-9:00	Welcome and State of the SANE
9:00-9:50	Sreyan Ghosh (Google) Towards Fully Open General Audio Intelligence
9:50-10:40	Berrak Sisman (Johns Hopkins University) Recent Advances in Expressive, Secure, and Health-Centric Speech Intelligence
10:40-12:20	Coffee break and Poster Session 1
12:20-13:05	Lunch
13:05-13:55	Henry Li (Google) Unsupervised Sound Separation Beyond Waveform Regression
13:55-14:00	Group Photo
14:00-14:50	George A. Saon (IBM Research) On the development of Granite Speech, IBM’s open-source, speech-aware LLMs
14:50-16:30	Coffee break and Poster Session 2
16:30-17:20	Bryan Pardo (Northwestern University) Creative Control for Artists in the Age of AI
17:20-18:10	Ruohan Gao (University of Maryland, College Park) Building Worlds of Sight and Sound
18:10-18:15	Closing remarks
18:15-.........	Drinks somewhere nearby

Registration

Registration is free but required. We are currently full but are accepting registrations on the waiting list. Please register as soon as possible by sending an email to with your name and affiliation.

Directions

The workshop will be hosted in the Thomas Tull Concert Hall at MIT, located in the new Edward & Joyce Linde Music Building (W18). The address is 201 Amherst St, Cambridge, MA 02142.
Public transportation: The nearest T station is the Kendall/MIT Red Line Station (~15 minutes walk), and the nearest bus stop is the 77 Massachusetts Ave stop on the 1 line.
Parking: Parking is available in the garage located under the Edward and Joyce Linde Music Building (W18) at a cost of $11 for the whole day.

Organizing Committee

Talks

Towards Fully Open General Audio Intelligence

Sreyan Ghosh

Google

What does it take for AI to understand the world through audio? In this talk, I will present our work on fully open large audio-language models, highlighting advances in architectures, training curricula, and internet-scale data curation. I will explore how unified encoders connect speech, sound, and music, and how new data and training infrastructure extend audio understanding from short clips to 30-minute recordings. Along the way, expert-level benchmarks reveal persistent gaps between current models and human audio reasoning. Finally, I will show how audio enriches real-world video comprehension, and discuss our work integrating audio into world models. Together, these directions highlight why listening is essential to AI systems that understand and model the world.

Sreyan Ghosh is a Research Scientist at Google DeepMind, where he works on audio and audio-visual intelligence for Gemini. His research focuses on speech-to-speech modeling, speech translation, large audio-language models, audio representation learning, and long-form audio understanding and reasoning. He completed his Ph.D. in Computer Science at the University of Maryland, College Park, advised by Dinesh Manocha and Ramani Duraiswami. During his Ph.D., he co-led the development of the Audio Flamingo family of models and the MMAU series of audio understanding and reasoning benchmarks. Previously, he held research internships at Nvidia, Adobe and Microsoft. He is a recipient of the NVIDIA Graduate Fellowship and UMD’s Outstanding Graduate Assistant Award.

Recent Advances in Expressive, Secure, and Health-Centric Speech Intelligence

Berrak Sisman

Johns Hopkins University

In this talk, Berrak Sisman presents recent research from her lab at the Johns Hopkins University Center for Language and Speech Processing (CLSP) on human-centered speech intelligence. The talk highlights their recent progress in controllable speech synthesis and voice conversion, robust speech deepfake detection, mechanistic interpretability of large audio-language models, and clinical applications for speech and language technologies, such as Alzheimer's and depression detection.

Berrak Sisman is an Assistant Professor of Electrical and Computer Engineering at Johns Hopkins University (JHU), where she leads the Smile Lab as part of the Center for Language and Speech Processing (CLSP). She received her Ph.D. from the National University of Singapore. Her research centers on human-centered speech and audio intelligence, focusing on expressive speech synthesis, voice conversion, speech deepfake detection, and the security, privacy, and health information encoded in the human voice. Her research has received multiple recognitions, including the 2026 Speech Communication Best Journal Paper Award, an NSF CAREER Award (2024), Amazon Faculty Awards (2023, 2025), and the Singapore Ministry of Education Tier 2 Award (2023). She is an Associate Editor for IEEE Transactions on Affective Computing, and an elected member of the IEEE Speech and Language Technical Committee (SLTC). She is the Program Co-chair of Interspeech 2026, and she serves on the ISCA Board.

Unsupervised Sound Separation Beyond Waveform Regression

Henry Li

Google

Single-channel sound separation typically requires clean, isolated training sources, a luxury unavailable in the wild. While unsupervised methods circumvent this constraint, prior frameworks rely on deterministic waveform regression, leaving them blind to source semantics and prone to regression-to-the-mean artifacts. This talk explores two complementary paradigms that advance unsupervised separation beyond signal-level regression: structured latent representations and continuous generative modeling. First, we show how generalizing mixture-invariant training to semantic embedding spaces (such as CLAP) decomposes mixtures into source-specific latent slots without text supervision, enabling prompt-free separation and fine-grained audio-text retrieval. Second, we derive a self-supervised flow matching framework through the lens of the classic Wake-Sleep algorithm by bootstrapping a generative student flow on teacher estimates. This approach bypasses unconditional prior training while consistently outperforming regression baselines across speech and universal sound benchmarks. Together, these paradigms demonstrate how rethinking representation spaces and generative dynamics enables high-fidelity, prompt-free separation directly from in-the-wild audio. We conclude by outlining failure modes and open challenges: permutation ambiguities along generative trajectories, acoustic mismatches from naive remixing, and the fundamental limits of independent-source priors.

Henry Li is a Research Scientist at Google focusing on generative modeling and acoustic scene decomposition. His work combines theoretical frameworks in inverse problems and flow matching with empirical algorithms to isolate, enhance, and semantically categorize sounds from unlabeled acoustic mixtures. While specialized in audio, his research foundation encompasses generative diffusion models more broadly, including theoretical work on volume-preserving maps, diffusion optimal control, and multimodal generation. His research has been presented at machine learning and vision venues such as NeurIPS, ICLR, ICML, and CVPR. He earned his Ph.D. in Applied Mathematics and a B.S. in Mathematics and Computer Science from Yale University. When not untangling audio mixtures, he enjoys running, drumming, and hiking mountains of decidedly moderate height.

On the development of Granite Speech, IBM’s open-source, speech-aware LLMs

George A. Saon

IBM Research

Granite Speech (https://huggingface.co/collections/ibm-granite/granite-speech) is a collection of open-source, highly accurate speech-aware LLMs specialized in multilingual ASR and bidirectional speech translation for major European languages and Japanese that can be used under a permissive license. They pair Conformer acoustic encoders trained with Connectionist Temporal Classification on publicly available multilingual ASR corpora, Q-former speech modality adapters and Granite text LLMs from 1B to 8B parameters. The role of the adapter is to perform temporal downsampling and to map the acoustic embeddings from the encoder to the embedding space of the LLM. The modality adapter and LLM LoRA parameters are trained jointly with next-token prediction CE loss on several tasks such as ASR, speech translation, ASR with word-level time stamps, speaker-attributed ASR and keyword-biased ASR by varying the text prompt and target transcripts.
The talk will dive deeply into several aspects related to the model architecture, training, and performance of Granite Speech. On the encoder side, we will discuss several optimizations such as block attention, self-conditioning and temporal subsampling within Conformer blocks as well as the use of a novel Muon optimizer during training which are exemplified in our newly released Granite 5.0 TurboCTC encoder-only model (https://huggingface.co/ibm-granite/granite-speech-5.0-470m-turboctc). On the speech LLM side, we will discuss balanced sampling, synthetic data generation, speculative decoding and a novel non-autoregressive architecture that edits the CTC hypothesis in a single forward pass through the LLM. Throughout the talk, we will report the performance of our models (accuracy and inverse real-time factor) on the Hugging Face Open ASR leaderboard as well as the Treble Far-Field ASR (FFASR) leaderboard that measures system performance in challenging acoustic conditions such as noise, reverberation and distant microphone speech.

George A. Saon is Manager of the AI Foundations Speech Technologies group at IBM Research's Thomas J. Watson Research Center in Yorktown Heights, NY, and has been a Distinguished Research Scientist at IBM since 2018. He joined IBM Research in 1998 and has worked on feature extraction, acoustic modeling, speaker adaptation, hypothesis search, and end-to-end sequence transduction for large vocabulary continuous speech recognition. He holds a Ph.D. in Computer Science from Université Henri Poincaré – Nancy 1 (1997) and an engineer diploma from the Polytechnic University of Bucharest (1995). Since 2001, Dr. Saon has been a key member of IBM's speech recognition team which participated in several U.S. government-sponsored evaluations for the EARS, SPINE, GALE, RATS and BOLT programs. He has published over 150 conference and journal papers and holds 20 patents in the field of ASR. Since 2024 he has led Granite Speech, IBM's speech large language modeling work. He is a Fellow of the IEEE and served on the IEEE Speech and Language Technical Committee from 2011 to 2014.

Creative Control for Artists in the Age of AI

Bryan Pardo

Northwestern University

The dominant paradigm in generative models for music is to place a huge generative model in a data center and design the interaction to support casual creation with minimal human oversight or input. This emphasis on centralized, low-touch generative modeling for casual users has resulted in a flood of generic music that risks homogenizing our culture and marginalizing the people whose works were used to train the generative models displacing them.
Bryan Pardo’s Interactive Audio Lab chooses a different path: the goal is to support, rather than supplant, human artistry. The tools we develop can be deployed locally, are controllable through multiple modalities (text, audio examples, hand gestures) and support both intuitive control and virtuosity. This gives creators (musicians, sound artists, podcasters, Foley artists) the tools to support new creative work. We also put strong emphasis on attribution and acknowledgment of human artists. In this talk, Bryan will touch on recent work from the lab that supports this ethos and direction.

Bryan Pardo is a professor of Computer Science at Northwestern University. He studies fundamental problems in generative modeling of music, speech and sound effects, computer audition, content-based audio search and also develops inclusive interfaces blind and low-vision users of audio production tools. He is head of Northwestern University’s Interactive Audio Lab and co-director of the Northwestern University Center for HCI+Design. He received a M. Mus. in Jazz Studies in 2001 and a Ph.D. in Computer Science in 2005, both from the University of Michigan. He has authored over 150 peer-reviewed publications. He has developed speech analysis software for the Speech and Hearing department of the Ohio State University, statistical software for SPSS and worked as a machine learning researcher for General Dynamics. His patented technologies have been productized by companies including Bose, Adobe, Lexi, and Ear Machine.

Building Worlds of Sight and Sound

Ruohan Gao

University of Maryland, College Park

Our everyday world is inherently multisensory: what we see shapes what we hear, while what we hear reveals physical properties and spatial structures that vision alone cannot capture. In this talk, I will explore how we can build computational worlds that jointly model sight and sound, progressing from individual objects to static spaces and, ultimately, dynamic environments. I will first discuss audio-visual objects, including differentiable impact sound modeling and our recent work that leverages visual information to model and render sounds from physical interactions. I will then move from objects to spaces, showing how visual cues from multi-view images can inform spatial acoustic modeling and enable the reconstruction of acoustic environments. Finally, I will discuss our recent efforts toward 3D and 4D audio-visual scene generation, including methods that jointly model visual appearance, geometry, static and dynamic sound sources, and spatial sound fields to create explorable environments that can be both seen and heard. Together, these efforts point toward machines capable of understanding, reconstructing, and generating rich worlds of sight and sound.

Ruohan Gao is an Assistant Professor of Computer Science at the University of Maryland, College Park, where he leads the UMD Multisensory Machine Intelligence Lab. Before joining UMD, he spent a year at Meta as a Research Scientist after his Postdoc at the Stanford Vision and Learning Lab. He received his Ph.D. in Computer Science from The University of Texas at Austin. He mainly works in the fields of computer vision and machine learning with a particular emphasis on multisensory machine intelligence involving sight, sound, and touch. His research has been recognized with the Sony Faculty Innovation Award, AAAI New Faculty Highlights, the Michael H. Granof Award for UT Austin’s Top 1 Doctoral Dissertation, the Google PhD Fellowship, the Adobe Research Fellowship, a Best Student Paper Award at WASPAA 2025, a Best Paper Award Runner Up at BMVC 2021, and a Best Paper Award Finalist at CVPR 2019.

Posters

Instructions for posters: most poster boards are 32"x40" (a few are 20"x30", thank you to those who volunteered to use these smaller boards), and can be placed in portrait or landscape orientation.

Poster Session 1

AM-1. Don't CLAP: Are Music-Text Models Bag-of-Words?
Yuan-Chiao Cheng, Alexander Lerch
Georgia Institute of Technology
Abstract
- Text-to-music systems are assessed on audio quality and on how faithfully the music follows its prompt, and the CLAP score, the cosine similarity between a music-text model's audio and text embeddings, is the standard objective metric of faithfulness. We ask how accurately that score reflects the text: when an attribute is linked to an instrument (e.g., distorted guitar), does the text embedding capture that binding? To find out, we introduce an attribute swap perturbation: the caption of a real recording is edited by exchanging exactly one property, timbre, lead versus accompaniment, or order of first appearance, between two instruments. We then test four contrastive music-text models and one large audio-language model on whether the audio scores higher against the original caption than against the perturbed one. No contrastive model distinguishes the two captions reliably. The audio-language model does better, but further experiments show that its advantage rests largely on audio-agnostic language priors. Our results thus provide compelling evidence that the CLAP score and related metrics do not capture fine-grained musical meaning or attribute bindings; their representation is closer to a bag-of-words that leaves them insensitive to meaning-changing perturbations of the caption.
AM-2. Full-page Handwritten Jazz Lead Sheet Recognition
Jiayi Wang, Alexander Lerch
Georgia Institute of Technology
Abstract
- Despite recent advances in Optical Music Recognition (OMR), the recognition of handwritten jazz lead sheets remains a challenging problem. Recent work has shown promising results on pre-segmented staff-level lead sheet transcription, yet chord recognition error rates remain high and full-page handwritten lead sheet recognition has remained unexplored. We present a full-page OMR system for handwritten jazz lead sheets under limited data by applying a curriculum learning strategy that progressively trains the model on inputs of increasing length, from single staves up to complete pages. By extending previous evaluation strategies to disentangle melodic and harmonic transcription performance, we show that the proposed full-page system consistently outperforms the state of the art on chord symbol recognition.
AM-3. SpliceGen: Multi-Track Loop Generation Using Flow Matching And Inversion With Non-Parallel Data
Alejandro Koretzky, Amantur Amatov, Anastasiia Herus, Andy Thompson, Daniel Gonzalez, Jeff Sontag, Julian Lenz, Kyle Billings, Nate Raw, Naveen Sasalu Rajashekharappa, Shahan Nercessian, Yurii Bondarenko
Splice
Abstract
- We introduce SpliceGen, a controllable neural audio resynthesis framework for generating musical loops. Our model decomposes audio into 3 axes—timbre, tonality, and timing—and enables independent control through disentangled representations and conditional flow matching. Leveraging a rectified flow objective, it supports both forward generation and inversion, allowing flexible transformations to existing audio without paired training data. The system enables music production tasks ranging from sample variation and key-aware resynthesis to stem generation and musical accompaniment. SpliceGen adheres strongly to source timbre while offering fine-grained control over musical structure, providing a practical and unified approach to controllable, musically grounded audio generation.
AM-4. Smorph: Playable Sound Morphing with Diffusion Models
Annie Chu*, Hugo Flores García*, Johannes Imort, Oriol Nieto, Bryan Pardo, Prem Seetharaman, Justin Salamon
Northwestern University, Adobe Research
Abstract
- Sound morphing, generating intermediate sounds that transition from one sonic identity to another, can be a powerful tool for musical sound design. Existing diffusion-based morphing approaches entangle temporal structure and timbral identity, offering no mechanism to hold one fixed while transforming the other. We present smorph, a training-free guidance framework that preserves how a sound behaves over time while transforming what the sound is, allowing users to morph, for instance from brass to strings at a fixed pitch. We demonstrate across three morphing modes: prompt-to-prompt, audio-to-prompt, and audio-to-audio. Evaluations across diverse datasets show that smorph effectively produces smooth morph trajectories while substantially improving temporal-structure and source preservation over baselines, albeit with more conservative target-ward transformation in some settings. In an exploratory case study, musicians found smorph trajectories to be expressive and playable, suggesting structural anchoring can serve as a productive constraint for instrumental interaction.
AM-5. The Vocal Eras Tour: Microtiming Trends Across Decades and Genres
Elena Georgieva, Daniel Fernandes, Ethan Cajetan Menezes, Valerian Coelho, Magdalena Fuentes, Pablo Ripollés, Brian McFee
New York University, Agnel Institute of Technology and Design
Abstract
- Recent advancements in audio processing have enabled large-scale, quantitative analysis of musical trends. While previous research has explored changes in music over time, the evolution of vocal performance characteristics remains underexplored. In this work, we analyze the microtiming of 88,357 vocal tracks from popular songs spanning 45 years (1965–2010). Using source-separated vocal stems, we analyze vocal timing deviations in songs with a duple metrical structure at both the beat (quarter-note) and beat subdivision (16th-note) levels. We identify vocal onsets to study microtiming deviations. Through statistical analysis, we find significant differences in microtiming between musical genres, as well as trends over time. Our analysis reveals that Rap music has the greatest number of vocal onsets on average, generally arriving later than those of other genres. Across the other genres, singers tend to anticipate the beat, especially on beats 1 and 3. However, these entries have shifted closer to metrical targets over time, possibly reflecting the influence of digital editing tools and evolving listener preferences. Our findings demonstrate the utility of quantitative analysis methods for studying vocal timing and contribute new insights to the understanding of how singing voice recordings have evolved in popular music. We believe these methods have the potential to be applied to the study of other instruments and more diverse musical styles.
AM-6. MulTTiPop: A Multitrack Transcription Benchmark for Pop Music
Nathan Pruyne, Benjamin Stoler, William Chen, Chien-yu Huang, Shinji Watanabe, Chris Donahue
Carnegie Mellon University
Abstract
- We present MulTTiPop, a benchmark for evaluating automatic music transcription systems, consisting of pop music segments and their associated multitrack MIDI transcriptions. MulTTiPop contains 572 segments of popular music totaling 3.5 hours of audio, and contains songs from diverse genres and decades from the 1930s to 2000s. To collect this benchmark, we perform 1) Metadata Matching on song segments from the Lakh MIDI and TheoryTab datasets, 2) Beat-based Alignment using audio beat-tracking to warp MIDI timing and 3) Human Anchor Beat Selection to identify alignment points between audio and MIDI. We evaluate state-of-the-art automatic music transcription models on MulTTiPop and find substantial room for improvement, with the best model achieving 42% Onset F1. More details and sound examples of MulTTiPop are available at https://gclef-cmu.org/multtipop.
AM-7. Text-to-music Generation via Structured Captions AND Interpretable Evaluation
Junkai Wu, Zhepei Wang, Ge Zhu, Sarah Hye-yeon Lee, Mari Ostendorf, Nicholas J. Bryan
Adobe, University of Washington
Abstract
- Text descriptions of music are central to the performance of text-to-music generation (TTM) methods, but creation of paired text information for music generation is relatively underexplored. Here, we propose training latent diffusion/flow-based music generation models conditioned on structured captions (SCAPs) and compare with typical free-form captions via a set of controlled ablations. We compare free-form vs. structured captions, partial vs. full information coverage, and global vs. temporal text information during training, by fine-tuning different configurations of a pretrained flow-based diffusion transformer (DiT) and using a multi-stage inference-time prompt re-writer. Further, we introduce a new text-alignment evaluation protocol inspired by Davidsonian Scene Graphs (DSGs) for a fine-grained, interpretable measure of faithfulness of TTM methods. With multiple metrics, we show 1) that fine-tuning our DiT on SCAPs significantly improves text-alignment on music generation outputs while also improving quality, and 2) our proposed Music DSG metric better correlates with human preference than CLAP.
AM-8. MadLibs: Analyzing the Distribution of Generative Music
Matthew Keating, Noah Schaffer, Michael Casey, Nikhil Singh
Dartmouth College
Abstract
- What music can a text-to-music model actually make? The answer to this question has consequences for individual creators, communities, and musical culture more broadly. We introduce MadLibs, a framework to measure the expressive support of state-of-the-art music generators: the subset of musical behaviors they will actually produce across the space of prompts. We first develop strategies for generating musical prompts using natural language processing methods and existing caption corpora. Then, we use a set of MIR techniques to study the corresponding output spaces in terms of interpretable and qualitatively meaningful elements like key, beat, tempo, instrumentation, and dynamics. We find that text-to-music models are not interchangeable instruments for the same prompt space. Rather, each model has learned idiosyncratic properties that constrain the regime of music reachable via prompting it, for example, defaulting to certain tempi, failing to produce silences, or oversampling canonical band arrangements. Because users interact with these systems one prompt at a time, these biases are often invisible in practice yet cumulatively shape what AI-assisted music sounds like, risking the production of a new auditory monoculture hidden behind the promise of universal describability. We thus urge the MIR community to further study and responsibly steer population-level properties of music generation models.
AM-9. AILive Mixer: A Deep Learning based Zero Latency Automatic Music Mixer for Live Music Performances
Devansh Zurale, Iris Lorente, Michael Lester, Alex Mitchell
Shure, Inc.
Abstract
- In this work, we present a deep learning-based automatic multitrack music mixing system catered towards live performances. In a live performance, channels are often corrupted with acoustic bleeds of co-located instruments. Moreover, audio-visual synchronization is of critical importance thus putting a tight constraint on the audio latency. In this work we primarily tackle these two challenges of handling bleeds in the input channels to produce the music mix with zero latency. Although there have been several developments in the field of automatic music mixing in recent times, most or all previous works focus on offline production for isolated instrument signals and to the best of our knowledge, this is the first end-to-end deep learning system developed for live music performances. Our proposed system currently predicts mono gains for a multitrack input, but its design along with the precedent set in past works, allows for easy adaptation to future work of predicting other relevant music mixing parameters.
AM-10. FX-JEPA: Learning Effect-Aware Audio Representations for Reference-Based Audio Effect Transfer
Michael Clemens, Oleksandr Tsymbalenko, Tim Stilson, Scott Van Duyne
Universal Audio
Abstract
- Learning representations of audio effects from music recordings is difficult, as the observed signal entangles musical content with the processing applied to it. To address this challenge, we present FX-JEPA, a joint-embedding predictive architecture trained on paired dry and processed multitrack stems. Within a shared embedding space, source-conditioned forward and inverse predictors relate audio and effect parameters, while a regression head estimates the settings needed to match a reference recording. On a held-out MUSDB18-HQ test set processed with Pedalboard effect chains of varying length and order, FX-JEPA reduced multi-resolution STFT error by 17% relative to the strongest frozen-encoder baseline (StemFX with a learned readout; 95% CI of the difference [−0.30, −0.13]). It also outperformed Fx-Encoder++, AFx-Rep, and RelFx under the same readout. These results suggest that paired supervision on stems yields effect representations that transfer to unseen effect implementations better than general-purpose encoders, though isolating the contribution of the predictive objective from direct regression remains future work.
AM-11. Pitch-Shift Equivariant Audio Autoencoder
Dimitrios Bralios, Jonah Casebeer, Paris Smaragdis
University of Illinois Urbana-Champaign, MIT
Abstract
- Neural audio autoencoders can compress audio with high fidelity but their latents can be difficult to interpret or control. Our goal is to preserve reconstruction quality while structuring latents so known audio transformations produce predictable latent changes. We propose a framework for learning transformation-structured latents and use pitch shifting as our test case. We find that naive equivariance regularization leads to unintended shortcut solutions either in encoding or decoding. We address this by regularizing the representation to track the applied transformation via an angle-aware objective. The resulting latents encode absolute pitch, enabling competitive pitch estimation while preserving high-quality reconstruction. They also support relative pitch manipulation, enabling pitch shift outside the training regime. These results suggest that one representation can simultaneously support simple reconstruction, analysis, and manipulation.
AM-12. Discrete Parameter Estimation for DDSP Source Separation
David Millard, Andrea Cogliati
University of Rochester
Abstract
- This work investigates discrete parameter estimation in differentiable digital signal processing (DDSP), with applications to source separation and multi-pitch estimation. We fit multiple Extended Karplus–Strong synthesizers to a single-channel mixture, jointly estimating integer delay lengths along with continuous feedback/feedforward parameters. We compare methods for handling discrete parameters, including continuous relaxation, straight-through Gumbel–Softmax, categorical REINFORCE, and discrete search. We evaluate how the estimation method affects recovery of individual sources and their pitches, alongside parameter accuracy, mixture reconstruction, computational cost, loss landscapes, and optimization stability. Controlled synthetic mixtures with known source counts provide ground-truth parameters for the multi-source separation experiments. The poster will present the comparative study and discuss the assumptions underlying this approach to source separation and directions for future work.
AM-13. Linear STFT Multi-Pitch Estimation and Role-Agnostic Voice Attribution for Vocal Ensembles
Junyoung Koh, Hao-Wen Dong
University of Michigan; Yonsei University
Abstract
- Vocal ensemble transcription aims to recover the pitches sung by multiple simultaneous voices and attribute them to individual voices. This task is challenging because vocal parts often occupy overlapping pitch ranges, causing their harmonics to overlap in the mixture. We address these challenges from two complementary perspectives, examining the spectral representation used for multi-pitch estimation and the modeling of voice identity for role-agnostic attribution. First, we show that the harmonic constant-Q transform, a standard input representation for vocal multi-pitch estimation, is not necessary. A linear STFT using only its first 360 frequency bins (0–7.7 kHz) achieves higher accuracy across held-out songs from five a cappella and choir corpora, improving pooled F1 by 0.020 while reducing training time by approximately 13×. Second, building on this representation, we introduce M0TET, a role-agnostic slot model that jointly predicts a pitch track, voicing activity, and a timbre embedding for each voice. Trained with slot-batch contrastive learning without singer or voice-part labels, M0TET achieves strong per-voice pitch estimation while enabling reference-free voice attribution within 0.007 F1 of oracle assignment. Together, these results demonstrate that vocal ensemble transcription can be performed using a simple spectral representation and learned voice identity without specialized harmonic inputs or predefined vocal roles. (Part of this work is under review at ICASSP 2027 and AAAI 2027.)
AM-14. Universal Short Time Fourier Transform
Manan Mittal, John R. Buck, Ryan M. Corey, Andrew C. Singer
Stony Brook University, University of Massachusetts Dartmouth, University of Illinois Chicago
Abstract
- Standard time-frequency analysis relies on the Short-Time Fourier Transform (STFT), a representation parameterized by a fixed analysis window length. This rigid geometric parameter forces a compromise between time and frequency resolution, obscuring the underlying dynamics of non-stationary signals. Existing adaptive windowing techniques attempt to mitigate this fundamental trade-off by dynamically shifting window lengths. However, they suffer from erratic temporal tracking and destroy the Linear Time-Invariant (LTI) envelope, preventing artifact-free signal reconstruction. The Universal Short Time Fourier Transform (U-STFT) resolves this challenge by integrating competitive sequential prediction directly into the time-frequency architecture. The framework implicitly maintains an exponentially large family of candidate window histories and dynamically re-weights them based on their cumulative loss. The theoretical formulation establishes a bound on the cumulative sequential regret relative to an omniscient oracle that selects the optimal piecewise-stationary sequence of transform window lengths in hindsight. Simulations on hyperbolic chirps and physical experiments on marine biological recordings demonstrate optimal tracking and artifact-free LTI signal reconstruction.
AM-15. Fast Uncertainty-Aware Estimation of Acoustic Source Location and Ocean Environmental Parameters Using Generative AI
Mark Goldwater, Daniel P. Zitterbart, Andrew R. McNeese, Preston S. Wilson, Julien Bonnel
MIT, Woods Hole Oceanographic Institution, The University of Texas at Austin, Scripps Institution of Oceanography
Abstract
- Many scientific inference problems require estimating hidden properties of a physical system from indirect sensor measurements while also quantifying uncertainty. In underwater acoustics, estimating parameters that define an acoustic source (e.g., source-receiver range of a whale) and the ocean environment (e.g., water depth) is traditionally done using Bayesian inference with Markov chain Monte Carlo (MCMC) algorithms. This approach repeatedly runs expensive physics simulations to compare hypothetical sources and environments against the observed data. Because each inference requires thousands to millions of simulations, due to the large number of unknown parameters, obtaining a posterior distribution can take hours or even days. This work presents an approach that amortizes this computation using a generative model. We train a conditional invertible neural network (cINN) to learn the posterior distribution over the source and environmental parameters directly from simulated data. Equipped with a trained cINN, we can perform Bayesian inference in a likelihood-free manner and without carrying out an optimization procedure for each new measurement. Once trained, the model produces calibrated posterior distributions from data collected by a single underwater sensor in just milliseconds. We demonstrate that the learned posterior distributions accurately capture uncertainty and reflect the amount of information contained in the observations. When applied to experimental data from a large-scale ocean experiment, the method produces results that closely agree with a traditional MCMC-based approach while reducing inference time by many orders of magnitude. More broadly, this work illustrates how generative models can combine physics-based simulation with modern machine learning to enable fast, uncertainty-aware inference for computationally intensive scientific inverse problems.
AM-16. Tracking individual mesopelagic animals with AUV-mounted echosounders
Camille Wardlaw, Julien Bonnel, Zhaozhong Zhuang, Weifeng (Gordon) Zhang, Andone Lavery
MIT, UCSD, Oregon State University, Woods Hole Oceanographic Institution (WHOI)
Abstract
- Active acoustics is the primary tool for observing animals in the mesopelagic zone (200–1000 m), which hosts one of the largest animal biomasses on Earth. Conventional shipboard echosounders, however, average backscatter over many animals and hundreds of meters, obscuring individual behavior. Mounting a split-beam echosounder on an autonomous underwater vehicle (AUV) brings the sensor to depth, where individual animals can be resolved and tracked in three dimensions. This requires detecting targets, associating them across pings, and estimating their motion. The mesopelagic zone adds a further challenge: dense and overlapping targets. This poster presents our tracking pipeline, which reconstructed ~20,000 individual trajectories at 300 m depth, and a case study in which it revealed rapid diving responses to seismic airgun noise that concurrent shipboard echosounders did not detect.
AM-17. Structured Pruning of Feedback Delay Networks
Facundo Franchino, Karolina Prawda
University of York, MIT
Abstract
- Feedback delay networks (FDNs) are recursive filter structures used to synthesise artificial reverberation. Their feedback matrix governs both the exchange of energy between delay lines and the work required at every sample. We develop a structured approximation method that connects these acoustic and computational concerns, allowing high-order dense orthogonal designs to be fitted to efficient implementations. The feedback matrix is replaced by stages of fixed permutations and small orthogonal blocks. We characterise how connectivity grows with depth and derive exact conditional block updates for geometric fitting, followed by refinement against the complete FDN magnitude response. Orthogonality is preserved by construction. The factors admit direct vectorised evaluation, with arithmetic and coefficient storage linear in network size for fixed block size and depth. On an Apple M2, measured feedback-multiplication speedups over per-call dense BLAS range from approximately three to more than twenty times across the tested network sizes, with substantial reductions in coefficient storage. Listeners rated the evaluated high-order configurations as nearly identical to their dense references. Analysis of processor utilisation and register reuse explains why execution time depends on more than the operation count. Block size and depth provide practical control over response fidelity, storage, and real-time cost, allowing acoustic design to precede the choice of an efficient implementation.
AM-18. FFASR: Benchmarking Far-Field Automatic Speech Recognition using High-Fidelity Simulated RIRs
Shivam Saini, Eric Bezzam, Georg Götz, Alessia Milo, Steinar Guðjónsson,Konstantinos Gkanos, Finnur Pind, Daniel Gert Nielsen
Treble Technologies
Abstract
- Far-field automatic speech recognition (ASR) degrades under reverberation, noise, and talker motion, yet the benchmarks that drive model selection emphasize close-microphone speech. We present FFASR, a held-out corpus of 15,637 utterances and an open leaderboard spanning nine conditions, each varying a single acoustic factor: anechoic near-field speech, a measured-versus-simulated office-lab pair, static far-field mixtures at high/mid/low signal-to-noise ratio (SNR), and moving-talker variants at matched SNR. Dry speech from 15 talkers is convolved with hybrid wave/geometrical-acoustics room impulse responses from 14 furnished rooms; because the speech is newly recorded and the test waveforms are never released, the corpus resists training-data contamination. Across contemporary systems, mean word error rate (WER) rises from 4.4% near-field to 41.3% in the static low-SNR condition; a moving talker adds a small but consistent penalty at matched SNR; and on the office-lab pair, measured and simulated WER agree to within about 1.7 pp on average. These results support high-fidelity simulation as a scalable proxy for measured far-field evaluation under the conditions we test.
AM-19. Few-Shot Room Impulse Response Interpolation In Latent Domains
Jackie Lin, Yoshiki Masuyama, Christoph Boeddeker, Julius Richter, Gordon Wichern, Minje Kim, Jonathan Le Roux
Mitsubishi Electric Research Laboratories (MERL), University of Illinois Urbana-Champaign
Abstract
- We address the problem of few-shot room impulse response (RIR) interpolation, where the goal is to estimate the RIR at an arbitrary receiver location given a small set of location-labeled reference RIRs from that same room. Naïve signal-domain methods, such as direct interpolation in the time domain, often produce inaccurate results. To overcome this, we leverage an audio variational autoencoder (VAE) to map RIRs into a compact and smooth latent space that is amenable to interpolation. We explore both training-free and network-based approaches in this latent domain, including distance-based kernel regression and geometry-informed attention pooling. Our results show that simple kernel methods serve as strong baselines, while lightweight neural networks further improve performance. We present a promising, flexible latent-domain framework for few-shot RIR interpolation and provide insight into the interpolation behavior of latent representations across diverse acoustic conditions.
AM-20. Reverberant Speech Conditioning for Material-Aware Acoustic Parameter Mapping
Yi Wu, Olha Townsend, Qianqian Fang, Magdalena Fuentes, You Zhang
Dolby Laboratories, Inc., New York University
Abstract
- Dense maps of room acoustic parameters (T30, EDT, C50, DRR) describe how a room sounds at every position. Scene-wide estimation predicts such maps from a floor plan but requires a reference impulse response, which is hard to obtain. Our system instead receives a floor plan and easily recorded reverberant speech from one talker-microphone pair. We compare four speech-derived conditioning inputs: MFCCs, an embedding from the blind estimator BERP, BERP’s scalar reverberation-time estimate, and an impulse response reconstructed by Rec-RIR. Our main benchmark is the MRAS dataset, where material assignments vary at fixed geometry. The reconstructed impulse response performs best among the four inputs, reducing T30 error from 31.5% with the floor plan alone to 19.9%, 83% of the reduction achieved by the reference response. We find that adding speech mainly sets the overall level of the map, with little improvement in predicting how the acoustics vary across the room.
AM-21. Benchmarking Orientation Encodings for Acoustic Neural Field
Jiawen Mao
University of Michigan, Ann Arbor
Abstract
- Source and receiver orientations parameterized by 2D spherical angles [azimuth, colatitude] are critical spatial conditioning inputs for training neural fields as spherical acoustic field representations. Existing approaches condition networks on latent embeddings learned from Euclidean coordinate encodings, based on raw angular coordinates or equivalent 3D Cartesian directional vectors. However, encodings rooted in Euclidean geometries may be less adapted to the spherical domain and lead to spectral and spatial artifacts, especially at the poles. Alternatively, spherical harmonics (SH)-based encodings have demonstrated success on spherically-defined geospatial data but are under-explored within acoustic domains. In this work, we benchmark a SIREN architecture paired with three distinct orientation encoders on the Aachen HRTF database: (1) direct encoding from raw spherical angles, (2) sinusoidal encoding of 3D directional vectors, and (3) SH encoding from raw spherical angles. We evaluate the models across varying spatial sampling densities and compare their convergence speed and spectral reconstruction fidelity at interpolation points.
AM-22. One-Step Voice Conversion by Learning kNN Transport in WavLM Space
Anton Selitskiy and David Millard
Stony Brook University, University of Rochester
Abstract
- Voice conversion (VC) systems fall into two families: non-parametric embedding-space methods, which need no trained model but degrade on short target utterances, and spectrogram-based neural architectures, which achieve strong quality via multi-module pipelines with tens of millions of parameters. We propose kNN-FM-VC, a single conditional flow-matching network that learns to approximate the kNN-VC mapping between WavLM embedding distributions of source and target speakers, replacing explicit pointwise kNN matching with a neural regressor trained on kNN-generated pairs. The model is conditioned on the target speaker via cross-attention and FiLM, and trained under three Gaussian conditional paths (Schrödinger bridge, straight line, and constant-variance Gaussian tube), enabling few-step sampling. Unlike Phoneme Hallucinator, which uses an upsampling stage followed by kNN matching, our 13M-parameter model performs conversion with a single learned network and supports one-step inference. On LibriSpeech, the one-step Gaussian Bridge achieves lower WER and higher estimated speech quality than FreeVC and Phoneme Hallucinator. Relative to kNN and kDOT, it substantially reduces WER. Preprint: arXiv 2609.27230
AM-23. Emotion Across Speech and Faces: Shared Affective Mechanisms in Multimodal Foundation Models
Xiutian Zhao, Luqi Sun, Björn Schuller, and Berrak Sisman
Johns Hopkins University, Technical University of Munich
Abstract
- Modern multimodal foundation models can recognize emotion from both speech and facial expressions, but it remains unclear whether they rely on shared affective mechanisms across these modalities. We investigate sparse emotion-sensitive neurons (ESNs) in the decoder MLPs of Gemma-4-12B-it, MiniCPM-o-4.5, and Qwen2.5-Omni-7B. Using speech emotion recognition and facial expression recognition as complementary probes, we identify acoustic and visual ESNs and validate their causal roles through activation deactivation and steering. We find that ESNs identified from speech and facial inputs exhibit emotion-matched overlap and similar layer-wise distributions. More importantly, transferring ESN interventions across modalities produces bidirectional, emotion-specific causal effects: speech-derived ESNs affect facial emotion recognition, while face-derived ESNs affect speech emotion recognition. These findings suggest that multimodal foundation models partially converge on shared decoder-level affective components, which can be localized and manipulated without additional training.
AM-24. A Benchmark for Controllable Speaking-Style Captioning in Audio-Language Models
Shreeram Suresh Chandra, Ashwyn Sharma, Liang Lu, Xiutian Zhao, Leda Sari, Björn Schuller, Berrak Sisman
Johns Hopkins University, Vocal Bridge, Otter.ai, Imperial College London
Abstract
- Speaking-style captioning (SSC) aims to generate natural language descriptions of how speech is delivered, capturing paralinguistic attributes such as vocal timbre, prosody, and expressivity. Many downstream applications, including conversational AI agents, controllable speech generation, and large-scale audio annotation, require dimension-specific style captions that describe targeted aspects of speech (e.g., speaker traits, emotion, or delivery style), rather than a single undifferentiated description. However, existing work lacks a unified formulation for controllable speaking-style captioning. We address this gap by formulating SSC as an instruction-following audio-language task, where explicit instructions specify the speaking-style dimensions to be described. We introduce StyleInstructCaps, the first standardized benchmark for controllable speaking-style captioning in audio-language models. StyleInstructCaps provides a dataset and evaluation framework that measures metadata groundedness, hallucination, instruction-following ability, generalization to unseen instructions and audio datasets, and speaker-style consistency. Across nine state-of-the-art audio-language models, we find that strong instruction-following ability does not necessarily imply grounded speaking-style understanding, and that SSC-specific instruction-diverse training substantially improves groundedness, controllability, and speaker consistency.
AM-25. TRACE-EVC: Text-Guided Relative Affective Control for Zero-Shot Emotional Voice Conversion
Zihan Zhang, Shreeram Suresh Chandra, Zongyang Du, Xiutian Zhao, Aurosweta Mahapatra, Hao Zhang, Philipp Koehn, and Berrak Sisman
Johns Hopkins University
Abstract
- Existing emotional voice conversion methods specify a target emotion through labels, reference speech, or style descriptions, but do not explicitly model how emotional expression should change relative to the source. We introduce instruction-guided relative emotional voice conversion, where natural-language instructions such as “make the speech slightly calmer” specify the direction and degree of affective change. We construct TRACE-Instruct, a dataset covering categorical emotion transitions, intensity modifications, and open-ended affective changes. Our framework, TRACE-EVC, uses an Emo-Compass module to model these transformations as source-anchored rectified flows, enabling conversion without target emotion labels or reference speech. Experiments on ESD and MEAD demonstrate effective relative emotion control and generalization to unseen speakers while preserving speaker identity and linguistic content. TRACE-EVC achieves stronger perceptual instruction following and better ordered control of valence and arousal than the evaluated large audio-language models, while remaining competitive with conventional systems on categorical emotion conversion.
AM-26. VoiceDesigner: Text-to-Voice Generation and Editing via Unified Diffusion Modeling and Data Augmentation
Jiarui Hai, Karan Thakkar, Ke Chen, Yunyun Wang, Jiaqi Su, Rithesh Kumar, Mounya Elhilali, Zeyu Jin
Johns Hopkins University, Adobe Research
Abstract
- We present VoiceDesigner, a unified framework for text-to-voice generation and editing that enables diverse and controllable voice design. VoiceDesigner supports generating voices from natural language descriptions, including both real-world and fictional voices, as well as editing existing voices through instructions such as modifying emotion, tone, and other vocal attributes. To achieve this, we introduce a hybrid data augmentation pipeline combining digital signal processing and generative models, along with an improved diffusion transformer architecture that unifies voice generation and editing. Our results demonstrate improved alignment with voice descriptions and editing instructions while maintaining high perceptual quality.
AM-27. Enrollment-Free Desired Speaker Extraction in Multi-Talker ASR via Multiple Serialization Orders
Chien-yu Huang, Alexander Polok, Jee-weon Jung, Takuya Higuchi, Li-Wei Chen, Zakaria Aldeneh, Barry-John Theobald, and Shinji Watanabe
Carnegie Mellon University, Brno University of Technology, Apple
Abstract
- Multi-talker automatic speech recognition (ASR) typically transcribes all speakers in a conversation, and serialized output training (SOT) enables this using a single sequence-to-sequence model. However, the desired speaker depends on the application: a summarizer may need to focus on the most talkative participant, while an assistant may need to respond to the most recent speaker. Target-speaker ASR can address this by identifying the desired speaker through enrollment. Yet, when the desired speaker is defined by a property of the recording itself, such as speaking time or recency, enrollment is unnecessary and may be impractical. We therefore propose enrollment-free desired-speaker extraction by extending the serialization order in SOT with a desired-speaker policy, such as earliest, most talkative, or loudest first. We further extend this approach to a multi-task model that switches the desired-speaker policy by replacing a task token at inference time. The proposed scheme is particularly efficient when only the first desired speaker is needed, as decoding can terminate after transcribing the desired speaker without processing the remaining undesired speakers. On the AMI corpus, our method successfully extracts the most talkative speaker in 84.2% of multi-speaker groups while making decoder inference up to 4x faster.
AM-28. Does Optimizing Predicted MOS Improve Human-Perceived Speech Quality?
Dominik Klement, Yoshiki Masuyama, Christoph Boeddeker, Julius Richter, Kohei Saijo, Gordon Wichern, Jonathan Le Roux
Mitsubishi Electric Research Laboratories (MERL), Brno University of Technology, Mitsubishi Electric Corporation
Abstract
- Neural estimators of human mean opinion score (MOS) are increasingly used as auxiliary losses in speech generation and enhancement systems to optimize perceptual quality. However, whether improvements in predicted MOS translate into better human-perceived speech quality remains insufficiently explored. In this work, we systematically investigate the reliability of MOS predictors as optimization objectives. First, we demonstrate that widely used MOS predictors can be artificially optimized through imperceptible waveform perturbations. Moreover, nine MOS predictors can be simultaneously optimized using a shared perturbation, revealing common adversarial vulnerabilities of these predictors. We then investigate MOS-aware model fine-tuning and find that increases in predicted MOS do not necessarily translate into perceptual improvements, as assessed through human listening tests. We further observe that optimizing one predictor can indirectly increase the scores of other non-optimized predictors, raising concerns about their reliability as independent evaluation metrics. Finally, jointly optimizing multiple predictors increases their scores without a corresponding statistically significant improvement in human-perceived quality. These findings illustrate Goodhart's law in neural speech quality assessment and highlight that agreement among multiple MOS predictors does not guarantee perceptual improvement. Our results underscore the importance of human listening tests, particularly when evaluating MOS-optimized speech systems.
AM-29. RVQHash: Structured Neural Codes are Hash Codes for Efficient Speech Enhancement
Cameron Churchwell, Li-Chia Yang, Mikolaj Kegler, Nils L. Westhausen, Shuo Zhang, Marko Stamenovic, Minje Kim
University of Illinois Urbana-Champaign, Bose Corporation
Abstract
- We introduce RVQHash, a novel hashing-based system which models speech enhancement (SE) in the space of discrete neural audio codec (NAC) tokens as a single hash table lookup per frame. As opposed to prior approaches to hashing-based SE, which utilize learned locality-sensitive hash functions to retrieve ideal STFT masks, RVQHash combines an off-the-shelf NAC with a masking strategy to construct a hash table directly mapping noisy tokens to enhanced tokens. We explore a range of masking patterns and further distinguish two proposed systems, RVQHash-Blend and RVQHash-Binary. RVQHash-Blend achieves best results and represents a low-cost, gradient-free alternative to neural networks for codec-space speech enhancement. RVQHash-Binary achieves slightly worse performance, but in exchange, enhancement costs only a small number of integer and binary operations per-frame to carry out the hash table lookup, representing effectively free enhancement for any system using a NAC.
AM-30. BSBWE: Joint Denoising, Transfer-Path Compensation, and Bandwidth Extension for Own-Voice Restoration From Noisy In-Ear Microphones
Yulun Wu, Mikolaj Kegler, Marko Stamenovic
University of Illinois Urbana-Champaign, Bose Corporation
Abstract
- Modern hearables often employ in-ear microphones which attenuate environmental noise while capturing the user’s own voice via body conduction, enabling noise-robust voice pickup from a single in-ear sensor. However, the voice picked up by the in-ear microphone is band-limited, nonlinearly reshaped by the occluded-ear path, and contaminated by residual noise, so recovering wide-band speech demands joint denoising, transfer-path compensation, and bandwidth extension. We propose BSBWE, a causal band-split dual-path model that performs all three tasks from the in-ear microphone input alone. Targeting efficient embedded deployment, we ablate the dual-path building blocks, comparing recurrent, state-space Mamba and self-attention operators. Despite Mamba’s efficiency benefits over long sequences, self-attention proves lighter and stronger for intra-frame modeling across the short sub-band axis. Paired with Mamba on the time axis, it outperforms the compact state-of-the-art EBEN baseline on every metric and condition of the Vibravox benchmark, with lower complexity, latency, and parameter count.
AM-31. Cocktail-Talker: Multi-Speaker Dialog Modeling in Noisy Social Environments with Turn Action GRPO
Xilin Jiang, Riki Shimizu, Sukru Samet Dindar, Junkai Wu, Zhongweiyang Xu, Nima Mesgarani
Columbia University, University of Washington, University of Illinois Urbana-Champaign
Abstract
- Spoken dialog systems are typically designed for clean, dyadic interactions in which a single user and an assistant take turns speaking. Real-world social conversations, however, are often more ambiguous: multiple speakers may participate in the same conversation amid irrelevant speech and background noise. Each utterance may be directed to the assistant, addressed to another speaker, or completely irrelevant. In such settings, the assistant must decide not only what to say, but also whether to speak at all. In this paper, we introduce Cocktail-Talker, a speech LLM framework for multi-speaker spoken dialog modeling in noisy social environments. We model the assistant's behavior with three action tokens: <|respond|>, <|listen|>, and <|ignore|>, placed before a response or silence. Cocktail-Talker is trained via supervised finetuning and reinforcement learning to generate the appropriate action token and, only in <|respond|> mode, a speech response. To prepare the training data, we develop Cocktail-DialogGen, an LLM-based data pipeline that simulates realistic multi-speaker dialogs with speaker roles across diverse social settings. Together, these components take a step toward spoken dialog systems that interact more naturally and selectively in complex social environments.
AM-32. Sympatheia: Emotionally Adaptive Voice Assistant with Continuous Affect Conditioning
Sukru Samet Dindar, Riki Shimizu, Xilin Jiang, Nima Mesgarani
Columbia University
Abstract
- Empathetic spoken dialogue systems must infer a user's emotional state to respond appropriately, yet everyday speech often carries weak, neutral, or ambiguous affective cues. To address this, we introduce Sympatheia, a speech-to-speech dialogue framework conditioned on affect inferred from the user's speech and, when available, explicit affect specifications provided as a continuous valence-arousal (VA) control signal by a multimodal sensing module or user interface. To train our model, we construct Sympatheia-18k, an emotion-conditioned synthetic spoken dialogue corpus with 12 emotion anchors. This dataset includes an emotional split for learning affective speech behavior, and a neutral split that pairs emotionally neutral queries with multiple emotion-conditioned responses to isolate explicit emotion control in emotionally ambiguous cases. Empirical results show that Sympatheia outperforms speech conversational baselines in generating responses whose semantic content and spoken delivery are both emotionally appropriate. We further show that the same VA interface can integrate emotion estimates from diverse sensing modules, including facial expression, biosignals, and textual affect descriptions, improving response alignment when speech alone provides limited emotional evidence. These results suggest that continuous affect conditioning is an effective practical step for building emotionally adaptive voice assistants.
AM-33. Audio-based automatic detection of tutor engagement
Jiani Wang, Camille Giordano, Zeyneb Sarioglan, Ryan S. Baker, Jacob Whitehill
Worcester Polytechnic Institute, University of Pennsylvania, Adelaide University
Abstract
- We define an annotation protocol to measure tutor engagement from short (2-3sec) segments of tutor speech that captures both what was said (content engagement) and how it was said (voice engagement). This protocol is inspired by prior work on teacher engagement and aims to capture the tutor's effort towards teaching and developing positive socio-emotional relationships with their students. Further, by harnessing speech foundation models (e.g., WavLM), we train automatic tutor engagement detectors: The cross-validation accuracy (Pearson r) of the detectors on the ANON dataset of math tutoring sessions containing 873 utterances from 33 tutors was r=0.6762 across all utterances and tutors (outperforming a strong baseline of low-level audio descriptors whose correlation was r=0.5468); r=0.5239 across the utterances within each tutor; r=0.8593 across different tutors. Additionally, we provide evidence that tutor engagement is associated with tutors' instructional discourse strategies (i.e., talk moves), as well as (but less consistently) the students' learning outcomes. Finally we discuss ethical concerns such as overreliance on automatic engagement measurement and privacy.
AM-34. One Model, Any Sensor: Sensor Fusion and Few-Shot Generalization for Body-Conducted Speech Enhancement
Kenneth Egan, Abia Hasan, Mehmet Ergezer
Wentworth Institute of Technology
Abstract
- Body-conducted microphones provide noise-robust speech sensing for wearable devices, but existing enhancement systems are generally tied to a specific sensor and must be retrained when the hardware changes. We introduce heterogeneous sensor-agnostic speech enhancement: a single model that enhances speech from variable subsets of physically distinct body-conducted sensors without retraining. Our 3.9M-parameter architecture combines a shared convolutional frontend, FiLM conditioning on learned sensor embeddings, cross-sensor attention, stochastic sensor dropout, and a causal Mamba backbone. Using five heterogeneous sensors from the Vibravox corpus, the model achieves 5.78% WER on clean speech and 7.58% on noisy speech, corresponding to a 59% relative reduction in noisy-speech WER compared with a matched single-sensor model. The model also remains robust when sensors are unavailable and can adapt to new sensor identities by fine-tuning only a sensor embedding with minutes of paired data. We additionally identify amplitude drift as a failure mode of scale-invariant enhancement objectives, highlighting a gap between conventional signal-quality metrics and downstream speech-recognition performance.
AM-35. Mixture-Derived Speaker Embeddings: Can They Replace Enrollment in Pre-trained Target Speech Extractors?
FNU Sidharth, Hao-Wen Dong, and Dhruv Jain
University of Michigan, Ann Arbor
Abstract
- Target speech extraction (TSE) isolates one talker from a mixture by conditioning a separation network on that talker’s speaker embedding, usually computed from a clean enrollment recording. In the settings where TSE could help most, such as live conversations and assistive listening for people with hearing loss, a clean recording of each talker is rarely available. A common workaround is diarize-then-enroll: find where each speaker talks alone and embed that audio. Recent methods instead estimate speaker embeddings directly from the multi-speaker mixture, but are mostly evaluated on clustering quality or with extractors trained jointly. We ask a practical question: can mixture-derived embeddings stand in for enrollment in an existing extractor without retraining it, and how do they compare with diarize-then-enroll? We train a multi-head encoder that predicts one embedding per speaker from the mixture, using permutation-invariant cosine distillation from a frozen single-speaker teacher. We train a DPCCN extractor only on clean teacher embeddings, freeze it, and swap in embeddings from each system at test time. We evaluate on noisy two-speaker LibriSpeech mixtures, reporting SI-SDR and how often the two embeddings select two different speakers. First, mixture-derived embeddings clearly outperform diarize-then-enroll (pyannote 3.1 with the same teacher) through the same frozen extractor, and the gap widens as noise increases. Diarization itself is not the bottleneck: even oracle solo regions barely help, because short, noisy solo segments give unreliable embeddings. Second, mixture-derived embeddings work as a drop-in replacement. They perform close to a clean embedding of the target’s own speech, whether the encoder is trained by distillation or with speaker labels and a fitted linear map. The approach carries over to a second (ECAPA-TDNN) embedding space, and a causal variant suitable for streaming costs only modestly. Third, the remaining failures, shared across approaches, come from both embeddings selecting the same speaker. A relational loss that pushes the two heads apart does not reduce these failures, which points to alignment-oriented objectives as the next step.
AM-36. Battery-Friendly AI-based Packet Loss Concealment for Mobile Voice: a Low-Latency, On-Device Add-on to Opus and WebRTC
Mahboob Karimian, Pierre Favrat, Nicolas Abelé
Sonix SA
Abstract
- Real-time voice over Wi-Fi and 4G/5G suffers from lost and delayed packets. Packet loss concealment (PLC) fills these gaps at the receiver, but the strongest neural concealers either wait for future audio, adding latency, or are too demanding to run on a phone during a call without draining the battery or taking CPU and GPU resources away from a game. Building on our earlier ultra-low-latency work, which let musicians play together remotely in real time, this project brings the same ambition to voice for mobile gaming and communication. Our goal is an AI concealer that builds on Opus, the codec used by WebRTC, as a pluggable receiver-side add-on that requires no changes to the sender, bitstream, or bandwidth; adds no latency by working strictly causally on 10 ms frames; improves quality beyond the state of the art; and is small enough to run on the neural processing units (NPUs) of mobile and battery-powered edge devices, so concealment does not cost energy or game performance. We benchmarked our approach against the concealers currently shipped with Opus using PLCMOS, Microsoft’s learned quality score for concealed speech, on speech never seen during training. Inside Opus, our neural predictor, integrated into the Opus Deep PLC pipeline, delivers a significant quality improvement over Opus’s own shipping model. In end-to-end audio-in/audio-out tests, our standalone concealer is still early in development, but already reconstructs lost speech more faithfully than Opus’s classic concealer at every tested loss rate across six languages, and performs on par with Opus Deep PLC under heavy, bursty loss. In the next research phase, the goal is to deploy the concealer on mobile NPUs.
AM-37. Distribution-Aware Activation Steering for Controllable Speech
Xingjian Du, Haoyu Zhang, Jonathan Le Roux, Zhiyao Duan
University of Rochester, University of Tokyo, Mitsubishi Electric Research Laboratories (MERL)
Abstract
- Controllable TTS often requires labeled data or model adaptation. We investigate training-free activation steering for gender, age, accent, and emotion in a frozen Qwen3-TTS model. Building on fixed-strength MeanDiff steering, we introduce GMM-based activation-dependent steering directions and a density-informed strength schedule. We also propose to evaluate these methods using hypervolume to capture the trade-off between speech quality and attribute controllability.
AM-38. Beyond Clean Text: A Benchmark for PII/PHI Detection Robustness to ASR Transcription Error
Raquel Harrison, Rachel Manzelli
Modulate
Abstract
- Voice-based systems routinely handle personally identifiable and health information (PII/PHI), but the benchmarks used to evaluate PII/PHI detection are almost universally built on clean, pre-transcribed text rather than the noisy, error-laden transcripts that real ASR pipelines actually produce. To close this gap, we build and release a 545-clip, human-validated English audio benchmark specifically designed to stress-test text-based privacy detectors under realistic transcription conditions, spanning two ASR systems with more than a threefold difference in word error rate (6.49% vs. 20.64%). Using this dataset, we run ground-truth text and both sets of ASR transcripts through three open-source detectors spanning distinct architectures — zero-shot NER, generative tagging, and fixed-taxonomy classification — to isolate how transcription noise, rather than detector design alone, shapes privacy-relevant performance. We find that detection degradation under ASR noise is not uniform across architectures: some absorb a large one-time hit that plateaus regardless of further transcription quality, others degrade smoothly in step with ASR error rate, and others remain near-insensitive within our tested range. These divergent failure modes suggest that architecture, not ASR quality alone, is the stronger predictor of how much privacy-relevant detection capability survives real-world voice deployment. We release our audio benchmark dataset publicly to support further research on PII/PHI detection under realistic acoustic and transcription conditions.
AM-39. Spatial Semantic Segmentation of Sound Scenes: Task Design, Results and Open Problems in DCASE 2025 and 2026
Masahiro Yasuda, Binh Thien Nguyen, Noboru Harada, Romain Serizel, Mayank Mishra, Marc Delcroix, Carlos Hernandez-Olivan, Shoko Araki, Daiki Takeuchi, Tomohiro Nakatani, Nobutaka Ono
NTT, Inc., Université de Lorraine, Tokyo Metropolitan University
Abstract
- Our goal is to decompose a multichannel recording of a sound scene into separate, labelled sound objects, for example a baby crying, a television and people chatting. Such a representation would support immersive remote communication, acoustic scene monitoring, and selective listening, where a listener hears only the sounds they care about. We call this problem spatial semantic segmentation of sound scenes (S5): recognize which sound events are present in a multichannel recording and extract a clean waveform for each of them. To open this problem to the research community, we organized S5 as a task of the DCASE Challenge, an open benchmark for environmental sound analysis, in 2025 and 2026, and provided shared data, a baseline system and an evaluation metric. In this poster, we discuss three aspects of these two editions: the task design, the results and the open problems.
AM-40. SAS: Scene-Annotated AudioSet Strong for Joint Scene-Event Analysis
Michel Olvera, Paraskevas Stamatiadis, Romain Serizel, Slim Essid, Gaël Richard
Télécom Paris, Université de Lorraine, NVIDIA
Abstract
- Context is central to human auditory perception, as listeners interpret sounds within their acoustic environment to resolve ambiguity. However, large-scale datasets that jointly capture sound events and acoustic scenes remain limited. We address this by augmenting AudioSet Strong with clip-level scene annotations under two taxonomies for joint scene-event modeling. A corpus-scale analysis shows that scene identity is better captured by events distinctive to a scene than by those that merely occur frequently. Reweighting event co-occurrences toward scene-specific associations produces representations that better align with human-derived taxonomies, while also exposing scenes where the acoustic and semantic views diverge. This geometric divergence predicts classification errors, while supervised validation confirms that the proposed scene annotations provide meaningful complementary information for developing scalable context-aware machine listening systems.
AM-41. MixCards: Evaluating Structured Descriptions of Multi-Source Sound Scenes
Sripathi Sridhar, Yujin Kim, Giovana Morais, Liang-Yuan Wu, Mehmet Atilay Kucukoglu, Prem Seetharaman, Lloyd May, Mark Cartwright, Magdalena Fuentes
New Jersey Institute of Technology, New York University, Adobe Systems, Monash University
Abstract
- Audio captioning benchmarks typically evaluate a single unstructured free-text description of an audio scene, providing limited insight into how a model fails to understand polyphonic audio. A plausible caption may omit a source, or conflate multiple sources into one, or assign an attribute of one source to another, and conventional captioning evaluation cannot locate these errors. We introduce MixCardBench, a benchmark for structured captioning of polyphonic audio, which evaluates models in two stages: first, identifying and temporally localizing the sources in a scene, and second, generating a structured MixCard describing the semantic, temporal, and acoustic properties of each source. The benchmark contains 4884 ecologically plausible synthetic soundscapes with up to eight overlapping speech, music and sound effects sources built from high-quality recordings and source-level annotations, where each source has a reference MixCard. We evaluate six state-of-the-art large audio language models and find substantial limitations in current models, which struggle considerably with speech and sound effects timing and detection, with the strongest model reaching only around 60% mean card-field score. Performance varies significantly across attributes, exposing capability differences that are hidden by aggregate caption scores. We assess the quality and ecological validity of MixCardBench through human validation, distribution matching and a listening study, and show that human corrections have little effect on model rankings, and our synthesized soundscapes are closer to in-the-wild YouTube videos than source guided but otherwise random synthesis.
AM-42. SoundTrace: What, When, and Where Sounds Occur in Videos
Mengyu Yang, Prem Seetharaman
Georgia Institute of Technology, Adobe Research
Abstract
- Real-world videos capture rich audio-visual scenes that highlight the close-knit associations between the two modalities. We define this as dense correspondences, and being able to identify them involves understanding the what, when, and where of a depicted audio-visual event. While foundational multimodal models, both open-source (e.g. Qwen3-Omni) and proprietary (e.g. Gemini Pro), demonstrate impressive audio-visual capabilities, it is largely limited to global understanding tasks like video captioning. Limitations still exist when these models are asked to capture more fine-grained details. To properly capture these dense correspondences, we introduce SoundTrace, a framework consisting of an automatic data annotation pipeline and a subsequent model capable of jointly describing, for each individual sound event within a mixture, what the sound is, when it occurred using explicit timestamps, and where it originates via bounding boxes. We present current results on benchmarks across a variety of tasks, including speaker attribution, sound localization, and timestamping and captioning accuracy.
AM-43. MRMAD: A Multi-Round Multi-Audio Benchmark for Evaluating Acoustic Degradation Perception in Large Audio-Language Models
Yize Li, Ningyuan Yang, Sile Yin, Sindhuja Thogarrati, Sung-En Chang, Andrew C. Singer, Xue Lin, Chuan-Che Huang, Shuo Zhang
Bose Corporation, Northeastern University, Stony Brook University
Abstract
- Large audio-language models (LALMs) have shown promising progress in understanding speech, music, and general sound events, yet their ability to reason about how audio signals are degraded remains underexplored. Existing benchmarks primarily evaluate semantic understanding, event recognition, or high-level audio reasoning, leaving a basic question unanswered: Do LALMs understand the differences in audio quality? We introduce MRMAD, a Multi-Round Multi-Audio Degradation benchmark for evaluating audio degradation perception and understanding in LALMs. MRMAD spans speech, music, and sound, and frames evaluation as multi-turn dialogues across multiple audio inputs, requiring models to identify types of degradation, compare severity, and perceive corruption changes across turns. Unlike current single-turn audio-language benchmarks, MRMAD evaluates whether LALMs can maintain consistent degradation hypotheses with new evidence and comprehend low-level acoustic phenomena over multi-turn dialogues. Through a systematic evaluation of 18 representative LALMs from non-thinking to reasoning and Omni models, we find that current models often recognize coarse content while failing to diagnose, compare, or reason about degradations reliably. Human evaluations further reveal a significant perception gap between LALMs and human listeners. MRMAD thus exposes a critical yet overlooked aspect of audio-language understanding and provides a diagnostic foundation for building future LALMs that are robust to real-world acoustic conditions.
AM-44. Efficient Text-to-Audio Generation via Pruning
Arshdeep Singh, Yi Yuan, Yun Chen, Wenwu Wang, Mark D. Plumbley
King’s College London (KCL), University of Surrey
Abstract
- Diffusion-based text-to-audio generative models such as AudioLDM achieve high perceptual quality and strong semantic consistency; however, their practical deployment is hindered by the substantial computational cost of the U-Net denoising backbone. In this work, we apply model pruning to improve the computational efficiency of AudioLDM, a U-Net-based text-conditioned audio latent diffusion model. We analyse parameter redundancy across U-Net convolutional blocks and evaluate a filter-pruning strategy. We find that pruning affects AudioLDM's ability to generate certain sound events including safety-critical sounds such as gunshots, sirens, and explosions, as well as mechanical sounds such as drills and sewing machines, and other sounds such as sprays and tick-tocks, which are mostly recovered by lightweight finetuning of the pruned model.
AM-45. Performance–Latency Trade-offs for Transformer and Mamba-Based Streaming Sound Event Detection
Dallin Gordon, Rachel Manzelli, Brian Kulis
Boston University, Modulate
Abstract
- Sound event detection (SED) on strongly labeled data requires identifying which sounds occur and exactly when, with onset and offset times for each event. The best current systems, such as ATST-F on RealDESED, are offline transformers that see the entire clip before predicting. We measure the cost of causality by presenting a causal Mamba-based model, built on a pretrained self-supervised audio Mamba encoder, that achieves 0.86 PSDS2 on the validation set with no lookahead, compared with 0.92 for our reproduction of the offline ATST-F transformer. Increasing the context of a sliding-window transformer requires keeping more past audio frames in memory and recomputing over all of them for each prediction. On a fixed compute budget, a sliding-window transformer implementation therefore faces a trade-off between frequency of prediction and performance: a wider window imposes a longer minimum time between predictions than a similar-sized Mamba-based model, which can throw away past data. A Mamba model is recurrent and only needs the most recent frames of audio to perform inference, with past context compressed into a fixed-size hidden state. We are benchmarking this trade-off across a range of compute budgets, comparing sliding-window transformers of varying context length against Mamba on both detection performance and how far predictions lag the incoming audio. We also evaluate the model on continuous streams, carrying its state across consecutive clips without resetting, to measure how long its performance holds when it runs indefinitely as a real-time detector would.
AM-46. Concept-Based Adversarial Audio: Query-Free Attacks on Audio Models via Their Public CLAP Dependency
Rasmus Torp, Manzi Fabrice Niyigaba, Adam Breuer
Dartmouth College
Abstract
- In this paper, we introduce query-free attacks that manipulate models across the diverse universe of audio AI by exploiting their common CLAP dependency, without accessing or querying them. Public Contrastive Language–Audio Pretraining (CLAP) models are widely reused as input audio encoders across otherwise independent systems for audio classification, generation, retrieval, captioning, and question answering. We study a query-free threat model where an adversary has white-box access only to the public CLAP checkpoints, but no access or queries to the target model and no knowledge of its task, architecture, parameters, or training data. We design two classes of attacks, both optimized entirely offline against the public checkpoints. First, we apply a small norm-bounded perturbation to the waveform to change CLAP's output using ensemble PGD, then test transfer to the unseen downstream system. Second, we introduce the first concept-based adversarial audio attack. These attacks generate replacement audio that stays within the same semantic concept as the original (e.g., sound of a slightly different gunshot; meow of a slightly different kitten) while being incorrectly encoded by CLAP. We show that both attacks transfer downstream, and concept attacks in particular transfer even to peripheral CLAP-family-dependent systems that share no public CLAP checkpoint parameters. These attacks can continue to survive under downsampling, low-pass filtering, MP3 compression, and over-the-air playback and re-recording on cheap consumer speakers/microphones outdoors amid heavy ambient noise in Hollywood’s recording district.

Poster Session 2

PM-1. Music Structural Representation Learning with Graph Neural Networks
Huiran Yu, Wei-Cheng Lin, Yiyang Wang, Zhiyao Duan
University of Rochester, Bosch LLC
Abstract
- Musical structures such as motifs, grooves, forms, and relational dependencies between events are fundamental to music analysis and generation. Learning representations that are sensitive to such structures remains a core challenge in symbolic music modeling. While recent sequence-based models have made good advances, they are designed to process linearized event streams and still struggle to model the rich structures underlying polyphonic music. In this work, we propose a Graph Neural Network (GNN)–based approach for symbolic music representation learning, where a sparse, interpretable graph among musical notes is learned through a masked node attribute reconstruction objective. Over this graph, note embeddings are also learned to carry musical structure information. We argue that this graph-based approach is natural and efficient in modeling simultaneous musical events and their relations in polyphonic music. We evaluate the learned note embeddings of our approach on multiple structure-required downstream tasks, including pattern matching, music segmentation, and harmony recognition. Results show that, with a significantly smaller model size, our approach achieves superior or competitive performance over state-of-the-art Transformer-based models. To facilitate future research on structure-aware music modeling, we additionally release a symbolic music dataset with fine-grained melodic pattern annotations.
PM-2. VIOLET: High-Fidelity Violin Synthesis with Techniques and Dynamics
Baotong Tian, Cynthia Lu, Vincent K.M. Cheung, Ting-Kang Wang, Jonathan Churchill, Zhiyao Duan
University of Rochester, Sony Computer Science Laboratories, National Taiwan University, Embertone
Abstract
- Neural synthesis for musical instruments has the potential to revolutionize current practices that use concatenative synthesis and a sample library. However, most research focused on piano synthesis and expressive performance generation; little work has been done on continuously articulated instruments like the violin, let alone rendering them with playing techniques and dynamics. We present VIOLET, a latent-diffusion framework for controllable violin synthesis, which uses a Diffusion Transformer (DiT) with rectified flow to synthesize high-fidelity audio from MIDI notes, playing techniques, and continuous dynamics. To train VIOLET, in addition to using a few existing datasets, we curate a new dataset named CSV-TD, which contains 39 h of 48 kHz synthetic audio and time-aligned annotations of MIDI notes, note-level techniques, and continuous dynamics curves. Objective and subjective evaluations show that VIOLET synthesizes violin performances with high technique adherence, accurate pitch and timing alignment, and good dynamics control. It outperforms the current state-of-the-art neural violin synthesis system and approaches a top commercial virtual instrument in terms of technique clarity, naturalness, and dynamics following.
PM-3. Scaling End-to-End Guitar Tablature Transcription
Frank Cwitkowitz, Randy Benitez, Zhiyao Duan
University of Rochester, ReWriter
Abstract
- Guitar Tablature Transcription (GTT) is a domain-specific music transcription task that aims to estimate the tablature corresponding to notes within a solo guitar recording, i.e., their string-fret combinations and playing techniques in addition to onsets and pitches. The primary bottleneck for this task is the lack of high-quality annotated data, typically obtained through poorly scalable methods involving guitars fitted with special devices (i.e., hexaphonic pickups), the application of monophonic pitch estimation algorithms, and manual correction. Previous work has shown the efficacy of using data synthesized from large-scale symbolic tablature datasets such as DadaGP for improving cross-dataset generalization. In this work, we refine these methods and re-render DadaGP through Guitar Pro’s native Realistic Sound Engine (RSE), which more faithfully synthesizes each score and its playing techniques. We pre-train a T5 transformer architecture on the synthetic data with a reduced DadaGP tokenization and fine-tune it on several smaller datasets of real recordings to demonstrate a proof-of-concept for end-to-end GTT with playing technique estimation.
PM-4. AIRPiano: Piano Score Following with Tabletop Hand Gestures
Jinxuan Zhu, Zhiyao Duan
University of Rochester
Abstract
- AIRPiano aims to turn the common practice of silently rehearsing piano music on a tabletop into sound. Guided by a known score, users make piano-like finger movements within a limited space without reproducing absolute keyboard positions. The score supplies pitches, while gestures control timing. We formulate this as visual score following, using a transformer with supervised attention to align hand motion with score events. We train on existing piano performance recordings, develop representations to facilitate transfer to tabletop gestures, and design a streaming inference framework for real-time interaction. We plan to present alignment evaluations on both piano and tabletop performances, examining the transfer from piano playing to tabletop gestures.
PM-5. End-to-end Historical Music Restoration in Latent Space
Steven Cho, Junghyun Koo, Tushar Dhyani, Raphael Lafargue, Eloi Moliner, Yuki Mitsufuji
ETH Zurich, Sony Corporate Technology Center, Sony Europe Ltd, Aalto University, Sony Group Corporation
Abstract
- Historical music restoration (HMR) has almost exclusively focused on constrained problems such as Super-Resolution or the restoration of solo pieces, under-exploring the general task of restoring orchestral historical music, which has multiple instruments. This under-exploration is largely because the HMR domain, early-20th-century recordings, has no pre-degradation ground-truth pairs, making the restoration task unsupervised and more challenging. This paper presents a supervised end-to-end orchestral HMR benchmark by exploring both the synthetic degradation functions and the end-to-end generative deep-learning restoration methods. We simulate the historical recording degradation chain more faithfully than prior work, which makes orchestral restoration into a tractable supervised problem. A latent flow-matching model trained on the resulting synthetic pairs outperforms existing HMR baselines on intrusive, non-intrusive, and subjective evaluations. We also curate and release a 9.3-hour license-free, unpaired, historical classical-music test set, along with code and audio demos.
PM-6. D-Composer: Language Modeling for Simultaneous Drum Transcription and Sound Event Decomposition
Patrick O'Reilly, Saumya Pailwan, Annie Chu, Jason Smith, Bryan Pardo
MIT, Northwestern University
Abstract
- Drum beats are often conceptualized and produced as arrangements of repeated single-drum sound events or one-shots. Drum Analysis-by-Synthesis (AbS) models combine this structural assumption with reconstruction-based training to perform MIR tasks such as drum transcription, drum source separation, and one-shot extraction, offering a flexible alternative to more traditional task-specific supervision. However, leading AbS approaches struggle to generalize to a broad range of drum timbres (e.g., electronic drum kits), rely on high-quality paired audio-transcription training data, and do not model intra-class timbral variation within a single arrangement (e.g., multiple kick sounds). To overcome these limitations, we propose D-Composer, an autoregressive language modeling framework that decomposes drum recordings into labeled sound events and their onset times. In drum transcription and one-shot extraction tasks, D-Composer performs comparably to or better than a state-of-the-art AbS system and task-specific baselines while generalizing to a broader range of drum timbres. Our results demonstrate that when equipped with abundant synthetic data and compact acoustic token representations, language modeling provides a simple recipe for unifying transcription and sound event decomposition.
PM-7. AI-tribution: Enabling Training Data Attribution for Users of Generative Music Models
Julia Barnett, Nathan Pruyne, Nicholas Diakopoulos, and Bryan Pardo
Apple, Carnegie Mellon University, Northwestern University
Abstract
- Generative music models are growing in popularity, but users remain in the dark about the influences on their supposedly novel creations. Existing systems provide no mechanisms to identify highly similar songs or to help users learn how model outputs are influenced by training data. We present AI-tribution: a tool designed to inform users of generative music models about the training songs most similar to their generated outputs. We detail the interface, methodology, and a user study ($n=26$) where musicians, media makers, and music consumers interact with the tool in 30-minute sessions. We analyze how the tool changes user knowledge of generated outputs, examine differences across user groups, and qualitatively analyze users' attitudes towards the tool. We find an overwhelmingly positive reception, show how the tool encourages deeper engagement with influences, and shed light on how different groups would use it in practice.
PM-8. Local Multimodal Music Alignment from Global Supervision
Irmak Bukey, Zachary Novack, Jongmin Jung, Dasaem Jeong, Chris Donahue
Carnegie Mellon University, University of California San Diego, Neutune, Sogang University
Abstract
- Understanding music requires understanding localized relationships across data modalities, e.g., how time in performance audio maps onto position in a score image. Yet supervision for such local correspondences is difficult to obtain in practice, we often only have access to coarser global supervision like paired segments of audio and images. To address this gap, we propose FuSiLi (Fused Sinkhorn-Localized Similarity), a similarity score for multimodal contrastive learning operating directly on local image patch and audio frame features via Sinkhorn-based soft alignment. We show that FuSiLi (i) effectively learns local relationships, (ii) requires only global supervision, and (iii) retains the global alignment capabilities of conventional contrastive approaches. We fine-tune pretrained CLIP and CLAP encoders on pairs of raw sheet music images and audio using a hybrid contrastive objective combining FuSiLi with conventional global similarity. We evaluate on cross-modal retrieval and frame-level alignment tasks against a range of global and local baselines, showing that our approach outperforms them on local alignment while remaining competitive on retrieval.
PM-9. Decomposer: Learning to Decompile Symbolic Music to Programs
Yewon Kim, Apurva Gandhi, David Chung, Graham Neubig, Chris Donahue
Carnegie Mellon University
Abstract
- Musical performance involves executing a set of high-level musical instructions, yet recovering those instructions from the performance is a challenging inverse problem. We present Decomposer, a post-training framework for symbolic music decompilation: the task of recovering executable, editable music programs from symbolic music. We instantiate the task as MIDI-to-Strudel decompilation, where the model takes symbolic MIDI as input and produces a program in Strudel, a music programming language, that reconstructs the input when executed. The task poses two challenges: Strudel is a low-resource language with little naturally paired MIDI-code data, and optimizing faithful reconstruction of MIDI alone can collapse to unreadable note-by-note transliteration. We address these challenges in two stages. First, we construct Strudel-Synth, a synthetic corpus of paired Strudel programs and rendered MIDI, and use it for supervised fine-tuning. Second, we refine the model with reinforcement learning on unpaired MIDI, optimizing rewards for both MIDI reconstruction faithfulness and code readability. Our evaluation across synthetic and real-world MIDI benchmarks shows that Decomposer achieves substantially higher MIDI reconstruction faithfulness than closed-source LLMs while producing more readable and diverse code than the heuristic converter.
PM-10. Magic Quill - A Real-time Score Engraving System for Live MIDI Performances
Lynn Ye, Roger Dannenberg, Chris Donahue
Carnegie Mellon University
Abstract
- We present Magic Quill, a browser-based system that visualizes live MIDI performances as an evolving engraved score. An online beat tracker produces provisional notation immediately, while an asynchronous offline model revises earlier measures as additional musical context becomes available. On consumer hardware, the online model requires approximately 0.13 ms per note and browser-based online beat-tracking inference completes within approximately 10 ms; offline refinement arrives after several seconds and achieves competitive transcription accuracy using substantially less context than prior offline systems. A demonstration of our proposed system is available at https://magicquill.github.io.
PM-11. Interpretable Destination-Aware Synthesizer Modulation Recovery
David Liu, Giulio Cengarle, David Cooper, Mark Vinton, Haici Yang
Dolby Laboratories, Boston University
Abstract
- Synthesizer programming is a challenging task, particularly in configuring modulation: the time-varying control of parameters such as pitch, filter cutoff, and oscillator level by modulators such as Low-Frequency Oscillators (LFOs). Prior work has shown that modulation can be reconstructed from the clean audio through curve matching, yet the recovered parameters are often untransferable to modern synthesizers. We present an interpretable destination-aware modulation and waveform recovery pipeline that mirrors how musicians often recreate sounds on a synthesizer. First, the system identifies the modulated destinations; then it recovers the LFO shape associated with each destination, and estimates the oscillator waveform to match the source timbre. The training is supported by a differentiable synthesizer that incorporates a noise oscillator, and more modulation options than previous work. We carefully train the models for optimal perceptual quality, with extensive exploration of perceptual losses and adversarial training. Through objective and subjective evaluation, we show that predicting the destinations correctly is essential, our model excels in the modulation focused inverse synthesis tasks, and our Gammatone loss and CQT discriminator significantly outperform the traditionally-used MSS loss in improving perceptual quality.
PM-12. BACHI: Boundary-Aware Symbolic Chord Recognition Through Masked Iterative Decoding on Pop and Classical Music
Mingyang Yao, Ke Chen, Shlomo Dubnov, Taylor Berg-Kirkpatrick
University of California, San Diego
Abstract
- Automatic chord recognition (ACR) via deep learning models has gradually achieved promising recognition accuracy, yet two key challenges remain. First, prior work has primarily focused on audio-domain ACR, while symbolic music (e.g., score) ACR has received limited attention due to data scarcity. Second, existing methods still overlook strategies that are aligned with human music analytical practices. To address these challenges, we make two contributions: (1) we introduce POP909-CL, an enhanced version of POP909 dataset with tempo-aligned content and human-corrected labels of chords, beats, keys, and time signatures; and (2) we propose BACHI, a symbolic chord recognition model that decomposes the task into different decision steps, namely boundary detection and iterative ranking of chord root, quality, and bass (inversion). This mechanism mirrors the human ear-training practices. Experiments demonstrate that BACHI achieves state-of-the-art chord recognition performance on both classical and pop music benchmarks, with ablation studies validating the effectiveness of each module.
PM-13. Harmonic-Percussive Disentangled Neural Audio Codec for Bandwidth Extension
Benoît Giniès, Xiaoyu Bie, Olivier Fercoq, Gaël Richard
Télécom Paris
Abstract
- In this poster, we will present an approach for bandwidth extension framed as an audio token prediction problem. Specifically, we train a transformer-based language model on the discrete representations produced by a disentangled neural audio codec, where the disentanglement is guided by a Harmonic–Percussive decomposition of the input signals, highlighting spectral structures particularly relevant for bandwidth extension. Our approach introduces a novel codec design that explicitly accounts for the downstream token prediction task. The results obtained highlight the importance of aligning codec disentanglement and representation learning with the generative modeling stage, and demonstrate the potential of global, representation-aware design for advancing bandwidth extension.
PM-14. RIME: Enabling Large-Scale Agentic Music Post-Production
Noah Schaffer, Nikhil Singh
Dartmouth College
Abstract
- Almost every piece of recorded music you have ever heard was modified before it reached you; commercial releases rarely spring fully formed from a musician's mind. Despite the promise of music generation models for one-shot output, such fine-grained iterative refinement workflows are a complementary problem, and largely out of their reach. There is also a gap for musicians: while they can express what they want to hear, not all have the facility with studio production tools to implement the complex set of actions needed to realize these intuitions. We formalize this task as agentic post-production, wherein individual aspects of a track are targeted, refined, and combined into a final version. We argue the bottleneck is data: existing corpora do not reflect how realistic post-production chains map onto the vocabulary musicians and engineers actually use. We observe that there is a language for modifying recorded music that is dense, consistent, and learnable, and to leverage this we introduce the Rule-Based Instructions for Music Editing (RIME) framework, which generates realistic paired edit-instruction data from any baseline music dataset grounded in canonical methods, design patterns, and constraints derived from real production workflows. RIME leverages POEMS, a new toolkit that combines stem separation, mixing, and common studio effects for use by multimodal agents. We use POEMS and RIME to generate 15,000 pairs of edit instructions and ground-truth audio, and use this data to evaluate existing multimodal LLMs as agents on this task, showing persistent challenges in current models' post-production capabilities. We also demonstrate RIME's ability to improve post-production agent performance via supervised fine-tuning using synthetic data generation. We see RIME as an early step toward iterative musical agents, collaborative systems that could transform music production much as interactive coding agents have reshaped software engineering. We include all code for RIME at: https://anonymous.4open.science/r/RIME-0116/.
PM-15. PLACE: Positional Latent Adaptation via Conditioned Embeddings for Binaural Audio Generation
Tiernon Riesenmy, You (Neil) Zhang, Gautam Bhattacharya, Andrea Fanelli
Georgia Institute of Technology, Dolby Laboratories
Abstract
- We present PLACE, a method that extends the pretrained any-to-audio model AudioX for binaural generation from arbitrary combinations of text, video, and optional audio prompts. PLACE augments video conditioning with Perception Encoder Core features, aligns text and video representations to derive spatial cues, and applies a conditioning-dependent low-rank transformation to the generated latent. The adapter is supervised via decoded-audio interaural level and time difference objectives. Trained on MRSAudio, PLACE improves most metrics over ViSAGe on FAIR-Play and achieves an improved SpatialCLAP score over SpatialSonic on the BEWO-1M Single Static test split. Listener evaluations favor PLACE for video-to-audio and out-of-distribution text-to-audio generation, demonstrating flexible multimodal control and improved spatial consistency.
PM-16. Learning Floor-Map-Conditioned Priors for Sparse Acoustic Field Reconstruction
Harshvardhan Takawale, Gautam Bhattacharya, Nirupam Roy, Avery Bruni
University of Maryland, College Park, Dolby Laboratories
Abstract
- Many spatial audio applications, such as rendering AR/VR acoustics or room correction, require knowledge of the sound field throughout a room, yet measuring it densely takes hundreds of microphone positions. With only a few microphones, many different fields fit the same measurements. Floor plans are often available and constrain the field before any measurement is taken. However, floor plans describe only the walls and doorways, not the furniture and materials that also shape the acoustic field. We therefore learn a distribution of sound fields consistent with a floor plan, using a flow-based generative prior that maps a cheap image-source field, computed from the floor plan alone, to high-fidelity wave-simulated fields. Given a few measurements, posterior sampling recovers the field of a specific room. Because the model is generative, the variance across posterior samples gives an uncertainty map along with the reconstruction. This map tracks the reconstruction error, so it can indicate where to place the next microphones. We evaluate on real multi-room apartment floor plans with wave-simulated acoustics. On floor plans unseen during training, our method reduces average reconstruction error by 5.5 dB with 128 microphones relative to kernel interpolation given the same floor-plan information, and the gain grows as microphones are added. With 16 microphones, it is at least as accurate as the same baseline with 64. Active sampling based on posterior uncertainty improves reconstruction by up to 0.7 dB over spatially uniform placement, reaching the same accuracy with 15% fewer microphones.
PM-17. Active Reverb Control with Flat-panel Loudspeakers
Ryan Bowering, Mark Bocko, Michael Heilemann, Zhiyao Duan
University of Rochester
Abstract
- Controlling room acoustics requires effective management of sound reflections. Current approaches rely on passive acoustic treatments or modifying room geometry, which can be costly and difficult to adapt. Active acoustic control is a flexible alternative that uses loudspeakers and microphones to modify sound fields in real time. Flat-panel loudspeakers are an appealing choice for scalable active acoustic control because their thin form factor and low manufacturing cost make them easy to integrate into walls, ceilings, and other surfaces. This work investigates real-time cancellation of reflections from a single flat-panel loudspeaker using the Filtered-x Least Mean Squares (FxLMS) algorithm. A reference microphone measures the incident sound from a source loudspeaker, and an error microphone measures the residual reflections from the panel. The controller is implemented on an Analog Devices SHARC DSP board for low-latency processing. System performance is evaluated by measuring the reduction in reflected sound level relative to the uncontrolled case.
PM-18. Perceptual Acoustic Parameter Fields from Point-Cloud Scans for Multi-Source Rendering
Yi Wu, Purnima Kamath, Agnieszka Roginska, Magdalena Fuentes
New York University
Abstract
- A field of acoustic parameters over a room describes how the room sounds from every position. Interactive audio systems render from such fields, which are obtained through hours of simulation using a material-annotated mesh that is unavailable for a newly scanned room. We predict the field from the scan instead. Given a semi-dense point cloud and a listener position, a convolutional network predicts six broadband perceptual parameters over a grid of source positions: onset time, early and late decay times, and direct, early, and late energy levels. One field per listener position supplies parameters for all sources to a parametric renderer, with source and listener motion handled through lookups. On held-out scenes, when compared with a room-average baseline computed from the simulated reference fields, our approach is more accurate in estimating onset, early decay, and direct and early levels. For reverberation time, which varies little within a room, the room average is more accurate than our prediction, while the two are comparable for the late energy level. In a listening test against the simulated reference response, renderings from our predicted fields were rated well above room-based reverb presets representative of current game and XR audio practice. Ratings were also equivalent whether the renderer used our predicted fields or the reference fields.
PM-19. Reconstructing the Vocal Tract with Differentiable Acoustic Simulation
Eric Ming Chen, Jin Woo Lee, Vincent Sitzmann
MIT
Abstract
- The vocal tract is the region of the human body responsible for filtering one's voice to create speech. However, creating intuitive models of how the geometry of the vocal tract affects speech remains a challenge. Little paired data between vocal tract geometry and its produced speech exist. To address this problem, this thesis presents a differentiable and GPU accelerated acoustic simulator for the vocal tract. The differentiable simulator synthesizes speech by propagating sound along an acoustic tube model of the vocal tract, and via its gradients, can solve the inverse problem: reconstructing the shape of the vocal tract solely from the sound it produces. Although learning the inverse mapping between geometry and sound is notoriously non-convex, we discover that gradient descent succeeds with three technical contributions: (1) we design a frequency domain formulation of the vocal tract's fluid dynamics that is 70x more GPU parallelizable than finite differences in time, (2) we integrate a differentiable model for turbulence to synthesize consonants, and (3) similar to prior work in implicit neural representations (INRs) and neural fields, we find that parameterizing the geometry with a neural network accelerates convergence and escapes local minima that trap discrete representations. Because the simulator is differentiable, it may be readily integrated with other deep learning pipelines to enable novel speech and medical imaging applications. First, we demonstrate self-supervised autoencoding of vocal tract shapes across 11 languages. Second, we couple our simulator with a generative model of MRI (magnetic resonance imaging) images to reconstruct one's vocal tract from only their speech, no paired data required.
PM-20. From Numbers to Perception: Multi-Band Energy Decay Curve Prediction
Imran Muhammad, Gerald Schuller
Applied Media Systems, Technische Universität Ilmenau
Abstract
- Predicting full room impulse responses (RIRs) is costly and high-dimensional. We instead predict multi-band energy decay curves (EDCs) directly from room geometry, source–receiver positions, and frequency-dependent absorption, then reconstruct a listening-ready RIR. Building on our previous LSTM (SANE 2025), a physically constrained 1D-ConvNet decoder cuts the model from ~90 M to 9 M parameters and estimates 24 one-third-octave bands (100 Hz–20 kHz). A log-domain loss with a slope penalty suppresses non-physical “staircase” artifacts and keeps decays monotonic. On 6,000 simulated shoebox rooms, T30 reaches R^2 = 0.90 with MAE 0.05 s, mostly within the 5% JND. RIRs are recovered by differentiating the EDC and assigning polarity with a random sign-sticky scheme. Code, weights, and data scripts: https://github.com/TUIlmenauAMS/LSTM-Model-Energy-Decay-Curves.
PM-21. FA-Bench: A Benchmark for Phone- and Word-Level Timestamp Accuracy in Forced Alignment and ASR on Clean and Noisy Speech
Wei Chu, Yuanzhe Dong, Ke Tan, Dong Han, Yichao Zhou, Ruchao Fan, Bingshen Mu, Jingbei Li, Vishwas Shetty, Sarthak Bisht, Ziyue Qiu, Massa Baali, Rita Singh, Bhiksha Raj
Olewave, Stanford University, Cartesia, Microsoft, Northwestern Polytechnical University, University of California, Los Angeles, Tsinghua University, Carnegie Mellon University
Abstract
- Forced alignment estimates the timestamps of each word, phone or character in speech given its transcript. Published comparisons normalize transcripts, split the data and match boundaries differently, so their numbers cannot be read together. We present FA-Bench, an open framework that fixes those choices once and releases the code, splits, phone mapping, text normalization and scoring script, with results published periodically. Track 1 gives every aligner the reference transcript and Track 2 gives it a recognizer's output, on the same audio, clean and degraded four ways, with 21 open models and 9 commercial APIs under a unified protocol. We score every boundary of an utterance and check the two labels beside it, so a word the recognizer missed or invented is charged. Using a tolerance-based F1 as our primary metric eliminates the 9% to 14% score inflation that standard MAE causes on recognition-dependent systems in conversational speech. We then group boundaries by position and how many adjacent words were recognized correctly, which shows where a system lost the score. We discovered systematic bias in how current systems time words, with Whisper about 150 ms early and several commercial ASR APIs over 50 ms late. Code and results are at this https URL.
PM-22. CoLMbo-SV: A Grounded Language Model for Explainable Speaker Verification
Massa Baali, Sarthak Bisht, Ziyue Qiu, Joseph Konan, Rita Singh, Bhiksha Raj
Carnegie Mellon University
Abstract
- Speaker verification systems achieve high accuracy but provide little account of the acoustic evidence behind their judgments. Making these systems inspectable requires exposing interpretable evidence while retaining the richer information on which their decisions depend. We present CoLMbo-SV, a speaker language model that combines strong speaker discrimination with structured, acoustically grounded comparison reports. By connecting a pretrained speaker encoder to a language model and supplying explicit acoustic measurements, CoLMbo-SV makes voice comparisons inspectable without restricting verification to the evidence verbalized in its reports. We additionally introduce VoxReason, paired recordings with measured acoustic properties and comparison reports filtered through numerical and qualitative checks, providing supervision for this combined capability. We also develop an evaluation framework that separates what acoustic information a speaker representation encodes, what influences the verification score, and what the generated report discusses. On VoxCeleb1-O, CoLMbo-SV achieves 0.99% EER, reducing verification error by approximately 80% relative to the strongest audio-language baseline fine-tuned on VoxReason, while attaining a numerical-grounding score of 0.82. Our analysis further demonstrates that acoustic correctness and decision relevance are distinct properties of an explanation, exposing a gap that numerical-grounding metrics miss. Together, these contributions substantially advance audio-language speaker verification, bring its accuracy toward that of dedicated speaker encoders while adding checkable acoustic reporting, and establish an empirical framework for connecting natural-language explanations to the decisions they explain.
PM-23. AffectDF: The Most Comprehensive Benchmark for Speech Deepfake Detection against Emotionally Expressive Attacks
Aurosweta Mahapatra, Xiutian Zhao, Shreeram Suresh Chandra, Zihan Zhang, Zongyang Du, Ismail Rasim Ulgen, Kong Aik Lee, Nicholas Andrews, Carlos Busso, Berrak Sisman
Johns Hopkins University, Hong Kong Polytechnic University, Carnegie Mellon University
Abstract
- AffectDF is the first comprehensive benchmark designed to evaluate speech deepfake detectors at scale under emotionally expressive spoofing conditions, with broad variation across attack types, emotional states, and speaking styles. It introduces paralinguistic variability that is largely absent from conventional benchmarks, which typically focus on neutral speech and other challenging factors such as channel variation, adversarial perturbations, and environmental noise. By evaluating detectors across both conventional and emotionally expressive benchmarks, AffectDF enables systematic analysis of cross-domain generalization and reveals whether current systems rely on dataset-specific artifacts or learn transferable spoofing cues. These findings underscore the need for models that can generalize robustly across both conventional and expressive speech conditions.
PM-24. DiffAnon: Diffusion-based Prosody Control for Voice Anonymization
Ismail Rasim Ulgen, Zexin Cai, Nicholas Andrews, Philipp Koehn, Berrak Sisman
Johns Hopkins University, Oracle
Abstract
- To preserve or not to preserve prosody is a central question in voice anonymization. Prosody conveys meaning and affect, yet is tightly coupled with speaker identity. Existing methods either discard prosody for privacy or lack a principled mechanism to control the utility-privacy trade-off, operating at fixed design points. We propose DiffAnon, a diffusion-based anonymization method with classifier-free guidance (CFG) that provides explicit, continuous inference-time control over prosody preservation. DiffAnon refines acoustic detail over semantic embeddings of an RVQ codec, enabling smooth interpolation between anonymization strength and prosodic fidelity within a single model. To the best of our knowledge, it is the first voice anonymization framework to provide structured, interpolatable inference-time prosody control. Experiments demonstrate structured trade-off behavior, achieving strong utility while maintaining competitive privacy across controllable operating points.
PM-25. Cross-Codec Transfer of Pretrained Speech Language Models
Junhyeok Lee, Jihwan Lee, Xiluo He, Minu Kim, Shrikanth Narayanan, Jesús Villalba, Najim Dehak
Johns Hopkins University, University of Southern California
Abstract
- Neural audio codecs define the token vocabulary, frame rate, and codebook structure used by speech language models, but whether pretrained models can adapt to a different codec remains largely unexplored. We investigate cross-codec transfer in Qwen3-TTS and Moshi, motivated by the Platonic Representation Hypothesis, that independently trained codecs may encode compatible acoustic information. For Qwen3-TTS, direct fine-tuning with 50 Hz JHCodec and DAC streams fails, whereas stacking four consecutive frames to restore its pretrained 12.5 Hz interface enables transfer, even with 24 of 28 Transformer layers frozen. For the spoken dialogue model Moshi, replacing its Mimi codec with the same-rate Qwen3-TTS-Tokenizer while freezing 24 of 32 temporal layers retains part of its spoken question-answering ability. These results demonstrate that pretrained speech language models can be reused across codec vocabularies, while highlighting the importance of temporal alignment and the limitations of matching frame rate alone.
PM-26. Efficient Conversational Multi-Talker ASR with Predicted Speaker-Agnostic Activity Streams
Xiluo He, Magdalena Rybicka, Laureano Moro-Velazquez, Jesús Villalba, Thomas Thebaud, and Najim Dehak
Johns Hopkins University, AGH University of Krakow, Universidad Carlos III de Madrid, Agence Ministerielle pour l'Intelligence Artificielle de Defense
Abstract
- Multi-talker automatic speech recognition (ASR) can be performed by conditioning a single-speaker recognizer on frame-level speaker activity. However, this approach requires a complete diarization pass to obtain the conditioning signals and one ASR pass per speaker, making both the frontend and recognition costs dependent on the number of speakers. We reduce these costs by using Heuristic Error Assignment Training (HEAT) to merge the activities of an arbitrary number of speakers into two speaker-agnostic streams. Conditioning on these streams limits recognition to two ASR passes regardless of the number of participants. Through an analysis of the resulting encoder representations, we show that under a stream-assignment heuristic suited to conversational speech, the recognizer represents a stream target much as it does a speaker target. Because the streams do not encode persistent speaker identities, we further replace the conventional sliding-window diarization and clustering frontend with a model that predicts both activity streams directly from the audio in a single pass. A combination of arrival-order and permutation-invariant training objectives over the two-stream powerset labels prevents the predictor from collapsing to identical streams. We also investigate post-hoc speaker attribution by matching decoded utterances to speaker activities within their assigned streams. Experiments on corpora with diverse conversational dynamics demonstrate a favorable efficiency--accuracy tradeoff. On AMI and ICSI, two-stream conditioning is respectively 44% and 87% faster than per-speaker conditioning using the same diarization frontend, with less than 2% relative WER degradation. Direct activity prediction increases these speedups to 79% and 132%, with relative WER degradations of 10% and 21%, respectively.
PM-27. Pretrained ASR Pseudo-labeling for Noisy Police Audio
Kaavya Chaparala, Su Huang, Stephen L. Morgan, Rhiannon N. Miller, Anjalie Field
Johns Hopkins University, Providence College
Abstract
- Pretrained ASR systems perform poorly on noisy Broadcast Police Communication (BPC), hindering efforts to understand police decision-making. Pseudo-labeling offers an unsupervised path to improve ASR without expensive human labels, but the efficacy of this approach on very noisy domains is not known. In this work, we systematically assess the opportunities and limits of pseudo-labeling to adapt foundation ASR models (Whisper and Qwen3-ASR) to noisy BPC domain corpora from Baltimore and Chicago. We demonstrate that existing internal confidence metrics (log-probabilities and STAR scores) fail to distinguish between high and low quality BPC pseudo-labels, and we introduce an external LLM-as-a-judge filtering paradigm that leverages parametric knowledge to discard contextually implausible transcripts. Our LLM-judging filters more aggressively than internal metrics and significantly reduces WER of the pseudo-labeled training sets across the Baltimore and Chicago BPC corpora, though a substantial gap remains relative to an oracle filter. We also introduce a new cross-model pseudo-labeling paradigm where one model is finetuned with pseudo-labels from the other, and we identify this method as a promising direction for future pseudo-labeling work.
PM-28. Beyond Intelligibility: Measuring Listening Effort in Multi-Talker Listening Using Behavior and Physiology
Sudan Duwadi, Yasaa Mohammad, Aneesa Diya Pathiyaparambath, David A. Boas, Meryem A. Yücel, Scott Isabelle, Elin Roverud, Virginia Best, Kamal Sen
Boston University, Amazon
Abstract
- The cocktail party problem (CPP) refers to the challenge that arises in crowded situations, where a listener must selectively attend to one talker among multiple competing talkers to understand a message of interest (Cherry, 1953; McDermott, 2009). This challenge varies with the target-to-masker ratio (TMR), which describes the difference in intensity between the attended speech (target) and the ignored speech (maskers). The CPP is most commonly quantified in terms of target speech intelligibility, measured here as the accuracy with which listeners report the target words, but it is also clear that following one talker while ignoring others requires cognitive resources, which manifests as listening effort in the CPP (Peelle, 2018). Certain hearing technologies, such as spatially selective algorithms, can improve the TMR in cocktail party environments, and thus have the potential to improve speech intelligibility and reduce listening effort. Here we propose to investigate the relationship between TMR, speech intelligibility, and listening effort in the CPP using whole-head high-density functional near-infrared spectroscopy (fNIRS) and pupillometry. The broad goal is to identify objective physiological correlates that capture a listener’s individual experience of the CPP and can be used to evaluate the real-world benefit of hearing technologies. Our experimental design follows a three-talker setup in which the target speaker is always directly in front of the participant and the maskers are at 30° to the left and right. Participants have their head stabilized with a chin rest and are instructed to always fixate on a fixation cross at the center of a frontally located monitor. There are 120 trials in total, where each trial presents three five-word sentences from a matrix-style corpus (Sensimetrics, 2014). Of the 120 trials, 60 are processed through a biologically oriented sound segregation algorithm (BOSSA) (Chou et al., 2022), which processes a given multi-talker mixture and isolates the speech from a desired location (center location in this case). These 60 trials are further divided between three TMRs (−5, 0, and 5 dB). The trials are presented in random order for each participant. Participants are asked to report the five words spoken by the target talker, by selecting from an on-screen word grid, and to provide a listening-effort rating using the 13-point ACALES scale (Krueger et al., 2017). During the behavioral task, physiological responses are recorded continuously using an in-house built whole-head high-density fNIRS system (O’Brien et al., 2024) and an eyetracker (Neon, Pupil Labs GmbH, Berlin, Germany). We measure oxygenated hemoglobin concentration in relevant brain areas such as premotor cortex and dorsolateral prefrontal cortex (Peelle, 2018), and pupil size as an indicator of listening effort (Choudhari et al., 2026). Our results reveal how behavioral and physiological measures vary with TMR and with BOSSA processing.
PM-29. Quieter Than the Room: Representation Drift and Task Robustness in Speech Encoders
Vsevolod Kovalev, Pranay Manocha
Boston University, Princeton University
Abstract
- Speech encoders are often evaluated by downstream task scores. Quiet non-speech sound can change the representation even when these scores change little. We study this response near and below a recording’s background by comparing embedding drift with controlled speech variation. Repeated takes and delivery changes in RAVDESS, with words and speaker fixed, define an encoder-specific reference corridor. Continuous interference in pauses reaches the repeat-take reference 24–34 dB below the estimated RAVDESS background on five self-supervised encoders, with changes extending into speech-frame representations. Across seven sounds added throughout each recording, the SNR at which drift reaches the delivery reference correlates with degradation on four tasks at mean Spearman 0.82-0.88. For white noise at 10 dB SNR, however, pause placement produces larger pooled drift on FSC, while speech placement degrades all four tasks more. The relationship between drift and task degradation therefore depends on placement: larger shifts can accompany smaller task losses.
PM-30. SpeechCritic: Learning a Diagnostic Speech Judge from Limited Human Preferences
Mingyue Huo, Shivam Mehta, Bhavin Jawade, Yinghong Lan, Haoqi Li
University of Illinois Urbana-Champaign, Netflix
Abstract
- Human speech conveys rich perceptual information, such as emotion and speaker identity, yet most automatic speech quality judges reduce it to a single naturalness score. We study diagnostic speech judges: given two candidates, a diagnostic judge decides which is better, along which perceptual dimensions (e.g., timbre, emotion, timing) they differ, and which audible cues support its decision. Learning such judges is challenging: expert annotation is costly, and simply prompting a frontier audio-language model to produce labels is unreliable: our probing reveals substantial errors and unstable instruction following. We introduce SpeechCritic, which learns a diagnostic judge in a reference-conditioned cross-lingual setting from only about 300 human-labeled comparisons. Rather than replacing the frontier model, SpeechCritic calibrates it with these labels: for each dimension, it selects the acoustic measurements that agree with human judgments, maps them to A/Tie/B probabilities, and passes these to the model as non-binding hints alongside the audio. Compared with the same model labeling without hints, this raises dimension-level agreement with humans by 6.3 points and cuts the mismatch with human Tie rates by 10.4 points. We then train a 7B judge on this supervision and find that different training signals shape different judge behaviors: SFT establishes the task, OPD transfers the teacher's dimension-level strengths and weaknesses, and RL helps most on clear-cut comparisons where human raters agree. Notably, human listeners also find that RL makes rationales cite more specific, localized acoustic cues, although it never directly rewards rationale text. Finally, we show that the pipeline is language-pair agnostic by instantiating it on both English-Japanese and English-Spanish. Together, these results demonstrate a path from limited human preferences to a diagnostic speech judge.
PM-31. Something from Nothing: Data Augmentation for Robust Severity Level Estimation of Dysarthric Speech
Jaesung Bae, Xiuwen Zheng, Minje Kim, Chang D. Yoo, Mark Hasegawa-Johnson
University of Illinois Urbana-Champaign, Korea Advanced Institute of Science and Technology (KAIST)
Abstract
- Dysarthric speech quality assessment (DSQA) is critical for clinical diagnostics and inclusive speech technologies. However, subjective evaluation is costly and difficult to scale, and the scarcity of labeled data limits robust objective modeling. To address this, we propose a three-stage framework that leverages unlabeled dysarthric speech and large-scale typical speech datasets to scale training. A teacher model first generates pseudo-labels for unlabeled samples, followed by weakly supervised pretraining using a label-aware contrastive learning strategy that exposes the model to diverse speakers and acoustic conditions. The pretrained model is then fine-tuned for the downstream DSQA task. Experiments on five unseen datasets spanning multiple etiologies and languages demonstrate the robustness of our approach. Our Whisper-based baseline significantly outperforms SOTA DSQA predictors such as SpICE, and the full framework achieves an average SRCC of 0.761 across unseen test datasets.
PM-32. Variable-Length Non-Autoregressive Speech-to-Speech Translation, With and Without a Pretrained Diffusion Language Model
Aniket Tathe, Zhongweiyang Xu, Shinji Watanabe
University of Illinois Urbana-Champaign, Carnegie Mellon University
Abstract
- Non-autoregressive generative approaches, such as masked diffusion language models and flow matching, have gained considerable traction in recent years: masked diffusion for text generation and code infilling, and flow matching for image generation and text-to-speech synthesis. Yet they remain rarely explored for speech-to-speech translation. A central obstacle is output length. Source and target utterances differ in length, and the relationship varies across language pairs, whereas in their standard form these models fill a canvas of output positions whose size must be fixed before decoding begins. In this work, we present several techniques for handling variable-length output in non-autoregressive speech-to-speech translation, adapting methods originally developed for other domains, such as code infilling and molecule generation. We study two settings: a large pretrained masked diffusion language model, LLaDA, together with variable-length extensions of it, and a much smaller model based on Branching Flows that does not rely on a large language model. All systems are trained and evaluated on the CVSS-C French--English benchmark. The Branching Flows model reaches 24.19 ASR-BLEU, slightly above the discrete diffusion-based model Dub-S2ST (23.35).
PM-33. Conversational Voice Aesthetic Model with Reinforcement Learning from Human Listeners
Xilin Jiang, Shun Zhang, Tejas Jayashankar, Yinghao Aaron Li, Osama Hanna
Columbia University, Meta
Abstract
- We introduce Conversational Voice Aesthetic Model, a speech large language model for describing the voice aesthetics of real or synthetic speech responses in natural conversational contexts. Given a context and a response speech, CVAM describes salient moments that characterize the voice and predicts nine categorical attributes spanning gender, pitch, pacing, emotion, and delivery. The key challenge lies in perceptual fields such as emotion and delivery, which are inherently subjective and lack definitive ground truth. Therefore, we collect ~10 human annotations for each of 3k real and synthetic responses derived from the CANDOR corpus. CVAM is supervised finetuned on synthesized aesthetic descriptions and labels, then optimized with Group Relative Policy Optimization on human judgments. Experiments show that CVAM better agrees with human listeners than Gemini 3.1 Pro and open-source speech LLMs, and outperforms single-human-vs.-rest agreement. Together, we demonstrate the importance of grounding voice aesthetics in human perception and propose a principled framework for human alignment.
PM-34. Reference-Free Mispronunciation Detection with Reinforcement Learning and Preference Optimization
Ju Lin, Faiaz Rahman, Jing Zheng, Florian Metze
Meta
Abstract
- Mispronunciation detection and diagnosis (MDD) systems are typically evaluated with the target script in hand, using it to force-align a canonical phoneme sequence against the learner's speech. That assumption fails on spontaneous speech, where the intended utterance is unknown at scoring time. We propose a reference-free framework built on a speech large language model. A single model decodes two channels from audio alone: a word channel that recovers the intended words and a phoneme channel that transcribes what was actually articulated. Grapheme-to-phoneme conversion of the word channel yields a canonical reference the model supplies for itself, so detection reduces to self-disagreement between the two channels. We then optimize the discrete detection criterion at the sequence level, comparing three objectives: a transcription-accuracy reward, a detection-$F_1$ reward with optional vowel and confusion-pair weighting, and an anti-normalization preference objective contrasting realized against canonical pronunciations. Reference-free phoneme-tier detection $F_1$ improves from 38.3 to 49.2 and word-tier $F_1$ from 50.1 to 60.3 over an in-domain supervised baseline, with the missed-vowel-error rate falling 39.2% and 63.9% relative at the two tiers. Discarding the ground-truth script costs only 2.4--4.8 and 0.7--1.5 $F_1$ respectively, indicating that self-supplied canonicals are a viable substitute for a known reference.
PM-35. CLEAR: Online Speech Content Leakage Estimation through Cross-ASR Disagreement
Tanvi Kandepuneni, Bhawana Chagglani, Jeremy Gummeson, Prashant Shenoy
University of Massachusetts, Amherst
Abstract
- Signal-level speech privacy mechanisms suppress linguistic content while preserving acoustic information for downstream sensing, but their privacy settings are typically fixed and cannot adapt to changing speech leakage at runtime. We present CLEAR, a reference-free method that estimates residual speech leakage using disagreement among heterogeneous ASR systems. As privacy transformations make speech less recoverable, ASR outputs increasingly disagree. Across configurable privacy settings, cross-ASR disagreement closely tracks transcript-based leakage, achieving a correlation of 0.8. CLEAR also examines latency–accuracy trade-offs and identifies potentially exposed words, enabling runtime feedback and dynamic adjustment of privacy strength without requiring ground-truth transcripts. This work is currently under review at ICASSP 2027.
PM-36. Revisiting Speech Enhancement Optimization with Quality Estimators: Opportunities and Pitfalls
Gasser Elbanna, Bryce Irvin, Marko Stamenovic, Mikolaj Kegler
Harvard University, MIT, Bose Corporation
Abstract
- Speech enhancement models are typically trained with signal-level reconstruction losses that correlate imperfectly with perceived quality. Speech-quality estimators offer perceptually aligned supervision, but directly optimizing their scores can produce severe artifacts. We show that this failure originates in the estimators themselves: waveforms optimized against a frozen estimator achieve high target scores while performing poorly on other metrics and remaining perceptually degraded. To address this, we propose a perceptual objective based on latent representations of deep speech-quality estimators trained to predict human ratings, combined with a signal-level reconstruction loss that anchors optimization toward the clean target. Among several recently proposed estimators, the anchored ScoreQ- and UTMOS-latent losses perform best, improving over either component alone, with the largest gains at low SNRs. A 3.6 M-parameter model outperforms discriminative and generative baselines up to an order of magnitude larger, while a listening study confirms its benefits for overall quality and speech preservation.
PM-37. Predicting Menstrual Cycle Phases from Speech: A Paralinguistic Approach
Anika A. Spiesberger, Andreas Triantafyllopoulos, Melanie Weirich, Björn Schuller
Technical University of Munich University Hospital, Munich Center for Machine Learning, Friedrich-Schiller-University Jena, Imperial College London, Munich Data Science Institute
Abstract
- Hormonal changes throughout the menstrual cycle can affect speech production. However, studies investigating differences in acoustic parameters have found inconsistent results. This could be because changes are likely subtle and multivariate. We therefore apply handcrafted and embedding-based features, combined with machine learning methods. We use a German read speech dataset of 76 participants in two cycle phases (ovulation and luteal). Additionally, we investigate whether speaker-level accuracy is correlated with hormonal levels, the magnitude of hormonal changes between phases, or age. Our results show small, non-significant effects of loudness, formant amplitudes, H1-H2, and spectral flux. We achieve 62.5 % accuracy with handcrafted features, but chance-level performance with learnt embeddings. Speaker-level accuracies show no correlation with hormonal levels or age. We suggest using more cycle phases and personalisation techniques in future research.
PM-38. P2Flow: Phoneme-Aware Progressive Flow Matching for Extreme Speech Super-Resolution
Ningyuan Yang, Yize Li, Pu Zhao, Diego A. Cuji, Kanad Sarkar, Ryan M. Corey, Xue Lin, Andrew C. Singer
Stony Brook University, Northeastern University, University of Illinois Urbana-Champaign, University of Illinois Chicago
Abstract
- We investigate extreme speech super-resolution, where severely band-limited speech at 1 or 2 kHz is restored to 16 kHz. We propose P2Flow, a phoneme-aware progressive flow matching framework that combines phoneme guidance, progressive frequency modeling, and vocoder post-training. Experiments on TIMIT and VCTK demonstrate consistent improvements over representative speech super-resolution baselines in spectral fidelity, perceptual quality, and intelligibility.
PM-39. CrossEdit: Cross-Modal Training Enables Rich Audio-Visual Editing
William Chen, Prem Seetharaman, Ke Chen, Oriol Nieto, Kevin Duarte, Siddharth Srinivasan Iyer, Mamshad Nayeem Rizve, Zhiwen Cao, Shinji Watanabe, Yuanjun Xiong, Jianming Zhang, Zeyu Jin, Justin Salamon
Carnegie Mellon University, Adobe
Abstract
- Multimedia editing requires generative models to interpret complex, compositional instructions and modify only the desired elements across modalities, a capability largely beyond existing systems. Current approaches are typically limited to simple, single-attribute edits within one modality, since diverse, high-quality editing pairs are hard to obtain, especially for cross-modal tasks where aligned audiovisual (AV) data is scarce. We present CrossEdit, a unified omni-modal editing model for images, audio, and video that performs AV movie scene edits zero-shot through cross-modal transfer. Our key observation is that complex instruction-following, once learned in any modality, generalizes to others. We exploit the additive nature of acoustic signals to procedurally generate large-scale audio editing pairs with compositional instructions, and show that fine-tuning on this synthetic data and self-supervised AV masked reconstruction, alongside a targeted set of curated cross-modal tasks, induces instruction-following that transfers zero-shot to unseen modality and instruction combinations. To evaluate this capability, we release CrossEditBench, a human-annotated benchmark of AV edits on movie scenes, and propose AV-FES, a metric that jointly scores instruction following and consistency. We show that our proposed techniques improve performance on zero-shot AV movie scene editing, while maintaining or improving performance on lip-synced speech editing and standard image, video, and audio editing benchmarks.
PM-40. ESPnet3: Infrastructure for Scalable Speech and Audio Research in the Foundation Model Era
Masao Someki, Alexander Polok, Carlos Carvalho, Chyi-Jiunn Lin, Da-Hee Yang, Jiatong Shi, Jinchuan Tian, Nelson Enrique Yalta Soplin, Samuele Cornell, Siddhant Arora, Francisco Teixeira, Wei Wang, William Chen, Alberto Abad, Chenda Li, Shinji Watanabe, Wangyou Zhang
Carnegie Mellon University, Brno University of Technology, Instituto Superior Técnico, Hanyang University, Hitachi Astemo, Shanghai Jiao Tong University
Abstract
- ESPnet3 is a new framework designed to support scalable speech and audio research in the foundation model era. It introduces a modular architecture that simplifies the integration of new models and tasks, along with flexible dataset management and large-scale training. ESPnet3 provides a unified workflow covering data preparation, training, fine-tuning, inference, evaluation, and demo creation. We demonstrate its capabilities through practical examples of speech and audio model development and adaptation.
PM-41. Segmental Posterior Decoding for Audio Moment Retrieval
Seungdeok Choi, Seongmin Choi, Inhan Choi, Junho Kim, Jeong-gyu Ban, and Yong-Hwa Park
Korea Advanced Institute of Science and Technology (KAIST)
Abstract
- Audio moment retrieval (AMR) identifies temporal segments in long recordings that best match a free-form text query. Existing systems largely rely on fixed-slot DETR decoders that assign proposal-level confidence scores without explicitly normalizing over competing explanations of the full timeline. We propose segmental posterior decoding, which defines a globally normalized distribution over temporal segmentations and scores each candidate moment by its exact segment marginal posterior computed through forward-backward inference. We further expand the training segmentation space by treating a foreground span and its adjacent subdivisions as distinct hypotheses, thereby increasing competition among alternative segmentations. On CASTELLA, our method achieves 41.15% R1@0.7 and 34.68% mAP, outperforming the same network decoded with DETR slot confidence by 10.91 and 9.20 percentage points, respectively.
PM-42. Projected Audio Tokens Gain Retrieval and Lose Grounded Generation in Multimodal LLMs
Ali Vosoughi, Jing Bi, Pinxin Liu, Yolo Y. Tang, Chenliang Xu
University of Rochester
Abstract
- Projecting audio tokens into CLIP space raises audio-to-video retrieval in a multimodal LLM and lowers how well its captions are grounded in the audio. We freeze LLaVA-1.6 with a Mistral-7B backbone and five audio encoders, and give the model either a raw audio token or one projected into CLIP space. On a 1,006-clip pool, projection raises recall at one for every encoder, with ImageBind rising from 0.10% to 15.7% (CLAP and Whisper values are lower bounds for their original configurations). Captions move the other way. With 150 visual tokens kept, a CLAP-based audio grounding score falls by 23–53% and recall of the sound sources falls by 31–48%. Across 3,568 paired clips, projected tokens show higher centered kernel alignment with visual embeddings for all five encoders, with less norm outside the visual subspace and less nearest-neighbor overlap with the raw audio graph. Raw tokens lead on most generation scores across five frozen backbones from 7B to 34B wherever raw-token generation is stable. These comparisons do not identify a causal mechanism. The work is under review at ICASSP 2027.
PM-43. ALM2Vec: Learning Audio Embeddings for Universal Audio Retrieval with Large Audio-Language Models
Fengjie Lu, Chenang Jiang, Jiarui Hai, Helin Wang, Aaron Yee
Zhejiang University, Johns Hopkins University
Abstract
- We present ALM2Vec, a universal audio embedding framework that leverages pretrained large audio-language models (LALMs) for flexible and instruction-aware audio retrieval. Unlike traditional contrastive audio-text embedding models that primarily focus on caption matching, ALM2Vec transfers the audio understanding, instruction-following, and reasoning capabilities of LALMs into a unified embedding space. This enables diverse retrieval tasks, including text-audio retrieval, audio question answering, and aspect-conditioned retrieval guided by natural-language instructions. Experiments demonstrate competitive performance on standard audio and speech retrieval benchmarks, along with promising compositional and controllable retrieval capabilities across different domains and tasks.
PM-44. OmniDream: Enabling Immersive Audio-Visual Experience from Any Video
Zitong Lan, Mutian Tong, Jiatao Gu, Mingmin Zhao
University of Pennsylvania
Abstract
- We present OmniDream, a training-free framework that transforms silent monocular videos into immersive 360° audio-visual experiences. OmniDream introduces an object-centric audio representation that disentangles sound content from scene-dependent acoustic effects, enabling independent sound generation, physics-based acoustic simulation, and spatial audio rendering. By leveraging visual scene understanding and acoustic propagation modeling, OmniDream generates spatially consistent soundscapes that adapt to the viewer's perspective. Our results demonstrate improved audio-visual alignment, spatial accuracy, and perceptual immersion, highlighting the potential of combining generative models with physics-based acoustics for immersive audio-visual content creation.
PM-45. An Evaluation Framework for Structured Audio Captions Validated by Controlled Perturbations
Liang-Yuan Wu, Sripathi Sridhar, Mark Cartwright, Magdalena Fuentes
New York University, New Jersey Institute of Technology
Abstract
- Recent advances in automated audio captioning (AAC) are driving a shift from monolithic sentences toward structured formats that disentangle acoustic and semantic properties, such as timestamped captions for different sound events. Such representations can support faceted sound search for creators and richer access to auditory information for Deaf and Hard of Hearing people. Yet, it remains unclear how to meaningfully evaluate these hybrid, structured captions. We propose an evaluation framework for structured audio descriptions, spanning five complementary axes: tag sets, descriptions, reasoning, numeric measurements, and spectral profiles. The framework combines large language model (LLM) judges for semantic fields with deterministic metrics for temporal and acoustic attributes. To validate these metrics, we introduce controlled perturbations that apply typed, graded changes to ground-truth annotations. Results show that the proposed metrics remain robust to meaning-preserving paraphrases while responding to genuine semantic and acoustic corruptions, enabling more reliable evaluation of structured captions.
PM-46. Audio-R1: Audio Reasoning through Reinforcement Learning on Self-Supervised Tasks
Jihoon Jeong, Mirco Ravanelli, Zhepei Wang, Cem Subakan
Mila-Québec AI Institute, Université Laval, Concordia University, Adobe Research
Abstract
- Reasoning is increasingly important for large audio-language models (LALMs), yet remains relatively underexplored compared with text-only and vision-language models. We introduce Audio-R1, a reinforcement learning (RL) post-training framework that revisits audio self-supervised learning as a source of verifiable rewards for improving audio understanding and reasoning task performance. Audio-R1 uses answers automatically determined during the construction of self-supervised tasks to verify model responses and compute rewards. Experiments across multiple LALMs and diverse audio benchmarks show consistent performance gains, while ablations demonstrate the benefits of each self-supervised task and additional gains from joint training. While both supervised fine-tuning and Audio-R1 improve self-supervised task accuracy, Audio-R1 further improves performance on downstream tasks distinct from the training tasks. These results highlight the potential of RL post-training with self-supervised audio tasks to improve downstream performance and reasoning-trace quality.