Shisa has been very lucky to have a crew of talented interns helping us for the summer 2026 season, and we’ve decided to turn the blog over to them so they can talk about the projects they’ve been working on during their time here. Today, we’re turning things over to Lucinda, who’s spent her summer doing critical early exploratory work for a Japanese-specific speech diarization system.
My name is Lucinda, and I interned at Shisa AI over the summer. For my project, I worked on Japanese-language-specific speech diarization.
What Is Diarization?
To improve speaker diarization, we must first have a solid understanding of what it is. Diarization is the component of the audio pipeline that reveals who spoke when. To succeed, diarization needs to determine when speech starts and stops (speech versus silence), identify which speaker is talking, and handle overlapping speech. What it does not give us is the identity of the speakers; it merely assigns them arbitrary labels such as Speaker 1, Speaker 2, and so on.
Speaker diarization is one component in the broader speech-processing pipeline:
Without diarization, transcripts would simply be a block of text with no indication of who is speaking.
A conventional diarization pipeline is itself made up of several components. It starts with voice activity detection to determine when speech is present. It then moves on to segmentation, where the system identifies speaker turns. Next, it creates numerical representations for each segment called embeddings, which record the unique characteristics of the speaker’s voice. Finally, the embeddings are used to assign speaker labels.
Why Is It Hard?
There are currently very few diarization models specifically designed for Japanese speech. The pyannote Community-1 pipeline supports Japanese in theory, but in our testing it performed worse than NVIDIA’s Sortformer, despite Sortformer being trained primarily on English.
We reached this conclusion after manually counting both models’ errors on a Japanese Zoom livestream dataset. We tracked misidentified speakers, missed speech, and crosstalk assigned to a single speaker. Sortformer made fewer errors in the first two categories, although it made slightly more crosstalk errors.
Sortformer performs diarization with a single end-to-end model, whereas pyannote Community-1 uses a pipeline consisting of separate segmentation, speaker-embedding, and clustering stages. Community-1 packages approximately 8.1 million neural parameters across its segmentation and speaker-embedding models, whereas the streaming Sortformer v2.1 model used here has 117 million parameters. This difference in model capacity may have contributed to Sortformer’s stronger zero-shot performance on our Japanese conversation data, suggesting that fine-tuning it for Japanese could yield further improvements.
| Sortformer (NVIDIA) | pyannote Community-1 | |
|---|---|---|
| Architecture | End-to-end streaming diarization model | Multi-stage pipeline (segmentation → speaker embeddings → VBx clustering) |
| Processing | Single jointly trained neural network with an Arrival-Order Speaker Cache | Separate learned models plus PLDA-backed clustering |
| Main backbone | 17-layer NEST encoder (Fast-Conformer-based) + 18-layer Transformer | Segmentation model + WeSpeaker ResNet34-LM embedding model + VBx clustering |
| Training objective | Hybrid Sort Loss + Permutation-Invariant Loss | Segmentation loss + embedding loss; clustering at inference |
| Neural parameter count | 117M | ~8.1M across segmentation and embedding |
| Strength | Better global speaker reasoning and assignment | Flexible and modular |
How I Approached It
Because Sortformer provided a strong baseline, I chose to fine-tune it rather than build a new model from scratch. This let us retrain the model with Japanese conversations while preserving the strengths of its large-scale pretraining.
Training was completed on two NVIDIA RTX 3090 GPUs using Distributed Data Parallel (DDP). I used PyTorch Lightning to handle the training loop, multi-GPU execution, checkpointing, logging, and validation throughout the experiments.
What Was Our Dataset?
For fine-tuning, I used the Creative Commons-licensed Fhrozen/CABankSakuraCHJP dataset, also known as CallHome, subject to the corpus’s applicable access and attribution terms. The dataset used for this project contains 18.3 hours of speech from 120 participants and features human-audited phone calls in Japanese. While many Japanese speech datasets are available, this dataset is particularly useful because it includes timestamps and speaker identification, making it well suited for diarization labels.
Corpus attribution: Linguistic Data Consortium (2008). CABank Japanese CallHome Corpus. TalkBank. https://doi.org/10.21415/T5H59V
I also trained with an internally generated synthetic diarization dataset built from Japanese audiobook-style source recordings. I first aligned the source utterances with Qwen3 Forced Aligner, then used a multi-speaker simulator to compose 90-second Japanese conversations with one to four speakers, controlled turn-taking, overlap, silence, and room-microphone acoustics.
Our synthetic dataset totaled 375 hours and was made up of 90-second clips to match Sortformer training expectations. I controlled speech density, overlap, silence, backchannels, and interruptions, and used noise, augmentation, and acoustic simulation to reproduce the desired environment.
Training
I ran a series of fine-tuning experiments to identify the best strategy for Japanese diarization. I tested several approaches, including encoder freezing versus full fine-tuning, separate encoder and decoder learning rates, warmup and decay schedules, and different checkpoint-selection strategies.
Early experiments used only synthetic data, but lower diarization error rate (DER) on the synthetic data did not translate to our real-world test data, so we shifted to CallHome. The best CallHome-only setup used full fine-tuning with a tied encoder and decoder learning rate of 1e-5.
Combining CallHome with our synthetic data produced further improvements on both CallHome and local video-conference evaluation data.
Results
To measure performance, I used diarization error rate, or DER, which summarizes missed speech, false alarms, and speaker-confusion errors. We used a 0.25-second collar, meaning the scorer ignored errors within 250 milliseconds before or after a reference speaker boundary to account for the fact that exact boundaries are somewhat subjective. A lower DER indicates better performance.
After CallHome-only fine-tuning, our best model improved from a baseline DER of 10.07% to 8.06% on the CallHome development data—a reduction of 2.01 percentage points.
I then ran training on a mix of CallHome and synthetic data and observed further DER improvements on both our CallHome evaluation data and our local video-conference data. The model improved from 10.07% to 7.78% on CallHome evaluation and from 9.95% to 9.18% on our local video evaluation.
Conclusion
Although Sortformer was trained primarily on English, it provided a stronger starting point than pyannote Community-1 on our evaluation data. By fine-tuning with Japanese conversational speech and running a series of training experiments, I reduced diarization error while preserving the strengths of Sortformer’s architecture.
For this evaluation, the results suggest that architecture mattered at least as much as language-specific pretraining: the end-to-end Sortformer model generalized well to Japanese conversations, indicating that strong speaker representations can transfer across languages. Fine-tuning with Japanese data then allowed the model to adapt to the relevant conversational characteristics.
This project also reinforced the importance of real, annotated conversational datasets. CallHome succeeded where purely synthetic training did not, while the best overall results came from combining real and synthetic data. Our experiments suggest that adding more real data could improve DER further.
Overall, this project demonstrates that adapting an existing diarization model can be an effective alternative to building a language-specific model from scratch. With a relatively modest amount of Japanese data, I was able to improve performance while building on a strong pretrained foundation.