In this paper, we introduce a novel framework for generating multi-speaker speech without relying on any audible inputs. Our approach leverages silent electromyography (EMG) signals to capture linguistic content, while facial images are used to align with the vocal identity of the target speaker. Notably, we present a pitch-disentangled content embedding that enhances the extraction of linguistic content from EMG signals. Extensive analysis demonstrates that our method can successfully generate multi-speaker speech without any audible inputs and confirms the effectiveness of the proposed pitch-disentanglement approach.

Figure : Overview of the proposed multi-speaker EMG-to-Speech generation framework. In the (a) Inference phase, content embedding is estimated from EMG signals, while speaker embedding and global pitch information are derived from a facial image. During the (b) Training phase, for training the (i) face-based voice conversion network, a predefiend speaker-wise global pitch is used to estimate frame-wise F0 values. However, since the global pitch values for unseen target speakers are not available during the inference, a (iii) face-based global pitch estimation network is independently trained using only face images. Additionally, as the speech-based content encoder is not available during inference, the (ii) EMG-based contents estimation network is trained. Each network is trained independently during the training phase.