While automatic speech recognition (ASR) systems are widely used in many real-world applications, they often do not generalize well to new domains and need to be fine-tuned with data from these domains. However, data from the target domain is generally not available in many scenarios. In this paper, we propose a new strategy to adapt ASR models to new target domains without text or speech from those domains. To achieve this, we propose a novel data synthesis process that uses a large language model (LLM) to generate a target-domain text corpus and a state-of-the-art controllable speech synthesis model to generate the corresponding speech. We propose a simple but effective instruction-in-context adjustment strategy to increase the effectiveness of LLM in generating text corpora for new domains. Experiments on the SLURP dataset show that the proposed method achieves an average relative word error rate improvement of 28% on unseen target domains without any performance drop on source domains.
