Consumer voice recognition systems do not work as well for many people with speech differences, such as stuttering, relative to the rest of the general population. However, what is not clear is to what extent these systems are not working, how they can be improved, or how many people want to use them. In this paper, we first address these questions using the results of a survey of 61 people who stutter and found that participants want to use speech recognition, but frequently get cut off, misinterpreted, or the speech predictions do not represent the intent. In a second study, in which 91 people who stutter recorded voice assistant commands and dictations, we quantified how disfluencies impede performance in a consumer-grade speech recognition system. Through three technical investigations, we demonstrated how many common errors can be prevented, resulting in a system that cuts expressions 79.1% less frequently and improves the rate of word errors from 25.4% to 9 .9%.