In practice, training using federated learning can be much slower than standard centralized training. This severely limits the amount of experimentation and tuning that can be done, making it difficult to perform well on a given task. Server-side proxy data can be used to run training simulations, for example for hyperparameter tuning. This can greatly speed up the training process by reducing the number of tuning runs that must be performed overall on real customers. However, it is difficult to ensure that these simulations accurately reflect the dynamics of real federated training. In particular, the proxy data used for simulations often comes as a single centralized data set without partitioning across different clients, and naively splitting this data can lead to simulations that poorly reflect actual federated training. In this article we address the challenge of how to partition centralized data in a way that reflects the statistical heterogeneity of true federated clients. We propose a theoretically justified, fully federated algorithm that efficiently learns the distribution of true clients and sees improved server-side simulations when it uses the inferred distribution to create simulated clients from centralized data.
