Convolution-BatchNorm (ConvBN) blocks are integral components in various computer vision tasks and other domains. A ConvBN block can operate in three modes: train, test, and deploy. While Train mode is indispensable for training models from scratch, Eval mode is suitable for transfer learning and more, and Deploy mode is designed for model deployment. This article focuses on the trade-off between stability and efficiency in ConvBN blocks: the implementation mode is efficient but suffers from training instability; The evaluation mode is widely used in transfer learning, but lacks efficiency. To resolve the dilemma, we theoretically reveal the reason behind the decreased training stability observed in Deploy mode. Subsequently, we propose a novel tuning mode to bridge the gap between evaluation mode and implementation mode. The proposed Tune mode is as stable as the Eval mode for transfer learning, and its computational efficiency closely resembles that of the Deploy mode. Through extensive experiments on object detection, classification, and generation of adversarial examples on datasets and model architectures, we demonstrate that the proposed Tune mode preserves performance while significantly reducing GPU memory footprint and build time. training, thus contributing efficient ConvBN blocks for transfer learning and more. Our method has been integrated into both PyTorch (general machine learning framework) and MMCV/MMEngine (computer vision framework). Professionals only need one line of code to enjoy our efficient ConvBN blocks thanks to PyTorch's built-in machine learning compilers.
