This article was accepted into the Self-Supervised Learning Workshop: Theory and Practice (SSLTP) at NeurIPS 2024.
The Image-based Joint Embedding Predictive Architecture (IJEPA) offers an attractive alternative to the masked autoencoder (MAE) for representation learning using the masked image modeling framework. IJEPA drives representations to capture useful semantic information by predicting in latent space rather than input space. However, IJEPA relies on carefully designed target contexts and windows to avoid representational collapse. The encoder modules in IJEPA cannot adaptively modulate the type of predicted and/or target features based on the feasibility of the masked prediction task, as they do not receive enough information from both the context and the targets. Based on the intuition that in natural images information has a strong spatial bias, with local spatial regions being highly predictive of each other compared to distant ones, we condition the target encoder and context encoder modules in IJEPA with positions of context and target windows respectively. . Our “conditional” encoders show performance improvements on several image classification benchmark datasets, improved robustness to context window size, and sampling efficiency during pre-training.
