Vision Models Leave Geometry Untapped
Vision foundation models (VFMs) excel at single-pass predictions like depth, pose, and point clouds. However, they often overlook the critical geometric relationships between multiple views of a scene. This oversight can lead to less accurate and inconsistent predictions, especially in challenging scenarios. A new technique, dubbed Self-Geometry, addresses this gap by enforcing explicit epipolar consistency during inference. This approach tightens predictions across diverse datasets by leveraging fundamental geometric principles, and remarkably, it achieves this without requiring full model retraining.
Traditional test-time methods typically rely on implicit self-consistency. This means they derive consistency signals solely from the model's own outputs. While this can yield some improvements, its effectiveness is limited, particularly when the pretrained VFM is already prone to inaccuracies. The signal is weak if the model's initial predictions are unreliable. Self-Geometry replaces this weak, implicit signal with a more robust approach. It utilizes pseudo ground-truth 2D correspondences and directly optimizes them against multi-view and epipolar losses. This explicit enforcement of geometric laws provides a stronger, more reliable constraint.
Epipolar Constraints Deliver Significant Gains
The effectiveness of Self-Geometry is demonstrated on the wide-baseline ETH3D benchmark, a challenging dataset known for its complex scenes and significant viewpoint changes. Here, Self-Geometry significantly boosts the pose accuracy of the VGGT model. Specifically, it lifts pose accuracy by 9.2% (AUC@30) and an impressive 37.3% (AUC@3). In contrast, a comparable model, presumably using a more standard or weaker test-time adaptation method, achieved gains of only 5.0% (AUC@30) and 25.1% (AUC@3). This stark difference highlights how direct enforcement of epipolar constraints translates into substantial improvements on difficult tasks.
The core idea behind epipolar geometry is that for any point in one image, its corresponding point in another image must lie on a specific line called the epipolar line. This line is determined by the relative pose and camera calibration between the two views. By ensuring that predicted correspondences adhere to this geometric rule, Self-Geometry effectively grounds the model's predictions in physical reality. This is akin to a painter ensuring their perspective lines converge correctly on a canvas; minor errors in individual lines can lead to a distorted overall image, but enforcing the underlying rules of perspective corrects these distortions.
How Self-Geometry Works
The Self-Geometry pipeline operates during the inference phase, meaning it doesn't alter the underlying pretrained VFM. Instead, it introduces an auxiliary module that refines the VFM's initial outputs. The process can be broken down into several key steps:
- Initial Prediction: The VFM processes an input image or set of images and generates initial predictions for depth, pose, and point clouds.
- Correspondence Generation: Using the VFM's outputs, the system generates pseudo 2D correspondences between different views. These are not perfect ground truth but are derived from the model's own understanding of the scene.
- Epipolar Loss Calculation: The generated correspondences are then evaluated against epipolar constraints. This involves calculating a loss function that penalizes deviations from the expected geometric relationships defined by epipolar geometry. Multi-view consistency losses are also applied, ensuring that predictions are coherent across all available views.
- Optimization: The pseudo correspondences are optimized directly to minimize these geometric losses. This optimization process refines the correspondences, making them more geometrically accurate.
- Prediction Refinement: The refined correspondences are used to guide or correct the VFM's initial predictions, resulting in tighter and more geometrically sound outputs.
This approach is particularly powerful because it leverages the VFM's learned features while imposing strong, interpretable geometric priors. Unlike methods that rely solely on the VFM's internal consistency, Self-Geometry introduces an external, physics-based validation step. This makes it robust even when the VFM's initial predictions are less accurate, as seen in the ETH3D benchmark results.
Broader Implications and Future Directions
The success of Self-Geometry suggests a promising direction for improving the reliability and accuracy of vision foundation models. By integrating explicit geometric reasoning at test time, developers can significantly enhance the performance of existing models without the costly process of full retraining. This is especially valuable for applications where precise spatial understanding is critical, such as robotics, autonomous driving, and augmented reality.
What remains to be fully explored is the scalability of this approach to even larger and more complex scenes, or to different types of geometric predictions beyond depth and pose. The current method focuses on 2D-to-2D correspondences derived from 3D predictions. Future work might involve direct 3D-to-3D consistency checks or incorporating other physical constraints. Furthermore, understanding the optimal balance between the VFM's learned representations and the imposed geometric priors will be key to unlocking further performance gains. The ability to fine-tune these constraints dynamically based on scene characteristics could lead to even more adaptable and robust vision systems.