Robust Foundation Model Architectures

Abstract

Foundation models -- large pre-trained models that serve as a base for a wide range of downstream tasks -- have achieved remarkable capability but face systematic robustness challenges: sensitivity to input distribution shift, adversarial vulnerability, miscalibration of uncertainty estimates, and degradation under out-of-distribution conditions. These robustness failures are not peripheral concerns but core limitations that prevent deployment in safety-critical applications where distributional robustness and calibrated uncertainty are non-negotiable. This study systematically evaluates robustness across six leading foundation model families (GPT-4, Claude 3.5, Gemini 1.5, LLaMA-3-70B, Mistral-Large, and CLIP-ViT-L/14) on five robustness dimensions: distribution shift robustness (ImageNet-C, WILDS, DomainBed), adversarial robustness (AutoAttack, AdvGLUE), calibration (ECE on domain-shifted data), out-of-distribution detection, and uncertainty quantification quality. Additionally, six robustness enhancement techniques are evaluated: adversarial training, data augmentation (AugMax, RandAugment), ensemble methods, test-time adaptation (TTT++, TENT), uncertainty calibration (temperature scaling, conformal prediction), and a proposed Robustness-Aware Fine-Tuning framework (RAFT). RAFT achieves the best combined robustness profile across all five dimensions. Temperature scaling with conformal prediction achieves the best calibration (ECE 0.042 on shifted data). Adversarial training improves AutoAttack robustness by 28.4pp but reduces clean accuracy by 4.2pp. Conformal prediction provides guaranteed coverage at 90% confidence level with minimal set size overhead. A robustness requirements framework for high-stakes foundation model deployment is proposed.