Multi-Modal Foundation Models

Abstract

Multi-modal foundation models -- large neural networks pretrained on aligned data spanning text, images, audio, video, and structured data -- represent a qualitative shift in AI capability. Where earlier AI systems required separate models for each modality and task, multi-modal foundation models learn shared representations that transfer across modalities, enable cross-modal generation and retrieval, and support zero-shot task composition. This study evaluates eight multi-modal foundation models -- GPT-4V, Gemini 1.5 Pro, Claude 3 Opus, LLaVA-1.6-34B, InternVL-Chat-V1.5, CogVLM2, Flamingo-80B, and ImageBind -- across six evaluation dimensions: vision-language understanding (VQA, captioning), cross-modal retrieval (image-text, audio-text), video understanding, document understanding (charts, tables, OCR), scientific figure interpretation, and multi-modal reasoning (mathematical diagrams, spatial reasoning). Benchmarks include MMMU, MMBench, SeedBench, ChartQA, DocVQA, VideoMME, and a purpose-built Scientific Figure Understanding Benchmark (SFUB, n=4,840 figures from 12 scientific disciplines). GPT-4V achieves the highest overall MMMU score (56.8%) and leads on scientific figure interpretation (SFUB: 62.4%). InternVL achieves the best ChartQA accuracy (83.6%) and DocVQA score (91.2%). ImageBind demonstrates unique cross-modal binding across six modalities including audio and 3D point clouds. Capability gaps in mathematical diagram reasoning (best model: 48.4%) and temporal video reasoning (best: 54.2%) are identified. A task-to-model routing framework for multi-modal deployment is proposed.