AI-Based Anomaly Detection in Big Data

Abstract

Anomaly detection -- identifying observations that deviate significantly from expected patterns -- is a foundational problem in big data analytics with applications spanning fraud detection, cybersecurity intrusion, industrial process monitoring, healthcare surveillance, and climate data quality assurance. The big data context introduces distinctive challenges: anomaly detectors must scale to billions of records, operate on streaming data with sub-second latency, adapt to concept drift without full retraining, and maintain low false positive rates that are operationally manageable in high-volume settings. This study evaluates twelve anomaly detection algorithms across five big data domains and four dataset scales (10^5, 10^6, 10^7, and 10^8 records). Algorithms include classical methods (Isolation Forest, LOF, One-Class SVM), deep learning approaches (AutoEncoder-AD, VAE-AD, DeepSVDD), streaming ML (River-AD, RRCF), graph-based (Graph-AD), transformer-based (TransAD), a large language model-guided detector (LLM-AD), and a proposed Scalable Adaptive Anomaly Detection framework (SAAD). SAAD achieves the best AUC across all five domains (mean 0.924) while maintaining sub-10ms per-record latency at 10^7 scale. Isolation Forest remains the most computationally efficient for batch detection. TransAD achieves the highest AUC on time-series domains (0.948). LLM-AD demonstrates novel capability for semantic anomaly detection (contextual deviations requiring natural language understanding) with AUC 0.884. A scalability and accuracy trade-off framework for big data anomaly detection deployment is proposed.