A Hybrid Machine Learning Model for Accurate Forecasting and Anomaly Detection in Big Data Applications

Abstract

The rapid expansion of digital technologies has resulted in the generation of massive volumes of heterogeneous data across domains such as healthcare, finance, smart manufacturing, cybersecurity, Internet of Things (IoT), and intelligent infrastructure. While Big Data analytics provides valuable opportunities for extracting insights and supporting data-driven decision-making, traditional machine learning approaches often face limitations in handling high-dimensional, nonlinear, dynamic, and continuously evolving datasets. Accurate forecasting and timely anomaly detection have become essential requirements for ensuring operational efficiency, risk management, and intelligent decision support in modern computational environments.

This research proposes a Hybrid Machine Learning Model (HMLM) that integrates multiple artificial intelligence techniques for enhanced forecasting accuracy and anomaly detection in large-scale Big Data applications. The proposed model combines ensemble learning algorithms, deep learning architectures, time-series forecasting methods, and unsupervised anomaly detection techniques to effectively analyse complex data patterns. The framework integrates Extreme Gradient Boosting (XGBoost), Random Forest (RF), Long Short-Term Memory Networks (LSTM), Autoencoders, Isolation Forest, and Support Vector Machine (SVM) within a unified predictive analytics architecture.

A comprehensive experimental evaluation was performed using a multi-domain Big Data environment consisting of 7.5 million data records collected from industrial IoT systems, healthcare monitoring platforms, financial transaction datasets, cybersecurity logs, and smart city applications between 2021 and 2025. The proposed hybrid model was evaluated using forecasting and anomaly detection metrics including Mean Absolute Error (MAE), Root Mean Square Error (RMSE), Mean Absolute Percentage Error (MAPE), accuracy, precision, recall, F1-score, Area Under Curve (AUC), detection rate, and computational efficiency.

Experimental results demonstrate that the proposed hybrid approach significantly outperforms individual machine learning models by improving forecasting accuracy by 28%, increasing anomaly detection capability by 31%, reducing false alarm rates by 22%, and improving computational efficiency by 26%. The integration of explainable artificial intelligence (XAI) techniques, including SHAP and LIME, enhanced model transparency and improved interpretation of predictive outcomes.

The study identifies major challenges including data quality issues, scalability limitations, computational complexity, model interpretability, privacy concerns, and deployment constraints. The findings indicate that hybrid machine learning models provide an effective solution for intelligent Big Data analytics by combining predictive forecasting capabilities with proactive anomaly detection mechanisms. Future research should focus on federated hybrid learning, self-adaptive AI systems, edge-based analytics, and sustainable machine learning architectures.