Derivative-Free vs. Sampling Based Optimization in Apache Spark MLlib
61st International Scientific Conference on Information, Communication and Energy Systems and Technologies, ICEST 2026, Nis, Sırbistan, 1 - 03 Temmuz 2026, ss.259-264, (Tam Metin Bildiri)
- Yayın Türü: Bildiri / Tam Metin Bildiri
- Doi Numarası: 10.1109/icest71230.2026.11623479
- Basıldığı Şehir: Nis
- Basıldığı Ülke: Sırbistan
- Sayfa Sayıları: ss.259-264
- Anahtar Kelimeler: Apache Spark MLlib, convergence analysis, derivative-free optimization, distributed machine learning, resourceaware computing, sampling-based optimization
- Süleyman Demirel Üniversitesi Adresli: Evet
Özet
Although great strides have been made in matrix computation, scaling techniques, and prediction models, there exists no systematic comparison of optimization methodologies specifically tailored for Apache Spark MLlib's distributed architecture. This study presents a novel comparative analysis of derivative-free and sampling-based optimization methods applied to Apache Spark MLlib. To that end, Cross-Entropy (CE), Monte Carlo, Nelder-Mead, Mesh Adaptive Direct Search (MADS), and Hooke-Jeeves (HJK) are involved in the comparative analysis. The experiment is conducted through four benchmark classification datasets including Dense, Microsoft, Payload, and Santander, and encompasses three key contributions: (1) convergence-iteration dynamics across MLlib algorithms, (2) comparative convergence rates providing practical insights for practitioners, and (3) comprehensive scalability analysis integrating CPU utilization and memory consumption metrics. Results reveal distinct performance patterns: MADS demonstrates optimal balance between computational efficiency and memory footprint, while Nelder-Mead exhibits unexpectedly high memory consumption despite moderate CPU requirements. Convergence analysis shows that GLM achieves rapid stabilization with CE optimization, whereas HJK displays characteristic instability patterns during early iterations. Notably, using deep neural networks requires much more computational resources when the optimization is done based on classification accuracy.