This study presents a hybrid computational framework that integrates radiactive heat transfer (RHT) physics, Design Expert–based response surface methodology (RSM), and machine learning (ML) to optimize combustion efficiency in gaseous flame systems. Radiactive heat exchange is modeled using the Stefan–Boltzmann law under gray-gas assumptions, while gas emissivity is characterized through temperature-composition-and soot-dependent correlations. To systematically explore nonlinear interactions among key combustion variables, a Design Expert central composite design (CCD) is employed to construct the experimental design space and develop predictive response surface models for combustion efficiency. In parallel, a Random Forest (RF) machine learning model (MLM) is trained using multi-modal datasets, including flame images, spectral radiation signatures, RGB intensity features, soot concentration, gas composition, and operating conditions such as flow rate and equivalence ratio. The model exhibits high predictive accuracy, achieving R² = 0.998 for flame temperature and R² = 0.978 for effective emissivity, demonstrating strong agreement with physics -based evaluations. Optimization through the Design Expert desirability function identifies a global optimal combustion condition where maximum thermal efficiency is achieved through an optimal balance of flame temperature, emissivity, and flow dynamics. At this optimal point, response surface analysis confirms a significant reduction in radiactive heat losses and improved energy utilization compared to baseline conditions. The close agreement between physics-based modeling and ML predictions validates the robustness of the hybrid framework. This integrated Design Expert–ML–physics approach enables accurate emissivity prediction, efficient parameter optimization, and potential real-time combustion control. The findings provide practical pathways for improving furnace design, reducing energy losses, and advancing data -driven combustion management in high-temperature thermal systems.