A novel approach to house price prediction using segmented bayesian optimization based ensemble models

Abstract

The housing market is a multifaceted system that not only satisfies the fundamental human need for shelter but also functions as a critical instrument for wealth accumulation, investment planning, and financial decision-making. Over the past decades, residential properties have evolved into complex economic assets, whose valuation is affected by a wide array of factors ranging from structural characteristics and location to broader economic conditions and market sentiment. The global financial crises, including the 2008 mortgage collapse and the subsequent disruptions caused by the Covid-19 pandemic, have vividly demonstrated the volatility and sensitivity of housing prices to macroeconomic shocks, inflationary pressures, and shifting consumer expectations. These dynamics are particularly pronounced in developing economies such as Türkiye, where economic fragility and rapid urbanization create an environment of heightened uncertainty, making reliable, data driven methods for predicting property values indispensable for buyers, sellers, investors, financial institutions, and policy regulators. This study addresses these challenges by developing a comprehensive, machine learning-based framework for predicting housing prices within the Istanbul real estate market. The dataset utilized includes a broad spectrum of property-specific, locational, and neighborhood-level attributes, such as net and gross square meter areas, number of rooms, building age, floor level, total floors, heating type, listing category, and district and neighborhood-level price indicators. To ensure data quality and analytical reliability, an extensive preprocessing pipeline was applied. This pipeline involved cleaning and standardization of entries, resolution of inconsistent or missing values, and mitigation of outliers through Winsorization, which reduces the influence of extreme values while preserving natural distributional characteristics. Categorical features, including district and neighborhood identifiers, were target-encoded in a manner designed to prevent information leakage, while the target variable itself underwent a log1p transformation to stabilize variance and enhance model learning efficiency. A critical component of the study was the development of engineered features to better capture underlying relationships that are not directly observable in raw data. Features such as floor ratios, area-per-room metrics, bathroom-to-area ratios, and binary indicators for properties within residential complexes were introduced to quantify nuanced aspects of property structure and usability. Additionally, average price-based encodings at district and neighborhood levels allowed the models to incorporate local market expectations, which are known to strongly influence individual property valuations. Correlation-based feature selection and redundancy elimination were further applied to retain only the most informative predictors, improving both model efficiency and interpretability.A novel aspect of the methodology was the segmentation of the dataset into distinct market segments, recognizing that Istanbul's housing market is inherently heterogeneous. Properties were clustered based on a combination of price ranges, structural characteristics, and neighborhood-level indicators, allowing each segment to reflect unique market dynamics. Segment-specific modeling enabled the identification of differentiated feature importance patterns and improved predictive performance, as individual models could adapt to the structural, locational, and economic nuances of their respective groups. This approach also provides meaningful insights into how property characteristics influence pricing differently across luxury, mid-range, and smaller-scale residential segments. Five ensemble-based regression algorithms—XGBoost, LightGBM, CatBoost, Gradient Boosting, and Random Forest—were trained and optimized using Bayesian hyperparameter search through Optuna. Evaluation employed multiple complementary metrics, including R², MAE, RMSE, and SMAPE, offering a comprehensive assessment of both accuracy and predictive stability. Segment-specific hyperparameter optimization further strengthened each model's adaptability, while an R²-weighted ensemble of the optimized models was constructed to combine their predictive strengths. This ensemble consistently outperformed individual models, demonstrating enhanced generalization, stability across heterogeneous property groups, and improved resistance to overfitting. In conclusion, this research presents a robust, interpretable, and highly adaptable machine learning framework for housing price prediction in dynamic urban markets. By combining rigorous preprocessing, advanced feature engineering, segment-specific modeling, and ensemble learning, the study not only delivers high predictive accuracy but also provides actionable insights into the determinants of property values. The proposed methodology offers practical benefits for buyers, sellers, financial institutions, and policymakers by enabling data-driven decision-making and fairer market assessments. Future extensions may involve incorporating temporal and macroeconomic indicators, spatial econometric techniques, and deeper hierarchical segmentation strategies, which could further enhance the framework's capacity to capture evolving market behaviors and complex nonlinear relationships in real estate pricing.