Data splitting method for building machine learning models
DOI: 10.21293/1818-0442-2024-27-4-88-96
Abstract: A method for splitting into training and validation samples for constructing predictive machine learning models is presented. The method is aimed at maintaining the proximity of samples to the original data. Proximity means minimal deviation of the characteristics of data features. Preserving proximity allows reducing information loss when constructing models, compared to random division, and thereby increasing the generalizing predictive ability. Using training data, many alternative models with structural characteristics of varying complexity are constructed, and the most accurate model is selected using validation data. Experiments were conducted to construct fuzzy classi-fiers with data division. The application of the method showed an increase in the accuracy of classification and interpretability of models compared to random division and without division of the original data.
Keywords: machine learning, classification, binary optimization algorithms, fuzzy classifier
Funding: The study was supported by the grant from the Russian Science Foundation No. 24-21-00168, https://rscf.ru/project/24-21-00168/.
For citation:
Sarin K. S. Data splitting method for building machine learning models. Doklady Tomskogo gosudarstvennogo universiteta sistem upravleniya i radioelektroniki, 2024, vol. 27, no. 4, pp. 88–96. DOI: 10.21293/1818-0442-2024-27-4-88-96
Authors and copyright holders:
- Sarin K. S. , Tomsk State University of Control Systems and Radioelectronics (Tomsk, Russia)
- 1. Hu X., Chu L., Pei J., Liu W., Bian J. Model complexity of deep learning: a survey. Knowledge and Information Systems, 2021, vol. 63, pp. 2585–2619.
- 2. Wen W., Ke W., Feng J., Liu S., Xu Z., Sheng X. Constructing Complexity Metrics for Measuring Generalization Ability of Deep Learning Models. 2024 10th International Conference on Big Data and Information Analytics (BigDIA), Chiang Mai, Thailand, 2024, pp. 9–16.
- 3. Bulso N., Marsili M., Roudi Y. On the complexity of logistic regression models. Neural Computation, 2019, vol. 31, no. 8, pp. 1592–1623.
- 4. Hu X. B., Liu W., Bian J., Pei J. Measuring Model Complexity of Neural Networks with Curve Activation Functions. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2020, pp. 1521–1531.
- 5. Buhrman H., De Wolf R. Complexity measures and decision tree complexity: a survey. Theoretical computer science, 2002, vol. 288, no. 1, pp. 21–43.
- 6. Gacto M.J., Alcala R., Herrera F. Interpretability of linguistic fuzzy rule-based system: An overview of interpretability measures. Information Science, 2011, vol. 181, pp. 4340–4360.
- 7. Hanin B., Rolnick D. Complexity of linear regions in deep networks. International conference on machine learning, PMLR. 2019. pp. 2596–2604.
- 8. Ortigossa E.S., Goncalves T., Nonato L.G. EXplainable Artificial Intelligence (XAI) – From Theory to Methods and Applications. IEEE Access, 2024, vol. 12, pp. 80799–80846.
- 9. Chehreghani M.H. A Review on the Impact of Data Representation on Model Explainability. ACM Computing Surveys, 2024, vol. 56, pp. 1–21.
- 10. Ying X. An Overview of Overfitting and its Solutions. Journal of Physics: Conference Series, 2019, vol. 1168, no. 2, pp. 1–6.
- 11. Ma C., Liu Y., Deng J., Xie L., Dong W., Xu C. Understanding and Mitigating Overfitting in Prompt Tuning for Vision-Language Models. IEEE Transactions on Circuits and Systems for Video Technology. 2022, vol. 33, pp. 4616–4629.
- 12. Monica Agrawal P. A Survey on Hyperparameter Optimization of Machine Learning Models. 2024 2nd International Conference on Disruptive Technologies (ICDT), Greater Noida, India, 2024, pp. 11–15.
- 13. Yang L., Shami A. On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice. Neurocomputing, 2020, vol. 415, pp. 295–316.
- 14. Raji I.D., Bello-Salau H., Umoh I.J., Onumanyi A.J., Adegboye M.A., Salawudeen A.T. Simple Deterministic Selection-Based Genetic Algorithm for Hyperparameter Tuning of Machine Learning Models. Applied Sciences, 2022, vol. 12, no. 3, pp. 1186.
- 15. Bergstra J., Bardenet R., Bengio Y., Kegl B. Algorithms for hyper-parameter optimization. Proceedings Advances in Neural Information Processing Systems, 2011, pp. 2546–2554.
- 16. James B., Yoshua B. Random search for hyperparameter optimization. Journal of Machine Learning Research. 2012, vol. 13, no. 1, pp. 281–305.
- 17. Snoek J., Larochelle H., Adams R. Practical Bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems, 2012, vol. 4, pp. 2951–2959.
- 18. Alcala R., Nojima Y., Herrera F., Ishibuchi H. Multiobjective genetic fuzzy rule selection of single granularitybased fuzzy classification rules and its interaction with the lateral tuning of membership functions. Soft Computing, 2011, vol. 15, pp. 2303–2318.
- 19. Fazzolari M., Alcala R., Herrera F. Multi-objective evolutionary method for learning granularities based on fuzzy discretization to improve the accuracy-complexity trade-off of fuzzy rule-based classification systems: D-MOFARC algorithm. Applied Soft Computing, 2014, vol. 24, pp. 470–481.
- 20. Alcala-Fdez J., Alcala R., Herrera F. A Fuzzy Association Rule-Based Classification Model for High-Dimensional Problems With Genetic Rule Selection and Lateral Tuning. IEEE Transactions on Fuzzy Systems, 2011, vol. 19, no. 5, pp. 857–872.
- 21. Alcala R., Alcala-Fdez J., Herrera F. A proposal for the genetic lateral tuning of linguistic fuzzy systems and its interaction with rule selection. IEEE Transactions of Fuzzy System. 2007, vol. 15, no. 4, pp. 616–635.
- 22. Sarin K.S. Discrete Optimization Algorithm Based on Probability Distribution with Transformation of Target Values. Programming and Computer Software, 2024, vol. 50, no. 6, pp. 445–456.
- 23. Sarin K.S. [Mixed-integer multiobjective optimization algorithm based on cuckoo search methaheuristic with genetic crossover operator]. Artificial Intelligence and Decision Making, 2024, no. 2, pp. 87–105 (in Russ.).
- 24. Sarin K., Bardamova M., Svetlakov M., Koryshev N., Ostapenko R., Hodashinskaya A., Hodashinsky I. A three-stage fuzzy classifier method for Parkinson’s disease diagnosis using dynamic handwriting analysis. Decision Analytics Journal, 2023, vol. 5, pp. 100274.
- 25. Demsar J. Statistical Comparisons of Classifiers over Multiple Data Sets. Journal of Machine Learning Research, 2006, vol. 7, pp. 1–30.
- 26. Garsia S., Herrera F. An Extension on «Statistical Comparisons of Classifiers over Multiple Data Sets» for all Pairwise Comparisons. Journal of Machine Learning Research, 2008, vol. 9, pp. 2677–2694.
- 27. Garsia S., Fernandez A., Luengo J., Herrera F. Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining: Experimental analysis of power. Information Sciences, 2010, vol. 180, pp. 2044–2064.
Last issue
-2026-1_%D0%BA%D0%BE%D0%BF%D0%B8%D1%8F.jpg)
Editorial office address
Executive Secretary of the Editor’s Office
Editor’s Office: 40 Lenina Prospect, Tomsk, 634050, Russia
Phone / Fax: + 7 (3822) 701-582
Viktor N. Maslennikov
Executive Secretary of the Editor’s Office
Editor’s Office: 40 Lenina Prospect, Tomsk, 634050, Russia
Phone / Fax: + 7 (3822) 51-21-21 / 51-43-02
Subscription for updates
