ROBUSTNESS OF TREE-BASED MACHINE LEARNING ALGORITHMS AGAINST ADVERSARIAL EXAMPLES IN NETWORK INTRUSION DETECTION SYSTEM (NIDS) A COMPARATIVE STUDY OF RANDOM FOREST AND XGBOOST

Abstract

The security of machine learning-based Network Intrusion Detection Systems (NIDS) is increasingly threatened by Adversarial Machine Learning (AML) attacks capable of manipulating input data to deceive models and evade detection. The RobEns study [1] built a comprehensive adversarial ensemble framework evaluating four black-box attacks against six models on IoT traffic data, ye left three research gaps unresolved: the accuracy of the substitute model used to represent Random Forest was never validated, XGBoost was excluded from the evaluation entirely, and the single-step adversarial training scheme suffered from label leakage. This study addresses those gaps through a direct comparative analysis between Random Forest and XGBoost on the ToN-IoT dataset, using two complementary black-box attack techniques, ZOO Attack and Genetic Adversarial Attack (GAA), together with Feature Squeezing as a data-level defense mechanism. The methodology comprises four phases: data collection and preprocessing, baseline model training, adversarial attack implementation under two feature-encoding schemes, and defense implementation with comparative evaluation using six metrics: Standard Accuracy (SA), Adversarial Accuracy (AA), Robust Accuracy (RA), Attack Success Rate (ASR), Robust Accuracy Gain (RAG), and SA Preservation Rate (SPR). Results show that both models strongly resist ZOO Attack (ASR=0%) owing to their non differentiable, piecewise-constant decision structure, but respond very differently to GAA: Random Forest retains strong robustness under One-Hot Encoding (ASR=0%) yet becomes highly vulnerable under Ordinal Encoding with StandardScaler (ASR = 99,90%), whereas XGBoost remains consistently vulnerable (ASR = 100%) regardless of the encoding scheme. Feature Encoding (RAG = 28.87%) but offers little benefit in the continuous feature space. These findings indicate that feature representation is as critical as model architecture in determining adversarial robustness, offering practical guidance for designing more resilient tree based NIDS.