Performance Results

Table of Links

Abstract and 1. Introduction

2. Related Work

3. Background on Autoencoder and SPRT and 3.1. Autoencoder

3.2. Sequential Probability Ratio Test

4. Design of CUMAD and 4.1. Network Model

4.2. CUMAD: Cumulative Anomaly Detection

5. Evaluation Studies and 5.1. Dataset, Features, and CUMAD System Setup

5.2. Performance Results

6. Conclusions and References

5.2. Performance Results

Table 2 shows the performance of CUMAD in detecting IoT devices, in terms of accuracy, recall, and F1 score [12]. From the table we can see that CUMAD achieves superior performance in all three metrics. For example, for 5 of the IoT devices, CUMAD is able to detect all the compromised cases (see the column of Recall). CUMAD is also able to detect vast majority of the compromised cases for the remaining two of the IoT devices, with recall scores of 0.999 and 0.994. Considering both detection precision of attack and benign traffic, we can see that CUMAD also performs very well, with an accuracy score ranging from 0.955 to 0.995 for all 7 IoT devices. The F1 scores, which is a weighted average of the precision and recall scores of a model, also confirm that CUMAD performs well in detecting compromised cases.

Figure 4 shows the false positive rates of an autoencoder based anomaly detection scheme and CUMAD. As shown in the figure, the false positive rates of the autoencoder-based anomaly detection scheme for the 7 IoT devices range from 0.77% to 11.22%, while the false positive rates of CUMAD range from 0.014% to 2.067%. On average the autoencoder based anomaly detection scheme has about 3.57% false positive rate, while the false positive rate of CUMAD is about 0.5%, which represents about 7 times performance improvement in terms of false positive rate for CUMAD over the autoencoder-based anomaly detection scheme.

For performance comparison, we also include in the table the performance results of the N-BaIoT scheme, with the same evaluation studies setup. We can see from the table that CUMAD and N-BaIoT performs comparably in terms of all three-performance metrics. However, N-BaIoT works on a fixed window size. Table 2 shows that N-BaIoT requires a relatively large window size, ranging from 20 to 82 (column with name Window Size). In contrast, CUMAD works in an online fashion and does not requires such a fixed window size. Table 2 shows the average number of observations required for CUMAD to reach a detection (column with name Mean Size); we can see from the table that it takes on average less than 5 observations for CUMAD to make a detection of a compromised case, much quicker than NBaIoT. In order to have a better understanding of the number of observations for CUMAD to make a detection of a compromised case, Figure 5 shows the cumulative distribution function (CDF) of required observations for CUMAD to make a detection for all the 7 IoT devices. We can see from the figure that the vast majority of detection requires less than 10 observations for all 7 IoT devices.

In summary, compared to simple anomaly detection schemes such as the ones only based on autoencoders, CUMAD can greatly reduce the false positive rates, making CUMAD much more attractive than simple anomaly detection schemes in the real-world deployment. Compared with window-based schemes such as N-BaIoT, CUMAD requires much less observations to reach a detection, and thus can detect compromised IoT devices much quicker.