Lifelong Intelligence Beyond the Edge using Hyperdimensional Computing: Variants of LifeHD

by Computational Technology for AllJuly 24th, 2024

Too Long; Didn't Read

LifeHD is an on-device lifelong learning system using Hyperdimensional Computing for efficient, unsupervised learning in dynamic IoT environments.

featured image - Lifelong Intelligence Beyond the Edge using Hyperdimensional Computing: Variants of LifeHD

Authors:

(1) Xiaofan Yu, University of California San Diego, La Jolla, California, USA ([email protected]);

(2) Anthony Thomas, University of California San Diego, La Jolla, California, USA ([email protected]);

(3) Ivannia Gomez Moreno, CETYS University, Campus Tijuana, Tijuana, Mexico ([email protected]);

(4) Louis Gutierrez, University of California San Diego, La Jolla, California, USA ([email protected]);

(5) Tajana Šimunić Rosing, University of California San Diego, La Jolla, USA ([email protected]).

Table of Links

Abstract and 1. Introduction

7 Evaluation of LifeHD

8 Evaluation of LifeHD semi and LifeHDa

9 Discussions and Future Works

10 Conclusion, Acknowledgments, and References

6 VARIANTS OF LIFEHD

While LifeHD is designed to cater to general IoT applications with streaming input and without supervision, real-world scenarios may vary. Some scenarios might have a few labeled samples in addition to the unlabeled stream, while others may require operation within strict power constraints. LifeHD offers extensibility to address these diverse needs. In this section, we introduce two software-based extensions: LifeHDsemi, which adds a separate processing path to manage labeled samples, and LifeHDa, which adaptively prunes the HDC model using masking to handle low-power scenarios.

6.1 LifeHDsemi

6.2 LifeHDa

While HDC computation is typically lightweight, there may be instances of energy scarcity (e.g., when powered by a solar panel) that call for a balance between accuracy and power efficiency. Similar to neural networks, one approach is to prune the HDC model using a mask, retaining the most crucial HDC dimensions postencoding [25]. Dimension importance can be determined by aggregating all class hypervectors into one and sorting the values across all dimensions. Notably, direct reduction of the encoding dimension should be avoided, as it can degrade HDC’s expressive capability and end up with corruption. Fig. 7 (left) visually demonstrates the impact of masking in supervised HDC tasks: retaining the top 6000 bits incurs only a 3% accuracy loss compared to using the full 10K-bit precision.

While the concept of masking has been employed in prior HDC studies [25], they are not directly applicable to LifeHD due to their offline training setting with iid data. With streaming non-iid data in LifeHD, the set of observed cluster HVs may only represent a subset of the potential classes, and the less significant bits could become crucial as new classes are introduced.

We introduce LifeHDa, which enhances LifeHD through an adaptive masking approach applied to all cluster HVs in working and long-term memory. Let 𝐷𝑎 represent the target dimension for reduction. The rationale behind LifeHDa is depicted in Fig. 7 (right). Whenever an original LifeHD detects novelty, we temporarily revert to the full dimension 𝐷 for 2 batches, which is sufficient for LifeHD to consolidate new patterns in its memory. After these two batches, we assess the long-term memory cluster HVs by aggregating them and ranking the dimensions, and derive a mask retaining 𝐷𝑎 dimensions with the largest absolute values. This mask is then applied to the following batches of 𝜙 (𝑋) immediately after encoding, up until the next novelty is detected. Novelty detection is executed with the masked hypervectors. Importantly, LifeHDa can utilize the same novelty detection sensitivity as LifeHD since the most significant dimensions dominate the similarity check. In other words, the similarity results in LifeHDa using 𝐷𝑎 dimension are similar as using the full dimension. LifeHDa offers an adaptive HDC model pruning interface with minimal accuracy loss and overhead.