| 1. |
- Abdukalikova, Anara, et al.
(author)
-
Detection of Atrial Fibrillation from Short ECGs : Minimalistic Complexity Analysis for Feature-Based Classifiers
- 2018
-
In: Computing in Cardiology 2018. - : IEEE.
-
Conference paper (peer-reviewed)abstract
- In order to facilitate data-driven solutions for early detection of atrial fibrillation (AF), the 2017 CinC conference challenge was devoted to automatic AF classification based on short ECG recordings. The proposed solutions concentrated on maximizing the classifiers F 1 score, whereas the complexity of the classifiers was not considered. However, we argue that this must be addressed as complexity places restrictions on the applicability of inexpensive devices for AF monitoring outside hospitals. Therefore, this study investigates the feasibility of complexity reduction by analyzing one of the solutions presented for the challenge.
|
|
| 2. |
- Alonso, Pedro, 1986-, et al.
(author)
-
HyperEmbed: Tradeoffs Between Resources and Performance in NLP Tasks with Hyperdimensional Computing Enabled Embedding of n-gram Statistics
- 2021
-
In: 2021 International Joint Conference on Neural Networks (IJCNN) Proceedings. - : IEEE.
-
Conference paper (peer-reviewed)abstract
- Recent advances in Deep Learning have led to a significant performance increase on several NLP tasks, however, the models become more and more computationally demanding. Therefore, this paper tackles the domain of computationally efficient algorithms for NLP tasks. In particular, it investigates distributed representations of n -gram statistics of texts. The representations are formed using hyperdimensional computing enabled embedding. These representations then serve as features, which are used as input to standard classifiers. We investigate the applicability of the embedding on one large and three small standard datasets for classification tasks using nine classifiers. The embedding achieved on par F1 scores while decreasing the time and memory requirements by several times compared to the conventional n -gram statistics, e.g., for one of the classifiers on a small dataset, the memory reduction was 6.18 times; while train and test speed-ups were 4.62 and 3.84 times, respectively. For many classifiers on the large dataset, memory reduction was ca. 100 times and train and test speed-ups were over 100 times. Importantly, the usage of distributed representations formed via hyperdimensional computing allows dissecting strict dependency between the dimensionality of the representation and n-gram size, thus, opening a room for tradeoffs.
|
|
| 3. |
- Balasubramaniam, Sasitharan, et al.
(author)
-
Exploiting bacterial properties for multi-hop nanonetworks
- 2014
-
In: IEEE Communications Magazine. - Piscataway, NJ, USA : IEEE Press. - 0163-6804 .- 1558-1896. ; 52:7, s. 184-191
-
Journal article (peer-reviewed)abstract
- Molecular communication is a relatively new communication paradigm for nanomachines where the communication is realized by utilizing existing biological components found in nature. In recent years researchers have proposed using bacteria to realize molecular communication because the bacteria have the ability to swim and migrate between locations, carry DNA contents (i.e. plasmids) that could be utilized for information storage, and interact and transfer plasmids to other bacteria (one of these processes is known as bacterial conjugation). However, current proposals for bacterial nanonetworks have not considered the internal structures of the nanomachines that can facilitate the use of bacteria as an information carrier. This article presents the types and functionalities of nanomachines that can be utilized in bacterial nanonetworks. A particular focus is placed on the bacterial conjugation and its support for multihop communication between nanomachines. Simulations of the communication process have also been evaluated, to analyze the quantity of bits received as well as the delay performances. Wet lab experiments have also been conducted to validate the bacterial conjugation process. The article also discusses potential applications of bacterial nanonetworks for cancer monitoring and therapy. © 2014 IEEE.
|
|
| 4. |
- Bandaragoda, Tharindu, et al.
(author)
-
Trajectory clustering of road traffic in urban environments using incremental machine learning in combination with hyperdimensional computing
- 2019
-
In: The 2019 IEEE Intelligent Transportation Systems Conference - ITSC. - : IEEE. - 9781538670248 - 9781538670255 ; , s. 1664-1670
-
Conference paper (peer-reviewed)abstract
- Road traffic congestion in urban environments poses an increasingly complex challenge of detection, profiling and prediction. Although public policy promotes transport alternatives and new infrastructure, traffic congestion is highly prevalent and continues to be the lead cause for numerous social, economic and environmental issues. Although a significant volume of research has been reported on road traffic prediction, profiling of traffic has received much less attention. In this paper we address two key problems in traffic profiling by proposing a novel unsupervised incremental learning approach for road traffic congestion detection and profiling, dynamically over time. This approach uses (a) hyperdimensional computing to enable capture variable-length trajectories of commuter trips represented as vehicular movement across intersections, and (b) transforms these into feature vectors that can be incrementally learned over time by the Incremental Knowledge Acquiring Self-Learning (IKASL) algorithm. The proposed approach was tested and evaluated on a dataset consisting of approximately 190 million vehicular movement records obtained from 1,400 Bluetooth identifiers placed at the intersections of the arterial road network in the State of Victoria, Australia.
|
|
| 5. |
- Bybee, Connor, et al.
(author)
-
Efficient optimization with higher-order ising machines
- 2023
-
In: Nature Communications. - : Nature Research. - 2041-1723. ; 14
-
Journal article (peer-reviewed)abstract
- A prominent approach to solving combinatorial optimization problems on parallel hardware is Ising machines, i.e., hardware implementations of networks of interacting binary spin variables. Most Ising machines leverage second-order interactions although important classes of optimization problems, such as satisfiability problems, map more seamlessly to Ising networks with higher-order interactions. Here, we demonstrate that higher-order Ising machines can solve satisfiability problems more resource-efficiently in terms of the number of spin variables and their connections when compared to traditional second-order Ising machines. Further, our results show on a benchmark dataset of Boolean k-satisfiability problems that higher-order Ising machines implemented with coupled oscillators rapidly find solutions that are better than second-order Ising machines, thus, improving the current state-of-the-art for Ising machines.
|
|
| 6. |
- Coelho Mollo, Dimitri, et al.
(author)
-
Beyond the Imitation Game : Quantifying and extrapolating the capabilities of language models
- 2023
-
In: Transactions on Machine Learning Research. ; :5
-
Journal article (peer-reviewed)abstract
- Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabilities are as yet poorly characterized. In order to inform future research, prepare for disruptive new model capabilities, and ameliorate socially harmful effects, it is vital that we understand the present and near-future capabilities and limitations of language models. To address this challenge, we introduce the Beyond the Imitation Game benchmark (BIG-bench). BIG-bench currently consists of 204 tasks, contributed by 442 authors across 132 institutions. Task topics are diverse, drawing problems from linguistics, childhood development, math, common-sense reasoning, biology, physics, social bias, software development, and beyond. BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models. We evaluate the behavior of OpenAI's GPT models, Google-internal dense transformer architectures, and Switch-style sparse transformers on BIG-bench, across model sizes spanning millions to hundreds of billions of parameters. In addition, a team of human expert raters performed all tasks in order to provide a strong baseline. Findings include: model performance and calibration both improve with scale, but are poor in absolute terms (and when compared with rater performance); performance is remarkably similar across model classes, though with benefits from sparsity; tasks that improve gradually and predictably commonly involve a large knowledge or memorization component, whereas tasks that exhibit "breakthrough" behavior at a critical scale often involve multiple steps or components, or brittle metrics; social bias typically increases with scale in settings with ambiguous context, but this can be improved with prompting.
|
|
| 7. |
- Dhole, Kaustubh, et al.
(author)
-
NL-Augmenter : A Framework for Task-Sensitive Natural Language Augmentation
- 2023
-
In: NEJLT Northern European Journal of Language Technology. - 2000-1533. ; 9:1, s. 1-41
-
Journal article (peer-reviewed)abstract
- Data augmentation is an important method for evaluating the robustness of and enhancing the diversity of training datafor natural language processing (NLP) models. In this paper, we present NL-Augmenter, a new participatory Python-based naturallanguage (NL) augmentation framework which supports the creation of transformations (modifications to the data) and filters(data splits according to specific features). We describe the framework and an initial set of117transformations and23filters for avariety of NL tasks annotated with noisy descriptive tags. The transformations incorporate noise, intentional and accidental humanmistakes, socio-linguistic variation, semantically-valid style, syntax changes, as well as artificial constructs that are unambiguousto humans. We demonstrate the efficacy of NL-Augmenter by using its transformations to analyze the robustness of popularlanguage models. We find different models to be differently challenged on different tasks, with quasi-systematic score decreases.The infrastructure, datacards, and robustness evaluation results are publicly available onGitHubfor the benefit of researchersworking on paraphrase generation, robustness analysis, and low-resource NLP.
|
|
| 8. |
- Diao, C., et al.
(author)
-
Generalized Learning Vector Quantization for Classification in Randomized Neural Networks and Hyperdimensional Computing
- 2021
-
In: <em>Proceedings of the International Joint Conference on Neural Networks</em>. - : Institute of Electrical and Electronics Engineers Inc..
-
Conference paper (peer-reviewed)abstract
- Machine learning algorithms deployed on edge devices must meet certain resource constraints and efficiency requirements. Random Vector Functional Link (RVFL) networks are favored for such applications due to their simple design and training efficiency. We propose a modified RVFL network that avoids computationally expensive matrix operations during training, thus expanding the network’s range of potential applications. Our modification replaces the least-squares classifier with the Generalized Learning Vector Quantization (GLVQ) classifier, which only employs simple vector and distance calculations. The GLVQ classifier can also be considered an improvement upon certain classification algorithms popularly used in the area of Hyperdimensional Computing. The proposed approach achieved state-of-the-art accuracy on a collection of datasets from the UCI Machine Learning Repository-higher than previously proposed RVFL networks. We further demonstrate that our approach still achieves high accuracy while severely limited in training iterations (using on average only 21% of the least-squares classifier computational costs).
|
|
| 9. |
- Frady, E. Paxon, et al.
(author)
-
A Theory of Sequence Indexing and Working Memory in Recurrent Neural Networks
- 2018
-
In: Neural Computation. - : MIT Press. - 0899-7667 .- 1530-888X. ; 30:6, s. 1449-1513
-
Journal article (peer-reviewed)abstract
- To accommodate structured approaches of neural computation, we propose a class of recurrent neural networks for indexing and storing sequences of symbols or analog data vectors. These networks with randomized input weights and orthogonal recurrent weights implement coding principles previously described in vector symbolic architectures (VSA) and leverage properties of reservoir computing. In general, the storage in reservoir computing is lossy, and cross-talk noise limits the retrieval accuracy and information capacity. A novel theory to optimize memory performance in such networks is presented and compared with simulation experiments. The theory describes linear readout of analog data and readout with winner-take-all error correction of symbolic data as proposed in VSA models. We find that diverse VSA models from the literature have universal performance properties, which are superior to what previous analyses predicted. Further, we propose novel VSA models with the statistically optimal Wiener filter in the readout that exhibit much higher information capacity, in particular for storing analog data. The theory we present also applies to memory buffers, networks with gradual forgetting, which can operate on infinite data streams without memory overflow. Interestingly, we find that different forgetting mechanisms, such as attenuating recurrent weights or neural nonlinearities, produce very similar behavior if the forgetting time constants are aligned. Such models exhibit extensive capacity when their forgetting time constant is optimized for given noise conditions and network size. These results enable the design of new types of VSA models for the online processing of data streams.
|
|
| 10. |
- Frady, E. P., et al.
(author)
-
Computing on Functions Using Randomized Vector Representations (in brief)
- 2022
-
In: ACM International Conference Proceeding Series. - New York, NY, USA : Association for Computing Machinery. - 9781450395595 ; , s. 115-122
-
Conference paper (peer-reviewed)abstract
- Vector space models for symbolic processing that encode symbols by random vectors have been proposed in cognitive science and connectionist communities under the names Vector Symbolic Architecture (VSA), and, synonymously, Hyperdimensional (HD) computing [22, 31, 46]. In this paper, we generalize VSAs to function spaces by mapping continuous-valued data into a vector space such that the inner product between the representations of any two data points approximately represents a similarity kernel. By analogy to VSA, we call this new function encoding and computing framework Vector Function Architecture (VFA). In VFAs, vectors can represent individual data points as well as elements of a function space (a reproducing kernel Hilbert space). The algebraic vector operations, inherited from VSA, correspond to well-defined operations in function space. Furthermore, we study a previously proposed method for encoding continuous data, fractional power encoding (FPE), which uses exponentiation of a random base vector to produce randomized representations of data points and fulfills the kernel properties for inducing a VFA. We show that the distribution from which components of the base vector are sampled determines the shape of the FPE kernel, which in turn induces a VFA for computing with band-limited functions. In particular, VFAs provide an algebraic framework for implementing large-scale kernel machines with random features, extending [51]. Finally, we demonstrate several applications of VFA models to problems in image recognition, density estimation and nonlinear regression. Our analyses and results suggest that VFAs constitute a powerful new framework for representing and manipulating functions in distributed neural systems, with myriad potential applications in artificial intelligence.
|
|
| 11. |
- Frady, Edward Paxon, et al.
(author)
-
Variable Binding for Sparse Distributed Representations : Theory and Applications
- 2023
-
In: IEEE Transactions on Neural Networks and Learning Systems. - : Institute of Electrical and Electronics Engineers Inc.. - 2162-237X .- 2162-2388. ; 34:5, s. 2191-2204
-
Journal article (peer-reviewed)abstract
- Variable binding is a cornerstone of symbolic reasoning and cognition. But how binding can be implemented in connectionist models has puzzled neuroscientists, cognitive psychologists, and neural network researchers for many decades. One type of connectionist model that naturally includes a binding operation is vector symbolic architectures (VSAs). In contrast to other proposals for variable binding, the binding operation in VSAs is dimensionality-preserving, which enables representing complex hierarchical data structures, such as trees, while avoiding a combinatoric expansion of dimensionality. Classical VSAs encode symbols by dense randomized vectors, in which information is distributed throughout the entire neuron population. By contrast, in the brain, features are encoded more locally, by the activity of single neurons or small groups of neurons, often forming sparse vectors of neural activation. Following Laiho et al. (2015), we explore symbolic reasoning with a special case of sparse distributed representations. Using techniques from compressed sensing, we first show that variable binding in classical VSAs is mathematically equivalent to tensor product binding between sparse feature vectors, another well-known binding operation which increases dimensionality. This theoretical result motivates us to study two dimensionality-preserving binding methods that include a reduction of the tensor matrix into a single sparse vector. One binding method for general sparse vectors uses random projections, the other, block-local circular convolution, is defined for sparse vectors with block structure, sparse block-codes. Our experiments reveal that block-local circular convolution binding has ideal properties, whereas random projection based binding also works, but is lossy. We demonstrate in example applications that a VSA with block-local circular convolution and sparse block-codes reaches similar performance as classical VSAs. Finally, we discuss our results in the context of neuroscience and neural networks.
|
|
| 12. |
- Grytsenko, Vladimir I., et al.
(author)
-
Neural Distributed Autoassociative Memories: A Survey.
- 2017
-
In: Cybernetics and Computer Engineering Journal. - : NASU-National Academy of Sciences of Ukraine. - 0454-9910 .- 2519-2205. ; 188:2, s. 5-35
-
Journal article (peer-reviewed)abstract
- Introduction. Neural network models of autoassociative, distributed memory allow storage and retrieval of many items (vectors) where the number of stored items can exceed the vector dimension (the number of neurons in the network). This opens the possibility of a sublinear time search (in the number of stored items) for approximate nearest neighbors among vectors of high dimension.The purpose of this paper is to review models of autoassociative, distributed memory that can be naturally implemented by neural networks (mainly with local learning rules and iterative dynamics based on information locally available to neurons).Scope. The survey is focused mainly on the networks of Hopfield, Willshaw and Potts, that have connections between pairs of neurons and operate on sparse binary vectors. We discuss not only autoassociative memory, but also the generalization properties of these networks. We also consider neural networks with higher-order connections and networks with a bipartite graph structure for non-binary data with linear constraints.Conclusions. In conclusion we discuss the relations to similarity search, advantages and drawbacks of these techniques, and topics for further research. An interesting and still not completely resolved question is whether neural autoassociative memories can search for approximate nearest neighbors faster than other index structures for similarity search, in particular for the case of very high dimensional vectors.
|
|
| 13. |
- Heddes, Mike, et al.
(author)
-
Torchhd: An Open Source Python Library to SupportResearch on Hyperdimensional Computing andVector Symbolic Architectures
- 2023
-
In: Journal of Machine Learning Research. ; 24, s. 1-10
-
Journal article (peer-reviewed)abstract
- Hyperdimensional computing (HD), also known as vector symbolic architectures (VSA), is a framework for computing with distributed representations by exploiting properties of random high-dimensional vector spaces. The commitment of the scientific community to aggregate and disseminate research in this particularly multidisciplinary area has been fundamental for its advancement. Joining these efforts, we present Torchhd, a highperformance open source Python library for HD/VSA. Torchhd seeks to make HD/VSA more accessible and serves as an efficient foundation for further research and application development. The easy-to-use library builds on top of PyTorch and features state-of-the art HD/VSA functionality, clear documentation, and implementation examples from wellknown publications. Comparing publicly available code with their corresponding Torchhd implementation shows that experiments can run up to 100× faster. Torchhd is available at: https://github.com/hyperdimensional-computing/torchhd.
|
|
| 14. |
- Karvonen, Niklas, 1979-, et al.
(author)
-
A Domain Knowledge-Based Solution for Human Activity Recognition : The UJA Dataset Analysis
- 2018
-
In: The 12th International Conference on Ubiquitous Computing and Ambient Intelligence (UCAmI 2018). - Basel Switzerland : MDPI.
-
Conference paper (peer-reviewed)abstract
- Detecting activities of daily living (ADL) allows for rich inference about user behavior, which can be of use in the care of for example, elderly people, chronic diseases, and psychological conditions. This paper proposes a domain knowledge-based solution for detecting 24 different ADLs in the UJA dataset. The solution is inspired by a Finite State Machine and performs activity recognition unobtrusively using only binary sensors. Each day in the dataset is segmented into: morning, day, evening in order to facilitate the inference from the sensors. The model performs the ADL recognition in two steps. The first step is to detect the sequence of activities in a given event stream of binary sensors, and the second step is to assign a starting and ending times for each of detected activities. Our proposed model achieved an accuracy of 81.3% using only a very small amount of operations, making it an interesting approach for resource-constrained devices that are common in smart environments. It should be noted, however, that the model can end up in faulty states which could cause a series of mis-classifications before the model is returned to the true state.
|
|
| 15. |
- Karvonen, Niklas, 1979-, et al.
(author)
-
Low-Power Classification using FPGA : An Approach based on Cellular Automata, Neural Networks, and Hyperdimensional Computing
- 2019
-
In: 2019 18th IEEE International Conference On Machine Learning And Applications (ICMLA). - : IEEE. - 9781728145501 ; , s. 370-375
-
Conference paper (other academic/artistic)abstract
- Field-Programmable Gate Arrays (FPGA) are hardware components that hold several desirable properties for wearable and Internet of Things (IoT) devices. They offer hardware implementations of algorithms using parallel computing, which can be used to increase battery life or achieve short response-times. Further, they are re-programmable and can be made small, power-efficient and inexpensive. In this paper we propose a classifier targeted specifically for implementation on FPGAs by using principles from hyperdimensional computing and cellular automata. The proposed algorithm is shown to perform on par with Naive Bayes for two benchmark datasets while also being robust to noise. It is also synthesized to a commercially available off-the-shelf FPGA reaching over 57.1 million classifications per second for a 3-class problem using 40 input features of 8 bits each. The results in this paper show that the proposed classifier could be a viable option for applications demanding low power-consumption, fast real-time responses, or a robustness against post-training noise.
|
|
| 16. |
- Kleyko, Denis, 1990-, et al.
(author)
-
A Comprehensive Study of Complexity and Performance of Automatic Detection of Atrial Fibrillation : Classification of Long ECG Recordings Based on the PhysioNet Computing in Cardiology Challenge 2017
- 2020
-
In: Biomedical Engineering & Physics Express. - : Institute of Physics Publishing (IOPP). - 2057-1976. ; 6:2
-
Journal article (peer-reviewed)abstract
- Objective: The 2017 PhysioNet/CinC Challenge focused on automatic classification of atrial fibrillation (AF) in short ECGs. This study aimed to evaluate the use of the data and results from the challenge for detection of AF in longer ECGs, taken from three other PhysioNet datasets.Approach: The used data-driven models were based on features extracted from ECG recordings, calculated according to three solutions from the challenge. A Random Forest classifier was trained with the data from the challenge. The performance was evaluated on all non-overlapping 30 s segments in all recordings from three MIT-BIH datasets. Fifty-six models were trained using different feature sets, both before and after applying three feature reduction techniques.Main Results: Based on rhythm annotations, the AF proportion was 0.00 in the MIT-BIH Normal Sinus Rhythm (N = 46083 segments), 0.10 in the MIT-BIH Arrhythmia (N = 2880), and 0.41 in the MIT-BIH Atrial Fibrillation (N = 28104) dataset. For the best performing model, the corresponding detected proportions of AF were 0.00, 0.11 and 0.36 using all features, and 0.01, 0.10 and 0.38 when using the 15 best performing features.Significance: The results obtained on the MIT-BIH datasets indicate that the training data and solutions from the 2017 Physionet/Cinc Challenge can be useful tools for developing robust AF detectors also in longer ECG recordings, even when using a low number of carefully selected features. The use of feature selection allows significantly reducing the number of features while preserving the classification performance, which can be important when building low-complexity AF classifiers on ECG devices with constrained computational and energy resources.
|
|
| 17. |
- Kleyko, Denis, 1990-, et al.
(author)
-
A Hyperdimensional Computing Framework for Analysis of Cardiorespiratory Synchronization during Paced Deep Breathing
- 2019
-
In: IEEE Access. - : IEEE. - 2169-3536. ; 7, s. 34403-34415
-
Journal article (peer-reviewed)abstract
- Objective: Autonomic function during deep breathing (DB) is normally scored based on the assumption that the heart rate is synchronized with the breathing. We have observed individuals with subtle arrhythmias during DB where autonomic function cannot be evaluated. This study presents a novel method for analyzing cardiorespiratory synchronization: feature-based analysis of the similarity between heart rate and respiration using principles of hyperdimensional computing. Methods: Heart rate and respiration signals were modeled using Fourier series analysis. Three feature variables were derived and mapped to binary vectors in a high-dimensional space. Using both synthesized data and recordings from patients/healthy subjects, the similarity between the feature vectors was assessed using Hamming distance (high-dimensional space), Euclidean distance (original space), and with a coherence-based index. Methods were evaluated via classification of the similarity indices into three groups. Results: The distance-based methods achieved good separation of signals into classes with different degree of cardiorespiratory synchronization, also providing identification of patients with low cardiorespiratory synchronization but high values of conventional DB scores. Moreover, binary high-dimensional vectors allowed an additional analysis of the obtained Hamming distance. Conclusions: Feature-based similarity analysis using hyperdimensional computing is capable of identifying signals with low cardiorespiratory synchronization during DB due to arrhythmias. Vector-based similarity analysis could be applied to other types of feature variables than based on spectral analysis. Significance: The proposed methods for robustly assessing cardiorespiratory synchronization during DB facilitate the identification of individuals where the evaluation of autonomic function is problematic or even impossible, thus, increasing the correctness of the conventional DB scores.
|
|
| 18. |
- Kleyko, Denis, et al.
(author)
-
A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part I: Models and Data Transformations
- 2023
-
In: ACM Computing Surveys. - : Association for Computing Machinery (ACM). - 0360-0300 .- 1557-7341. ; 55:6
-
Journal article (peer-reviewed)abstract
- This two-part comprehensive survey is devoted to a computing framework most commonly known under the names Hyperdimensional Computing and Vector Symbolic Architectures (HDC/VSA). Both names refer to a family of computational models that use high-dimensional distributed representations and rely on the algebraic properties of their key operations to incorporate the advantages of structured symbolic representations and distributed vector representations. Notable models in the HDC/VSA family are Tensor Product Representations, Holographic Reduced Representations, Multiply-Add-Permute, Binary Spatter Codes, and Sparse Binary Distributed Representations but there are other models too. HDC/VSA is a highly interdisciplinary field with connections to computer science, electrical engineering, artificial intelligence, mathematics, and cognitive science. This fact makes it challenging to create a thorough overview of the field. However, due to a surge of new researchers joining the field in recent years, the necessity for a comprehensive survey of the field has become extremely important. Therefore, amongst other aspects of the field, this Part I surveys important aspects such as: known computational models of HDC/VSA and transformations of various input data types to high-dimensional distributed representations. Part II of this survey [84] is devoted to applications, cognitive computing and architectures, as well as directions for future work. The survey is written to be useful for both newcomers and practitioners.
|
|
| 19. |
- Kleyko, Denis, et al.
(author)
-
A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part II : Applications, Cognitive Models, and Challenges
- 2023
-
In: ACM Computing Surveys. - : Association for Computing Machinery. - 0360-0300 .- 1557-7341. ; 55:9
-
Journal article (peer-reviewed)abstract
- This is Part II of the two-part comprehensive survey devoted to a computing framework most commonly known under the names Hyperdimensional Computing and Vector Symbolic Architectures (HDC/VSA). Both names refer to a family of computational models that use high-dimensional distributed representations and rely on the algebraic properties of their key operations to incorporate the advantages of structured symbolic representations and vector distributed representations. Holographic Reduced Representations [321, 326] is an influential HDC/VSA model that is well known in the machine learning domain and often used to refer to the whole family. However, for the sake of consistency, we use HDC/VSA to refer to the field.Part I of this survey [222] covered foundational aspects of the field, such as the historical context leading to the development of HDC/VSA, key elements of any HDC/VSA model, known HDC/VSA models, and the transformation of input data of various types into high-dimensional vectors suitable for HDC/VSA. This second part surveys existing applications, the role of HDC/VSA in cognitive computing and architectures, as well as directions for future work. Most of the applications lie within the Machine Learning/Artificial Intelligence domain; however, we also cover other applications to provide a complete picture. The survey is written to be useful for both newcomers and practitioners.
|
|
| 20. |
- Kleyko, Denis, 1990-, et al.
(author)
-
Autoscaling Bloom filter : controlling trade-off between true and false positives
- 2020
-
In: Neural Computing & Applications. - : Springer. - 0941-0643 .- 1433-3058. ; 32:8, s. 3675-3684
-
Journal article (peer-reviewed)abstract
- A Bloom filter is a special case of an artificial neural network with two layers. Traditionally, it is seen as a simple data structure supporting membership queries on a set. The standard Bloom filter does not support the delete operation, and therefore, many applications use a counting Bloom filter to enable deletion. This paper proposes a generalization of the counting Bloom filter approach, called “autoscaling Bloom filters”, which allows adjustment of its capacity with probabilistic bounds on false positives and true positives. Thus, by relaxing the requirement on perfect true positive rate, the proposed autoscaling Bloom filter addresses the major difficulty of Bloom filters with respect to their scalability. In essence, the autoscaling Bloom filter is a binarized counting Bloom filter with an adjustable binarization threshold. We present the mathematical analysis of its performance and provide a procedure for minimizing its false positive rate.
|
|
| 21. |
- Kleyko, Denis, et al.
(author)
-
Brain-like classifier of temporal patterns
- 2014
-
In: International Conference on Computer and Information Sciences, ICCOINS 2014, Kuala Lumpur, Malaysia, June 03-05, 2014. Proceedings. - Piscataway, NJ : IEEE Communications Society. - 9781479943913 ; , s. 1-6
-
Conference paper (peer-reviewed)abstract
- In this article we present a pattern classification system which uses Vector Symbolic Architecture (VSA) for representation, learning and subsequent classification of patterns, as a showcase we have used classification of vibration sensors measurements to vehicles types. On the quantitative side the proposed classifier requires only 1 kB of memory to classify an incoming signal against of several hundred of training samples. The classification operation into N types requires only 2*N+1 arithmetic operations this makes the proposed classifier feasible for implementation on a low-end sensor nodes. The main contribution of this article is the proposed methodology for representing temporal patterns with distributed representation and VSA-based classifier.
|
|
| 22. |
- Kleyko, Denis, et al.
(author)
-
Cellular Automata Can Reduce Memory Requirements of Collective-State Computing
- 2022
-
In: IEEE Transactions on Neural Networks and Learning Systems. - : Institute of Electrical and Electronics Engineers Inc.. - 2162-237X .- 2162-2388. ; 33:6, s. 2701-2713
-
Journal article (peer-reviewed)abstract
- Various nonclassical approaches of distributed information processing, such as neural networks, reservoir computing (RC), vector symbolic architectures (VSAs), and others, employ the principle of collective-state computing. In this type of computing, the variables relevant in computation are superimposed into a single high-dimensional state vector, the collective state. The variable encoding uses a fixed set of random patterns, which has to be stored and kept available during the computation. In this article, we show that an elementary cellular automaton with rule 90 (CA90) enables the space-time tradeoff for collective-state computing models that use random dense binary representations, i.e., memory requirements can be traded off with computation running CA90. We investigate the randomization behavior of CA90, in particular, the relation between the length of the randomization period and the size of the grid, and how CA90 preserves similarity in the presence of the initialization noise. Based on these analyses, we discuss how to optimize a collective-state computing model, in which CA90 expands representations on the fly from short seed patterns--rather than storing the full set of random patterns. The CA90 expansion is applied and tested in concrete scenarios using RC and VSAs. Our experimental results show that collective-state computing with CA90 expansion performs similarly compared to traditional collective-state models, in which random patterns are generated initially by a pseudorandom number generator and then stored in a large memory.
|
|
| 23. |
- Kleyko, Denis, 1990-, et al.
(author)
-
Classification and Recall With Binary Hyperdimensional Computing : Tradeoffs in Choice of Density and Mapping Characteristics
- 2018
-
In: IEEE Transactions on Neural Networks and Learning Systems. - : IEEE. - 2162-237X .- 2162-2388. ; 29:12, s. 5880-5898
-
Journal article (peer-reviewed)abstract
- Hyperdimensional (HD) computing is a promising paradigm for future intelligent electronic appliances operating at low power. This paper discusses tradeoffs of selecting parameters of binary HD representations when applied to pattern recognition tasks. Particular design choices include density of representations and strategies for mapping data from the original representation. It is demonstrated that for the considered pattern recognition tasks (using synthetic and real-world data) both sparse and dense representations behave nearly identically. This paper also discusses implementation peculiarities which may favor one type of representations over the other. Finally, the capacity of representations of various densities is discussed.
|
|
| 24. |
- Kleyko, Denis, et al.
(author)
-
Comparison of Machine Learning Techniques for Vehicle Classification using Road Side Sensors
- 2015
-
In: Proceedings of the 2015 IEEE 18th International Conference on Intelligent Transportation Systems. - Piscataway, NJ : IEEE Communications Society. - 9781467365956 ; , s. 572-577
-
Conference paper (peer-reviewed)abstract
- The main contribution of this paper is a comparison of different machine learning algorithms for vehicle classification according to the "Nordic system for intelligent classification of vehicles" standard using measurements of road surface vibrations and magnetic field disturbances caused by vehicles. The algorithms considered are logistic regression, neural networks, and support vector machines. They are evaluated on a large dataset, consisting of 3074 samples and hence, a good estimate of the actual classification rate is obtained. The results show that for the considered classification problem logistic regression is the best choice with an overall classification rate of 93.4%.
|
|
| 25. |
- Kleyko, Denis, et al.
(author)
-
Density Encoding Enables Resource-Efficient Randomly Connected Neural Networks
- 2021
-
In: IEEE Transactions on Neural Networks and Learning Systems. - : Institute of Electrical and Electronics Engineers Inc.. - 2162-237X .- 2162-2388. ; 32:8, s. 3777-3783
-
Journal article (peer-reviewed)abstract
- The deployment of machine learning algorithms on resource-constrained edge devices is an important challenge from both theoretical and applied points of view. In this brief, we focus on resource-efficient randomly connected neural networks known as random vector functional link (RVFL) networks since their simple design and extremely fast training time make them very attractive for solving many applied classification tasks. We propose to represent input features via the density-based encoding known in the area of stochastic computing and use the operations of binding and bundling from the area of hyperdimensional computing for obtaining the activations of the hidden neurons. Using a collection of 121 real-world data sets from the UCI machine learning repository, we empirically show that the proposed approach demonstrates higher average accuracy than the conventional RVFL. We also demonstrate that it is possible to represent the readout matrix using only integers in a limited range with minimal loss in the accuracy. In this case, the proposed approach operates only on small ${n}$ -bits integers, which results in a computationally efficient architecture. Finally, through hardware field-programmable gate array (FPGA) implementations, we show that such an approach consumes approximately 11 times less energy than that of the conventional RVFL.
|
|
| 26. |
- Kleyko, Denis, et al.
(author)
-
Dependable MAC layer architecture based on holographic data representation using hyper-dimensional binary spatter codes
- 2012
-
In: Multiple Access Communications. - Heidelberg : Encyclopedia of Global Archaeology/Springer Verlag. - 9783642349751 - 9783642349768 ; , s. 134-145
-
Conference paper (peer-reviewed)abstract
- In this article we propose the usage of binary spatter codes and distributed data representation for communicating loss and delay sensitive data in event-driven sensor and actuator networks. Using the proposed data representation technique along with the medium access control protocol the mission critical control information can be transmitted with assured constant delay in deployments exposing below 0 dB signal-to-noise ratio figures.
|
|
| 27. |
- Kleyko, Denis, 1990-, et al.
(author)
-
Distributed Representation of n-gram Statistics for Boosting Self-organizing Maps with Hyperdimensional Computing
- 2019
-
In: Perspectives of System Informatics. - Cham : Springer. ; , s. 64-79, s. 64-79
-
Conference paper (peer-reviewed)abstract
- This paper presents an approach for substantial reduction of the training and operating phases of Self-Organizing Maps in tasks of 2-D projection of multi-dimensional symbolic data for natural language processing such as language classification, topic extraction, and ontology development. The conventional approach for this type of problem is to use n-gram statistics as a fixed size representation for input of Self-Organizing Maps. The performance bottleneck with n-gram statistics is that the size of representation and as a result the computation time of Self-Organizing Maps grows exponentially with the size of n-grams. The presented approach is based on distributed representations of structured data using principles of hyperdimensional computing. The experiments performed on the European languages recognition task demonstrate that Self-Organizing Maps trained with distributed representations require less computations than the conventional n-gram statistics while well preserving the overall performance of Self-Organizing Maps.
|
|
| 28. |
- Kleyko, Denis, et al.
(author)
-
Efficient Decoding of Compositional Structure in Holistic Representations
- 2023
-
In: Neural Computation. - : MIT Press Journals. - 0899-7667 .- 1530-888X. ; 35:7, s. 1159-1186
-
Journal article (peer-reviewed)abstract
- We investigate the task of retrieving information from compositional distributed representations formed by hyperdimensional computing/vector symbolic architectures and present novel techniques that achieve new information rate bounds. First, we provide an overview of the decoding techniques that can be used to approach the retrieval task. The techniques are categorized into four groups. We then evaluate the considered techniques in several settings that involve, for example, inclusion of external noise and storage elements with reduced precision. In particular, we find that the decoding techniques from the sparse coding and compressed sensing literature (rarely used for hyperdimensional computing/vector symbolic architectures) are also well suited for decoding information from the compositional distributed representations. Combining these decoding techniques with interference cancellation ideas from communications improves previously reported bounds (Hersche et al., 2021) of the information rate of the distributed representations from 1.20 to 1.40 bits per dimension for smaller codebooks and from 0.60 to 1.26 bits per dimension for larger codebooks.
|
|
| 29. |
- Kleyko, Denis, et al.
(author)
-
Fault Detection in the Hyperspace : Towards Intelligent Automation Systems
- 2015
-
In: IEEE International Conference on Industrial Informatics. - Piscataway, NJ : IEEE Communications Society. - 9781479966493 ; , s. 1219-1224
-
Conference paper (peer-reviewed)abstract
- This article presents a methodology for intelligent, biologically inspired fault detection system for generic complex systems of systems. The proposed methodology utilizes the concepts of associative memory and vector symbolic architectures, commonly used for modeling cognitive abilities of human brain. Compared to classical methods of artificial intelligence used in the context of fault detection the proposed methodology shows an unprecedented performance, while featuring zero configuration and simple operations.
|
|
| 30. |
- Kleyko, Denis, et al.
(author)
-
Fly-The-Bee: A Game Imitating Concept Learning in Bees
- 2015
-
In: Procedia Computer Science. - : Elsevier BV. - 1877-0509. ; 71, s. 25-30
-
Journal article (peer-reviewed)abstract
- This article presents a web-based game functionally imitating a part of the cognitive behavior of a living organism. This game is a prototype implementation of an artificial online cognitive architecture based on the usage of distributed data representations and Vector Symbolic Architectures. The game emonstrates the feasibility of creating a lightweight cognitive architecture, which is capable of performing rather complex cognitive tasks. The cognitive functionality is implemented in about 100 lines of code and requires few tens of kilobytes of memory for its operation, which make the concept suitable for implementing in low-end devices such as minirobots and wireless sensors.
|
|
| 31. |
- Kleyko, Denis, et al.
(author)
-
Generalized Key-Value Memory to Flexibly Adjust Redundancy in Memory-Augmented Networks
- 2023
-
In: IEEE Transactions on Neural Networks and Learning Systems. - : Institute of Electrical and Electronics Engineers (IEEE). - 2162-237X .- 2162-2388. ; 34:12, s. 10993-10998
-
Journal article (peer-reviewed)abstract
- Memory-augmented neural networks enhance a neural network with an external key-value (KV) memory whose complexity is typically dominated by the number of support vectors in the key memory. We propose a generalized KV memory that decouples its dimension from the number of support vectors by introducing a free parameter that can arbitrarily add or remove redundancy to the key memory representation. In effect, it provides an additional degree of freedom to flexibly control the tradeoff between robustness and the resources required to store and compute the generalized KV memory. This is particularly useful for realizing the key memory on in-memory computing hardware where it exploits nonideal, but extremely efficient nonvolatile memory devices for dense storage and computation. Experimental results show that adapting this parameter on demand effectively mitigates up to 44% nonidealities, at equal accuracy and number of devices, without any need for neural network retraining.
|
|
| 32. |
- Kleyko, Denis, et al.
(author)
-
Holographic Graph Neuron: a Bio-Inspired Architecture for Pattern Processing
- 2017
-
In: IEEE Transactions on Neural Networks and Learning Systems. - : IEEE. - 2162-237X .- 2162-2388. ; 28:6, s. 1250-1262
-
Journal article (peer-reviewed)abstract
- This article proposes the use of Vector Symbolic Architectures for implementing Hierarchical Graph Neuron, an architecture for memorizing patterns of generic sensor stimuli. The adoption of a Vector Symbolic representation ensures a one-layered design for the approach, while maintaining the previously reported properties and performance characteristics of Hierarchical Graph Neuron, and also improving the noise resistance of the architecture. The proposed architecture enables a linear (with respect to the number of stored entries) time search for an arbitrary sub-pattern.
|
|
| 33. |
- Kleyko, Denis, 1990-, et al.
(author)
-
Hyperdimensional computing in industrial systems : the use-case of distributed fault isolation in a power plant
- 2018
-
In: IEEE Access. - : IEEE. - 2169-3536. ; 6, s. 30766-30777
-
Journal article (peer-reviewed)abstract
- This paper presents an approach for distributed fault isolation in a generic system of systems. The proposed approach is based on the principles of hyperdimensional computing. In particular, the recently proposed method called Holographic Graph Neuron is used. We present a distributed version of Holographic Graph Neuron and evaluate its performance on the problem of fault isolation in a complex power plant model. Compared to conventional machine learning methods applied in the context of the same scenario the proposed approach shows comparable performance while being distributed and requiring simple binary operations, which allow for a fast and efficient implementation in hardware.
|
|
| 34. |
- Kleyko, Denis, et al.
(author)
-
Imitation of honey bees’ concept learning processes using Vector Symbolic Architectures
- 2015
-
In: Biologically Inspired Cognitive Architectures. - : Elsevier BV. - 2212-683X .- 2212-6848. ; 14, s. 57-72
-
Journal article (peer-reviewed)abstract
- This article presents a proof-of-concept validation of the use of Vector Symbolic Architectures as central component of an online learning architectures. It is demonstrated that Vector Symbolic Architectures enable the structured combination of features/relations that have been detected by a perceptual circuitry and allow such relations to be applied to novel structures without requiring the massive training needed for classical neural networks that depend on trainable connections.The system is showcased through the functional imitation of concept learning in honey bees. Data from real-world experiments with honey bees (Avarguès-Weber et al., 2012) are used for benchmarking. It is demonstrated that the proposed pipeline features a similar learning curve and accuracy of generalization to that observed for the living bees. The main claim of this article is that there is a class of simple artificial systems that reproduce the learning behaviors of certain living organisms without requiring the implementation of computationally intensive cognitive architectures. Consequently, it is possible in some cases to implement rather advanced cognitive behavior using simple techniques.
|
|
| 35. |
- Kleyko, Denis, et al.
(author)
-
Integer Echo State Networks : Efficient Reservoir Computing for Digital Hardware
- 2022
-
In: IEEE Transactions on Neural Networks and Learning Systems. - : Institute of Electrical and Electronics Engineers Inc.. - 2162-237X .- 2162-2388. ; 33:4, s. 1688-1701
-
Journal article (peer-reviewed)abstract
- We propose an approximation of echo state networks (ESNs) that can be efficiently implemented on digital hardware based on the mathematics of hyperdimensional computing. The reservoir of the proposed integer ESN (intESN) is a vector containing only n-bits integers (where n< 8 is normally sufficient for a satisfactory performance). The recurrent matrix multiplication is replaced with an efficient cyclic shift operation. The proposed intESN approach is verified with typical tasks in reservoir computing: memorizing of a sequence of inputs, classifying time series, and learning dynamic processes. Such architecture results in dramatic improvements in memory footprint and computational efficiency, with minimal performance loss. The experiments on a field-programmable gate array confirm that the proposed intESN approach is much more energy efficient than the conventional ESN.
|
|
| 36. |
- Kleyko, Denis, et al.
(author)
-
Integer Factorization with Compositional Distributed Representations
- 2022
-
In: ACM International Conference Proceeding Series. - New York, NY, USA : Association for Computing Machinery. - 9781450395595 ; , s. 73-80
-
Conference paper (peer-reviewed)abstract
- In this paper, we present an approach to integer factorization using distributed representations formed with Vector Symbolic Architectures. The approach formulates integer factorization in a manner such that it can be solved using neural networks and potentially implemented on parallel neuromorphic hardware. We introduce a method for encoding numbers in distributed vector spaces and explain how the resonator network can solve the integer factorization problem. We evaluate the approach on factorization of semiprimes by measuring the factorization accuracy versus the scale of the problem. We also demonstrate how the proposed approach generalizes beyond the factorization of semiprimes; in principle, it can be used for factorization of any composite number. This work demonstrates how a well-known combinatorial search problem may be formulated and solved within the framework of Vector Symbolic Architectures, and it opens the door to solving similarly difficult problems in other domains.
|
|
| 37. |
- Kleyko, Denis, 1990-, et al.
(author)
-
Integer Self-Organizing Maps for Digital Hardware
- 2019
-
In: 2019 International Joint Conference on Neural Networks (IJCNN). - : IEEE. - 9781728119854
-
Conference paper (peer-reviewed)abstract
- The Self-Organizing Map algorithm has been proven and demonstrated to be a useful paradigm for unsupervised machine learning of two-dimensional projections of multidimensional data. The tri-state Self-Organizing Maps have been proposed as an accelerated resource-efficient alternative to the Self-Organizing Maps for implementation on field-programmable gate array (FPGA) hardware. This paper presents a generalization of the tri-state Self-Organizing Maps. The proposed generalization, which we call integer Self-Organizing Maps, requires only integer operations for weight updates. The presented experiments demonstrated that the integer Self-Organizing Maps achieve better accuracy in a classification task when compared to the original tri-state Self-Organizing Maps.
|
|
| 38. |
- Kleyko, Denis, et al.
(author)
-
Modality Classification of Medical Images with Distributed Representations Based on Cellular Automata Reservoir Computing
- 2017
-
In: Proceedings - International Symposium on Biomedical Imaging. - Piscataway, NJ : Institute of Electrical and Electronics Engineers (IEEE). - 9781509011711 ; , s. 1053-1056
-
Conference paper (peer-reviewed)abstract
- Modality corresponding to medical images is a vital filter in medical image retrieval systems. This article presents the classification of modalities of medical images based on the usage of principles of hyper-dimensional computing and reservoir computing. It is demonstrated that the highest classification accuracy of the proposed method is on a par with the best classical method for the given dataset (83% vs. 84%). The major positive property of the proposed method is that it does not require any optimization routine during the training phase and naturally allows for incremental learning upon the availability of new training data.
|
|
| 39. |
- Kleyko, Denis, et al.
(author)
-
Modification of Holographic Graph Neuron using Sparse Distributed Representations
- 2016
-
In: Procedia Computer Science. - : Elsevier. - 1877-0509. ; 88, s. 39-45
-
Journal article (peer-reviewed)abstract
- This article presents a modification of the recently proposed Holographic Graph Neuron approach for memorizing patterns of generic sensor stimuli. The original approach represents patterns as dense binary vectors, where zeros and ones are equiprobable. The presented modification employs sparse binary distributed representations where the number of ones is less than zeros. Sparse representations are more biologically plausible because activities of real neuronsare sparse. Performance was studied comparing approaches for different sizes of dimensionality.
|
|
| 40. |
- Kleyko, Denis, et al.
(author)
-
Modified Algorithm of Dynamic Frequency Hopping (DFH) in the IEEE 802.22 Standard
- 2014
-
In: Multiple Access Communications. - Cham : Springer. - 9783319102610 - 9783319102627 ; , s. 75-83
-
Conference paper (peer-reviewed)abstract
- IEEE 802.22 Cognitive Wireless Regional Area Networks is a first standard of wireless terrestrial system relying on cognitive radio concept and operating as an opportunistic system in the the vacant unoccupied frequency spaces of the licensed TV-frequency band. Concept of the proposed standard assumes special functionality to protect the operation of the primary licensed subscribers. Dynamic Frequency Hopping is the mechanism for providing connectionless operation of Wireless Regional Area Networks systems while ensuring protection of transmissions from the primary users. During its operation regular time gaps appear on the involved frequency channels. This paper introduces the concept of the efficient reuse of the vacant frequency resources appearing when using the Dynamic Frequency Hopping mode. The scheme for consecutive-parallel inclusion of the new Dynamic Frequency Hopping Communities-members in the Dynamic Frequency Hopping mode is presented. The proposed approach allows significantly decrease time of inclusion the new members into a new Dynamic Frequency Hopping Communities. © Springer International Publishing Switzerland 2014
|
|
| 41. |
- Kleyko, Denis, et al.
(author)
-
No Two Brains Are Alike : Cloning a Hyperdimensional Associative Memory Using Cellular Automata Computations
- 2018
-
In: Biologically Inspired Cognitive Architectures (BICA) for Young Scientists. - Cham : Springer. - 9783319639390 - 9783319639406 ; , s. 91-100
-
Conference paper (peer-reviewed)abstract
- This paper looks beyond of the current focus of research on biologically inspired cognitive systems and considers the problem of replication of its learned functionality. The considered challenge is to replicate the learned knowledge such that uniqueness of the internal symbolic representations is guaranteed. This article takes a neurological argument “no two brains are alike” and suggests an architecture for mapping a content of the trained associative memory built using principles of hyperdimensional computing and Vector Symbolic Architectures into a new and orthogonal basis of atomic symbols. This is done with the help of computations on cellular automata. The results of this article open a way towards a secure usage of cognitive architectures in a variety of practical application domains.
|
|
| 42. |
- Kleyko, Denis, et al.
(author)
-
On bidirectional transitions between localist and distributed representations : The case of common substrings search using Vector Symbolic Architecture
- 2014
-
In: Procedia Computer Science. - : Elsevier BV. - 1877-0509. ; 41, s. 104-113
-
Journal article (peer-reviewed)abstract
- The contribution of this article is twofold. First, it presents an encoding approach for seamless bidirectional transitions between localist and distributed representation domains. Second, the approach is demonstrated on the example of using Vector Symbolic Architecture for solving a problem of finding common substrings. The proposed algorithm uses elementary operations on long binary vectors. For the case of two patterns with respective lengths L1 and L2 it requires Θ(L1 + L2 – 1) operations on binary vectors, which is equal to the suffix trees approach – the fastest algorithm for this problem. The simulation results show that in order to be robustly detected by the proposed approach the length of a common substring should be more than 4% of the longest pattern.
|
|
| 43. |
- Kleyko, Denis, et al.
(author)
-
On Methodology of Implementing Distributed Function Block Applications using TinyOS WSN nodes
- 2014
-
In: Proceedings of 2014 IEEE 19th International Conference on Emerging Technologies & Factory Automation (ETFA 2014). - Piscataway, NJ : IEEE Communications Society. - 9781479948451
-
Conference paper (peer-reviewed)abstract
- This paper presents a feasibility study of implementing parts of a distributed function block application as TinyOS modules running on Wireless Sensors as a part of Wireless Sensor Network. The paper first briefly describes underlying technologies and gives motivation for implementation of function blocks in TinyOS. The paper then presents implementation details about TinyOS realization of the one of the function block, which is a part of bigger distributed control application with the help of distributed function block application.
|
|
| 44. |
- Kleyko, Denis
(author)
-
Pattern Recognition with Vector Symbolic Architectures
- 2016
-
Licentiate thesis (other academic/artistic)abstract
- Pattern recognition is an area constantly enlarging its theoretical and practical horizons. Applications of pattern recognition and machine learning can be found in many areas of the present day world including health-care, robotics, manufacturing, economics, automation, transportation, etc. Despite some success in many domains pattern recognition algorithms are still far from being close to their biological vis-a-vis – human brain. New possibilities in the area of pattern recognition may be achieved by application of biologically inspired approaches. This thesis presents the usage of a bio-inspired method of representing concepts and their meaning – Vector Symbolic Architectures – in the context of pattern recognition with possible applications in intelligent transportation systems, automation systems, and language processing. Vector Symbolic Architectures is an approach for encoding and manipulating distributed representations of information. They have previously been used mainly in the area of cognitive computing for representing and reasoning upon semantically bound information. First, it is shown that Vector Symbolic Architectures are capable of pattern classification of temporal patterns. With this approach, it is possible to represent, learn and subsequently classify vehicles using measurements from vibration sensors.Next, an architecture called Holographic Graph Neuron for one-shot learning of patterns of generic sensor stimuli is proposed. The architecture is based on implementing the Hierarchical Graph Neuron approach using Vector Symbolic Architectures. Holographic Graph Neuron shows the previously reported performance characteristics of Hierarchical Graph Neuron while maintaining the simplicity of its design. The Holographic Graph Neuron architecture is applied in two domains: fault detection and longest common substrings search. In the area of fault detection the architecture showed superior performance compared to classical methods of artificial intelligence while featuring zero configuration and simple operations. The application of the architecture for longest common substrings search showed its ability to robustly solve the task given that the length of a common substring is longer than 4% of the longest pattern. Furthermore, the required number of operations on binary vectors is equal to the suffix trees approach, which is the fastest traditional algorithm for this problem. In summary, the work presented in this thesis extends understanding of the performance proprieties of distributed representations and opens the way for new applications.
|
|
| 45. |
- Kleyko, Denis, et al.
(author)
-
Perceptron Theory Can Predict the Accuracy of Neural Networks
- 2023
-
In: IEEE Transactions on Neural Networks and Learning Systems. - 2162-237X .- 2162-2388.
-
Journal article (peer-reviewed)abstract
- Multilayer neural networks set the current state of the art for many technical classification problems. But, these networks are still, essentially, black boxes in terms of analyzing them and predicting their performance. Here, we develop a statistical theory for the one-layer perceptron and show that it can predict performances of a surprisingly large variety of neural networks with different architectures. A general theory of classification with perceptrons is developed by generalizing an existing theory for analyzing reservoir computing models and connectionist models for symbolic reasoning known as vector symbolic architectures. Our statistical theory offers three formulas leveraging the signal statistics with increasing detail. The formulas are analytically intractable, but can be evaluated numerically. The description level that captures maximum details requires stochastic sampling methods. Depending on the network model, the simpler formulas already yield high prediction accuracy. The quality of the theory predictions is assessed in three experimental settings, a memorization task for echo state networks (ESNs) from reservoir computing literature, a collection of classification datasets for shallow randomly connected networks, and the ImageNet dataset for deep convolutional neural networks. We find that the second description level of the perceptron theory can predict the performance of types of ESNs, which could not be described previously. Furthermore, the theory can predict deep multilayer neural networks by being applied to their output layer. While other methods for prediction of neural networks performance commonly require to train an estimator model, the proposed theory requires only the first two moments of the distribution of the postsynaptic sums in the output neurons. Moreover, the perceptron theory compares favorably to other methods that do not rely on training an estimator model.
|
|
| 46. |
- Kleyko, Denis, et al.
(author)
-
Recognizing Permuted Words with Vector Symbolic Architectures: A Cambridge Test for Machines
- 2016
-
In: Procedia Computer Science. - : Elsevier BV. - 1877-0509. ; 88, s. 169-175
-
Journal article (peer-reviewed)abstract
- This paper proposes a simple encoding scheme for words using principles of Vector Symbolic Architectures. The proposed encoding allows finding a valid word in the dictionary for a given permuted word (represented using the proposed approach) using only a single operation - calculation of Hamming distance to the distributed representations of valid words in the dictionary. The proposed encoding scheme can be used as an additional processing mechanism for models of word embedding, which also form vectors to represent the meanings of words, in order to match the distorted words in the text to the valid words in the dictionary.
|
|
| 47. |
- Kleyko, Denis, 1990-, et al.
(author)
-
Vector-Based Analysis of the Similarity Between Breathing and Heart Rate During Paced Deep Breathing
- 2018
-
In: Computing in Cardiology 2018. - : IEEE.
-
Conference paper (peer-reviewed)abstract
- The heart rate (HR) response to paced deep breathing (DB) is a common test of autonomic function, where the scoring is based on indices reflecting the overall heart rate variability (HRV), where high scores are considered as normal findings but can also reflect arrhythmias. This study presents a method based on hyperdimensional computing for assessment of the similarity between feature vectors derived from the HR and breathing signals. The proposed method was used to identify subjects where HR did not follow the paced breathing pattern in recordings from DB tests in 174 healthy subjects and 135 patients with cardiac autonomic neuropathy. Subjects were classified in 4 similarity classes, where the lowest similiarity class included 35 patients and 3 controls. In general, the autonomic function cannot be evaluated in subjects in the lowest similarity class if they also present with high HRV scores, since this combination is a strong indicator of the presence of arrhythmias. Thus, the proposed vector-based similarity analysis is one tool to identify subjects with high HRV but low cardiorespiratory synchronization during the DB test, which falsely can be interpreted as normal autonomic function.
|
|
| 48. |
- Kleyko, Denis, 1990-
(author)
-
Vector Symbolic Architectures and their applications : Computing with random vectors in a hyperdimensional space
- 2018
-
Doctoral thesis (other academic/artistic)abstract
- The main focus of this thesis lies in a rather narrow subfield of Artificial Intelligence. As any beloved child, it has many names. The most common ones are Vector Symbolic Architectures and Hyperdimensional Computing. Vector Symbolic Architectures are a family of bio-inspired methods of representing and manipulating concepts and their meanings in a high-dimensional space (hence Hyperdimensional Computing). Information in Vector Symbolic Architectures is evenly distributed across representational units, therefore, it is said that they operate with distributed representations. Representational units can be of different nature, however, the thesis concentrates on the case when units have either binary or integer values. This thesis includes eleven scientific papers and extends the research area in three directions: theory of Vector Symbolic Architectures, their applications for pattern recognition, and unification of Vector Symbolic Architectures with other neural-like computational approaches. Previously, Vector Symbolic Architectures have been used mainly in the area of cognitive computing for representing and reasoning upon semantically bound information, for example, for analogy-based reasoning. This thesis significantly extends the applicability of Vector Symbolic Architectures to an area of pattern recognition. Pattern recognition is the area constantly enlarging its theoretical and practical horizons. Applications of pattern recognition and machine learning can be found in many areas of the present day world including health-care, robotics, manufacturing, economics, automation, transportation, etc. Despite the success in many domains pattern recognition algorithms are still far from being close to their biological vis-a-vis – the brain. In particular, one of the challenges is a large amount of training data required by conventional machine learning algorithms. Therefore, it is important to look for new possibilities in the area via exploring biologically inspired approaches.All application scenarios, which are considered in the thesis, contribute to the development of the global strategy of creating an information society. Specifically, such important applications as biomedical signal processing, automation systems, and text processing were considered. All applications scenarios used novel methods of mapping data to Vector Symbolic Architectures proposed in the thesis.In the domain of biomedical signal processing, Vector Symbolic Architectures were applied for three tasks: classification of a modality of medical images, gesture recognition, and assessment of synchronization of cardiovascular signals. In the domain of automation systems, Vector Symbolic Architectures were used for a data-driven fault isolation. In the domain of text processing, Vector Symbolic Architectures were used to search for the longest common substring and to recognize permuted words.The theoretical contributions of the thesis come in four aspects. First, the thesis proposes several methods for mapping data from its original representation into a distributed representation suitable for further manipulations by Vector Symbolic Architectures. These methods can be used for one-shot learning of patterns of generic sensor stimuli. Second, the thesis presents the analysis of an informational capacity of Vector Symbolic Architectures in the case of binary distributed representations. Third, it is shown how to represent finite state automata using Vector Symbolic Architectures. Fourth, the thesis describes the approach of combining Vector Symbolic Architectures and a cellular automaton.Finally, the thesis presents the results of unification of two computational approaches with Vector Symbolic Architectures. This is one of the most interesting cross-disciplinary contributions of the thesis. First, it is shown that Bloom Filters – an important data structure for an approximate membership query task – can be treated in terms of Vector Symbolic Architectures. It allows generalizing the process of building the filter. Second, Vector Symbolic Architectures and Echo State Networks (a special kind of recurrent neural networks) were combined together. It is possible to implement Echo State Networks using only integer values in network’s units and much simpler operation for a recurrency operation while preserving the entire dynamics of the network. It results in a simpler architecture with lower requirements on memory and operations.
|
|
| 49. |
- Kleyko, Denis, et al.
(author)
-
Vector Symbolic Architectures as a Computing Framework for Emerging Hardware
- 2022
-
In: Proceedings of the IEEE. - : Institute of Electrical and Electronics Engineers Inc.. - 0018-9219 .- 1558-2256. ; 110:10, s. 1538-1571
-
Journal article (peer-reviewed)abstract
- This article reviews recent progress in the development of the computing framework vector symbolic architectures (VSA) (also known as hyperdimensional computing). This framework is well suited for implementation in stochastic, emerging hardware, and it naturally expresses the types of cognitive operations required for artificial intelligence (AI). We demonstrate in this article that the field-like algebraic structure of VSA offers simple but powerful operations on high-dimensional vectors that can support all data structures and manipulations relevant to modern computing. In addition, we illustrate the distinguishing feature of VSA, 'computing in superposition,' which sets it apart from conventional computing. It also opens the door to efficient solutions to the difficult combinatorial search problems inherent in AI applications. We sketch ways of demonstrating that VSA are computationally universal. We see them acting as a framework for computing with distributed representations that can play a role of an abstraction layer for emerging computing hardware. This article serves as a reference for computer architects by illustrating the philosophy behind VSA, techniques of distributed computing with them, and their relevance to emerging computing hardware, such as neuromorphic computing.
|
|
| 50. |
- Kleyko, Denis, et al.
(author)
-
Vehicle Classification using Road Side Sensors and Feature-free Data Smashing Approach
- 2016
-
In: 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC). - Piscataway : IEEE. - 9781509018895 - 9781509018888 - 9781509018901 ; , s. 1988-1993
-
Conference paper (peer-reviewed)abstract
- The main contribution of this paper is a study of the applicability of data smashing - a recently proposed data mining method - for vehicle classification according to the "Nordic system for intelligent classification of vehicles" standard, using measurements of road surface vibrations and magnetic field disturbances caused by passing vehicles. The main advantage of the studied classification approach is that it, in contrast to the most of traditional machine learning algorithms, does not require the extraction of features from raw signals. The proposed classification approach was evaluated on a large dataset consisting of signals from 3074 vehicles. Hence, a good estimate of the actual classification rate was obtained. The performance was compared to the previously reported results on the same problem for logistic regression. Our results show the potential trade-off between classification accuracy and classification method's development efforts could be achieved.
|
|
