Gesture recognition and customization on textile-based pressure sensor array

Abstract

The tactile sensation plays an essential part in perceiving and interacting with our surroundings, making touch-based technologies increasingly significant in everyday life. The technologies cover a wide spectrum from cellphones and tablets to sensors made entirely of textiles. When creating tactile sensing systems, multiple parameters need to be taken into account. Although touchscreens are the ideal choice for systems that need visual feedback, wearable technology requires devices that are soft, flexible, adjustable to the human body's shapes, and free from safety concerns. Textile-based capacitive pressure sensor array is selected for pressure sensing in gesture recognition system because of its accurate pressure detection capability and lightweight design. A pressure sensor array consisting of 11x2 sensors has been manufactured that rely on the principle of determining the location of pressure applied through variations in capacitance. It generates 22-dimensional capacitance data vector. In order to detect regions with higher capacitance when pressure is applied with fingertips, a sequence of data processing procedures, such as calibration, scaling, and flattening are executed. The manipulated data reveal a series of consecutively pressed cells on the textile sensor. In order to analyze the patterns of pressure applied cells, a deep learning model called Long Short-Term Memory (LSTM) and a machine learning model Hidden Markov Model (HMM) are utilized. The results of the two models were compared, and based on the obtained results, a high level of accuracy was achieved. In considering the difficulties caused by memorizing gestures or the inability of users to execute pre-defined gestures, it was considered crucial to enable the creation of new gestures and customization of them. In order to solve this issue, a class-incremental approach was implemented. The proposed approach deals with two primary problems: missing previous data and the inability to identify a new class of data. Changes were implemented to the LSTM layer and output layer of the existent model. The amount of new data sample is a factor that affects the equilibrium between usability and accuracy. As the amount of data sample grows, the duration of training increases and the usability decreases. On the other hand, when there are fewer data samples, the accuracy of the model reduces. In order to tackle this issue, a compromise was made by gathering a small number of data samples from the user and then enlarging the dataset through the utilization of data augmentation techniques. In order to reduce the risk of forgetting previous classes, the model was enhanced by using past data as inputs. An experiment was carried out in two phases, involving a total of 20 people. During the first phase, the participants were instructed to execute four predetermined movements -up, down, pinch, and zoom—. During the second phase, participants were instructed to repeatedly execute a new gesture in order to collect data for the creation of the model of the new gesture. Afterwards, they were instructed to execute both predetermined and newly defined gestures in order to confirm the recognition of the new data and ensure that the previous data were not forgotten. This phase was repeated to demonstrate the capacity to establish several new gestures. The recognition rate of the predetermined movements in the first phase gave accuracy values of 95.3% for deep learning model, 91.5% for machine learning model. After including one new gesture, the overall system achieved an accuracy of 96.3% for deep learning model, 91.7% for machine learning model. Furthermore, after introducing a second new gesture, the accuracy dropped slightly to 94.7% for deep learning model, 89.4% for machine learning model. Further studies will investigate the utilization of this technology as a controller.