Analyzing the Effects of Data Augmentation on Single and Multimodal Biometrics

Now-a-days, in the field of machine learning the data augmentation techniques are common in use, especially with deep neural networks, where a large amount of data is required to train the network. The effectiveness of the data augmentation technique has been analyzed for many applications; however, it has not been analyzed separately for the multimodal biometrics. This research analyzes the effects of data augmentation on single biometric data and multimodal biometric data. In this research, the features from two biometric modalities: fingerprint and signature, have been fused together at the feature level. The primary motivation for fusing biometric data at feature level is to secure the privacy of the user’s biometric data. The results that have been achieved by using data augmentation are presented in this research. The experimental results for the fingerprint recognition, signature recognition and the feature-level fusion of fingerprint with signature have been presented separately. The results show that the accuracy of the training classifier can be enhanced with data augmentation techniques when the size of real data samples is insufficient. This research study explores that how the effectiveness of data augmentation gradually increases with the number of templates for the fused biometric data by making the number of templates double each time until the classifier achieved the accuracy of 99%.


INTRODUCTION
he field of biometrics has always been an attractive area for the research community. It includes a diverse variety of biometric traits or modalities such as fingerprint, face, iris, palmprint, palm vein, gait, signature, keystroke dynamics, voice, and DNA (Deoxyribonucleic Acid) are among the different types of biometrics. In recent years, trends are diverting towards multimodal biometrics [1]. In multimodal biometric various biometric modalities 648 may be at score level where the matching scores for two or more biometric modalities can be fused to generate a final score for the recognition [4]. This type of fusion is termed as score-level fusion. The score-level fusion is basically used to make the biometric systems more accurate and reliable, It is efficient when one of the input biometric traits are not present in fully correct format like scars on fingerprints or change of the voice due to weather conditions and the system is still required to recognize the users reliably. Likewise, the features from two or more biometric traits may also be fused together to generate a new combined feature set based on those modalities [5]. The feature level fusion scheme combines features from one or more biometric modalities and create a single new biometric template that consists of the properties from both of the fused modalities or biometric traits. This type of fusion is mainly used to increase the security and privacy of human biometrics. As the features of two or more modalities will be combined into a single template along with applying some encryption to secure the template it will be nearly impossible to get either fingerprint data or signature data separately from that fused template. This is called feature level fusion and this approach is used for the experiments performed for this research study. In this research, the multimodal biometric approach is described that consists of fingerprints and signatures. Both types of biometric traits are fused at the feature level. The main motivation behind this fusion is to enhance the privacy and security of user data thus making the biometric system more reliable.
Fingerprint recognition is the most acceptable physiological biometric trait. A fingerprint consists of a unique combination of ridges and valleys. During the fingerprint recognition process, unique features are extracted from a fingerprint. These unique features are called minutiae. Various types of minutiae have been identified by the researchers but only a few have been used in most of the research works. One of the most important types of minutiae is called bifurcation. When a ridge is divided into two ridges this feature is called ridge-bifurcation. Similarly, the point where the ridge is terminated is termed as ridge-ending. In the same way, if a very small ridge is present in a fingerprint, then this is called a short-ridge. Likewise, there may be a small point of the ridge and that is called a dot.
The second biometric modality that is used in this research is one of the oldest and most user-friendly behavioral biometric modality that is signature. A signature is a written identity, mostly handwritten name of a person that is used for the recognition or verification process. There are two kinds of signatures, one is called online signature that is made on the screen of a touch sensible device and the other is termed as the offline signature, that is mostly made on paper and then transferred to the computer with the help of camera or scanner to perform the recognition. This research study considers the offline signature.
The experiments performed for this research are based on two types of data; one is data without augmentation and the other is data with augmentation. In very simple words data augmentation may be defined as increasing the number of images in a dataset, by taking the images of that dataset and making copies of the images with slight difference in attributes of the images like rotation angle, height, width, and zooming. The data augmentation techniques, in terms of images, are used to increase the dataset to generate a sufficient amount of images that are necessary for efficiently training the classifier. A larger dataset is generated with data augmentation techniques from a small number of actual data. The efficiency of data augmentation techniques has been well proved by research community [6]. Data augmentation has been proved a helpful tool, not only to increase the accuracy of the classifiers but it may also aid in better training of the classifier with the low quality input images [7]. Fig.1 represents a sample of real and augmented data for the fingerprint and signature.
Data augmentation can be used to increase the efficiency of the system by training the system with large data samples even with the smaller number of real samples available. The larger samples can be generated by a smaller number of samples with programming techniques. It may be clearly noted from the results presented in Table 1 that the data augmentation increased the accuracy of the data. A brief account of literature review is discussed in Section II of this paper. Section III describes the databases used for the experiments during this research along with the organization and augmentation of data throughout the experiments. Experimental results are presented and discussed in section IV of this paper. Section V concludes the paper and describes the future directions for this research.

LITERATURE REVIEW
Various literature studies reflect that the multimodal biometrics provides better security and accuracy as compared with the traditional unimodal biometric system [10]. A unique approach to fuse the features is proposed by [11] that suggests the fusion of the palmprint information with its own reproduced copy. The authors first made the copies of the captured palmprint by varying its phase 45-degree every time then that information is fused with its reproduced templates and a new fused template is generated. By using these types of data augmentation techniques the security of the biometric data is ensured.
It is also evident from the research in [12] that the biometric fusion at the initial stages of the recognition process like in feature level fusion is more effective Fingerprints have been combined with signatures by some of the researchers by score level fusion to increase the accuracy of existing systems. Like in [13] score level fusion has been used to enhance the accuracy of the system. However, for the purpose of enhancing the security rather than accuracy, one must consider combining the features from multiple biometric traits instead of combining recognition scores. It is discussed in [14] that security risks to biometric data increase by storing separate data templates in the system, and it is better to fuse the templates and then store the fused templates in the system. Literature reveals that the feature level fusion has successfully been achieved for various biometric traits.
In recent years the fingerprints were fused with iris to create a new fused template called iris-print [15]. That template looks like the fingerprint and is able to be recognized by any conventional fingerprint recognition system, however, actually the fingerprint data in the template is modified and mixed with the features from the iris. These types of fusion mechanisms are also used to enhance the security of biometric data.
A multimodal system proposed by [16] integrates palmprint and Finger Knuckle-Print for person recognition. A matching algorithm by applying Phase-Correlation Function with both the palmprint and Finger Knuckle-Print modalities is combined and a score level fusion is developed. The results reported show the efficiency of the system which is more efficient than the unimodal biometric systems. In the same way [17] proposed a novel method to combine fingerprint and face for electronic voting system by using cascaded Generalized Principal Component Analysis (GPCA) and KNN (K-Nearest Neighbor) algorithms. The reported accuracy of the system outperformed the previous systems with 91% accuracy in very nominal light conditions for face recognition. Similarly, a rank-level fusion method proposed by [18] applies principal component analysis and Fisher's linear discriminant techniques to match individual biometrics i.e. face, signature, and ear. Individual biometrics rank is combined by applying highest rank, logistic regression and borda count methods. The ranklevel fusion technique is then used to authenticate the consolidated results achieved from individual biometric matchers. The results reported indicate that by making the fusion of individual biometrics, the overall performance of the system may be improved even when the quality of the data is low. Vein pattern matching techniques have also become more common and popular among the researchers as these patterns are unique, stable and immune to frauds. A multimodal biometric system based on hand vein patterns proposed by [19] implements a dorsal and palmar vein features. Principle Component Analysis is used for representing the features of veins. Initially, separate scores are increased by every matcher and then these scores are tested to measure the efficiency of the system. A scorelevel fusion technique is then applied to combine these individual scores. The performance of both the unimodal and multimodal biometric systems is compared. While the fusion of different biometric has improved the biometric recognition accuracy, image augmentation also plays a vital role in training the models for winning algorithms especially for deep neural networks [20]. A recent research study demonstrates that the biometric recognition using augmented data achieved two times greater accuracies as compared with the data without augmentation [21]. The researchers have taken a dataset of 50 images and expanded that dataset to 2000 images with the help of data augmentation techniques like reflecting and rotating the image and adjusting the intensity and the color of the image. In the same way in another research investigation on liveness detection of multimodal biometrics, the data augmentation is used to increase the performance of the classifier [22].   [8]. A similar method was followed for fingerprints to initially take 8 copies of fingerprints for each of the 20 users. After that fingerprints were generated with the help of data augmentation and a complete dataset of 2400 fingerprints was generated for the experiments during this research study. For experiment the complete dataset has been divided into two sets: 80% for the training set and 20% for the testing set. The data augmentation techniques used for this research includes the rotation of the input image randomly from 1-35 degrees. Further, the size of the image has been increased by a zoom range of 0.1. Width and height for the augmentation of the data are also set to 0.1.

DATABASE AND EXPERIMENTAL SETUP
The experiments are performed using the Python programming language using the Anaconda3 platform. It is elaborated with the help of Fig. 3 that the proposed system consists of two phases namely enrollment and verification.
These two phases are further divided into various steps and sub-steps. These steps are briefly described in this section. During the enrollment phase first of all the images of the fingerprints and signatures are provided as the input to the proposed system. After the system receives the signature and fingerprint as the input the preprocessing steps are performed. The preprocessing operations include morphological operations like binarization and thinning. The preprocessing steps for the proposed research first include the size normalization. The size for both the fingerprint and the signature has been set to 388 pixels in width and 388 pixels in height. After size normalization the images were binnarized, an image after binarization contains only two colors that are black and white. After binarization each pixel of the image can be represented with one of two values either 'True' or 'False'. After binarization, the image is skeletonized, in such a way that the width of the ridges in a fingerprint image or writing strokes in a signature becomes only one pixel wide. The skeletonized image has been obtained by Guo-Hall thinning algorithm [24].
After completing the preprocessing steps the image becomes ready for the features extraction. The features have been extracted from these images using a Histogram of Oriented Gradients (HOG) technique as it helps in the efficient extraction of features from the images [25]. HOG is a technique which splits the image into a number of segments or cells, and each cell computes the orientations or positions of pixels to find the edges. The number of cells can be varied to get the better results and in our case, after testing different values of the cells the best results were achieved by setting cells equal to 9. After feature extraction, the fingerprint data and the signature data are combined by concatenating the features from both fingerprint and offline signature. Each user's fingerprint has been fused with a unique signature to generate a virtual multimodal database with the help of Sum-Rule technique for feature fusion and saved to the database. In the Sum-Rule feature fusion for biometric data, the features from two or more biometric modalities are fused together by simple arithmetic addition. All the experiments have been performed on that virtual multimodal database.
For the verification phase represented by Fig. 3, all the initial steps are same as discussed in detail for the enrollment phase. However, after the fusion of the features from fingerprint and signature, the fused template is compared with those already available in the database. This comparison or classification has been made with the help of the Random Forest Classifier (RFC) to search for the best possible matching class. RFC is a classifier that uses ensemble approach and combines more than one classifiers usually the search trees, to obtain the best possible results [26]. The RFC classifier has been selected for this research for the reason that ensemble classifiers yield a better accuracy results as compared to the conventional classifiers like Artificial Neural Networks (ANNs) and kNNs [27][28]. Finally, the decision is made on the result from the RFC classifier.

EXPERIMENTAL RESULTS AND DISCUSSIONS
The statistics of the dataset and the testing accuracy for the fingerprint, signature and fused data, is described in Table 1. The results are shown without data augmentation and with data augmentation. It can be noted that the data augmentation has helped to increase 653 the accuracy of the system. The accuracy of the fingerprint without data augmentation was 40% while with the data augmentation it increased and reached to the accuracy of 98%. The accuracy of the signature increased with the help of data augmentation from 82-94%. While the fusion of both fingerprint and signature leads to the accuracy of 99%.
The fingerprint and signature dataset used in the experiments contributes 160 original fingerprints and signatures and the remaining fingerprints and signatures were generated with the help of data augmentation to make up the dataset of 2400 fingerprints and 2400 signatures. It is evident from the results that the overall testing accuracy of 98% was achieved for the fingerprint and the testing accuracy of 94.4% was accomplished for the signature data. However, the fused data produced an accuracy of 99%. Although the accuracy of the fused data is not much higher than that of the single biometric trait, the fusion aids in preserving the privacy of the biometric data providers because if the template is compromised even then the fused data template cannot be easily reused for either of two biometric modalities. Table 1 shows that the accuracy of fused data with 1200 templates is 94% and when the number of templates doubled to 2400 with data augmentation the accuracy increases to 97%. Yet the accuracy for 4800 fused templates leads to the accuracy of 99%, Experimental results of this research are also calculated in terms of precision, recall, and F1-score. The results are presented in Table 2, for the precision, recall and f1-score for fingerprint, signature and fusion of fingerprint and signature. It may be clearly noted from Table 2 that the F1-score for the fingerprint alone is 99% while the F1-score for the signature alone is 94%. However, when the fingerprint and signature data is fused the F1-score is 99%.
One of the simple ways to illustrate the performance of any classifier is to represent the results in the form of a confusion matrix. In a confusion matrix the classification results of a test dataset are examined against the already provided class labels to verify the accuracy of the trained classifier. In other words, the confusion matrix represents how many times the classifier was successful to classify the correct label or class for a given input. For example, if the user1 is given 20 times to the classifier and the classifier predicted that as the user1 for all the 20 times then it will be represented with the top most color from the color bar that is the color yellow and if the system fails to predict the user1 as the user1 for all the 20 times then it will be represented with the lowest color of the color bar that is color purple. The confusion matrix for the recognition of fused data for this research is displayed in Fig. 4.

654
template and there will be no need to store the separate template data for either fingerprint or signature in order to ensure the security of the separate data templates. Experimental results have been presented for separate fingerprint recognition, signature recognition and the fused data for fingerprint and signature. The overall accuracy of 99% for the fused data is acquired. The importance of data augmentation has also been discussed and the results have been presented for data with augmentation and data without augmentation.
There is a possibility to improve the accuracy of the developed system and make it more reliable by using more details in fingerprints in the form of sweat pores. In future, the fingerprint dataset will be used with the liveness detection techniques in order to provide more unique and detailed identification of the user. That database will include the data about active sweat pores [33].

FUTURE WORK
In future, the same data that is used in this research can be used to perform the research experiments for deep learning such as Convolutional Neural Networks (CNNs). This will certainly increase the accuracy of the system as deep learning techniques are outperforming other conventional methods for more than a decade [34].In the future, the same developed system will be modified to work on another type of signature that is an online signature. By introducing online signature there is a clear possibility of an increase in accuracy as online signature stores much more information compared to the offline signature.