Texture feature column scheme for single‐ and multi‐script writer identification

IET BiometricsVolume 10, Issue 2 p. 179-193 ORIGINAL RESEARCH PAPEROpen Access Texture feature column scheme for single- and multi-script writer identification Faycel Abbas, Corresponding Author Faycel Abbas faycel.abbas@univ-tebessa.dz LIMPAF Laboratory, Akli Mohand Oulhadj University, Bouira, Algeria Department of Mathematics and Computer Science, Larbi Tebessi University, Tebessa, Algeria Correspondence Faycel Abbas, LIMPAF Laboratory, Akli Mohand Oulhadj University, Bouira, Algeria. Email: faycel.abbas@univ-tebessa.dzSearch for more papers by this authorAbdeljalil Gattal, Abdeljalil Gattal Department of Mathematics and Computer Science, Larbi Tebessi University, Tebessa, AlgeriaSearch for more papers by this authorChawki Djeddi, Chawki Djeddi Department of Mathematics and Computer Science, Larbi Tebessi University, Tebessa, Algeria LITIS Laboratory, University of Rouen, Rouen, FranceSearch for more papers by this authorImran Siddiqi, Imran Siddiqi Vision & Learning Lab, Bahria University, Islamabad, PakistanSearch for more papers by this authorAmeur Bensefia, Ameur Bensefia Higher Colleges of Technology, CIS Division, Abu Dhabi, UAESearch for more papers by this authorKamel Saoudi, Kamel Saoudi Department of Electrical Engineering, Akli Mohand Oulhadj University, Bouira, AlgeriaSearch for more papers by this author Faycel Abbas, Corresponding Author Faycel Abbas faycel.abbas@univ-tebessa.dz LIMPAF Laboratory, Akli Mohand Oulhadj University, Bouira, Algeria Department of Mathematics and Computer Science, Larbi Tebessi University, Tebessa, Algeria Correspondence Faycel Abbas, LIMPAF Laboratory, Akli Mohand Oulhadj University, Bouira, Algeria. Email: faycel.abbas@univ-tebessa.dzSearch for more papers by this authorAbdeljalil Gattal, Abdeljalil Gattal Department of Mathematics and Computer Science, Larbi Tebessi University, Tebessa, AlgeriaSearch for more papers by this authorChawki Djeddi, Chawki Djeddi Department of Mathematics and Computer Science, Larbi Tebessi University, Tebessa, Algeria LITIS Laboratory, University of Rouen, Rouen, FranceSearch for more papers by this authorImran Siddiqi, Imran Siddiqi Vision & Learning Lab, Bahria University, Islamabad, PakistanSearch for more papers by this authorAmeur Bensefia, Ameur Bensefia Higher Colleges of Technology, CIS Division, Abu Dhabi, UAESearch for more papers by this authorKamel Saoudi, Kamel Saoudi Department of Electrical Engineering, Akli Mohand Oulhadj University, Bouira, AlgeriaSearch for more papers by this author First published: 14 February 2021 https://doi.org/10.1049/bme2.12010Citations: 1AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract Identification of writers from images of handwriting is an interesting research problem in the handwriting recognition community. Application of image analysis and machine learning techniques to this problem allows development of computerised solutions which can facilitate forensic experts in reducing the search space against a questioned document. This article investigates the effectiveness of textural measures in characterising the writer of a handwritten document. A novel descriptor by crossing the local binary patterns (LBP) with different configurations that allows capturing the local textural information in handwriting using a column histogram is introduced. The representation is enriched with the oriented Basic Image Features (oBIFs) column histogram. Support vector machine (SVM) is employed as the classifier, and the experimental study is carried out on five different datasets in single as well as multi-script evaluation scenarios. Multi-script evaluations allow evaluating the hypothesis that writers share common characteristics across multiple scripts and the reported results validate the effectiveness of textural measures in capturing this script-independent, writer-specific information. 1 INTRODUCTION Study of handwriting and hand-drawn shapes has always remained an attractive subject for historians, document examiners, psychologists and forensic experts. In addition to the semantic content, handwriting is known to carry rich information on the individual producing it [1, 2]. The most notable of these is the unique set of writing preferences depicted in the writing style. These writer-specific characteristics allow identifying and authenticating the authorship of a handwriting sample; more commonly known as writer identification and writer verification respectively. Formally, the problem of writer identification involves a set of writing samples with known writers and finding the writer of a handwriting in question, matching it with those in the reference set. The closely related problem of writer verification includes concluding whether a given set of two writing samples come from the same or different writers. Writer identification and verification form an important component of behavioural biometrics [3, 4] and although the performance of handwriting as a biometric modality is less impressive as opposed to physiological biometrics (iris, finger prints, DNA etc.), acquisition of handwriting is non-invasive and does not require any specialised hardware. With the advancements in image analysis and pattern classification, efforts have been made to develop computerised solutions to analyse digitised images of handwriting. Such systems aim to map the domain knowledge of forensic experts into computational features and these features are employed to characterise the writer of a handwritten document [1, 5, 6]. From the view point of practical applications, such systems can be employed to facilitate the domain experts by retrieving a hit-list of probable candidates from a large reference set, given a questioned document. This reduces the search space and allows the forensic experts to focus on a limited and manageable set of writing samples for manual analysis and decisive conclusions. Writer identification can be carried out from online handwriting or offline images in text-dependent or text-independent mode [7, 8]. From the perspective of offline, text-independent writer identification, computerised solutions exploit the visual differences in handwriting to characterise the writer. These difference include allographic variations, inter-word and intra-word spacings, slope of lines, slant of characters, legibility and cursiveness. Few of such visual differences between the writing samples of two writers can be observed in Figure 1. FIGURE 1Open in figure viewerPowerPoint Samples of two different writers—Images in each row represent samples by the same writer This article presents an offline writer identification technique leveraging the rich textural information in handwriting and considering the handwriting of an individual as a unique texture. In other words the technique relies on 'how' the text is written rather than 'what' is written to characterise the writer. Textural measures based on column histograms of local binary patterns (LBP) and oriented basic image features (oBIFs), computed from images of handwriting, are employed to identify the writer using Support Vector Machine (SVM) as classifier. The proposed technique is evaluated in a single as well as a multi-script mode and high identification rates are realised in a number of experimental scenarios. The hypothesis that individuals share common characteristics across multiple scripts is inspired by the handwriting generation process. The individuality in handwriting is contributed by many factors which, in addition to the learned copy-book style, include structure of hand, pen grip, muscular strength and properties of the central nervous system [9]. Handwriting is produced by a combination of writing strokes—the fundamental units of movement of hand (writing instrument). While the alphabets and words in one script are morphologically very different from those in another script, when produced by the same writer, the same writer-specific combination of writing strokes is employed to generate the final product. Hence, this idea can be employed to carry out writer identification in a script-independent scenario. The existence of common patterns across handwriting samples (of an individual writer) in multiple scripts has also been validated by a number of previous studies [10-12] and makes the subject of our current study as well. The key highlights of this study are listed in the following: Investigation of textural measures in characterising an individual from handwriting. Crossing multiple configurations of LBP and oBIFs to produce column histograms hence enriching the feature representation. Experimental study on five different datasets in a single as well as multi-script scenario. Validation of the hypothesis that individuals share common characteristics across multiple scripts. Realisation of high identification rates outperforming the current state-of-the-art on this problem. The article is organised as follows. In the next section, we discuss the recent advancements in offline writer identification. Section 3 introduces the textural features employed to characterise the writer along with the classification details. Experimental study, quantitative performance and a detailed analysis of the reported results is presented in Section 4. Section 5 concludes the article with a discussion on key findings and insights into open research problems on this subject. 2 RELATED WORKS Automatic writer identification has matured significantly over the years; thanks to the research endeavours of the pattern classification and handwriting recognition community. While the preliminary attempts targeted isolated characters and text-dependent identification, robust text-independent writer identification techniques have been developed reporting close to 100% identification rates on a hitlist of 10, with reference base of the order of 103 matching the target performance for most forensic applications [9]. Most of the research on writer identification focused on writing samples in a single-script [1, 13-15]. A relatively recent trend, however, has been to investigate this problem in a multi-script scenario where same individuals provide writing samples in more than one script [16, 17]. Investigating common, writer-specific writing characteristics across multiple scripts indeed represents an interesting research problem. Among various computational features, texture has been a popular choice of researchers for writer characterisation. Bertolini et al. [13], for instance, exploit Local phase quantisation (LPQ) and LBP features extracted from normalised blocks of handwriting to identify the writer and report high identification rates on the IAM dataset and the Brazilian Forensic Letter (BFL) dataset. In another similar work, Hannad et al. [18] rather than extracting textural measures from blocks, consider small fragments (windows) of writing. In addition to LBP and LPQ, the representation is enriched by a third textural descriptor, the Local Ternary Patterns (LTP). The features are evaluated in English (IAM) as well as Arabic (IFN/ENIT) writing samples and a comparative performance analysis reveals that LPQ outperforms other descriptors on both the datasets. Among other textural measures, oBIFs [19] computed using a bank of multiple Derivative-of-Gaussian filters, have also shown effective writer identification performance. Another notable category of writer identification techniques is the codebook-based methods. These methods extract frequent writing patterns to characterise the writer [14]. A codebook of fragmented connected-component contours (FCO3) extracted from character fragments ('fraglets'), for example is studied in Ref. [20]. The codebook is produced by grouping the fragments using Self Organising Maps (SOMs) and the probability distribution of producing the codebook patterns is used to identify the writer. A similar idea is presented by Siddiqi et al. [21] but rather than considering contour fragments, the codebook is generated using small fragments of writing. In another study [22], a synthetic codebook is employed along with a feature selection scheme to reduce its size. He et al. [23] identify the junction points in handwriting and produce a codebook of 'junclets'. The idea of codebooks was extended to an ensemble of codebooks (of graphemes) followed by dimensionality reduction in Ref. [24]. In another related work, Khan et al. [25] exploit the bagged discrete cosine transform (BDCT) descriptors to produce multiple codebooks. The final decision on the identity of the writer is reached through majority voting. In a recent investigation [26], key points in the handwriting are identified and patches around the key points are grouped into a codebook to characterise the writer. In a number of recent studies, automatic feature learning using deep neural networks has also been investigated for writer identification. Authors in Ref. [27], for example exploit convolutional neural networks to learn robust feature representations from writing samples and identify the writer in an end-to-end trainable system. A number of other similar studies [28-31] also demonstrate the effectiveness feature learning for writer characterisation and demonstrate the robustness of learned features over traditional methods. It is, however, important to recall that for a problem like writer identification, while the overall dataset sizes could be of the order of 103, the number of samples per (writer) class is fairly limited. Especially, from the view point of real-world forensic applications, a very limited amount of text (in some cases few lines or words only) is likely to be available. With such limited amount of training data per class, training CNNs to learn features could be challenging. Furthermore, from the perspective of applications for forensic experts, an 'explainable' solution is needed for the acceptance of computerised applications in their daily practices. Hand-crafted features are relatively more 'explainable' to domain experts as opposed to machine-learned features and the experts can also correlate these hand-engineered features with what they employ in handwriting analysis. In other words, the mapping of domain knowledge to computations features is much more evident. This serves to reduce to hesitancy of these experts in accepting automated solutions to facilitate their analysis. In the context of multi-script writer identification, Djeddi et al. [32] exploit run-length features with KNN and SVM classifier to identify writers in multi-script experimental settings. Evaluations on a collection of English and Greek writing samples from 126 different writers report promising identification rates. In another work considering the multi-script scenario, Graian et al. [10] employ two-dimensional (2D) auto-regression coefficients to characterise the writer from writing samples in French and Bengali. Likewise, Bertolini et al. [11] study the effectiveness of textural features (LBP and LPQ) in identifying writers in a multi-script experimental protocol with Arabic and English samples of 475 writers from the QUWI database. In another similar work [12], edge-hinge and run-length features are combined to identify writers through samples in Arabic, German, English and French. International competitions on multi-script writer identification have also been organised, the most notable of these include the ICDAR 2015 [16] and the ICFHR 2018 [17] competition. A summary of well-known contributions to writer identification reported in the literature is presented in Table 1. It is observed that identification of writers from writing samples in a single-script has been thoroughly investigated in the literature and high identification rates have been reported. Writer identification in multi-script settings, on the other, remains relatively less explored. The reported identification rates are also relatively low in many cases. Among popular datasets, IAM and CVL have been widely employed for English while IFN/ENIT and KHATT for Arabic writing samples. It is, however, important to mention that not all of these datasets were collected targeting the writer identification problem. IAM and IFN/ENIT, for instance, were primarily developed for tasks like segmentation and recognition of handwriting. However, since writer information was also stored during data collection, many researchers employed these datasets for evaluation of the writer identification systems. Datasets specifically developed for the writer identification task include the CEDAR letter [1], the BFL dataset [33], ICDAR 2015 writer identification dataset [16] and ICFHR 2018 writer identification dataset [17]. While CEDAR [1] and BFL [33] are collections of single-script samples, many of the recent datasets target writer identification in a multi-script scenario [16,17] and the same is being addressed in our current study. TABLE 1. An overview of well-known writer identification systems Features Study Database Language Number of writers Classification rates (%) Textural features Bertolini et al. (2013) IAM English 650 96.70 BFL Portuguese 315 99.20 Hannad et al. (2016) IFN/ENIT Arabic 411 94.89 IAM English 657 89.54 CVL English 310 96.20 Codebook-based Siddiqi and Vincent (2010) IAM English 650 91.00 He et al. (2015) IAM English 650 91.10 Firemaker Dutch 250 89.80 CERUG-MIXED Chinese and English 105 96.20 Abdi and Khemakhem (2015) IFN/ENIT Arabic 411 90.00 Khalifa et al. (2015) IAM English 650 92.00 Khan et al. (2017) IAM English 650 97.20 CVL English 310 99.60 IFN/ENIT Arabic 411 76.00 Bennour et al. (2019) BFL Portuguese 300 98.33 CVL English 300 94.00 KHAT Arabic 1000 62.81 Local descriptors Christlein et al. (2017) IAM English 650 88.00 CVL English 310 99.20 KHATT Arabic 1000 98.80 CNN Nguyen et al. (2019) JEITA-HP Japanese 400 93.82 Firemaker English 250 92.38 IAM English 650 90.12 Multi-script studies AR coefficients Graian et al. (2009) RIMES + ISI French/Bengali 422 62.10 Edge-hinge and run-length Djeddi et al. (2012) GRDS German/English/French 26 99.50 IFN/ENIT+GRDS German/English/French/Arabic 301 93.30 Run-length features Djeddi et al. (2013) Latin & Greek Docs. English/Greek 126 73.41/76.59 Textural features Bertolini et al. (2016) QUWI Arabic/English 475 77.90/98.70 English/Arabic 475 70.90/94.50 Zernike Moments ICDAR 2015 Winner QUWI Arabic/English 300 55.00 English/Arabic 300 29.00 3 METHODS Like all pattern classification systems, our writer identification technique relies on two key components, feature extraction and classification. Since we target multi-script writer identification, globally computed features that capture the overall writing style are not likely to be effective as the overall visual appearance of writing samples in different scripts would be totally different (even if they come from the same writer). Local features, on the other hand, allow capturing low-level stroke information in handwriting and this information is independent of script as an individual employs the same set of writing gestures to produce characters and words irrespective of the script. We, therefore, locally compute textural descriptors to characterise the writer. We introduce a novel representation of the local binary patterns (LBP column histogram) and combine it with the oBIFs column histogram to enrich the representation. Classification is carried out using the SVM classifier. Each of these components is presented in detail in the following. 3.1 Column scheme for textural features Extracting robust feature representations is the core component of any classification system. Given a collection of data samples (handwritten documents in our case) and the corresponding class labels (writer IDs), the idea is to seek effective representation in a feature space where samples of the same class (writer) tend to cluster together while those from other classes are far apart. In our study, we investigate two textural features, LBP and oBIFs. LBP captures the local spatial patterns in the handwriting. Likewise, oBIFs capture the local symmetry information. Furthermore, in order to capture the same information and different scales of observations, we compute these features for different parameter settings and then compute the column histogram, hence further enriching the representation. 3.2 LBP column histogram In continuation of the efforts [34] to seek robust textural representations, LBP descriptor was first proposed in Refs. [35, 36]. It encodes each pixel in an image with a decimal number as a function of the neighbouring pixel values. The encoding works by subtracting the reference pixel from each of its (8) neighbours. If the result is a negative value, the respective neighbour is assigned 1 and 0 otherwise. The resulting binary sequence is then considered as a number and represents the LBP code of the central pixel. The originally proposed LBP descriptor is considered a local neighbourhood of 3 × 3 and could not capture the structural information at distant scales of observations. The descriptor was therefore extended to multiple resolutions [37] by using neighbourhoods of different sizes. The neighbourhood is generalised by considering a circle centred at the pixel to be labelled and is defined by two parameters, the radius R of the circle and the number of (equally spaced) sampling points P in the neighbourhood (Figure 2). Points which do not represent true pixels are interpolated and the neighbourhood is typically represented by the pair (P, R). FIGURE 2Open in figure viewerPowerPoint Circular neighbourhood for two combinations of (P, R) pairs The coordinates of P neighbours (xp, yp) on the boundary of the circle with radius R are computed with respect to the central pixel (xc, yc) as follows. x p = x c + R c o s ( 2 π p P ) (1) y p = y c + R s i n ( 2 π p P ) (2) If the intensity value of the centre pixel is gc and the intensity values of its neighbours are gp with p = 0, 1, 2, …, P − 1, then the texture T considers the signs of the differences in the local neighbourhood of pixel (xc, yc) and is defined as: T ≈ ( s ( g 0 − g c ) , s ( g 1 − g c ) , … , s ( g P − 1 , g c ) ) (3)where, s(gp − gc) = 1 if (gp − gc) ≥ 0 and 0 otherwise. The local textural information is hence represented as a joint distribution of the value of the centre pixel and the differences. The extended LBP operator [37] is computed by assigning a binomial weight 2p to each s(gp − gc) as follows. L B P P , R ( x c , y c ) = ∑ p = 0 P − 1 s ( g p − g c ) . 2 p (4)where, P is the number of neighbouring pixels and R is the radius of the circle around (xc, yc). The histogram of these binary numbers is then used to describe the texture of the image. LBP is further characterised into uniform and non-uniform patterns. A pattern is termed as uniform if it has at most two bitwise transitions from 0 to 1 (or vice versa) when the pattern is considered a circular string. This results in P × (P − 1) + 3 unique values for P bits. For instance, uniform mapping of 8 sampling points produce 59 unique values while with 16 sampling points, 243 different values are produced. From the view point of representing a handwriting sample using LBP descriptors, LBP images for different combinations of sampling points P and radius R are computed. These LBP images are then stacked representing each pixel as a column vector. The image is then encoded using a (normalised) histogram of these columns. Figure 3 illustrates the process of computing the LBP column histogram from LBP images corresponding to two sets of (P, R) values. The column histogram enriches the LBP descriptor by taking into account multiple resolutions (scales of observations) corresponding to different values of P and R. While the original LBP method considers the local neighbourhood a pixel for a given pair of (P, R) values, we propose to enhance the basic LBP feature by computing two different LBP images for two pairs of (P, R) values, stacking them together and then computing the LBP column histogram that captures much richer local textural and symmetry information in the (handwriting) image. Different combinations of (P, R) values in computing the column histograms represent different scales at which the handwriting is analysed; combined in a single and enhanced feature vector. FIGURE 3Open in figure viewerPowerPoint Different steps of the LBP column scheme: (a) Original image; (b) Both LBPs images at different parameters (LBP image (p = 4, R = 16), LBP image (p = 8, R = 2)) are crossed to form columns at each location; (c) Histogram is computed with columns 3.2.1 oBIF column histogram Another set of textural measures investigated in our study comprises the oBIF. These features have been successfully applied to problems like texture classification [38], digit recognition [39] and identification of writers [19]. In our previous studies, we employed oBIFs for classification of gender from handwriting [40] and identification of writers from historical manuscripts [41]. In the present study, the objective of employing these features is to complement the LBP column histograms and enrich the textural representation of handwriting. The oBIFs represent an extension to the Basic Image Features (BIFs) [42, 43]. The key idea is to label each location in the image with one of the seven local symmetry classes. The features are computed by applying a bank of derivative of Gaussian filters controlled by a scale parameter σ. An additional parameter ϵ is employed to classify a location as 'flat'. Three of these symmetry types are accompanied by n possible orientations, the slope class has 2n possible orientations while three of the classes do not have any orientation. This gives an oBIF vector of 5n + 3. More details on computational aspects of oBIFs can be found in our previous work [40]. In the present work, we quantise the orientation into n = 4 directions, hence producing 23 entries (5 × 4 + 3) in the oBIFs dictionary. Similar to the LBP descriptor, oBIFs at two different scales are stacked (by ignoring the symmetry type flat) and the column histogram is computed. The histogram has a total of (5n + 2)2 = 484 bins. The scale parameter is chosen from the set σ ∈{1, 2, 4, 8, 16}, while ϵ is picked from a set of three small values {0.1, 0.01, 0.02}. The histogram is finally normalised for subsequent processing. The LBP and oBIFs column histograms are computed for writing samples of all individuals. By varying the parameters in computation of features, different configurations of LBP and oBIFs are produced and more details on these configurations are presented in Section 4. 3.3 Classification For the identification task, we employ SVM as the classifier [44, 45]. The LBP and oBIFs column histograms extracted from the writing sample(s) of an individual are used to train the learning algorithm. SVM is trained with the radial basis kernel function (RBF), the kernel parameter is selected in the range 0, 100], while the soft margin parameter C is fixed to 10. For each feature x fed to the trained SVM classifier, n decision scores fnj(x) for j = 1, 2, …, n are produced. n is the total number of writers and the decision on the identity of the writer is taken as follows. f m a x ( x ) = m a x ( f n j ( x ) ; j = 1 , 2 , … , n ) (5)fmax(x) is the maximum value selected from n responses produced by the classifier and fn(x) is the mapping of original SVM output values to the interval [0, 1] as follows. f n ( x ) = 1 1 + e − f ( x ) (6) In addition to making the identification decisions on each of the LBP and OBIFs column histograms, we also report identification performance based on picking the maximum value from the normalised decision scores on each of the two features. There are different ways to combine decisions of classifiers in an attempt to enhance the overall classification rates. Commonly employed methods include majority voting, Borda count, the sum, product and max rules and so on. Since we combine the decisions based on two distinct types of features, combinations using majority voting or Borda count would not be very meaningful (as there are only two scores). For each type of feature (LBP and oBIF columns), the SVM classifier outputs n similarity scores corresponding to n writers in the reference base. As mentioned earlier, these scores are mapped to the interval [0, 1]; higher the score, more confident the classifier is in predicting the respective class label (writer). The max rule predicts the output class by choosing the class corresponding to the maximum confidence value among all the participating classifiers. In other words, for the problem under study, the system reports two scores SLBP and SoBIFs. Selecting the maximum [max(SLBP, SoBIFs)] score ensures that we pick the decision corresponding to the classifier which is more confident. This serves to compensate the errors and improves the overall classification performance. The usefulness of combining decisions of a