Learning DALTS for cross‐modal retrieval

CAAI Transactions on Intelligence TechnologyVolume 4, Issue 1 p. 9-16 Research ArticleOpen Access Learning DALTS for cross-modal retrieval Zheng Yu, Zheng Yu School of Electronic and Computer Engineering, Shenzhen Graduate School, Peking University, Shenzhen, People's Republic of ChinaSearch for more papers by this authorWenmin Wang, Corresponding Author Wenmin Wang wangwm@ece.pku.edu.cn School of Electronic and Computer Engineering, Shenzhen Graduate School, Peking University, Shenzhen, People's Republic of ChinaSearch for more papers by this author Zheng Yu, Zheng Yu School of Electronic and Computer Engineering, Shenzhen Graduate School, Peking University, Shenzhen, People's Republic of ChinaSearch for more papers by this authorWenmin Wang, Corresponding Author Wenmin Wang wangwm@ece.pku.edu.cn School of Electronic and Computer Engineering, Shenzhen Graduate School, Peking University, Shenzhen, People's Republic of ChinaSearch for more papers by this author First published: 18 February 2019 https://doi.org/10.1049/trit.2018.1051Citations: 14AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Cross-modal retrieval has been recently proposed to find an appropriate subspace, where the similarity across different modalities such as image and text can be directly measured. In this study, different from most existing works, the authors propose a novel model for cross-modal retrieval based on a domain-adaptive limited text space (DALTS) rather than a common space or an image space. Experimental results on three widely used datasets, Flickr8K, Flickr30K and Microsoft Common Objects in Context (MSCOCO), show that the proposed method, dubbed DALTS, is able to learn superior text space features which can effectively capture the necessary information for cross-modal retrieval. Meanwhile, DALTS achieves promising improvements in accuracy for cross-modal retrieval compared with the current state-of-the-art methods. 1 Introduction The task of cross-modal retrieval is beginning to attract more and more attention recently. That is, given an image (text) query, we aim to search for the most relevant text (image). However, multimedia data is heterogeneous intrinsically and thus hard for us to measure the similarity directly. So the main challenge remaining in cross-modal retrieval is how to embed heterogeneous multimedia data into a homogeneous space, so that their similarity can be measured directly. More specifically, the main challenge consists of the following two sub-problems. The first problem is how to learn efficient features for multimedia data, which evolves from hand-crafted features to deep features gradually. As for image, with the great success achieved by convolutional neural networks (CNNs), Sharif Razavian et al. [[1]] argue that a pre-trained deep CNN is an effective image feature extractor for many computer vision tasks including cross-modal retrieval. However, do off-the-shelf CNNs provide sufficient information for cross-modal retrieval? Most existing works employ off-the-shelf CNNs such as VGGNet [[2]] and ResNet [[3]] to extract image features. However, these models are usually pre-trained for classification and thus only need to consider the category information contained in an image. Therefore, they are inevitable to miss detailed cues such as how the objects relate to each other as well as their attributes and the activities they are involved in, which may play an indispensable role in cross-modal retrieval. As shown in Fig. 1, given two different input images, a pre-trained CNN can only recognise the objects contained in each image which are similar to each other such as 'man', 'surfboard' and 'wave'. However, it tends to miss some crucial cues which are totally dissimilar for each other such as how do the man surf the wave. With the great progress achieved in image captioning task recently, we can get sensible descriptive sentences corresponding to an input image, which contain nouns and verbs. That is, image captioning models are able to not only recognise the objects in the image (nouns) but also preserve rich relation information among different objects (verbs). Therefore, we adopt image captioning models to make up for the shortcomings of the traditional CNN features. Fig. 1Open in figure viewerPowerPoint Illustration of the problem of using pre-trained CNNs to extract image features. Such classification models extract similar features for two images with different interactions among objects ('jumping off' versus 'surfing, paddling toward') As for text, Word2Vec [[4]], Latent Dirichlet allocation (LDA) [[5]] and FV [[6]] are all popular choices for text representation. However, they are all pre-trained on some specified corpora which are totally different from the datasets adopted in cross-modal retrieval. As such, instead of using off-the-shelf models, we employ recurrent neural network (RNN) to learn text features from scratch. Given efficient features for image and text, the second problem is how to find a homogeneous space. Since in this paper we only focus on the retrieval between image and text, cross-modal retrieval can be achieved by a common space [[7]–[18]], a text space [[19]–[21]] or an image space [[22]]. Considering the way people perform cross-modal retrieval, different modalities are processed asymmetrically in the brain. It refers to the well-known semantic gap [[23]] reflecting the fact that textual features are closer to human understanding (and language) than the pixel-based features [[19]]. Therefore, the textual features provide more accurate information than the pixel-based features during retrieval. Moreover, it is more straightforward for brains to understand the text than the image because nature language is the result of high-level abstraction of image content. Accordingly, we propose a feature embedding network to explore the possibility of performing cross-modal retrieval in a text space. The text space is highly discriminative. If a linear classifier is trained after the text space to predict whether a vector comes from an image or a sentence, we can achieve near 100% accuracy. That is, we can fit a hyperplane in the text space to near perfectly separate out images and sentences. This property violates the original goal to find a homogeneous space. Thus, since the source image space and the target text space can be regarded as two different domains, we propose a domain classifier to further minimise the diversity among the features from different modalities, similar to the idea of domain adaptation in [[24]]. That is, the domain classifier tries to discriminate the difference between the source domain (the original image space) and the target domain (the text space) during training, while the feature embedding network tries to learn domain-invariant features and confuse the domain classifier. Therefore, an additional adversarial loss will be back-propagated to the feature embedding network in order to guide the network to learn domain-invariant text space features for image and text. The text space is essentially a vector space spanning by a set of base vectors which are also known as different Chinese characters or English words. For Chinese, there are no exact numbers for Chinese characters, which is close to 100,000. Meanwhile, the emergence of enormous new words every year makes the size of the text space continue to grow. In addition to Chinese, similar phenomenon has appeared in other languages such as English. According to incomplete statistics, the number of English words has exceeded 1,000,000, and it is still growing by thousands every year. Therefore, natural language is inherently divergent. It is almost impossible to learn a complete and unlimited text space. However, in most cases, people only need to remember some of the commonly used Chinese characters and English words to meet their daily needs. For example, many English linguists argue that about 3650 commonly used English words can accomplish more than 95% of the tasks of expressing ideas and communication. The 'National Dictionary of Modern Chinese' published by the National Board of Education in November 1987 proposes that the number of commonly used words in modern Chinese is 2500, accounting for more than 99% of the daily use of Chinese. Therefore, this paper ensures the convergence of the proposed algorithm by learning a limited text space (LTS) with a fixed vocabulary. The ability for the LTS to understand is affected by the size of the vocabulary. The bigger the vocabulary, the stronger the understanding ability. Increasing the number of words blindly will not improve the retrieval performance but increase the complexity of the algorithm in time and space. Our core contributions are summarised as follows: We propose a novel model domain-adaptive LTS (DALTS) to perform cross-modal retrieval humanly in a DALTS, which can better imitate the human behaviour. Moreover, we give a brief explanation on the LTS. In contrast to the commonly used pre-trained features for both image and text, DALTS is able to learn task-specific features. To further minimise the diversity between the source domain (the original image space) and the target domain (the LTS), the idea of domain adaptation is applied to the model to learn a DALTS. The rest of this paper is organised as follows. We review the related work for cross-modal retrieval in Section 2. Then in Section 3, we propose our own model and describe it in detail. To emphasise the effectiveness of DALTS, Section 4 and 5 show extensive experiments on three benchmark datasets. Finally, we make a summary of this paper in Section 6. 2 Related work 2.1 Multi-modal feature learning For cross-modal retrieval, most existing works directly use off-the-shelf features to represent images [[8], [11], [14]–[17]]. However, the pre-trained features are likely to leave out some crucial information which may be the key to cross-modal retrieval. Recently, image captioning models [[25]–[28]] can be used to learn task-specific features to provide more information that is useful to cross-modal retrieval. Given an input image, before decoding it to a descriptive sentence, image captioning models first try to map the image into a text space. Thus, the text space feature for an image contains not only category information but also rich relation information among different objects. Typically, multi-modal RNN (m-RNN) [[25]], neural image caption (NIC) [[26]], deep visual-semantic alignments [[27]] and unifying visual-semantic embeddings (VSEs) [[28]] are all representative methods for image captioning. As for text, similarly, typical methods such as Word2Vec [[4]], LDA [[5]] and FV [[6]] are all pre-trained on some specified corpora which are totally different from the benchmark datasets in cross-modal retrieval. Recently, with the great progress on machine translation [[29]], RNN is found to be a more powerful tool for language modelling which can be trained from scratch and thus more suitable for cross-modal retrieval. 2.2 Homogeneous space learning The mainstream approach tries to learn a common space by affine transformations on both image and text sides. Typically, canonical correlation analysis [[15]] tries to learn a common space by maximising the correlations between relevant image and text features. Karpathy et al. [[10]] break down both image and text into fragments and embeds them into a common multi-modal space which utilises fine-grained alignments between image and text. Niu et al. [[13]] address the problem of dense VSE that maps not only full sentences and whole images but also phrases within sentences and salient regions within images into a multi-modal embedding space. Wang et al. [[16]] propose deep structure-preserving embeddings (DSPEs) for image and text which extends pairwise ranking loss to model the intra-modal relationship and adopts a complicated data sample scheme. Nam et al. [[12]] propose dual attention networks (DANs) which jointly leverage visual and textual attention mechanisms to capture fine-grained interplay between image and text. In addition to a common space, in the DeViSE model developed by Frome et al. [[20]], a text space is formed by a pre-trained Word2Vec model. The text space vector of an image is obtained by a convex combination of the word embedding vectors of the visual labels predicted to be the most relevant to the image. However, the visual labels only reflect the objects contained in an image but ignore how these objects relate to each other as well as their attributes and the activities they are involved in. Thus, the Word2Vec space is not an effective text space for cross-modal retrieval. Recently, a distributional visual embedding space provided by Word2VisualVec [[22]] is found to be an effective space to perform cross-modal retrieval by embedding the text into an image space. 2.3 Domain adaptation In the absence of labelled data for a certain task, domain adaptation often provides an attractive option given that labelled data of similar nature but from a different domain are available. Ganin and Lempitsky [[24]] propose a new approach to domain adaptation in deep architectures that can learn features that are discriminative for the main learning task on the source domain and invariant with respect to the shift between the domains, which can be achieved by a domain classifier and a simple gradient reversal layer. Inspired by Goodfellow in Generative Adversarial Nets [[30]], there exists an alternative adversarial training strategy rather than using the gradient reversal layer. Recently, Park and Im [[14]] try to learn a common space for cross-modal retrieval based on domain adaptation and have achieved competitive experimental results. 3 Proposed method The general framework of DALTS is shown in Fig. 2 a, which contains a feature extraction network, a feature embedding network and a domain classifier. Fig. 2Open in figure viewerPowerPoint Overview of DALTS. The overall loss function contains the traditional pairwise ranking loss (the blue dashed lines) and the additional adversarial loss (the brown dashed lines) a General framework of DALTS, which contains a feature extraction network, a feature embedding network and a domain classifier b Detailed illustration of the feature embedding network c Detailed illustration of the domain classifier 3.1 Feature extraction Image representation: The network for image feature extraction consists of two branches: VGGNet that is pre-trained for image classification and NIC [[26]] that is pre-trained for image captioning. As mentioned earlier, VGGNet tends to capture rich category information but leave out some detailed cues for cross-modal retrieval. Conversely, NIC has the innate advantage of mining rich relation information among different objects contained in an image. So they are perfectly complementary to each other for cross-modal retrieval. Accordingly, we aim to design the network for image feature extraction as a combination of these two separate models. As shown in Fig. 3 a, the blue and green dashed boxes represent NIC and VGGNet, respectively, which are pre-trained on image captioning and image classification task. Given an input image, a forward pass of the pre-trained VGGNet produces a 4096-dimension feature . As for NIC, in order to avoid the information loss during decoding, we regard the 512-dimension output of the image embedding layer as the image feature . Finally, we denote a 4608-dimension feature as the feature for the input image, which is the concatenation of and . In practise, we have tried a further step to fine-tune the parameters of NIC but no significant gains were observed, so we decided to leave them fixed. Fig. 3Open in figure viewerPowerPoint Detailed illustration of the feature extraction network a Image feature extraction b Text feature extraction Text representation: We employ Long Short Term Memory networks (LSTM) to learn d -dimensional text features as shown in Fig. 3 b. Here, d is also denoted as the dimensionality of the LTS. Let be an input text with length , where we represent each word as a one-hot vector of dimension equals to the size of the dictionary. Note that we denote by as a special end word which designates the end of the text. Before fed into the LSTM, should be embedded into a denser space (1) where is a word embedding matrix. Then we feed the vectors into LSTM, which take the form (2) where denote the input, forget, output, memory cell and hidden state of the LSTM at time step t, respectively. Here, is the input word at time step t and is the hidden state of the LSTM at the last time step . denotes the sigmoid function and indicates element-wise multiplication. W, U and b represent the trainable parameters of LSTM. Thus, the feature for S can be obtained from the hidden state of the LSTM at time , that is, . 3.2 Domain classifier We adopt the concept from [[14], [24]]. Instead of using the gradient reversal layer, we advocate the adversarial training strategy by designing a domain classifier. Specifically, the domain classifier is a simple feed-forward neural network that has three fully connected layers as shown in Fig. 2 c. Given image and text features in the LTS, the domain classifier tries to predict the domain label for each input, for example, for the image features and for the text features. During training, we minimise the cross-entropy loss for a better domain discrimination with the parameters (3) where and represent the input feature and its corresponding domain label, respectively. The mapping function is able to predict the domain label given an input feature . 3.3 Feature embedding The feature embedding network aims to learn an LTS with parameters . As shown in Fig. 2 b, we design two mapping functions to transform and to d -dimensional text space features and , respectively, denoted as and . Similar to and , and are complementary to each other as well. Therefore, we add a fusion layer on top to combine the two features by summation. The whole process can be defined as (4) where are the LTS features for an input image. Note that the procedure of text feature extraction from scratch by LSTM is equivalent to embedding text into an LTS. Therefore, involves the parameters of LSTM. After embedding image and text into an LTS, the next step is to compare their similarities. We define a scoring function , where v and t represent image and text features, respectively. To make s equivalent to cosine similarity, v and t are first scaled to have unit norm by the layer. Then, two kinds of loss functions are exploited to train the embedding network: pairwise ranking loss and adversarial loss . Pairwise ranking loss is widely adopted for cross-modal retrieval. Let denote all the parameters to be learnt. We optimise the following pairwise ranking loss: (5) where is a negative text for a given image v and is a negative image for a given text . To obtain the non-matching terms, we choose them randomly from the training set and re-sample every epoch. Meanwhile, the adversarial loss will be back-propagated to the feature embedding network simultaneously. Since the feature embedding network tries to maximise in order to learn domain-invariant features, the optimisation goals of these two loss functions are opposite. Therefore, the overall loss function for the feature embedding network can be defined as (6) where is an adaptation factor varying from 0 to 1 in order to suppress noisy signal from the domain classifier at the early stages of the training procedure. Following Ganin and Lempitsky [[24]], we update by the following equation: (7) where p is the fraction of current step in maximum training steps. 3.4 Training procedure The training procedure can be divided into five stages. We denote the parameters of domain classifier and feature embedding network as and , respectively. Stage 1: During the first training stage, we pre-train NIC on image captioning using the benchmark datasets in cross-modal retrieval such as Flickr30K and Microsoft Common Objects in Context (MSCOCO). After the training complete, we can learn efficient image features. Stage 2: After extracting features for all images, we start stage 2 to learn an LTS. Given loss function L for the feature embedding network, we fix and try to update by the following rule: (8) where is the learning rate. Stage 3: After stage 2, we start stage 3 to enhance the discriminating ability of the domain classifier. Given loss function for the domain classifier, we fix and try to update by the following rule: (9) where is the learning rate. Stage 4: For each training batch, repeat stage 2 and stage 3 until DALTS converges. Stage 5: We can further fine-tune the parameters of NIC. 4 Experiments In this section, we perform extensive experiments on Flickr8K [[31]], Flickr30K [[32]] and MSCOCO [[33]] datasets following the dataset splits in [[10]]. Evaluation is performed using Recall@K (with K = 1, 5, 10), which computes the mean number of images (texts) for which the correct texts (images) are ranked within the top-K retrieved results. 4.1 Implementation details For image feature extraction, we first pre-train NIC on image captioning task using Flickr30K and MSCOCO and fix the parameters of NIC and VGGNet during the whole training procedure. In practise, we have tried a further step to fine-tune the parameters of NIC but no significant gains were observed, so we decided to leave them fixed. More specifically, we first rescale the image to , and then use a single centre crop of size to compute 1-crop VGG image feature. For text feature extraction, we set the dimensionality of the LTS to 1024. Meanwhile, the dimensionality of word embedding is set to 1024 as well. The feature embedding network contains two functions, and . For , is a matrix and is a matrix. Among various layers, Rectified Linear Unit (ReLU) is adopted to be the activation function and a dropout layer is added right after ReLU with probability = 0.5 in order to reduce overfitting. For , is a matrix. The margin is set to 0.3 in all our experiments. To accelerate the training and also make gradient updates more stable, we apply batch normalisation right after each mapping function. We employ a three-layer feed-forward neural network activated by ReLU for the domain classifier. The output dimensions of intermediate layers and are set to . Softmax layer is added right after the last layer . During training, we adopt Adam optimiser to optimise the model with learning rate 0.0002 for the first 15 epochs and then decay the learning rate by 0.1 for the remaining 15 epochs. We use a mini-batch size of 128 in all our experiments. 4.2 Comparison with the state of the art In this section, we report experimental results for cross-modal retrieval including image-to-text retrieval (Img2Text) and text-to-image retrieval (Text2Img) on the benchmark Flickr8K, Flickr30K and MSCOCO datasets. For Flickr8K, experimental results are presented in Table 1. Comparing DALTS with the current state-of-the-art method Hierarchical Multiscale Long Short Term Memory Networks (HMLSTM) [[13]], we observe that our model achieves new state-of-the-art results on image-to-text retrieval. However, it performs slightly inferiorly to HMLSTM on text-to-image retrieval. Since, instead of the global features we use with massive redundant information, HMLSTM extracts features for phrases within sentences and salient regions within images, as well as embeds them into a denser space. Table 1. Bidirectional retrieval results on Flickr8K Methods Img2Text Text2Img R@1 R@5 R@10 R@1 R@5 R@10 DeViSE [[20]] 4.8 16.5 27.3 5.9 20.1 29.6 m-RNN [[25]] 14.5 37.2 48.5 11.5 31.0 42.4 DeepFrag [[10]] 12.6 32.9 44.0 9.7 29.6 42.5 VSE [[28]] 22.3 48.7 59.8 14.9 38.3 51.6 NIC [[26]] 20.0 — 61.0 19.0 — 64.0 HMLSTM [[13]] 27.7 — 68.6 24.4 — 68.1 DALTS 30.8 60.9 75.2 23.4 53.6 68.0 The bold values in Tables 1–6 denote the best experimental results. On Flickr30K, the best performing competitor model becomes DAN_ vgg [[12]] on both tasks, as shown in Table 2. Only DAN_vgg outperforms DALTS on image-to-text retrieval. As for the text-to-image retrieval, DALTS achieves new state-of-the-art results. Owing to the application of attention mechanism, DAN is able to focus on certain aspects of data sequentially and aggregate essential information over time to infer the results. On the contrary, we use global features to represent both image and text which are likely to contain noisy or unnecessary information. Table 2. Bidirectional retrieval results on Flickr30K Methods Img2Text Text2Img R@1 R@5 R@10 R@1 R@5 R@10 DeViSE [[20]] 4.5 18.1 29.2 6.7 21.9 32.7 m-RNN [[25]] 35.4 63.8 73.7 22.8 50.7 63.1 DeepFrag [[10]] 14.2 37.7 51.3 10.2 30.8 44.2 NIC [[26]] 17.0 – 56.0 17.0 – 57.0 DSPE [[16]] 40.3 68.9 79.9 29.7 60.1 72.1 DAN_vgg [[12]] 41.4 73.5 82.5 31.8 61.7 72.5 VSE++ [[7]] 31.9 – 68.0 23.1 – 60.7 HMLSTM [[13]] 38.1 – 76.5 27.7 – 68.8 smLSTM [[9]] 42.4 67.5 79.9 28.2 57.0 68.4 DALTS 43.0 68.4 81.0 30.7 61.9 73.6 As shown in Table 3, with enough training data, DALTS achieves about 1 and 2% improvement in R@5 and R@10, respectively, on image-to-text retrieval, compared with DSPE. However, DALTS performs slightly inferiorly to smLSTM [[9]], which utilises attention mechanism similar to DAN. As for the text-to-image retrieval, DALTS performs slightly inferiorly to DSPE. One possible reason may be that the chain structured LSTM is likely to miss the intrinsic hierarchical structure of texts and thus shows weaker ability to learn text features than Fisher vector, which is learned by external text corpora. In particular, DALTS performs better than MRLA [[14]], which shows that pairwise ranking loss is more suitable for cross-modal retrieval rather than category classification loss. Table 3. Bidirectional retrieval results on MSCOCO Methods Img2Text Text2Img R@1 R@5 R@10 R@1 R@5 R@10 m-RNN [[25]] 41.0 73.0 83.5 29.0 42.2 77.0 VSE [[28]] 43.4 75.7 85.8 31.0 66.7 79.9 DSPE [[16]] 50.1 79.7 89.2 39.6 75.2 86.9 VSE++ [[7]] 43.6 — 84.6 33.7 — 81.0 HMLSTM [[13]] 43.9 — 87.8 36.1 — 86.7 smLSTM [[9]] 52.4 81.7 90.8 38.6 73.4 84.6 MRLA [[14]] 14.3 40.5 55.8 12.7 39.0 57.2 DALTS 46.2 81.0 90.0 38.1 73.5 86.1 For a fair comparison, the results of VSE++ [[7]] in Tables 2 and 3 are based on 1-crop VGG image features without fine-tuning. Note that if 10-crops ResNet image features are used to train the models with fine-tuning, the experimental results could be further improved. As shown in the Appendix, Fig. 4 shows some qualitative results for cross-modal retrieval on Flickr8K. To emphasise the efficiency of our proposed model, the retrieval results for each query are listed from left to right according to three variants of our proposed model, DALTS (VGG + BLSTM), DALTS (NIC + BLSTM) and DALTS (VGG + NIC + BLSTM). We can observe that the retrieval results from left to right obtain significant improvement especially from DALTS (VGG + BLSTM) to DALTS (NIC + BLSTM). Furthermore, the incorrectly retrieved results are reasonable as well, compared with the ground truth. Fig. 4Open in figure viewerPowerPoint Qualitative cross-modal retrieval results on Flickr8K. The first column lists image and text queries for retrieval. The second column to the fourth column shows top five retrieved results for each query by our proposed model DALTS (VGG + BLSTM), DALTS (NIC + BLSTM) and DALTS (VGG + NIC + BLSTM) respectively. For image-to-text retrieval, the correctly retrieved texts for each image query are denoted in red. As for the text-to-image retrieval, the image with a hook represents the correctly retrieved image for a text query In general, DALTS achieves promising improvements in accuracy for cross-modal retrieval compared with the current state-of-the-art methods, though it has some obvious limitations. In the future, we will employ a stronger CNN (ResNet) for experiments. Meanwhile, attention mechanism will be applied to reduce the negative impact of redundant information. 5 Further study on DALTS 5.1 Importance of different components To demonstrate the impact of different components in DALTS, we report results for the following variants in Table 4 : DALTS (VGG + LSTM): In this setting, we remove NIC while keep the remaining part fixed. DALTS (NIC + LSTM): In contrast to DALTS (VGG + LSTM)}, we remove VGGNet while keep the remaining part fixed. DALTS (VGG + NIC + LSTM): Network as in Fig. 2 a. DALTS (VGG + NIC + BLSTM): The network structure is as above but LSTM is replaced by BLSTM. For Flickr8K, we observe that changing the image feature extractor from VGG to NIC improves the accuracy by about 22% for image-to-text retrieval and about 20% for text-to-image retrieval. It demonstrates that, rather than VGG, NIC can better capture the information we need for cross-modal retrieval such as the interaction information among differe