| 1. |
- Weegar, Rebecka, et al.
(författare)
-
Linking Entities Across Images and Text
- 2015
-
Ingår i: Proceedings of the 19th Conference on Computational Language Learning. - Stroudsburg, PA, USA : Association for Computational Linguistics. - 9781941643778 ; , s. 185-193
-
Konferensbidrag (refereegranskat)abstract
- This paper describes a set of methods to link entities across images and text. Asa corpus, we used a data set of images, where each image is commented by a short caption and where the regions in the images are manually segmented and labeled with a category. We extracted the entity mentions from the captions and we computed a semantic similarity between the mentions and the region labels. We also measured the statistical associations between these mentions and the labels and we combined them with the semantic similarity to produce mappings in the form of pairs consisting of a region label and a caption entity. In a second step, we used the syntactic relationships between the mentions and the spatial relationships between the regions to rerank the lists of candidate mappings. To evaluate our methods, we annotated a test set of 200 images, where we manually linked the image regions to their corresponding mentions in the captions. Eventually, we could match objects in pictures to their correct mentions for nearly 89 percent of the segments, when such a matching exists.
|
|
| 2. |
- Tegen, Agnes, et al.
(författare)
-
Image Segmentation and Labeling Using Free-Form Semantic Annotation
- 2014
-
Ingår i: [Host publication title missing]. - 1051-4651. ; , s. 2281-2286
-
Konferensbidrag (refereegranskat)abstract
- In this paper we investigate the problem of segmenting images using the information in text annotations. In contrast to the general image understanding problem, this type of annotation guided segmentation is less ill-posed in the sense that for the output there is higher consensus among human annotations. In the paper we present a system based on a combined visual and semantic pipeline. In the visual pipeline, a list of tentative figure-ground segmentations is first proposed. Each such segmentation is classified into a set of visual categories. In the natural language processing pipeline, the text is parsed and chunked into objects. Each chunk is then compared with the visual categories and the relative distance is computed using the word-net structure. The final choice of segments and their correspondence to the chunked objects are then obtained using combinatorial optimization. The output is compared to manually annotated ground-truth images. The results are promising and there are several interesting avenues for continued research.
|
|
| 3. |
- Weegar, Rebecka, et al.
(författare)
-
Visual Entity Linking: A Preliminary Study
- 2014
-
Ingår i: Cognitive computing for augmented human intelligence, papers presented at the Twenty-Eighth AAAI Conference on Artificial Intelligence. - 9781577356646 ; , s. 46-49
-
Konferensbidrag (refereegranskat)abstract
- In this paper, we describe a system that jointly extracts entities appearing in images and mentioned in their ac- companying captions. As input, the entity linking pro- gram takes a segmented image together with its cap- tion. It consists of a sequence of processing steps: part- of-speech tagging, dependency parsing, and coreference resolution that enables us to identify the entities as well as possible textual relations from the captions. The pro- gram uses the image regions labelled with a set of pre- defined categories and computes WordNet similarities between these labels and the entity names. Finally, the program links the entities it detected across the text and the images. We applied our system on the Segmented and Annotated IAPR TC-12 dataset that we enriched with entity annotations and we obtained a correct as- signment rate of 55.48%
|
|
