onr:"swepub:oai:DiVA.org:hh-51777"
Search: onr:"swepub:oai:DiVA.org:hh-51777" > DILF :
- 1 of 1
- Previous record
- Next record
- To hitlist
DILF : Differentiable rendering-based multi-view Image–Language Fusion for zero-shot 3D shape understanding
-
- Ning, Xin (author)
- Chinese Academy Of Sciences, Beijing, China
-
- Yu, Zaiyang (author)
- Chinese Academy Of Sciences, Beijing, China; University Of Chinese Academy Of Sciences, Beijing, China
-
- Li, Lusi (author)
- Old Dominion University, Norfolk, United States
- show more...
- show less...
- (creator_code:org_t)
- Amsterdam : Elsevier, 2024
- 2024
- English.
- In: Information Fusion. - Amsterdam : Elsevier. - 1566-2535 .- 1872-6305. ; 102, s. 1-12
- Journal article (peer-reviewed)
Abstract
Subject headings
Close
- Zero-shot 3D shape understanding aims to recognize “unseen” 3D categories that are not present in training data. Recently, Contrastive Language–Image Pre-training (CLIP) has shown promising open-world performance in zero-shot 3D shape understanding tasks by information fusion among language and 3D modality. It first renders 3D objects into multiple 2D image views and then learns to understand the semantic relationships between the textual descriptions and images, enabling the model to generalize to new and unseen categories. However, existing studies in zero-shot 3D shape understanding rely on predefined rendering parameters, resulting in repetitive, redundant, and low-quality views. This limitation hinders the model's ability to fully comprehend 3D shapes and adversely impacts the text–image fusion in a shared latent space. To this end, we propose a novel approach called Differentiable rendering-based multi-view Image–Language Fusion (DILF) for zero-shot 3D shape understanding. Specifically, DILF leverages large-scale language models (LLMs) to generate textual prompts enriched with 3D semantics and designs a differentiable renderer with learnable rendering parameters to produce representative multi-view images. These rendering parameters can be iteratively updated using a text–image fusion loss, which aids in parameters’ regression, allowing the model to determine the optimal viewpoint positions for each 3D object. Then a group-view mechanism is introduced to model interdependencies across views, enabling efficient information fusion to achieve a more comprehensive 3D shape understanding. Experimental results can demonstrate that DILF outperforms state-of-the-art methods for zero-shot 3D classification while maintaining competitive performance for standard 3D classification. The code is available at https://github.com/yuzaiyang123/DILP. © 2023 The Author(s)
Subject headings
- NATURVETENSKAP -- Data- och informationsvetenskap -- Språkteknologi (hsv//swe)
- NATURAL SCIENCES -- Computer and Information Sciences -- Language Technology (hsv//eng)
Keyword
- Differentiable rendering
- Information fusion
- Text–image fusion
- Zero-shot 3D shape understanding
Publication and Content Type
- ref (subject category)
- art (subject category)
Find in a library
- Information Fusion (Search for host publication in LIBRIS)
To the university's database
- 1 of 1
- Previous record
- Next record
- To hitlist
Find more in SwePub
- By the author/editor
- Ning, Xin
- Yu, Zaiyang
- Li, Lusi
- Li, Weijun
- Tiwari, Prayag, ...
- About the subject
-
- NATURAL SCIENCES
- NATURAL SCIENCES
- and Computer and Inf ...
- and Language Technol ...
- Articles in the publication
- Information Fusi ...
- By the university
- Halmstad University
Search outside SwePub
- Extend your search to:
- Google Book Search
- Google Scholar
Kungliga biblioteket hanterar dina personuppgifter i enlighet med EU:s dataskyddsförordning (2018), GDPR. Läs mer om hur det funkar här.
Så här hanterar KB dina uppgifter vid användning av denna tjänst.
- Copyright © LIBRIS - National Library Systems
- LIBRIS.kb.se
