DILF: Differentiable rendering-based multi-view Image–Language Fusion for zero-shot 3D shape understanding

↓ Direkt till sidans innehåll
↓ Direkt till sidans sekundära innehåll (sidomenyn)

Sökning: onr:"swepub:oai:DiVA.org:hh-51777" > DILF :

1 av 1
Föregående post
Nästa post
Till träfflistan

Ning, XinChinese Academy Of Sciences, Beijing, China (författare)

DILF : Differentiable rendering-based multi-view Image–Language Fusion for zero-shot 3D shape understanding

Artikel/kapitelEngelska2024

Förlag, utgivningsår, omfång ...

Amsterdam :Elsevier,2024
printrdacarrier

Nummerbeteckningar

LIBRIS-ID:oai:DiVA.org:hh-51777
https://urn.kb.se/resolve?urn=urn:nbn:se:hh:diva-51777URI
https://doi.org/10.1016/j.inffus.2023.102033DOI

Kompletterande språkuppgifter

Språk:engelska
Sammanfattning på:engelska

Ingår i deldatabas

SwePubSwePub

Klassifikation

Ämneskategori:ref swepub-contenttype
Ämneskategori:art swepub-publicationtype

Anmärkningar

Funding agency:National Natural Science Foundation of China (NSFC) Grant number: 6237334Beijing Natural Science Foundation Grant number: L233036
Zero-shot 3D shape understanding aims to recognize “unseen” 3D categories that are not present in training data. Recently, Contrastive Language–Image Pre-training (CLIP) has shown promising open-world performance in zero-shot 3D shape understanding tasks by information fusion among language and 3D modality. It first renders 3D objects into multiple 2D image views and then learns to understand the semantic relationships between the textual descriptions and images, enabling the model to generalize to new and unseen categories. However, existing studies in zero-shot 3D shape understanding rely on predefined rendering parameters, resulting in repetitive, redundant, and low-quality views. This limitation hinders the model's ability to fully comprehend 3D shapes and adversely impacts the text–image fusion in a shared latent space. To this end, we propose a novel approach called Differentiable rendering-based multi-view Image–Language Fusion (DILF) for zero-shot 3D shape understanding. Specifically, DILF leverages large-scale language models (LLMs) to generate textual prompts enriched with 3D semantics and designs a differentiable renderer with learnable rendering parameters to produce representative multi-view images. These rendering parameters can be iteratively updated using a text–image fusion loss, which aids in parameters’ regression, allowing the model to determine the optimal viewpoint positions for each 3D object. Then a group-view mechanism is introduced to model interdependencies across views, enabling efficient information fusion to achieve a more comprehensive 3D shape understanding. Experimental results can demonstrate that DILF outperforms state-of-the-art methods for zero-shot 3D classification while maintaining competitive performance for standard 3D classification. The code is available at https://github.com/yuzaiyang123/DILP. © 2023 The Author(s)

Ämnesord och genrebeteckningar

NATURVETENSKAP Data- och informationsvetenskap Språkteknologi hsv//swe
NATURAL SCIENCES Computer and Information Sciences Language Technology hsv//eng
Differentiable rendering
Information fusion
Text–image fusion
Zero-shot 3D shape understanding

Biuppslag (personer, institutioner, konferenser, titlar ...)

Yu, ZaiyangChinese Academy Of Sciences, Beijing, China; University Of Chinese Academy Of Sciences, Beijing, China (författare)
Li, LusiOld Dominion University, Norfolk, United States (författare)
Li, WeijunChinese Academy Of Sciences, Beijing, China (författare)
Tiwari, Prayag,1991-Högskolan i Halmstad,Akademin för informationsteknologi(Swepub:hh)pratiw (författare)
Chinese Academy Of Sciences, Beijing, ChinaChinese Academy Of Sciences, Beijing, China; University Of Chinese Academy Of Sciences, Beijing, China (creator_code:org_t)

Sammanhörande titlar

Ingår i:Information FusionAmsterdam : Elsevier102, s. 1-121566-25351872-6305

Internetlänk

Hitta via bibliotek

Information Fusion (Sök värdpublikationen i LIBRIS)

Till lärosätets databas

1 av 1
Föregående post
Nästa post
Till träfflistan

Hitta mer i SwePub

Av författaren/redakt...: Ning, Xin; Yu, Zaiyang; Li, Lusi; Li, Weijun; Tiwari, Prayag, ...

Om ämnet

NATURVETENSKAP: NATURVETENSKAP; och Data och informa ...; och Språkteknologi

Artiklar i publikationen: Information Fusi ...

Av lärosätet: Högskolan i Halmstad

Sök utanför SwePub

Sök vidare i:: Google; Google Book Search; Google Scholar

Kungliga biblioteket hanterar dina personuppgifter i enlighet med EU:s dataskyddsförordning (2018), GDPR. Läs mer om hur det funkar här.
Så här hanterar KB dina uppgifter vid användning av denna tjänst.

LIBRIS.kb.se