A confidence-based interface for neuro-symbolic visual question answering

Eiter, ThomasInstitute of Logic and Computation, Vienna University of Technology, Vienna, Austria (author)

We present a neuro-symbolic visual question answering (VQA) approach for the CLEVR dataset that is based on the combination of deep neural networks and answer-set programming (ASP), a logic-based paradigm for declarative problem solving. We provide a translation mechanism for the questions included in CLEVR to ASP programs. By exploiting choice rules, we consider deterministic and non-deterministic scene encodings. In addition, we introduce a confidence-based interface between the ASP module and the neural network which allows us to restrict the non-determinism to objects classified by the network with high confidence. Our experiments show that the non-deterministic scene encoding achieves good results even if the neural networks are trained rather poorly in comparison with the deterministic approach. This is important for building robust VQA systems if network predictions are less-than perfect.

Higuera, NelsonInstitute of Logic and Computation, Vienna University of Technology, Vienna, Austria (author)
Oetsch, JohannesInstitute of Logic and Computation, Vienna University of Technology, Vienna, Austria(Swepub:hj)oetjoh (author)
Pritz, MichaelInstitute of Logic and Computation, Vienna University of Technology, Vienna, Austria (author)
Institute of Logic and Computation, Vienna University of Technology, Vienna, Austria (creator_code:org_t)

In:Combining learning and reasoning: Programming languages, formalisms, and representations

By the author/editor: Eiter, Thomas; Higuera, Nelson; Oetsch, Johannes; Pritz, Michael

About the subject

NATURAL SCIENCES: NATURAL SCIENCES; and Computer and Inf ...; and Computer Science ...

Kungliga biblioteket hanterar dina personuppgifter i enlighet med EU:s dataskyddsförordning (2018), GDPR. Läs mer om hur det funkar här.
Så här hanterar KB dina uppgifter vid användning av denna tjänst.