| 1. |
- Bauer, Jakob Johannes, et al.
(författare)
-
Neuro-symbolic Visual Graph Question Answering with LLMs for language parsing
- 2023
-
Konferensbidrag (refereegranskat)abstract
- Images containing graph-based structures are an ubiquitous and popular form of data representation that, to the best of our knowledge, have not yet been considered in the domain of Visual Question Answering (VQA). We provide arespective novel dataset and present a modular neuro-symbolic approach as a first baseline. Our dataset extends CLEGR, an existing dataset for question answering on graphs inspired by metro networks. Notably, the graphs there are given in symbolic form, while we consider the more challenging problem of taking images of graphs as input. Our solution combines optical graph recognition for graph parsing, a pre-trained optical character recognition neural network for parsing node labels, and answer-set programming for reasoning. The model achieves an overall average accuracy of 73% on the dataset. While regular expressions are sufficient to parse the natural language questions, we also study various large-language models to obtain a more robust solution that also generalises well to variants of questions that are not part of the dataset. Our evaluation provides further evidence of the potential of modular neuro-symbolic systems, in particular with pre-trained models, to solve complex VQA tasks.
|
|
| 2. |
|
|
| 3. |
- Eiter, Thomas, et al.
(författare)
-
A confidence-based interface for neuro-symbolic visual question answering
- 2022
-
Ingår i: Combining learning and reasoning: Programming languages, formalisms, and representations.
-
Konferensbidrag (refereegranskat)abstract
- We present a neuro-symbolic visual question answering (VQA) approach for the CLEVR dataset that is based on the combination of deep neural networks and answer-set programming (ASP), a logic-based paradigm for declarative problem solving. We provide a translation mechanism for the questions included in CLEVR to ASP programs. By exploiting choice rules, we consider deterministic and non-deterministic scene encodings. In addition, we introduce a confidence-based interface between the ASP module and the neural network which allows us to restrict the non-determinism to objects classified by the network with high confidence. Our experiments show that the non-deterministic scene encoding achieves good results even if the neural networks are trained rather poorly in comparison with the deterministic approach. This is important for building robust VQA systems if network predictions are less-than perfect.
|
|
| 4. |
- Eiter, Thomas, et al.
(författare)
-
A Logic-based Approach to Contrastive Explainability for Neurosymbolic Visual Question Answering
- 2023
-
Ingår i: IJCAI International Joint Conference on Artificial Intelligence. - : International Joint Conferences on Artificial Intelligence. - 9781956792034 ; , s. 3668-3676
-
Konferensbidrag (refereegranskat)abstract
- Visual Question Answering (VQA) is a well-known problem for which deep-learning is key. This poses a challenge for explaining answers to questions, the more if advanced notions like contrastive explanations (CEs) should be provided. The latter explain why an answer has been reached in contrast to a different one and are attractive as they focus on reasons necessary to flip a query answer. We present a CE framework for VQA that uses a neurosymbolic VQA architecture which disentangles perception from reasoning. Once the reasoning part is provided as logical theory, we use answer-set programming, in which CE generation can be framed as an abduction problem. We validate our approach on the CLEVR dataset, which we extend by more sophisticated questions to further demonstrate the robustness of the modular architecture. While we achieve top performance compared to related approaches, we can also produce CEs for explanation, model debugging, and validation tasks, showing the versatility of the declarative approach to reasoning.
|
|
| 5. |
- Eiter, Thomas, et al.
(författare)
-
A Modular Neurosymbolic Approach for Visual Graph Question Answering
- 2023
-
Ingår i: Proceedings of the 17th International Workshop on Neural-Symbolic Learning and Reasoning La Certosa di Pontignano, Siena, Italy, July 3-5, 2023. - : CEUR-WS. ; , s. 139-149
-
Konferensbidrag (refereegranskat)abstract
- Images containing graph-based structures are a ubiquitous and popular form of data representation that, to the best of our knowledge, have not yet been considered in the domain of Visual Question Answering (VQA). We use CLEGR, a graph question answering dataset with a generator that synthetically produces vertex-labelled graphs that are inspired by metro networks. Structured information about stations and lines is provided, and the task is to answer natural language questions concerning such graphs. While symbolic methods suffice to solve this dataset, we consider the more challenging problem of taking images of the graphs instead of their symbolic representations as input. Our solution takes the form of a modular neurosymbolic model that combines the use of optical graph recognition for graph parsing, a pretrained optical character recognition neural network for parsing node labels, and answer-set programming, a popular logic-based approach to declarative problem solving, for reasoning. The implementation of the model achieves an overall average accuracy of 73% on the dataset, providing further evidence of the potential of modular neurosymbolic systems in solving complex VQA tasks, in particular, the use and control of pretrained models in this architecture.
|
|
| 6. |
- Eiter, Thomas, et al.
(författare)
-
A Neuro-Symbolic ASP Pipeline for Visual Question Answering
- 2022
-
Ingår i: Theory and Practice of Logic Programming. - : Cambridge University Press. - 1471-0684 .- 1475-3081. ; 22:5, s. 739-754
-
Tidskriftsartikel (refereegranskat)abstract
- We present a neuro-symbolic visual question answering (VQA) pipeline for CLEVR, which is a well-known dataset that consists of pictures showing scenes with objects and questions related to them. Our pipeline covers (i) training neural networks for object classification and bounding-box prediction of the CLEVR scenes, (ii) statistical analysis on the distribution of prediction values of the neural networks to determine a threshold for high-confidence predictions, and (iii) a translation of CLEVR questions and network predictions that pass confidence thresholds into logic programmes so that we can compute the answers using an answer-set programming solver. By exploiting choice rules, we consider deterministic and non-deterministic scene encodings. Our experiments show that the non-deterministic scene encoding achieves good results even if the neural networks are trained rather poorly in comparison with the deterministic approach. This is important for building robust VQA systems if network predictions are less-than perfect. Furthermore, we show that restricting non-determinism to reasonable choices allows for more efficient implementations in comparison with related neuro-symbolic approaches without losing much accuracy.
|
|
| 7. |
- Eiter, Thomas, et al.
(författare)
-
ALASPO : An Adaptive Large-Neighbourhood ASP Optimiser
- 2022
-
Ingår i: KR 2022. - : IJCAI Organization. - 9781956792010 ; , s. 565-569
-
Konferensbidrag (refereegranskat)abstract
- We present the system ALASPO which implements Adaptive Large-neighbourhood search for Answer Set Programming (ASP) Optimisation. Large-neighbourhood search (LNS) is a meta-heuristic where parts of a solution are destroyed and reconstructed in an attempt to improve an overall objective. ALASPO currently supports the ASP solver clingo, as well as its extensions clingo-dl and clingcon for difference and full integer constraints, and multi-shot solving for an efficient implementation of the LNS loop. Neighbourhoods can be defined in code or declaratively as part of the ASP encoding. While the method underlying ALASPO has been described in previous work, ALASPO also incorporates portfolios for the LNS operators along with self-adaptive selection strategies as a technical novelty. This improves usability considerably at no loss of solution quality, but on the contrary often yields benefits. To demonstrate this, we evaluate ALASPO on different optimisation benchmarks.
|
|
| 8. |
- Eiter, Thomas, et al.
(författare)
-
An open challenge for exact job scheduling with reticle batching in photolithography
- 2022
-
Konferensbidrag (refereegranskat)abstract
- We consider scheduling solutions for photolithography, an important sub-task in semi-conductor production, where patterns are transferred to wafers using reticles. The problem can be modelled as job scheduling on unrelated parallel machines with sequence-dependent setup times and release dates. The reticles add auxiliary-resource constraints for processing jobs. Equipping machines with the right reticles using transport robots from stockers in time renders this problem extremely difficult for exact solvers that use a declarative model. The latter would be attractive as such models tend to be compact and easy to maintain. We present a solver-independent MiniZinc model and provide 500 new benchmark instances. However, only small instances can be solved with state-of-the-art MIP and CP solvers. Consequently, we present this problem as an open challenge with considerable potential for driving improvements towards industrial applications.
|
|
| 9. |
- Eiter, Thomas, et al.
(författare)
-
Large-Neighbourhood Search for ASP Optimisation
- 2022
-
Ingår i: Electronic Proceedings in Theoretical Computer Science, EPTCS. - : Open Publishing Association. ; , s. 163-165
-
Konferensbidrag (refereegranskat)abstract
- While answer-set programming (ASP) is a successful approach to declarative problem solving optimisation can still be a challenge for it. Large-neighbourhood search (LNS) is a metaheuristic technique where parts of a solution are alternately destroyed and reconstructed, which has high but untapped potential for ASP solving. We present a framework for LNS optimisation for ASP, in which neighbourhoods can be specified either declaratively as part of the ASP encoding, or automatically generated by code. In the full paper, we illustrate the framework on different optimisation problems, some of which are notoriously difficult, including shift planning and a parallel machine scheduling problem from semi-conductor production which demonstrate the effectiveness of the LNS approach.
|
|
| 10. |
- Eiter, Thomas, et al.
(författare)
-
Large-Neighbourhood Search for Optimisation in Answer-Set Solving
- 2022
-
Ingår i: Proceedings of the 36th AAAI Conference on Artificial Intelligence, AAAI 2022. - : Association for the Advancement of Artificial Intelligence. - 1577358767 - 9781577358763 ; , s. 5616-5625
-
Konferensbidrag (refereegranskat)abstract
- While Answer-Set Programming (ASP) is a prominent approach to declarative problem solving, optimisation problems can still be a challenge for it. Large-Neighbourhood Search (LNS) is a metaheuristic for optimisation where parts of a solution are alternately destroyed and reconstructed that has high but untapped potential for ASP solving. We present a framework for LNS optimisation in answer-set solving in which neighbourhoods can be specified either declaratively as part of the ASP encoding or automatically generated by code. To effectively explore different neighbourhoods, we focus on multi-shot solving as it allows to avoid program regrounding. We illustrate the framework on different optimisation problems some of which are notoriously difficult, including shift planning and a parallel machine scheduling problem from semi-conductor production, which demonstrate the effectiveness of the LNS approach.
|
|
