| 1. |
- Marta, Daniel, et al.
(författare)
-
Aligning Human Preferences with Baseline Objectives in Reinforcement Learning
- 2023
-
Ingår i: 2023 IEEE INTERNATIONAL CONFERENCE ON ROBOTICS AND AUTOMATION (ICRA 2023). - : Institute of Electrical and Electronics Engineers (IEEE).
-
Konferensbidrag (refereegranskat)abstract
- Practical implementations of deep reinforcement learning (deep RL) have been challenging due to an amplitude of factors, such as designing reward functions that cover every possible interaction. To address the heavy burden of robot reward engineering, we aim to leverage subjective human preferences gathered in the context of human-robot interaction, while taking advantage of a baseline reward function when available. By considering baseline objectives to be designed beforehand, we are able to narrow down the policy space, solely requesting human attention when their input matters the most. To allow for control over the optimization of different objectives, our approach contemplates a multi-objective setting. We achieve human-compliant policies by sequentially training an optimal policy from a baseline specification and collecting queries on pairs of trajectories. These policies are obtained by training a reward estimator to generate Pareto optimal policies that include human preferred behaviours. Our approach ensures sample efficiency and we conducted a user study to collect real human preferences, which we utilized to obtain a policy on a social navigation environment.
|
|
| 2. |
- Marta, Daniel, et al.
(författare)
-
VARIQuery: VAE Segment-Based Active Learning for Query Selection in Preference-Based Reinforcement Learning
- 2023
-
Ingår i: 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2023. - : Institute of Electrical and Electronics Engineers (IEEE). ; , s. 7878-7885
-
Konferensbidrag (refereegranskat)abstract
- Human-in-the-loop reinforcement learning (RL) methods actively integrate human knowledge to create reward functions for various robotic tasks. Learning from preferences shows promise as alleviates the requirement of demonstrations by querying humans on state-action sequences. However, the limited granularity of sequence-based approaches complicates temporal credit assignment. The amount of human querying is contingent on query quality, as redundant queries result in excessive human involvement. This paper addresses the often-overlooked aspect of query selection, which is closely related to active learning (AL). We propose a novel query selection approach that leverages variational autoencoder (VAE) representations of state sequences. In this manner, we formulate queries that are diverse in nature while simultaneously taking into account reward model estimations. We compare our approach to the current state-of-the-art query selection methods in preference-based RL, and find ours to be either on-par or more sample efficient through extensive benchmarking on simulated environments relevant to robotics. Lastly, we conduct an online study to verify the effectiveness of our query selection approach with real human feedback and examine several metrics related to human effort.
|
|
| 3. |
- Mitsioni, Ioanna, 1991-, et al.
(författare)
-
Safe Data-Driven Model Predictive Control of Systems with Complex Dynamics
- 2023
-
Ingår i: IEEE Transactions on robotics. - : Institute of Electrical and Electronics Engineers (IEEE). - 1552-3098 .- 1941-0468. ; 39:4, s. 3242-3258
-
Tidskriftsartikel (refereegranskat)abstract
- In this article, we address the task and safety performance of data-driven model predictive controllers (DD-MPC) for systems with complex dynamics, i.e., temporally or spatially varying dynamics that may also be discontinuous. The three challenges we focus on are the accuracy of learned models, the receding horizon-induced myopic predictions of DD-MPC, and the active encouragement of safety. To learn accurate models for DD-MPC, we cautiously, yet effectively, explore the dynamical system with rapidly exploring random trees (RRT) to collect a uniform distribution of samples in the state-input space and overcome the common distribution shift in model learning. The learned model is further used to construct an RRT tree that estimates how close the model's predictions are to the desired target. This information is used in the cost function of the DD-MPC to minimize the short-sighted effect of its receding horizon nature. To promote safety, we approximate sets of safe states using demonstrations of exclusively safe trajectories, i.e., without unsafe examples, and encourage the controller to generate trajectories close to the sets. As a running example, we use a broken version of an inverted pendulum where the friction abruptly changes in certain regions. Furthermore, we showcase the adaptation of our method to a real-world robotic application with complex dynamics: robotic food-cutting. Our results show that our proposed control framework effectively avoids unsafe states with higher success rates than baseline controllers that employ models from controlled demonstrations and even random actions.
|
|
| 4. |
- Pek, Christian, et al.
(författare)
-
SpaTiaL : monitoring and planning of robotic tasks using spatio-temporal logic specifications
- 2023
-
Ingår i: Autonomous Robots. - : Springer Nature. - 0929-5593 .- 1573-7527. ; 47:8, s. 1439-1462
-
Tidskriftsartikel (refereegranskat)abstract
- Many tasks require robots to manipulate objects while satisfying a complex interplay of spatial and temporal constraints. For instance, a table setting robot first needs to place a mug and then fill it with coffee, while satisfying spatial relations such as forks need to placed left of plates. We propose the spatio-temporal framework SpaTiaL that unifies the specification, monitoring, and planning of object-oriented robotic tasks in a robot-agnostic fashion. SpaTiaL is able to specify diverse spatial relations between objects and temporal task patterns. Our experiments with recorded data, simulations, and real robots demonstrate how SpaTiaL provides real-time monitoring and facilitates online planning. SpaTiaL is open source and easily expandable to new object relations and robotic applications.
|
|
| 5. |
- Vahs, Matti, et al.
(författare)
-
Belief Control Barrier Functions for Risk-Aware Control
- 2023
-
Ingår i: IEEE Robotics and Automation Letters. - : Institute of Electrical and Electronics Engineers Inc.. - 2377-3766. ; 8:12, s. 8565-8572
-
Tidskriftsartikel (refereegranskat)abstract
- Ensuring safety in real-world robotic systems is often challenging due to unmodeled disturbances and noisy sensors. To account for such stochastic uncertainties, many robotic systems leverage probabilistic state estimators such as Kalman filters to obtain a robot's belief, i.e. a probability distribution over possible states. We propose belief control barrier functions (BCBFs) to enable risk-aware control, leveraging all information provided by state estimators. This allows robots to stay in predefined safety regions with desired confidence under these stochastic uncertainties. BCBFs are general and can be applied to a variety of robots that use extended Kalman filters as state estimator. We demonstrate BCBFs on a quadrotor that is exposed to external disturbances and varying sensing conditions. Our results show improved safety compared to traditional state-based approaches while allowing control frequencies of up to 1 kHz.
|
|
| 6. |
- Vahs, Matti, et al.
(författare)
-
Risk-aware Spatio-temporal Logic Planning in Gaussian Belief Spaces
- 2023
-
Ingår i: Proceedings - ICRA 2023. - : Institute of Electrical and Electronics Engineers (IEEE). ; , s. 7879-7885
-
Konferensbidrag (övrigt vetenskapligt/konstnärligt)abstract
- In many real-world robotic scenarios, we cannot assume exact knowledge about a robot’s state due to unmodeled dynamics or noisy sensors. Planning in belief space addresses this problem by tightly coupling perception and planning modules to obtain trajectories that take into account the environment’s stochasticity. However, existing works are often limited to tasks such as the classic reach-avoid problem and do not provide risk awareness. We propose a risk-aware planning strategy in belief space that minimizes the risk of violating a given specification and enables a robot to actively gather information about its state. We use Risk Signal Temporal Logic (RiSTL) as a specification language in belief space to express complex spatio-temporal missions including predicates over Gaussian beliefs. We synthesize trajectories for challenging scenarios that cannot be expressed through classical reach-avoid properties and show that risk-aware objectives improve the uncertainty reduction in a robot’s belief.
|
|
| 7. |
- van Waveren, Sanne, et al.
(författare)
-
Generating Scenarios from High-Level Specifications for Object Rearrangement Tasks
- 2023
-
Ingår i: 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2023. - : Institute of Electrical and Electronics Engineers (IEEE). ; , s. 11420-11427
-
Konferensbidrag (refereegranskat)abstract
- Rearranging objects is an essential skill for robots. To quickly teach robots new rearrangements tasks, we would like to generate training scenarios from high-level specifications that define the relative placement of objects for the task at hand. Ideally, to guide the robot’s learning we also want to be able to rank these scenarios according to their difficulty. Prior work has shown how generating diverse scenario from specifications and providing the robot with easy-to-difficult samples can improve the learning. Yet, existing scenario generation methods typically cannot generate diverse scenarios while controlling their difficulty. We address this challenge by conditioning generative models on spatial logic specifications to generate spatially-structured scenarios that meet the specification and desired difficulty level. Our experiments showed that generative models are more effective and data-efficient than rejection sampling and that the spatially-structured scenarios can drastically improve training of downstream tasks by orders of magnitude.
|
|
| 8. |
- van Waveren, Sanne, et al.
(författare)
-
Increasing perceived safety in motion planning for human-drone interaction
- 2023
-
Ingår i: HRI 2023. - New York, NY, USA : Association for Computing Machinery (ACM). ; , s. 446-455
-
Konferensbidrag (refereegranskat)abstract
- Safety is crucial for autonomous drones to operate close to humans. Besides avoiding unwanted or harmful contact, people should also perceive the drone as safe. Existing safe motion planning approaches for autonomous robots, such as drones, have primarily focused on ensuring physical safety, e.g., by imposing constraints on motion planners. However, studies indicate that ensuring physical safety does not necessarily lead to perceived safety. Prior work in Human-Drone Interaction (HDI) shows that factors such as the drone's speed and distance to the human are important for perceived safety. Building on these works, we propose a parameterized control barrier function (CBF) that constrains the drone's maximum deceleration and minimum distance to the human and update its parameters on people's ratings of perceived safety. We describe an implementation and evaluation of our approach. Results of a withinsubject user study (Ng= 15) show that we can improve perceived safety of a drone by adjusting to people individually.
|
|
