| 1. |
- Adib Yaghmaie, Farnaz, et al.
(författare)
-
A New Result on Robust Adaptive Dynamic Programming for Uncertain Partially Linear Systems
- 2019
-
Ingår i: 2019 IEEE 58TH CONFERENCE ON DECISION AND CONTROL (CDC). - : IEEE. - 9781728113982 - 9781728113999 ; , s. 7480-7485
-
Konferensbidrag (refereegranskat)abstract
- In this paper, we present a new result on robust adaptive dynamic programming for the Linear Quadratic Regulation (LQR) problem, where the linear system is subject to unmatched uncertainty. We assume that the states of the linear system are fully measurable and the matched uncertainty models unmeasurable states with an unspecified dimension. We use the small-gain theorem to give a sufficient condition such that the generated policies in each iteration of on-policy and off-policy routines guarantee robust stability of the overall uncertain system. The sufficient condition can be used to design the weighting matrices in the LQR problem. We use a simulation example to demonstrate the result.
|
|
| 2. |
- Adib Yaghmaie, Farnaz, et al.
(författare)
-
Differential graphical games for H-infinity control of linear heterogeneous multiagent systems
- 2019
-
Ingår i: International Journal of Robust and Nonlinear Control. - : WILEY. - 1049-8923 .- 1099-1239. ; 29:10, s. 2995-3013
-
Tidskriftsartikel (refereegranskat)abstract
- Differential graphical games have been introduced in the literature to solve state synchronization problem for linear homogeneous agents. When the agents are heterogeneous, the previous notion of graphical games cannot be used anymore and a new definition is required. In this paper, we define a novel concept of differential graphical games for linear heterogeneous agents subject to external unmodeled disturbances, which contain the previously introduced graphical game for homogeneous agents as a special case. Using our new formulation, we can solve both the output regulation and H-infinity output regulation problems. Our graphical game framework yields coupled Hamilton-Jacobi-Bellman equations, which are, in general, impossible to solve analytically. Therefore, we propose a new actor-critic algorithm to solve these coupled equations numerically in real time. Moreover, we find an explicit upper bound for the overall L2-gain of the output synchronization error with respect to disturbance. We demonstrate our developments by a simulation example.
|
|
| 3. |
- Adib Yaghmaie, Farnaz, et al.
(författare)
-
Linear Quadratic Control Using Model-Free Reinforcement Learning
- 2023
-
Ingår i: IEEE Transactions on Automatic Control. - : IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC. - 0018-9286 .- 1558-2523. ; 68:2, s. 737-752
-
Tidskriftsartikel (refereegranskat)abstract
- In this article, we consider linear quadratic (LQ) control problem with process and measurement noises. We analyze the LQ problem in terms of the average cost and the structure of the value function. We assume that the dynamics of the linear system is unknown and only noisy measurements of the state variable are available. Using noisy measurements of the state variable, we propose two model-free iterative algorithms to solve the LQ problem. The proposed algorithms are variants of policy iteration routine where the policy is greedy with respect to the average of all previous iterations. We rigorously analyze the properties of the proposed algorithms, including stability of the generated controllers and convergence. We analyze the effect of measurement noise on the performance of the proposed algorithms, the classical off-policy, and the classical Q-learning routines. We also investigate a model-building approach, inspired by adaptive control, where a model of the dynamical system is estimated and the optimal control problem is solved assuming that the estimated model is the true model. We use a benchmark to evaluate and compare our proposed algorithms with the classical off-policy, the classical Q-learning, and the policy gradient. We show that our model-building approach performs nearly identical to the analytical solution and our proposed policy iteration based algorithms outperform the classical off-policy and the classical Q-learning algorithms on this benchmark but do not outperform the model-building approach.
|
|
| 4. |
- Adib Yaghmaie, Farnaz, 1987-, et al.
(författare)
-
Online Optimal Tracking of Linear Systems with Adversarial Disturbances
- 2023
-
Ingår i: Transactions on Machine Learning Research. - 2835-8856. ; :04
-
Tidskriftsartikel (refereegranskat)abstract
- This paper presents a memory-augmented control solution to the optimal reference tracking problem for linear systems subject to adversarial disturbances. We assume that the dynamics of the linear system are known and that the reference signal is generated by a linear system with unknown dynamics. Under these assumptions, finding the optimal tracking controller is formalized as an online convex optimization problem that leverages memory of past disturbance and reference values to capture their temporal effects on the performance. That is, a (disturbance, reference)-action control policy is formalized, which selects the control actions as a linear map of the past disturbance and reference values. The online convex optimization is then formulated over the parameters of the policy on its past disturbance and reference values to optimize general convex costs. It is shown that our approach outperforms robust control methods and achieves a tight regret bound O(√T) where in our regret analysis, we have benchmarked against the best linear policy.
|
|
| 5. |
- Adib Yaghmaie, Farnaz, et al.
(författare)
-
Output regulation of unknown linear systems using average cost reinforcement learning
- 2019
-
Ingår i: Automatica. - : PERGAMON-ELSEVIER SCIENCE LTD. - 0005-1098 .- 1873-2836. ; 110
-
Tidskriftsartikel (refereegranskat)abstract
- In this paper, we introduce an optimal average cost learning framework to solve output regulation problem for linear systems with unknown dynamics. Our optimal framework aims to design the controller to achieve output tracking and disturbance rejection while minimizing the average cost. We derive the Hamilton-Jacobi-Bellman (HJB) equation for the optimal average cost problem and develop a reinforcement algorithm to solve it. Our proposed algorithm is an off-policy routine which learns the optimal average cost solution completely model-free. We rigorously analyze the convergence of the proposed algorithm. Compared to previous approaches for optimal tracking controller design, we elevate the need for judicious selection of the discounting factor and the proposed algorithm can be implemented completely model-free. We support our theoretical results with a simulation example. (C) 2019 Elsevier Ltd. All rights reserved.
|
|
| 6. |
- Adib Yaghmaie, Farnaz, et al.
(författare)
-
Using Reinforcement Learning for Model-free Linear Quadratic Control with Process and Measurement Noises
- 2019
-
Ingår i: 2019 IEEE 58TH CONFERENCE ON DECISION AND CONTROL (CDC). - : IEEE. - 9781728113982 - 9781728113999 ; , s. 6510-6517
-
Konferensbidrag (refereegranskat)abstract
- In this paper, we analyze a Linear Quadratic (LQ) control problem in terms of the average cost and the structure of the value function. We develop a completely model-free reinforcement learning algorithm to solve the LQ problem. Our algorithm is an off-policy routine where each policy is greedy with respect to all previous value functions. We prove that the algorithm produces stable policies given that the estimation errors remain small. Empirically, our algorithm outperforms the classical Q and off-policy learning routines.
|
|
| 7. |
- Modares, Amir, et al.
(författare)
-
Safe Reinforcement Learning via a Model-Free Safety Certifier
- 2023
-
Ingår i: IEEE Transactions on Neural Networks and Learning Systems. - : IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC. - 2162-237X .- 2162-2388.
-
Tidskriftsartikel (refereegranskat)abstract
- This article presents a data-driven safe reinforcement learning (RL) algorithm for discrete-time nonlinear systems. A data-driven safety certifier is designed to intervene with the actions of the RL agent to ensure both safety and stability of its actions. This is in sharp contrast to existing model-based safety certifiers that can result in convergence to an undesired equilibrium point or conservative interventions that jeopardize the performance of the RL agent. To this end, the proposed method directly learns a robust safety certifier while completely bypassing the identification of the system model. The nonlinear system is modeled using linear parameter varying (LPV) systems with polytopic disturbances. To prevent the requirement for learning an explicit model of the LPV system, data-based $\lambda$ -contractivity conditions are first provided for the closed-loop system to enforce robust invariance of a prespecified polyhedral safe set and the systems asymptotic stability. These conditions are then leveraged to directly learn a robust data-based gain-scheduling controller by solving a convex program. A significant advantage of the proposed direct safe learning over model-based certifiers is that it completely resolves conflicts between safety and stability requirements while assuring convergence to the desired equilibrium point. Data-based safety certification conditions are then provided using Minkowski functions. They are then used to seemingly integrate the learned backup safe gain-scheduling controller with the RL controller. Finally, we provide a simulation example to verify the effectiveness of the proposed approach.
|
|
