| 1. |
- Colin, Kevin, et al.
(författare)
-
Regret Minimization for Linear Quadratic Adaptive Controllers Using Fisher Feedback Exploration
- 2022
-
Ingår i: IEEE Control Systems Letters. - : Institute of Electrical and Electronics Engineers (IEEE). - 2475-1456. ; 6, s. 2870-2875
-
Tidskriftsartikel (refereegranskat)abstract
- In this letter, we study the trade-off between exploration and exploitation for linear quadratic adaptive control. This trade-off can be expressed as a function of the exploration and exploitation costs, called cumulative regret. It has been shown over the years that the optimal asymptotic rate of the cumulative regret is in many instances O(root T). In particular, this rate can be obtained by adding a white noise external excitation, with a variance decaying as O(1/root T). As the amount of excitation is pre-determined, such approaches can be viewed as open loop control of the external excitation. In this contribution, we approach the problem of designing the external excitation from a feedback perspective leveraging the well known benefits of feedback control for decreasing sensitivity to external disturbances and system-model mismatch, as compared to open loop strategies. We base the feedback on the Fisher information matrix which is a measure of the accuracy of the model. Specifically, the amplitude of the exploration signal is seen as the control input while the minimum eigenvalue of the Fisher matrix is the variable to be controlled. We call such exploration strategies Fisher Feedback Exploration (F2E). We propose one explicit F2E design, called Inverse Fisher Feedback Exploration (IF2E), and argue that this design guarantees the optimal asymptotic rate for the cumulative regret. We provide theoretical support for IF2E and in a numerical example we illustrate benefits of IF2E and compare it with the open loop approach as well as a method based on Thompson sampling.
|
|
| 2. |
- Ferizbegovic, Mina, et al.
(författare)
-
Bayes control of hammerstein systems
- 2021
-
Ingår i: 19th IFAC Symposium on System Identification, SYSID 2021. - : Elsevier BV. ; , s. 755-760, s. 755-760
-
Konferensbidrag (refereegranskat)abstract
- In this paper, we consider data driven control of Hammerstein systems. For such systems a common control structure is a transfer function followed by a static output nonlinearity that tries to cancel the input nonlinearity of the system, which is modeled as a polynomial or piece-wise linear function. The linear part of the controller is used to achieve desired disturbance rejection and tracking properties. To design a linear part of the controller, we propose a weighted average risk criterion with the risk being the average of the squared L2 tracking error. Here the average is with respect to the observations used in the controller and the weighting is with respect to how important it is to have good control for different impulse responses. This criterion corresponds to the average risk criterion leading to the Bayes estimator and we therefore call this approach Bayes control. By parametrizing the weighting function and estimating the corresponding hyperparameters we tune the weighting function to the information regarding the true impulse response contained in the data set available to the user for the control design. The numerical results show that the proposed methods result in stable controllers with performance comparable to the optimal controller, designed using the true input nonlinearity and true plant.
|
|
| 3. |
- Ferizbegovic, Mina
(författare)
-
Dual control concepts for linear dynamical systems
- 2022
-
Doktorsavhandling (övrigt vetenskapligt/konstnärligt)abstract
- We study simultaneous learning and control of linear dynamical systems. In such a setting, control policies are derived with respect to two objectives: i) to control the system as well as possible, given the current knowledge of system dynamics (exploitation), and ii) to gather as much information as possible about the unknown system that can improve control (exploration).These two objectives are often in conflict, and this phenomenon is known as the exploration-exploitation trade-off.More specifically, in the context of simultaneous learning and control, we consider: linear quadratic regulation (LQR) problem, model reference control, and data-driven control based on Willems \textit{et al.}'s fundamental lemma. First, we consider the LQR problem with unknown dynamics. We present robust and certainty equivalence (CE) model-based control methods that balance exploration and exploitation. We focus on control policies that can be iteratively updated after sequentially collecting data.We propose robust (with respect to parameter uncertainty) LQR design methods. To quantify uncertainty, we derive a methodbased on Bayesian inference, which is directly applicable to robust control synthesis. To begin, we derive a robust controller to minimize the worst-case cost, with high probability, given the empirical observation of the system. This robust controller synthesis is then used to derive a robust dual controller, which updates its control policy after collecting data. An episode in which data is collected is called exploration, and the episode using an updated control policy called exploitation. The objective is to minimize the worst-case cost of the updated control policy, requiring that a given exploration budget constrains the worst-case cost during exploration. Additionally, we derive methods that balance exploration and exploitation to minimize the cumulative worst-case cost for a fixed number of episodes. In this thesis, we refer to such a problem as robust reinforcement learning. Essentially, it is a robust dual controller aiming to minimize the cumulative worst-case cost, and that updates its control policy in each episode.Numerical experiments show that the proposed methods perform better than existing state-of-the-art algorithms. Moreover, experiments also indicate that the exploration prioritizes the uncertainty reduction in the parameters that matter most for control.A control policy using the CE principle for LQR consists of a sum of an optimal controller calculated using estimated dynamics at time $t$, and an additive external excitation. It has been shown over the years that the optimal asymptotic rate of regret is in many instances $\mathcal{O}(\sqrt{T})$. In particular, this rate can be obtained by adding a white noise external excitation, with a variance decaying as $\gamma/\sqrt{T}$, where $\gamma$ is a predefined constant. As the amount of excitation is pre-determined, such approaches can be viewed as open-loop control of the external excitation. In this thesis, we approach the problem of designing the external excitation from a feedback perspective leveraging the well-known benefits of feedback control for decreasing sensitivity to external disturbances and system-model mismatch, as compared to open-loop strategies. The benefits of this approach over the open-loop approach can be seen in the case of unmodeled dynamics and disturbances. However, even when using the benefits of feedback control, we do not calculate the optimal amount of external excitation. To find the optimal amount of external excitation, we suggest exploration strategies that are based on a time-dependent scaling $\gamma_t$ and can attain cumulative regret similar to or lower than cumulative regret obtained for optimal scaling $\gamma^*$ according to numerical examples.Second, we consider the model reference control problem with the goal of proposing a data-driven robust control design method based on an average risk criterion, which we call Bayes control. We show that this approach has very close ties to the Bayesian kernel-based method, but the conceptual difference lies in the use of a deterministic respective stochastic setting for the system parameters. Finally, we consider data-driven control using Willems \textit{et al.}'s fundamental lemma. First, we propose variations of the fundamental lemma that, instead of a data trajectory, utilize correlation functions in the time domain, as well as power spectra of the input and the output in the frequency domain. Since data-driven control using the fundamental lemma can become a very expensive computation task for large datasets, the proposed variations are easy to computeeven for large datasets and can be efficient as a data compression technique. Second, we study connections of data informativity conditions between the results based on the fundamental lemma (finite time), and classical system identification. We show that finite time informativity conditions for state-space systems are closely linked to the identifiability conditions derived from the fundamental lemma. We prove that the obtained persistency of excitation conditions for infinite time are sufficient conditions for finite time informativity. Moreover, we reveal that the obtained conditions for a finite time in closed-loop are stricter than in classical system identification. This is a consequence of the noiseless data setting in the fundamental lemma that precludes the possibility of noise to excite the system in a feedback setting.
|
|
| 4. |
- Ferizbegovic, Mina, et al.
(författare)
-
Learning Robust LQ-Controllers Using Application Oriented Exploration
- 2020
-
Ingår i: IEEE Control Systems Letters. - : Institute of Electrical and Electronics Engineers (IEEE). - 2475-1456. ; 4:1, s. 19-24
-
Tidskriftsartikel (refereegranskat)abstract
- This letter concerns the problem of learning robust LQ-controllers, when the dynamics of the linear system are unknown. First, we propose a robust control synthesis method to minimize the worst-case LQ cost, with probability 1-δ , given empirical observations of the system. Next, we propose an approximate dual controller that simultaneously regulates the system and reduces model uncertainty. The objective of the dual controller is to minimize the worst-case cost attained by a new robust controller, synthesized with the reduced model uncertainty. The dual controller is subject to an exploration budget in the sense that it has constraints on its worst-case cost with respect to the current model uncertainty. In our numerical experiments, we observe better performance of the proposed robust LQ regulator over the existing methods. Moreover, the dual control strategy gives promising results in comparison with the common greedy random exploration strategies.
|
|
| 5. |
- Ferizbegovic, Mina, et al.
(författare)
-
Nonlinear FIR Identification with Model Order Reduction Steiglitz-McBride⁎
- 2018
-
Ingår i: IFAC-PapersOnLine. - : Elsevier B.V.. - 2405-8963. ; 51:15, s. 646-651
-
Tidskriftsartikel (refereegranskat)abstract
- In system identification, many structures and approaches have been proposed to deal with systems with non-linear behavior. When applicable, the prediction error method, analogously to the linear case, requires minimizing a cost function that is non-convex in general. The issue with non-convexity is more problematic for non-linear models, not only due to the increased complexity of the model, but also because methods to provide consistent initialization points may not be available for many model structures. In this paper, we consider a non-linear rational finite impulse response model. We observe how the prediction error method requires minimizing a non-convex cost function, and propose a three-step least-squares algorithm as an alternative procedure. This procedure is an extension of the Model Order Reduction Steiglitz-McBride method, which is asymptotically efficient in open loop for linear models. We perform a simulation study to illustrate the applicability and performance of the method, which suggests that it is asymptotically efficient.
|
|
| 6. |
- Ferizbegovic, Mina
(författare)
-
Robust learning and control of linear dynamical systems
- 2020
-
Licentiatavhandling (övrigt vetenskapligt/konstnärligt)abstract
- We consider the linear quadratic regulation problem when the plant is an unknown linear dynamical system. We present robust model-based methods based on convex optimization, which minimize the worst-case cost with respect to uncertainty around model estimates. To quantify uncertainty, we derive a methodbased on Bayesian inference, which is directly applicable to robust control synthesis.We focus on control policies that can be iteratively updated after sequentially collecting data. More specifically, we seek to design control policies that balance exploration (reducing model uncertainty) and exploitation (control of the system) when exploration must be safe (robust).First, we derive a robust controller to minimize the worst-case cost, with high probability, given the empirical observation of the system. This robust controller synthesis is then used to derive a robust dual controller, which updates its control policy after collecting data. An episode in which data is collected is called exploration, and the episode using an updated control policy is exploitation. The objective is to minimize the worst-case cost of the updated control policy, requiring that a given exploration budget constrains the worst-case cost during exploration.We look into robust dual control in both finite and infinite horizon settings. The main difference between the finite and infinite horizon settings is that the latter does not consider the length of the exploration and exploitation phase, but it rather approximates the cost using the infinite horizon cost. In the finite horizon setting, we discuss how different exploration lengths affect the trade-off between exploration and exploitation.Additionally, we derive methods that balance exploration and exploitation to minimize the cumulative worst-case cost for a fixed number of episodes. In this thesis, we refer to such a problem as robust reinforcement learning. Essentially, it is a robust dual controller aiming to minimize the cumulative worst-case cost, and that updates its control policy in each episode.Numerical experiments show that the proposed methods have better performance compared to existing state-of-the-art algorithms. Moreover, experiments also indicate that the exploration prioritizes the uncertainty reduction in the parameters that matter most for control.
|
|
| 7. |
- Ferizbegovic, Mina, et al.
(författare)
-
Weighted Null-Space Fitting for Cascade Networks with Arbitrary Location of Sensors and Excitation Signals
- 2018
-
Ingår i: 57th IEEE Conference on Decision and Control. - : Institute of Electrical and Electronics Engineers (IEEE). - 9781538613955 ; , s. 4707-4712
-
Konferensbidrag (refereegranskat)abstract
- Identification of a complete dynamic network affected by sensor noise using the prediction error method is often too complex. One of the reasons for this complexity is the requirement to minimize a non-convex cost function, which becomes more difficult with more complex networks. In this paper, we consider serial cascade networks affected by sensor noise. Recently, the Weighted Null-Space Fitting method has been shown to be appropriate for this setting, providing asymptotically efficient estimates without suffering from non-convexity; however, applicability of the method was subject to some conditions on the locations of sensors and excitation signals. In this paper, we drop such conditions, proposing an extension of the method that is applicable to general serial cascade networks. We formulate an algorithm that describes application of the method in a general setting, and perform a simulation study to illustrate the performance of the method, which suggests that this extension is still asymptotically efficient.
|
|
| 8. |
- Ferizbegovic, Mina, et al.
(författare)
-
Willems' fundamental lemma based on second-order moments
- 2021
-
Ingår i: 2021 60TH IEEE CONFERENCE ON DECISION AND CONTROL (CDC). - : Institute of Electrical and Electronics Engineers (IEEE). - 9781665436595 ; , s. 396-401
-
Konferensbidrag (refereegranskat)abstract
- In this paper, we propose variations of Willems' fundamental lemma that utilize second-order moments such as correlation functions in the time domain and power spectra in the frequency domain. We believe that using a formulation with estimated correlation coefficients is suitable for data compression, and possibly can reduce noise. Also, the formulations in the frequency domain can enable modeling of a system in a frequency region of interest.
|
|
| 9. |
- Fonken, Stefanie, et al.
(författare)
-
Consistent identification of dynamic networks subject to white noise using Weighted Null-Space Fitting
- 2020
-
Ingår i: 21st IFAC World Congress on Automatic Control - Meeting Societal Challenges. - : Elsevier BV.
-
Konferensbidrag (refereegranskat)abstract
- Identification of dynamic networks has been a flourishing area in recent years. However, there are few contributions addressing the problem of simultaneously identifying all modules in a network of given structure. In principle the prediction error method can handle such problems but this methods suffers from well known issues with local minima and how to find initial parameter values. Weighted Null-Space Fitting is a multi-step least-squares method and in this contribution we extend this method to rational linear dynamic networks of arbitrary topology with modules subject to white noise disturbances. We show that WNSF reaches the performance of PEM initialized at the true parameter values for a fairly complex network, suggesting consistency and asymptotic efficiency of the proposed method.
|
|
| 10. |
- Galrinho, Miguel, et al.
(författare)
-
Weighted Null-Space Fitting for Identification of Cascade Networks⁎
- 2018
-
Ingår i: IFAC-PapersOnLine. - : Elsevier B.V.. - 2405-8963. ; 51:15, s. 856-861
-
Tidskriftsartikel (refereegranskat)abstract
- For identification of systems embedded in dynamic networks, the prediction error method (PEM) with a correct parametrization of the complete network provides asymptotically efficient estimates. However, the network complexity often hinders a successful application of PEM, which requires minimizing a non-convex cost function that can become more intricate for more complex networks. For this reason, identification in dynamic networks often focuses in obtaining consistent estimates of modules of interest. A downside of these approaches is that splitting the network in several modules for identification often costs asymptotic efficiency. In this paper, we consider dynamic networks with the modules connected in serial cascade, with measurements affected by sensor noise. We propose an algorithm that estimates all the modules in the network simultaneously without requiring the minimization of a non-convex cost function. This algorithm is an extension of Weighted Null-Space Fitting (WNSF), a weighted least-squares method that provides asymptotically efficient estimates for single-input single-output systems. We illustrate the performance of the algorithm with simulation studies, which suggest that a network WNSF method may also be asymptotically efficient when applied to cascade structures. Finally, we discuss the possibility of extension to more general networks affected by sensor noise.
|
|
