Abstract—This article investigates a lifting method enhanced modified Q-Learning to be known as the special case of reinforcement learning (RL) for optimal control design of a class of periodic systems. Due to the purpose of investigating periodic systems with many sub-equations by only one dynamic equation, a lifting method is utilized to transfer the periodic LQR to time-invariant LQR in an augmented system. After obtaining the corresponding model to be described by only one dynamic equation, partition technique is developed to achieve easier optimal control design. Due to the difficulty in analytically solving Hamilton-Jacobi-Bellman equation, adaptive reinforcement learning (ARL) is studied using iteration algorithm. The model-free Q-learning solution with the advantage of considering the Bellman function of two variables is proposed with the expanded system and the convergence analysis is discussed by considering the poles position on the complex plane as well as Lyapunov stability theory. The proposed Q-Learning method is realized online to find the optimal controller based on the system states' data collection, and the computation of Bellman function and control policy is only in one step of the proposed algorithm. The tracking and performance of the proposed methods are illustrated for spacecraft systems with appropriate simulation results.

Index Terms—lifting technique, Q-learning, discrete-time linear periodic systems, LQR, adaptive reinforcement learning

Cite: Phuong Nam Dao, Hong Quang Nguyen, "Q-Learning Algorithms in Control Design of Discrete-Time Linear Periodic Systems by Lifting Technique," International Journal of Mechanical Engineering and Robotics Research, Vol. 10, No.11, pp. 620-625, November 2021. DOI: 10.18178/ijmerr.10.11.620-625

Copyright © 2021 by the authors. This is an open access article distributed under the Creative Commons Attribution License (CC BY-NC-ND 4.0), which permits use, distribution and reproduction in any medium, provided that the article is properly cited, the use is non-commercial and no modifications or adaptations are made.