Theoretical foundations

This project will investigate and characterise the differences between dynamic models based on open-loop experimental data, and those derived from closed-loop operation. Open-loop configurations lead to predictive models with errors that can reliably be characterised, while feedback configurations lead to characterisations in which the errors are heavy-tailed. The standard measures of model suitability are not applicable in such situations making the objective of minimising variance or mean-square error poorly posed. We will develop modelling frameworks and methods tailored to characterising errors based on the form of the stochastic error distribution.

The project addresses the question of optimal system layout for facilitating decentralized control and optimal operation of a network of power converters with generators, storages and loads. Special emphasis is placed on the system capability of operating in a decentralized fashion to meet specific local control objectives (e.g. keeping the local voltage within prescribed limits) and system-wide control objectives (e.g. fair proportional load sharing among the inverters, provision of ancillary services). Sensitivity analysis will enable to define the boundaries of a completely decentralized structure and where adding a hierarchical level enables a significant performance improvement.

Manufacturing tasks are often repetitive, involving the same processing steps being repeated over and over to produce identical parts. This makes them well suited for data driven methods that exploit repetition, like the ones developed for autonomous racing. This project addresses deploying the data-assisted predictive control methods in manufacturing applications. The resulting optimisation problems are often of formidable complexity, making them challenging to solve in real time. This calls for efficient solution methods that inherit the stability and performance guarantees of the predictive control design even when the search for the optimal solution has to be terminated prematurely.

The DeepGreen snooker robot project aims to build a robot capable of challenging the best human players. Snooker (similar to billiard) is a game that combines both advanced strategy and physical skills. A single game can be abstractly represented as a zero-sum dynamical game with an extremely highly dimensional state-space (position of each ball, score, and current player), a continuous action space (angle, speed and position of the cue) and nonlinear hybrid dynamics (a combination of the games rules and physics governing the interaction of the cue, the balls and the table). All the above characteristics make snooker the ideal testbed for approximate dynamic programming or reinforcement learning algorithms that have the ambition of filling the “reality gap” and transition from digital simulations to implementation in the real world. The principal objective of the project is the design of the strategy policy for the robotic player. The main challenges are (i) the need of a suitable function approximation scheme for the policy or value function to exploit the characteristics of the game and reduce the dimensionality of the decision problem and (ii) combining the use of data from both a physics engine and the real physical system to reduce the “reality gap” and obtain a strategy that performs well (and improves over time) on the physical robot and not only in a simulated environment.

In this project we aim to transfer the powerful modern methods of distributionally robust optimization to dynamic decision problems and optimal control problems. Important tasks to be addressed include the derivation of tractable convex reformulations or approximations for the original distributionally robust optimization problems, the design of customized first-order methods for their efficient solution, the derivation of out-of-sample performance and asymptotic consistency guarantees, and the study of the time consistency properties of the new control models.

A key limitation of current deep reinforcement learning (RL) approaches is their need for accurate computational models and focus on fully observable domains.  In many real-world domains, such as those considered in the NCCR, only approximate dynamics models exist, and partial observability is paramount. In our research, we will investigate connections between reinforcement learning and causal inference. We explore novel approaches for off-policy evaluation based on ideas from causal inference, and study the use of experimental design for causal discovery for safe exploration in RL. We plan to also utilize ideas from meta-learning in order to transfer inductive biases across multiple related tasks.

Decentralized algorithms for governing multi-modal transport networks with multiple entities and operators might not work properly when congestion is unevenly distributed. As we are scaling-up from the local bottleneck-oriented issues we must consider how local regulators (e.g. traffic lights, ramp meters, pricing) can best communicate in a distributed manner to reach an agreement on their control decisions, so as to ensure local cooperation with its geographical neighbours towards realization of city-level mobility objectives. Establishing proper rules and mechanisms within each subsystem allows self-organization of individual components while ensuring coherence.

We study how flexible devices, consumers and producers can participate in an automated fashion in an energy market that yields efficient market outcomes. As the availability of distributed resources in terms of production and flexibility is uncertain, probabilistic approaches for the automated bidding but also for the determination of the required reserves are necessary. Given that distributions of the willingness of the participants to accept prices are unknown, we intend to formulate and explore distributionally robust optimization models.

Cyberphysical systems are composed by subsystems linked through physical interconnections and communication channels enabling distributed control architectures. By leveraging recent results on optimal control, we will develop distributed algorithms for the data-driven generation of networked regulators adapted to the communication topology. Special attention will be devoted to the decentralization of the design process, so as to facilitate the addition and removal of subsystems in a plug-and-play fashion. As benchmark examples, we will consider energy systems and manufacturing processes, which are key applications in the NCCR.

In spite of several success stories, standard deep networks  can lead to unstable forward propagation or suffer from an ill-posed learning process, which might result in vulnerability to adversarial attacks. In this project we will study deep network architectures based on ODEs for mitigating these problems. Furthermore, we will investigate the use of these networks for parametrizing regulators that can be tuned from available data. The new methods will be tested for the control of inverters, smart loads and microgrid clusters, which are relevant for the energy systems considered in the NCCR.

We will investigate the theoretical and practical challenges of using energy harvesting to power nodes distributed control systems. Combining energy sources such as temperature and vibration with battery systems and wireless links enables the placement of sensor nodes where they are needed for the best data quality, regardless of the availability of wired power or communication. In the context of the NCCR, we will jointly investigate some of the open challenges in the design of autonomous, energy-neutral automation systems. In a second phase, we will investigate suitable demonstrators and applications such as wireless sensing and control in motor control and energy systems.

This project will study formal methods for reduced complexity design and verification of embedded optimization techniques for the control of high-speed nonlinear constrained systems. We will focus on machine learning techniques for differentiable parametric optimization and their application to the control of fast, nonlinear dynamic systems with a power conversion system taken as an important exemplar case study.

For temporally varying electric distribution networks, reliable information about the system topology and parameters may be missing or outdated. In the project we will develop online estimation algorithms for the network reconstruction, so as to automatically track changes, even in the presence of noisy and incomplete data. Furthermore, we will blend online estimation algorithms with higher-level controllers commonly used in distribution networks  and apply our methods to the moon-shot testbed developed in the NCCR. Our ultimate goal is to contribute to the development of autonomous electric networks that self-adapt in real-time to changing ambient conditions.

Recent results on data-based control exploit alternative parametrizations of linear and nonlinear systems, such as those provided by behavioral theory and deep neural networks. In this project we focus on Input/Output Behavioral (IOB) models that have been used for control by assuming noiseless measurement or deterministic bounded disturbances. The goal of the project is to study the relation between IOB-based control methods and the existing approaches for stochastic systems, including system identification and stochastic model predictive control. New controllers will be tested on the energy systems relevant for the NCCR.

The goal of the project is to study data-driven modelling and control frameworks for large-scale systems stemming from the interconnection of several subsystems, which are of primary interest to the NCCR. For this purpose, we will develop (i) innovative probabilistic machine learning algorithms, based on Gaussian Processes (GPs), for batch and online estimation of models complying with the coupling structure of the system and (ii) model-based Distributed Reinforcement Learning (DRL) algorithms for the generation of networked control policies. Our DRL schemes will exploit the epistemic model uncertainty to drive exploration while ensuring stability of the distributed controllers.

This project uses ideas from mathematical optimization and learning theory to derive statistical guarantees for reinforcement learning (RL). While there has recently been significant progress in the understanding of finite-sample guarantees for the linear quadratic regulator (LQR) problem, it remains unclear how to generalize these results beyond the LQR setting. We plan to establish a principled approach to RL based on distributionally robust optimization with the goal to derive statistical guarantees for RL problems beyond the LQR setup. We also aim to develop efficient computational algorithms.

Emerging shared-mobility systems create additional opportunities to decrease car ownership and congestion. Asymmetric demand creates imbalances in the distribution of vehicles for these systems. To maximize the covered demand and decrease the waiting time of passengers, vehicle distribution has to be rebalanced with relocations. The potentially very high number of vehicles necessitates using distributed control algorithms to efficiently solve this problem. Presence of multiple companies competing to serve the same demand can be addressed via game theoretic approaches. Μatching algorithms to create shared rides can be combined with the coverage problem to increase coverage in areas with high demand. Fairness in covering low demand areas will also be investigated.

The project aims at developing an autonomous and reconfigurable control architecture where each converter operates in plug-and-play mode, i.e., without additional communication overhead or detailed model knowledge, and under variable system conditions. This requires in a first step that each converter estimates online the equivalent circuit as seen at its terminals. Based on such estimates, the converter controller reconfigures its structure and adapts online its parameters in order to optimally perform under the different grid configurations and operating conditions.

Recent years have seen a surge of academic and industrial interest in distributionally robust optimization, where the probability distribution of the uncertain problem parameters is itself uncertain and one seeks decisions that are optimal in view of the most adverse distribution within a given ambiguity set. In this project we will study online distributionally robust optimization with streaming data. The key motivation is that dynamic stochastic processes (as encountered in control, estimating, and filtering problems) demand recursive and online solutions with real-time computational constraints. We plan to implement our approaches on various energy system platforms.

Online feedback optimization refers to the design of feedback controllers that asymptotically steer a physical system to the solution of an optimization problem while respecting physical and operational constraints. Here we are interested in exploring self-interested agents that do not want to cooperate for the sake of achieving a common goal but first and foremost have their own interest in mind. A relevant real-world example is selfish and uncoordinated congestion control by different power transmission system operators. We will investigate distributed Nash-seeking algorithms to solve the resulting antagonistic decision-making problems, and also deploy them in numerical and real-world case studies.

We want to develop continuous-time optimality-seeking algorithms on the space of probability distributions. We envision our approach to result in practically useful algorithms to probabilistic (e.g. distributionally robust) optimization problems. Furthermore, our framework is directly applicable to optimization and control of traffic flow in a density modelling approach or a continuum modelling of multi-agent systems.

We will demonstrate that multiple competitive agents can efficiently share a mobility infrastructure without the need for an external coordinator. Standard game-theoretic approaches to this problem fall short in case of dynamic systems as encountered in autonomous mobility, coordinated use of the mobility space, traffic congestion control, etc.  We will develop a new mathematical formalism and computational methods blending the concept of game-theoretic and dynamic equilibria. Autonomous mobility is an important application due to the importance of fairness and efficiency in resource use, the large number of interacting agents, and the need for automated and scalable solutions.

We investigate how classical methods from system identification, dynamical systems, and control can be combined with techniques from non- parametric machine learning to develop a rigorous framework for continuous-time model-based deep reinforcement learning. Concretely, we will learn a continuous-time dynamics model via neural ordinary differential equations, along with associated uncertainty bounds that can be used to trade exploration and exploitation. Ultimately, our framework should enable control design for complex and uncertain systems, and we plan to apply it to problems in building automation and grid-connected converters.

Our goal is to develop novel algorithms for risk-aware Bayesian optimization and demonstrate them in a case study for high-precision linear drives requiring position accuracy and stability with less than 5 nm deviation. For this system, as for many other applications, it is crucial to optimize expected performance and ensure reliable outcomes.  We will develop computationally efficient algorithms trading expected performance with the variance in the outcome, considering risk measures on the optimization constraints, and compiling thi acquisition functions into neural network policies.