Via Automation - a small tour of automatic control: Photographs, filters and AI
At first glance, this might appear unrelated to automatic control. Yet the area of digital photography is filled with ideas that are deeply connected to our field.
In fact, nearly every picture you take nowadays involves some form of automatic estimation, filtering or optimization. Your phone continuously decides how to focus, how long the exposure should be, how to stabilise motion and even which image among several candidates is the ‘bestʼ one. Most of this happens in fractions of a second and, as Karl Johan Åström might say, largely as a hidden technology [1].
Let us try a small experiment.
Take a picture in some room with little light, while deliberately moving your phone. The resulting image will typically contain two imperfections: noise and blur. Noise appears because the camera sensor receives only a limited amount of light. Blur appears because the scene moves while the shutter is effectively ‘openʼ. Now repeat the experiment using your phoneʼs night mode. You will likely notice that the result is better.
What happened?
If your phone is new enough, it does not rely on a single photograph. Instead, it will rapidly capture multiple images and combine them ‘intelligentlyʼ, known as burst processing [2,5]. Roughly speaking, the software estimates which structures belong to the true scene and which structures are merely noise or motion artefacts. This is filtering in a very broad sense: extracting useful information from imperfect measurements.
Filtering has a long and fascinating history within automatic control and signal processing. During the 1960s, Rudolf E Kalman introduced (with colleagues) what is now called the Kalman filter—arguably one of the most influential algorithms ever developed in engineering [3]. The idea is as follows. Suppose we have a mathematical model describing how a dynamical system evolves, together with noisy measurements of that system, meaning, measurements of the output of the system. It turns out that we can optimally reconstruct the states of the underlying system under some structural assumption, for instance, when the underlying system is linear and the noise is Gaussian (ie a bell curve). Effectively, the model provides a first prediction, then, the filter uses incoming measurements and the noise statistics to correct this prediction.
Kalman filtering became instrumental in navigation, aerospace and robotics. Famously, Kalman filtering was of great use during the Apollo missions [4]. Now, conceptually, your phone performs similar estimation tasks whenever it stabilizes a video or tracks a moving face [5].
You can try another experiment yourself. Open your phone camera and record a short video while walking. Then compare this with old handheld footage from twenty years ago or so. If you are a student, ask your parents.
Now, shifting gears, mobile computational photography (ie your phone as opposed to a shitter camera with film) provides us with one of the most clear examples when it comes to the impossibility of deriving reasonable first-principle models. If we want to describe the motion of something like a gondola, we can often write down reasonably accurate differential equations. Then, these equations are of use for prediction and control.
Physics provides us with the structure. But now consider a different question:
What exactly is a ‘goodʼ photograph? This turns out to be much harder to formalise. Similarly, what precisely defines a dog in terms of just pixels? Describing the motion of a gondola is easier!
It is precisely in these situations—where clear mathematical models are absent, incomplete or simply unclear—that modern AI machinery became of great use [6].
Instead of working with explicit definitions, we let algorithms infer patterns directly from data. Large collections of images provide examples, and the AI system gradually learns structure implicitly available in the data. This is why AI has become so effective for tasks such as object recognition and the automatic curation of photo libraries; do we even have other options?
Indeed, many recent phone features rely heavily on this philosophy. For example, based on examples, ‘magic eraserʼ tools estimate what should plausibly exist behind an unwanted object.
You may do another simple experiment.
Take an old blurry photograph and apply one of the modern ‘unblurʼ tools that you will find on your phone [7]. Text might become readable, you can recognise faces and so forth. However, if you inspect the result carefully, you may also notice, occasionally, strange and unnatural behaviour; we have all seen the pictures with people having two lefts hands. The software is not merely recovering information; it is partially generating ‘plausibleʼ information. It tries to fill the gaps. This should be contrasted with classical signal processing, where usually the goal is to remove noise from an existing signal.
Of course, this also raises many questions. If an algorithm generates details of an image that are possibly never truly captured, are we still connected to reality? Are we doing something that is ‘closeʼ to reality? And is this approach informative or only fun?
As we mentioned before, all of this is recent. This is in part because these methods only became practical recently. Modern AI systems are huge, often involving billions of parameters and nontrivial optimization procedures. Much of the recent progress was enabled by transformer architectures and advances in large-scale computation [8]. The resulting models operate in extremely high-dimensional spaces, where geometry and probability interact in non-intuitive (but beautiful) ways. This relates to current research.
Now, from the perspective of automatic control, this development should not be interpreted as replacing modelling altogether. Not quite.
When accurate physical models exist, they remain very valuable. Gravity does not need to be relearned from YouTube videos. Physics remains useful, of course.
Hence, one should view AI as complementary rather than universal. AI excels particularly in extracting implicit structure when explicit descriptions are difficult to formulate or otherwise hidden. Control theory, meanwhile, provides principled tools for dynamics, stability, robustness and decision making. Increasingly, we see the push towards a combination, or better yet, integration.
A self-driving car illustrates this well. Neural networks may identify cyclists and their most likely action from some camera images. We have no other tools to effectively do this. On the other hand, the vehicle still requires classical feedback control and state estimation to operate. LLMs will not directly control your motor.
So perhaps the story of AI is not that models disappeared (or will disappear), but rather that we discovered how to proceed when models are incomplete or unavailable. That observation is clearly important far beyond photography and is something we must embrace.
Now, as we plan our descend from the summit and scroll through our newly generated photo collection (or sadly, emails), it is worth remembering that the backbone of these seemingly magical tools is a mix of classical estimation, filtering and optimization—concepts deeply intertwined with automatic control, whether visible or not [9].
References
- https://link.springer.com/content/pdf/10.1007/978-1-4471-0853-5_1.pdf
- https://store.google.com/ideas/articles/what-is-an-ai-camera/ (We are not promoting Googleʼs technology in particular, the author happens to have a Pixel phone.)
- https://asmedigitalcollection.asme.org/fluidsengineering/article/82/1/35/397706/A-New-Approach-to-Linear-Filtering-and-Prediction The interested technical reader should note the beautiful duality with optimal control.
