Rice cookers are fascinating machines. I’ve owned one for years, as rice is a significant part of my regular diet, and it completely removes the stress of preparing rice. They also operate on a simple principle that can help us operate cloud infrastructure – the control loop.

Control loops are pretty straightforward things. In its simplest format, you have an input, a sensor, a process variable and a set-point. The purpose of the loop is to measure the process variable via the sensor, applying or removing input until the desired set-point is reached.

Control loops can broadly be split into two categories: open and closed. An open loop has no feedback mechanism – the input does not change depending on the output. In our case, if the set-point was reached but we allowed more input to be applied, then it would be an open loop. Conversely, a closed loop has a feedback mechanism, so the input is controlled depending on the process variable reaching the set-point or not.

You really, really want closed loops. Open loops are a common cause of “runaway” systems. I’ve seen open loops be responsible for a number of failures in complex systems, and sometimes these loops can be quite innocuous. It is also very easy to create open loops.

An open loop example

For a classic open loop, consider how rice is made over a stove (or a 竃 kamado). Water and rice is added to the pot, and heat is applied (via gas, electric heating rings or induction).

Imagine that you are an inattentive cook, and you walk away from the pot and go and do something else more engrossing. You return some time later to find that the water has completely boiled off, and your rice burned beyond recognition.

If we say that our input is the heat applied to the pot, the process variable is the temperature of the pot itself, and the set-point is the boiling point of water, we can model it like so:

#!/usr/bin/env python

import time

# this is very simplified for the sake of illustration
water_volume = 1000 # 1000g of water
water_boiling_point = 100 # 100ºC
pot_temperature = 22 # about room temperature

while True:
    if (pot_temperature == water_boiling_point) and (water_volume > 0):
        water_volume -= 25
    else:
        pot_temperature += 2

    print(f"{time.clock_gettime(time.CLOCK_MONOTONIC)},{pot_temperature},{water_volume}")
    time.sleep(0.5)

This outputs a set of numbers, when graphed, gives the following:

As can be seen, we reach a steady state of pot temperature while there is water remaining in the pot and we have reached boiling point. Once the water has boiled off, the pot temperature continues to rise.

Shifting towards a closed loop

When cooking rice over the stove, one of three things happens: you have too much water for the amount of rice (resulting in over-cooking the rice and reaching the texture of congee), you have too little water (resulting in uncooked rice that burns against the pan) or you nail it and you have perfectly cooked rice.

Wouldn’t it be easier if instead of having to manage the input, we could just walk away and have pretty great rice with minimal effort?

And so, the first automated commercial rice cooker was created in 1956 by Toshiba, using a mechanical bi-metal switch that above a certain temperature, would deform and switch off the heating element. We can model that like this:

#!/usr/bin/env python

import time

# this is very simplified for the sake of illustration
water_volume = 1000 # 1000g of water
water_boiling_point = 100 # 100ºC
pot_temperature = 22 # about room temperature
set_point = 110
element_on = 100

print(f"time,pot_temperature,water_volume,element_on")
while True:
    if element_on > 0:
        if pot_temperature > set_point:
            element_on = 0
        if (pot_temperature == water_boiling_point) and (water_volume > 0):
            water_volume -= 25
        else:
            pot_temperature += 2
    else:
        pot_temperature -= 2

    print(f"{time.clock_gettime(time.CLOCK_MONOTONIC)},{pot_temperature},{water_volume},{element_on}")
    time.sleep(0.5)

Plotted, this results in a graph like the following:

Here, we can see clearly at after the pot reaches the set-point of 110ºC, the switch is disengaged and the heating element is turned off, resulting in the steady cooling of the pot.

As the input is now controlled by the output, we have a closed loop – a much more desirable state of affairs.

Maintaining a steady state

This is a significant improvement for rice-lovers everywhere, but what if we want to keep it warm? Food safety regulations suggest that we keep food at 65ºC or above and fortunately for us, rice generally needs to be cooked at 70ºC or more to make progress in any reasonable amount of time.

This is a much taller order than having a bi-metal switch deform to disengage the heating element. Commerical rice cookers typically advertise fuzzy logic features for this, which take advantage of microcomputers to take into account variables like “almost” the right water-to-rice ratio, ambient temperature fluctuations, hotspots in the cooking pot and so on. They also allow for keeping the rice warm at a food safe temperature.

In order to model this, we require a bit more sophistication. To do this, we’re going to turn to a three-term controller, also known as a PID loop.

The PID Loop

I love PID loops. The individual compoenents, or all together, are extremely useful concepts when managing cloud infrastructure, as well as cooking rice in an automated fashion.

So how can we model a fancy rice cooker, like the Zojirushi ones?

In the simplified model, we still have our input (heat), sensor (thermocouple) and process variable (pot temperature). However, we need to approach out set-point a little differently. In the first case, our set-point needs to be 100ºC to cook the rice. But afterwards, we need to adjust our set-point down to 65ºC.

For the purposes of this model, I’m going to use the simple-pid package. This is very straightforward implementation of a PID, and going deeply in to the minutiae of the implementation is out of scope for this post.

#!/usr/bin/env python

import time
from simple_pid import PID

# this is very simplified for the sake of illustration


water_boiling_point = 100
keep_warm_temp = 65
rice_cooking_time = 5


class RiceCooker:
    def __init__(self) -> None:
        self.water_mass = 1000
        self.pot_temperature = 22

    def update(self, element_power) -> int:
        if element_power > 0:
            self.pot_temperature += 1 * element_power
        else:
            self.pot_temperature -= 2

        if (self.pot_temperature >= water_boiling_point) and (self.water_mass > 0):
            self.water_mass -= 25

        return self.pot_temperature


def new_pid(setpoint) -> PID:
    pid = PID(0.1, 0.001, 0.01, setpoint=setpoint)
    pid.output_limits = (0, water_boiling_point)
    pid.sample_time = 0.1
    return pid


def execute(pid, cooker, pv) -> tuple:
    power = pid(pv)
    pv = cooker.update(power)

    return power, pv


cooker = RiceCooker()
boiling_pid = new_pid(setpoint=water_boiling_point)
warm_pid = new_pid(setpoint=keep_warm_temp)

pot_temperature = cooker.pot_temperature
above_boiling_duration = 0

print(f"time,pot_temperature,water_volume,element_power")
while True:
    while 0 <= pot_temperature <= (water_boiling_point * 1.10):
        current_time = time.time()

        if (water_boiling_point * 0.9) <= pot_temperature > water_boiling_point:
            above_boiling_duration += current_time - last_time

        if above_boiling_duration >= rice_cooking_time:
            break

        pid = boiling_pid
        power, pot_temperature = execute(pid, cooker, pot_temperature)
        print(f"{current_time},{pot_temperature},{cooker.water_mass},{power}")
        last_time = current_time
        time.sleep(pid.sample_time)

    pid = warm_pid
    power, pot_temperature = execute(pid, cooker, pot_temperature)
    print(f"{current_time},{pot_temperature},{cooker.water_mass},{power}")
    time.sleep(pid.sample_time)

Here, we use an inner loop for the boiling controller and another for the hot hold controller. Once the cooking process is started, the boiling controller applies power to the heating element proportionally to the set-point of 100ºC. Once that is reached, it applies power in proportion to what is needed to keep it at sufficient temperature for the cooking time.

Once the cooking time is exceeded, we switch to the hot hold controller. Here, we need to allow the pot to cool until we can apply power to the heating element in proportion with the new set-point, that of 65ºC.

Deriving the proportional, integral and derivative factors requires tuning. In this case, the Ziegler–Nichols heuristic was used.

Plotted, this results in a graph like the following:

Here, we can see the heating process on the left hand size to get up to cooking temperature, followed by a steady state while cooking at the first set-point, then the period of cooling to the second set-point, where it is held.

Wait, I thought this was about cloud infrastructure?

Operating cloud infrastructure effectively is a lot like building a good automated rice cooker:

1. You want to avoid open loops as much as possible – input should always be constrained.
2. Be aware of your inputs and outputs, especially if they indicate open loops.
3. Using sophiscated controllers, like PID loops, independently or as part of a larger system gives you a large amount of control.

As an example, consider a Postgres database. Postgres uses write-ahead logging as part of its standard operation, and typically you’ll configure Postgres to archive the WAL segments to remote storage after they have been written. However, should there be a failure in your archival process and the WAL segments are not moved to remote storage, you have an open loop where the disk will continue to fill up as more segments are written.

Similarily, should the rate of WAL segment creation be greater than that of the rate of archival, you have an open loop and the disk will fill up. This is where a proportional controller can be useful – I have previously implemented one controlling the maximum number of concurrent connections in order to reduce write volume.

As another example, consider maintaining a cache hit rate across multiple memcached servers. In order to increase the cache size, we add more servers. Again, we can model this with a control loop, with our input as the number of servers, our process variable is the cache hit rate, and our output is a moving average hit rate. We can use a PID controller to scale this on demand, but requires some careful tuning and treatment of time – do we want to consider “event time” (that is, each cache request advances a logical clock) or “wall time”?

Wrap-up

Control loops are a powerful thing to understand when operating cloud infrastructure, and come in a number of varieties. Controllers such a PID controllers can help express complex system dynamics, as well as provide composability. Indeed, many modern industrial control systems have a number of such controllers, and even simpler devices like a rice cooker take advantage of these control systems.

For further reading, I can heavily recommend Janert, P. K. (2013) Feedback Control for Computer Systems, which includes additional simulations and exploration of other PID tuning methods, such as AMIGO and Cohen-Coon.

Time to enjoy some freshly made rice, thanks to the efforts of Yoshitada Minami and Fumiko Minami back in 1956. My choice is the classic tamago kake gohan (“egg over rice”).