Hazardous conditions, as we have seen, are defined by constraints. To stay out of hazardous conditions, we have the system maintain such safety constraints. In general, though, the environment often tries to tip the system into breaking these constraints, and it often does this in unpredictable ways. This means we cannot declare in advance a sequence of steps the system should follow that will always maintain constraints.
Instead, maintaining constraints is a dynamic control problem. There are multiple controllers interacting with the system to try to keep it out of hazardous conditions. They observe feedback, i.e. information on where the system is now; they execute mental models, i.e. run simulations of where the system is going in the future; and then they issue control actions, i.e. try to adjust the system to maintain constraints based on their predictions.
Whenever a system enters a hazardous condition, it is because there were problems with the control structure, specifically one of the three components listed above:
- Feedback to controllers can be insufficient, which means the controllers do not understand what is going on with the system at some specific moment.
- Mental models can be insufficient, which means the controllers understand what’s going on with the system, but they are unable to predict something that will happen in the future.
- Control actions can be insufficient, which means the controllers know what they need to do to the system to maintain constraints, but it does not have an effect of the desired strength.3 This could be because the effect is too weak – or too strong!
We can also see combinations of these problems. When all three of them are problematic, we might actually be looking at an entire controller missing that should be present.
Controllers exist on all levels. For aircraft maintaining fuel constraints, controllers include the fadec inside the jet engines, the flight management computer, pilots, ground crew, dispatchers at the airline, training programmes for pilots, air traffic controllers, as well as national and international regulatory boards.4 For my child among rocks, controllers include their balance, their strength, their extremely limited sense of self-preservation, my instruction, my supervision, the places I decide to take us, etc.
Low-level controllers are often automated, in hardware or software. High-level controllers are often social, cultural, and legal in nature.