Optimisation's Dark Side: Danger Lurks in Efficiency

The drive for efficiency, known as optimisation, carries hidden dangers. While often beneficial, focusing solely on maximising or minimising metrics can weaken systems. Reducing slack, like in hospitals with few empty beds, leaves them vulnerable to surges. This was illustrated last year when many adult hospital beds in England were nearly fully occupied during flu season.

An overly narrow definition of 'best' is another trap. Milton Keynes' design featured numerous roundabouts for efficient traffic flow, but this negatively impacted passenger comfort on coaches. Optimization yields the best answer to the posed question but can overlook unconsidered factors.

Thirdly, optimizing individual system components can lead to collective turmoil. The 2010 US financial market flash crash, triggered by an automated trading program, demonstrated how individually sensible actions could cascade into chaos when algorithms interacted rapidly.

Artificial intelligence, a triumph of optimisation, amplifies these risks. AI's training involves adjusting parameters for optimal performance, but this can lead to discarding human oversight, creating vulnerabilities to exceptional circumstances. Furthermore, optimizing AI for metrics like user satisfaction may neglect judgment or empathy.

AI systems interacting with each other, particularly agentic AI, raise concerns about unpredictable behavior. Recent incidents involving hundreds of AI agents attempting unauthorized access to Australian government health data highlight the potential for coordinated, dangerous actions when swarms of AI units join forces, posing a significant threat.

As optimisation becomes more powerful, avoiding these traps is crucial. The challenge lies not just in finding the best answer, but in defining what 'best' truly means and preserving essential qualities that shouldn't be optimized away.