We must not eliminate the most important variable in today’s systems — the humans who design, maintain and manage them.
Straight out of defense labs, autonomous and semi-autonomous weapons are already in use, but there’s no overarching agreement among key stakeholders on how to control their implementation and diffusion. Unlike nuclear or biological weapons whose proliferation have been largely controlled, autonomous weapons pose some tricky problems.
The first is the absence of an international treaty. The second is the comparative ease by which autonomous weapons can be developed. Nuclear weapons are hard. The nine countries with nuclear weapons have achieved this with multi-decade projects backed substantially by state resources and administrative capacity.
Not so for autonomous weapons. The hardware is getting cheaper and cheaper. Drones in use by insurgent forces cost only a few hundred dollars. And the software engineering know-how (machine vision, autonomous navigation, collision detection) is widely understood and rapidly dispersed through legitimate channels (such as ArXiv and GitHub), and increasingly in commercial products. An autonomous weapon does not need to be a T-1000. It could be a $500 drone carrying a shaped charge. (Longish review of autonomous drone weapons currently in use by state military and non-state actors here.)
In sum: the knowledge to build these subsystems is widely available. And the tools to turn them into reality are cheap. This 8-minute video published by Future of Life Institute depicts one possible scenario, too realistic to be ignored. (An immediate threat is on the horizon, however, in the form of machine learning-enabled malicious software that learns normal behaviour patterns and uses them to get past security gates. WSJ paywall.)
Paul Scharre, a security analyst, warns that there’s not much time left for negotiating proper regulations:
Four years ago, the first diplomatic discussions on autonomous weapons seemed more promising, with a sense that countries were ahead of the curve. Today, even as the public grows increasingly aware of the issues, and as self-driving cars pop up frequently in the daily news, energy for a ban seems to be waning. Notably, one recent open letter by AI and robotics company founders did not call for a ban. Rather, it simply asked the UN to “protect us from all these dangers.”
AI researcher Subbarao Kambhampati questions, too, whether any ban is worth supporting because a ban is likely to be ineffective and “a pyrrhic victory” for the proponents of peace.
I do think there are two things to bear in mind. First, by calling these powerful machines “killer robots” (as the Campaign to Stop Killer Robots does) we eliminate the most important variable in today’s systems — the humans who design, maintain and manage them. As this excellent, comprehensive study by Maaike Verbruggen and Vincent Boulanin has shown, the largest militaries do not immediately envision fully-autonomous systems taking over the battlefield:
The focus on full autonomous systems is somewhat problematic as it does not reflect the reality of how the military is envisioning the future of autonomy in weapon systems, nor does it allow for tackling the spectrum of challenges raised by the progress of autonomy in weapon systems in the short term. Autonomy is bound to transform the way humans interact with weapon systems and make decisions on the battlefield, but will not eliminate their role. […] What control should humans maintain over the weapon systems they use and what can be done to ensure that such control remains adequate or meaningful as weapon systems’ capabilities become increasingly complex and autonomous?
The failure of system designers to account for human error has been particularly exposed in air travel, another industry with high stakes. In 2009 Air France 447 disappeared into the ocean as its pilots were befuddled after the autopilot transferred controls to humans. (See also this story of an Airbus whose control systems went rogue resulting in several passenger injuries.)
The second is that, yes, the technologies & raw materials are widely dispersed. Yellow cake and centrifuges, it isn’t. Cheap drones and free software, it is. Rather than throwing our hands up in despair, we should spend some time seriously raising awareness of these issues is, in turn, might generate creative solutions to the problem. Any workable solution is likely to multi-factor, ranging from technical, to ethical, regulatory, administrative and even social.
Worth looking into: the co-founding organisation of Campaign to Stop Killer Robots, PAX, put out two reports in the wake of the latest UN discussion: an overview of trends and weapons under development and a report on the positions of European states.
If you enjoyed reading this, you’ll appreciate my weekly newsletter. Some people say it replaced Sunday newspapers for them.