Who Pulls the Trigger? Autonomous Weapons, Meaningful Human Control, and the Search for Accountability under International Humanitarian Law

Who Pulls the Trigger?

Autonomous Weapons, Meaningful Human Control, and the Search for Accountability under International Humanitarian Law

AUTHOR’S NAME: ISHITA BHUSHAN

Introduction

Warfare has always been ahead of the law, but this tendency to outpace may prove a critical weakness in the case of autonomous systems. The question of whether weaponry should be able to select and attack targets without human intervention, while hotly debated, is no longer theoretical. A switch in March 2020, when Turkey’s Kargu-2 loitering munition, in Libya, according to a UN Panel of Experts report, had been used to kill and injure several people without requiring data-link connection from the operator, making it capable of independently identifying and attacking targets, raises the possibility that the first autonomous killing may already have taken place.[1]

This article explores the implications of international humanitarian law (‘IHL’) for the development and use of autonomous weapons systems (‘AWS’). It identifies the emerging consensus in international law and policy discourse on the need for maintaining ‘meaningful human control’ over AWS, and critically appraises the law’s ability to assign responsibility for unlawful harm directed at civilians. After briefly discussing the evolution of relevant treaties – notably the Convention on Certain Conventional Weapons – the article focuses on two areas. Firstly, it reviews the key principles of IHL, including the distinction between civilian objects and military targets, the principle of proportionality, and precautionary measures, to determine how they might be applied to AWS. Secondly, it addresses the challenge of responsibility: if AWS cause unlawful harm, but no human agent had meaningful control over their targeting function at the point of attack, who is responsible for the direct and indirect consequences of such harm? In the final part, the article provides a critical assessment of the law’s response to the AWS challenge, and whether a new treaty on autonomous weapons systems would be more appropriate than trying to interpret IHL in a way that fits with existing principles.

Background and Legal Framework

There is no universal definition or treaty language for what constitutes an autonomous weapon system, because the lack of one has proven to be one of the most intractable problems in attempting to regulate them. The ICRC uses the following functional definition: An AWS is “a weapon that – after initial human input – selects and applies force against targets by way of sensors and a generalized target description, without further direct human input.”[2] In other words, the difference between an autonomous and a non-autonomous weapon lies not in the ability to navigate or even to fly without a human at the controls – since many aircraft have been able to do that for decades – but rather in the “autonomy in the critical functions of targeting and firing.”

The human rights organization Human Rights Watch has a similar definition, which they developed for their seminal 2012 report on AWS, which is generally considered as the catalyst for the discussion on autonomous weapons systems that has since taken off, becoming part of mainstream discourse on disarmament and human rights.[3]

The initial assumption on which my own discussion of the law is based rests on the position that international humanitarian law (IHL) governs AWS in the same way as any other instrument or means of warfare; no one supposes that a projectile becomes entirely governed by a different law merely because its trigger is pulled by software rather than a soldier. The Geneva Conventions of 1949 and Additional Protocol I of 1977 provide the basic framework for such a discussion. Article 48 of Additional Protocol I enshrines the principle of distinction, requiring that parties to an armed conflict distinguish at all times between civilian objects and combatants and, in their conduct of operations, attack only military objectives.[4]Additional Protocol I, Articles 51(5)(b) and 57(2)(a)(iii), address the principles of proportionality and precaution, which require that attacks not be launched if they anticipate civilian casualties to be excessive in relation to the concrete and direct military advantage anticipated and that those planning and launching attacks take all feasible measures to ensure that targets are, indeed, military objectives and to limit collateral damage[5]. None of the IHL treaties took into consideration a self-directing weapon, and that is the problem in need of examination that will proceed.

A second, less obvious, obligation derives from Additional Protocol I, Article 36, which requires each State party to review anew all weapons, means or methods of warfare to determine whether they may be prohibited, in whole or in part, under international law. [6]Although Article 36 reviews are intended to serve as the first line of defence against unlawful weapons becoming fielded, these reviews are conducted in great secrecy by States, and there is no international oversight. The Martens Clause provides the necessary backing for prohibitions not covered by treaties: the International Court of Justice explicitly stated in its nuclear weapon’s advisory opinion that even if a weapon is not proscribed by treaty, its use may still violate the principles of humanity and the dictates of public conscience, as required by customary international humanitarian law[7]. The Martens Clause routinely serves as the fallback position for States, including in the context of AWS, in situations where treaties fail to explicitly ban a certain type of weapon or method of warfare. Target selection functions are not mentioned anywhere in the treaties, and so States routinely turn to the Martens Clause to justify bans of AWS capabilities that have not been formally proscribed by treaties.

The multilateral process has so far taken place mainly within the framework of the Convention on Certain Conventional Weapons (‘CCW’), through which various protocols have been adopted, and, since 2014, through informal and formal expert discussions on lethal autonomous weapons systems (‘LAWS’). The catalyst for this process was the 2013 report by the UN Special Rapporteur on extrajudicial, summary or arbitrary executions, Christof Heyns, calling for a moratorium on the development and deployment of fully autonomous weapons until international humanitarian law could be adapted to regulate them[8], and the work of the NGO Article 36, which in the same year proposed the concept of ‘meaningful human control’ that has since framed the debate.[9]

The emergence of ‘Meaningful Human Control’

It is, according to the author of the work, the closest common denominator on which it is possible to rely today among the international community regarding AWS. This approach does not require elaborating on what the term autonomy means; instead, it focuses on the question of whether a human is sufficiently involved in the decision-making process for the weapon to employ force, so they can be held accountable for it. While the idea itself is almost universally appealing, it has also been critiqued as being too vague and difficult to implement. Namely, Rebecca Crootof argued that the phrase does not provide a “meaningful floor” and serves as nothing more than a “catch-all” that all States would be able to agree to but interpret differently.[10] The U.S. took the position within the CCW that focusing on the principle of control was distracting and ineffective, arguing that it was more important to ensure that humans are appropriately involved (“meaningful human judgement”) in a weapon’s lifecycle, including its development, testing, and eventual use.

The ICRC together with Stockholm International Peace Research Institute conducted an in-depth study in 2020 that involved identifying particular design and operational features of human control over AWS, including predictability and reliability of the system behaviour, comprehensibility of the behaviour to the human operator and predictability in its deactivation within the time-scales of the engagement[11]. The ICRC itself contributed to the debate with the May 2021 position paper that called upon states to develop new binding agreements banning AWS “with unpredictable effects or designed or used to directly attack the civilian population” while imposing limitations on all other AWS in terms of specific targets, time and area of operation, and specific circumstances of their use. [12]Such a differentiated approach, which was later widely endorsed by CCW, is now regarded as a preferred option within the treaty-based framework for regulating AWS.

It is important to note that there are few states, if any, that would propose giving machines absolute and unrestrained power over deciding whose lives to value over whose. In fact, the disagreement is precisely over the threshold beyond which machines, when used in killings, must be under the direct control of humans. The question is whether such threshold should be established by an explicit treaty (as the UK position in the CDLP, that insists upon weapons being operated by humans at all times, but does not propose adding anything to the existing treaties) or left to the individual countries’ doctrines (as stated in the Franco-German position in the CCW, that only prohibits autonomous weapons that completely dispense with a responsible command).

Targeting Rules and the Limits of Autonomy

Whatever one chooses to label the minimum necessary level of human involvement, it must be sufficient to meet the substantive requirements of targeting doctrine as discussed above, which an autonomous system would have trouble doing, as each of the three principles provides unique difficulties.

Discrimination requires making a qualitative assessment of the target, such as determining whether a given target is a combatant or civilian or whether a combatant has surrendered and is therefore hors de combat, which is not always straightforward. Although machine-learning algorithms can be trained to make accurate classifications on well-defined data sets, hostilities in urban areas and other settings deliberately chosen by states for their tactical value are likely to include civilians, the use of decoys, and other factors that complicate discrimination. Proportionality is even more challenging, as it requires comparing the value of civilian lives lost to the military benefit to be gained and therefore requires value judgments that are beyond the capacity of existing systems and that few scholars believe can ever be encoded in software. Finally, although a human may select the target for an autonomous attack, he or she cannot take responsibility for the attack itself if there is insufficient time for reconsideration, as is likely to be the case with unmanned and remotely controlled weapons systems that do not require constant communication with their human controllers, such as the weapons used in the recent strikes on Libya.

None of this means that existing IHL is entirely inadequate to regulate AWS. It does mean however, that the factors determining legality are to a large extent decided long before the system is deployed: during its design, testing, and particularly the definition of its target profiles, constraints, and permitted operating environments. This is why the Article 36 weapons review obligation is the focal point of contention in this debate – a rigorous review process is the one specific mechanism by which States can ensure legality across the potential spectrum of targets, engagement scenarios, and operational environments for any autonomous weapon system they develop.

The Accountability Gap: Individual, Command and State Responsibility

Suppose that an independent entity was able to strike unlawful targets. Who, if anyone, would be responsible for such crimes, and how such responsibility could be established? This is the question that has attracted the most attention from international law scholars, as up to this point, all legal frameworks have been designed with the assumption that only humans are capable of forming intent or a lack thereof.

International criminal law provides with three ways of establishing individual criminal responsibility. Direct perpetration is the main mode of responsibility according to Article 25(3) of the Rome Statute[13]. It involves the responsibility of the accused for each crime committed through their direct actions, and the liability is based on the perpetrator’s mens rea and knowledge about the crime’s illegality at the time of its commission. When it comes to the unlawful use of force by automated systems, the direct perpetration of unlawful attack is possible if the operator initiated the attack when the probability of an unlawful strike was not denied by all reasonable people. In turn, the situation when the system’s behaviour is unpredictable, for instance, due to the use of machine learning mechanisms in robots operating in dynamic environments, may undermine the operator’s liability due to the impossibility to prove his mens rea with regard to the specific criminal outcome.

The second possibility is command responsibility, established by article 28 of the Rome statute[14], according to which a commander may be responsible for crimes committed or attempted by forces placed under one’s command when he or she knew or should have known that they had committed or were about to commit them and failed to take action to prevent these acts. This doctrine, developed by the ICTY in Čelebići case[15], was applied (with little success) by the ICC in the Bemba case, which was later overturned on appeal for reasons unrelated to this particular issue[16]. The application of command responsibility as a liability doctrine for autonomous “subordinates” is far from being logical, as a machine is not a person and therefore cannot give the superior a sense of responsibility in the way outlined by the statute and the case law. However, some scholars have argued that an omission-based variant of command responsibility may still be applicable, as a commander who places such systems under direct control and fails to exercise the level of monitoring that could have prevented the attacks would not have met his or her duty of care as stipulated by article 28.

The third option, which is less problematic in terms of autonomy, is the responsibility of the state. According to the Articles on the Responsibility of States for International Wrongful Acts of the International Law Commission, an internationally wrongful act is an act of a state which breaches an international obligation, irrespective of whether any state official was at fault.[17] Thus, if a state uses an autonomous weapon system, which commits an unlawful attack, this state will be internationally responsible for such an attack in breach of IHL, irrespective of whether any military personnel can be prosecuted. In other words, state responsibility provides a basis for reparations in all situations in which a state uses AWS in violation of IHL, even if no state official is directly at fault. The option of state responsibility is a relatively weak form of accountability for AWS misuse because state responsibility cannot be punishing enough, and in addition, it does not respond directly to the crime by the specific perpetrator. Thus, if the crime constitutes a war crime, state responsibility would not be sufficient in terms of individual criminal responsibility for such an offense.

Put together, these three cases create what can be called in the literature, the accountability or responsibility gap; situations where a State has acted in violation of IHL, but no individual can be convicted with a probability sufficient to meet the standards of criminal law, as the proximate decision causing the violation was made by a system the specific input of which in the decision-making process no individual could have predicted. This is not always the case, as AWSs that are not entirely autonomous or do not operate in unpredictable manners will often still have a responsible agent who can be found to have exercised the requisite mens rea for a finding of liability. But in general, these problems are likely to occur more often for systems displaying greater autonomy, adaptability, and speed, and these qualities are what militaries are expected to install in future AWSs.

The Regulatory Response: The CCW, the Guiding Principles, and Beyond

Formal international consultation on LAWS within the CCW framework began in 2014 and led, in 2016, to the establishment of a permanent (but informal) Group of Governmental Experts (‘GGE’) on LAWS. In 2019, the GGE agreed on the eleven Guiding Principles which have since been formally adopted by the CCW Meeting of High Contracting Parties.[18] The Guiding Principles include recognition of the continuing applicability of Additional Protocol I to LAWS, reaffirmation of the central role of human agency in all decisions to kill and injure and the need to embed risk assessment and mitigation processes throughout the lifecycle of autonomous weapon systems. Nevertheless, the principles are essentially aspirational, and subsequent GGE meetings have failed to agree on further steps, due to a fundamental policy divide between States that want to ban or heavily regulate LAWS and others, including some states with the most developed military AI capabilities, which believe that existing IHL is entirely applicable to LAWS.

Civil society has been much more persistent than the diplomatic process. The Campaign to Stop Killer Robots, working with Human Rights Watch and the Harvard Law School International Human Rights Clinic, has long promoted a treaty banning the development and use of weapons systems that lack direct human control and requiring positive obligations to maintain direct human control over all other weapons.[19] This view was also echoed by the ICRC. The combined pressure has borne fruit in the diplomatic arena. On 22 December 2023, the UN General Assembly approved Resolution 78/241, which states that international law, including the UN Charter, IHL and international human rights law, applies to autonomous weapons systems and urges the Secretary-General to seek States’ views on the implications for international security of emerging technologies, in particular lethal autonomous weapon systems.[20] A similar resolution is being tabled almost a year later. [21]Both are designed to complement the work of the CCW, which reflects the lack of consensus on both the need and the solution among the states that serve on the commission.

National policy, however, has developed more or less independently of any treaty, for the time being. The best public example is the United States Department of Defence Directive 3000.09 on autonomous weapons systems, first issued in 2012 and updated in 2023[22]. It requires that autonomous and semi-autonomous weapon systems be designed to “provide appropriate levels of human judgment over the use of force” and to undergo “rigorous testing and senior leadership review” before deployment. However, limited its value may be in terms of transparency (the review process being at least theoretically confidential), DoD Directive 3000.09 is nevertheless an acknowledgment, by nations hostile to a binding international treaty, that unrestrained autonomy in weaponized artificial intelligence is not desirable. The discrepancy between the domestic policy and the international legal framework, at least as far as the law is concerned, is the real issue.

Critical Analysis and Evaluation

Three points emerge from the previous discussion. First, the thesis that existing IHL is “adequate to the task, without further ado,” is difficult to defend once the accountability gap is properly understood. While IHL prescribes what actors may and may not do, it says relatively little about who is responsible for what, and in particular, who is responsible for actions that no civilian could have predicted in a semi-autonomous or autonomous system. State responsibility comes into play here, but states are only one type of actor, and often the least immediately tangible one in international law – and it is individual responsibility, not just official responsibility, for unlawful acts, particularly for those unlawful acts that would be considered otherwise crimes of war if they were committed by humans.

Second, ‘meaningful human control’ seems to be doing more work politically than legally at this stage. As such, it has proven a fruitful diplomatic concept in that it allows States with divergent military capabilities to agree on something each believes it already possesses. This is not necessarily a weakness, since vagueness can be a virtue in diplomacy, but it does mean that the concept cannot deliver on its potential as a stabilizing legal framework without further substantial conceptual development of the kind currently under way at SIPRI, the ICRC and elsewhere. A treaty requiring States to exercise ‘meaningful human control’ without detailed elaboration on the predictability of control measures, their timing, or the adequacy of testing regimes would be as ineffective as many of its critics believe.

Thirdly, the review process required by Article 36, which is probably the most underutilized of the existing legal norms, seems to have greatest potential for filling the current capability gaps in the immediate future. The process mandated by Article 36 encompasses precisely the kind of precautions that many of the proposed new legal rules would have us adopt, and the most important reform in this domain in the near term is likely to consist in a norm requiring that Article 36 reviews of autonomous weapons systems take place in a manner that applies a common methodology (however detailed) and is subject to external audit. This would close many of the loopholes while still leaving room for subsequent negotiations about more controversial policy questions.

None of this is meant to dismiss the case for a new treaty. The two-tier model preferred by the ICRC and the Campaign to Stop Killer Robots is undoubtedly desirable as a way to draw a bright line, a prohibition, in a domain where bright lines are otherwise hard to come by. Whether states can be relied upon, in the context of the consensus-driven CCW process, to agree upon such a bright line is a separate question, but on the basis of a decade of GGE meetings, such a consensus seems unlikely to come soon.

Conclusion

It is wrong to assume that autonomous weapons systems are completely separate from international humanitarian law; on the contrary, the relevant articles of the Geneva Conventions, Additional Treaty I, practice, and the Martens Clause continue to apply. The real problem is not with what law applies to autonomous weapons systems but rather with how international humanitarian law can best hold sway where there is no control over individual action. In other words, how to ensure accountability if the direct cause of death and injury is an object, a machine, whose actions were unpredictable? The concept of meaningful human control has proven to be a good starting point for addressing these issues. It will serve as an effective basis for action only if states adopt specific measures that make human accountability concrete. And to do that, they need to focus on predictability, understandability, and timely human intervention. These principles are not set in stone but should be prioritized while waiting for the regulation of autonomous weapons systems. A possible way out of this dilemma is to be found not in new prohibitions but in ensuring that existing humanitarian law, notably the review of new weapons under Article 36, is applied rigorously.

 

Frequently Asked Questions (FAQs)

1. Does international humanitarian law (IHL) apply to autonomous weapons systems at all?

IHL applies to all humanitarian issues related to the use of autonomous weapon systems. There is no differentiation made between autonomous and human guided weapons in the Geneva Conventions, Additional Protocol I and customary IHL. Where the specific rules do not apply, common article 1 of the Geneva Conventions, the Martens Clause and the Guiding Principles on the use of lethal force provide a general framework. The UN General Assembly and CCW have explicitly stated that the IHL treaties apply to autonomous weapons systems.

2. What do you mean by ‘meaningful human control’?

‘Meaningful human control’ is not defined explicitly in the Guiding Principles, but is covered by a set of requirements that most proposals put forward on this issue largely share. These require that the weapon’s actions be predictable and verifiable, that the actions of the weapon be understandable to the human using it, and that a human have the ability to stop the weapon from firing within the ‘decision cycle’ of the engagement. The ICRC/SIPRI framework provides a detailed breakdown of this concept.

3. Who can be prosecuted if an autonomous weapon commits a war crime?

It will depend on how far removed any individual human operator was from the decision to commit the unlawful act. An operator who knowingly launches a system foreordained to commit certain unlawful acts is a proximate causal agent and can be directly prosecuted. A commander who fails to maintain appropriate human control over a known autonomous system, however, can be prosecuted on command responsibility grounds. And if the particular system did not perform an unlawful act reasonably foreseeable to any human by virtue of its design, no individual may be directly prosecuted for it – and that is the accountability deficit addressed by this paper.

4. Can a State be held responsible if no individuals can be prosecuted?

Yes. Responsibility under International Law Commission Articles does not require specification of an individual wrongdoer or proof of mens rea; it only requires the conduct of the State to be unlawful. The State which operates an autonomous system committing unlawful attacks is responsible for those attacks and can be ordered to make reparation even when no soldier, commander, or civilian operator can be individually prosecuted.

5. Is there an international treaty banning autonomous weapons?

Not yet. The CCW Group of Governmental Experts has been discussing this matter for more than a decade, resulting in only eleven non-binding Guiding Principles in 2019, while the UN General Assembly adopted two further resolutions on the matter last year. Meanwhile, campaigners from the Campaign to Stop Killer Robots are pushing for a two-tier international treaty banning the most egregious autonomous weapons while regulating the rest. But so far, no treaties banning autonomous weapons systems have been concluded.

6. What is the Article 36 weapons review, and why is it relevant to the discussion?

Article 36 of Additional Protocol I requires States to review new weapons developments to determine whether their development or use would violate international humanitarian law. It focuses on the weapon’s impact on civilian populations, requiring states to assess whether it may violate the principles of distinction, proportionality, and precautions in attack. Since states are obligated to make that review and publish its results, few do it transparently or in detail, while no international institution is empowered to review those reviews. This is why many campaigners believe that the universal strengthening of state review practices, rather than waiting for a new treaty, is the most immediate step towards ensuring that autonomous weapons do not violate IHL.

References

1. Treaties and Instruments

• Geneva Conventions I–IV, concluded on 12 August 1949.
• Additional Protocol I to the Geneva Conventions (1977), articles 36, 48, 51(5)(b), 57(2)(a)(iii).
• Convention on Certain Conventional Weapons (1980), 1342 UNTS 137.
• Rome Statute of the International Criminal Court (1998), articles 8(2)(b), 25(3), 28.
• International Law Commission, Articles on Responsibility of States for Internationally Wrongful Acts (2001), appended to UNGA Res 56/83.
• UNGA Res 78/241, ‘Lethal Autonomous Weapons Systems’, UN Doc A/RES/78/241 (22 December 2023).
• UNGA Res 79/62 (2 December 2024).

2. Cases

• Legality of the Threat or Use of Nuclear Weapons (Advisory Opinion) [1996] ICJ Rep 226.
• Prosecutor v Delalić and Others (Čelebići Case) (Trial Judgment) ICTY-96-21-T (16 November 1998).
• Prosecutor v Jean-Pierre Bemba Gombo (Judgment Pursuant to Article 74) ICC-01/05-01/08 (21 March 2016).

3. Reports, Policy Documents and Commentary

• Christof Heyns, Report of the Special Rapporteur on Extrajudicial, Summary or Arbitrary Executions, UN Doc A/HRC/23/47 (9 April 2013).
• Article 36, ‘Killing by Machine: Key issues for understanding meaningful human control’ (April 2013).
• Human Rights Watch, ‘Losing Humanity: The case against killer robots’, (19 November 2012).
• Human Rights Watch and Harvard Law School International Human Rights Clinic, ‘Heed the call: A moral and legal imperative to ban killer robots’, (21 August 2018).
• ICRC and SIPRI, ‘Limits on autonomy in weapon systems: identifying practical elements of human control’, (June 2020).
• ICRC, ‘ICRC Position on autonomous weapon systems: position and background paper’, (12 May 2021).
• CCW Group of Governmental Experts, Report of the 2019 Session, UN Doc CCW/GGE.1/2019/3, Annex IV (Guiding Principles).
• CCW Group of Governmental Experts, Report of the 2021 Session, UN Doc CCW/GGE.1/2021/CRP.1.
• Letter dated 8 March 2021 from the Panel of Experts on Libya addressed to the President of the Security Council, UN Doc S/2021/229.
• US Department of Defence, Directive 3000.09, ‘Autonomy in weapon systems’ (21 November 2012, updated 25 January 2023).

Rebecca Crootof, ‘A meaningful floor for meaningful human control’ (2016) 30 Temple International & Comparative Law Journal 53.

[1] Letter dated 8 March 2021 from the Panel of Experts on Libya addressed to the President of the Security Council, UN Doc S/2021/229.

[2] ICRC and Stockholm International Peace Research Institute, ‘Limits on autonomy in weapon systems: identifying practical elements of human control’ (June 2020).

[3] Human Rights Watch, ‘Losing Humanity: The case against killer robots’ (19 November 2012).

[4] Additional Protocol I to the Geneva Conventions (1977), art 48.

[5] Additional Protocol I to the Geneva Conventions (1977), arts 51(5)(b), 57(2)(a)(iii).

[6] Additional Protocol I to the Geneva Conventions (1977), art 36.

[7] Legality of the Threat or Use of Nuclear Weapons (Advisory Opinion) [1996] ICJ Rep 226

[8] Christof Heyns, Report of the Special Rapporteur on Extrajudicial, Summary or Arbitrary Executions, UN Doc A/HRC/23/47 (9 April 2013).

[9] Article 36, ‘Killing by Machine: Key issues for understanding meaningful human control’ (April 2013).

[10] Rebecca Crootof, ‘A meaningful floor for meaningful human control’ (2016) 30 Temple International & Comparative Law Journal 53.

[11] ICRC and Stockholm International Peace Research Institute, ‘Limits on autonomy in weapon systems: identifying practical elements of human control’ (June 2020).

[12] ICRC, ‘ICRC Position on autonomous weapon systems: position and background paper’ (12 May 2021).

[13] Rome Statute of the International Criminal Court (1998), art 25(3).

[14] Rome Statute of the International Criminal Court (1998), art 28.

[15] Prosecutor v Delalić and Others (Čelebići Case) (Trial Judgment) ICTY-96-21-T (16 November 1998).

[16] Prosecutor v Jean-Pierre Bemba Gombo (Judgment Pursuant to Article 74) ICC-01/05-01/08 (21 March 2016).

[17] International Law Commission, Articles on Responsibility of States for Internationally Wrongful Acts (2001), appended to UNGA Res 56/83.

[18] CCW Group of Governmental Experts, Report of the 2019 Session, UN Doc CCW/GGE.1/2019/3, Annex IV (Guiding Principles).

[19] Human Rights Watch and Harvard Law School International Human Rights Clinic, ‘Heed the call: A moral and legal imperative to ban killer robots’ (21 August 2018).

[20] UNGA Res 78/241, ‘Lethal Autonomous Weapons Systems’, UN Doc A/RES/78/241 (22 December 2023).

[21] UNGA Res 79/62 (2 December 2024).

[22] US Department of Defence, Directive 3000.09, ‘Autonomy in weapon systems’ (21 November 2012, updated 25 January 2023).