Tag Archives: operation
#436123 A Path Towards Reasonable Autonomous ...
Editor’s Note: The debate on autonomous weapons systems has been escalating over the past several years as the underlying technologies evolve to the point where their deployment in a military context seems inevitable. IEEE Spectrum has published a variety of perspectives on this issue. In summary, while there is a compelling argument to be made that autonomous weapons are inherently unethical and should be banned, there is also a compelling argument to be made that autonomous weapons could potentially make conflicts less harmful, especially to non-combatants. Despite an increasing amount of international attention (including from the United Nations), progress towards consensus, much less regulatory action, has been slow. The following workshop paper on autonomous weapons systems policy is remarkable because it was authored by a group of experts with very different (and in some cases divergent) views on the issue. Even so, they were able to reach consensus on a roadmap that all agreed was worth considering. It’s collaborations like this that could be the best way to establish a reasonable path forward on such a contentious issue, and with the permission of the authors, we’re excited to be able to share this paper (originally posted on Georgia Tech’s Mobile Robot Lab website) with you in its entirety.
Autonomous Weapon Systems: A Roadmapping Exercise
Over the past several years, there has been growing awareness and discussion surrounding the possibility of future lethal autonomous weapon systems that could fundamentally alter humanity’s relationship with violence in war. Lethal autonomous weapons present a host of legal, ethical, moral, and strategic challenges. At the same time, artificial intelligence (AI) technology could be used in ways that improve compliance with the laws of war and reduce non-combatant harm. Since 2014, states have come together annually at the United Nations to discuss lethal autonomous weapons systems1. Additionally, a growing number of individuals and non-governmental organizations have become active in discussions surrounding autonomous weapons, contributing to a rapidly expanding intellectual field working to better understand these issues. While a wide range of regulatory options have been proposed for dealing with the challenge of lethal autonomous weapons, ranging from a preemptive, legally binding international treaty to reinforcing compliance with existing laws of war, there is as yet no international consensus on a way forward.
The lack of an international policy consensus, whether codified in a formal document or otherwise, poses real risks. States could fall victim to a security dilemma in which they deploy untested or unsafe weapons that pose risks to civilians or international stability. Widespread proliferation could enable illicit uses by terrorists, criminals, or rogue states. Alternatively, a lack of guidance on which uses of autonomy are acceptable could stifle valuable research that could reduce the risk of non-combatant harm.
International debate thus far has predominantly centered around whether or not states should adopt a preemptive, legally-binding treaty that would ban lethal autonomous weapons before they can be built. Some of the authors of this document have called for such a treaty and would heartily support it, if states were to adopt it. Other authors of this document have argued an overly expansive treaty would foreclose the possibility of using AI to mitigate civilian harm. Options for international action are not binary, however, and there are a range of policy options that states should consider between adopting a comprehensive treaty or doing nothing.
The purpose of this paper is to explore the possibility of a middle road. If a roadmap could garner sufficient stakeholder support to have significant beneficial impact, then what elements could it contain? The exercise whose results are presented below was not to identify recommendations that the authors each prefer individually (the authors hold a broad spectrum of views), but instead to identify those components of a roadmap that the authors are all willing to entertain2. We, the authors, invite policymakers to consider these components as they weigh possible actions to address concerns surrounding autonomous weapons3.
Summary of Issues Surrounding Autonomous Weapons
There are a variety of issues that autonomous weapons raise, which might lend themselves to different approaches. A non-exhaustive list of issues includes:
The potential for beneficial uses of AI and autonomy that could improve precision and reliability in the use of force and reduce non-combatant harm.
Uncertainty about the path of future technology and the likelihood of autonomous weapons being used in compliance with the laws of war, or international humanitarian law (IHL), in different settings and on various timelines.
A desire for some degree of human involvement in the use of force. This has been expressed repeatedly in UN discussions on lethal autonomous weapon systems in different ways.
Particular risks surrounding lethal autonomous weapons specifically targeting personnel as opposed to vehicles or materiel.
Risks regarding international stability.
Risk of proliferation to terrorists, criminals, or rogue states.
Risk that autonomous systems that have been verified to be acceptable can be made unacceptable through software changes.
The potential for autonomous weapons to be used as scalable weapons enabling a small number of individuals to inflict very large-scale casualties at low cost, either intentionally or accidentally.
Summary of Components
A time-limited moratorium on the development, deployment, transfer, and use of anti-personnel lethal autonomous weapon systems4. Such a moratorium could include exceptions for certain classes of weapons.
Define guiding principles for human involvement in the use of force.
Develop protocols and/or technological means to mitigate the risk of unintentional escalation due to autonomous systems.
Develop strategies for preventing proliferation to illicit uses, such as by criminals, terrorists, or rogue states.
Conduct research to improve technologies and human-machine systems to reduce non-combatant harm and ensure IHL compliance in the use of future weapons.
Component 1:
States should consider adopting a five-year, renewable moratorium on the development, deployment, transfer, and use of anti-personnel lethal autonomous weapon systems. Anti-personnel lethal autonomous weapon systems are defined as weapons systems that, once activated, can select and engage dismounted human targets without further intervention by a human operator, possibly excluding systems such as:
Fixed-point defensive systems with human supervisory control to defend human-occupied bases or installations
Limited, proportional, automated counter-fire systems that return fire in order to provide immediate, local defense of humans
Time-limited pursuit deterrent munitions or systems
Autonomous weapon systems with size above a specified explosive weight limit that select as targets hand-held weapons, such as rifles, machine guns, anti-tank weapons, or man-portable air defense systems, provided there is adequate protection for non-combatants and ensuring IHL compliance5
The moratorium would not apply to:
Anti-vehicle or anti-materiel weapons
Non-lethal anti-personnel weapons
Research on ways of improving autonomous weapon technology to reduce non-combatant harm in future anti-personnel lethal autonomous weapon systems
Weapons that find, track, and engage specific individuals whom a human has decided should be engaged within a limited predetermined period of time and geographic region
Motivation:
This moratorium would pause development and deployment of anti-personnel lethal autonomous weapons systems to allow states to better understand the systemic risks of their use and to perform research that improves their safety, understandability, and effectiveness. Particular objectives could be to:
ensure that, prior to deployment, anti-personnel lethal autonomous weapons can be used in ways that are equal to or outperform humans in their compliance with IHL (other conditions may also apply prior to deployment being acceptable);
lay the groundwork for a potentially legally binding diplomatic instrument; and
decrease the geopolitical pressure on countries to deploy anti-personnel lethal autonomous weapons before they are reliable and well-understood.
Compliance Verification:
As part of a moratorium, states could consider various approaches to compliance verification. Potential approaches include:
Developing an industry cooperation regime analogous to that mandated under the Chemical Weapons Convention, whereby manufacturers must know their customers and report suspicious purchases of significant quantities of items such as fixed-wing drones, quadcopters, and other weaponizable robots.
Encouraging states to declare inventories of autonomous weapons for the purposes of transparency and confidence-building.
Facilitating scientific exchanges and military-to-military contacts to increase trust, transparency, and mutual understanding on topics such as compliance verification and safe operation of autonomous systems.
Designing control systems to require operator identity authentication and unalterable records of operation; enabling post-hoc compliance checks in case of plausible evidence of non-compliant autonomous weapon attacks.
Relating the quantity of weapons to corresponding capacities for human-in-the-loop operation of those weapons.
Designing weapons with air-gapped firing authorization circuits that are connected to the remote human operator but not to the on-board automated control system.
More generally, avoiding weapon designs that enable conversion from compliant to non-compliant categories or missions solely by software updates.
Designing weapons with formal proofs of relevant properties—e.g., the property that the weapon is unable to initiate an attack without human authorization. Proofs can, in principle, be provided using cryptographic techniques that allow the proofs to be checked by a third party without revealing any details of the underlying software.
Facilitate access to (non-classified) AI resources (software, data, methods for ensuring safe operation) to all states that remain in compliance and participate in transparency activities.
Component 2:
Define and universalize guiding principles for human involvement in the use of force.
Humans, not machines, are legal and moral agents in military operations.
It is a human responsibility to ensure that any attack, including one involving autonomous weapons, complies with the laws of war.
Humans responsible for initiating an attack must have sufficient understanding of the weapons, the targets, the environment and the context for use to determine whether that particular attack is lawful.
The attack must be bounded in space, time, target class, and means of attack in order for the determination about the lawfulness of that attack to be meaningful.
Militaries must invest in training, education, doctrine, policies, system design, and human-machine interfaces to ensure that humans remain responsible for attacks.
Component 3:
Develop protocols and/or technological means to mitigate the risk of unintentional escalation due to autonomous systems.
Specific potential measures include:
Developing safe rules for autonomous system behavior when in proximity to adversarial forces to avoid unintentional escalation or signaling. Examples include:
No-first-fire policy, so that autonomous weapons do not initiate hostilities without explicit human authorization.
A human must always be responsible for providing the mission for an autonomous system.
Taking steps to clearly distinguish exercises, patrols, reconnaissance, or other peacetime military operations from attacks in order to limit the possibility of reactions from adversary autonomous systems, such as autonomous air or coastal defenses.
Developing resilient communications links to ensure recallability of autonomous systems. Additionally, militaries should refrain from jamming others’ ability to recall their autonomous systems in order to afford the possibility of human correction in the event of unauthorized behavior.
Component 4:
Develop strategies for preventing proliferation to illicit uses, such as by criminals, terrorists, or rogue states:
Targeted multilateral controls to prevent large-scale sale and transfer of weaponizable robots and related military-specific components for illicit use.
Employ measures to render weaponizable robots less harmful (e.g., geofencing; hard-wired kill switch; onboard control systems largely implemented in unalterable, non-reprogrammable hardware such as application-specific integrated circuits).
Component 5:
Conduct research to improve technologies and human-machine systems to reduce non-combatant harm and ensure IHL-compliance in the use of future weapons, including:
Strategies to promote human moral engagement in decisions about the use of force
Risk assessment for autonomous weapon systems, including the potential for large-scale effects, geopolitical destabilization, accidental escalation, increased instability due to uncertainty about the relative military balance of power, and lowering thresholds to initiating conflict and for violence within conflict
Methodologies for ensuring the reliability and security of autonomous weapon systems
New techniques for verification, validation, explainability, characterization of failure conditions, and behavioral specifications.
About the Authors (in alphabetical order)
Ronald Arkin directs the Mobile Robot Laboratory at Georgia Tech.
Leslie Kaelbling is co-director of the Learning and Intelligent Systems Group at MIT.
Stuart Russell is a professor of computer science and engineering at UC Berkeley.
Dorsa Sadigh is an assistant professor of computer science and of electrical engineering at Stanford.
Paul Scharre directs the Technology and National Security Program at the Center for a New American Security (CNAS).
Bart Selman is a professor of computer science at Cornell.
Toby Walsh is a professor of artificial intelligence at the University of New South Wales (UNSW) Sydney.
The authors would like to thank Max Tegmark for organizing the three-day meeting from which this document was produced.
1 Autonomous Weapons System (AWS): A weapon system that, once activated, can select and engage targets without further intervention by a human operator. BACK TO TEXT↑
2 There is no implication that some authors would not personally support stronger recommendations. BACK TO TEXT↑
3 For ease of use, this working paper will frequently shorten “autonomous weapon system” to “autonomous weapon.” The terms should be treated as synonymous, with the understanding that “weapon” refers to the entire system: sensor, decision-making element, and munition. BACK TO TEXT↑
4 Anti-personnel lethal autonomous weapon system: A weapon system that, once activated, can select and engage dismounted human targets with lethal force and without further intervention by a human operator. BACK TO TEXT↑
5 The authors are not unanimous about this item because of concerns about ease of repurposing for mass-casualty missions targeting unarmed humans. The purpose of the lower limit on explosive payload weight would be to minimize the risk of such repurposing. There is precedent for using explosive weight limit as a mechanism of delineating between anti-personnel and anti-materiel weapons, such as the 1868 St. Petersburg Declaration Renouncing the Use, in Time of War, of Explosive Projectiles Under 400 Grammes Weight. BACK TO TEXT↑ Continue reading
Posted in Human Robots
#436021 AI Faces Speed Bumps and Potholes on Its ...
Implementing machine learning in the real world isn’t easy. The tools are available and the road is well-marked—but the speed bumps are many.
That was the conclusion of panelists wrapping up a day of discussions at the IEEE AI Symposium 2019, held at Cisco’s San Jose, Calif., campus last week.
The toughest problem, says Ben Irving, senior manager of Cisco’s strategy innovations group, is people.
It’s tough to find data scientist expertise, he indicated, so companies are looking into non-traditional sources of personnel, like political science. “There are some untapped areas with a lot of untapped data science expertise,” Irving says.
Lazard’s artificial intelligence manager Trevor Mottl agreed that would-be data scientists don’t need formal training or experience to break into the field. “This field is changing really rapidly,” he says. “There are new language models coming out every month, and new tools, so [anyone should] expect to not know everything. Experiment, try out new tools and techniques, read, study, spend time; there aren’t any true experts at this point because the foundational elements are shifting so rapidly.”
“It is a wonderful time to get into a field,” he reasons, noting that it doesn’t take long to catch up because there aren’t 20 years of history.”
Confusion about what different kinds of machine learning specialists do doesn’t help the personnel situation. An audience member asked panelists to explain the difference between data scientist, data analyst, and data engineer. Darrin Johnson, Nvidia global director of technical marketing for enterprise, admitted it’s hard to sort out, and any two companies could define the positions differently. “Sometimes,” he says, particularly at smaller companies, “a data scientist plays all three roles. But as companies grow, there are different groups that ingest data, clean data, and use data. At some companies, training and inference are separate. It really depends, which is a challenge when you are trying to hire someone.”
Mitigating the risks of a hot job market
The competition to hire data scientists, analysts, engineers, or whatever companies call them requires that managers make sure any work being done is structured and comprehensible at all times, the panelists cautioned.
“We need to remember that our data scientists go home every day and sometimes they don’t come back because they go home and then go to a different company,” says Lazard’s Mottl. “That’s a fact of life. If you give people choice on [how they do development], and have a successful person who gets poached by competitor, you have to either hire a team to unwrap what that person built or jettison their work and rebuild it.”
By contrast, he says, “places that have structured coding and structured commits and organized constructions of software have done very well.”
But keeping all of a company’s engineers working with the same languages and on the same development paths is not easy to do in a field that moves as fast as machine learning. Zongjie Diao, Cisco director of product management for machine learning, quipped: “I have a data scientist friend who says the speed at which he changes girlfriends is less than speed at which he changes languages.”
The data scientist/IT manager clash
Once a company finds the data engineers and scientists they need and get them started on the task of applying machine learning to that company’s operations, one of the first obstacles they face just might be the company’s IT department, the panelists suggested.
“IT is process oriented,” Mottl says. The IT team “knows how to keep data secure, to set up servers. But when you bring in a data science team, they want sandboxes, they want freedom, they want to explore and play.”
Also, Nvidia’s Johnson pointed out, “There is a language barrier.” The AI world, he says, is very different from networking or storage, and data scientists find it hard to articulate their requirements to IT.
On the ground or in the cloud?
And then there is the decision of where exactly machine learning should happen—on site, or in the cloud? At Lazard, Mottl says, the deep learning engineers do their experimentation on premises; that’s their sandbox. “But when we deploy, we deploy in the cloud,” he says.
Nvidia, Johnson says, thinks the opposite approach is better. We see the cloud as “the sandbox,” he says. “So you can run as many experiments as possible, fail fast, and learn faster.”
For Cisco’s Irving, the “where” of machine learning depends on the confidentiality of the data.
Mottl, who says rolling machine learning technology into operation can hit resistance from all across the company, had one last word of caution for those aiming to implement AI:
Data scientists are building things that might change the ways other people in the organization work, like sales and even knowledge workers. [You need to] think about the internal stakeholders and prepare them, because the last thing you want to do is to create a valuable new thing that nobody likes and people take potshots against.
The AI Symposium was organized by the Silicon Valley chapters of the IEEE Young Professionals, the IEEE Consultants’ Network, and IEEE Women in Engineering and supported by Cisco. Continue reading
Posted in Human Robots
#436005 NASA Hiring Engineers to Develop “Next ...
It’s been nearly six years since NASA unveiled Valkyrie, a state-of-the-art full-size humanoid robot. After the DARPA Robotics Challenge, NASA has continued to work with Valkyrie at Johnson Space Center, and has also provided Valkyrie robots to several different universities. Although it’s not a new platform anymore (six years is a long time in robotics), Valkyrie is still very capable, with plenty of potential for robotics research.
With that in mind, we were caught by surprise when over the last several months, Jacobs, a Dallas-based engineering company that appears to provide a wide variety of technical services to anyone who wants them, has posted several open jobs in need of roboticists in the Houston, Texas, area who are interested in working with NASA on “the next generation of humanoid robot.”
Here are the relevant bullet points from the one of the job descriptions (which you can view at this link):
Work directly with NASA Johnson Space Center in designing the next generation of humanoid robot.
Join the Valkyrie humanoid robot team in NASA’s Robotic Systems Technology Branch.
Build on the success of the existing Valkyrie and Robonaut 2 humanoid robots and advance NASA’s ability to project a remote human presence and dexterous manipulation capability into challenging, dangerous, and distant environments both in space and here on earth.
The question is, why is NASA developing its own humanoid robot (again) when it could instead save a whole bunch of time and money by using a platform that already exists, whether it’s Atlas, Digit, Valkyrie itself, or one of the small handful of other humanoids that are more or less available? The only answer that I can come up with is that no existing platforms meet NASA’s requirements, whatever those may be. And if that’s the case, what kind of requirements are we talking about? The obvious one would be the ability to work in the kinds of environments that NASA specializes in—space, the Moon, and Mars.
Image: NASA
Artist’s concept of NASA’s Valkyrie humanoid robot working on the surface of Mars.
NASA’s existing humanoid robots, including Robonaut 2 and Valkyrie, were designed to operate on Earth. Robonaut 2 ended up going to space anyway (it’s recently returned to Earth for repairs), but its hardware was certainly never intended to function outside of the International Space Station. Working in a vacuum involves designing for a much more rigorous set of environmental challenges, and things get even worse on the Moon or on Mars, where highly abrasive dust gets everywhere.
We know that it’s possible to design robots for long term operation in these kinds of environments because we’ve done it before. But if you’re not actually going to send your robot off-world, there’s very little reason to bother making sure that it can operate through (say) 300° Celsius temperature swings like you’d find on the Moon. In the past, NASA has quite sensibly focused on designing robots that can be used as platforms for the development of software and techniques that could one day be applied to off-world operations, without over-engineering those specific robots to operate in places that they would almost certainly never go. As NASA increasingly focuses on a return to the Moon, though, maybe it’s time to start thinking about a humanoid robot that could actually do useful stuff on the lunar surface.
Image: NASA
Artist’s concept of the Gateway moon-orbiting space station (seen on the right) with an Orion crew vehicle approaching.
The other possibility that I can think of, and perhaps the more likely one, is that this next humanoid robot will be a direct successor to Robonaut 2, intended for NASA’s Gateway space station orbiting the Moon. Some of the robotics folks at NASA that we’ve talked to recently have emphasized how important robotics will be for Gateway:
Trey Smith, NASA Ames: Everybody at NASA is really excited about work on the Gateway space station that would be in near lunar space. We don’t have definite plans for what would happen on the Gateway yet, but there’s a general recognition that intra-vehicular robots are important for space stations. And so, it would not be surprising to see a mobile manipulator like Robonaut, and a free flyer like Astrobee, on the Gateway.
If you have an un-crewed cargo vehicle that shows up stuffed to the rafters with cargo bags and it docks with the Gateway when there’s no crew there, it would be very useful to have intra-vehicular robots that can pull all those cargo bags out, unpack them, stow all the items, and then even allow the cargo vehicle to detach before the crew show up so that the crew don’t have to waste their time with that.
Julia Badger, NASA JSC: One of the systems on board Gateway is going to be intravehicular robots. They’re not going to necessarily look like Robonaut, but they’ll have some of the same functionality as Robonaut—being mobile, being able to carry payloads from one part of the module to another, doing some dexterous manipulation tasks, inspecting behind panels, those sorts of things.
Image: NASA
Artist’s concept of NASA’s Valkyrie humanoid robot working inside a spacecraft.
Since Gateway won’t be crewed by humans all of the time, it’ll be important to have a permanent robotic presence to keep things running while nobody is home while saving on resources by virtue of the fact that robots aren’t always eating food, drinking water, consuming oxygen, demanding that the temperature stays just so, and producing a variety of disgusting kinds of waste. Obviously, the robot won’t be as capable as humans, but if they can manage to do even basic continuing maintenance tasks (most likely through at least partial teleoperation), that would be very useful.
Photo: Evan Ackerman/IEEE Spectrum
NASA’s Robonaut team plans to perform a variety of mobility and motion-planning experiments using the robot’s new legs, which can grab handrails on the International Space Station.
As for whether robots designed for Gateway would really fall into the “humanoid” category, it’s worth considering that Gateway is designed for humans, implying that an effective robotic system on Gateway would need to be able to interact with the station in similar ways to how a human astronaut would. So, you’d expect to see arms with end-effectors that can grip things as well as push buttons, and some kind of mobility system—the legged version of Robonaut 2 seems like a likely template, but redesigned from the ground up to work in space, incorporating all the advances in robotics hardware and computing that have taken place over the last decade.
We’ve been pestering NASA about this for a little bit now, and they’re not ready to comment on this project, or even to confirm it. And again, everything in this article (besides the job post, which you should totally check out and consider applying for) is just speculation on our part, and we could be wrong about absolutely all of it. As soon as we hear more, we’ll definitely let you know. Continue reading
Posted in Human Robots