Tag Archives: object

#435683 How High Fives Help Us Get in Touch With ...

Posted on September 16, 2019 by Android

The human sense of touch is so naturally ingrained in our everyday lives that we often don’t notice its presence. Even so, touch is a crucial sensing ability that helps people to understand the world and connect with others. As the market for robots grows, and as robots become more ingrained into our environments, people will expect robots to participate in a wide variety of social touch interactions. At Oregon State University’s Collaborative Robotics and Intelligent Systems (CoRIS) Institute, I research how to equip everyday robots with better social-physical interaction skills—from playful high-fives to challenging physical therapy routines.

Some commercial robots already possess certain physical interaction skills. For example, the videoconferencing feature of mobile telepresence robots can keep far-away family members connected with one another. These robots can also roam distant spaces and bump into people, chairs, and other remote objects. And my Roomba occasionally tickles my toes before turning to vacuum a different area of the room. As a human being, I naturally interpret this (and other Roomba behaviors) as social, even if they were not intended as such. At the same time, for both of these systems, social perceptions of the robots’ physical interaction behaviors are not well understood, and these social touch-like interactions cannot be controlled in nuanced ways.

Before joining CoRIS early this year, I was a postdoc at the University of Southern California’s Interaction Lab, and prior to that, I completed my doctoral work at the GRASP Laboratory’s Haptics Group at the University of Pennsylvania. My dissertation focused on improving the general understanding of how robot control and planning strategies influence perceptions of social touch interactions. As part of that research, I conducted a study of human-robot hand-to-hand contact, focusing on an interaction somewhere between a high five and a hand-clapping game. I decided to study this particular interaction because people often high five, and they will likely expect robots in everyday spaces to high five as well!

I conducted a study of human-robot hand-to-hand contact, focusing on an interaction somewhere between a high five and a hand-clapping game. I decided to study this particular interaction because people often high five, and they will likely expect robots to high five as well!

The implications of motion and planning on the social touch experience in these interactions is also crucial—think about a disappointingly wimpy (or triumphantly amazing) high five that you’ve experienced in the past. This great or terrible high-fiving experience could be fleeting, but it could also influence who you interact with, who you’re friends with, and even how you perceive the character or personalities of those around you. This type of perception, judgement, and response could extend to personal robots, too!

An investigation like this requires a mixture of more traditional robotics research (e.g., understanding how to move and control a robot arm, developing models of the desired robot motion) along with techniques from design and psychology (e.g., performing interviews with research participants, using best practices from experimental methods in perception). Enabling robots with social touch abilities also comes with many challenges, and even skilled humans can have trouble anticipating what another person is about to do. Think about trying to make satisfying hand contact during a high five—you might know the classic adage “watch the elbow,” but if you’re like me, even this may not always work.

I conducted a research study involving eight different types of human-robot hand contact, with different combinations of the following: interactions with a facially reactive or non-reactive robot, a physically reactive or non-reactive planning strategy, and a lower or higher robot arm stiffness. My robotic system could become facially reactive by changing its facial expression in response to hand contact, or physically reactive by updating its plan of where to move next after sensing hand contact. The stiffness of the robot could be adjusted by changing a variable that controlled how quickly the robot’s motors tried to pull its arm to the desired position. I knew from previous research that fine differences in touch interactions can have a big impact on perceived robot character. For example, if a robot grips an object too tightly or for too long while handing an object to a person, it might be perceived as greedy, possessive, or perhaps even Sméagol-like. A robot that lets go too soon might appear careless or sloppy.

In the example cases of robot grip, it’s clear that understanding people’s perceptions of robot characteristics and personality can help roboticists choose the right robot design based on the proposed operating environment of the robot. I likewise wanted to learn how the facial expressions, physical reactions, and stiffness of a hand-clapping robot would influence human perceptions of robot pleasantness, energeticness, dominance, and safety. Understanding this relationship can help roboticists to equip robots with personalities appropriate for the task at hand. For example, a robot assisting people in a grocery store may need to be designed with a high level of pleasantness and only moderate energy, while a maximally effective robot for comedy roast battles may need high degrees of energy and dominance above all else.

After many a late night at the GRASP Lab clapping hands with a big red robot, I was ready to conduct the study. Twenty participants visited the lab to clap hands with our Baxter Research Robot and help me begin to understand how characteristics of this humanoid robot’s social touch influenced its pleasantness, energeticness, dominance, and apparent safety. Baxter interacted with participants using a custom 3D-printed hand that was inlaid with silicone inserts.

The study showed that a facially reactive robot seemed more pleasant and energetic. A physically reactive robot seemed less pleasant, energetic, and dominant for this particular study design and interaction. I thought contact with a stiffer robot would seem harder (and therefore more dominant and less safe), but counter to my expectations, a stiffer-armed robot seemed safer and less dominant to participants. This may be because the stiffer robot was more precise in following its pre-programmed trajectory, therefore seeming more predictable and less free-spirited.

Safety ratings of the robot were generally high, and several participants commented positively on the robot’s facial expressions. Some participants attributed inventive (and non-existent) intelligences to the robot—I used neither computer vision nor the Baxter robot’s cameras in this study, but more than one participant complimented me on how well the robot tracked their hand position. While interacting with the robot, participants displayed happy facial expressions more than any other analyzed type of expression.

Photo: Naomi Fitter

Participants were asked to clap hands with Baxter and describe how they perceived the robot in terms of its pleasantness, energeticness, dominance, and apparent safety.

Circling back to the idea of how people might interpret even rudimentary and practical robot behaviors as social, these results show that this type of social perception isn’t just true for my lovable (but sometimes dopey) Roomba, but also for collaborative industrial robots, and generally, any robot capable of physical human-robot interaction. In designing the motion of Baxter, the adjustment of a single number in the equation that controls joint stiffness can flip the robot from seeming safe and docile to brash and commanding. These implications are sometimes predictable, but often unexpected.

The results of this particular study give us a partial guide to manipulating the emotional experience of robot users by adjusting aspects of robot control and planning, but future work is needed to fully understand the design space of social touch. Will materials play a major role? How about personalized machine learning? Do results generalize over all robot arms, or even a specialized subset like collaborative industrial robot arms? I’m planning to continue answering these questions, and when I finally solve human-robot social touch, I’ll high five all my robots to celebrate.

Naomi Fitter is an assistant professor in the Collaborative Robotics and Intelligent Systems (CoRIS) Institute at Oregon State University, where her Social Haptics, Assistive Robotics, and Embodiment (SHARE) research group aims to equip robots with the ability to engage and empower people in interactions from playful high-fives to challenging physical therapy routines. She completed her doctoral work in the GRASP Laboratory’s Haptics Group and was a postdoctoral scholar in the University of Southern California’s Interaction Lab from 2017 to 2018. Naomi’s not-so-secret pastime is performing stand-up and improv comedy. Continue reading →

Posted in Human Robots

#435664 Swarm Robots Mimic Ant Jaws to Flip and ...

Posted on September 15, 2019 by Android

Small robots are appealing because they’re simple, cheap, and it’s easy to make a lot of them. Unfortunately, being simple and cheap means that each robot individually can’t do a whole lot. To make up for this, you can do what insects do—leverage that simplicity and low-cost to just make a huge swarm of simple robots, and together, they can cooperate to carry out relatively complex tasks.

Using insects as an example does set a bit of an unfair expectation for the poor robots, since insects are (let’s be honest) generally smarter and much more versatile than a robot on their scale could ever hope to be. Most robots with insect-like capabilities (like DASH and its family) are really too big and complex to be turned into swarms, because to make a vast amount of small robots, things like motors aren’t going to work because they’re too expensive.

The question, then, is to how to make a swarm of inexpensive small robots with insect-like mobility that don’t need motors to get around, and Jamie Paik’s Reconfigurable Robotics Lab at EPFL has an answer, inspired by trap-jaw ants.

Let’s talk about trap-jaw ants for just a second, because they’re insane. You can read this 2006 paper about them if you’re particularly interested in insane ants (and who isn’t!), but if you just want to hear the insane bit, it’s that trap-jaw ants can fire themselves into the air by biting the ground (!). In just 0.06 millisecond, their half-millimeter long mandibles can close at a top speed of 64 meters per second, which works out to an acceleration of about 100,000 g’s. Biting the ground causes the ant’s head to snap back with a force of 300 times the body weight of the ant itself, which launches the ant upwards. The ants can fly 8 centimeters vertically, and up to 15 cm horizontally—this is a lot, for an ant that’s just a few millimeters long.

Trap-jaw ants can fire themselves into the air by biting the ground, causing the ant’s head to snap back with a force of 300 times the body weight of the ant itself

EPFL’s robots, called Tribots, look nothing at all like trap-jaw ants, which personally I am fine with. They’re about 5 cm tall, weighing 10 grams each, and can be built on a flat sheet, and then folded into a tripod shape, origami-style. Or maybe it’s kirigami, because there’s some cutting involved. The Tribots are fully autonomous, meaning they have onboard power and control, including proximity sensors that allow them to detect objects and avoid them.

Photo: Marc Delachaux/EPFL

EPFL researchers Zhenishbek Zhakypov and Jamie Paik.

Avoiding objects is where the trap-jaw ants come in. Using two different shape-memory actuators (a spring and a latch, similar to how the ant’s jaw works), the Tribots can move around using a bunch of different techniques that can adapt to the terrain that they’re on, including:

Vertical jumping for height
Horizontal jumping for distance
Somersault jumping to clear obstacles
Walking on textured terrain with short hops (called “flic-flac” walking)
Crawling on flat surfaces

Here’s the robot in action:

Tribot’s maximum vertical jump is 14 cm (2.5 times its height), and horizontally it can jump about 23 cm (almost 4 times its length). Tribot is actually quite efficient in these movements, with a cost of transport much lower than similarly-sized robots, on par with insects themselves.

Working together, small groups of Tribots can complete tasks that a single robot couldn’t do alone. One example is pushing a heavy object a set distance. It turns out that you need five Tribots for this task—a leader robot, two worker robots, a monitor robot to measure the distance that the object has been pushed, and then a messenger robot to relay communications around the obstacle.

Image: EPFL

Five Tribots collaborate to move an object to a desired position, using coordination between a leader, two workers, a monitor, and a messenger robot. The leader orders the two worker robots to push the object while the monitor measures the relative position of the object. As the object blocks the two-way link between the leader and the monitor, the messenger maintains the communication link.

The researchers acknowledge that the current version of the hardware is limited in pretty much every way (mobility, sensing, and computation), but it does a reasonable job of demonstrating what’s possible with the concept. The plan going forward is to automate fabrication in order to “enable on-demand, ’push-button-manufactured’” robots.

“Designing minimal and scalable insect-inspired multi-locomotion millirobots,” by Zhenishbek Zhakypov, Kazuaki Mori, Koh Hosoda, and Jamie Paik from EPFL and Osaka University, is published in the current issue of Nature.
[ RRL ] via [ EPFL ] Continue reading →

Posted in Human Robots

#435658 Video Friday: A Two-Armed Robot That ...

Posted on September 15, 2019 by Android

Video Friday is your weekly selection of awesome robotics videos, collected by your Automaton bloggers. We’ll also be posting a weekly calendar of upcoming robotics events for the next few months; here’s what we have so far (send us your events!):

ICRES 2019 – July 29-30, 2019 – London, U.K.
DARPA SubT Tunnel Circuit – August 15-22, 2019 – Pittsburgh, Pa., USA
IEEE Africon 2019 – September 25-27, 2019 – Accra, Ghana
ISRR 2019 – October 6-10, 2019 – Hanoi, Vietnam
Ro-Man 2019 – October 14-18, 2019 – New Delhi, India
Humanoids 2019 – October 15-17, 2019 – Toronto, Canada
Let us know if you have suggestions for next week, and enjoy today’s videos.

I’m sure you’ve seen this video already because you read this blog every day, but if you somehow missed it because you were skiing across Antarctica (the only valid excuse we’re accepting today), here’s our video introducing HMI’s Aquanaut transforming robot submarine.

And after you recover from all that frostbite, make sure and read our in-depth feature article here.

[ Aquanaut ]

Last week we complained about not having seen a ballbot with a manipulator, so Roberto from CMU shared a new video of their ballbot, featuring a pair of 7-DoF arms.

We should learn more at Humanoids 2019.

[ CMU ]

Thanks Roberto!

The FAA is making it easier for recreational drone pilots to get near-realtime approval to fly in lightly controlled airspace.

[ LAANC ]

Self-reconfigurable modular robots are usually composed of multiple modules with uniform docking interfaces that can be transformed into different configurations by themselves. The reconfiguration planning problem is finding what sequence of reconfiguration actions are required for one arrangement of modules to transform into another. We present a novel reconfiguration planning algorithm for modular robots. The algorithm compares the initial configuration with the goal configuration efficiently. The reconfiguration actions can be executed in a distributed manner so that each module can efficiently finish its reconfiguration task which results in a global reconfiguration for the system. In the end, the algorithm is demonstrated on real modular robots and some example reconfiguration tasks are provided.

[ CKbot ]

A nice design of a gripper that uses a passive thumb of sorts to pick up flat objects from flat surfaces.

[ Paper ] via [ Laval University ]

I like this video of a palletizing robot from Kawasaki because in the background you can see a human doing the exact same job and obviously not enjoying it.

[ Kawasaki ]

This robot cleans and “brings joy and laughter.” What else do we need?

I do appreciate that all the robots are named Leo, and that they’re also all female.

[ LionsBot ]

This is less of a dishwashing robot and more of a dishsorting robot, but we’ll forgive it because it doesn’t drop a single dish.

[ TechMagic ]

Thanks Ryosuke!

A slight warning here that the robot in the following video (which costs something like $180,000) appears “naked” in some scenes, none of which are strictly objectionable, we hope.

Beautifully slim and delicate motion life-size motion figures are ideal avatars for expressing emotions to customers in various arts, content and businesses. We can provide a system that integrates not only motion figures but all moving devices.

[ Speecys ]

The best way to operate a Husky with a pair of manipulators on it is to become the robot.

[ UT Austin ]

The FlyJacket drone control system from EPFL has been upgraded so that it can yank you around a little bit.

In several fields of human-machine interaction, haptic guidance has proven to be an effective training tool for enhancing user performance. This work presents the results of psychophysical and motor learning studies that were carried out with human participant to assess the effect of cable-driven haptic guidance for a task involving aerial robotic teleoperation. The guidance system was integrated into an exosuit, called the FlyJacket, that was developed to control drones with torso movements. Results for the Just Noticeable Difference (JND) and from the Stevens Power Law suggest that the perception of force on the users’ torso scales linearly with the amplitude of the force exerted through the cables and the perceived force is close to the magnitude of the stimulus. Motor learning studies reveal that this form of haptic guidance improves user performance in training, but this improvement is not retained when participants are evaluated without guidance.

[ EPFL ]

The SAND Challenge is an opportunity for small businesses to compete in an autonomous unmanned aerial vehicle (UAV) competition to help NASA address safety-critical risks associated with flying UAVs in the national airspace. Set in a post-natural disaster scenario, SAND will push the envelope of aviation.

[ NASA ]

Legged robots have the potential to traverse diverse and rugged terrain. To find a safe and efficient navigation path and to carefully select individual footholds, it is useful to predict properties of the terrain ahead of the robot. In this work, we propose a method to collect data from robot-terrain interaction and associate it to images, to then train a neural network to predict terrain properties from images.

[ RSL ]

Misty wants to be your new receptionist.

[ Misty Robotics ]

For years, we’ve been pointing out that while new Roombas have lots of great features, older Roombas still do a totally decent job of cleaning your floors. This video is a performance comparison between the newest Roomba (the S9+) and the original 2002 Roomba (!), and the results will surprise you. Or maybe they won’t.

[ Vacuum Wars ]

Lex Fridman from MIT interviews Chris Urmson, who was involved in some of the earliest autonomous vehicle projects, Google’s original self-driving car among them, and is currently CEO of Aurora Innovation.

Chris Urmson was the CTO of the Google Self-Driving Car team, a key engineer and leader behind the Carnegie Mellon autonomous vehicle entries in the DARPA grand challenges and the winner of the DARPA urban challenge. Today he is the CEO of Aurora Innovation, an autonomous vehicle software company he started with Sterling Anderson, who was the former director of Tesla Autopilot, and Drew Bagnell, Uber’s former autonomy and perception lead.

[ AI Podcast ]

In this week’s episode of Robots in Depth, Per speaks with Lael Odhner from RightHand Robotics.

Lael Odhner is a co-founder of RightHand Robotics, that is developing a gripper based on the combination of control and soft, compliant parts to get better grasping of objects. Their work focuses on grasping and manipulating everyday human objects in everyday environments.This mimics how human hands combine control and flexibility to grasp objects with great dexterity.

The combination of control and compliance makes the RightHand robotics gripper very light-weight and affordable. The compliance makes it easier to grasp objects of unknown shape and differs from the way industrial robots usually grip. The compliance also helps in a more unstructured environment where contact with the object and its surroundings cannot be exactly predicted.

[ RightHand Robotics ] via [ Robots in Depth ] Continue reading →

Posted in Human Robots

#435619 Video Friday: Watch This Robot Dog ...

Posted on September 13, 2019 by Android

Video Friday is your weekly selection of awesome robotics videos, collected by your Automaton bloggers. We’ll also be posting a weekly calendar of upcoming robotics events for the next few months; here’s what we have so far (send us your events!):

IEEE Africon 2019 – September 25-27, 2019 – Accra, Ghana
RoboBusiness 2019 – October 1-3, 2019 – Santa Clara, CA, USA
ISRR 2019 – October 6-10, 2019 – Hanoi, Vietnam
Ro-Man 2019 – October 14-18, 2019 – New Delhi, India
Humanoids 2019 – October 15-17, 2019 – Toronto, Canada
ARSO 2019 – October 31-1, 2019 – Beijing, China
ROSCon 2019 – October 31-1, 2019 – Macau
IROS 2019 – November 4-8, 2019 – Macau
Let us know if you have suggestions for next week, and enjoy today’s videos.

Team PLUTO (University of Pennsylvania, Ghost Robotics, and Exyn Technologies) put together this video giving us a robot’s-eye-view (or whatever they happen to be using for eyes) of the DARPA Subterranean Challenge tunnel circuits.

[ PLUTO ]

Zhifeng Huang has been improving his jet-stepping humanoid robot, which features new hardware and the ability to take larger and more complex steps.

This video reported the last progress of an ongoing project utilizing ducted-fan propulsion system to improve humanoid robot’s ability in stepping over large ditches. The landing point of the robot’s swing foot can be not only forward but also side direction. With keeping quasi-static balance, the robot was able to step over a ditch with 450mm in width (up to 97% of the robot’s leg’s length) in 3D stepping.

[ Paper ]

Thanks Zhifeng!

These underacuated hands from Matei Ciocarlie’s lab at Columbia are magically able to reconfigure themselves to grasp different object types with just one or two motors.

[ Paper ] via [ ROAM Lab ]

This is one reason we should pursue not “autonomous cars” but “fully autonomous cars” that never require humans to take over. We can’t be trusted.

During our early days as the Google self-driving car project, we invited some employees to test our vehicles on their commutes and weekend trips. What we were testing at the time was similar to the highway driver assist features that are now available on cars today, where the car takes over the boring parts of the driving, but if something outside its ability occurs, the driver has to take over immediately.

What we saw was that our testers put too much trust in that technology. They were doing things like texting, applying makeup, and even falling asleep that made it clear they would not be ready to take over driving if the vehicle asked them to. This is why we believe that nothing short of full autonomy will do.

[ Waymo ]

Buddy is a DIY and fetchingly minimalist social robot (of sorts) that will be coming to Kickstarter this month.

We have created a new arduino kit. His name is Buddy. He is a DIY social robot to serve as a replacement for Jibo, Cozmo, or any of the other bots that are no longer available. Fully 3D printed and supported he adds much more to our series of Arduino STEM robotics kits.

Buddy is able to look around and map his surroundings and react to changes within them. He can be surprised and he will always have a unique reaction to changes. The kit can be built very easily in less than an hour. It is even robust enough to take the abuse that kids can give it in a classroom.

[ Littlebots ]

The android Mindar, based on the Buddhist deity of mercy, preaches sermons at Kodaiji temple in Kyoto, and its human colleagues predict that with artificial intelligence it could one day acquire unlimited wisdom. Developed at a cost of almost $1 million (¥106 million) in a joint project between the Zen temple and robotics professor Hiroshi Ishiguro, the robot teaches about compassion and the dangers of desire, anger and ego.

[ Japan Times ]

I’m not sure whether it’s the sound or what, but this thing scares me for some reason.

[ BIRL ]

This gripper uses magnets as a sort of adjustable spring for dynamic stiffness control, which seems pretty clever.

[ Buffalo ]

What a package of medicine sees while being flown by drone from a hospital to a remote clinic in the Dominican Republic. The drone flew 11 km horizontally and 800 meters vertically, and I can’t even imagine what it would take to make that drive.

[ WeRobotics ]

My first ride in a fully autonomous car was at Stanford in 2009. I vividly remember getting in the back seat of a descendant of Junior, and watching the steering wheel turn by itself as the car executed a perfect parking maneuver. Ten years later, it’s still fun to watch other people have that experience.

[ Waymo ]

Flirtey, the pioneer of the commercial drone delivery industry, has unveiled the much-anticipated first video of its next-generation delivery drone, the Flirtey Eagle. The aircraft designer and manufacturer also unveiled the Flirtey Portal, a sophisticated take off and landing platform that enables scalable store-to-door operations; and an autonomous software platform that enables drones to deliver safely to homes.

[ Flirtey ]

EPFL scientists are developing new approaches for improved control of robotic hands – in particular for amputees – that combines individual finger control and automation for improved grasping and manipulation. This interdisciplinary proof-of-concept between neuroengineering and robotics was successfully tested on three amputees and seven healthy subjects.

[ EPFL ]

This video is a few years old, but we’ll take any excuse to watch the majestic sage-grouse be majestic in all their majesticness.

[ UC Davis ]

I like the idea of a game of soccer (or, football to you weirdos in the rest of the world) where the ball has a mind of its own.

[ Sphero ]

Looks like the whole delivery glider idea is really taking off! Or, you know, not taking off.

Weird that they didn’t show the landing, because it sure looked like it was going to plow into the side of the hill at full speed.

[ Yates ] via [ sUAS News ]

This video is from a 2018 paper, but it’s not like we ever get tired of seeing quadrupeds do stuff, right?

[ MIT ]

Founder and Head of Product, Ian Bernstein, and Head of Engineering, Morgan Bell, have been involved in the Misty project for years and they have learned a thing or two about building robots. Hear how and why Misty evolved into a robot development platform, learn what some of the earliest prototypes did (and why they didn’t work for what we envision), and take a deep dive into the technology decisions that form the Misty II platform.

[ Misty Robotics ]

Lex Fridman interviews Vijay Kumar on the Artifiical Intelligence Podcast.

[ AI Podcast ]

This week’s CMU RI Seminar is from Ross Knepper at Cornell, on Formalizing Teamwork in Human-Robot Interaction.

Robots out in the world today work for people but not with people. Before robots can work closely with ordinary people as part of a human-robot team in a home or office setting, robots need the ability to acquire a new mix of functional and social skills. Working with people requires a shared understanding of the task, capabilities, intentions, and background knowledge. For robots to act jointly as part of a team with people, they must engage in collaborative planning, which involves forming a consensus through an exchange of information about goals, capabilities, and partial plans. Often, much of this information is conveyed through implicit communication. In this talk, I formalize components of teamwork involving collaboration, communication, and representation. I illustrate how these concepts interact in the application of social navigation, which I argue is a first-class example of teamwork. In this setting, participants must avoid collision by legibly conveying intended passing sides via nonverbal cues like path shape. A topological representation using the braid groups enables the robot to reason about a small enumerable set of passing outcomes. I show how implicit communication of topological group plans achieves rapid covergence to a group consensus, and how a robot in the group can deliberately influence the ultimate outcome to maximize joint performance, yielding pedestrian comfort with the robot.

[ CMU RI ]

In this week’s episode of Robots in Depth, Per speaks with Julien Bourgeois about Claytronics, a project from Carnegie Mellon and Intel to develop “programmable matter.”

Julien started out as a computer scientist. He was always interested in robotics privately but then had the opportunity to get into micro robots when his lab was merged into the FEMTO-ST Institute. He later worked with Seth Copen Goldstein at Carnegie Mellon on the Claytronics project.

Julien shows an enlarged mock-up of the small robots that make up programmable matter, catoms, and speaks about how they are designed. Currently he is working on a unit that is one centimeter in diameter and he shows us the very small CPU that goes into that model.

[ Robots in Depth ] Continue reading →

Posted in Human Robots

#435589 Construction Robots Learn to Excavate by ...

Posted on September 11, 2019 by Android

Pavel Savkin remembers the first time he watched a robot imitate his movements. Minutes earlier, the engineer had finished “showing” the robotic excavator its new goal by directing its movements manually. Now, running on software Savkin helped design, the robot was reproducing his movements, gesture for gesture. “It was like there was something alive in there—but I knew it was me,” he said.

Savkin is the CTO of SE4, a robotics software project that styles itself the “driver” of a fleet of robots that will eventually build human colonies in space. For now, SE4 is focused on creating software that can help developers communicate with robots, rather than on building hardware of its own.
The Tokyo-based startup showed off an industrial arm from Universal Robots that was running SE4’s proprietary software at SIGGRAPH in July. SE4’s demonstration at the Los Angeles innovation conference drew the company’s largest audience yet. The robot, nicknamed Squeezie, stacked real blocks as directed by SE4 research engineer Nathan Quinn, who wore a VR headset and used handheld controls to “show” Squeezie what to do.

As Quinn manipulated blocks in a virtual 3D space, the software learned a set of ordered instructions to be carried out in the real world. That order is essential for remote operations, says Quinn. To build remotely, developers need a way to communicate instructions to robotic builders on location. In the age of digital construction and industrial robotics, giving a computer a blueprint for what to build is a well-explored art. But operating on a distant object—especially under conditions that humans haven’t experienced themselves—presents challenges that only real-time communication with operators can solve.

The problem is that, in an unpredictable setting, even simple tasks require not only instruction from an operator, but constant feedback from the changing environment. Five years ago, the Swedish fiber network provider umea.net (part of the private Umeå Energy utility) took advantage of the virtual reality boom to promote its high-speed connections with the help of a viral video titled “Living with Lag: An Oculus Rift Experiment.” The video is still circulated in VR and gaming circles.

In the experiment, volunteers donned headgear that replaced their real-time biological senses of sight and sound with camera and audio feeds of their surroundings—both set at a 3-second delay. Thus equipped, volunteers attempt to complete everyday tasks like playing ping-pong, dancing, cooking, and walking on a beach, with decidedly slapstick results.

At outer-orbit intervals, including SE4’s dream of construction projects on Mars, the limiting factor in communication speed is not an artificial delay, but the laws of physics. The shifting relative positions of Earth and Mars mean that communications between the planets—even at the speed of light—can take anywhere from 3 to 22 minutes.

A long-distance relationship

Imagine trying to manage a construction project from across an ocean without the benefit of intelligent workers: sending a ship to an unknown world with a construction crew and blueprints for a log cabin, and four months later receiving a letter back asking how to cut down a tree. The parallel problem in long-distance construction with robots, according to SE4 CEO Lochlainn Wilson, is that automation relies on predictability. “Every robot in an industrial setting today is expecting a controlled environment.”
Platforms for applying AR and VR systems to teach tasks to artificial intelligences, as SE4 does, are already proliferating in manufacturing, healthcare, and defense. But all of the related communications systems are bound by physics and, specifically, the speed of light.
The same fundamental limitation applies in space. “Our communications are light-based, whether they’re radio or optical,” says Laura Seward Forczyk, a planetary scientist and consultant for space startups. “If you’re going to Mars and you want to communicate with your robot or spacecraft there, you need to have it act semi- or mostly-independently so that it can operate without commands from Earth.”

Semantic control
That’s exactly what SE4 aims to do. By teaching robots to group micro-movements into logical units—like all the steps to building a tower of blocks—the Tokyo-based startup lets robots make simple relational judgments that would allow them to receive a full set of instruction modules at once and carry them out in order. This sidesteps the latency issue in real-time bilateral communications that could hamstring a project or at least make progress excruciatingly slow.
The key to the platform, says Wilson, is the team’s proprietary operating software, “Semantic Control.” Just as in linguistics and philosophy, “semantics” refers to meaning itself, and meaning is the key to a robot’s ability to make even the smallest decisions on its own. “A robot can scan its environment and give [raw data] to us, but it can’t necessarily identify the objects around it and what they mean,” says Wilson.

That’s where human intelligence comes in. As part of the demonstration phase, the human operator of an SE4-controlled machine “annotates” each object in the robot’s vicinity with meaning. By labeling objects in the VR space with useful information—like which objects are building material and which are rocks—the operator helps the robot make sense of its real 3D environment before the building begins.

Giving robots the tools to deal with a changing environment is an important step toward allowing the AI to be truly independent, but it’s only an initial step. “We’re not letting it do absolutely everything,” said Quinn. “Our robot is good at moving an object from point A to point B, but it doesn’t know the overall plan.” Wilson adds that delegating environmental awareness and raw mechanical power to separate agents is the optimal relationship for a mixed human-robot construction team; it “lets humans do what they’re good at, while robots do what they do best.”

This story was updated on 4 September 2019. Continue reading →

Posted in Human Robots