Tag Archives: camera

#437182 MIT’s Tiny New Brain Chip Aims for AI ...

The human brain operates on roughly 20 watts of power (a third of a 60-watt light bulb) in a space the size of, well, a human head. The biggest machine learning algorithms use closer to a nuclear power plant’s worth of electricity and racks of chips to learn.

That’s not to slander machine learning, but nature may have a tip or two to improve the situation. Luckily, there’s a branch of computer chip design heeding that call. By mimicking the brain, super-efficient neuromorphic chips aim to take AI off the cloud and put it in your pocket.

The latest such chip is smaller than a piece of confetti and has tens of thousands of artificial synapses made out of memristors—chip components that can mimic their natural counterparts in the brain.

In a recent paper in Nature Nanotechnology, a team of MIT scientists say their tiny new neuromorphic chip was used to store, retrieve, and manipulate images of Captain America’s Shield and MIT’s Killian Court. Whereas images stored with existing methods tended to lose fidelity over time, the new chip’s images remained crystal clear.

“So far, artificial synapse networks exist as software. We’re trying to build real neural network hardware for portable artificial intelligence systems,” Jeehwan Kim, associate professor of mechanical engineering at MIT said in a press release. “Imagine connecting a neuromorphic device to a camera on your car, and having it recognize lights and objects and make a decision immediately, without having to connect to the internet. We hope to use energy-efficient memristors to do those tasks on-site, in real-time.”

A Brain in Your Pocket
Whereas the computers in our phones and laptops use separate digital components for processing and memory—and therefore need to shuttle information between the two—the MIT chip uses analog components called memristors that process and store information in the same place. This is similar to the way the brain works and makes memristors far more efficient. To date, however, they’ve struggled with reliability and scalability.

To overcome these challenges, the MIT team designed a new kind of silicon-based, alloyed memristor. Ions flowing in memristors made from unalloyed materials tend to scatter as the components get smaller, meaning the signal loses fidelity and the resulting computations are less reliable. The team found an alloy of silver and copper helped stabilize the flow of silver ions between electrodes, allowing them to scale the number of memristors on the chip without sacrificing functionality.

While MIT’s new chip is promising, there’s likely a ways to go before memristor-based neuromorphic chips go mainstream. Between now and then, engineers like Kim have their work cut out for them to further scale and demonstrate their designs. But if successful, they could make for smarter smartphones and other even smaller devices.

“We would like to develop this technology further to have larger-scale arrays to do image recognition tasks,” Kim said. “And some day, you might be able to carry around artificial brains to do these kinds of tasks, without connecting to supercomputers, the internet, or the cloud.”

Special Chips for AI
The MIT work is part of a larger trend in computing and machine learning. As progress in classical chips has flagged in recent years, there’s been an increasing focus on more efficient software and specialized chips to continue pushing the pace.

Neuromorphic chips, for example, aren’t new. IBM and Intel are developing their own designs. So far, their chips have been based on groups of standard computing components, such as transistors (as opposed to memristors), arranged to imitate neurons in the brain. These chips are, however, still in the research phase.

Graphics processing units (GPUs)—chips originally developed for graphics-heavy work like video games—are the best practical example of specialized hardware for AI and were heavily used in this generation of machine learning early on. In the years since, Google, NVIDIA, and others have developed even more specialized chips that cater more specifically to machine learning.

The gains from such specialized chips are already being felt.

In a recent cost analysis of machine learning, research and investment firm ARK Invest said cost declines have far outpaced Moore’s Law. In a particular example, they found the cost to train an image recognition algorithm (ResNet-50) went from around $1,000 in 2017 to roughly $10 in 2019. The fall in cost to actually run such an algorithm was even more dramatic. It took $10,000 to classify a billion images in 2017 and just $0.03 in 2019.

Some of these declines can be traced to better software, but according to ARK, specialized chips have improved performance by nearly 16 times in the last three years.

As neuromorphic chips—and other tailored designs—advance further in the years to come, these trends in cost and performance may continue. Eventually, if all goes to plan, we might all carry a pocket brain that can do the work of today’s best AI.

Image credit: Peng Lin Continue reading

Posted in Human Robots

#437150 AI Is Getting More Creative. But Who ...

Creativity is a trait that makes humans unique from other species. We alone have the ability to make music and art that speak to our experiences or illuminate truths about our world. But suddenly, humans’ artistic abilities have some competition—and from a decidedly non-human source.

Over the last couple years there have been some remarkable examples of art produced by deep learning algorithms. They have challenged the notion of an elusive definition of creativity and put into perspective how professionals can use artificial intelligence to enhance their abilities and produce beyond the known boundaries.

But when creativity is the result of code written by a programmer, using a format given by a software engineer, featuring private and public datasets, how do we assign ownership of AI-generated content, and particularly that of artwork? McKinsey estimates AI will annually generate value of $3.5 to $5.8 trillion across various sectors.

In 2018, a portrait that was christened Edmond de Belamy was made in a French art collective called Obvious. It used a database with 15,000 portraits from the 1300s to the 1900s to train a deep learning algorithm to produce a unique portrait. The painting sold for $432,500 in a New York auction. Similarly, a program called Aiva, trained on thousands of classical compositions, has released albums whose pieces are being used by ad agencies and movies.

The datasets used by these algorithms were different, but behind both there was a programmer who changed the brush strokes or musical notes into lines of code and a data scientist or engineer who fitted and “curated” the datasets to use for the model. There could also have been user-based input, and the output may be biased towards certain styles or unintentionally infringe on similar pieces of art. This shows that there are many collaborators with distinct roles in producing AI-generated content, and it’s important to discuss how they can protect their proprietary interests.

A perspective article published in Nature Machine Intelligence by Jason K. Eshraghian in March looks into how AI artists and the collaborators involved should assess their ownership, laying out some guiding principles that are “only applicable for as long as AI does not have legal parenthood, the way humans and corporations are accorded.”

Before looking at how collaborators can protect their interests, it’s useful to understand the basic requirements of copyright law. The artwork in question must be an “original work of authorship fixed in a tangible medium.” Given this principle, the author asked whether it’s possible for AI to exercise creativity, skill, or any other indicator of originality. The answer is still straightforward—no—or at least not yet. Currently, AI’s range of creativity doesn’t exceed the standard used by the US Copyright Office, which states that copyright law protects the “fruits of intellectual labor founded in the creative powers of the mind.”

Due to the current limitations of narrow AI, it must have some form of initial input that helps develop its ability to create. At the moment AI is a tool that can be used to produce creative work in the same way that a video camera is a tool used to film creative content. Video producers don’t need to comprehend the inner workings of their cameras; as long as their content shows creativity and originality, they have a proprietary claim over their creations.

The same concept applies to programmers developing a neural network. As long as the dataset they use as input yields an original and creative result, it will be protected by copyright law; they don’t need to understand the high-level mathematics, which in this case are often black box algorithms whose output it’s impossible to analyze.

Will robots and algorithms eventually be treated as creative sources able to own copyrights? The author pointed to the recent patent case of Warner-Lambert Co Ltd versus Generics where Lord Briggs, Justice of the Supreme Court of the UK, determined that “the court is well versed in identifying the governing mind of a corporation and, when the need arises, will no doubt be able to do the same for robots.”

In the meantime, Dr. Eshraghian suggests four guiding principles to allow artists who collaborate with AI to protect themselves.

First, programmers need to document their process through online code repositories like GitHub or BitBucket.

Second, data engineers should also document and catalog their datasets and the process they used to curate their models, indicating selectivity in their criteria as much as possible to demonstrate their involvement and creativity.

Third, in cases where user data is utilized, the engineer should “catalog all runs of the program” to distinguish the data selection process. This could be interpreted as a way of determining whether user-based input has a right to claim the copyright too.

Finally, the output should avoid infringing on others’ content through methods like reverse image searches and version control, as mentioned above.

AI-generated artwork is still a very new concept, and the ambiguous copyright laws around it give a lot of flexibility to AI artists and programmers worldwide. The guiding principles Eshraghian lays out will hopefully shed some light on the legislation we’ll eventually need for this kind of art, and start an important conversation between all the stakeholders involved.

Image Credit: Wikimedia Commons Continue reading

Posted in Human Robots

#436263 Skydio 2 Review: This Is the Drone You ...

Let me begin this review by saying that the Skydio 2 is one of the most impressive robots that I have ever seen. Over the last decade, I’ve spent enough time around robots to have a very good sense of what kinds of things are particularly challenging for them, and to set my expectations accordingly. Those expectations include things like “unstructured environments are basically impossible” and “full autonomy is impractically expensive” and “robot videos rarely reflect reality.”

Skydio’s newest drone is an exception to all of this. It’s able to fly autonomously at speed through complex environments in challenging real-world conditions in a way that’s completely effortless and stress-free for the end user, allowing you to capture the kind of video that would be otherwise impossible, even (I’m guessing) for professional drone pilots. When you see this technology in action, it’s (almost) indistinguishable from magic.

Skydio 2 Price
To be clear, the Skydio 2 is not without compromises, and the price of $999 (on pre-order with delivery of the next batch expected in spring of 2020) requires some justification. But the week I’ve had with this drone has left me feeling like its fundamental autonomous capability is so far beyond just about anything that I’ve ever experienced that I’m questioning why I would every fly anything else ever again.

We’ve written extensively about Skydio, beginning in early 2016 when the company posted a video of a prototype drone dodging trees while following a dude on a bike. Even three years ago, Skydio’s tech was way better than anything we’d seen outside of a research lab, and in early 2018, they introduced their first consumer product, the Skydio R1. A little over a year later, Skydio has introduced the Skydio 2, which is smaller, smarter, and much more affordable. Here’s an overview video just to get you caught up:

Skydio sent me a Skydio 2 review unit last week, and while I’m reasonably experienced with drones in general, this is the first time I’ve tried a Skydio drone in person. I had a pretty good idea what to expect, and I was absolutely blown away. Like, I was giggling to myself while running through the woods as the drone zoomed around, deftly avoiding trees and keeping me in sight. Robots aren’t supposed to be this good.

A week is really not enough time to explore everything that the Skydio can do, especially Thanksgiving week in Washington, D.C. (a no-fly zone) in early winter. But I found a nearby state park in which I could legally and safely fly the drone, and I did my best to put the Skydio 2 through its paces.

Note: Throughout this review, we’ve got a bunch of GIFs to help illustrate different features of the drone. To fit them all in, these GIFs had to be heavily compressed. Underneath each GIF is a timestamped link to this YouTube video (also available at the bottom of the post), which you can click on to see the an extended cut of the original 4K 30 fps footage. And there’s a bunch of interesting extra video in there as well.

Skydio 2 Specs

Photo: Evan Ackerman/IEEE Spectrum

The Skydio 2 is primarily made out of magnesium, which (while light) is both heavier and more rigid and durable than plastic. The offset props (the back pair are above the body, and the front pair are below) are necessary to maintain the field of view of the navigation cameras.

The Skydio 2 both looks and feels like a well-designed and carefully thought-out drone. It’s solid, and a little on the heavy side as far as drones go—it’s primarily made out of magnesium, which (while light) is both heavier and more rigid and durable than plastic. The blue and black color scheme is far more attractive than you typically see with drones.

Photo: Evan Ackerman/IEEE Spectrum

To detect and avoid obstacles, the Skydio 2 uses an array of six 4K hemispherical cameras that feed data into an NVIDIA Jetson TX2 at 30 fps, with the drone processing a million points in 3D space per second to plan the safest path.

The Skydio 2 is built around an array of six hemispherical obstacle-avoidance cameras and the NVIDIA Jetson TX2 computing module that they’re connected to. This defines the placement of the gimbal, the motors and props, and the battery, since all of this stuff has to be as much as possible out of the view of the cameras in order for the drone to effectively avoid obstacles in any direction.

Without the bottom-mounted battery attached, the drone is quite flat. The offset props (the back pair are above the body, and the front pair are below) are necessary to maintain the field of view of the obstacle-avoidance cameras. These hemispherical cameras are on the end of each of the prop arms as well as above and below the body of the drone. They look awfully exposed, even though each is protected from ground contact by a little fin. You need to make sure these cameras are clean and smudge-free, and Skydio includes a cleaning cloth for this purpose. Underneath the drone there are slots for microSD cards, one for recording from the camera and a second one that the drone uses to store data. The attention to detail extends to the SD card insertion, which has a sloped channel that guides the card securely into its slot.

Once you snap the battery in, the drone goes from looking streamlined to looking a little chubby. Relative to other drones, the battery almost seems like an afterthought, like Skydio designed the drone and then remembered, “oops we have to add a battery somewhere, let’s just kludge it onto the bottom.” But again, the reason for this is to leave room inside the body for the NVIDIA TX2, while making sure that the battery stays out of view of the obstacle avoidance cameras.

The magnetic latching system for the battery is both solid and satisfying. I’m not sure why it’s necessary, strictly speaking, but I do like it, and it doesn’t seem like the battery will fly off even during the most aggressive maneuvers. Each battery includes an LED array that will display its charge level in 25 percent increments, as well as a button that you push to turn the drone on and off. Charging takes place via a USB-C port in the top of the drone, which I don’t like, because it means that the batteries can’t be charged on their own (like the Parrot Anafi’s battery), and that you can’t charge one battery while flying with another, like basically every other drone ever. A separate battery charger that will charge two at once is available from Skydio for an eyebrow-raising $129.

I appreciate that all of Skydio’s stuff (batteries, controller, and beacon) charges via USB-C, though. The included USB-C adapter with its beefy cable will output at up to 65 watts, which’ll charge a mostly depleted battery in under an hour. The drone turns itself on while charging, which seems unnecessary.

Photo: Evan Ackerman/IEEE Spectrum

The Skydio 2 is not foldable, making it not nearly as easy to transport as some other drones. But it does come with a nice case that mitigates this issue somewhat, and the drone plus two batteries end up as a passably flat package about the size of a laptop case.

The most obvious compromise that Skydio made with the Skydio 2 is that the drone is not foldable. Skydio CEO Adam Bry told us that adding folding joints to the arms of the Skydio 2 would have made calibrating all six cameras a nightmare and significantly impacted performance. This makes complete sense, of course, but it does mean that the Skydio 2 is not nearly as easy to transport as some other drones.

Photo: Evan Ackerman/IEEE Spectrum

Folded and unfolded: The Skydio 2 compared to the Parrot Anafi (upper left) and the DJI Mavic Pro (upper right).

The Skydio 2 does come with a very nice case that mitigates this issue somewhat, and the drone plus two batteries end up as a passably flat package about the size of a laptop case. Still, it’s just not as convenient to toss into a backpack as my Anafi, although the Mavic Mini might be even more portable.

Photo: Evan Ackerman/IEEE Spectrum

While the Skydio 2’s case is relatively compact, the non-foldable drone is overall a significantly larger package than the Parrot Anafi.

The design of the drone leads to some other compromises as well. Since landing gear would, I assume, occlude the camera system, the drone lands directly on the bottom of its battery pack, which has a slightly rubberized pad about the size of a playing card. This does’t feel particularly stable unless you end up on a very flat surface, and made me concerned for the exposed cameras underneath the drone as well as the lower set of props. I’d recommend hand takeoffs and landings—more on those later.

Skydio 2 Camera System

Photo: Evan Ackerman/IEEE Spectrum

The Skydio 2’s primary camera is a Sony IMX577 1/2.3″ 12.3-megapixel CMOS sensor. It’s mounted to a three-axis gimbal and records 4K video at 60 fps, or 1080p video at 120 fps.

The Skydio 2 comes with a three-axis gimbal supporting a 12-megapixel camera, just enough to record 4K video at 60 fps, or 1080p video at 120 fps. Skydio has provided plenty of evidence that its imaging system is at least as good if not better than other drone cameras. Tested against my Mavic Pro and Parrot Anafi, I found no reason to doubt that. To be clear, I didn’t do exhaustive pixel-peeping comparisons between them, you’re just getting my subjective opinion that the Skydio 2 has a totally decent camera that you won’t be disappointed with. I will say that I found the HDR photo function to be not all that great under the few situations in which I tested it—after looking at a few muddy sunset shots, I turned it off and was much happier.

Photo: Evan Ackerman/IEEE Spectrum

The Skydio 2’s 12-megapixel camera is solid, although we weren’t impressed with the HDR option.

The video stabilization is fantastic, to the point where watching the video footage can be underwhelming because it doesn’t reflect the motion of the drone. I almost wish there was a way to change to unstabilized (or less-stabilized) video so that the viewer could get a little more of a wild ride. Or, ideally, there’d be a way for the drone to provide you with a visualization of what it was doing using the data collected by its cameras. That’s probably wishful thinking, though. The drone itself doesn’t record audio because all you’d get would be an annoying buzz, but the app does record audio, so the audio from your phone gets combined with the drone video. Don’t expect great quality, but it’s better than nothing.

Skydio 2 App
The app is very simple compared to every other drone app I’ve tried, and that’s a good thing. Here’s what it looks like:

Image: Skydio

Trackable subjects get a blue “+” sign over them, and if you tap them, the “+” turns into a spinny blue circle. Once you’ve got a subject selected, you can choose from a variety of cinematic skills that the drone will execute while following you.

You get the controls that you need and the information that you need, and nothing else. Manual flight with the on-screen buttons works adequately, and the double-tap to fly function on the phone works surprisingly well, making it easy to direct the drone to a particular spot above the ground.

The settings menus are limited but functional, allowing you to change settings for the camera and a few basic tweaks for controlling the drone. One unique setting to the Skydio 2 is the height floor—since the drone only avoids static obstacles, you can set it to maintain a height of at least 8 feet above the ground while flying autonomously to make sure that if you’re flying around other people, it won’t run into anyone who isn’t absurdly tall and therefore asking for it.

Trackable subjects get a blue “+” sign over them in the app, and if you tap them, the “+” turns into a spinny blue circle. Once you’ve got a subject selected, you can choose from a variety of cinematic skills that the drone will execute while following you, and in addition, you can select “one-shot” skills that involve the drone performing a specific maneuver before returning to the previously selected cinematic skill. For example, you can tell the drone to orbit around you, and then do a “rocket” one-shot where it’ll fly straight up above you (recording the whole time, of course), before returning to its orbiting.

After you’re done flying, you can scroll through your videos and easily clip out excerpts from them and save them to your phone for sharing. Again, it’s a fairly simple interface without a lot of options. You could call it limited, I guess, but I appreciate that it just does a few things that you care about and otherwise doesn’t clutter itself up.

The real limitation of the app is that it uses Wi-Fi to connect to the Skydio 2, which restricts the range. To fly much beyond a hundred meters or so, you’ll need to use the controller or beacon instead.

Skydio 2 Controller and Beacon

Photo: Evan Ackerman/IEEE Spectrum

While the Skydio 2 controller provides a better hands-on flight experience than with the phone, plus an extended range of up to 3.5 km, more experienced pilots may find manual control a bit frustrating, because the underlying autonomy will supersede your maneuvers when you start getting close to objects.

I was looking forward to using the controller, because with every other drone I’ve had, the precision that a physically controller provides is, I find, mandatory for a good flying experience and to get the photos and videos that you want. With Skydio 2, that’s all out the window. It’s not that the controller is useless or anything, it’s just that because the drone tracks you and avoids obstacles on its own, that level of control precision becomes largely unnecessary.

The controller itself is perfectly fine. It’s a rebranded Parrot Skycontroller3, which is the same as the one that you get with a Parrot Anafi. It’s too bad that the sticks don’t unscrew to make it a little more portable, and overall it’s functional rather than fancy, but it feels good to use and includes a sizeable antenna that makes a significant difference to the range that you get (up to 3.5 kilometers).

You definitely get a better hands-on flight experience with the controller than with the phone, so if you want to (say) zip the drone around some big open space for fun, it’s good for that. And it’s nice to be able to hand the controller to someone who’s never flown a drone before and let them take it for a spin without freaking out about them crashing it the whole time. For more experienced pilots, though, the controller is ultimately just a bit frustrating, because the underlying autonomy will supersede your control when you start getting close to objects, which (again) limits how useful the controller is relative to your phone.

I do still prefer the controller over the phone, but I’m not sure that it’s worth the extra $150, unless you plan to fly the Skydio 2 at very long distances or primarily in manual mode. And honestly, if either of those two things are your top priority, the Skydio 2 is probably not the drone for you.

Photo: Evan Ackerman/IEEE Spectrum

The Skydio 2 beacon uses GPS tracking to help the drone follow you, extending range up to 1.5 km. You can also fly the with the beacon alone, no phone necessary.

The purpose of the beacon, according to Skydio, is to give the drone a way of tracking you if it can’t see you, which can happen, albeit infrequently. My initial impression of the beacon was that it was primarily useful as a range-extending bridge between my phone and the drone. But I accidentally left my phone at home one day (oops) and had to fly the drone with only the beacon, and it was a surprisingly decent experience. The beacon allows for full manual control of a sort—you can tap different buttons to rotate, fly forward, and ascend or descend. This is sufficient for takeoff, landing, to make sure that the drone is looking at you when you engage visual tracking, and to rescue it if it gets trapped somewhere.

The rest of the beacon’s control functions are centered around a few different tracking modes, and with these, it works just about as well as your phone. You have fewer options overall, but all the basic stuff is there with just a few intuitive button clicks, including tracking range and angle. If you’re willing to deal with this relatively minor compromise, it’s nice to not have your phone available for other things rather than being monopolized by the drone.

Skydio 2 In Flight

GIF: Evan Ackerman/IEEE Spectrum

Hand takeoffs are simple and reliable.
Click here for a full resolution clip.

Starting up the Skydio 2 doesn’t require any kind of unusual calibration steps or anything like that. It prefers to be kept still, but you can start it up while holding it, it’ll just take a few seconds longer to tell you that it’s ready to go. While the drone will launch from any flat surface with significant clearance around it (it’ll tell you if it needs more room), the small footprint of the battery means that I was more comfortable hand launching it. This is not a “throw” launch; you just let the drone rest on your palm, tell it to take off, and then stay still while it gets its motors going and then gently lifts off. The lift off is so gentle that you have to be careful not to pull your hand away too soon—I did that once and the drone, being not quite ready, dropped towards the ground, but managed to recover without much drama.

GIF: Evan Ackerman/IEEE Spectrum

Hand landings always look scary, but the Skydio 2 is incredibly gentle. After trying this once, it became the only way I ever landed the drone.
Click here for a full resolution clip.

Catching the drone for landing is perhaps very slightly more dangerous, but not any more difficult. You put the drone above and in front of you facing away, tell it to land in the app or with the beacon, and then put your hand underneath it to grasp it as it slowly descends. It settles delicately and promptly turns itself off. Every drone should land this way. The battery pack provides a good place to grip, although you do have to be mindful of the forward set of props, which (since they’re the pair that are beneath the body of drone) are quite close to your fingers. You’ll certainly be mindful after you catch a blade with your fingers once. Which I did. For the purposes of this review and totally not by accident. No damage, for the record.

Photo: Evan Ackerman/IEEE Spectrum

You won’t be disappointed with the Skydio 2’s in-flight performance, unless you’re looking for a dedicated racing drone.

In normal flight, the Skydio 2 performs as well as you’d expect. It’s stable and manages light to moderate wind without any problems, although I did notice some occasional lateral drifting when the drone should have been in a stationary hover. While the controller gains are adjustable, the Skydio 2 isn’t quite as aggressive in flight as my Mavic Pro on Sport Mode, but again, if you’re looking for a high-speed drone, that’s really not what the Skydio is all about.

The Skydio 2 is substantially louder than my Anafi, although the Anafi is notably quiet for a drone. It’s not annoying to hear (not a high-pitched whine), but you can hear it from a ways away, and farther away than my Mavic Pro. I’m not sure whether that’s because of the absolute volume or the volume plus the pitch. In some ways, this is a feature, since you can hear the drone following you even if you’re not looking at it, you just need to be aware of the noise it makes when you’re flying it around people.

Obstacle Avoidance
The primary reason Skydio 2 is the drone that you want to fly is because of its autonomous subject tracking and obstacle avoidance. Skydio’s PR videos make this capability look almost too good, and since I hadn’t tried out one of their drones before, the first thing I did with it was exactly what you’d expect: attempt to fly it directly into the nearest tree.

GIF: Evan Ackerman/IEEE Spectrum

The Skydio 2 deftly slides around trees and branches. The control inputs here were simple “forward” or “turn,” all obstacle avoidance is autonomous.
Click here for a full resolution clip.

And it just won’t do it. It slows down a bit, and then slides right around one tree after another, going over and under and around branches. I pointed the drone into a forest and just held down “forward” and away it went, without any fuss, effortlessly ducking and weaving its way around. Of course, it wasn’t effortless at all—six 4K cameras were feeding data into the NVIDIA TX2 at 30 fps, and the drone was processing a million points in 3D space per second to plan the safest path while simultaneously taking into account where I wanted it to go. I spent about 10 more minutes doing my level best to crash the drone into anything at all using a flying technique probably best described as “reckless,” but the drone was utterly unfazed. It’s incredible.

What knocked my socks off was telling the drone to pass through treetops—in the clip below, I’m just telling the drone to fly straight down. Watch as it weaves its way through gaps between the branches:

GIF: Evan Ackerman/IEEE Spectrum

The result of parking the Skydio 2 above some trees and holding “down” on the controller is this impressive fully autonomous descent through the branches.
Click here for a full resolution clip.

Here’s one more example, where I sent the drone across a lake and started poking around in a tree. Sometimes the Skydio 2 isn’t sure where you want it to go, and you have to give it a little bit of a nudge in a clear direction, but that’s it.

GIF: Evan Ackerman/IEEE Spectrum

In obstacle-heavy environments, the Skydio 2 prudently slows down, but it can pick its way through almost anything that it can see.
Click here for a full resolution clip.

It’s important to keep in mind that all of the Skydio 2’s intelligence is based on vision. It uses cameras to see the world, which means that it has similar challenges as your eyes do. Specifically, Skydio warns against flying in the following conditions:

Skydio 2 can’t see certain visually challenging obstacles. Do not fly around thin branches, telephone or power lines, ropes, netting, wires, chain link fencing or other objects less than ½ inch in diameter.
Do not fly around transparent surfaces like windows or reflective surfaces like mirrors greater than 60 cm wide.
When the sun is low on the horizon, it can temporarily blind Skydio 2’s cameras depending on the angle of flight. Your drone may be cautious or jerky when flying directly toward the sun.

Basically, if you’d have trouble seeing a thing, or seeing under some specific flight conditions, then the Skydio 2 almost certainly will also. It gets even more problematic when challenging obstacles are combined with challenging flight conditions, which is what I’m pretty sure led to the only near-crash I had with the drone. Here’s a video:

GIF: Evan Ackerman/IEEE Spectrum

Flying around very thin branches and into the sun can cause problems for the Skydio 2’s obstacle avoidance.
Click here for a full resolution clip.

I had the Skydio 2 set to follow me on my bike (more about following and tracking in a bit). It was mid afternoon, but since it’s late fall here in Washington, D.C., the sun doesn’t get much higher than 30 degrees above the horizon. Late fall also means that most of the deciduous trees have lost their leaves, and so there are a bunch of skinny branches all over the place. The drone was doing a pretty good job of following me along the road at a relatively slow speed, and then it clipped the branch that you can just barely see in the video above. It recovered in an acrobatic maneuver that has been mostly video-stabilized out, and resumed tracking me before I freaked and told it to land. You can see another example here, where the drone (again) clips a branch that has the sun behind it, and this clip shows me stopping my bike before the drone runs into another branch in a similar orientation. As the video shows, it’s very hard to see the branches until it’s too late.

As far as I can tell, the drone is no worse for wear from any of this, apart from a small nick in one of the props. But, this is a good illustration of a problematic situation for the Skydio 2: flying into a low sun angle around small bare branches. Should I not have been flying the drone in this situation? It’s hard to say. These probably qualify as “thin branches,” although there was plenty of room along with middle of the road. There is an open question with the Skydio 2 as to exactly how much responsibility the user should have about when and where it’s safe to fly—for branches, how thin is too thin? How low can the sun be? What if the branches are only kinda thin and the sun is only kinda low, but it’s also a little windy? Better to be safe than sorry, of course, but there’s really no way for the user (or the drone) to know what it can’t handle until it can’t handle it.

Edge cases like these aside, the obstacle avoidance just works. Even if you’re not deliberately trying to fly into branches, it’s keeping a lookout for you all the time, which means that flying the drone goes from somewhat stressful to just pure fun. I can’t emphasize enough how amazing it is to be able to fly without worrying about running into things, and how great it feels to be able to hand the controller to someone who’s never flown a drone before and say, with complete confidence, “go ahead, fly it around!”

Skydio 2 vs. DJI Mavic

Photo: Evan Ackerman/IEEE Spectrum

Both the Skydio 2 and many models of DJI’s Mavic use visual obstacle avoidance, but the Skydio 2 is so much more advanced that you can’t really compare the two systems.

It’s important to note that there’s a huge difference between the sort of obstacle avoidance that you get with a DJI Mavic, and the sort of obstacle avoidance that you get with the Skydio 2. The objective of the Mavic’s obstacle avoidance is really there to prevent you from accidentally running into things, and in that capacity, it usually works. But there are two things to keep in mind here—first, not running into things is not the same as avoiding things, because avoiding things means planning several steps ahead, not just one step.

Second, there’s the fact that the Mavic’s obstacle detection only works most of the time. Fundamentally, I don’t trust my Mavic Pro, because sometimes the safety system doesn’t kick in for whatever reason and the drone ends up alarmingly close to something. And that’s actually fine, because with the Mavic, I expect to be piloting it. It’s for this same reason that I don’t care that my Parrot Anafi doesn’t have obstacle avoidance at all: I’m piloting it anyway, and I’m a careful pilot, so it just doesn’t matter. The Skydio 2 is totally and completely different. It’s in a class by itself, and you can’t compare what it can do to what anything else out there right now. Period.

Skydio 2 Tracking
Skydio’s big selling point on the Skydio 2 is that it’ll autonomously track you while avoiding obstacles. It does this visually, by watching where you go, predicting your future motion, and then planning its own motion to keep you in frame. The works better than you might expect, in that it’s really very good at not losing you. Obviously, the drone prioritizes not running into stuff over tracking you, which means that it may not always be where you feel like it should be. It’s probably trying to get there, but in obstacle dense environments, it can take some creative paths.

Having said that, I found it to be very consistent with keeping me in the frame, and I only managed to lose it when changing direction while fully occluded by an obstacle, or while it was executing an avoidance maneuver that was more dynamic than normal. If you deliberately try to hide from the drone it’s not that hard to do so if there are enough obstacles around, but I didn’t find the tracking to be something that I had to worry about it most cases. When tracking does fail and you’re not using the beacon, the drone will come to a hover. It won’t try and find you, but it will reacquire you if you get back into its field of view.

The Skydio 2 had no problem tracking me running through fairly dense trees:

GIF: Evan Ackerman/IEEE Spectrum

The Skydio 2 had no problem chasing me around through these trees, even while I was asking it to continually change its tracking angle.
Click here for a full resolution clip.

It also managed to keep up with me as I rode my bike along a tree-lined road:

GIF: Evan Ackerman/IEEE Spectrum

The Skydio 2 is easily fast enough to keep up with me on a bike, even while avoiding tree branches.
Click here for a full resolution clip.

It lost me when I asked it to follow very close behind me as I wove through some particularly branch-y trees, but it fails more or less gracefully by just sort of nope-ing out of situations when they start to get bad and coming to a hover somewhere safe.

GIF: Evan Ackerman/IEEE Spectrum

The Skydio 2 knows better than to put itself into situations that it can’t handle, and will bail to a safe spot if things get too complicated.
Click here for a full resolution clip.

After a few days of playing with the drone, I started to get to the point where I could set it to track me and then just forget about it while I rode my bike or whatever, as opposed to constantly turning around to make sure it was still behind me, which is what I was doing initially. It’s a level of trust that I don’t think would be possible with any other drone.

Should You Buy a Skydio 2?

Photo: Evan Ackerman/IEEE Spectrum

We think the Skydio 2 is fun and relaxing to fly, with unique autonomous intelligence that makes it worth the cost.

In case I haven’t said it often enough in this review, the Skydio 2 is an incredible piece of technology. As far as I know (as a robotics journalist, mind you), this represents the state of the art in commercial drone autonomy, and quite possibly the state of the art in drone autonomy, period. And it’s available for $999, which is expensive, but less money than a Mavic Pro 2. If you’re interested in a new drone, you should absolutely consider the Skydio 2.

There are some things to keep in mind—battery life is a solid but not stellar 20 minutes. Extra batteries are expensive at $99 each (the base kit includes just one). The controller and the beacon are also expensive, at $150 each. And while I think the Skydio 2 is definitely the drone you want to fly, it may not be the drone you want to travel with, since it’s bulky compared to other options.

But there’s no denying the fact that the experience is uniquely magical. Once you’ve flown the Skydio 2, you won’t want to fly anything else. This drone makes it possible to get pictures and videos that would be otherwise impossible, and you can do it completely on your own. You can trust the drone to do what it promises, as long as you’re mindful of some basic and common sense safety guidelines. And we’ve been told that the drone is only going to get smarter and more capable over time.

If you buy a Skydio 2, it comes with the following warranty from Skydio:

“If you’re operating your Skydio 2 within our Safe Flight guidelines, and it crashes, we’ll repair or replace it for free.”

Skydio trusts their drone to go out into a chaotic and unstructured world and dodge just about anything that comes its way. And after a week with this drone, I can see how they’re able to offer this kind of guarantee. This is the kind of autonomy that robots have been promising for years, and the Skydio 2 makes it real.

Detailed technical specifications are available on Skydio’s website, and if you have any questions, post a comment—we’ve got this drone for a little while longer, and I’d be happy to try out (nearly) anything with it.

Skydio 2 Review Video Highlights
This video is about 7 minutes of 4K, 30 fps footage directly from the Skydio 2. The only editing I did was cutting clips together, no stabilization or color correcting or anything like that. The drone will record in 4K 60 fps, so it gets smoother than this, but I, er, forgot to change the setting.

[ Skydio ] Continue reading

Posted in Human Robots

#436220 How Boston Dynamics Is Redefining Robot ...

Gif: Bob O’Connor/IEEE Spectrum

With their jaw-dropping agility and animal-like reflexes, Boston Dynamics’ bioinspired robots have always seemed to have no equal. But that preeminence hasn’t stopped the company from pushing its technology to new heights, sometimes literally. Its latest crop of legged machines can trudge up and down hills, clamber over obstacles, and even leap into the air like a gymnast. There’s no denying their appeal: Every time Boston Dynamics uploads a new video to YouTube, it quickly racks up millions of views. These are probably the first robots you could call Internet stars.

Spot

Photo: Bob O’Connor

84 cm HEIGHT

25 kg WEIGHT

5.76 km/h SPEED

SENSING: Stereo cameras, inertial measurement unit, position/force sensors

ACTUATION: 12 DC motors

POWER: Battery (90 minutes per charge)

Boston Dynamics, once owned by Google’s parent company, Alphabet, and now by the Japanese conglomerate SoftBank, has long been secretive about its designs. Few publications have been granted access to its Waltham, Mass., headquarters, near Boston. But one morning this past August, IEEE Spectrum got in. We were given permission to do a unique kind of photo shoot that day. We set out to capture the company’s robots in action—running, climbing, jumping—by using high-speed cameras coupled with powerful strobes. The results you see on this page: freeze-frames of pure robotic agility.

We also used the photos to create interactive views, which you can explore online on our Robots Guide. These interactives let you spin the robots 360 degrees, or make them walk and jump on your screen.

Boston Dynamics has amassed a minizoo of robotic beasts over the years, with names like BigDog, SandFlea, and WildCat. When we visited, we focused on the two most advanced machines the company has ever built: Spot, a nimble quadruped, and Atlas, an adult-size humanoid.

Spot can navigate almost any kind of terrain while sensing its environment. Boston Dynamics recently made it available for lease, with plans to manufacture something like a thousand units per year. It envisions Spot, or even packs of them, inspecting industrial sites, carrying out hazmat missions, and delivering packages. And its YouTube fame has not gone unnoticed: Even entertainment is a possibility, with Cirque du Soleil auditioning Spot as a potential new troupe member.

“It’s really a milestone for us going from robots that work in the lab to these that are hardened for work out in the field,” Boston Dynamics CEO Marc Raibert says in an interview.

Atlas

Photo: Bob O’Connor

150 cm HEIGHT

80 kg WEIGHT

5.4 km/h SPEED

SENSING: Lidar and stereo vision

ACTUATION: 28 hydraulic actuators

POWER: Battery

Our other photographic subject, Atlas, is Boston Dynamics’ biggest celebrity. This 150-centimeter-tall (4-foot-11-inch-tall) humanoid is capable of impressive athletic feats. Its actuators are driven by a compact yet powerful hydraulic system that the company engineered from scratch. The unique system gives the 80-kilogram (176-pound) robot the explosive strength needed to perform acrobatic leaps and flips that don’t seem possible for such a large humanoid to do. Atlas has inspired a string of parody videos on YouTube and more than a few jokes about a robot takeover.

While Boston Dynamics excels at making robots, it has yet to prove that it can sell them. Ever since its founding in 1992 as a spin-off from MIT, the company has been an R&D-centric operation, with most of its early funding coming from U.S. military programs. The emphasis on commercialization seems to have intensified after the acquisition by SoftBank, in 2017. SoftBank’s founder and CEO, Masayoshi Son, is known to love robots—and profits.

The launch of Spot is a significant step for Boston Dynamics as it seeks to “productize” its creations. Still, Raibert says his long-term goals have remained the same: He wants to build machines that interact with the world dynamically, just as animals and humans do. Has anything changed at all? Yes, one thing, he adds with a grin. In his early career as a roboticist, he used to write papers and count his citations. Now he counts YouTube views.

In the Spotlight

Photo: Bob O’Connor

Boston Dynamics designed Spot as a versatile mobile machine suitable for a variety of applications. The company has not announced how much Spot will cost, saying only that it is being made available to select customers, which will be able to lease the robot. A payload bay lets you add up to 14 kilograms of extra hardware to the robot’s back. One of the accessories that Boston Dynamics plans to offer is a 6-degrees-of-freedom arm, which will allow Spot to grasp objects and open doors.

Super Senses

Photo: Bob O’Connor

Spot’s hardware is almost entirely custom-designed. It includes powerful processing boards for control as well as sensor modules for perception. The ­sensors are located on the front, rear, and sides of the robot’s body. Each module consists of a pair of stereo cameras, a wide-angle camera, and a texture projector, which enhances 3D sensing in low light. The sensors allow the robot to use the navigation method known as SLAM, or simultaneous localization and mapping, to get around autonomously.

Stepping Up

Photo: Bob O’Connor

In addition to its autonomous behaviors, Spot can also be steered by a remote operator with a game-style controller. But even when in manual mode, the robot still exhibits a high degree of autonomy. If there’s an obstacle ahead, Spot will go around it. If there are stairs, Spot will climb them. The robot goes into these operating modes and then performs the related actions completely on its own, without any input from the operator. To go down a flight of stairs, Spot walks backward, an approach Boston Dynamics says provides greater stability.

Funky Feet

Gif: Bob O’Connor/IEEE Spectrum

Spot’s legs are powered by 12 custom DC motors, each geared down to provide high torque. The robot can walk forward, sideways, and backward, and trot at a top speed of 1.6 meters per second. It can also turn in place. Other gaits include crawling and pacing. In one wildly popular YouTube video, Spot shows off its fancy footwork by dancing to the pop hit “Uptown Funk.”

Robot Blood

Photo: Bob O’Connor

Atlas is powered by a hydraulic system consisting of 28 actuators. These actuators are basically cylinders filled with pressurized fluid that can drive a piston with great force. Their high performance is due in part to custom servo valves that are significantly smaller and lighter than the aerospace models that Boston Dynamics had been using in earlier designs. Though not visible from the outside, the innards of an Atlas are filled with these hydraulic actuators as well as the lines of fluid that connect them. When one of those lines ruptures, Atlas bleeds the hydraulic fluid, which happens to be red.

Next Generation

Gif: Bob O’Connor/IEEE Spectrum

The current version of Atlas is a thorough upgrade of the original model, which was built for the DARPA Robotics Challenge in 2015. The newest robot is lighter and more agile. Boston Dynamics used industrial-grade 3D printers to make key structural parts, giving the robot greater strength-to-weight ratio than earlier designs. The next-gen Atlas can also do something that its predecessor, famously, could not: It can get up after a fall.

Walk This Way

Photo: Bob O’Connor

To control Atlas, an operator provides general steering via a manual controller while the robot uses its stereo cameras and lidar to adjust to changes in the environment. Atlas can also perform certain tasks autonomously. For example, if you add special bar-code-type tags to cardboard boxes, Atlas can pick them up and stack them or place them on shelves.

Biologically Inspired

Photos: Bob O’Connor

Atlas’s control software doesn’t explicitly tell the robot how to move its joints, but rather it employs mathematical models of the underlying physics of the robot’s body and how it interacts with the environment. Atlas relies on its whole body to balance and move. When jumping over an obstacle or doing acrobatic stunts, the robot uses not only its legs but also its upper body, swinging its arms to propel itself just as an athlete would.

This article appears in the December 2019 print issue as “By Leaps and Bounds.” Continue reading

Posted in Human Robots

#436180 Bipedal Robot Cassie Cal Learns to ...

There’s no particular reason why knowing how to juggle would be a useful skill for a robot. Despite this, robots are frequently taught how to juggle things. Blind robots can juggle, humanoid robots can juggle, and even drones can juggle. Why? Because juggling is hard, man! You have to think about a bunch of different things at once, and also do a bunch of different things at once, which this particular human at least finds to be overly stressful. While juggling may not stress robots out, it does require carefully coordinated sensing and computing and actuation, which means that it’s as good a task as any (and a more entertaining task than most) for testing the capabilities of your system.

UC Berkeley’s Cassie Cal robot, which consists of two legs and what could be called a torso if you were feeling charitable, has just learned to juggle by bouncing a ball on what would be her head if she had one of those. The idea is that if Cassie can juggle while balancing at the same time, she’ll be better able to do other things that require dynamic multitasking, too. And if that doesn’t work out, she’ll still be able to join the circus.

Cassie’s juggling is assisted by an external motion capture system that tracks the location of the ball, but otherwise everything is autonomous. Cassie is able to juggle the ball by leaning forwards and backwards, left and right, and moving up and down. She does this while maintaining her own balance, which is the whole point of this research—successfully executing two dynamic behaviors that may sometimes be at odds with one another. The end goal here is not to make a better juggling robot, but rather to explore dynamic multitasking, a skill that robots will need in order to be successful in human environments.

This work is from the Hybrid Robotics Lab at UC Berkeley, led by Koushil Sreenath, and is being done by Katherine Poggensee, Albert Li, Daniel Sotsaikich, Bike Zhang, and Prasanth Kotaru.

For a bit more detail, we spoke with Albert Li via email.

Image: UC Berkeley

UC Berkeley’s Cassie Cal getting ready to juggle.

IEEE Spectrum: What would be involved in getting Cassie to juggle without relying on motion capture?

Albert Li: Our motivation for starting off with motion capture was to first address the control challenge of juggling on a biped without worrying about implementing the perception. We actually do have a ball detector working on a camera, which would mean we wouldn’t have to rely on the motion capture system. However, we need to mount the camera in a way that it would provide the best upwards field of view, and we also have develop a reliable estimator. The estimator is particularly important because when the ball gets close enough to the camera, we actually can’t track the ball and have to assume our dynamic models describe its motion accurately enough until it bounces back up.

What keeps Cassie from juggling indefinitely?

There are a few factors that affect how long Cassie can sustain a juggle. While in simulation the paddle exhibits homogeneous properties like its stiffness and damping, in reality every surface has anisotropic contact properties. So, there are parts of the paddle which may be better for juggling than others (and importantly, react differently than modeled). These differences in contact are also exacerbated due to how the paddle is cantilevered when mounted on Cassie. When the ball hits these areas, it leads to a larger than expected error in a juggle. Due to the small size of the paddle, the ball may then just hit the paddle’s edge and end the juggling run. Over a very long run, this is a likely occurrence. Additionally, some large juggling errors could cause Cassie’s feet to slip slightly, which ends up changing the stable standing position over time. Since this version of the controller assumes Cassie is stationary, this change in position eventually leads to poor juggles and failure.

Would Cassie be able to juggle while walking (or hovershoe-ing)?

Walking (and hovershoe-ing) while juggling is a far more challenging problem and is certainly a goal for future research. Some of these challenges include getting the paddle to precise poses to juggle the ball while also moving to avoid any destabilizing effects of stepping incorrectly. The number of juggles per step of walking could also vary and make the mathematics of the problem more challenging. The controller goal is also more involved. While the current goal of the juggling controller is to juggle the ball to a static apex position, with a walking juggling controller, we may instead want to hit the ball forwards and also walk forwards to bounce it, juggle the ball along a particular path, etc. Solving such challenges would be the main thrusts of the follow-up research.

Can you give an example of a practical task that would be made possible by using a controller like this?

Studying juggling means studying contact behavior and leveraging our models of it to achieve a known objective. Juggling could also be used to study predictable post-contact flight behavior. Consider the scenario where a robot is attempting to make a catch, but fails, letting the ball to bounce off of its hand, and then recovering the catch. This behavior could also be intentional: It is often easier to first execute a bounce to direct the target and then perform a subsequent action. For example, volleyball players could in principle directly hit a spiked ball back, but almost always bump the ball back up and then return it.

Even beyond this motivating example, the kinds of models we employ to get juggling working are more generally applicable to any task that involves contact, which could include tasks besides bouncing like sliding and rolling. For example, clearing space on a desk by pushing objects to the side may be preferable than individually manipulating each and every object on it.

You mention collaborative juggling or juggling multiple balls—is that something you’ve tried yet? Can you talk a bit more about what you’re working on next?

We haven’t yet started working on collaborative or multi-ball juggling, but that’s also a goal for future work. Juggling multiple balls statically is probably the most reasonable next goal, but presents additional challenges. For instance, you have to encode a notion of juggling urgency (if the second ball isn’t hit hard enough, you have less time to get the first ball up before you get back to the second one).

On the other hand, collaborative human-robot juggling requires a more advanced decision-making framework. To get robust multi-agent juggling, the robot will need to employ some sort of probabilistic model of the expected human behavior (are they likely to move somewhere? Are they trying to catch the ball high or low? Is it safe to hit the ball back?). In general, developing such human models is difficult since humans are fairly unpredictable and often don’t exhibit rational behavior. This will be a focus of future work.

[ Hybrid Robotics Lab ] Continue reading

Posted in Human Robots