Computer vision technology is complementing GPS sensors in visually smarter drones in a quest to have autonomous features like object tracking, environment sensing, collisions avoiding and more. At the same time, 4K video, now a must-have feature, and other functions like 3D depth map creation pose significant computational challenges for the high-quality image- and video-processing pipeline in drone electronics.
So, while computer vision clearly dominates the design work in flying robots, it’s still algorithmically challenging and requires greater intelligence and analytical capabilities for drones to know what they are filming. In fact, software algorithms are continuously improving to overcome the computational complexity in vision applications, and that calls for a flexible hardware platform for leveraging these improvements.
Moreover, computer vision can make up for the lack of training or even understanding of the rules of airspace, so that drones don’t knock out power for hundreds of West Hollywood residents or collides with helicopter. The twisted portrayal of delivery drones in Audi’s commercial further underlines the hidden dangers in the brave new world of these flying machines in the sky.
The darker side of drone technology takeoff clearly highlights two imperatives. First, drones need to further boost the autonomous features so that they don’t crash too often and become a danger to society. Second, drones are required to increase their functionality to open up new opportunities in the commercial arena.
According to a study from BCC Research, released in September 2015, the global drone market is expected to grow from $639.9 million in 2014 to $725.5 million in 2015. The drone market is further expected to grow to $1.2 billion by 2020 with a compound annual growth rate (CAGR) of 11.4 percent during the forecast period of 2015-2020.
Collision Avoidance Systems
A comprehensive collision avoidance system is now essential to prohibit drones flying into everything from power lines to trees to windmills. In the near future, there may be as many as 10,000 drones flying over a city on a given day.
So the drones not only need to avoid flying into buildings, trees and commercial aircraft, they also need to avoid other drones. That entails the recognition of objects through complex software algorithms and subsequently achieve a situational awareness to avoid object collision.
Computer vision enables drones to sense and avoid obstacles
The software can detect the objects of interest, track these objects frame-by-frame, and carry out the analysis to recognize the behaviour. Here, depth information can be used to construct 3D depth map based on the data captured from two stereo cameras in order to avoid object collision.
Moreover, the flying objects like remote-control drones move around in open space, so the video has to be stabilized from all three axes and six degrees of motion and rotation. That includes X, Y and Z axes—up-down, right-left and forward-backwards—and Pitch, Roll and Yaw (tilting forward, tilting sideways, and rotating around the middle).
How Dual-Camera Enablers New Markets
Drones are now open to serving a myriad of new markets from mining to agriculture to construction. For example, drones can be used to track the oil delivery while they measure the speed of the vehicle and locate leaks in oil and gas pipelines. However, for that, drones need to sense the surrounding environment, identify objects, and respond to situations in an instant.
Inevitably, drones will require a number of improvements. For a start, drones have to be made easier to fly and land, and for that, these flying machines require features like depth map to minimize the risk of crashing into other objects as well as ensure a safe landing.
Computer vision-equipped drones can evaluate the progress of a crop
It’s worth remembering that drones use CMOS sensors to record video, and that leads to “rolling shutter” effects when capturing fast moving objects. Next, wide-angle lens requires real-time lens distortion correction. Then, there is shake in the video due to drone motor or wind, and solutions like gimbal for vibration dampening and isolation contraptions come at various prices.
The viable engineering solutions to above problems include real-time digital video stabilization and other features for enhancing image and video processing using dedicated hardware accelerators. Moreover, there could be two cameras instead of one to allow drones to switch between daylight and thermal video streams.
A dual-camera-enabled drone with daylight and thermal cameras, for example, can fly over a parking lot and find cars that still have a hot engine. It can scan their number plates for payment or security purposes. Some drones companies already offers object tracking for “Follow Me” and “Point of Interest” features for various outdoor applications.
The Battery Conundrum
However, these new features will inevitably require more battery power, which is already a hanging sword on the future of drones. The battery life of a drone is generally limited to 15 to 30 minutes of flying time, and that has to change if drones are to reach a wider spectrum of applications.
The greater emphasis on image capture and vision processing will lead to more advanced camera subsystems, and that will cause greater strain on the drone batteries. Here, a DSP solution can handle significant compute workloads for image and video processing at much less power and die area on a chipset because of ISA tailored for specific applications such as aerial photography or 4K video post-processing.
XM4 vision processor outperforms GPUs by consuming 10% of power and 5% of die area
Take CEVA-XM4 vision processor, for instance, which has been designed from grounds up to run complex imaging and vision algorithms in a battery efficient manner. It’s a DSP and memory subsystem IP core that boasts a vision-oriented low-power instruction set along with programmable-wide vector architecture, multiple simultaneous scalar units, and support for both fixed- and floating-point math.
The CEVA-XM4 imaging and vision DSP, which is fully optimized for convolutional neural network (CNN) layers, software libraries and APIs, can facilitate designs of autonomous drones by enabling object detection and recognition features and thus help avoid collisions and ensure safe landings. The CEVA Deep Neural Network (CDNN) software framework complements the XM4 processor core with an easy migration of pre-trained Deep Learning networks like Caffe into designs like autonomous drones.
Find out more about computer vision system design and applications in the CEVA-XM4 white paper about XM4 intelligent vision processor.
Moreover, click here to watch CEVA’s webinar about implementing machine vision in embedded systems, including a deep dive into CDNN.
Ceva