NERDS: Heavy-lift Drone with low latency Command and Control over Cellular 5G and 4k Video Stream In Realtime

Image1: Project Team Picture After The Successful Handover

Note: This article post is a rewritten version of a white paper I wrote while working with the company, however, I tried as much to remove the typical corporate lingo and also added extra media and technical details for anyone interested.

Abstract

NERDs is an acronym for Network-Enhanced Real-Time Drones Project. It was a research and development project lead by myself while working at Indrorobotics with technical support from Ericsson and financial support from the Ontario Centre of Innovation (OCI)¹. OCI's mission is to drive economic benefit, job creation and growth in Ontario. Specifically, OCI is committed to helping "Ontario innovators get connected with the researchers, industry partners and funding needed to commercialize the next generation of made-in-Ontario IP and solutions."

Project Background

This project took place under the umbrella of the ENCQOR 5G program, which provides SMEs and academia with access to a 5G testbed (the ENCQOR 5G network is a 5G network running through a Quebec/Ontario corridor). (ENCQOR)² The NERDs project was based in Ottawa, out of Area X.O, to utilize the ENCQOR 5G network, as well as other infrastructure for Beyond Visual Line-of-Sight (BVLOS) testing. The project commenced in the fall of 2020 and concluded successfully at the end of March 2022.

While drone systems have advanced tremendously in the past decade, most commercial and industrial drone systems are operated over Radio Frequency (RF) bandwidths. These frequencies are generally between 900 MHz and 5.8 GHz and limit both the range of the drone's communication system (known as Command and Control, or C2), along with data bandwidth. These low-power networks are created by and exist only within the range of capabilities of the controller module/remote control.

Current Canadian and international regulations restrict drone flights, except under approved circumstances, to what is known as Visual Line-Of-Sight (VLOS). This means the aircraft must be seen, unaided, by the drone pilot or a nearby visual observer for the entirety of the mission. The rationale is that operating a drone Beyond Visual Line-Of-Sight (BVLOS) could lead to conflict with traditional aviation, property, or people on the ground. But this also means that long-range drone operations require special permission and are the exception rather than the norm. Although the rules exist for good reasons, they are a significant barrier to the adoption of drones at an industrial scale for inspections, deliveries, etc. Meanwhile, drone technology has advanced significantly in recent years. Drones can now carry out a multitude of data-intensive tasks useful to industry, including:

• Capturing 4K video
• Inspections using thermal and multispectral imaging
• Non-destructive testing
• Volumetric calculations
• Laser scanning to produce digital 3D 'twins'
• Air quality sampling.

Fig0: High-level NERDs Layout, note: while this layout gives most of the components/design, this was the first version and several iterations changed later, open in a new tab to zoom in

Yet two factors impede utilizing these capabilities at scale: The regulatory environment that prohibits most BVLOS flights, and bandwidth limitations that restrict real-time wireless data delivery. (Most data captured is stored onboard drones with MicroSD cards or Solid-State Drives.) Significantly higher data bandwidths have the potential to overcome both limitations. Greater data throughput enables technologies that enhance drone safety in BVLOS flights and allow for long-range, low-latency teleoperations. A large data pipeline also enables the real-time simultaneous transmission of uncompressed data to the ground and cloud-based servers. In short, enabling drones to operate over 5G networks (3GPP (3GPP, n.d.) mobile broadband standard) opens the door to safer BVLOS flights that can be teleoperated from great distances, plus more efficient data capture, processing and direct-to-cloud transmission. These are the technological hurdles the NERDs project attempted to solve.

The NERDs project charter states:

The goal of project NERDs is to conceptualize, design and test a 5G compatible module for use with Remotely Piloted Aircraft Systems (RPAS) to allow for remote command and control, as well as aviation radio detection, transmission, and edge processing to increase situational awareness in Beyond Visual Line Of Sight (BVLOS) operations

While there were successful previously flown teleoperated BVLOS missions utilizing LTE networks before, LTE does not have sufficient bandwidth for dense, real-time data transmission. LTE also poses issues with latency for the drone operator. Minimizing latency to the greatest extent possible is important for the safe operation of BVLOS flights.

Project Requirements

The NERDs project had specific technical requirements. These challenges were designed to fully test the potential capabilities of a 5G-enabled drone. InDro Robotics (the company I was working in that time, and also this project was the foundation for this company's expansion in the east side of Canada) was selected to develop a state-of-the-art 5G bidirectional module capable of the following:

• A Software Defined Radio capable of transmitting to nearby private aircraft
• Transmission of RF data back to the control station (signal strength, cell ID, 3GPP SNR, etc.)
• Client-side web integration to back-end drone dispatch service
• Transmission of live adaptive bitrate (720p, 1080p, 4k) video stream from the drone to the control station over 5G. Low-latency transcoding and transmission are key factors to success
• End-to-end video processing latency target of less than 100 ms with 1080p
• Transmit RTK (Real-Time Kinematic) data over 5G cellular link to the drone for high-precision positioning (< 5-10 cm).

Fig1: Project Requirements

Multiple components integrated into a single module

The drone airframe for this project was the FreeFly AltaX . The 20-kilogram platform has a lift capacity of 15 kg with a maximum flight time of 55 minutes. We previously carried out BVLOS flights with the same drone over LTE. The module being built for the project would eventually be known as RPAS2.

image2: Drone airframes are modified from FreeFly AltaX

5G/LTE Cellular Modem

To enable operations over 5G, a Quectel RM50XQ-AE series modem was selected (Fig.2). The 5G module, compatible with both Stand-Alone (SA) and Non-Stand-Alone (NSA) 5G networks, has been developed for IoT applications and was responsible for all onboard communications with the cellular network. The Quectel³ is capable of maximum download speeds of 2.5 Gbps and upload speeds of 650 Mbps under optimal conditions, supporting multiple 5G bands, with fallback to LTE and MIMO (Multi-Input Multi-Output) support with dual transmitter for enhanced upload throughput, with a built-in GNSS -not be confused with drone GNSS.

Fig2: Quectel Modem RM50XQ-AE

The Process of testing and surveying the network was challenging, for one, the private cellular networks have a very specific location/s to carry out the tests, unlike the typical public/commercial networks where in theory you can test it as long as there is coverage, mix that with the fact that the facility where ENCQOR network is located, was also closed during the pandemic, so the only way to access, test, benchmark, and basically anything must be done either completely outside regardless of the weather conditions or inside my car as you can see in the pictures below.

image3: Testing the network inside my car

image4: Or while enjoying the rain outside, high safe bench test as you can tell :)

Another challenge was the chip shortage that also happened during that time, and the logistics to secure that modem along other needed components were also challenging.

Additionally, I conducted a series of survey tests while driving around to collect extra parameters the you can not being stationary like location, speed factor, and network towers nearby cells among others.

To do that, I built a portable testing device made out of Rpi4, SDR BladeRF, and Quectel 5G modem primarily, in addition to some code to automate the process.

SBC

After several researches between the tradeoffs, I decided to choose a NVIDIA Jetson Nano as a companion computer for edge processing and was responsible for all drone C2 processing, including telemetry and autonomous flight functions. In addition, the NVIDIA was responsible for 4K video processing, First-Person View video streams used by the pilot, gimbal controls, as well as the modulation and demodulation of Airband communications. The design excluded other types of SBC (Single Board Computer) to enable high processing power while maintaining the form factor and low power consumption, excluding NUCs (Next Unit of Computing) or Jetson Xavier.

Fig3: Project Testing Area and ENCQOR Network

SDR

The project also required the integration of a Software Defined Radio, or SDR. This would carry out the 'Hear and be heard' function (Fig.4), transmitting a voice message to nearby crewed (traditional) aircraft. This was a critical piece of the challenge, as the SDR function provides those piloting small aircraft nearby with notification that a drone is flying in their vicinity. Such a function could help ensure the safety of Beyond Visual Line-Of-Sight missions. After researching to find one that fullfil the design including the power delivery, I selected SDR (bladeRF xA9) enabling bi-directional communications with pilots in the vicinity, and another at the Ground Control Station (GCS) to mimic the pilot's radio and receive the NERDs drone communication. The SDR is full-duplex and supports MIMO as well.

Fig4: High Level Collision Avoidance Using SDR

These SDR boards incorporate Field Programmable Gate Array integrated circuits, enabling custom programming to alter functionality. Additionally, I designed and selected a special antenna, designed for the frequency spectrum assigned to aviation communications (108-136 MHz). The Airband communications were complex, requiring detailed calculations and simulations. This helped to ensure the selected antenna would have optimal placement, EMC (electromagnetic compatibility), power, and antenna scattering RCS (Radar Cross-Section)

Fig5: Pilot to on-board SDR Communications to other Pilots in the vicinity

On-Board Antenna

In addition, I had to ensure the selected antenna would fit the airframe and function properly regardless of the position in space of the drone. This distance (r) is determined with Near Field and Far Field Pattern equation:
r=(2D^2)⁄λ(1) Volakis ³, where D is the dimension of the antenna, λ is the wavelength.

Fig6: Antenna Design, Analysis, and Simulation

The FEM simulation (Finite Element Method) was not appropriate due to the complexity of the drone's shape. Instead, the Multilevel Fast Multipole Method (MLFMM) was used to determine the best parameters.

Fig7: EMC Scatter For The Antenna

Finally, calculations for EM (Electro-Magnetic) compatibility and EM scatter were carried out before selecting the optimal antenna for the project. A blade VHF antenna proved to be the best choice, satisfying all criteria.

4K Realtime Video Streaming

For the 4K video streaming challenge, and after some research, I selected the Sony A7s III, as it was not straight forward to find a camera that was able to output raw 4k directly from its HDMI port at that time. This camera features a full-frame CMOS sensor and produces dense, high-bitrate 4K video streams. The ability to stream 4K video or other high-bitrate data has two advantages: In the case of 4K video, it enables greater situational awareness for the pilot, who can see surroundings from the sky with far greater clarity than common 720p feeds. (In fact, 4K provides nine times the pixel density of 720p). It also enables, in conjunction with the 5G modem, the ability to stream dense data to the cloud and ground stations in real-time. This has been, until now, a bottleneck that has required many industrial drone operators to record data on MicroSD cards, and then manually upload the data following the missions.

vid2: Several tests of different encoding streamed over different networks.

The 4K stream data rates can reach up to 70mbps in real-time, posing the following challenges for this project:

• Network: While 5G is now emerging, most 5G networks are currently NSA (Non-Stand-Alone, meaning they share the same LTE backbone). Moreover, cellular networks are fragile in nature (compared to FTTH/Fiber-To-The-Homes, for example). Other factors can further complicate things: Bandwidths can change due to regulations, and multiple UEs (user equipment, like a phone) connecting at peak periods can lead to network congestion. External factors such as weather - even sun coronas and solar flares - can lead to interference or, potentially, a temporary but total outage of the system.

• Limited technology: Because of the drone platform, the only viable option for real-time processing is to use an SBC (single-board computer) for real-time processing. At the time of writing this article, the most current and powerful SBC uses the Tegra architecture (NVENC with 4 cores on an ARM64 chip). This eliminates harnessing desktop/enterprise architecture like Ampere's B-frame, which utilizes previous and forward frames, as well as the support of highly efficient encoders (like AV1). This presents a challenge. In addition, the on-board SBC is also responsible for other tasks while the drone is flying, including C2/Telem (command and control and telemetry) links, positioning, as well as other cameras on the system (including First Person View camera).

• Limited Resources: The drone platform (size and takeoff weight) is a limiting factor - meaning it cannot bear a fully-fledged server rack. So there are limitations to what can be put onboard for real-time data processing, etc.

The ENCQOR network (the private network used for communication to control the NERDs drone) can reach up to 45mbps data rates maximum upload for limited periods of time. This data rate is not sufficient for the low-latency, real-time 4K video streaming to satisfy the NERDs challenge. As a result, I developed an algorithm for the FEC (forward error correction) based on the draft IETF payload scheme 20⁴. Though the technicality, development, and implementation of this algorithm are involved, the concept boils down to the idea of sending the buffer of the 4K video in a multi-dimensional interleaved pipeline instead of the linear one, allowing the possibility of sending repair packets 'on the fly'

Fig8: FEC Algorithm for realtime 4K stream

The Sony A7s III was installed on a Gremsy T3 gimbal (seen below in image5 holding the payload), which allowed the pilot to change the camera view from the ground station over 5G (tilt, pan), and within InDro Pilot - the software platform powering the drone, more about it below.

image5: The RPAS2, and the Gremsy T3 gimbal seen here holding the camera/payload

RTK Positioning

Positioning accuracy is important for several reasons: It ensures that geo-referenced data is tagged with far greater accuracy, leading to more robust datasets to produce digital twins, etc. Greater accuracy also improves safety for autonomous missions (i.e., centimeter-level precision landings, waypoints, etc.) further contributing to the safety of missions and the confidence of regulators.

System Architecture

NERDs system architecture: For this project, I built from the ground-up software platform to enable and extend all project requirements. The platform is called InDroPilot, within the NERDs system architecture (Refer to InDroPilot section below and the Indropilot project for more details). It enables all on-board as well as cloud/GCS communication and processing.

Fig9: Project NERDs Architecture (Later Indropilot) - open the image in a new tab to zoom in

Latency

Latency plays a crucial factor in the NERDs project - and for multiple reasons. On the C2 Link, for example, low latency means a pilot using a real-time joystick to control the drone and camera will have a smooth, responsive, and safe flight over 5G. It's also important for telemetry data and positioning - even for the FPV (First-Person View) camera for real-time viewing or 4K streaming. The pilot needs to have the feel that they are directly in the virtual cockpit, with near-zero latency.

This project achieved and delivered very low latency over 5G networks of sub 75ms up to 40ms (vs. sub-400ms over LTE). This enabled low-latency C2/Telem links, as well as video streams for both 4K and FHD in real-time, in addition to the real-time bi-directional communication within the aviation airwaves.

To achieve such low latency, I built a custom tunnel p2p network that leverages software-defined networking for multicast, rules, etc., the whole component in Indropilot platform named (InDro Link) More in Indropilot project first establishes a node in the cloud to authenticate all parties connected to this link. It then enables low-latency peer-to-peer connection with end-to-end encryption (AES-256 cipher with RSA-4096 handshake encryption and SHA-512 hash authentication on both control and data channels). Latency is reported and displayed in real-time within the InDroPilot platform and updated nearly every second - without adding any extra overhead packets in the platform.

vid2: Flight Test of the drone over 5G. Note: This is one of the early tests with a choppy 4k video stream, but it shows the realtime latency, also, the video is resized to 1080

Such low latency enabled and opened huge opportunities, one of them is drone delivery More on Drone Delivery Project, as well as direct control by joystick, and it was field tested on several scenarios More in Indropilot project where the pilot in one scenario was in Texas, US and flying a drone in ON, Canada as if they were standing in front of it, another one the pilot was in BC, Canada, and a third time the pilot was in Australia flying with sub 100ms latency for a drone located in ON, Canada!

Fig. 10/11 shows the latency over both ENCQOR 5G and Commercial LTE network over time, every second. The first chart shows both latencies, orange one is the LTE and the blue one is the 5G latency. The next charts are the latency over time for each.

Fig10: Latency Comparison Between Commercial LTE (orange) and Private 5G (blue) Networks, lower is better, left axis is in ms

On average, the latency over 5G was around 46ms and 231ms over LTE, graphs below summarize a quantitative distribution with five standard statistics: The minimum value, first quartile, median, third quartile, and maximum value.

Fig11: Average Latency Comparison Between Commercial and Private Network

As for the latency correlation and distribution over time, there are many contributing factors. This includes ground station machine performance decoding the 4K stream, plus the on-board edge computing unit. However, overall correlations from the tests were more-or-less consistent.

Fig12: Latency correlation and distribution over time

Next is the latency data point cloud with estimated density over time. Darker spots indicate high occurrence compared to light ones.

Fig13: Latency data Point cloud, darker areas mean more frequent "hits"

Cellular Survey

Cellular networks, in general, have been created for ground-based users (cellphones, etc.). Although surveying the cellular network itself wasn't part of the NERDs project requirements, as you can see above the efforts I put into testing the network on the ground with the portable test unit I built, I decided to add this ability to the software (Indropilot) so you can scan the sky as well by flying the drone, since it already has the automated feature to scan when I was driving, the feature 'map' cellular strength at altitude for both 5G (SA/NSA) and LTE. This provided a more thorough understanding of network infrastructure, as well as ensuring that no dead zones would be encountered during missions, either for this project or future ones.

Drone Flight Path

While surveying the cellular network, the drone was flown at three levels: 30m, 40m, and 50m AGL (Above Ground Level).

Fig14: The path the drone took to scan the cellular network

Data was collected every second, resulting in a massive dataset. The following measurement statistics define the quality of the cellular signal (RSSI): RSRP, RSRQ, and SINR.

Fig15: RSRP, RSRQ, and SINR thresholds.

RSSI: Received Signal Strength Indicator.

Fig16: RSSI data set during the flight, we can see the RSSI gets better the higher the drone fly/Higher AGL

RSRP: Reference Signal Received Power.

Fig17: RSRP data set during the flight, just like the RSSI, it gets better the higher the drone fly/Higher AGL

RSRQ: Reference Signal Received Quality

Fig18: RSRQ data set during the flight, higher AGL, better RSRQ

SINR: Signal to Interference plus Noise Ratio

Fig19: SINR data set during the flight, higher AGL, better SINR

InDro Pilot

At the early stages of the project, the initial plan was to use a off the shelf software to power the NERDs drone, since building the C2/TELEM software side was not a requirement, however, a few months passed and after evaluating the current options, it was clear none of them will have the flexibility and the integration that I was looking for, mainly for 4K realtime stream and Airband comms, and the usual buy/rent analysis was done and it was clear it is the time to build from scratch, and it is when IndroPilot platform was born!

InDro Pilot is a next-generation modular platform that empowers and enables control over cellular networks (5G/LTE). It can be used on drones for BVLOS flight, and even on Uncrewed Ground Vehicles (UGV). The platform emphasizes modularity, expandability, and reliability while still being user-friendly with minimum configurations required to be up in the air. It enables C2/Telem as well as modular integration of any payload per mission requirements. More in Indropilot project

InDro Pilot is an ecosystem platform, not a single program. Three major components comprise InDro Pilot:

• On-board (named InDro Captain): Responsible for on-board, low-level processing, positioning, navigation, and interfacing with payloads
• Network (named InDro Link): Establishes a secure, low-latency link between the GCS and edge computers for drones or UGVs
• On-ground (named InDro Base): Responsible for the GCS side of the pilot/operator

The platform I built from scratch, software, APIs, embedded drivers, hardware design, UI, Architecture, and even the names you read above and the icons you see below I hand-crafted them! (I used Figma/Illustrator for that) However, I consider the platform as the first iteration only, and soon will release a superior one.

Fig20: Indropilot eco-system

Platform high-level features are:

• Next-gen platform empowering the mission with command and controls, telemetry links, and all comms to ground stations with mission autonomy and centimeter-position accuracy (depends on airframe GPS)
• Supports 5G out of the box, both NSA (Non-Standalone) and SA (Standalone), with LTE/3G fallback
• System agnostic, works on all architectures (ARMv8, ARMv7, AMD64, etc.)
• Drone agnostic: Supports all Pixhawk-based drones
• Modular, supports a wide range of integrations with different payloads
• Reliable for mission-critical operations and BVLOS
• Ultra-reliable low-latency connection. All links are encrypted with state-of-the-art crypto (AES-256 cipher with RSA-4096 handshake encryption and SHA-512 hash authentication on both control and data channels) and in compliance with ISO27000 and NIST800
• Powerful edge-computing units powered by NVIDIA for AI/ML, real time transcoding, and processing

Fig21: Screenshot of on-board components that some of them were used for NERDs project.

Challenges

In addition to the technical challenges mentioned in this here, the project also faced other challenges. These are summarized as follows:

• The project was awarded, executed, and delivered during the worldwide COVID-19 pandemic. This impacted the project in multiple ways. Engineering staff including myself were required to work from home in isolation for long periods and faced additional challenges with restricted access to Area X.O facilities, my home was at some point a workshop with all sort of drones and tools!

image6: Yep, that was my development desk, both hardware and software during the lockdowns!

• The global chip shortage impacted orders, shipments and deliveries

• The pandemic also closed access to the Bayview Yard, the location of Ericsson's ENCQOR network. This was off-limits until Feb 2022, a month prior to the delivery date. This also meant I was accessing the network during months when extreme weather conditions were the norm, as seen above by utilizing my car as a portable test environment

• With the project located in Canada, flights had to be carried out during sub-optimal weather as the deadline approached, and every test was time consuming as it required several individuals to complete all the calibrations and preflight checklists, image below shows the process of calibrating the drone before each test.

image7: The process of calibrating the drone before each test, was conducted by my two colleagues

Implications/Conclusion

The successful outcome of this project carries significant implications for the company. clients (Ericsson, OCI), and the broader drone industry.

As noted, most industrial drones are currently operated within Visual Line-Of-Sight and over the Radio Frequency bandwidths of the remote control. Some manufacturers have just begun to release drones that operate over 4G. 5G is the next territory, and one that will enable greater data throughput and vastly greater efficiencies for industrial applications (i.e. direct cloud uploads of dense data during flight). 5G is the door that will truly make drones and ground robots part of the Internet of Things.

But high-speed, low-latency data uploads are not enough to unlock the promise of Beyond Visual Line-Of-Sight flights. Regulators must be satisfied that drone missions are being operated with a minimal chance of conflict with crewed aircraft, people on the ground or property.

To that end, this project made other significant breakthroughs that work in tandem with the potential of 5G. They are:

• Near-zero latency 4K video streaming, enabling greater situational awareness for the pilot
• Software Defined Radio voice transmissions, alerting pilots of traditional aircraft in the vicinity of the mission of drone operations
• Centimeter-accurate positioning data for autonomous flights and datasets
• The InDroPilot software suite

Collectively, these are significant achievements for the industry. Improving the safety of situational awareness for both the pilot and crewed aircraft is a solid step toward more routine BVLOS flights. The dramatic improvement in low-latency data transmission will also be embraced by industrial users, who have been searching for a more efficient solution to data handling and cloud processing.

In addition, the 5G/6G promise is not limited to this project. Industrial applications such as inspections and surveillance, volumetric calculations at facilities such as cement plants or mining operations, will continue to grow. We have noted significant growth in private 5G networks for industry, and innovations such as those detailed in the NERDs project, will generate even greater demand.

In summary, the NERDs project was a clear success. It is not an understatement to suggest that, collectively, these achievements mark an inflection point in the sphere of drones as they relate to BVLOS flights and the IoT.

image8 a/b/c: FUN FUN!

Back to top