Archives of Design Research

Copyright : This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/3.0/), which permits unrestricted educational and non-commercial use, provided the original work is properly cited.

“Drone” is the name given by the military to an unmanned aerial vehicle (UAV). In recent times, helicopter-type drones equipped with multiple rotors have been commercialized and used in various fields such as photography, agriculture, distribution, and disaster management, as well as for leisure (Shin et al., 2014). However, with the widespread use of these devices, drone-related accidents are increasing. The accident rate and the possibility of property damage and injury from drones are known to be about ten and 100 times greater, respectively, than those for aircrafts (Weibel & Hansman, 2006). The most frequent cause of drone accidents is poor handling by pilots (Williams, 2004; Hing & Oh, 2009). Several cases of pilot deaths or serious injuries due to pilot errors have been reported overseas (Cho et al., 2016). Therefore, it is very important to design a controller that can be easily and safely used by pilots to control drones and reduce the risk of accidents (Hing & Oh, 2009; Cho et al., 2016).

There are three main factors that contribute to the difficultly of controlling drones. First, it is difficult for pilots to maintain situational awareness owing to the limited availability of sensory information. Because the pilot is physically separated from the drone, it is necessary to manipulate drones using only visual information without the possibility of acquiring information through vestibular, tactile, and auditory senses (McCarley & Wickens, 2005; Lam et al., 2007, Hing et al., 2009). Lam et al. (2007) showed that supporting the sensory information through a haptic interface increases the workload and task activity as compared to that when relying exclusively on visual information. However, it effectively reduces collision frequency and ensures stability. In addition, providing integrated vestibular feedback with visual feedback for the degree of drone rotation is helpful to the situational awareness of the pilot (Giordano et al., 2010).

Second, control difficulties arise from inconsistencies between the viewpoints of the drone and the pilot. The drone pilot generally has access to one of two viewpoints. One is the third-person view (TPV), in which the pilot controls the drone while visually viewing it from the ground. The other is the first-person view (FPV), in which the pilot control the drone as if riding in the drone's cockpit. The FPV corresponds to the concept of egocentric view or internal piloting, while the TPV is corresponds to that of allocentric view or external piloting (Brewer & Pears, 1993; Soechting et al., 1996). Drone manipulation from the FPV is challenging in that the situation perception of drones may be insufficient or inaccurate owing to limited visibility, slow control response and feedback, and lack of sensory information (Williams, 2006; Hing & Oh, 2009).

The pilot attempts to manipulate the drone assuming that his/her forward direction and the forward direction of the drone are identical. However, when the drone rotates in the TPV mode, this is not the case. At such a time, because the forward direction of the drones is different from that assumed by the pilot, the pilot's intended direction of manipulation and the drone's actual movement often differ. This is called the reference-frame misalignment problem between controllers and drones (Williams, 2006). For example, if a drone is at an angle of 90˚ to the left with respect to the forward direction of the pilot, as shown in Figure 1, the drone will move toward the left of the pilot when the pilot moves the drone in the forward direction. In this case, the pilot must control the drone by consciously considering its perspective (Cho et al., 2016). Thus, if the pilot wants the drone to move straight in the forward direction, he/she must move it towards the right. Such mental rotation manipulation is known to be cognitively difficult and can cause several malfunctions (Arthur & Hancock, 2001; Gugerty & Brooks, 2004; McCarley & Wickens, 2005; Williams, 2006). In particular, the greater the angle of the required mental rotation, the worse the drone operation performance (Aretz & Wickens, 1992; Gugerty & Brooks, 2004). The reference-frame misalignment problem can be solved by the automation of the drone control, a function to always match the front of the pilot and the front of the drone (i.e., headless function), and drone manipulation from the FPV (Williams, 2006).

Figure 1 Reference-frame misalignment problem when controlling a drone

Third, owing to the lack of intuitiveness, it is difficult to operate a joystick controller, which is the most common type of drone controller, shown in Figure 2 (left). Drones move with propeller propulsion, and the attitude is controlled by three-axis rotation. A joystick controller uses the left joystick to control the yaw and throttle for rotation and up and down movement, while the right stick controls the pitch and roll to move the drones forward, backward, left, or right. However, as mentioned above, the drone control using a joystick is not intuitive, and it requires considerable training time for precise flights (Higuchi & Rekimoto, 2013). This is because drone control requires three-dimensional movement, unlike the two-dimensional movement common to other joystick-controlled devices such as remote-controlled (RC) cars and model ships. Essentially, learning and manipulation are challenging because it is difficult to cognitively map the two-dimensional motion of a joystick to the three-dimensional motion of drones. In particular, in the case of eight-way flight, which requires simultaneous operation of more than three axes including the yaw axis while maintaining the altitude of drones, the operation difficulty is very high and requires considerable learning. Unlike the two drone control difficulties mentioned above, the joystick controller makes it difficult to manipulate the drone regardless of the pilot’s viewpoints, i.e., TPV or FPV. Thus, it is very important to improve the drone controller to allow intuitive control, and several related studies are under-way (Vincenzi et al., 2015). The types of intuitive drone controllers that are superior to joysticks can be broadly classified into gesture, pointing, and motion-control methods. Studies have also been conducted to naturally manipulate drones using the human gaze or voice (Hansen et al., 2014; Yu et al., 2014; Fayjie et al., 2017; Menshchikov et al., 2019), or even brain signals (Jeong et al., 2020; Chen et al., 2020).

Figure 2 Joystick drone controller

Gesture control is a method of manipulating an unmanned device by mapping specific movements of the operator’s hand, arm, or head to the movement of the device. For example, Higuchi and Rekimoto (2013) proposed a manipulation method in which a drone moves forward when the pilot’s head is lowered, rotates when his or her head is rotated, and descends when his or her body is in a downward position. Several studies proposed a system for controlling a drone using pilot’s hand and finger gestures recognized by a Leap Motion device (Graff, 2016; Fernández et al., 2016; Sarkar et al., 2016; Gubcsi & Zsedrovits, 2018; Zhao et al., 2018; Bandala et al., 2019). Gesture-based natural controls are more intuitive and efficient than joystick controls in indoor environments (Lambrecht et al., 2011; Sanna et al., 2013; Mashood et al., 2015). Rognon et al. (2018) proposed a soft exoskeleton to control a drone with upper body gestures, and they showed that participants using the exoskeleton felt more immersed, had a greater sensation of flying, and reported less fatigue. DelPreto and Rus (2020) developed a gesture control system for manipulating a drone using wearable muscle and motion sensors. However, while gesture control may be intuitive for specific commands that require simple manipulation, it may not be appropriate for complex manipulation (Pfeil et al., 2013). This is because the more complex the flight trajectory is, the more gestures are required, and the greater the flight speed, the more difficult it will be to quickly change gestures according to the changing situation.

Pointing control is a manipulation method in which the pilot maintains a certain distance from the drone and moves it to the desired position by pointing to the corresponding point within the virtual hemisphere in front of the pilot. Chen et al. (2019) proposed an egocentric drone controller that allows pilots to arbitrarily position and rotate a flying drone using pointing interactions on a see-through mobile augmented reality (AR). Gromov et al. (2019) developed and validated a system for controlling the position of a drone in 3D space using pointing gestures. It is very easy to point to the desired location, but the greater the distance between the drone and the pilot, the less accurate will be the operation. In addition, from the FPV where the pilot’s forward view is blocked with a head-mounted display (HMD), he/she cannot accurately grasp the direction of the drone. Thus, this type of control method cannot be applied in this case.

Motion control is a method of mapping the movement of the controller to the motion of the drone; the drone rotates by following the angle of movement of the controller and moves forward as the controller moves forward (Figure 3). This type of operation is natural and intuitive for the manipulation of physical objects such as drones. Shin et al. (2014) implemented a controller system that controls a drone using the motion of a smartphone with a built-in G-sensor. Cho et al. (2016) showed that automatic drone rotation synchronizing with the direction of the joystick controller was highly effective because it could eliminate the misalignment problem. Morishita et al. (2016) proposed a control method that matches the movement of a drone with the head motion of a pilot wearing an HMD. This head-synced drone control could reduce virtual reality (VR) sickness (Watanabe & Takahashi, 2020). Such a motion control allows novices to learn quicker than they would with conventional controls and enables the stable operation of drones.

Figure 3 Motion-matching drone controller

There were researches in which some movements of the controller (e.g. rotation) were matched with some of the movements of the drone, but there was no controller that perfectly matched the 3-axis movement of the controller and the 3-axis movement of the drone. Therefore, this study proposes and develops the motion-matching drone controller that completely matches the motion of the drone. Then, this study confirms that the motion-matching drone controller in both TPV and FPV modes is easy to use and intuitive even for beginners.

In a previous study, Chang and Kim (2018) analyzed appropriate forms of motion-matching drone controllers and presented five representative controller types, as shown in Figure 4. They assessed their usability after implementing physical models with 3D printers. Their experimental results showed that users prefer the type shown in the second image in Figure 4, in which the throttle is manipulated with the thumb. Based on this, in this study, the shape of the motion controller was improved, and a design and prototype were developed, as shown in Figure 5. The main body of the motion controller is designed to have a curved shape with a natural wrist angle when gripped. It is designed symmetrically for easy operation with either hand, and it remains stable with the finger support between the forefinger and middle finger, even when the grip is loosened. The throttle dial switch is located at the center, and the two buttons responsible for sensor initialization and yaw-axis compensation functions are placed on either side of the dial switch. All three operating switches are placed in the line of the thumb at an angle of approximately 30˚ so that they can be operated comfortably while holding the controller. The power switch is placed below reach to prevent accidental pressing, and the battery is placed at the bottom so that the center of gravity is at the bottom.

Figure 4 Representative forms of a motion-matching drone controller

Figure 5 Prototype design and final rendering of the proposed motion-matching drone controller

Figure 6 shows the hardware configuration of the developed motion-matching controller. The controller consists of a control chip for motion control, a wheel switch for throttle control, a tactile switch for initializing the motion sensor, a slide switch for power control, a battery, and an antenna. The control chip is from E2BOX Ltd., which has a Cortex-M3 processor and an inertia measurement unit (IMU) sensor, MPU6050. It is placed inside the body of the controller to control the drone through wireless communication. The communication between the controller and the drone uses a 2.4GHz RF Transceiver, which is the same as the communication used by a wireless joystick controller.

Figure 6 Hardware structure of the motion-matching drone controller

The embedded software of the controller and drone, which was written in C language, functions as follows. When the user turns on the controller or presses the reset button, the controller's posture at that time matches the horizontal position of the drone looking forward. This determines the initial position shown in Figure 7. Following this, the controller continuously senses the values of the three axes (pitch, yaw, and roll) of the MPU6050 sensor together with the throttle value and transmits them to the drone. Then, the drone adjusts the speed by adjusting number of motor rotations based on the throttle value received from the controller. In addition, the drone compares its current values with the three received axes values and adjusts the motor accordingly to maintain the same posture as the controller at all times. Thus, the drone's posture always matches the controller's posture (Figure 7).

Figure 7 Manipulation of the motion-matching drone controller

Interestingly, the aforementioned reference-frame misalignment problem can be resolved using the motion controller. When rotating the drone, the pilot can rotate the wrist or arm holding the motion controller or rotate the body itself (Figure 8). However, when the angle to be rotated is high, the latter is more comfortable and natural than the former. If the pilot rotates their body to rotate the drone, their forward direction and the forward direction of the drone always coincide (the rightmost picture in Figure 8), thereby eliminating the reference-frame misalignment. This is similar to the headless function.

Figure 8 The relationship between the operator's posture and the solution of the reference-frame misalignment problem

This study proposed the motion-matching controller that controls the drone by perfectly matching the posture of the controller and the drone in a three-dimensional space. Then, this study demonstrated that the motion controller is more intuitive and usable than the conventional joystick controller. In the conducted experiments, participants performed the task of driving a given course with actual developed controllers and drones. The results showed that the motion-matching controller achieved a higher success rate than the joystick controller in maintaining the altitude and rotating the drone. The motion-matching controller was found to be superior to the joystick controller in terms of task success rate, number of collisions, task completion time, number of failed attempts, and subjective evaluation, particularly in the FPV mode. Specifically, it was observed that participants naturally used the motion-matching controller to perform the complicated manipulations that require controlling two or more axes simultaneously, while these were difficult to perform with the joystick controller.

As far as we know, there is no such type of motion controller as suggested in this study. Moreover, there have been no prior studies showing how the motion-matching controllers are useful in two viewpoints (TPV, FPV) during manipulating real drones. The experimental results showed that the motion-matching controller is useful in both the TPV and FPV, and particularly so in the FPV. It appears that the motion-matching controller, which enables position control, can more easily and accurately control the rotational position than the joystick, which enables speed control.

The introduction of this paper lists three reasons for the challenges faced in drone control, including low situational awareness, reference-frame misalignment, and the lack of intuitiveness of a joystick controller. The motion-matching controller in this study intends to address the latter two difficulties. The motion-matching controller naturally requires the pilot to rotate his or her body to rotate the drone, and thus, the drone's forward direction and the body's forward direction always coincide, thereby eliminating the reference-frame misalignment problem. Additionally, the motion-matching controller is highly intuitive, because the movements of the pilot or controller exactly match the movements of the drone. The experimental results showed that the motion controller failed less often than the joystick controller. In particular, the number of failed attempts for the joystick-FPV combination was very high. This indicates that the participants had difficulty in controlling the drone in the FPV using the joystick controller. This difference between the two controllers in the FPV is largely due to the rotation manipulation of drones. With the joystick controller, it is necessary to stop the movement of the drone, rotate, and then move it again. The motion-matching controller allows quicker rotation because it can curve naturally without stopping the movement of the drone.

However, this study has some limitations. Some performance differences between the developed motion-matching controller and existing joystick controller may affect the experimental results. In addition, if the drones had a hovering function that enabled them to maintain altitude easily, the results of the experiment may have changed. Furthermore, because the experimental driving course was considerably simplified considering the fatigue of participants, it may be difficult to generalize the results to an actual driving environment with a complicated route. Therefore, further experiments are required considering the hovering function and the complexity of the running course.

These findings demonstrate that if the motion-matching drones are manufactured using more precise sensors and their sensitivity is adjusted to allow more stable control, they can be highly effective for leisure or in industrial applications utilizing an FPV mode.