Archives of Design Research

Producing a high-quality user experience for an interface by adjusting its transfer function is primarily achieved for commercial products. Although the literature is scarce, a few research papers discuss improving the interactivity quality and user experience by manipulating transfer functions. For example, Roussel et al. (2012) proposed subpixels that are controlled by manipulating the transfer function to increase the sensitivity of pointing input devices. Making changes to the transfer function enabled researchers to control the level of precision and sensitivity of the input without altering the total characteristics of interactivity at the operating system level. High user performance in needs-specific settings was reported, such as browsing in a constrained display and editing sophisticated details of digital pictures.

Although people have recognized the importance of transfer functions in determining interactivity quality, few tools are available for designing or manipulating transfer functions. These tools can be categorized into two types: 1) implementing the function by manipulating its mathematical formula or 2) modifying the graph of the function using a graph-editing tool. A related example is demonstrated in Interactivity Sketcher (Woo, et al., 2011). This tool enables a designer to modify a graph that defines the mapping between input values and output values. In this case, designers need to be capable of reading graphs to interpret the interactivity quality that they can create.

In our study, we attempt to overcome the limitation of current practices by introducing a new way of crafting interactive quality for interaction designers. In the following sections, we explain how we developed this new approach in detail.

Ideally, IC should be utilized for any applications in designing transfer functions that relate to these applications. However, for this research, one application was implemented for the IC prototype with the objective of investigating whether interaction designers can intuitively modify transfer functions without focusing on how to numerically define them. Example application should also be familiar to designers so that they can focus on designing the interactivity without having to determine the application. The selected application is the Pen Tool application. The Pen Tool enables a user to draw anything with a pen on the canvas of a mobile device. Using the S-pen SDK, two pairs of input-output relationships for transfer functions were employed to determine the “feel” (Swink, 2008) of the pen interactivity of Pen Tool. The mappings between the pen input and the line output in Pen Tool included the mapping between the pen-pressure input and the pen-thickness output and the mapping between the pen-movement velocity input and the pen-thickness output. The mappings also demonstrated that more than one transfer function can be required to determine a more sensitive quality of the pen interactivity.

The Pen Tool is an appropriate example of virtually experiencing an interaction that exists in the physical world: we can mimic the quality of a real pen in the physical world via the manipulation of its interactivity quality. This approach has also been employed in other studies, for instance, in modifying the transfer function in the mouse interface to give the user the feeling of hurdles or holes in the virtual surface (Lécuyer, Burkhardt, and Etienne, 2004). Similarly, we can create a pen interactivity that simulates the use of an expensive fountain pen or a Chinese calligraphy brush by modifying the transfer functions. Because many applications mimic the physical world to create a more intuitive interaction experience, this type of application is meaningful for IC testing. Two functional features in the Pen Tool were implemented: drawing simple freeform strokes and erasing simple freeform strokes. Table 1 shows the set of input-output pairs selected for the application.

The IC prototype consists of the App, the Controller, the Tester, and the Server (Figure 1). The Controller and the Tester modify the transfer functions of the input-output relationships. Each component independently functions: either the Controller or the Tester can change the transfer functions. The Server is responsible for transferring, synchronizing, and processing the data exchanged between the Controller and the Tester.

The Controller and the Tester are developed as applications on the Samsung Galaxy Note; the Server is developed as a Java application on a PC. The Tester is the application to be tested, i.e., the Pen Tool.

3. 2. 1. Tester with the Interactive Feedback Method

Figure 1 Architecture of IC

Figure 2 Main screen of the Pen Tool Tester

The Tester is the most important part of IC. The main feature of the Tester is to enable interaction designers to intuitively change the transfer function by interactively providing feedback to the system while they experience the interactivity of the application content being designed. The interactive feedback is the core feature to accommodate the intention of the designer to achieve the desired interactivity quality.

Figure 2 shows an example of the test space in the Pen Tool. The test space can be utilized in two modes: 1) the canvas mode for drawing and 2) the feedback mode for interactive feedback. When the Pen Tool is in canvas mode, users can freely draw and modify black freeform strokes with a stylus pen. Users can erase the drawing using their finger. The thickness of the stroke is determined by two input values: the velocity of the pen’s movement and the pressure of the pen’s tip. Each input value is translated through different transfer functions; as a result, the two input values collectively determine the thickness of the dot drawn by the pen tip. The thickness is calculated as

T = a*T_v + b*T_p + c,

where T is the final thickness, T_v is the thickness determined by the velocity of pen movement, T_p is the thickness determined by the pressure of the pen tip, and a and b are proportional factors (coefficients) for balancing the thickness values from two different input sources. The constant c is the value that guarantees the minimum thickness of the stroke. The formula for determining the thickness of a stroke for the Pen Tool varies. In this case, several pilot trials were conducted to ensure that the two input sources of thickness determination were perceptually felt to influence the final thickness in a relatively equivalent manner.

Figure 3 Screen of the interactive feedback mode

Two buttons enable the Pen Tool (Figure 2(b)) to change the mode of the test space from the canvas mode to the feedback mode. In the feedback mode, a user can provide feedback for modifying the transfer functions. The left button is used to provide feedback about the transfer function of the input velocity, whereas the right button is used to provide feedback about the transfer function of the input pressure. When either button is clicked, the test space changes the canvas mode to the feedback mode, as shown in Figure 3. The greatest challenge in this area is to derive a solution that enables a designer to vividly remember the current interactivity quality of the Pen Tool while providing modification feedback. Interactivity is an invisible quality that can only be experienced when interaction occurs. To support a user’s recall of the interactivity experienced when modifying the transfer function, the last stroke is repeatedly animated in feedback mode while a designer modifies the function. During the animation, a designer can directly provide feedback: when the touch is dragged upward, the stroke becomes thicker and vice versa (Figure 3). The rate of change in the thickness of the stroke is proportional to the distance of the dragging. By touching the “feedback” button located on the lower left side on the screen in Figure 3, the feedback data are applied to the current transfer function for modification. After providing feedback, a designer can return to the drawing mode to verify the changed interactivity. Via iterative feedback, a designer can achieve the desired quality. Every time the feedback is applied, all data are saved in the Server and synced across the Tester and the Controller. Whenever this sync occurs, a “Data Sync” pop-up message (the bottom of Figure 2(a)) is shown. The filename of the saved data, which can also be modified, is shown in the textbox area ( Figure 2(c)). This interactive feedback is designed to support the practice of crafting the quality via reflection-in-action (Schön, 1983).

3. 2. 2. Controller with the Graph-Editing Method

The primary purpose of the Controller is to compare the interactive feedback method with the traditional method, which utilizes a graphical representation of the transfer function and the graph manipulation menus for review and editing. Thus, we can determine how our interactive feedback method in the Tester differs from the traditional method in our experiments. Figure 4 shows the main screen of the Controller, where the user can review and edit the transfer function by manipulating its graphical shape. The x-axis corresponds to the input values; the y-axis corresponds to the output values. Because the input values are scaled, the value 0 on the x-axis is mapped to the minimum input value, whereas the value 100 on the x-axis is mapped to the maximum input value. The range of the input values is divided into 100 units using integer values between 0 and 100. The range in the y-axis is also divided into real numbers between 0 and 100.

Figure 4 Main screen of the Controller for the Pen Tool

The graph area (Figure 4(a)) is freely editable with a pen or touch. When the edited graph is saved (Figure 4(c)), the program also updates the transfer function of the App on the Tester by saving the data on the Server. A set of pre-defined transfer function templates, which is provided in this study (Figure 4(b)), can also be modified. The two buttons on top of the graph area (Figure 4(d)) are used for selecting the individual input-output mapping that defines the transfer function. These buttons correspond to two input-output mappings of the Pen Tool: the velocity-thickness and the pressure-thickness. When using the Tester functionality, these buttons are also used to select the transfer function to edit. To verify and experience the changed interactivity of the App, the user has to launch the App on the Tester.

The Gaussian Process Regression (GPR) method was applied to adapt the transfer function according to the interactive feedback (Gibbs and MacKay, 1997). GPR is a modern machine learning technique frequently employed for function approximation (i.e., regression) problems. Although classical function approximation methods assume a parametric class of functions (e.g., linear or quadratic) to fit the data, GPR is a non-parametric method because it does not restrict the class of functions and is purely data-driven. GPR also trades off between the complexity of the learned function and the amount of available data using the Bayesian framework. One small disadvantage of using GPR is that it requires all data to compute the output of the function even when the learning is completed because it does not explicitly represent the function²⁾. In this case, however, the transfer function is learned by GPR using the interactive feedback data; thus, the amount of data is typically small enough to compute the output of the transfer function in real-time. We employed an open-source implementation of GPR³⁾.

GPR is also employed to adapt the transfer function to the feedback in the following manner: Assume that the initial transfer function graph is as shown in Figure 5(a). This function is represented as a set of points, which are shown as the black dots in the figure. The user provides feedback by dragging the touch upwards to increase the output value (e.g., make the stroke thicker). Because this feedback applies to the last input, the goal is to increase the output values that correspond to the last input values (e.g., velocity values in the last stroke). We employ the range bounded by the minimum and maximum of the input values (the blue area in the figure) to identify the region of input values whose output values should be increased. The points in the region are translated in the positive direction of the y-axis and are represented by the red dots in the figure. We perform GPR to smoothly fit the data points to obtain the results depicted in the red curve in Figure 5. The repeated feedback causes the transfer function to gradually change toward the intention of the user.

Figure 5 Predicting feedback intention by GPR

We recruited participants who are versatile in using smart mobile devices and have experience in interaction design. The ages of the participants ranged from 24 to 31, and the average age was 27. Two females and two males participated. They were asked to change the initial transfer function to the ground-truth transfer function using only the Tester.

We provided two Samsung Galaxy Note devices to each participant side-by-side as shown in Figure 6. The Tester on Device A was initially set with the initial transfer function, as shown on the left of Figure 7. The Pen Tool on Device B was set with the ground-truth transfer function, as shown on the right of Figure 7, which the participant needed to reach using the Tester on Device A. At the beginning of the study, the participants utilized the Tester on Device A to modify the transfer function to attain the same quality of interactivity produced by the ground-truth transfer function. The Pen Tool on Device B was employed as a reference point. The actual graphs of the transfer functions were hidden from the participants; only the Tester was utilized for transfer function modification.

Figure 6 User study for the tool reliability evaluation

The participants were asked to only modify the velocity-thickness mapping to help them deliver a clear intention. The pressure-thickness mapping was fixed during the testing process.

Figure 7 Initial transfer function (left) and the ground-truth transfer function (right) of the velocity-thickness mapping

Participants were allowed as much modification as desired to achieve the ground-truth transfer function. Prior to the task, a 5-minute tutorial about the use of this tool was presented. During this tutorial session, the participants were allowed to freely use the tool to become comfortable enough to complete the testing task.

All four participants completed the task when they were convinced that the interactivity qualities of both Devices A and B were identical. The final transfer function produced by each participant was stored as a numeric text file that could be checked as a graph format on the Controller. To evaluate the proximity of the participants to the ground-truth transfer function, the root mean square error (RMSE) was calculated for every intermediate result produced via the feedback-based modification. This calculation was employed to measure the difference between the two transfer function graphs. The RMSE is calculated using the formula

where n is the number of sample points on a graph and y_j is the value of y on the graph when x=j. Because we have 100 points of (x, y) on the transfer function, we set n=100.

The RMSE was utilized to check whether the modification process via the interactive feedback method successfully enabled the users to obtain the ground-truth transfer function. Figure 8 shows the results of the RMSE change over the feedback cycles for each participant. The possible range of the RMSE value is between 0 and 100. The original RMSE between the initial transfer function and the ground-truth transfer function was 23.4. The RMSE values for all participants quickly reduced to less than 10 after two or three feedback cycles. After this point, the results show fluctuation with decreasing widths until the completion of the task. The final RMSE values for the participants were 3.2, 3.6, 5.1, and 8.2. One participant attained a value of 3.2 after only eight feedback cycles. Conversely, another participant used 24 feedback cycles to attain an RMSE of 5.1.

Figure 8 RMSE results: each line represents the trend of the RMSE values for each participant (y-axis) plotted for every feedback cycle during the task (x-axis)

The fluctuation does not imply that the modification process (i.e., the feedback cycles) does not help in obtaining the ground-truth transfer function. Even for small RMSE values, few input values were obtained for which the participants feel very different from the output of the ground-truth transfer function. When the participants discovered these cases, they attempted to modify the output value for these input values to smooth the differences. These modifications can yield slightly larger differences for other input values. Although no definitive answer regarding how small the RMSE value should be to allow the two functions to be perceived as identical has been provided, the study confirmed that the participants can achieve the intended transfer function by providing feedback over time based on both the RMSE reduction results and verbal confirmations.

We recruited ten designers with experience in interaction design or a related background and designers who were accustomed to using smart mobile devices. The participants included five males and five females. The average age of the participants was 24.9 years old (SD=2.1). Five of the ten participants were asked to modify the transfer function using the Tester and subsequently modify the transfer function with the Controller. The remaining five participants were asked to complete the same task in the opposite order. Similar to Experiment 1, the participants were instructed to modify the provided initial transfer function to achieve the ground-truth transfer function (Figure 9). For this study, two devices were provided (Figure 10): the first device was designated for modifying the transfer function and the second device was designated for checking the interactivity quality of the App (i.e., Pen Tool) that was set with the ground-truth transfer function. For the task with the Controller, the participants only used the Controller to change the transfer function. With the Tester, participants only used the Tester. All participants were aware that the ground-truth transfer function for both tasks was identical.

Figure 9 The initial transfer function (left) and the ground-truth transfer function (right) for Experiment

Figure 10 Task set-up for using the graph editing method (left) and the interactive feedback method (right)

A 10-minute tutorial session was provided to each participant as provided for Experiment 1. For both tasks, the graph of the ground-truth transfer function was not revealed to the participants. For Experiment 2, a slightly more difficult transfer function (Figure 9) than the transfer function for Experiment 1 was employed as the ground-truth to demonstrate a more realistic situation.

After completing the task using one of the methods, the participants completed a survey with the following questions:

• Q1. Was the method⁴⁾ helpful for completing the task?
• Q2. Was the method easy to use when completing the task?
• Q3. Was the method intuitive when applied to complete the task?
• Q4. Was the method useful as an interactivity design approach?

For each question, participants utilized a Likert scale that ranged from 1 to 5, where 1 denoted “not at all” and 5 denoted “very much.”

The data were analyzed from the survey using the Wilcoxon signed-rank test for data, which cannot be assumed to be parametric. Regarding the intuitiveness of the method using data from Q3, the test results showed significance (z=2.132, p=0.033<0.05), which concluded that the interactive feedback method is more intuitive than the graph editing method for successfully modifying a transfer function. Although significance was only demonstrated for Q3, the feedback method was evaluated more positively than the graph method, with the exception of Q4 (Figure 11). However, the average score of the answers was positive, as determined by an average score greater than 3.

Figure 11 Results from the survey in Experiment 2

In addition to the survey results, the ability of each participant to attain the ground-truth transfer function using each of the methods was examined. The results were obtained by measuring the RMSE between the initial graph and the final graph⁵⁾ produced by each participant (Table 2). The original RMSE between the initial transfer function and the ground-truth transfer function (Figure 9) was 48.529, whereas the final RMSE values for the participants from both cases ranged from 2.249 to 18.711. To determine the difference in the RMSE values between the case of the feedback method and the graph editing method, a paired sample t-test was employed. This test indicated distinct significance (t=3.198, p=0.011<0.05). Figure 12 shows that the feedback method yielded a much smaller average RMSE, which indicates that the participants attained the ground-truth transfer function via the interactive feedback method more closely than the graph-editing method. This result implies that the interactive feedback method is significantly more effective for achieving the desired transfer function than the graph method.

Figure 12 RMSE values from Experiment 2

We recruited five professional interaction designers with more than three years of professional interaction design experience. Their ages ranged between 25 years old and 42 years old. The participants included two female designers and one male designer, none of whom had a programming background; two additional male designers with programming experience were recruited.

The duration of the total study session per participant was approximately one hour to one and a half hours, including the tutorial time, the description of the mission using the Tester, and the debriefing interview. Because the focus was the effectiveness and usefulness of the Tester, only the Tester component was examined for this study. For the tutorial session, the focus was helping the participants become accustomed to the Tester and developing strategies for its use. Participants were given approximately 10 to 20 minutes for the tutorial session until they were comfortable enough to begin the main mission.

In the main session (Figure 13), the participants were instructed to design several interactivity qualities for certain situations for the Pen Tool. The missions were as follows:

Design any pen interactivity that you like most.
Design a pen interactivity that feels like a sharp pencil.
Design a pen interactivity that feels like a brush.
Design the interactivity of an imaginary pen that may not be able to exist in the physical world.

Figure 13 Main session of Experiment 3

The duration of the main session was approximately 30 to 40 minutes per participant. The participants were asked to think aloud during the design activities and rely on their design expertise to accomplish these missions.

During the debriefing session, each participant was interviewed about their experience with the Tester. The feedback included their impression, limitations, and the usability and usefulness of the interactive feedback method. In this session, the Controller of IC was briefly explained to extract deeper discussions about methods to modify interactivity. The duration of the session was approximately 15 to 20 minutes per participant.