Home > Vol. 29, No. 3

Characterizing Natural User Interface with Wearable Smart Watches
  • Raeseong Kang : Department of Creative Design Engineering, Ulsan National Institute of Science and Technology, Ulsan, Korea
  • Yoonki Ahn : Department of Creative Design Engineering, Ulsan National Institute of Science and Technology, Ulsan, Korea
  • Marije Meijer : Department of Industrial Engineering and Innovation Sciences, Eindhoven University of Technology, Eindhoven, the Netherland
  • Tufail Muhammad : Department of Creative Design Engineering, Ulsan National Institute of Science and Technology, Ulsan, Korea
  • Yoonee Park : Department of Creative Design Engineering, Ulsan National Institute of Science and Technology, Ulsan, Korea
  • Joonsang Baek : Department of Creative Design Engineering, Ulsan National Institute of Science and Technology, Ulsan, Korea
  • Chajoong Kim : Department of Creative Design Engineering, Ulsan National Institute of Science and Technology, Ulsan, Korea

Background The emergence of new interaction paradigms makes the use of technology inrealizing the users’ natural ways of exploring the real world the ultimate goal of designers today.Research on interactive and immersive technologies for user interface design is still a challenging chore for engineers and scientists when it comes to designing natural interaction for wearable smart devices. To address the challenge, our study aims to develop guidelines for design practitioners in designing wearable smart watches that could offer natural user experiences.

Methods To better understand natural user experiences with smart watches, an extensive literature review was conducted. A quantitative survey with 80 participants was conducted, of which the focus was on the expected functions of smart watches. Based on the survey results, we selected eight participants in terms of technology familiarity. To achieve the objectives of our research, three studies were conducted: a design workshop (Study 1), a cultural probe (Study 2), and a focus group interview (Study 3). The design workshop was created to figure out the needs and wishes people have forsmart watches. In the cultural probe, the focus was on figuring out natural interactions with smart watches. Finally, the focus group interview aimed to gain more insights from the results of the cultural probe in terms of natural user interaction with particular functions.

Results To address the needs and wishes of the users toward wearable smartwatches, we made a subdivision into three categories, such as functions, input measures, and notification (feedback) methods.According to the results, participants wanted weather notification, health monitoring, and identification as expected functions. Regarding the methodof input, voice command and touch screen were preferred. In order to get feedback, most of the participantswanted vibrations, particularly as a reaction tocompleting the commands or inputs. There was also a suggestion to customize their smart watch. For example, users can select the functions and build their own command system, and even choose the notificationmethods. Considering natural user interface with respect to functions (weather, answering a call, navigation, health monitoring, taking a picture and messaging), specific natural user interfaces were mentioned for particular functions.

Conclusions Throughout the study, people’s needs and wishes and their perceptions about natural interaction were identified and the characteristics of natural user interfacesweredetermined. Based on the results, tenperceptions were specifically defined to provide a better understanding of smart watches in terms of natural interaction: user affinity of form, awareness by familiarity, reality correspondence, behavioral extension, purpose orientation, easiness of performance, timeliness, routine acceptance, generality, and rule of thumb. In addition to that, natural user interfaces were categorized into five groups: user familiarity, realistic interaction, accomplishment assistance, contextual appropriateness, and social awareness. In this study,we tried to identify what constitutes anatural interaction and how it should be created. The limitations and further study are discussed at the end.

Design Ethics, Moral Matters, Internalization, Good Design.
pISSN: 1226-8046
eISSN: 2288-2987
Publisher: Korean Society of Design Science
Received: 06 Jan, 2016
Revised: 05 Apr, 2016
Accepted: 04 Jun, 2016
Printed: Aug, 2016
Volume: 29 Issue: 3
Page: 45 ~ 61
DOI: https://doi.org/10.15187/adr.2016.
Corresponding Author: Chajoong Kim (cjkim@unist.ac.kr)
PDF Download:

Funding Information ▼
Citation : Kang, R., Ahn, Y., Meijer, M., Muhammad, T., Park, Y., Baek, J., & Kim, C. (2016). Characterizing Natural User Interface with Wearable Smart Watches. Archives of Design Research, 29 (3), 45-61.

Copyright : This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/3.0/), which permits unrestricted educational and non-commercial use, provided the original work is properly cited.

1. Introduction

Owing to the development of new mobile and wireless technology, attention around wearable devices have been increasingly paid. Among many types of wearable devices, wearable smart watch has been often mentioned because it is familiar like our wrist watches and at the same time has various functions such as health monitoring, message, weather forecasting, and navigation. As those functions have been varied, the popularity of wearable smart watches has been increased and welcomed as a newbridge that brings technologies to the people (Isacson, 2015). However, as wearable smart watches are getting complicated in terms ofsuch functions, more cognitive load is required to use the product. What makes it worse is that the user should operate functions with a couple of buttons and the tiny touch screen of the wearable smart watch. As an alternative, natural user interface, which means interface that mainly operates with natural interactions through expressing gesture, acting or noticing the surroundings by sensors or corporal parts (Valli, 2008), is an emerging topic for human computer interaction technology. It has been developed with the aims of incorporating the wider scope of multimodal interaction (Liu, 2010). There have been some studies in which it was investigated how natural user interface is applicable with wearable smart watches. For instance, Knibbe et al. (2014) attempted to see the possibility of natural user interface with smart watches by extending the interactive surface for a smart watch to the back of the hand in order to reduce screen occlusion by enabling off-device gestural interaction. In another study done by Chi, Chen, Liu,and Chu(2007), our everyday objects in our living environment such as writs watch and fridge augmented with digital technology were investigated tosee whether augumented interaction matches or conflicts with its original (i.e. natural) interaction. Also, designs of wearable smart watches that aim to provide natural interaction may not properly fit, or even it might be unnatural to wear a smart watch on the wrist (Han, Han, & Kim, 2015).Besides them, there would be many other issues regarding delivering better user experience with wearable smart watches.

Therefore, this study attempts to figure out issues related to natural user interface of wearable smart watches. Based on the findings, it also aims to develop guidelines which can provide a better understanding of natural user interface of wearable smart watches with design practitioners. In order to achieve the goals, three research questions were formulated: 1) what are the people’s needs and wishes with wearable smart watches? 2) how can wearable smart watches ideally offer natural user experience considering the needs and wishes, and 3) in which way user interface can be designed in order to deliver natural interaction?

2. Literature Review
Natural User Interface

Natural interaction deals with the issues about skills of people used to interact with the real world and manipulate the traditional objects, considering ways of using current available technology without cognitive loads (Valli, 2008). In the past, researchers in the field of HCI have developed interface styles to provide natural interaction called WIMP(Window, Icon, Menu, and Pointer). The styles, however, was hard to be said as natural interaction, because keyboard or mouse itself was already far from natural way that human interacts (Tavares, Medeiros, de Castro, & dos Anjos, 2013). Computer scientists have developed new interaction styles (post-WIMP), to improve user experience and understand human psychology (Jacob et al., 2008). The work of Jacob et al. (2008) aimed to combine post-WIMP interfaces and tangible user interfaces, in order to understand and analyze what makes specific interaction intuitive and simple. In recent years, the efforts of research community in computer science seeking to provide more natural and human centered ways of interacting with technology have been introduced as ‘Natural User Interface’ as a new interaction standard after post- WIMP. Natural user interface is the use of natural interaction based on the concept of naturalness in its construction (Vieira et al., 2013).

Based on the above studies, it was found that the main concept in natural interaction is knowledge and skills of the physical world involved in human computer interfaces. We considered the notion of reality based on interaction in our study because the framework, put forward by Jacob et al. (2008),explains such interfaces that normally build on people’s understanding and skills of the everyday in the physical world. They argued that there are four classes of skills in the physical world, which are exploited by reality based interaction designs, because they allow people to take advantage of their real world skills in interacting with smart computing devices, reducing cognitive load and the required mental efforts. Furthermore, they claimed that this may enhance learning, improve performance and foster extemporization and exploration, since people are not knuckled down to learn interface-specific skills and expertise. These four classes of skills are described as follows (Jacob et al., 2008):

• Naïve Physics: people use their common knowledge of how the physical world works without knowing the mathematical laws of Physics.

• Body Awareness and Skills: People have a natural understanding of their own bodies and the abilities to move their bodies.

• Environmental Awareness and Skills: People have normal awareness associated with a physical environment and the ability to move around it and interact with it.

• Social Awareness and Skills: People use their social and collaborative skills to interact with other people in their surroundings.

It can be argued that the framework of Jacob et al. (2008) may explore the design of wearable smart watches to produce an intuitive user experience by swapping the physical world into an interface and getting digital information to the intimate world, where it turns observable and conformable.

Wearable smart watch interface as a new way to interaction

There were studies which noticed already that wearable smart watch or similar concept of device could be a next form of interaction platform. Several groups have addressed wearable smart watcheswith different functions and characteristics. For example, Raghunath and Narayanaswami(2002) described a wearable computing platform in a wristwatch and discussed the design of applications and user interface for it. They added that a wearable smart watch is a good format of wearable computer which is more portable and can be directly watched with a jerk of the wrist, compared with smartphones and other smart devices which are usually kept in pockets, and require to be picked up and turned on before opened.

Maurer et al. (2006) proposed a wearable watch named ‘eWatch’, which can notify user to aware context with its sensors. They described the functions of eWatch that it performed multi-sensory notification, such as recognizing and recording surrounding situational information. In addition to the various functions performed by a wearable watch, Martin (2002) discussed the wearable watch in terms of wearability as a function of fashion as well as human anatomy. He concluded that the user interface of a wearable watch would be simplified if people become familiar with computers and the societal impact of a wearable watch would be expanded in terms of information, because of the ubiquitous features of wearable computers. Degen et al. (2003) developed a wrist watch ‘Speedy’ for the purpose of a fall detector for elderly people. They aimed that the device gets alert or place a call to a service center in case the user were in trouble or unconscious after a fall.

The form of interaction in smart watch have also actively being studied. Some studies were conducted to address health monitoring and user contexts. Seong et al. (2014) proposed the design for a watch-type device to calculate and analyze activities of users. They developed a platform with various functions in order to understand the user context. This research aimed that the watch can provide various services for the old or the weak as well as ordinary people. Kim et al. (2007) introduced a wearable watch called ‘Gesture Watch’ that makes user to control other devices by hand gestures. The main problem with these devices was that various functions had made the device multifaceted and required interface learning. However, the work of Loclair et al. (2010) presented ‘PinchWatch’ that offerred single purpose of functions controlled by one hand to support micro interactions. This device offerred simplified functions which requires less effort to learn and no particular skills to operates the device. Bonino et al. (2012) introduced ‘dWatch’, a wearable device with the functions of notification and controlling other devices in a smart home system. Angelini et al. (2013) proposed the design process of a smart watch ‘bracelet’ for personal assistant in everyday life, and it had functions of health monitoring and medication reminder to enhance the life of elderly.

Knibbe et al. (2014) proposed a smart watch with new interaction method, using user’s hands and bimanual gestures. They claimed that the approach could reduce finger occlusion and add haptic feedback through body interaction.

Apart from those studies focusing on the functions of wearable watches, some studies have been conducted to address interface design of wearable watches and dealt with gesture and tactile interactions as well. Ashbrook et al. (2010) studied the interface design of touchscreen and explored users’ performance with a rounded touchscreen of a watch. Moreover, they developed a tool called ‘MAGIC’ to design gesture-based interactions and discussed the evaluation of the tool. Blasko&Feiner(2004) developed a system that created rich-functional multimodal interfaces. They discussed that the interaction systems could be relevant for wearable computer systems, such as wrist-worn watch-style devices. Rekimoto(2001) extended the scope of interaction to wrist strap by using sensors with high capacitance to detect large gestures in the air. They introduced two user interface designs ‘Gesturewrist’ and ‘Gesturepad’ with unobtrusive features, in order that people can control without any social pressure.

As a way to understand the concept of natural user interface, the four classes of skills described above were a useful source. Therefore, they were taken into account as dimensions to figure out what kinds of natural interaction were preferred and why the people came up with such interaction in the study. In addition, the ways of new interaction and characteristics for wearable smart watches were used to recognize current limitations and to explore future possibilities in the project.

3. Methods

The study aimed to identify needs and wishes in relation to wearable smart watches, to reveal optimum ways of delivering natural interaction, and based on the findings to make guidelines for the watch in terms of natural interaction. To achieve the goals, the study was comprised of four parts: survey, design workshop, cultural probe, and focus group interview. The survey was for getting to know expected functions of smart watch. The design workshop was planned to identify what user needs and wishes about wearable smart watches. A list of functions from the survey was provided with participants to sensitize their imagination in smart watch usage. Since participants’ opinions were gathered before they first experience a smart watch, the opinions could contain vague expectation or imagination about smart watch of potential users. Cultural probe was conducted given smart watches for hands-on experience to trace overall user experiences. By observing the users’ processes of getting closer with smart watch, user could think about natural and convenient ways to achieve a certain function. The data from cultural probe was a source of discussion in the focus group interview, of which focus was on needs and wishes and natural interaction after hands-on experience.

3. 1. Survey

In order to prepare for the design workshop and recruit participants, a survey was conducted with 80 respondents who were recruited through social networking services. They were asked to freely answer five functions which their smart watch should have. A total of 51 expected functions were collected from the survey. They were categorized into 20 function groups in terms of similarity (Table 1). These functions were suggested as reference to the participants when the design workshop was conducted.

Table 1
A list of the expected functions of smart watch

Function Description
Clock Watch Clock /Alarm
Notification Alerts Messages /Sound /Haptic
Phone Call Phone Call /Dial Option /Mobile Phone Call
Health Monitoring Health Care Management / Exercise Trackers
Message SMS /Mobile Phone SMS
Music Music Player /Mobile Phone Music Player
Map Navigation /GPS
Energy Management Power Management /Power Saving Option
Schedule Schedule /Planner
Memo Memo Note
Pairing Connect With Other Device /Mobile Phone
Payment E-Payment /Shopping
Internet Internet Connectivity / Web
Interface stuff Interface Styles / Interaction
Security Security Software /Antivirus /Firewalls
Recording Voice Voice And Sound Recorder
Photo Image Gallery
Flashlight Torch Light
Weather Weather Forecasting
Account book Bank Book /Transaction Recorder

Among 80 respondents from the survey, eight participants were selected considering gender balance and the extent to which technology is familiar. The gender balance was considered to avoid a possible bias resulting from a particular gender. Technology familiarity was considered because user groups of smart watch could vary from those who are very familiar to technology to those who are very unfamiliar to technology. It could influence the needs and wishes for smart watch. It was measured through a survey. Two extreme cases in terms of technology familiarity were considered and the four respondents whose scores were highest and the other four ones whose scores were lowest were selected at the end. The sample continuouslyparticipated in three studies: the design workshop, cultural probe, and focus group interview.

3. 2. Design Workshop

The design workshop was carried out for two major purposes: one was to catch users’ latent needs and wishes in relation to smart watch, and the other was to find out in which way they tended to operate or achieve the functions of a smart watch.


In order to let the participants make their own smart watch freely in a way to accommodate five functions they would like to use, materials were provided such as pieces of felt, glue, strap, paper board, and so on (Figure 1).

Figure 1 Materials provided for the workshop (left) and the design session in the workshop (right)

The participants were invited to an open studio at school of design and human engineering, UNIST, South Korea. The aim of the workshop and the definition of natural interaction were introduced. The workshop consisted of three parts: idea generation, design, and imaginary acting with retrospective interview. In the idea generation session, the list of functions (Table 1) was provided with the participants in order to let them choose five functions they like to experience with smart watch. With five functions, the participants generated ideas considering three steps: appearance and scenario at the starting phase of using a function, the way of interaction while using the function, and appearance and scenario at the end of using the function. Subsequently,theparticipants were asked to design their own smart watches based on the generated ideas (functions, interactions, interfaces, etc.) from the previous stage. To emphasize thinking beyond the box and give them insight to make their own models, we showed them the designs of various concepts and existing design of wearable smart watches. They made their own smart watch with various kinds of materials such as pieces of felt, glue, strap, and paper board (Figure 2). Finally, they presented their smart watch showing how it works and how he/she and the watch interact by acting. At the same time, questions on the reason of the design were asked and answered.

Figure 2 MOTO360 (left) and a page of the diary for cultural probe (right)
3. 3. Cultural Probe

The cultural probe was designed to observe user experience during a relatively long period, and to compare before and after getting close with smart watches. By giving the participants enough time and opportunity to get used to it, it was expected that they could easily figure out pros and cons of their smart watch.


Two types of smart watches were chosen considering the popularity in the market at the time of the study: MOTO360 and Pebble smart watches. A user diary was developed to trace the daily use and experience with the smart watch (Figure 2).


Participants were provided with the two types of smart watches to wear for a week. At the same time, we asked them to write a diary about their daily experience with the smart watch. For example, they were asked to describe basic functions (i.e. clock, calendar, weather, phone call, or music) they used on every single day, rate the difficulty of interaction with the device, and write comments on each of the functions. After collecting the diaries from the participants, a short retrospective interview was conducted to gain further insights from their one-week experience.

3. 4. Focus Group Interview

The focus group interview aimed to get in-depth insights of the experience of the participants from the cultural probe. It was done mainly focusing on the results of the cultural probe in which users’ experience and opinions were reflected. All the participants were invited to the open studio at the university (Figure 3). They were asked to share their overall experiences, feelings, and opinions on particular functions and interface of the smart watches, then asked to discuss and assess them in terms of naturalness (see the questions in Table 2).

Figure 3 Focus group interview conducted in the study
Table 2
User assessment in natural interaction

Follow up questions
1 Could you tell us about your experience, while using the smart watch for a week?
2 Are there any differences from your previous expectation? If so, what differences are there?
Topic of discussion: ‘natural interaction’ (intuitiveness + familiarity)
3 From this perspective, was there any specific experience which was considered as natural or unnatural?
4 You wrote down something in the diary. Was it natural/unnatural experience? Why? (This question was adjusted according to participants’ answer)
5 Do you think the way of manipulating the device is natural?
6 Was the device natural to accept it as part of your body?
7 When you use the device outside, was there any experience you felt it is unnatural?
8 When you use this device, was there any moment you aware of others?
9 If you can redesign, is there any part you would like to keep, adjust or remove?
10 Could you define ‘natural design’ with your own language?

3. 5. Data Analysis

From an open-ended question of the survey, a total of 51 functions which the participants would like to have with smart watches were collected. The functions were grouped into 20 categories in terms of similarity such as feature and purpose, and each category was ranked according to the frequency. As a result, five function categories which had the highest frequency were selected because it has a characteristic of an exploratory study.

Based on the video of the workshop, a protocol analysis was conducted focusing on functional features, way of input, and feedback. The data from the workshop was also used to qualitatively figure out the ways of intituitive interaction with smartwatch.

The data from cultural probes and retrospective interview were used as materials to facilitate the discussion in the focus group interview. The results of the focus group interview were qualitatively analyzed in a way to come up with desirable characteristics of natural interaction with smartwatch.

4. Results
4. 1. Needs and wishes on wearable smart watch

The results from the design workshop contained users’ expectations, needs and wishes on wearable smart watches. Through the functions gathered from the workshop, it was able to guess what kind of roles the participants expected on their smart watch, what they want to achieve with smart watch in their life, and what position of smart watch between similar wearable devices would be expected. The quick and dirty prototypes the participants made could reflect not only form and style that the users prefer, but also the natural ways of interaction with smart watches that they prefer(Figure 4).

Figure 4 Smart watches made by the participants in the design workshop

Since they were asked to explain howfunctions would be delivered with their prototypes, they should also consider characteristics of interfaces. In order to address the needs and wishes of the participants towards wearable smart watches, the results from the workshop sessionwere classified intofollowing three categories: functions, way of input and notification (feedback). The overall categorization of needs and wishes in three stages is shown in Figure 5.

Figure 5 Categorization based the participants’ needs and wishes in three stages

One of the most frequently mentioned function between the participants was a weather notification function, especially operated by voice commands in their wearable watches. They also expected to make phone call through their smart watches. Similar with many wearable devices lays their emphasis on health functions, they showed their wishes to have a smart watch with health monitoring. Identification functions to secure payment transactions by personalized fraud detection checks and biometric information was also mentioned as expected function. Navigation or map functions were also the demand of the participants to have in their own smart watches.

Ways of Input

Even though there were differences between ways of delivering functions, voice commands were most easily and commonly used as input system by the participants, followed by interactions through a touch screen. Those two ways were mentioned by almost all the participants, and only several participants additionally wanted to write on the screen with their finger or smart-pen. There were also some opinions using gestures, such as shaking wrist, to operate functions. Interestingly one participant suggested her own command system to start function. For example, if active movement such as swinging her arm is detected by smart watch, then smart watch would start to measure the biophysical stats with sensors.

Notification (feedback)

The most participants wanted to get feedbacks with vibrations, especially, as a sign of completing the commands or inputs. Screen itself were used as important element to inform for user. Some participants wanted to get feedbacks through the screen such as ‘popping up information’ for special occasion, and they also wanted to get ‘constant informing’ of the time or temperature information.

Auditory notification such as artificial voice or beep sounds were also considered as a way of notification.

4. 2. Natural interaction with respect to functions

In this section, results regarding natural interactions for each functions are discussed. The results from the cultural probe and focus group interview were mainly about users’ experiences on applications and interface of the smart watches in terms of their intuitiveness and comfortableness. The focus group interview was held after the cultural probe and as such we expected participants to have become biased towards the smart watch they were given; in the sense that they would show preference on the way of interaction used in their smart watch and indicate them as natural. This expectation, however, turned out to be false; all participants were thinking very out of the box. They were asked mostly about natural gestures to interact with the functions, and received the following most interesting responses:


In this first function, participants voiced that they would like to be able to ask the smart watch what the weather today would be. Instead of seeing the text on the screen they preferred to get a spoken reply. They wanted their social surroundings to know that they were talking with the smart watch, not just unilateral command, and believed that they would feel more comfortable to talk to the device if it was able to reply back naturally, similarly with a conversation.

In terms of gestural interactions, participants replied that stretching their arm towards the sky would be a natural way to activate the weather function, because of the relation between sky and the weather. Another interesting interaction was being aware of some actions by the device taken by the user typically before going outside. They gave the great example of putting on shoes; the participants were all from South Korea in which it is common to take of your shoes inside, so naturally when you go outside it is a natural thing to do.

Answering a call

The participants indicated that they wanted to be able to answer a phone call with their smart watch by shaking their wrist.

More interestingly, they also noted that when making a call with the smart watch a user has to bring the smart watch up to their mouth to speak into the speaker; therefore, they felt it was natural to answer the phone call by bringing the smart watch up to their mouth. After then they received the call by clearing their throat, blowing on the watch face, or ‘making a big voice’. A natural way they mentioned to decline the call was to cover the watch face with their hand; similar to how we can now mute a phone call by turning the phone to face a surface.


Similarly with way that currently navigation operates on a smartphone, users indicated they wanted the angle of the map to change according to the direction its user was facing. A participant related this to an interesting way of activating navigation; as the smart watch has to let user be aware of the direction the user is facing, they felt it would be natural for a user to activate navigation by spinning around on the spot; this interaction can be related to typical natural behavior when one is lost, in which we typically also spin around on the spot trying to regain our sense of direction.

Health monitoring

Participants related this function to current medical interactions between patient and doctor; they felt it would be natural to activate this function by ‘taking your pulse’. The strap of watch would be aware of the gesture and activate the function, or display the current pulse of the user. Considering that this function is also commonly used when exercising, they felt it was also natural to make a throwing gesture which could be made during exercises to activate the function, indicating to the device the user is going to be exercising.

Taking a picture

Participants indicated it would be natural for them to use gestures already commonly used in smartphones, such as grabbing and stretching to manipulate the angle. Interestingly, they also felt it would be natural if the device could recognize the ‘V’ sign commonly used when taking pictures.


It was also apparent that it is a difficult function to implement according to the answers given by participants. They replied that the smart watch could perhaps project a keyboard on the floor or on the opposite palm, or that users could use a pointer to operate a keyboard. Interestingly enough, similar to an answer given in the design workshop, writing letters on the screen was mentioned again as well.

5. Discussion

As above results shows, the optimal way of interactions could be differed by functions, as they had been delivered and achieved in different ways before theappearance of smart watch. In other words, however, it was also able to draw a conclusion that a natural interface could be designed if designers observe how potential users achieve their purposes in real world. As a conclusion of this study, features of users’ behavioral or verbal expressions and connectivity between reality and virtual interface were mainly observed and discussed. They can be summarized as follows:

5. 1. Natural interaction with wearable smart watches

There were totally ten categories of insight that was found in this research, and each of them was described with examples from the results of the study as follows:

• User affinity of form: vague expectations or preference on interaction elements or product appearance could be included in this category. For instance, someone would think ‘Watch is rounded shape’ because it is interchangeable by the experience. Their affinity of form influenced people’s perception about natural interaction.

• Awareness by familiarity: recognizing similarity affects user’s familiarity and people were likely to feel natural when they once recognized the similar factors. It is usually based on user’s experiences as some participant said, ‘It had many similar interfaces and functions compared to my smartphone so it is natural to use.’

• Reality correspondency: it is correspondence between virtual reality and reality. It is considered natural if the virtual reality reflects reality well so that user can recognize how to use or what it is naturally. It was mentioned as natural interaction that the navigation map guided from from user’s perspective so user did not need to worry about direction.

• Behavioral extension: it becomes just like extension of user’s behavior so user does not need to bother from devices. For instance, the exercise apps automatically turning on when the user starts to make movements of exercise can be an example.

• Purpose orientation: participants also thought that errors or malfunction hinder natural interaction. The function should be purpose oriented and always follow users’ intention.

• Easiness of performance: it was considered natural if user can manipulate without any instruction. Following is how participants described the natural interaction in terms of easiness of performance. ‘it seemed that it would light on if I push this button, and it worked’

• Timeliness: the timely feedback or function in appropriate situation was considered natural. Some participant mentioned ‘Rather than checking weather constantly, I just want to be naturally informed when weather changes significantly’ and the common need in this characteristics was receiving timely information without intentional manipulation.

• Routine acceptance: if the function hinders people’s routine, it would not be considered as natural interaction. Participants felt bothered if there was some difference from their normal activity. ‘I used not to wear watch during sleeping.’ or ‘It is needed to charge too frequently’ are examples for low routine acceptability of smart watch.

• Generality: people are likely to consider it is not natural if the using activity would be seen by people abnormal. For instance, someone mentioned ‘I will never use the voice command in public because people would see me strangely.

• Rule of thumb: contrary to ‘generality’, sometimes using watch before other people considered more natural. Someone said ‘I could read text message during the meeting time without any neglecting etiquette because it does not require checking cell phone’. This kind of social etiquette also can be an influencing factor on natural interaction.

5. 2. Characteristics of Natural User Interface

From the general point of view, the findings on natural interaction could be adopted into designing other kinds of interactive devices. The new categories were built based on the key concepts driven throughout user research to better fit generally on the context of natural user interface. Ten key concepts were grouped by using affinity diagram. As a result of grouping, five categories with significant different characteristics were generated as follows:

User familiarity

It combines two concepts, ‘User affinity of form’ and ‘Awareness by familiarity’. The user Familiarity involves the familiarity from experience and intuitiveness. In either way, people are likely to prefer familiar thing and consider it more natural.

Realistic interaction

This is made by combining ‘reality correspondency’ and ‘behavioral extension’. Reality Correspondence means criteria of how well user’s physical movement, behavior or cognition in reality connected and extended to the virtual manipulation. The more this gap between reality and virtually is closed, the more user feel the interaction is natural.

Accomplishment assistance

It come from integrating both ‘purpose orientation’ and ‘easiness of performance’. It is about the clarity of manipulation to achieve user’s purpose while using a product. Well-functioning can be considered taken for granted but it is significant because the error or malfunction critically influences on hindering naturalness. On the other hand, attaining his/her object without additional manual or instruction or organizing information simply but effectively can be considered natural.

Contextual appropriateness

Both ‘timeliness’ and ‘routine acceptance’ refers to the term. Contextual Appropriateness is criteria of how a device fits well to its using environment or context in daily life. People are sensitive to something different within their routine. If using device is different in a negative way, they would consider it unnatural.

Social awareness

It combines ‘generality’ and ‘rule of thumb’ from natural interaction. Social consciousness is criteria about how the using activity is naturally accepted in public. The function, characteristic of a product, or its usage needs to be considered in the social context as well. It is closer to social etiquette or people’s general awareness rather than specific rules. If it hinders this etiquette or it is seen abnormal, it is not natural to use and people would be afraid to use device.

Throughout the study, people’s needs and wishes and their perceptions about natural interaction were identified and the characteristics of natural user interface were made in order to find what the natural interaction is and what it will look like. Although there might be some skeptical perspectives as well, it is worth continuing this study and finding further opportunities in natural interaction of smart watch because of the needs of naturalness in wearable device. As further study, the categories characterizing natural user interface could be more elaborated and developed into an assessment tool of natural user interface.


We would like to thank all those who provided their time to participate in this study. This work was supported by the 2015 Research Fund (1.150128.01) of UNIST (Ulsan National Institute of Science & Technology).

  1. 1 . Angelini, L., Caon, M., Carrino, S., Bergeron, L., Nyffeler, N., Jean-Mairet, M., & Mugellini, E. (2013). Designing a desirable smart bracelet for older adults. Proceedings of the 2013 ACM Conference on Pervasive and Ubiquitous Computing Adjunct Publication - UbiComp' 13 Adjunct, 425-434. [https://doi.org/10.1145/2494091.2495974]
  2. 2 . Ashbrook, D., & Starner, T. (2010). MAGIC: A Motion Gesture Design Tool. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 2159-2168). New York, NY, USA: ACM. [https://doi.org/10.1145/1753326.1753653]
  3. 3 . Blasko, G., & Feiner, S. (2004). An interaction system for watch computers using tactile guidance and bidirectional segmented strokes. Eighth International Symposium on Wearable Computers, 1, 0-3. [https://doi.org/10.1109/iswc.2004.6]
  4. 4 . Bonino, D., Corno, F., & De Russis, L. (2012). DWatch: A personal wrist watch for smart environments. In Procedia Computer Science (Vol. 10, pp. 300-307). [https://doi.org/10.1016/j.procs.2012.06.040]
  5. 5 . Chi, P. Y. P., Chen, J. H., Liu, S. Y., & Chu, H. H. (2007). Designing Smart Living Objects-Enhancing vs. Distracting Traditional Human-Object Interaction. In Human-Computer Interaction. Interaction Platforms and Techniques (pp. 788-797). Springer Berlin Heidelberg. [https://doi.org/10.1007/978-3-540-73107-8_87]
  6. 6 . Degen, T., Jaeckel, H., Rufer, M., & Wyss, S. (2003, October). SPEEDY: A Fall Detector in a Wrist Watch. In ISWC (pp. 184-189).
  7. 7 . Han, S. C., Han, Y. H., & Kim, H. S. (2014, December). Characteristics and perspectives of wearable smart devices and industrial ecosystem. In Parallel and Distributed Systems (ICPADS), 2014 20th IEEE International Conference on (pp. 911-914). IEEE. [https://doi.org/10.1109/PADSW.2014.7097907]
  8. 8 . Isacson, D. (2015). Application development for smartwatches. Retrieved from http://www.divaportal.org/smash/get/diva2:826851/FULLTEXT01.pdf.
  9. 9 . Jacob, R. J., Girouard, A., Hirshfield, L. M., Horn, M. S., Shaer, O., Solovey, E. T., & Zigelbaum, J. (2008, April). Reality-based interaction: a framework for post-WIMP interfaces. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 201-210). ACM. [https://doi.org/10.1145/1357054.1357089]
  10. 10 . Kim, J., He, J., Lyons, K., & Starner, T. (2007). The Gesture Watch: A wireless contact-free Gesture based wrist interface. In Proceedings - International Symposium on Wearable Computers, ISWC (pp. 15-22). [https://doi.org/10.1109/iswc.2007.4373770]
  11. 11 . Knibbe, J., Martinez Plasencia, D., Bainbridge, C., Chan, C. K., Wu, J., Cable, T., ... & Coyle, D. (2014, April). Extending interaction for smart watches: enabling bimanual around device control. In CHI'14 Extended Abstracts on Human Factors in Computing Systems (pp. 1891-1896). ACM. [https://doi.org/10.1145/2559206.2581315]
  12. 12 . Liu, W. (2010). Natural user interface-next mainstream product user interface. In 2010 IEEE 11th International Conference on Computer-Aided Industrial Design & Conceptual Design 1.
  13. 13 . Loclair, C., Gustafson, S., & Baudisch, P. (2010). PinchWatch: a wearable device for one-handed microinteractions. In Proc. MobileHCI (Vol. 10).
  14. 14 . Martin, T. L. (2002). Time and time again: Parallels in the development of the watch and the wearable computer. In Wearable Computers, 2002.(ISWC 2002). Proceedings. Sixth International Symposium on (pp. 5-11). IEEE.
  15. 15 . Maurer, U., Rowe, A., Smailagic, A., & Siewiorek, D. P. (2006, April). eWatch: a wearable sensor and notification platform. In Wearable and Implantable Body Sensor Networks, 2006. BSN 2006. International Workshop on (pp. 4-pp). IEEE. [https://doi.org/10.1109/BSN.2006.24]
  16. 16 . Raghunath, M. T., & Narayanaswami, C. (2002). User interfaces for applications on a wrist watch. Personal and Ubiquitous Computing, 6 (1), 17-30. [https://doi.org/10.1007/s007790200002]
  17. 17 . Rekimoto, J. (2001). Gesturewrist and gesturepad: Unobtrusive wearable interaction devices. In Wearable Computers, 2001. Proceedings. Fifth International Symposium on (pp. 21-27). IEEE. [https://doi.org/10.1109/iswc.2001.962092]
  18. 18 . Seong, K. E., Lee, K. C., & Kang, S. J. (2014, January). Self M2M based wearable watch platform for collecting personal activity in real-time. In Big Data and Smart Computing (BIGCOMP), 2014 International Conference on (pp. 286-290). IEEE.
  19. 19 . Tavares, T. A., Medeiros, A., de Castro, R., & dos Anjos, E. (2013). The use of natural interaction to enrich the user experience in telemedicine systems. In HCI International 2013-Posters’ Extended Abstracts (pp. 220-224). Springer Berlin Heidelberg. [https://doi.org/10.1007/978-3-642-39476-8_46]
  20. 20 . Valli, A. (2008). The design of natural interaction. Multimedia Tools and Applications, 38(3), 295-305. [https://doi.org/10.1007/s11042-007-0190-z]
  21. 21 . Vieira, E. S. F., Passos, M.,Oliveira, S. S.,Melo, E. G.,Motta, G. H. M. B., ...& Souza, Filho, G. L. D. (2013). Uma Ferramenta para Gerenciamento e Transmissão de Fluxos de Vídeo em Alta Definição para Telemedicina. Salão de Ferramentas do SimpósioBrasileiro de Redes de Computadores, 2012.