Design for Sharing Emotional Touches during Phone Calls: A Quantitative Evaluation of Four Tactile Representations

1. Introduction

As the smart phones became one of the essential medium in our everyday lives, the ways for delivering the users’ intention and emotions are diverging. Beyond conveying words and information, the phones help us to feel connected with others (Normal, 2004) as our life style is being more separated and individualized. In particular, still the most of our remote conversation rely on traditional voice calls, and the emotional communication using the voice depends on phonetic features such as amplitude variation, pitch inflections, tempo, duration, filtration, tonality and rhythm (Knapp and Hall, 1978). To provide harmonized conversation at a distance, it is important to consider verbal and non-verbal behaviors such as facial expressions, touching behaviors and gestures. Compared to other non-verbal signals, our daily phone conversation is restricted to touch or to be touched when human-to-human interactions are being mediated by communication devices. In order to make our mediated daily communication more private and emotional, it is important to consider touch for exchanging contextual and nonverbal cues (Wang and Quek, 2010). The addition of a touch channel to communication devices can be a unique and immediate channel for affective interaction when the context is required. Since touch is indispensably related to intimacy, touch allows for personal and intimate interactions in ways that words, sounds or images presumably cannot (Haans and Ijsselsteijin, 2006). Video phones are progress in the aspect of supporting non-verbal communication, but they introduce other problems of privacy and an unnatural communication posture. On the other hand, the posture of holding a voice phone on cheek has been naturally accepted for long time by most users without the problems of privacy and additional physical demands in hardware and users interaction. In addition to make emotional mobile phone communication more natural and effective, it is important to consider maintaining natural voice communication by keeping the receiver on the ear and the transmitter on the mouth (Fukumoto and Tonomura, 1999) - strictly speaking, a device on the cheek.

In this paper, we present ways to deliver touches, especially, pat, slap, tickle and kiss while holding phones to the cheeks as is done in typical phone calls. We report a result of a quantitative evaluation using CheekTouch interaction technique (Figure 1), to discover ways to deliver pat, slap, tickle and kiss by using the technologies (vibration motors and sound) in the existing phones. Lastly, by suggesting possible scenarios of how pat, slap, tickle and kiss with CheekTouch can be used in phone calls, we discuss the potentials of future tactile phone conversation and issues for improvements.

2. Related Work

A representative framework related to our study was proposed by Haans and Ijsselsteijn (2006), focusing on mediated social touch which supports non-verbal and emotional communication between remote persons using tactile input-output devices. There are also other studies that developed special input/output devices for mediated social touch. Brave et al. (1998) made a device having three rollers to allow users to convey their presence to each other by moving the device and receiving the movement in real time. Hashimoto and Kajimoto (2008) suggested a method to deliver emotional gestures (e.g., tapping, tickling, pushing and caressing) by sensing using a proximity sensor and by rendering using a woofer. Tsetserukou et al. (2010) devised a wearable device to be worn on users’ chest to make them feel hugging and hitting of their digital avatars in Second Life with vibration and heat. Wang and Quek (2010) found that if the tactile channel is added to voice communication then it amplifies positive emotions and reduces negative emotions by using an armband-type inflatable wearable tactile device. However, their idea involves a practical problem the user should wear additional equipment other than a mobile phone. Several studies have applied current mobile phone technologies to share touches for remote interpersonal communication. Chang et al. (2002) developed a pair of pads on the desk that can deliver the intensity of each finger’s pressure with vibrotactile feedback during a voice call. Brown and Williams developed a mobile messaging system called Shake2Talk (2007) that shakes the phone differently to create and deliver audio-tactile message. Their results showed that Shake2Talk could deliver positive emotions like playful messages or practical information like call-for-action messages (2009). Also, Park et al. (2012) have showed the qualitative results through investigating the possible usages of vibrotactile stimulation during real-time phone conversations.

New forms of tactile feedback technologies have been developed to give actual tactile feelings like human touch in HCI. Harrison (2009) and Wang (2010) used thin shape memory alloys (SMA). Because their movement is delicate, slow, and quiet, actuators using SMA do not cause discomfort to users wrapped with fabrics like woven elastic, spandex, cotton, and latex, thus they are good for developing wearable devices. Recently, Park et al. (2015) have investigated the potentials of tactile and gestural phone conversations by using SMA embedded shape-changing mobile devices.

Poupyrev et al. (2002) developed thin paper-like tactile actuators that can output subtle and ambient tactile feelings without giving displeasure to the users. Li et al. (2008) developed a prototype to provide tapping and rubbing feelings using a voice coil motor and a small hammer. Bau et al. (2009) developed a thin tactile output device to provide poking feelings by applying the principles of electromagnetic stimulation. In our study, we used coin-type vibrotactile actuators that were widely used by HCI researchers (Change et al. 2002, Tsetserukou et al. 2010) because they are already embedded in current mobile phones and can be applied more easily than other actuators. Additionally, their vibration is strong enough to stimulate the cheek; they are cheap and small enough to be used in arrays on existing equipment. They can also be used in sponge or cloth to make various forms of tactile output devices.

The studies above shows interesting insights on how touch can assist social interaction and suggest the potential of using tactile mediators in emotional communication remotely; however, the mapping between expression and perception of specific touches using the existing technologies of current phones has not been discussed. In addition, most studies have proposed alternative devices for sharing touches which might require unnatural ways of interaction and demand that users carry specific devices for sharing touch. In our study, CheekTouch and our evaluation results clearly show the feasibility and possibility of sharing the four touches (pat, slap, tickle and kiss) during a conventional way of voice phone conversations.

3. Selection of Four Tactile Representations: Pat, Slap, Tickle and Kiss

In this research, we focused on specific touches that are used in everyday human-to-human interactions rather than investigating abstract touches. Among various touch behaviors, first we considered touches that can be expressed and felt in a same time during phone calls. Those were chosen among sixteen touches that were most common in Western culture (Argyle, 1988). Second, we selected the types of touches that can be expressed with multi-finger gestures during mobile phone conversations (holding the device on the cheek). We expected that users can intuitively express various touch behaviors on mobile phones’ screen using the same fingers that hold the phone while speaking in the typical fashion. Then, we classified the types of touches that can be outputted to the cheek, because we naturally use the cheek for mobile voice communication, rich receptors on the cheek are suitable for detecting various affective finger gestures. Finally, we selected four touches that can be effectively felt with vibration motors that are normally used in phones and are easy to distinguish (pat, slap, tickle and kiss). It was because these touches have differences in frequency and duration. We summarized social meanings of those touches to understand their importance in our face-to-face communications (Pisano, 1986) (Table 1).

Table 1.

The social meanings of touches on the cheeks

4. CheekTouch

To deliver vibrotactile representations, we designed and developed an interaction technique, CheekTouch (Figure. 1), that can express and sense pat, slap, tickle and kiss during a phone call. The detail design and technical implementation have been introduced in a previous study (Park et al, 2010, 2012), and in this study the device was used as an apparatus in the experiment. In sum, CheekTouch is a way of sharing touches by transmitting multi-finger touch gestures with sender’s touch screen on a mobile phone to a receiver’s cheek using vibrotactile and sound stimulations, in real time, during a call. There are few important considerations for this interaction technique, and those are as on-the-cheek interaction, one-hand interaction, localized tactile feedback in real-time and sound stimulation of touches behaviors. To deliver different types of vibrotactile patterns, nine coin-type vibrotactile actuators (10mm diameter, 3mm thickness) in a 3x3 grid. The distance between each of the nine actuators was 7mm because the two-point discrimination threshold on the cheek was 7mm average for cheeks (Weinstein, 1968).

Figure 1

Representative examples to deliver pat, slap, tickle and kiss during phone conversations using vibrotactile stimulations and sound effect

5. Evaluation

In this section we evaluate how well CheekTouch can deliver pat, slap, tickle and kiss to the users and find best ways to deliver the four touches by sound and vibrotactile stimulation provided from CheekTouch. This allows users to share pat, slap, tickle and kiss over phone calls, gives a sense of feasibility in doing so.

Figure 2

Left: Holding CheekTouch and providing stimulations, Right: CheekTouch device (3x3 vibrotactile motors on the cheek-side, touch screen on the holding finger-side)

5. 3. Stimuli

• Sound only: The sound stimulations of pat, slap, tickle and kiss were given using the natural effect sound of the four touches that are recorded from when human touches the other person on the cheek.

• Vibrotactile only: The used vibrotactile stimulation of pat, slap, tickle and kiss was expressed through the 3x3 vibrotactile display of the CheekTouch device. The vibrotactile stimulations of the four touches (Figure 3) were average patterns which were extracted from users who participated in the preliminary experiment (See details in APPENDIX).

• Combined stimulation: The combined stimulations of four touches were given by synchronous stimulation of sound and vibrotactile stimulation (Figure 3).

Figure 3

Sound / vibrotactile stimulation of pat, slap, tickle and kiss

6. Results

First, we used confusion matrix (Table 2) to know the users have felt pat, slap, tickle and kiss with no confusion with other touches. Then we analyzed the data with the one-way, repeated measures ANOVA (within subjects) to know what type of stimulations had delivered the feeling of pat, slap, tickle and kiss well (Figure 4).

• Pat: There was significant difference among the three types of stimulations regarding the pat (F1, 59=10.199, p < 0.001). The combined stimulation for pat delivered their feelings well in the aspect of small confusion with other touches (51% perception (Table 2)) and delivering its realistic feelings (M=3.700, SD=0.962) (Figure 4).

• Slap: There was no significant difference among the three types of stimulations regarding the slap (F1, 59=2.210, n.s.). However, the result from confusion matrix shows sound stimulation of slap was perceived well (48%) without serious confusion. But kiss caused confusion (22%) when sound stimulation of slap was given.

Combined stimulation for slap is recommended even though the sound of slap showed least confusion, because the confused feeling was punch and pat that is similar to slap in some respects.

• Tickle: In case of tickle, there was significant difference among the three types of stimulations (F1, 59=3.557, p<0.05). However, the result from the confusion matrix shows tickle was perceived well with vibrotactile only (31% perception) and delivered its feelings (M=2.300, SD=1.442) (Figure 4) relatively well among three types of stimulations.

• Kiss: There was significant difference among the three types of stimulations (F1, 59=248.091, p<0.001). Kiss sound alone delivered its meaning clearly and had small confusion with other touches (74% perception (Table 2)) and delivering its realistic feelings (M=4.020, SD=1.033). Kiss was not conveyed at all with only vibrotactile stimulation (M=1.070, SD=0.362) (Figure 4).

Figure 4

Average level of perception (1~5 likert scale) of pat, slap, tickle and kiss according to three stimulations.

Table 2.

The confusion matrix of the four touches. The number of selection with score higher than 2 point (the percentage of occurrence of the user selection)

7. Discussion

The results of our evaluation demonstrate that pat, slap, tickle and kiss with combined stimulation were not always better to convey their realistic feelings compared to sound/vibrotactile only. In other words, the amount of information that is transferred was not adequate to feel the touches. Additionally, the sound stimulations of four touches were not always dominant to feel the touches. Bresciani et al (2005) argued that when the tactile and sound signal is given at the same time, people were more affected by the sound signal rather than the tactile signal. In case of pat and kiss, sound helped to perceive their feelings; however tickle was perceived well with vibrotactile only. On the other hand, touch like kiss was delivered more clearly with only sound rather than with vibrotactile stimulations. This finding might induce limitations in delivering some touches with vibrotactile motors and provide a certain guide in delivering touches during phone conversations. This shows that technologies that are in the mobile phones might decide the types of touches that can be conveyed, and in current situation, it is expected that there will be touches that will be delivered more clearly with vibrotactile motors or sound, also the touches used in certain context of conversation are needed to be investigated. In relation with this, we could see that there are similar touches which could be distinguished better in the context of a conversation. The most representative example is slap and punch, because they are rather similar already. Users were confused when the sound of touches were too similar. 22% confused slap sound as light kiss. And when tickle was delivered with sound only, stroke (32%) and groom (29%) caused dominant confusion to feel tickle (Table 2). In addition, the rotating feature of vibration actuator affected to feel touches. When kiss was delivered with vibrotactile only, shake (26%) and hold & shake (23%) caused dominant confusion to feel kiss (Table 2). The feeling of tickle with vibrotactile only was weak (M = 2.300) compared to kiss sound/pat combined, however there was 31% perception (Table 2), and many users mentioned from the interview that the tickle feeling was playful and fun. These findings provide the potential to use tickle in joyful/playful conversations or shifting negative conversations in a phone call.

8. Possible Usage Scenarios

We considered scenarios how those touches can be used in phone calls.

• Direct touch expression: Typical mobile phone calls are limited to express and feel touches. However, in our face-to-face communications we express our feelings or opinions by accompanying touch with words. Sometimes, during phone conversations we express the act of touching or sound of touches with voices. At this point, if users express pat, slap and kiss with words by CheekTouch and the other recognizes them in real time, the dialogue between the people can be more effective. In addition, when the user wants to play a joke or pester to the other, tickle or slap could support emotional conversation.

• Enriching emotions: Depending on the person or situation touching another’s cheeks could amplify positive / negative emotions. Emotional conversations that become richer can contain both positive and negative emotional exchange. Wang and Quek (2010) showed that remote touch reduces sadness but in Chang et al’s (2002) research, remote touch can be used to amplify the emotions. If tickle is delivered in a phone call, it could amplify joyful emotions. When a slap or simple pat is delivered to the other’s cheeks in negative contexts, those can amplify the discussion or, when a pat or kiss is delivered, they can offer comfort in melancholy moments.

• Replacing spoken words: As the environment/context changes in face-to-face conversations, we use touch when we think it can be more effective and emotional than saying words. For example, patting someone on the back elicits stronger emotional comfort than simply saying a word like “Cheer up!” When the mother pats her child in a phone call, the child can feel more relaxedness than saying words even if they are in a remote situation. According to Chang et al (2002) and Brown et al (2009), touch can be used for calling attention of the others, instead of saying with words, they can pat and slap for calling attention of the other during phone conversations.

9. Conclusion

We presented ways to deliver pat, slap, tickle and kiss on cheek for phone calls using our CheekTouch interaction technique. An evaluation was conducted to propose the most appropriate type of stimulation to deliver pat, slap, tickle and kiss using vibrotactile motors and sound which are the technologies in the traditional smart phones. We showed that best ways to deliver pat was vibrotactile combined with sound; however tickle was delivered best with only vibrotactile. Kiss and slap, on the other hand, were delivered best when there was only a sound. Although, it is still challenging to share touches at a distance through phones, however we believe the results upon the four touches might augment the phone conversation in richer way. Considering the current mobile phones, finding ways to deliver the touches with existing technologies (vibrotactile motors and sound) will show higher feasibility, also believe the certain level of ambiguity will guide the users to interpret the stimulation in their own way with the partner during their specific conversation context.

Acknowledgments

This work was supported by the 2015 Research Fund(1.150065.01) of UNIST(Ulsan National Institute of Science and Technology).

References

• Argyle, M. (1988). Bodily communication. Methuen.
• Bau, O., Petrevski, U., & Mackay, W. (2009, April). BubbleWrap: a textile-based electromagnetic haptic display. In CHI'09 Extended Abstracts on Human Factors in Computing Systems (pp. 3607- 3612). ACM. [https://doi.org/10.1145/1520340.1520542]
• Brave, S., Ishii, H., & Dahley, A. (1998, November). Tangible interfaces for remote collaboration and communication. In Proceedings of the 1998 ACM conference on Computer supported cooperative work (pp. 169-178). ACM. [https://doi.org/10.1145/289444.289491]
• Bresciani, J. P., Ernst, M. O., Drewing, K., Bouyer, G., Maury, V., & Kheddar, A. (2005). Feeling what you hear: auditory signals can modulate tactile tap perception. Experimental brain research, 162(2), 172-180. [https://doi.org/10.1007/s00221-004-2128-2]
• Brown, L. M., Sellen, A., Krishna, R., & Harper, R. (2009, April). Exploring the potential of audio-tactile messaging for remote interpersonal communication. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 1527-1530). ACM.
• Brown, L. M., & Williamson, J. (2007). Shake2Talk: multimodal messaging for interpersonal communication. In Haptic and Audio Interaction Design (pp. 44-55). Springer Berlin Heidelberg. [https://doi.org/10.1007/978-3-540-76702-2_6]
• Chang, A., O'Modhrain, S., Jacob, R., Gunther, E., & Ishii, H. (2002, June). ComTouch: design of a vibrotactile communication device. In Proceedings of the 4th conference on Designing interactive systems: processes, practices, methods, and techniques (pp. 312-320). ACM. [https://doi.org/10.1145/778712.778755]
• Fukumoto, M., & Tonomura, Y. (1999, May). Whisper: a wristwatch style wearable handset. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems (pp. 112-119). ACM. [https://doi.org/10.1145/302979.303009]
• Haans, A., & IJsselsteijn, W. (2006). Mediated social touch: a review of current research and future directions. Virtual Reality, 9(2-3), 149-159. [https://doi.org/10.1007/s10055-005-0014-2]
• Harrison, C., & Hudson, S. E. (2009, April). Texture displays: a passive approach to tactile presentation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 2261-2264). ACM. [https://doi.org/10.1145/1518701.1519047]
• Hashimoto, Y., & Kajimoto, H. (2008, August). Emotional touch: a novel interface to display emotional tactile information to a palm. In ACM SIGGRAPH 2008 new tech demos (p. 15). ACM. [https://doi.org/10.1145/1401615.1401630]
• Knapp, M., & Hall, J. (1978). Nonverbal communication in human interaction. Holt, Rinehart and Winston New York.
• Li, K. A., Baudisch, P., Griswold, W. G., & Hollan, J. D. (2008, October). Tapping and rubbing: exploring new dimensions of tactile feedback with voice coil motors. In Proceedings of the 21st annual ACM symposium on User interface software and technology (pp. 181-190). ACM. [https://doi.org/10.1145/1449715.1449744]
• Norman, D. A. (2005). Emotional design: Why we love (or hate) everyday things. Basic books.
• Park, J., Park, Y. W., & Nam, T. J. (2014, April). Wrigglo: shape-changing peripheral for interpersonal mobile communication. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 3973-3976). ACM. [https://doi.org/10.1145/2556288.2557166]
• Park, Y. W., Bae, S. H., & Nam, T. J. (2012, May). How do couples use CheekTouch over phone calls?. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 763-766). ACM. [https://doi.org/10.1145/2207676.2207786]
• Park, Y. W., Baek, K. M., & Nam, T. J. (2013, April). The roles of touch during phone conversations: long-distance couples' use of POKE in their homes. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 1679-1688). ACM. [https://doi.org/10.1145/2470654.2466222]
• Park, Y. W., Lim, C. Y., & Nam, T. J. (2010, April). CheekTouch: an affective interaction technique while speaking on the mobile phone. In CHI'10 Extended Abstracts on Human Factors in Computing Systems (pp. 3241-3246). ACM. [https://doi.org/10.1145/1753846.1753965]
• Pisano, M. D., Wall, S. M., & Foster, A. (1986). Perceptions of nonreciprocal touch in romantic relationships. Journal of Nonverbal Behavior, 10(1), 29-40. [https://doi.org/10.1007/BF00987203]
• Poupyrev, I., Maruyama, S., & Rekimoto, J. (2002, October). Ambient touch: designing tactile interfaces for handheld devices. In Proceedings of the 15th annual ACM symposium on User interface software and technology (pp. 51-60). ACM. [https://doi.org/10.1145/571985.571993]
• Tsetserukou, D., Neviarouskaya, A., Prendinger, H., Ishizuka, M., & Tachi, S. (2010, April). iFeel_IM: innovative real-time communication system with rich emotional and haptic channels. In CHI'10 Extended Abstracts on Human Factors in Computing Systems (pp. 3031-3036). ACM. [https://doi.org/10.1145/1753846.1753911]
• Wang, R., & Quek, F. (2010, January). Touch & talk: contextualizing remote touch for affective interaction. In Proceedings of the fourth international conference on Tangible, embedded, and embodied interaction (pp. 13-20). ACM. [https://doi.org/10.1145/1709886.1709891]
• Weinstein, S. (1968). Intensive and extensive aspects of tactile sensitivity as a function of body part, sex and laterality. In the First Int'l symp. on the Skin Senses, 1968.