![Archives of Design Research](/images/basic/pubreader_back.gif)
The Effect of Handwriting and Physical Representation in Computer-Mediated Text Communication
Background While digital text media (e.g., e-mail, instant messaging) is widely used for communication due to its swiftness and convenience, handwriting is still preferred in situations that call for sincerity (e.g., love letter, greeting card). However, outside particular situations, handwritten media is seldom used due to its slowness. In this paper, we investigated the effect of handwriting and physical representation in digital text communication using a novel prototype of a new hybrid communication medium, Tele-Handwriter (TH), which allows one’s handwriting to be instantly and physically delivered to the opponent through pen and paper.
Methods To investigate the user experience of Tele-Handwriter (TH), especially on the effect of using handwriting and physical representation, we conducted a comparative user study with eight couples who were in a relationship, and weused three different types of text media(e.g., TH, Shared Canvas, instant messaging). We asked them to have casual conversations and to write a long and affectionate message to each other. Participants were interviewed about their experience and the pros and cons of each medium. The interviews were thematically analyzed using a bottom-up affinity diagram.
Results The findings on TH largely regarded increased intimacy, wholly delivered effort and sincerity, and contextual cues being naturally added to the message. The use of handwritingenabled text-based communication with messages having a distinct characteristic that naturally incorporates the sender’s identity, the attitude in writing, and effort. Furthermore, physically representing the message using a pen and paper was effective in delivering the sender’s effort and commitment to the receiver in its entirety. It also allowed the sender to fully deliver the contextual cues within the message.
Conclusions We investigated the effect of adopting handwriting and physical representation in text-based communication using a novel hybrid communication medium, TH. We identified the effects of handwriting and the physical medium, which are the essential features of TH. Based on these findings, we discussed the applicable area of TH and the design insights to design physical handwriting systems for computer-mediated text communication.
Keywords:
Handwriting, Emotional Text Communication, Computer-mediated Communication, Instant Messaging, Research through Design1. Introduction
Digital text media (e.g., e-mail, instant messaging) is widely used for communication due to its swiftness and convenience. However, despite such advantages, using digital text media in situations that call for sincerity is not yet considered appropriate (Leppänen, 2001). With no situational flexibility, messages sent in plain text are often considered emotionally dull or even insincere by the receiver. In situations where the sender wants to make sure that sincerity is conveyed (e.g., a love letter or greeting card), a handwritten mediumor a message handwritten on paper is considered more appropriate and sincere by today’s standards (Höflich and Gebhardt, 2005).
Handwritten media, by default, requires effort from the sender, including the act of writing. This kind of effort acts as a means to communicate the sender’s sincerity to the receiver. Also, handwritten media presents the sender’s unique handwriting, which is advantageous when sending a personalized message (Gooch and Kelly, 2016). Furthermore, the physicality of media adds special value to the message, such as the sense of texture and the desire for possession. However, handwritten media must be delivered directly from the sender to the receiver or via a consignment service, and the message’s delivery is inevitably slow. Due to this downside compared to digital text media, which boasts quick delivery, handwritten media is seldom used these days.
Our study originates from the idea of complementing the pros and cons of the two media. Similar attempts on the combined use of digital text media and handwriting have been made. First, the Apple Inc. iOS10 function known as Write It Yourself (iOS, 2017) can transmit any handwritten message on a device’s screen to a receiver’s screen. However, simply presenting the handwritten message on the display fails to capture the value and effect of handwritten media, such as physicality. Second, The Indian Handwritten Letter Co. (2017) and Bond (2017) provide services where a sender simply gives content, and the handwriting is completed for the sender and is delivered to the receiver. Unfortunately, the slowness of delivery still remains problematic, and it is somewhat deceitful because the sender’s effort has been replaced by someone else’s effort. Lastly, from the research perspective, while the studies investigating the relationship between digital text media and handwritten media support many of the current research’s premises, they could not provide a direct solution. A case that supports instant communication between remote users by simply handwriting a message on paper is yet to be investigated.
In this paper, we present Tele-Handwriter (TH), a text-based communication system that allows remote users to instantly communicate with handwriting on a paper. The TH system consists of a pair of terminals connected by a network, and when the user handwrites messages on paper at one terminal, another terminal physically reproduces the message on paper by a pen. In order to investigate the effect of handwriting and the physical representation of TH, we have conducted a comparative study using three different text-based communication media, including TH, IM, and Shared Canvas (SC). We found that using handwriting increases the effect of intimacy and that using a physical medium wholly delivers the sender’s effort and sincerity and adds new contextual cues to the message. We also discuss the issues increatingemotional computer-mediated text communication and future works.
2. Related Works
Nowadays, digital text media (e.g., e-mail, IM) takes up a dominant portion of our daily communication. Besides the efficiency of communication, text-based media is also preferred due to its simplicity (i.e., unnecessary visual and audio cues are excluded from communication). The preference of text-based media over face-to-face communication hasalso been found (Valkenburg and Peter, 2009). While various messages are continuously delivered through digital text media, they often fail to effectively deliver the sender’s emotional information (i.e., sincere or heartfelt message). Similary, Leppänen (2001) reported that people consider e-mail an inappropriate media in cases when sincerity is required. Furthermore, Riche et al. (2010) found that digital messages are considered less valuable than conventional handwritten letters because they are easily written with less effort.
Accordingly, there has been a strong need for digital text media that supports rich emotional communication. As the convenience and efficiency of IM arestill significant advantages, various studies have been conducted to supplement emotional limitations and to expand usage. The early attempts include emojis and kinetic typographies (Bodine and Pignol, 2003; Forlizzi et al., 2003; Lee et al., 2002). Furthermore, MobiMood (Church et al., 2010) is an IM application that allows users to share their moods by selecting an icon; the study’s results show that mood sharing had a positive effect on communication. Kuber et al. (2013) presented an IM system that recognizes users’ emotions from a wearable headset and visually expresses emotional information in real time. GamIM (Pong et al., 2014) visualizes the history of the conversation, especially on the emotional exchanges during communication. Still, these approaches are still unintuitive to use, and the diversity of emotional expressions is limited. Furthermore, they are passively used (i.e., users have to consciously select and apply them to the message); hence, the emotions or context in the messages can be easily filtered or distorted.
This research is differentiated from prior works, in that we investigated the design opportunity for using physical handwriting as the medium for remote and instant communication. While handwritten communication has gradually deprecated due to the lengthy delivery time and lack of efficiency as a communication media, it is considered the most appropriate for sincere emotional communication. When delivering a romantic message, for instance, people prefer handwritten letters among various communication media (Wells and Dennis, 2016). Handwriting has been applied to multiple application domains from various studies; most commonly, handwriting is used as the system’s input modality. Whittaker et al. (1994) and Harit et al. (2007) adopted handwriting to instantly and conveniently add notes to digital materials. Poupyrev et al. (1998) used handwriting for a communication interface in virtual environments. Nonetheless, handwriting, in the perspective of a remote communication medium, was not particularly investigated. In our study, we focused on instant and remote communication through handwriting on paper by developing and experimenting with a novel computer-mediated text communication system.
3. Tele-Handwritter (TH)
TH is a system that allows the sender to instantly and remotelyexchange physical handwriting (Figure 1). We have designed TH in such a way that it functions as if the sender’s hand is actually writing on the paper with a pen, which is delivered to the receiver in a remote place (Figure 2).
![Figure 2 Figure 2](/xml/19708/ADR_2019_v32n2_45_f002.jpg)
Tele-Handwriter: a text-based communication system that allows remote users to instantly communicate with handwriting on a paper
3. 1. Features and Design Rationales
Our system allows users to write on paper with a pen, send messages, and read text written by a pen plotter. Also, to create the ordinary feeling of writing on paper that is magically delivered to the receiver, the exterior of the TH system is designed to look similar to an ordinary table, thus hiding the display under the top covering surface.
A message in a digital form can be instantly represented in a display when a message is received. However, representing a received message on paper with a pen inevitably requires time. In order to preserve our concept of instantly delivering handwriting, we chose to send and represent the message at the moment the message is written. In our setting, the receiver’s TH terminal moves in realtime according to the motion of the sender’s pen.
TH aims to deliver the sender’s full hand movements to the receiver. The TH system not only logs the pen’s movements and pathways but also the time intervals between the movements. In this way, when the plotter reproduces the handwritten message, the tempo and rhythm of the handwriting are reflected.
3. 2. Implementation
TH consists of a pair of two identical terminals connected via a network, and it is capable of physically reproducing the handwriting of one user for the other user (Figure 1). Each terminal has both input and output components. For the input component, Cintiq 21ux, a touch tablet from Wacom Co., Ltd., was used to sense the handwriting with a stylus. The nib of the stylus has been removed and replaced with a 1.4mm-thick graphite nib so that it can be used on paper while the positions and movementsare registered on the tablet. Tophysically reproduce the message, we adopted Axidraw v2, a pen plotter from Evil Mad Science LLC. To control the motors in the pen plotter, we used a combination of Arduino (https://www.arduino.cc/) and Arduino compatible grblShield.
The software was developed using Processing (https://processing.org/), an open-source programming language. Processing’s oscP5 library was used for remote communication between the terminals. When the tablet obtains the pathway of the stylus and the pressure, the software transmits them to another terminal through the network. The same software on the other terminal then receives this information, creates G-code, and transfers the code to grblShield to reproduce the handwriting.
4. User Study
4. 1. Study Setting
To investigate how the TH communication via handwriting and the utilization of handwriting and a physical medium such as pen and paper affect people in text-based communication, we conducted a comparative study using three text communication media depending on writing styles and the physical representation.
We introduce three forms of media in our comparative study (Table 1). Tele-HandWriter(TH) delivers handwritten messages through a physical medium. The second medium is Shared Canvas(SC), which delivers handwritten messages on a screen. The last medium is Instant Messaging(IM), a typical digital text media with features that contrast completely with those of TH. For the combination of typing and screen display, we can imagine a system that resembles the analogue typewirtter. However, considering the goal of our study, this was deemed unnecessary and has been left out.
4. 2. Study Procedure
The comparative study was conducted with eight couples in a relationship (8 males, 8 females, average age 24.6) using the three text media that comprise the study setting. We specifically recruited couples who were in a relationship because we wanted to investigate the effects of each text medium on various conversational situations, including day-to-day conversations and affectionate conversations.
Our study was made up of two sessions depending on the nature of the conversation. In the first session, participants engaged in day-to-day conversations, exchanging short paragraphs using three different text media for about seven minutes. For the second session, participants were asked to send one long, affectionate message to each other using each text medium without a time limit (i.e., a situation that calls for sincerity). For the experimental environment, the participants’ rooms were separated by means of partitioning walls to form pseudo remoteness, and vocal conversation between the participants during the test was prohibited. The order by which the participants used the media in each session was randomly shuffled, and the chat logs between the participants were not collected to create a natural chatting experience and to ensure privacy. Lastly, the couples were interviewed on their general experience of each medium and the pros and cons of each medium at the end of the two sessions.
4. 3. Analysis
For the in-depthunderstanding on the effect of each media, we took the qualiatative approach on collecting and analyzing the study results.We employed thematic analysis with the transcribed interview data. The authors read the transcripts to familiarize themselves with the data. Afterwards, they coded the data using the bottom-up affinity diagramming technique.
3. Study Findings
We report on our study’s findings regarding the effects of TH from two perspectives: the effect of handwriting and the effect of the physical representation.
5. 1. The Effect of Handwriting
Through the comparison between typing (IM) and handwriting (SC and TH), we were able to understand the effects of handwriting in text-based communication. The use of handwriting for communication allowed the sender to include his or her identity into the message and increased theseriousness ofthe sender’s attitude, and this change in identity and attitude was efficiently delivered to the receiver. Unlike digital text media where the text is presented in a standardized font that bleaches the sender’s identity, handwritten media is able to include the sender’s identity in the form of handwriting. “When the sender has no profile picture or an empty name in IM, it is hard to know who that person is, but with handwriting, I would have some idea about who.” The sender’s handwriting naturally embeds his or her identity in the message and allows for a personalized, intimate communication between two people. Furthermore, this effect increased even more when the two were already familiar with each other’s handwriting: “In the IM, messages are presented in the same fonts, regardless of who the sender is. When the messages are handwritten, familiar handwritings from my lover or my families stand out from others, and this makes the message feel more special.”
Handwritten messages were able to show what was written in the text and to express contextual information, such as the sender’s attitude. “When I received a long and affectionate message with TH, I saw that at first a lot of effort is put into it. But as the message progresses, it was easy to notice that the eagerness dies away. It was funny how I could notice this in the handwriting (showing the message received from TH). Also, at first, the writing was nicely aligned, but near the end, it went all over the paper.” Handwriting exposes whether the sender has put effort into “writing nicely” or whether he or she has written in a lackluster fashion, and this has an additional effect on the sender in changing his or her attitude, thus making it more serious. This serious attitude seems to have affected both the handwriting and the content of the message. “Handwriting requires more effort and work than typing, and this made me write things that really matter and are meaningful.”
In a nutshell, handwriting, compared to typing, has a distinct characteristic that naturally incorporates the sender’s identity, the attitude of the writing, and the effort. This distinction made the sender more serious and encouraged him or her to write meaningful messages; as a result, the receiver and the sender increased their intimacy throughout the conversation.
5. 2. The Effect of the Physical Representation
Similar to one of the effects of handwriting, senders felt inclined to dedicate more effort when writing their messages. We found that a physical medium effectively delivers the sender’s commitment to the receiver. Interestingly, while the sender did not show noticeable differences between SC and TH, the receivers preferred TH, as they commented that they were able to feel more effort and heart in the sender’s message when using the TH setup. This phenomenon can be interpreted as the receiver acknowledging the sender’s effort based on the busy movement of the pen plotter. “SC feels like it has been written effortlessly, but with TH, I can see how much effort was put in by the sender from the machine’s movement.”
On the other hand, it can also be interpreted as a way to reminisceon pre-digital media, such as the exchange of letters, which adds a higher emotional value to messages received by TH: “I prefer paper because it reminds me of memories when I used to write letters,” and, “Usually, when I grab a pen and write a letter to my lover, I think of many things, including the content, the writing style, and many other things. The same thing went on when I wrote the affectionate message with TH.”
A physical medium such as pen and paper, compared to a digital medium such as a screen, communicates the sender’s effort and commitment to the receiver in its entirety.
We found there is an additional effect from the active movement of the machine when TH reproduces the text. The machine’s movement not only delivers what is written but also the contextual meaning of how the text was written: “Looking at the machine as it wrote the message and then suddenly stopped for a while made me think, ‘Ah, this is where he stopped and thought about what to write.’” “Unlike instant messaging where the message appears completely, the message came letter by letter, which made me picture her stopping and pondering, and I could feel her heart that could not be expressed in words.” Furthermore, the varying motion of the machine expressing the contextual information also created the sense of a remote presence that reminded the sender of the receiver: “Because the writing speed changed back and forth between slow and fast, it felt like it was really handwriting the messages, which is completely different from a printer,” and, “It writes like a human would, unlike a printer, and the speed of writing changes from slow to fast, and it really felt like it was her hand.”
To sum up, conventional digital text media compresses the sender’s contextual information into plain text, thereby quenching it in the process; however, TH physically reproduces the text naturally, contains and enriches the sender’s contextual cues, and deliversthem fully, which is a great advantage.
6. Discussion
6. 1. Role of Tele-Handwriter as Affective Communication Media
Through the study, we were able to determine the effects of TH. In addition to this, we discuss the applicable area of TH between digital text media and handwritten media.
The participants commented that using handwritten media for routine conversations, such as in the first session of the study, did not differ that much from using digital text media, and it was more inconvenient in some ways. This is because, for everyday conversations, people expect a fast and convenient exchange of information rather than the delivery of emotions based on the role of the medium they use. However, all participants commonly stated that, as in the second session, TH effectively delivered sincerityfor emotional situations, and they evaluated it as useful.
We do not envision that TH will replace conventional digital text media completely but will serve as alternative media in situations where people’s only choiceis digital text media due to the inconvenience and slowness of handwritten media. Thus, TH can be utilized in some situations where affective but slow communication would be more appropriate: 1) the older generation and families who are not familiar with digital media; 2) long-distance couples or people who are intimate but physically far from each other, such as a students studying abroad and their family; and 3) children who need to learn how to write by hand.
6. 2. Improvement Areas of Tele-Handwriter
Based on the development of the TH prototype and the user study, we identified several issues that can further improve TH. We designed TH to be used on paper freely without any limitations. By doing so, participants were able to draw shapes and pictures and freely annotate the previously written messages. However, some users stated that this lack of format is uncomfortable, and they wanted a layout-correction function for the text’s alignment.
Also, the participants positively commented on our design choice for the real-time transmission of text, as it made them more thoughtful when writing. However, some participants stated that the inability to make amendments to the messages was inconvenient. Based on this, we considered implementing a delete function. Furthermore, people with illegible handwriting were reluctant to communicate using handwriting. One possible improvement could be to adda correction procedure to make the handwriting more legible.
Lastly, the current TH reproduced the text at a slower rate than the sender’s actual writing speed due to the mechanical constraints of the TH’s plotting system. Also, it doesn’t have a proper feedback interface that indicates how much of the written text was transmitted and represented to the other side. The combination of these two problems caused some confusion in terms of turn-taking while the participants werecommunicating. We plan to improve the feedback interface to show the progress of the text transmission and to provide rich awareness of the remote partner’s status.
6. 3. Future Work
In this research, we investigated the effect of handwriting and physical representation in computer-mediated text communication. The use of handwriting and a physical medium can be one way to recover the lack of emotional values in digital text media.
However, apart from those two elements, we believe there are still more elements that enrich the emotional experience in text-based digital communication. One of the interesting elements mentioned from the user study is the implementation of a function with changing paper layouts and pens: “If it were an actual letter and not just ordinary A4 paper, I would really treasure it. It would be nice if I could use something other than pencils, like a fountain pen.” Such a function with changeable layouts (i.e., the envelope and stamp for a letter) and writing tools is expected to enhance the level of sincerity in the message, yet we did not explore it. In future work, we plan to explore these unstudied elements with their variations and devise a way to embed such elements into TH and to investigate their effect.
7. Conclusion
In this paper, we investigated the effect of handwriting and physical representation in computer-mediated text communication.The contribution of this paper is twofold. First,we presented TH, a novel text-based communication system that allows remote users to instantly communicate with handwriting on paper. Second,we reported and discussed the results of a comparative user study using three different types of text media(e.g., TH, SC, IM). Our results showed thatthe use of handwriting and physical representation had the following effects: increased intimacy, wholly delivered effort and sincerity, and contextual cues being naturally added to the message.By reflecting on the study’s results, we discussed the applicable area of TH and proposed design insights to design physical handwriting systems for computer-mediated text communication. We argue that the TH development, study results, and discussion add insight to the development of slow technology (Odom et al., 2012) in the field of digital communication.
Acknowledgments
This work was supported by the Ministry of Education of the Republic of Korea and the National Research Foundation of Korea. (NRF-2017S1A5A2A01025327)
Notes
Copyright : This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/3.0/), which permits unrestricted educational and non-commercial use, provided the original work is properly cited.
References
-
Bodine, K., & Pignol, M. (2003, April). Kinetic typography-based instant messaging. In CHI'03 extended abstracts on Human factors in computing systems (pp. 914-915). ACM.
[https://doi.org/10.1145/765891.766067]
- Bond, (2017), Home. Retrieved January, 2017, from https://bond.co/.
-
Church, K., Hoggan, E., & Oliver, N. (2010, October). A study of mobile mood awareness and communication through MobiMood. In Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries (pp. 128-137). ACM.
[https://doi.org/10.1145/1868914.1868933]
-
Forlizzi, J., Lee, J., & Hudson, S. (2003, April). The kinedit system: affective messages using dynamic texts. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 377-384). ACM.
[https://doi.org/10.1145/642675.642677]
-
Gooch, D., & Kelly, R. (2016, May). Season's Greetings: An Analysis of Christmas Card Use. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (pp. 2105-2111). ACM.
[https://doi.org/10.1145/2851581.2892341]
-
Harit, G., Mankar, V., & Chaudhury, S. (2007, September). Patra: A novel document architecture for integrating handwriting with audio-visual information. In Document Analysis and Recognition, 2007. ICDAR 2007. Ninth International Conference on (Vol. 2, pp. 699-703). IEEE.
[https://doi.org/10.1109/ICDAR.2007.4377005]
-
Höflich, J. R., & Gebhardt, J. (2005). Changing cultures of written communication: Letter-e-mail-sms. In The Inside Text(pp. 9-31). Springer, Dordrecht.
[https://doi.org/10.1007/1-4020-3060-6_2]
- iOS, iOS 10, Apple. (2017), Write It Yourself. Retrieved January, 2017, from http://www.apple.com/ios/ios-10/.
-
Kuber, R., & Wright, F. P. (2013). Augmenting the instant messaging experience through the use of brain-computer interface and gestural technologies. International Journal of Human-Computer Interaction, 29(3), 178-191.
[https://doi.org/10.1080/10447318.2012.702635]
-
Lee, J. C., Forlizzi, J., & Hudson, S. E. (2002, October). The kinetic typography engine: an extensible system for animating expressive text. In Proceedings of the 15th annual ACM symposium on User interface software and technology (pp. 81-90). ACM.
[https://doi.org/10.1145/571985.571997]
- Leppänen, S. (2001). The Relationship between Electronic Mail and Paper Mail. Digital Media Institute (Ed.): Case Study: Changes in Postal Services-Paper or Bytes, 51-80.
-
Odom, W., Banks, R., Durrant, A., Kirk, D., & Pierce, J. (2012, June). Slow technology: critical reflection and future directions. In Proceedings of the Designing Interactive Systems Conference (pp. 816-817). ACM.
[https://doi.org/10.1145/2317956.2318088]
-
Pong, K. C., Wang, C. A., & Hsu, S. H. (2014, April). GamIM: affecting chatting behavior by visualizing atmosphere of conversation. In CHI'14 Extended Abstracts on Human Factors in Computing Systems (pp. 2497-2502). ACM.
[https://doi.org/10.1145/2559206.2581168]
- Poupyrev, I., Tomokazu, N., & Weghorst, S. (1998, March). Virtual Notepad: handwriting in immersive VR. In Proceedings. IEEE 1998 Virtual Reality Annual International Symposium (Cat. No. 98CB36180) (pp. 126-132). IEEE.
-
Riche, Y., Henry Riche, N., Isenberg, P., & Bezerianos, A. (2010, April). Hard-to-use interfaces considered beneficial (some of the time). In CHI'10 Extended Abstracts on Human Factors in Computing Systems (pp. 2705-2714). ACM.
[https://doi.org/10.1145/1753846.1753855]
- The Indian Handwritten Letter Co, (2017), Services. Retrieved January, 2017, from http://www.tihlc.com/.
-
Valkenburg, P. M., & Peter, J. (2009). Social consequences of the Internet for adolescents: A decade of research. Current directions in psychological science, 18(1), 1-5.
[https://doi.org/10.1111/j.1467-8721.2009.01595.x]
-
Wells, T. M., & Dennis, A. R. (2016). To email or not to email: The impact of media on psychophysiological responses and emotional content in utilitarian and romantic communication. Computers in Human Behavior, 54, 1-9.
[https://doi.org/10.1016/j.chb.2015.07.036]
-
Whittaker, S., Hyland, P., & Wiley, M. (1994, April). Filochat: Handwritten notes provide access to recorded conversations. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 271-277). ACM.
[https://doi.org/10.1145/259963.260380]