Archives of Design Research
[ Article ]
Archives of Design Research - Vol. 37, No. 5, pp.29-49
ISSN: 1226-8046 (Print) 2288-2987 (Online)
Print publication date 30 Nov 2024
Received 02 Aug 2024 Revised 07 Sep 2024 Accepted 07 Sep 2024
DOI: https://doi.org/10.15187/adr.2024.11.37.5.29

Transforming Everyday Objects into IoT Control Interfaces: Design and Evaluation of the 'e-Rings' System

Changmin Kim
Department of Industrial Design, Postdoctoral Researcher, KAIST, Deajeon, Korea

Correspondence to: Changmin Kim peterkim12@kaist.ac.kr

Abstract

Background The concept of a smart environment has been a topic of discussion for decades, but recent advancements in Internet of Things (IoT) technology have made it more achievable. However, while IoT devices are becoming more prevalent in our living spaces, their current usage is still rudimentary. To truly make these devices ‘smart’, they need to be able to automatically respond to users’ needs, which requires effective IoT coordination. Despite the existence of tools such as if-this-then-that (IFTTT), their capabilities are yet limited in covering vast and diversified user needs.

Methods This paper introduces ‘e-Rings,’ a wearable IoT coordination system that transforms everyday objects into control interfaces for IoT devices. Using the system as a probe for investigate the actual user experience of the proposed IoT coordination approach, an evaluation study involving eight participants was conducted in an office workspace. The study resulted in successful creation of 40 IoT coordination cases using office accessories and four types of IoT devices.

Results The study’s findings underscored the usability and acceptability of the ‘e-Rings’ system, as participants successfully created unique and personalized IoT applications, demonstrating both ease of use and the capability to integrate IoT coordination seamlessly into their routines. The system’s demonstration-based approach and wearable platform provided an effective solution to the challenges posed by traditional IoT coordination methods. Participants also appreciated the playful and engaging nature of tangible interactions in the resulting smart environment.

Conclusions The ‘e-Rings’ system contributes to achieving effective IoT coordination through the intuitive use of everyday objects. This research underscores the importance of creating seamless and engaging smart environments with more personalized and varied IoT applications. Future work should considers long-term deployment studies in broader context to further validate and refine this approach.

Keywords:

Smart Environment, Internet of Things, IoT Coordination Tools, Everyday Objects, Wearables, Research through Design, Interaction Design

1. Introduction

The concept of a ‘smart environment’—a living space that uses technology to support its inhabitants—has been discussed for decades. Recent advancements in the Internet of Things (IoT) have brought this concept closer to reality, introducing a variety of smart IoT devices to the market. As a result, 77% of adults now own at least one IoT device (techUK, 2022). However, current usage of these devices remains rudimentary, with users often controlling them manually through smartphone apps rather than having the devices proactively serve their needs (Bitkom Research, 2022). While IoT devices can be controlled passively like traditional devices, their true potential lies in coordinated use. For instance, a NEST thermostat can more effectively manage heating and cooling when coordinated with the user’s whereabouts, and Philips Hue lighting can adjust the room’s mood based on the user’s activities. By coordinating IoT devices and contexts, it becomes possible to create sophisticated and truly intelligent environments. Consequently, there is a growing need for self-customizing smart functionalities for IoT devices, and solutions for IoT coordination are in high demand (Markopoulos et al., 2017).

Currently, IoT coordination is mostly achieved through trigger-action programming tools like IFTTT (If This Then That). Users select a contextual ‘trigger’ (e.g., after 11 pm) and a desired ‘action’ for an IoT device (e.g., turn off the light) and insert both into a predefined programming template. While these tools enable end-users to experience some level of coordinated IoT device usage, their capabilities are limited in addressing the diverse needs of users (Ur et al., 2014, 2016). Despite users’ creative ideas for IoT coordination, the available triggers are restricted to a small range of sensor module values and time-related parameters (Funk et al., 2018), forcing users to scale back their ideas to partial solutions. Moreover, users struggle to translate their ideas into the unfamiliar programming terms and logic required (Funk et al., 2018). Although trigger-action programming is relatively simple, end-users do not naturally think in terms of sensor values and thresholds. Without prior knowledge of physical computing and programming, managing sensor modules and developing programming logic for IoT coordination remain challenging tasks for general end-users (Woo and Lim, 2015;Jakobi et al., 2017; Funk et al., 2018).

Given these challenges, this paper explores the research proposition that everyday objects can serve as natural and intuitive interfaces for controlling IoT devices; thus the diversity and familiarity of everyday objects can potentially resolve the challenges in end-user IoT coordination. Building on prior work that examined various beneficial interactions between everyday objects and IoT devices (Kim and Nam, 2022), this paper delves deeper into designing a system that brings this concept to an experiential level. The paper introduces ‘e-Rings,’ a wearable IoT coordination system that transforms everyday objects into control interfaces for IoT devices. It details the hardware and software design of the ‘e-Rings’ system and presents the results of a user study conducted in a realistic context. The study’s findings highlight the usability and acceptability of the ‘e-Rings’ system, as participants successfully created unique and personalized IoT applications, demonstrating both ease of use and the capability to integrate IoT coordination seamlessly into their routines in a playful and engaging way. Lastly, the paper discusses the potentials of using everyday objects in end-user IoT coordination and suggests directions for system improvements.

The contributions of this paper lie in the exploration and investigation of the potential role of everyday objects in future smart environments. It proposes a novel end-user IoT coordination approach that utilizes everyday objects as control interfaces for IoT devices, offering a fresh perspective on integrating non-digital artifacts into IoT ecosystems. The system prototype, ‘e-Rings’, demonstrates the feasibility of this approach, providing a practical reference for IoT designers incorporating non-digital artifacts through wearable devices. The effectiveness of this approach is supported by empirical evidence from the studies conducted with ‘e-Rings’, demonstrating its potential for improving the user experience in smart environments. Through the investigation of the potential challenges posed by this approach, this research also offers insights into the implications of bringing the proposed IoT coordination approach to reality. Through its findings, prototypes, and design implications, this paper contributes to the broader discourse on IoT user experience, providing useful knowledge for IoT designers and researchers aiming to refine future IoT ecosystems.


2. Related Works

2. 1. Challenges in End-User IoT Coordination

The emergence of commercial IoT devices has extended smart environments beyond controlled laboratory settings (e.g., Aware Home, 2002; MIT’s House, 2002) into real-life living spaces. Given the diverse and personalized preferences for IoT setups (Hwang and Hoey, 2012), various end-user IoT coordination tools have been investigated to enable non+technical users to configure their smart environments. The predominant approach has been the rule-based trigger-action paradigm, with efforts to simplify rule configuration (Olmo et al., 2010; Huang and Cakmak, 2015). Commercialized services like IFTTT, SmartThings (2014), and MiHome (2016) have been widely adopted, facilitating ‘in-the-wild’ studies to better understand several shortcomings of IoT coordination from a layperson’s viewpoint.

Despite the advancements, IoT coordination remains limited to a small set of obvious combinations of triggers and actions. Usable triggers fall into categories such as manual triggers through apps or voice commands, sensor value thresholds, and time-related parameters (Ur et al., 2014). These limited triggers restrict users’ control over IoT devices, forcing them to scale back their initial needs to partial solutions. An analysis of 224,500 IFTTT rules revealed that only 3.3 percent of trigger-action combinations were unique, with the majority being duplicates (Ur et al., 2016). This highlights the need for more diverse triggers to increase the variety of rules. To address this issue, researchers have developed specialized monitoring sensors, such as water activity sensing (Froehlich et al., 2009), electrical event sensing (Gupta et al., 2010), and occupancy sensing (Scott et al., 2011). Nevertheless, simply adding trigger instances is inadequate to meet the diverse needs of users, highlighting the necessity for more comprehensive solutions.

Additionally, the unaccustomed programming thinking required for IoT coordination creates a mismatch between user needs and resulting setups. Funk et al. (2018) noted that it is not intuitive for users to translate their needs into programming terms like sensor values and thresholds. Despite the relative simplicity of trigger-action programming, technical barriers in constructing sensor networks lead to failures (Jakobi et al., 2017), often requiring technical experts to develop IoT applications (Woo and Lim, 2015), emphasizing the importance of coherent and clear interface for smart environment configuration (Liu et al., 2023). This paper aims to overcome these challenges by integrating everyday objects into the IoT ecosystem. Leveraging the diversity and familiarity of everyday objects, this approach seeks to enable end-users to effortlessly pursue their desired contexts and apply their creative ideas to IoT coordination.

2. 2. Techniques for Sensing Everyday Objects

A key aspect of the smart environment is to effectively and unobtrusively infer users’ contexts in their environment (Weiser, 1999). As for inferring people’s daily activities, everyday objects have been long studied by various researchers. A variety of approaches are available for monitoring the usage of everyday objects, each with its own unique strengths and weaknesses. One straightforward approach is attaching sensor modules directly to objects. Wilson and Atkeson (2005) used pressure sensing modules throughout a home to gather location data and infer user activity. Laput et al. (2017) demonstrated using vibration motors on objects, allowing a smartwatch to recognize them based on vibro-acoustic signals. Programmable tokens like T4tags (Bellucci et al., 2016), T4tags2.0 (Bellucci et al., 2019), and Sens.se (2019) are Wi-Fi-connected with embedded sensors that can trigger events based on object movements. However, their size and battery requirements limit their practicality for analog objects.

Another technique involves attaching passive tags, such as RFID, NFC, or magnetic tokens, to objects. These can be sensed through wearable devices or environmental installations. Philipose et al. (2004) designed wearable RFID readers to detect interactions with tagged objects. Buettner et al. (2009) and Li et al. (2015) explored ultra-high frequency (UHF) RFID tags and readers, which, despite their extended range, had low resolution for detecting object movements. Gummeson et al. (2017) improved this by installing UHF RFID readers within lightbulbs, balancing reader distance and object movement detection. Passive tags are low-cost and straightforward for tagging objects, though they might impact the design and usability of the objects they are attached to.

Other techniques for sensing objects include vision, acoustics, and electromagnetic (EM) signals. Cameras can recognize objects through computer vision, initially using visual markers like barcodes and QR codes. Recent advancements allow for object recognition in natural scenes without markers (Cohn et al., 2011). However, their effectiveness depends on line of sight and lighting conditions, thus raise privacy concerns. Acoustic-based recognition has also been extensively researched. Viband (Laput et al., 2016) used smartwatches to identify objects through vibro-acoustic signals, while Event Listener (Wu et al., 2020) used microphone arrays and algorithms to infer user activities with objects through audio. Uninstrumented techniques include EM-Sense (Laput et al., 2015) and Deus EM Machina (Xiao et al., 2017), which identify objects by measuring unique electromagnetic noises emitted by electrically powered objects.

Each technique has limitations in coverage and reliability, but combination of wearable devices with passive tags offers a promising solution for the proposed IoT coordination systems. Passive tags enable detection of objects and the wearable device can further capture user activities while holding objects. Together, they offer a comprehensive and unobtrusive solution for detecting and inferring various types of contexts in everyday objects.


3. Utilizing Everyday Objects for IoT Control Interfaces

The following presents a breakdown of the proposal for an end-user IoT coordination approach in three parts: (1) utilizing everyday objects as contextual triggers to be coordinated with IoT devices, (2) employing user demonstrations on objects as a coordination method, and (3) incorporating wearables as a sensing platform to facilitate this process.

3. 1. Everyday Objects as Contextual Triggers for IoT Devices

The key idea of the research proposal is to leverage various types of everyday objects (Fig 1) and their usage behavior to control IoT devices, thereby enhancing the range of features and possibilities for IoT coordination. As users interact with various types of everyday objects in their daily lives, these interactions implies highly precise user contexts and activities Philipose et al. (2004). Despite the perceived incompatibility between analog everyday objects and digital appliances, there are certain combinations of everyday objects and IoT devices that can naturally work together in a single user activity. This implies that everyday objects can act as contextual triggers for controlling the functionalities of IoT devices. Additionally, this approach opens up new opportunities for seamlessly integrating the functionalities of IoT devices into nearby everyday objects, allowing users to interact with and control them in an intuitive and convenient manner, resulting in a natural and user-friendly experience. An example illustrating such IoT coordination is as follows:

Figure 1

Utilizing various types of everyday objects as a control interface for IoT devices.

  • ● When the user pulls a pen out from its cover to write something down on the paper, the desk lamp is automatically turned on and adjust its brightness and color temperature.

3. 2. User Demonstration as a Method for Coordinating Objects

To address the challenges and difficulties associated with current end-user IoT coordination, the research proposes a demonstration-based method that enables users to effectively coordinate everyday objects with IoT devices within a physical environment. The proposal involves an IoT coordination system that recognizes the sequential usage of everyday objects and IoT devices, and transcribes those actions into an IoT application (Fig 2). By applying the programming by demonstration (PBD) approach to the coordination method, users can simply repeat the sequential behavior of using an everyday object and changing the state of the IoT device to create the desired coordination. An example scenario of creating such coordination can be illustrated as follows:

Figure 2

Systemetic overview of demonstration based IoT coordination approach.

  • ● To coordinate a pen to a desk lamp, the user repeatedly demonstrates the sequential behavior of pulling a pen out from its cover and then adjusting the state of the desk lamp.

Subsequently, users can naturally trigger the change in the state of the coordinated IoT device by re-performing the previously demonstrated usage of the everyday object.

3. 3. Wearables as a Platform for Sensing Objects

In order to realize these propositions, techniques must be employed to identify non-digital everyday objects and connect them to the network. This research proposes using wearables, such as smartwatches and smart rings, as the platform to accomplish this (Fig 3). Although there are various potential methods for object sensing, wearables are expected to provide additional synergy by capturing user behaviors and gestures related to everyday objects. With the emergence of wearable devices, their built-in sensing capabilities, and network connectivity, they are believed to serve as a viable gateway for linking everyday objects to the IoT ecosystem.

Figure 3

Adopting wearable devices as a sensing platform for everyday objects.


4. E-Rings: A Wearable System for Coordinating Everyday Objects and IoT Devices

The objective of the ‘e-Rings’ system development is to create a functional proof-of-concept prototype that embodies the minimum viable solution of the IoT coordination approach that has been proposed in prior chapter. The ‘e-Rings’ system serves as a tangible representation of the concept, aiming to demonstrate the technical feasibility of the concept and providing a platform for testing and validating its effectiveness.

4. 1. System Requirements and Design Decisions

4. 1. 1. Unobtrusive and Comprehensive Object Sensing Techniques

Among various sensing techniques reviewed, attaching sensors or processors to objects was deemed inappropriate due to their need for power and battery recharging. Techniques using camera vision, vibration, acoustic, and EM signals to recognize objects were limited in applicability. The most suitable approach was to attach small, non-invasive, battery-free tags to objects, detected by external sensing modules. The combination of small RFID tags and a wearable RFID reader proved feasible, simple, and economical. A watch-type wearable RFID reader was chosen based on its ability to cover the majority of required object detections, informed by a workshop finding (Kim and Nam, 2022) that 72% of total ideation involved handheld objects.

4. 1. 2. Understanding the Usage Behaviors on Everyday Objects

Understanding object usage behaviors is crucial for developing the proposed IoT coordination system, as users incorporate various gestures with everyday objects. Inertial measuring sensors and machine learning algorithms typically detect these complex gestures. An accelerometer was integrated into the wearable device to sense movements while holding objects. For gesture recognition, options include Dynamic Time Warping, Hidden Markov Models, Support Vector Machines, Neural Networks, Decision Trees, and Random Forests. Dynamic Time Warping, known for its quick computation time in small sample sizes (Carmona and Climent, 2012), was adopted in the software system.

4. 1. 3. In-Situated and User-Friendly Methods for Coordination

The proposed IoT coordination approach expects users to improvise rules linking everyday objects, gestures, and target IoT devices. Efficiently handling the coordination of objects and IoT devices within the environment is necessary. Users need to engage in a machine learning process and undergo real-time data sampling to create a gesture classifier. A few-shot learning approach combined with programming by demonstration (PBD) can simplify the process for non-expert users. This approach allows users to create IoT coordination by repeatedly demonstrating object and device use until the system detects the pattern. Few-shot learning offers benefits like quickly teaching the system new gestures with minimal data and high accuracy. Demonstrating in the actual environment is effective for coordinating objects and IoT devices. References to a pet training analogy for AI (Objectifier, 2016) and a hurdle metaphor for gesture authoring (Kim et al., 2016) help make the gesture training process comprehensive for non-experts.

4. 2. Hardware Implementation

Figure 4

(left) Watch-with-ring type wearable hardware device, (right) a custom IoT device controller attached on the lamp.

The hardware device comprises an RFID reader (ID-12, 125kHz), a small-sized computer with an embedded Wi-Fi module (Adafruit Feather Huzzah ESP32 board), an accelerometer (ADXL-345), and a lithium polymer battery with 1200mAh capacity. These modules are organized and placed into a 3D printed enclosure, with a watch strap to wear around the wrist. In addition, a ring-type antenna was added to extend the sensing range of the RFID reader. Currently, the ring and watch devices are connected via a wire, although they could be integrated into a single device or have the wire removed in future prototypes. To provide system status feedback to the user, an LED and small vibration motor were embedded inside the case. The hardware module transmits data streams of RFID string readings and accelerometer values to the system via Wi-Fi at a sampling rate of 60Hz. The device identifies grasped everyday objects by sensing a small RFID capsule (Sen-09416) attached to the surface of the object. Furthermore, to enable sensing of how and which IoT devices are manipulated by the user, a custom RFID tag-attached controller was attached to each IoT device. This custom controller module features a tact switch and rotary encoder that are mapped to the IoT device’s state using vendor-provided APIs.

4. 3. Software Implementation

The software system was developed using the Express server built with Node.js. Socket. io with custom protocols was utilized for the communication between the hardware (wearables and IoT devices) and the server. As the coordination method is primarily based on demonstrations, users do not need to interact with the software system. As a result, the software was developed as a headless system and the interface, solely for monitoring and debugging purposes. The software system displays five variables, namely the RFID reading, axial acceleration values, state of the object, volatility index, and acceleration vector sum. The utilization of these variables is explained in detail in the following section.

4. 4. System Features

The ‘e-Rings’ system coordinates everyday objects and IoT devices by two main features: (a) proactive coordination by user demonstration which automatically generates and IoT coordination data from the user behavior, and (b) real-time behavior classification and IoT control, which detects the user gesture and change the state of the IoT device.

4. 4. 1. Proactive Coordination by User Demonstration
Figure 5

Simplified feature diagram for training an IoT coordination data.

This feature automatically selects and parses a specific portion of a continuous data stream and converts it into IoT Coordination Data. The IoT Coordination Data consists of (1) an everyday object, (2) a set of acceleration data (AccD) from the user’s behavior, and (3) a target IoT device and its status. When a user grabs a new object, it triggers the data parsing process. Using the upper Bollinger band as a threshold, the system collects a minimum of 5 sets of AccD. It then checks for similarities in the data by calculating the distance between data points using the Dynamic Time Warping (DTW) algorithm. The system determines similarity by comparing the average distance (AvgDist) and the maximum distance found in the AccD. Once the similarity of the collected AccD is confirmed, the hardware module vibrates and instructs the user to select and control the IoT device. Using the modified status of the IoT device as the final piece of information, the IoT Coordination Data (ICD) is created and saved to the system.

4. 4. 2. Real-time Behavior Classification and IoT Control
Figure 6

Simplified feature diagram for classifying gestures on objects and triggering the IoT devices.

This feature compares real-time data with previously saved IoT Coordination Data to compute similarities. The system first filters the IoT Coordination Data based on the currently held object identified by the real-time RFID string. It then calculates the distance between the real-time accelerometer data and the accelerometer datasets (AccD) in the IoT Coordination Data (ICD). For the IoT Coordination Data (ICD) that shows the highest similarity and the shortest distance to the real-time accelerometer data, the system triggers a change in the state of the IoT device according to the target IoT device and its status saved in the IoT Coordination Data (ICD).

4. 5. Example IoT Coordination Walkthrough

An example coordination process using the ‘e-Rings’ system from an end-user’s perspective can be shown as below (this is also illustrated with the supplementary video). In this example, the user intends to coordinate a pen with a desk lamp to create a smart office application, where the lamp is controlled when the user starts writing something. Specifically, the user desires the desk lamp to turn on automatically and adjust its brightness and color whenever the pen is pulled from its cover.

  • 1. The user places an RFID tag on the pen cover and grabs the pen, pulling it out of the cover.
  • 2. To help the system recognize the gesture, the user repeats the demonstration of grabbing and pulling out the cover at least 5 times.
  • 3. When the gesture is recognized and the hardware vibrates, the user releases the pen.
  • 4. The user turns on the desk lamp and adjusts the brightness to the desired state.
  • 5. By releasing the controller, the IoT coordination of the pen and the desk lamp is created.

Afterward, whenever the user picks up the pen and pulls it out from its cover, the system recognizes the object and use behavior, and shifts the state of the desk lamp to the user’s defined brightness and color.


5. Evaluation of e-Rings in an Office Workspace

The study aims to investigate the actual user experience of the proposed IoT coordination approach by using the ‘e-Rings’ system, and identify any problems that may only be apparent through field testing. The ‘e-Rings’ system acts as a probe to explore the applied environment, akin to an experience prototype (Buchenau and Suri, 2000). The primary aim was not to assess system performance, such as software precision, but to evaluate the feasibility of the IoT coordination approach and the resulting smart environment.

5. 1. Study Environment: Office Workspace

To create an IoT coordination using the implemented system, a space reflecting specific prerequisites is required. First, everyday objects in that space must be tagged with RFID so they can be recognized through the wearable device. Second, the IoT devices in that space have to be customized so the manual control of those devices can be recognized by the software system. As an example space reflecting the prerequisites, a typical office workspace has been selected as it is one of a daily space where people reside and spend their time.

Inside the office workspace, four types of IoT devices were prepared (Fig. 7a); an IoT lamp (the color, brightness, and on/off state can be controlled) and IoT ceiling light (on/off state can be controlled), an IoT fan (the wind speed can be controlled), and an IoT coffee port (on/off state can be controlled). The controllers of these IoT devices were replaced with custom controllers (Fig. 4 right) in which to be used along with the exemplary system. Office accessories and other objects (e.g., pen, cup, book, table and etc.) were placed in the office workspace with RFID tags pre-attached to the grasp positions. Extra RFID tags were also prepared to let participants attach the tags to their personal belongings and use them in the coordination (Fig. 7b).

Figure 7

(a) Office workspace equipped with four types of IoT devices; (b) office accessories attached with RFID tags.

5. 2. Methods and Participants

The study followed a one-on-one usability testing format, where participants were asked to perform given tasks using the study apparatus, and their behaviors and feedback were observed. The study was then followed by questionnaires and in-depth interviews. The study recruited eight office workers aged 22-32 (Avg=25.88) to participate, as they were expected to naturally project their lifestyles into the study environment. Prior to the study, participants were given time to understand the concept of coordinating everyday objects and IoT devices through a list example of IoT coordination cases. The study then proceeded in the following steps.

  • 1. Familiarization with the ‘e-Rings’ system: Participants were introduced to the ‘e-Rings’ system by following instructions to create a walkthrough example, coordinating a pen with a desk lamp. They attached RFID tags to the pen cover, performed the pulling gesture, and pressed the desk lamp’s customized controller to turn on the light by pulling the pen cover out from the pen.
  • 2. Creating 5 IoT coordination cases: Participants created five smart office applications using the system. The given contexts were: arriving at the office, working, resting, leaving the office, and a situation of their choice. Created IoT coordination cases were repeatedly tested and adjusted (up to 1.5 hours) until participants were satisfied with the results.
  • 3. In-depth interview: User experience was debriefed through a 40-minute in-depth interview. Questions focused on: (1) the most preferred and interesting smart office IoT application, (2) the effects and benefits of the new IoT coordination approach, and (3) the general usability of the ‘e-Rings’ system.
  • 4. SUS questionnaire: The usability of the ‘e-Rings’ system was further investigated using a modified system usability scale (SUS) questionnaire (Brooke et al., 1996). For any items deviating from the mean score, additional questions were asked to understand the reason. Challenges encountered and participants’ ideas for system improvements were also collected.

For data analysis, the created IoT coordination cases were organized into a table, having the selected and discarded objects and time for completion as table headers. The recorded interviews were also transcribed into text for further analysis. The iterative analytic induction method was primarily used, starting with a tentative hypothesis regarding the overall usability issues, system benefits, and the resulting IoT coordination experience.

5. 3. Evaluation Results

Figure 8

Participants creating IoT applications on electric kettle.

Overall, the exemplary system were successfully used in creating and testing IoT coordination in the smart office workspace context. Participants expressed a slight sense of incompatibility about using the ‘e-Rings’ system in the initial trial. However, after creating the first coordination and confirming that it worked well, the participants adapted progressively and became confident in using the system. Every participant succeeded in creating five coordination cases by themselves.

5. 3. 1. Usability of e-Rings System

The results of the modified SUS questionnaire show that the ‘e-Rings’ system was very usable (Fig 9, scores: mean values). Overall system usability was rated 4.0 on average. The understandability (4.25), learnability (4.12), and complexity (inverted, 3.88) of the proposed coordination method were rated high on average. Confidence in using the system (3.12) received a mediocre rating due to unfamiliarity with the method. Still, participants progressively adapted to the system by repeatedly testing it. In detail, participants complimented the combination of utilizing everyday objects with the demonstrationbased IoT coordination method and said it was effective for novice users like themselves. P3 explained, “It was easier to understand how the ‘e-Rings’ system works because of the simple narrative. ‘The system creates a rule that mimics my behaviors or habits of using objects in the office.’” P1, who had experience using the IFTTT system, added, “The ‘e-Rings’ system smartly recognizes my behaviors of using objects and automatically inserts them into a predefined template such as a Recipe in IFTTT.” Some participants complimented the ‘e-Rings’ system for not having a display-based programming environment and for being able to interact with real objects during the coordination. P3 mentioned, “The concept that this system proposes felt like ripping off the IoT device controller application from the smartphone and placing it in the appropriate places (or objects) in the office.” Other participants also liked the benefit of creating IoT coordination in situ (office). P8 said, “There was no need to sit in front of a monitor and write codes. Coordinating things in the middle of the office felt very handy.”

Figure 9

Summary of system usability scale result.

Notably, the system error recovery (2.63) had relatively low scores; it was identified that the current combination of a headless hardware device and the demonstration-based coordination method has weaknesses in debugging errors. The coordination process of the ‘e-Rings’ system requires repeated user demonstrations, which take at least five continuous repetitions of gestures. Meanwhile, light errors occurred; for example, the hardware failed to read the RFID tag (i.e., the RFID tag was not in the sensing range of the antenna), the software system did not recognize the user behavior (e.g., participants’ gestures were not consistent during the demonstration), and the software system required more user demonstrations to create a gesture classifier. Participants were confused when errors occurred, and debugging these errors was troublesome in the current headless system. Participants insisted on visual cues in the current hardware setup to indicate errors in the system and help them recover from the errors.

5. 3. 2. Values in IoT Coordination Achieved through ‘e-Rings’ System

(1) Personalized IoT Applications: Participants highly valued their ability to apply personalized rules to control IoT devices, turning habits into automated applications or adding new functions to their belongings. For example, P5 created a coordination to turn off the lamp when lifting his backpack to leave the office, stating, “Looking at the ceiling light going off when I lifted the backpack from the chair, it felt as if the room was seeing me off.” Similarly, P1 used a toothbrush to adjust the ceiling light’s color temperature to a warmer tone, noting, “Brushing my teeth in the office means I had a quick dinner and will be working late. It was imposing to see the light respond to my personal routines.” These impressions also points that the satisfaction of IoT coordination often come from creating personal connections with the environment rather than simple automation rules.

(2) Privately Controlled IoT Devices: Participants appreciated the potential for privately controlled IoT devices. The current hardware setup authenticates users by responding only to those wearing the wearable device, allowing multiple users’ personal IoT control rules to coexist in a public space. P4 attached an RFID tag to the office doorknob, making the desk lamp turn on when the doorknob was tilted downward, stating, “This wearable setup can privately manage various IoT coordination in public spaces for many people.” P1 suggested sharing IoT coordination with grouped members, adding, “It would be exciting to experience a smart office that only my colleagues and I can exclusively use.”

(3) Playful Environment with Tangibles and Gestures: The physicality of the triggers was well-received by participants. Many commented on the interaction benefit of utilizing tangibles to control IoT devices. P7 compared commercial sensor kits (e.g., light sensors, motion sensors) with everyday objects, stating, “... environmentally installed sensor modules try to passively infer my activity ... usage of tangible objects is a more direct input and it made the resulting IoT application more robust and intuitive.” Participants also highlighted the playfulness of tangible interaction. P1 noted, “Objects combined with gestures added activeness to the experience ... this made the IoT automation more interesting and the office environment more engaging.”

(4) Semi-Volitional Control Interface: Participants evaluated everyday objects as a new type of controller for IoT devices, coining the term ‘semi-volitional controller’ for this concept. These objects can serve as both volitional controllers, like physical switches, and non-volitional controllers, like installed sensors. P8 pointed out the stability of semi-volitional control, stating, “Performing a specific gesture while holding a particular object during a specific activity is natural and habitual but also intentional.” For instance, P2 controlled a desk lamp by picking up a book and gently tapping it on the desk. Participants appreciated not needing to reach for a physical controller or launch a control application. P2 noted, “The controller for the desk lamp naturally comes in hand, and all I need is to slightly tap,” resulting in an intuitive and seamless control experience.

Figure 10

Desk lamp controlled by picking up a book and gently tapping it on the desk.

5. 3. 3. Remaining Interaction Challenges in the e-Rings System

Current implementation of the ‘e-Rings’ system has limitations in covering all possible areas of coordination between everyday objects and IoT devices. During the study, there were some cases of IoT coordination that participants attempted to create but could not be achieved through the current ‘e-Rings’ system. These included:

  • ● Coordination involving gestures with subtle movements or randomness. For example, flicking a finger on the cup to turn on the electric kettle.
  • ● Coordination involving multiple everyday objects or IoT devices. For example, grabbing a book and then a pen to turn on the desk lamp.
  • ● Coordination requiring the demonstration of using objects located at long distances. For example, coordinating an ornament in the car with an electrical fan in the office.
  • ● Coordination requiring the demonstration of using objects in separate timelines. For example, swinging a cup to turn on the electric kettle 10 minutes later.

Participants also expressed a desire to expand the range of supported everyday objects, beyond the currently supported graspable objects, and to combine them with other conventional contextual triggers such as sensors and time-related parameters. While the study did not identify any instances of false-positive triggering of IoT devices, participants did express concerns about the possibility of unintended operations.


6. Discussion

To encourage users to adopt a system with technical and usage complexities, it must offer significant value. In the study, participants proposed various original and creative IoT coordination cases using everyday objects. However, most cases remained focused on utilitarian applications centered on automation and efficiency. Often, users preferred traditional controllers over everyday objects for IoT control. The following section discusses potential values that can further motivate users to engage in IoT coordination this way.

6. 1. IoT Coordination beyond Productivity and Efficiency

IoT technology is growing in our daily lives, but there is a lack of precision around what kind of values are at the center of the technology. This lack of precision often simplifies the perception of IoT at best idealized and at worst stereotyped. Currently, only fundamental values such as productivity and efficiency are at the forefront of IoT, which are important values but may not always translate into significant changes for users in small-scale environments such as homes or offices. Users do not expect the routinized automation offered by IoT products to dramatically enhance their lives. Therefore, it is important to further investigate a more diverse set of values that IoT can provide from a layperson’s perspective.

Accordingly, research has been conducted to examine laypeople’s needs and motivations for adopting IoT technologies. Studies on speculative smart environments (Ambe et al., 2019), idiosyncratic ideation on smart objects (Berger et al., 2019), and the alternative avenues for IoT (Desjardins et al., 2019, 2020) have shown that possible user needs extend beyond efficiency: building personal connections with the environment, enhancing emotional quality of life, intimate communication with other living people, personal or familial well-being, and self-reflection. In these respects, the potential in augmenting new functions to the long-used personal belongings can be a distinctive value of the proposed IoT coordination approach. Inviting such items in IoT would result in a more favorable smart environment as they embody personal history, emotional attachment, and reflect the user’s identity. Alternatively, the fun and playfulness that come from using tangible objects and embodied gestures can be a unique advantage (such as naturalness or intuitiveness) in the resulting smart environment. Nevertheless, it is important to observe whether such emotional values of non-stereotypical IoT applications can be sustained in native environments and continue to be welcomed by users over the long term.

6. 2. Enabling Gradual and Sustainable Smart Environment

Users may be hesitant to adopt IoT technologies in their environment for several reasons. One of the practical concerns is the cost, as IoT devices can be expensive and users may not see the value in upgrading their current devices. Another factor relates to the necessity of IoT devices, particularly for users who lack a strong interest in the features that new devices offer. On the other hand, users may simply be resistant to change and may prefer to stick with familiar technologies. Users may have an emotional attachment to their possessions and may refuse to discard old devices, even if they are outdated. Additionally, people often value the routines and habits supported by objects within their living space and may resist changes that threaten these familiarities (Gómez, 2015). The result of a home visit survey by Brush et al. (2011) finds that structural change (i.e., installation of IoT appliances and sensors) in homes is still the biggest challenge in broad adoption of IoT technology for layperson users.

There have been several research studies in the IoT domain that have attempted to preserve existing objects and respect the routines and habits associated with them. Retrofab (Ramakers et al., 2016), IoTIZ (Kim and Nam, 2019), and the concept of the Internet of Old Things (Cho et al., 2021) introduce ideas for enabling the IoT experience with existing appliances through remote-controllable add-on hardware. Also, there have been projects that augment new IoT functions to specific appliances that people commonly own (kettle: Brereton et al., 2015; speaker: Baharin et al., 2015). Utilizing everyday objects aligns with the aforementioned studies in terms of avoiding the need for installing new IoT sensor devices in the environment. Thus, as IoT coordination involves habitual gestures on objects, it would also help in avoiding behavior changes caused by the sensors. With the aim of achieving gradual and sustainable smartification of the environment, it may be possible to reconceptualize smart environments and IoT devices in a way that minimizes user reluctance towards their adoption.

6. 3. Breaking Free from Centralized Smartphone Applications

With the desire for instant access over every device, control interfaces have transitioned from being part of the product itself to being centralized in smartphones. Initially, appliances were operated by wired switches until the first remote controls emerged in the 1960s. Today, technology allows multiple appliances to be controlled via a single universal controller, typically a smartphone. Appliance manufacturers now prefer providing companion smartphone apps with their products because these apps are cost-effective, save space, and contribute to a more compact design (Becker et al., 2020). As market trends have evolved, smartphones have gradually taken control of the majority of appliances and IoT devices.

The proposed IoT coordination offers an alternative approach that aims to remove controllers from smartphones and redistribute them within the physical environment and context. There are several reasons to deviate from the trend of centralized IoT device control. First, using smartphones can lead to a loss of context as users must summon the control interface to the screen, diverting attention from the physical environment. As the number of IoT devices grows, users may need more time to navigate through numerous control applications, exacerbating the problem. Xiao et al. (2017) highlight that such interaction bottlenecks can negatively impact IoT device adoption and act as a barrier to integrating new smart features into daily life. The problem is likely to become more complex as IoT device capabilities expand, enabling a wider range of control scenarios. Additionally, with the growing feasibility of joint usage of connected devices, situations requiring simultaneous control of multiple devices are likely to emerge.

Figure 11

(left) Conventional means for appliance controls. (right) Redistributing the controls in environment and context by utilizing everyday objects as IoT controllers.

To effectively manage upcoming smart environments, exploring new types of control interfaces is crucial. Otherwise, users might have to individually control apps to switch their TV displays to theater mode, set their speakers to a low-frequency EQ, lower their window blinds, and dim their ceiling lights to create an immersive movie-watching experience. This tedious and time-consuming process may discourage users from creating a smart environment. Providing control at any time and from anywhere is insufficient; it is equally important to provide control in the right way (Fischer, 2012). A viable solution is to have user-configured controllers well-distributed throughout the environment and summoned in the right context. This concept applies not only to everyday objects explored in this research but also to decentralized controls in general, such as programmable switches and distributed sensors. As research into various methods of returning controls to their original context and environment advances, it will further strengthen the vision of IoT technology being truly accessible and impactful in users’ daily lives.

6. 4. Limitations and Future Works

While the ‘e-Rings’ system enabled the testing and exploration of these cases, the investigation was limited to short-term user experiences. Although the thesis findings demonstrate the potential benefits of incorporating everyday objects into IoT ecosystem, the question remains as to whether these advantages can be maintained in real-world user environments. Conducting long-term investigations or a deployment study in future work would uncover new opportunities and challenges. By analyzing IoT coordination cases that can persist, as well as cases that are difficult to sustain in real-life situations, a more detailed categorization of everyday objects can be established. These categories may also provide insights into what and how the IoT coordination system should be refined and optimized. Next, the IoT coordination applications created by the ‘e-Rings’ system were limited to the idea of a single person in a personal space. In a multi-user environment, which is much more complex than the office environment considered in this study, there are not only personal belongings, but also the belongings of others, shared items, and public IoT appliances to consider. Further investigation into these opportunities and challenges is expected to advance the proposed IoT coordination approach into a more robust concept


7. Conclusion

The concept of a smart environment had been a topic of discussion for decades, but recent advancements in Internet of Things (IoT) technology made it more achievable. However, while IoT devices were becoming more prevalent in our living spaces, their usage was still rudimentary. The ‘e-Rings’ system aimed at effective and intuitive IoT coordination by leveraging everyday objects as natural control interfaces. The study’s findings highlights the usability and acceptability of the ‘e-Rings’ system, as participants successfully created unique and personalized IoT applications, demonstrating both ease of use and the capability to integrate IoT coordination seamlessly into their routines in playful and engaging way. The system’s demonstration based approach and wearable platform provided an effective solution to the challenges posed by traditional IoT coordination methods. Future work considers long-term deployment studies to better understand the sustainability and real-world applicability of this approach, ultimately creating a more integrated and seamless smart environment for users. This research contributes to the design of intuitive IoT interfaces by offering a fresh perspective on integrating non-digital artifacts into future IoT ecosystems, enriching the broader discourse on IoT user experience and highlighting the importance of user-centered design in the evolution of smart environments.

Notes

Citation: Kim, C. (2024). Transforming Everyday Objects into IoT Control Interfaces: Design and Evaluation of the ‘e-Rings’ System. Archives of Design Research, 37(5), 29-49.

Copyright : This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/3.0/), which permits unrestricted educational and non-commercial use, provided the original work is properly cited.

References

  • Abowd, G. D., Bobick, A. F., Essa, I. A., Mynatt, E. D., & Rogers, W. A. (2002). The aware home: A living laboratory for technologies for successful aging.
  • Ambe, A. H., Brereton, M., Soro, A., Buys, L., & Roe, P. (2019, May). The adventures of older authors: Exploring futures through co-design fictions. In Proceedings of the 2019 CHI Conference on Human Factors in Computing systems (pp. 1-16). [https://doi.org/10.1145/3290605.3300588]
  • Baharin, H., Viller, S., & Rintel, S. (2015). Sonicair: supporting independent living with reciprocal ambient audio awareness. ACM Transactions on Computer-Human Interaction (TOCHI), 22(4), 1-23. [https://doi.org/10.1145/2754165]
  • Becker, V., Rauchenstein, F., & Sörös, G. (2020). Connecting and controlling appliances through wearable augmented reality. Augmented Human Research, 5(1), 2. [https://doi.org/10.1007/s41133-019-0019-0]
  • Bellucci, A., Vianello, A., Florack, Y., & Jacucci, G. (2016). Supporting the serendipitous use of domestic technologies. IEEE Pervasive Computing, 15(2), 16-25. [https://doi.org/10.1109/MPRV.2016.26]
  • Bellucci, A., Vianello, A., Florack, Y., Micallef, L., & Jacucci, G. (2019). Augmenting objects at home through programmable sensor tokens: A design journey. International Journal of Human-Computer Studies, 122, 211-231. [https://doi.org/10.1016/j.ijhcs.2018.09.002]
  • Berger, A., Odom, W., Storz, M., Bischof, A., Kurze, A., & Hornecker, E. (2019, May). The inflatable cat: Idiosyncratic ideation of smart objects for the home. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (pp. 1-12). [https://doi.org/10.1145/3290605.3300631]
  • Bitkom Research. (2022). Which device do you use to operate your smart home applications? Retrieved 02/19/2024 from https://de.statista.com/statistik/daten/studie/1168831/umfrage/smart-home-geraete-bediengeraete/.
  • Brereton, M., Soro, A., Vaisutis, K., & Roe, P. (2015, April). The messaging kettle: Prototyping connection over a distance between adult children and older parents. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (pp. 713-716). [https://doi.org/10.1145/2702123.2702462]
  • Brush, A. B., Lee, B., Mahajan, R., Agarwal, S., Saroiu, S., & Dixon, C. (2011, May). Home automation in the wild: challenges and opportunities. In proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 2115-2124). [https://doi.org/10.1145/1978942.1979249]
  • Buchenau, M., & Suri, J. F. (2000, August). Experience prototyping. In Proceedings of the 3rd conference on Designing interactive systems: processes, practices, methods, and techniques (pp. 424-433). [https://doi.org/10.1145/347642.347802]
  • Buettner, M., Prasad, R., Philipose, M., & Wetherall, D. (2009, September). Recognizing daily activities with RFID-based sensors. In Proceedings of the 11th international conference on Ubiquitous computing (pp. 51-60). [https://doi.org/10.1145/1620545.1620553]
  • Cho, H., Kim, H. J., Lee, J., Kim, C. M., Bae, J., & Nam, T. J. (2021, June). IoTIZER: A Versatile Mechanical Hijacking Device for Creating Internet of Old Things. In Proceedings of the 2021 ACM Designing Interactive Systems Conference (pp. 90-103). [https://doi.org/10.1145/3461778.3461996]
  • Cohn, G., Morris, D., Patel, S. N., & Tan, D. S. (2011, May). Your noise is my command: sensing gestures using the body as an antenna. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 791-800). [https://doi.org/10.1145/1978942.1979058]
  • Desjardins, A., Biggs, H. R., Key, C., & Viny, J. E. (2020, April). IoT data in the home: Observing entanglements and drawing new encounters. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (pp. 1-13). [https://doi.org/10.1145/3313831.3376342]
  • Desjardins, A., Viny, J. E., Key, C., & Johnston, N. (2019, May). Alternative avenues for IoT: Designing with non-stereotypical homes. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (pp. 1-13). [https://doi.org/10.1145/3290605.3300581]
  • Fischer, G. (2012, May). Context-aware systems: the 'right' information, at the 'right' time, in the 'right' place, in the 'right' way, to the 'right' person. In Proceedings of the international working conference on advanced visual interfaces (pp. 287-294). [https://doi.org/10.1145/2254556.2254611]
  • Froehlich, J. E., Larson, E., Campbell, T., Haggerty, C., Fogarty, J., & Patel, S. N. (2009, September). HydroSense: infrastructure-mediated single-point sensing of whole-home water activity. In Proceedings of the 11th international conference on Ubiquitous computing (pp. 235-244). [https://doi.org/10.1145/1620545.1620581]
  • Funk, M., Chen, L. L., Yang, S. W., & Chen, Y. K. (2018). Addressing the need to capture scenarios, intentions and preferences: Interactive intentional programming in the smart home. International Journal of Design, 12(1), 53-66.
  • García-Herranz, M., Haya, P. A., & Alamán, X. (2010). Towards a Ubiquitous End-User Programming System for Smart Spaces. Journal of Universal Computer Science, 16(12), 1633-1649.
  • Gummeson, J., Mccann, J., Yang, C., Ranasinghe, D., Hudson, S., & Sample, A. (2017). RFID light bulb: Enabling ubiquitous deployment of interactive RFID systems. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 1(2), 1-16. [https://doi.org/10.1145/3090077]
  • Gupta, S., Reynolds, M. S., & Patel, S. N. (2010, September). ElectriSense: single-point sensing using EMI for electrical event detection and classification in the home. In Proceedings of the 12th ACM international conference on Ubiquitous computing (pp. 139-148). [https://doi.org/10.1145/1864349.1864375]
  • Huang, J., & Cakmak, M. (2015, September). Supporting mental model accuracy in trigger-action programming. In Proceedings of the 2015 acm international joint conference on pervasive and ubiquitous computing (pp. 215-225). [https://doi.org/10.1145/2750858.2805830]
  • Hwang, A., & Hoey, J. (2012, October). Smart home, the next generation: Closing the gap between users and technology. In 2012 AAAI Fall Symposium Series, 14-21.
  • Intille, S. S. (2002). Designing a home of the future. IEEE pervasive computing, 1(2), 76-82. [https://doi.org/10.1109/MPRV.2002.1012340]
  • Jakobi, T., Ogonowski, C., Castelli, N., Stevens, G., & Wulf, V. (2017, May). The catch (es) with smart home: Experiences of a living lab field study. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (pp. 1620-1633). [https://doi.org/10.1145/3025453.3025799]
  • Kim, J. W., Kim, H. J., & Nam, T. J. (2016, May). M. gesture: an acceleration-based gesture authoring system on multiple handheld and wearable devices. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (pp. 2307-2318). [https://doi.org/10.1145/2858036.2858358]
  • Kim, C. M., & Nam, T. J. (2022, June). Exploration on Everyday Objects as an IoT Control Interface. In Proceedings of the 2022 ACM Designing Interactive Systems Conference (pp. 1654-1668). [https://doi.org/10.1145/3532106.3533472]
  • Laput, G., Xiao, R., & Harrison, C. (2016, October). Viband: High-fidelity bio-acoustic sensing using commodity smartwatch accelerometers. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (pp. 321-333). [https://doi.org/10.1145/2984511.2984582]
  • Laput, G., Yang, C., Xiao, R., Sample, A., & Harrison, C. (2015, November). Em-sense: Touch recognition of uninstrumented, electrical and electromechanical objects. In Proceedings of the 28th annual ACM symposium on user interface software & technology (pp. 157-166). [https://doi.org/10.1145/2807442.2807481]
  • Laput, G., Zhang, Y., & Harrison, C. (2017, May). Synthetic sensors: Towards general-purpose sensing. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (pp. 3986-3999). [https://doi.org/10.1145/3025453.3025773]
  • Li, H., Ye, C., & Sample, A. P. (2015, April). IDSense: A human object interaction detection system based on passive UHF RFID. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (pp. 2555-2564). [https://doi.org/10.1145/2702123.2702178]
  • Liu, X., Shi, Y., Yu, C., Gao, C., Yang, T., Liang, C., & Shi, Y. (2023). Understanding In-Situ Programming for Smart Home Automation. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 7(2), 1-31. [https://doi.org/10.1145/3596254]
  • Ludwig, T., Boden, A., & Pipek, V. (2017). 3D printers as sociable technologies: taking appropriation infrastructures to the internet of things. ACM Transactions on Computer-Human Interaction (TOCHI), 24(2), 1-28. [https://doi.org/10.1145/3007205]
  • Philipose, M., Fishkin, K. P., Perkowitz, M., Patterson, D. J., Fox, D., Kautz, H., & Hahnel, D. (2004). Inferring activities from interactions with objects. IEEE pervasive computing, 3(4), 50-57. [https://doi.org/10.1109/MPRV.2004.7]
  • Ramakers, R., Anderson, F., Grossman, T., & Fitzmaurice, G. (2016, May). Retrofab: A design tool for retrofitting physical interfaces using actuators, sensors and 3d printing. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (pp. 409-419). [https://doi.org/10.1145/2858036.2858485]
  • Scott, J., Bernheim Brush, A. J., Krumm, J., Meyers, B., Hazas, M., Hodges, S., & Villar, N. (2011, September). PreHeat: controlling home heating using occupancy prediction. In Proceedings of the 13th international conference on Ubiquitous computing (pp. 281-290). [https://doi.org/10.1145/2030112.2030151]
  • techUK. (2022). The state of the Connected Home. Retrieved August 14, 2023 from https://www.techuk.org/resource/state-of-the-connected-home-2022.html.
  • Ur, B., McManus, E., Pak Yong Ho, M., & Littman, M. L. (2014, April). Practical trigger-action programming in the smart home. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 803-812). [https://doi.org/10.1145/2556288.2557420]
  • Ur, B., Pak Yong Ho, M., Brawner, S., Lee, J., Mennicken, S., Picard, N., ... & Littman, M. L. (2016, May). Trigger-action programming in the wild: An analysis of 200,000 ifttt recipes. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (pp. 3227-3231). [https://doi.org/10.1145/2858036.2858556]
  • Weiser, M. (1999). The computer for the 21st century. ACM SIGMOBILE mobile computing and communications review, 3(3), 3-11. [https://doi.org/10.1145/329124.329126]
  • Wilson, D. H., & Atkeson, C. (2005, May). Simultaneous tracking and activity recognition (STAR) using many anonymous, binary sensors. In International Conference on Pervasive Computing (pp. 62-79). Berlin, Heidelberg: Springer Berlin Heidelberg. [https://doi.org/10.1007/11428572_5]
  • Woo, J. B., & Lim, Y. K. (2015, September). User experience in do-it-yourself-style smart homes. In Proceedings of the 2015 ACM international joint conference on pervasive and ubiquitous computing (pp. 779-790). [https://doi.org/10.1145/2750858.2806063]
  • Wu, J., Harrison, C., Bigham, J. P., & Laput, G. (2020, April). Automated class discovery and one-shot interactions for acoustic activity recognition. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (pp. 1-14). [https://doi.org/10.1145/3313831.3376875]
  • Wu, J., Harrison, C., Bigham, J. P., & Laput, G. (2020, April). Automated class discovery and one-shot interactions for acoustic activity recognition. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (pp. 1-14). [https://doi.org/10.1145/3313831.3376875]

Figure 1

Figure 1
Utilizing various types of everyday objects as a control interface for IoT devices.

Figure 2

Figure 2
Systemetic overview of demonstration based IoT coordination approach.

Figure 3

Figure 3
Adopting wearable devices as a sensing platform for everyday objects.

Figure 4

Figure 4
(left) Watch-with-ring type wearable hardware device, (right) a custom IoT device controller attached on the lamp.

Figure 5

Figure 5
Simplified feature diagram for training an IoT coordination data.

Figure 6

Figure 6
Simplified feature diagram for classifying gestures on objects and triggering the IoT devices.

Figure 7

Figure 7
(a) Office workspace equipped with four types of IoT devices; (b) office accessories attached with RFID tags.

Figure 8

Figure 8
Participants creating IoT applications on electric kettle.

Figure 9

Figure 9
Summary of system usability scale result.

Figure 10

Figure 10
Desk lamp controlled by picking up a book and gently tapping it on the desk.

Figure 11

Figure 11
(left) Conventional means for appliance controls. (right) Redistributing the controls in environment and context by utilizing everyday objects as IoT controllers.