Research on the Usability Test of Interface Design in e-Government – Focused on Qingdao e-Government Website

1. Introduction

E-government considerations play a crucial role in the last two decades related to the issue of the smart city (Fietkiewicz & Stock, 2015; Mainka et al., 2013; Stock, 2011). The governmental website is an information release platform for e-government. The data and massive amounts of information disclosed on government websites involve all aspects of public life. As a result, how to grant users better access and use has become a key issue. From the user's point of view, the high accessibility of government websites can make users’ experience of websites effective, efficient and satisfactory. Therefore, a scientific usability evaluation of government websites from the perspective of users is key if designers are to help bring citizens’ smart city life.

According to the data of the seventh national census, there are 21 megacities (cities whose permanent population exceeds 10 million) in China. At present, 17 cities have established their own government websites to build smart cities. According to the results of a long-term public opinion survey released by the research team of Harvard Grey Center (2020) and the 2022 Edelman Trust Barometer (2022), Chinese citizens have more than 91% confidence in the government. According to a survey on the degree of trust in different countries in 2021, approximately 80% of Chinese respondents said they trusted the media, which is the highest proportion of any country surveyed (Statista, 2022). Given this, how to evaluate the usability of e-government websites from the perspective of users, what scientific and effective methods are used to evaluate the usability of government websites, and what needs to be optimized in government websites are all issues that must be considered when building a city smarter.

The Qingdao Government Open Data Network was selected for participants testing because it represents the highest level of data openness for government websites in mainland China (2021), according to the China Open Data Index 2021 report. In this paper, the usability of the Qingdao government websites is assessed, and the experiment utilizes the CIF criteria of the usability test scheme.

2. Literature Review

The Working Group on E-government in the Developing World (Pacific Council on International Policy, 2021) denotes that the goal of e-government is to allow greater public access to information, facilitate more accessible government services, and make government more accountable to citizens. It includes governmental websites, governmental social media channels, and other digital governmental services (Palvia & Sharma, 2007). This paper focuses on governmental websites.

According to Jacob Nielsen (1994b), usability refers to a computer system's capacity to be used effectively. Learnability, Efficiency, Errors, Memorability, and Satisfaction are the five aspects that constitute usability. Its goal is to determine participant satisfaction with the product by collecting quantitative (e.g., successful completion rates, errors, and time on task) and qualitative (e.g., likes or dislikes, and comments) data (Nist, 2021).

Website Usability is the extent to which a site can be used by a specified group of users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use (Powell, 2002). Therefore, website usability is critical in the design and development of government portal websites. From the perspective of the residents “many of the primary e-government functions towards citizens involve the web-based provision of government information and services” (Manoharan & Carrizales, 2011, p. 284). If governmental websites fail to perform readily from a usability standpoint and instead block less-knowledgeable citizens from satisfactory contact with their government, the evolution of e-Government will be stymied (Baker, 2009).

The construction of the criteria system is the main premise for investigating the usability of government websites and this has become the focus of a number of researchers and practitioners. Huang and Chao (2001) compiled a list of 18 criteria of website usability issues, evaluated 33 government websites using these criteria, and proposed seven specific improvement solutions as a result. David L. Baker (2009) employed a content analysis approach to extract and compare government website usability evaluation criteria based on navigation, online help, and accessibility. Stowers' (1999) study, which used expert interviews, presents numerous characteristics of the usability of e-government websites, such as “user help,” “online service,” and “search and navigation”. Eighmey and McCord (1998) have proposed 17 website usability criteria, such as personal participation, valuable information, a straightforward structure, a desire to create communication, etc.

Numerous researchers have assessed the usability of government websites and discussed and examined the existing dimensions of evaluation as well as the most appropriate assessment techniques and tools. This has led many to employ automated methods of assessing government websites. For example, Ellison (2004) utilized Bobby software and other tools to test the usability of White House official websites and uncover challenges that hindered users from successfully accessing them. Li et al. (2011) evaluated website accessibility regulations and standards, as well as related evaluation research. They employed the Achecker online software tool to evaluate the accessibility of government websites in four recognized international cities, as well as Beijing and Shanghai, in accordance with the WCAG2.0 (Website Content Availability Guide) Level A standard.

Some researchers have investigated the usability of government websites from the perspective of the user. Jong and Lentz (2006) evaluated the usability of government portals in 15 cities by creating scenarios, completing surveys and identifying potential issues with the human-computer interaction interface and website navigation. This study examined the feasibility of a new expert-centred method for evaluating municipal (and other government) websites, and assisted experts in understanding the demands of users (Jong & Lentz, 2006).

An analysis of relevant research literature on usability evaluation of government websites accounts for large populations reveals that usability evaluation methods can be roughly divided into two categories: automated evaluation and non-automated evaluation. The automated testing software has mostly been used for the automatic evaluation. This includes tools such as Ask Alice, Bobby, WCAG2.0 or WebXact, which are designed to check government websites for faults that could lead to accessibility problems. Nevertheless, automated testing programs can erroneously categorize components as accessible or inaccessible, do not account for individual variations, do not address usability or functionality issues, do not address compatibility issues with assistive technologies, and overlook numerous accessibility issues that a person might recognize, among other restrictions (Jaeger, 2003; Mueller, 2008; Slatin & Rush, 2003; Sloan et al., 2002). Expert testing and user testing are the two basic types of non-automated evaluation. Expert testing consists mostly of heuristic and discussion evaluations, whereas user testing consists primarily of observation, questionnaire survey, and interview methods.

Prior studies suggest that employing the users' usability testing approach might help researchers evaluate website usability more objectively and efficiently. Jaeger (2006) argued that user behaviour is a record of real activity rather than a subjective understanding of designers or experts, so evaluating user behaviour can better reflect the objectivity of usability. At present, there are a few experimental studies based on users' actual experience in China, and the relationship between usability of government websites and users' actual experience has not been established.

3. Methodology

3. 2. Userbility Test

3. 2. 1. Participants for the Test

According to CIF, the participants in the test should have basic knowledge in the usability of the tested products. The groups targeted by government websites should have basic computer skills and experience in surfing the Internet. A significant number of participants must be tested to generate valid summative statistical analyses. For this purpose, 8 or more subjects/cells (segment) are recommended. 10 eligible residents in Qingdao were selected for this test (see Table 1).

Table 1

Information about the participants

3. 2. 2. Task for the Test

Stowers (2002) conceptually integrates the e-government factors along six dimensions of website usability: online services, quality of user-help features, quality of service navigation tools, legitimacy features, and accessibility. Baker (2009) goes further to integrate the research of Nielsen (1994), Pearrow (2000) and Nielsen and Tahir (2001). Baker reviews these aspects and typical variables: online services; user-help; navigation; legitimacy; information architecture, and accessibility accommodations. In addition, according to the relevant standards (Slatin & Rush, 2003), 10 corresponding test tasks were set up, which covered the browsing, searching, updating and retrieval of government website information and the interaction between users and websites. The first eight tasks were designed according to the existing web page menu categories (scenarios, domains) in the website, and the remaining two tasks were from the aforementioned usability testing criteria for government websites (as shown in Table 2).

Table 2

Classification of test tasks

4. Result

4. 1. Overall analysis of task completion

Table 3 reflects the overall situation from the perspective of participants after ten tasks on the Qingdao e-government website.

Table 3

Overall situation of participants' task completion

(1) Effectiveness metrics

Table 3 details the statistics for effectiveness metrics. The average frequency of independent completion of all participants is 87%, and the corresponding average frequency of assists and frequency of errors are 45% and 5%, respectively. The average number of help times was 4.7. Each tester sought help in the test. The above data shows to a certain extent that the participants in the test will encounter certain difficulties when they complete the task independently, and they need additional assistance from the experimental administrator to effectively complete the task. In addition, the average number of errors is as high as 9.7, and the standard deviation of the number of errors is only 3.47. The participants have a low degree of volatility, which shows that the probability of each person making errors is quite high when completing the web page test. Furthermore, the standard deviations of independent completion rate, assisted completion rate and failure rate are 10.59, 13.54 and 7.07, respectively, which indicates that the performance of participants in the test fluctuates greatly.

(2) Efficiency metrics

The average time for testers to complete all tasks on the Qingdao Government Open Data Network website is 16.51 minutes, with a standard deviation of 5.67 and an average task time of 1.65 minutes. Among them, the participants who took the longest time spent an average of 2.36 minutes to complete a task, and the shortest time was 0.76 minutes. Another data point reflecting the efficiency index: the completion rate/completion time is only 6.67%, and the standard deviation is 2.99. These figures show that the efficiency index of the website is also low.

(3) Satisfaction metrics

The questionnaire adopts a Likert 7-point scale. The descriptive analysis of SPSS (Table 4) indicates that there are no outliers in the current data, so we can directly describe and analyse the average value. The average score of each question is then converted from a seven point scale to a hundred point scale (illustrated in the diagram). The statistical results show that after the conversion to the percentile system the overall average degree of satisfaction is 54. According to the results of the percentile satisfaction questionnaire, only two questions yielded a degree of satisfaction exceeding 60 points, proving that the testers' satisfaction degree with all aspects of the website is generally low.

Table 4

Descriptive analysis of satisfaction questionnaire

The satisfaction test results indicate that the participants were relatively satisfied with the visual experience of the website, navigation menu and content layout of the webpage (Tasks 2, 7 and 8). According to the performance of the testers in the tasks, the testers seek help and make mistakes in the assigned tasks with a high help rate of 45%, which also illustrates to some degree that the testers will encounter more difficulties when completing the corresponding tasks on this website, which will affect the completion of the tasks.

4. 2. Specific analysis of task completion

In analysing the overall situation, it is important to analyse the specific completion of different tasks, so as to find out more specific differences.

(1) Analysing the results of Tasks 1, 2, 5, 6 and 8 confirms that the main purpose of these tasks is to determine whether the testers can easily find and browse relevant practical information on the website. If the testers can quickly and conveniently find various relevant information on the website, it shows that the information layout is reasonable and practical, and data is relatively easy to find, leading users to have a convenient experience. During the test, most testers searched for related tasks through search engines. The keywords of Task 5 and Task 8 are special and not easy to confuse, so the average time to complete these two tasks is relatively short. Participants’ Mean Time-On-Task time is less than 1 minute, and their independent completion rates are 90% and 100% respectively. However, the keywords of Task 1, Task 2 and Task 6 are not rich enough, and the average time to complete this task is high. The Mean Time-On-Task is 3.87 minutes, 3.32 minutes and 2.58 minutes respectively, and the independent completion rates are 70% and 60% respectively. Because keywords have no fuzzy search function, testers can not find the results efficiently, which greatly increases the error rate and help rate. The problem of there being no keyword fuzziness affects the completion of all tasks, and half of the participants clearly indicated it in the feedback and comments in Table 5.

Table 5

Analysis of the completion results of all tasks

(2) Analysing the results of Tasks 3, 4 and 7 reveals that different information retrieval and searching methods will affect the efficiency of users in locating the required information and completing the corresponding tasks, thus directly affecting the website’s usability. When tackling these tasks, the testers can find that the information inquiry service of the Qingdao Government Open Data Network has a special column named “Map Service” in the middle of the homepage. The retrieval mode of this column belongs to integrated retrieval, which has the function of classified inquiry, and its retrieval objects include the inquiries of the subway, shopping malls, shops and so on.

According to the website settings, the testers can easily retrieve the required content by selecting the corresponding retrieval object and inputting the corresponding retrieval words in the retrieval box. However, at present, the board does not function properly, so the testers can only find related tasks through search engines, which increases the access time and error rate of each task. Furthermore, when searching for the target places close to the participants' residences, because the website does not provide corresponding visual maps, the participants can only query information using their own experience and other helpful software. This greatly increases the browsing time and affects the access efficiency, which is unfavourable for users to browse (40% of the participant's feedback in the satisfaction questionnaire).

(3) Task 9 result analysis: In terms of user registration, this task reflects whether the website can respond well to relevant information from users and how easy it is for users to complete registration. It is the most efficient task of all tasks, which is reflected in the result analysis of task 8. In the process of this task, the registration entrance of the Qingdao Government Open Data Network is prominently located at the top of the homepage, and the location of the entrance also affects the efficiency of task completion to a certain extent.

(4) Task 10 result analysis: The website’s online help feature is a tool to help users effectively find website information. Online help is an important part of the website information layout, and whether the information related to user help on the website can be easily found is an important factor that affects the user experience. The “Smart Help” feature of the webpage is prominently displayed on the right side of the homepage, which is easy for testers to find. It is in line with the usability standard guide and the “user help” dimension mentioned by Stowers (2002), and the independent completion rate of users reaches 100%.

(5) Analysis of other problems: 60% of users mentioned in the website satisfaction feedback that the information was not updated in a timely manner and the latest information could not be found. Infrequent information updates seriously affect the usability and efficiency of government websites. 50% of users mentioned the lack of rational information distribution and classification, and insufficient related words, which also seriously affected the human services and use efficiency of the website. Based on the analysis of the different performances and reasons for website usability during the task experiment, it is apparent that the main factors that affect the user experience and website usability of testers are information (keyword) retrieval mode, map/visual function, information layout and interactivity, etc.

5. Conclusion

In this study, the CIF is used to objectively and quantitatively measure and compare the usability of the website of the Qingdao Public Data Open Network. The test results are consistent with users' subjective feelings and reactions, and reflect the actual usability level of the website to a large extent.

(1) Compatibility of access users

Government websites are provided to satisfy the needs of different access users, with information and services divided according to user categories, thus improving the order of the content and thus the usability of the website. The completion of the tasks reveals that the effectiveness, efficiency and other related criteria of the "Qingdao Government Open Data Network" website are low. The user satisfaction of the testers during the task completion is not high, which also shows that the usability criteria are directly proportional to the user satisfaction to a certain extent. In addition, by comparing the variance of relevant data such as independent completion rate, error times and help times, it becomes clear that the standard deviation associated with the Qingdao Government Open Data Network website is relatively large. It indicates that within this environment, the testers' performance fluctuates vastly, which means the government website needs to be better compatible and personalized for users from various industries. The division of relevant information and services according to user categories increases the orderliness of the content of the website and thus can improve the usability of the website. Although the website follows the WCAG 2.0 international standard and the Chinese Technical Requirements for Web Accessibility Design and related specifications in providing links to people with disabilities — mobility impairments, visual impairments and reduced reading ability due to aging etc. — the entrance to this access point is not obvious.

(2) Keyword search method (fuzzy search)

The absence of an fuzzy search function on the website increases the search time and affects user loyalty. Comparing the completion of different tasks of the testers in the experiment reveals that when a complex keyword search is not involved, such as Task 5, Task 8, Task 9 and Task 10, the independent completion rate, completion time, effectiveness and efficiency of the tasks are all high. In addition, the analysis and comparison of the experimental process of each specific task demonstrates that the main factor that affect users' actual experience is the way in which information is retrieved by keywords and the need to provide keyword fuzzy search functionality. These factors play an important role in the positive experience of government website users and effective access to government information and services. Providing keyword fuzzy search can therefore greatly enhance the efficiency of using government websites.

(3) Information retrieval

Information retrieval is directly related to the efficiency with which users can find information and the usability of the website. Through the experiment, it can be found that firstly, the information retrieval function should be set in the homepage to meet the users' retrieval needs. Also, the information retrieval mode affects the users' experience, among which the retrieval mode that integrates the information of the whole website and can be categorized can be more convenient for users to find information and improve the retrieval efficiency. Although the Qingdao government website has set up the classification of the search content, the section is not yet running properly, which greatly affects the search efficiency. For example, task 4, although all participants could complete it independently, it took 1.5 minutes, and the failure rate was 0.6. Secondly, the setting of information keywords will also affect the user experience. For example, when completing Task 8 (without confusing keywords), compared with Task 2, the average time for participants to complete tasks is reduced by 2.66 minutes. The independent completion rate also increases by 30%, and the failure rate is zero. In addition, in the satisfaction survey, the average score of the third question “easy to use” was 54, which is low. Furthermore, in the satisfaction feedback, half of the participants mentioned that the search function was flawed. This further demonstrates that proper classification of information retrieval can make it easier for users to find information and improve retrieval efficiency.

(4) Visualization function

The lack of certain visualization functions can make users significantly less efficient in their use and increase the time taken to complete tasks. Proper visualization is helpful for users in locating information, and rich visualization functions can improve the accessibility of websites. For example, when Task 7 was completed, the test participants did not provide the corresponding visual map and navigation location during the searching process, so the user could not quickly find the address nearest to their location. From the experimental results, the independent completion rate of this task is only 80%, and the average number of errors is 0.5. In addition, throughout the duration of the experiment, when the participants completed the task, all of them made mistakes or failed due to the missing map. Therefore, the placement and optimization of visualization functions in government websites are critical in order to improve their usability.

(5) Task path link length and location

The more layers of links a user has to complete a task, the more likely they are to become confused. For example, when the participants completed Task 10, they could quickly find “User Help” on the homepage and complete the task after one link. The independent completion rate was 100%, but because the location was not obvious in the sidebar, the average time was 0.43 minutes; while in the completion of Task 9, the user registration task, the registration portal is located in a prominent position on the home page (the top), so the average time was 0.38 minutes, the fastest among all tasks. Therefore, the number of link layers required for users to complete each task should be minimized in the e-government website. This saves the response time of the page and the user's access time, avoids the generation of information confusion, and thus improves the efficiency of the user's task completion. Therefore, the number of link layers required for users to complete each character should be minimised in government websites, thus saving page response time and user access time and avoiding the creation of information confusion as a way to improve the efficiency of user task completion.

(6) Information layout of web pages

A simple website interface provides a relaxing experience for the user, whereas a complex and cluttered interface tends to create a sense of tension and thus directly affects the user's efficiency in using the website. The layout of information on the webpage directly affects the usability of government websites and the users’ access behaviour. Important operation portals on government websites (such as “user help”) should be clearly visible, and the classification of information on government websites should be clear and reasonable, so as to reduce users' cognitive and searching burden. According to the survey satisfaction data, the participants considered the layout of the website to be clear and logical. Although the participants in the tesst were influenced by some objective questions about keywords in the experiment, because the commonly used information was reasonably classified and its title was eye-catching, the participants were nonetheless able to accurately find it when completing the corresponding tasks. Through this study, the need to improve the interface of the Qingdao government website was confirmed, and the user's information and service efficiency should be improved.

In summary, government websites need to be improved in the following areas to enhance usability: categorise different information to improve user compatibility; add fuzzy search functions to enhance efficiency; run categorisable information retrieval functions; add appropriate visualization to reduce the error rate of searching; reduce the number of levels of links required for users to complete each task to save user access time and web access time; and a streamlined, clear website layout to help users complete tasks more accurately.

References

• Baker, D. L. (2009). Advancing e-government performance in the United States through enhanced usability benchmarks. Government Information Quarterly, 26(1), 82-88. [https://doi.org/10.1016/j.giq.2008.01.004]
• CHINA OPEN DATA INDEX. (2021). Retrieved from http://ifopendata.fudan.edu.cn/.
• Ellison, J. (2004). Assessing the accessibility of fifty United States government Web pages: Using Bobby to check on Uncle Sam. First Monday. [https://doi.org/10.5210/fm.v9i7.1161]
• Huang, C. J., & Chao, M.-H. (2001). Managing WWW in public administration: Uses and misuses. Government Information Quarterly, 18(4), 357-373. [https://doi.org/10.1016/S0740-624X(01)00085-5]
• Jaeger, P. (2003). The importance of measuring the accessibility of the federal e-government: What studies are missing and how these issues can be addressed. Information Technology and Disabilities, 9(1).
• Jaeger, P. T. (2006). Assessing Section 508 compliance on federal e-government Web sites: A multi-method, user-centered evaluation of accessibility for persons with disabilities. Government Information Quarterly, 23(2), 169-190. [https://doi.org/10.1016/j.giq.2006.03.002]
• Jong, M. D., & Lentz, L. (2006). Scenario evaluation of municipal Web sites: Development and use of an expert-focused evaluation tool. Government Information Quarterly, 23(2), 191-206. [https://doi.org/10.1016/j.giq.2005.11.007]
• Long-term survey reveals Chinese government satisfaction - Harvard Gazette. (2020). Retrieved from https://news.harvard.edu/gazette/story/2020/07/long-term-survey-reveals-chinese-government-satisfaction/.
• Mainka, A., Fietkiewicz, K., Kosior, A., Pyka, S., & Stock, W. (2013). Maturity and usability of e-government in informational world cities. 292.
• Manoharan, A., & Carrizales, T. J. (2011). Recent trends in e-government: States' and local governments' utilisation of websites. International Journal of Electronic Governance, 4(4), 283-303. [https://doi.org/10.1504/IJEG.2011.046014]
• Mueller, J. (2008). Accessibility for everybody: Understanding the Section 508 accessibility requirements. Apress.
• Nielsen, J. (1994a). 10 Usability Heuristics for User Interface Design. Nielsen Norman Group. Retrieved from https://www.nngroup.com/articles/ten-usability-heuristics/.
• Nielsen, J. (1994b). Usability engineering. Morgan Kaufmann. [https://doi.org/10.1016/B978-0-08-052029-2.50009-7]
• Nist. (2021, April 12). Research Methods. NIST. Retrieved from https://www.nist.gov/itl/iad/visualization-and-usability-group/research-methods.
• Palvia, S. C. J., & Sharma, S. S. (2007). E-government and e-governance: Definitions/domain framework and status around the world. 5, 1-12.
• Pearrow, M. (2000). Web site usability handbook. Rockland, MA: Charles River Media. Inc.
• Powell, T. (2002). Web design. McGraw-Hill Professional Publishing.
• Scholtz, J. C. (2000). Common Industry Format for Usability Test Reports. Retrieved from https://www.nist.gov/publications/common-industry-format-usability-test-reports. [https://doi.org/10.1145/633292.633470]
• Slatin, J. M., & Rush, S. (2003). Maximum accessibility: Making your web site more usable for everyone. Addison-Wesley Professional.
• Sloan, D., Gregor, P., Booth, P., & Gibson, L. (2002). Auditing accessibility of UK Higher Education web sites. Interacting with Computers, 14(4), 313-325. [https://doi.org/10.1016/S0953-5438(01)00056-X]
• Stowers, G. N. (2002). The state of federal websites. Price Waterhouse Coopers Endowment for the Business of Government.