RESEARCH ARTICLE

 

Factors that influence the usability of a participatory IVR crowdsourcing system in a smart city

 

 

Liezel Cilliers^I; Stephen Flowerday^II

^IDepartment of Information Systems, University of Fort Hare, South Africa.lcilliers@ufh.ac.za
^IIDepartment of Information Systems, University of Fort Hare, South Africa.sflowerday@ufh.ac.za

 

 

---

ABSTRACT

'Smart Cities' is a new and inventive approach that allows city management to use current infrastructure and resources more effectively. Participatory crowdsourcing is an effective method to collect data from the citizens, as it does not require costly new infrastructure and can be used by all citizens, regardless of their literacy level. To date, very few studies have investigated the usability of these participatory crowdsourcing systems in a developing country context. The focus of this paper is then to provide a model for the usability of the IVR system to collect information from citizens to improve public safety in the city. The study makes use of a quantitative survey method. A questionnaire was completed by 361 participants of a public safety project hosted in East London, South Africa. The data analysis was completed making use of factor analysis. The results indicated that efficiency and perceived satisfaction with the system were important elements that determined the usability of the system. The recommendation of the study is that the city management must take these two elements into account when designed or developing a participatory crowdsourcing system.

Keywords: smart city, participatory crowdsourcing, public safety, usability, effectiveness, efficiency, user experience

---

 

 

1 INTRODUCTION

In recent times, the human-computer interaction (HCI) field, and specifically user experience, has become an important research field as the interaction between citizens and technology increases (Park, Han, Kim, Oh, & Moon, 2013). User experience can be defined as

all of the factors that influence the relationship between the end user and the organisation, especially when a product mediates that relationship. (Kuniavsky, 2007, p. 4)

One of the most important elements of user experience is the usability of the product (Park et al., 2013). Bevan, Carter, and Harker explain the difference between user experience and usability as follows:

Usability deals with goals shared within a user group while user experience focuses on individual goals including how the user encounters changes with repeated use. (2015, p. 2)

According to this definition, this study investigates the usability of a participatory mobile crowd-sourcing system among the citizens of East London, the user group. According to Brooke (2013) and Coursaris and Kim (2011), the ISO/DIS 9241-11:2015 standard (International Organisation for Standardisation, 2015) is one of the most popular and widely used definitions of usability. This standard defines usability as the combination and convergence of efficiency, effectiveness and satisfaction when a user makes use of a product to obtain a specific goal within a specified context of use.

 

2 BACKGROUND

It is projected that 80% of the population in the world will be living in cities by the year 2050 (McConnachie, 2012). This trend toward urbanisation means that there is an increased demand on available resources with which to provide basic services within cities (Washburn et al., 2009). Many cities have started to incorporate information and communication technologies (ICT) to improve their existing resource management and thus become 'smarter' (Harrison & Donnelly, 2011). While a generally accepted definition of a smart city seems to be elusive, one of the most commonly cited definitions in literature is "a city that uses ICT to be more interactive, efficient and making citizens more aware of what is happening in the city" (Azkuna, 2012, p. 2). Smart cities rely on large amounts of data used for predictive analytics to determine how resources can be used more efficiently and effectively.

Data can be collected through implicit methods (involuntary crowdsourcing) where sensors are used to collect data, or explicit methods (participatory crowdsourcing) where citizens (the user group) report information to the city making use of their own devices (such as mobile phones) in real time (Doan, Franklin, Kossmann, & Kraska, 2011). When the information that citizens report is used to improve the circumstances they live in, for example public safety, it will contribute to their quality of life. Making use of participatory crowdsourcing, citizens provide information using their mobile devices to call the city in order to ensure continuous data harvesting can take place (Cardone et al., 2013). There are several advantages when mobile devices are used to collect information from users. Existing mobile networks can be used which means that no expensive infrastructure is needed. This contributes to the cost effectiveness of using citizens as sensors. In addition, users can report events, such as emergencies or accidents, in a large geographical area (Christin, Reinhardt, Kanhere, & Hollick, 2011). This approach is particularly useful to report public safety issues to the city.

A further consideration of usability is the context in which the product is being used. Hussain, Mkpojiogua, and Hussain define context as "... any information that can be used to characterize the situation of an entity" (2015, p. 1). In this regard the entity can be any person, place or object relevant to the interaction between a user and the application in question. In this study, mobile phones were chosen to report public safety issues making use of an interactive voice response (IVR) system. As more citizens in Africa has access to mobile phones, this has become an important avenue for citizens to access services and content (Botha, Calteaux, Herselman, Grover, & Barnard, 2012). In addition, an IVR system is a suitable technology that can be used with the existing telephone infrastructure in developing countries (Kumar, Agarwal, & Manwani, 2010). IVR systems can be used to access, disseminate and collect information making use of mobile phones (Patel, Chittamuru, Jain, Dave, & Parikh, 2010).

There are several benefits that the system provides to developing countries. Citizens that are not familiar with the technology can complete the reports at their own pace, while literature has shown that even illiterate citizens can make user of the IVR system as they intuitively make use of a mobile phone. Furthermore, IVR systems do not require a smart phone as they can be accessed with a basic mobile handset. This is also a cost effective option as the service can be made available through a toll free number, which includes the unemployed or poor citizens of a city (Sherwani et al., 2007). Even though IVR systems have been reported as being time consuming and frustrating, it has become a widely accepted standard used in commercial organisations due to the high percentage of citizens that own mobile phones.

While the usability of the technology is acknowledged as one of the factors that will determine the adoption and use of IVR systems, there is limited empirical research available on this topic. However, with the greater emphasis on cities to make use of ICT innovations in order to manage resources more efficiently, it has become very important to provide guidelines that will enable the successful implementation of such systems (Botha et al., 2012). The focus of this paper is then to provide a model for the usability of the IVR system to collect information from citizens to improve public safety in the city.

 

3 THE PUBLIC SAFETY SMART CITY PROJECT

The University of Fort Hare, together with IBM, implemented a crowdsourcing system which the citizens of East London could use to report public safety concerns in their immediate environment. An IVR system was deemed the most suitable due to the benefits listed in the previous section. The project ran from 2012 to 2015 and the citizens were asked to participate in the study making use of a marketing campaign. The campaign made use of newspaper advertisements, social media posts and flyers that were delivered to the homes of the citizens. Figure 1 provides a graphical representation of the project.

3.1 The technology

Brooke et al. (1996) states that usability does not exist in isolation but can only exist with reference to a particular context. This implies that the characteristics that underline usability must be specified before any evaluation can be done (Speicher, 2015). An IVR system was deemed appropriate for this study due to the high mobile penetration rate (91%) of mobile phones in South Africa. The IVR system is also not dependent on the mobile phone device or operator (Kumar et al., 2010).

As mentioned in the previous section, illiterate and low income citizens are not excluded from participating in the project as the IVR content can be translated into the local languages. Kumar et al. (2010) previously reported that illiterate users are intuitively able to make use of a telephone keypad. Furthermore, a speech-based IVR system that can be accessed through a mobile device is familiar and easy to use as opposed to interacting with a computer or mobile interface (Botha et al., 2012). These types of services also have the further advantage that they can be used ubiquitously. The IVR system was thus deemed to be appropriate in a developing country context as it would be inclusive of all citizens including the illiterate and poor whom will benefit the most from the improved quality of life if public safety is improved.

3.2 The nature of the speech user interface

One of the most used dialogue style for speech-based telephone systems were identified in the literature as the IVR system. The IVR system initiates the conversation with the citizen and directs the dialogue making use of various response options. These response options allow the citizen to participate in the system with the greatest accuracy of speech recognition (Karat, Lai, Stewart, & Yankelovich, 2012). The structure and interaction flow of the IVR system is depicted in Figure 2.

The flow comprises of seven key steps:

1. Firstly, the caller is prompted to record their information after an audible beep (DD1).

2. Once the citizen has recorded the information, the system accepts the speech input.

3. The system alerts the user that their recording will be played back to them (EE1).

4. The system physically plays the recording back to the caller.

5. The citizen is then notified that they can accept the recording or have the option to re-record the message (FF1).

6. Depending on the input from the previous phase, the system will either proceed to step 7, return to step 1, or return to step 5 (in the case of invalid input).

7. The system plays a message that thanks the caller for their report (GG1).

The system and interaction flows were tested extensively during a pilot phase. The results from the pilot study were used to fine-tune the IVR system to ensure robustness in the event of incorrect or invalid user input.

 

4 THEORETICAL BACKGROUND

Usability has been widely investigated in the past. However, most studies agree that usability can only be determined if the user interacts with the product or service (Speicher, 2015). While there are different definitions for usability, the common thread among all of these is the user context and their satisfaction with the information system. Most authors will agree that the concept of usability must include these two subjective attributes and social aspects which extend beyond the simple task-related activities that citizens will use the IVR system for (Botha et al., 2012).

There are three factors that will affect the user experience of the citizen with the crowdsourcing system. The first is the user, or citizen, that reports information about their city through the IVR system. The second is the actual system that is being used - in this study the IVR system - and the third refers to the context or setting in which the citizen wants to evaluate the usability of the product (Hassenzahl & Tractinsky, 2006). Additionally, there are two types of goals that can be identified when system usability is evaluated (Botha et al., 2012). The first is do-goals which are pragmatic dimensions that refer to the product's perceived ability to support the achievement of tasks. The second goal is called be-goals and refers to the hedonic dimension. This goal is based on the satisfaction the user feels when making use of the system. These two goals are not interrelated as the do-goals are not always a prerequisite of be-goals. For example, a citizen can be satisfied with the IVR system even if they failed to accomplish the task to report to the system and vice versa (Speicher, 2015). The next section will discuss the ISO/DIS 9241-11:2015 standard that was used as the standard for usability in this study.

 

5 ISO/DIS 9241-11:2015 STANDARD

The ISO/DIS 9241-11 standard is the internationally accepted standard for understanding and applying usability. The standard is also used in and referenced in research and industry. The ISO/DIS 9241-11:2015 standard states that usability is an outcome of interaction rather than a specific property of a product and can therefore be described in terms of performance (effectiveness and efficiency) and satisfaction in a particular context of use. This means that the user and business requirements are directly related as effectiveness will measure the IVR system's success in achieving goals, efficiency will test if the citizen feels that their time was wasted when reporting to the IVR system, while satisfaction will determine the citizen's willingness to make use of the system in future (Bevan et al., 2015).

Effectiveness relates to whether citizens can complete their tasks and achieve their goals by using the IVR system (Brooke, 2013). Previously, effectiveness was only defined in terms of the accuracy and completeness of the system, but more recently the appropriateness of the system to achieve the citizen's goal has been added as an additional consideration (Bevan et al., 2015).

Efficiency refers to the extent to which citizens expend their resources in accomplishing their goals of using the IVR system (Bevan et al., 2015). These resources can include time, human effort, costs and material resources, which becomes important consideration if low income users are expected to participate in the study as not to exclude them (Brooke, 2013). Efficiency also refers to the degree to which the IVR system enables citizens to complete a task successfully without wasting time or energy (Park et al., 2013).

Satisfaction is the degree of comfort and delight that users experience while achieving their goals or as they use the IVR system (Brooke, 2013). Bevan et al. (2015) define satisfaction as

the extent to which attitudes related to the use of a system, product or service and the emotional and physiological effects arising from use are positive or negative (2015, p. 6)

The next section will discuss the hypotheses formulated for this study.

 

6 HYPOTHESES

Based on the aforementioned literature pertaining to the usability of IVR systems in a smart city, public safety crowdsourcing project, the following three hypotheses can be formulated:

H1. Efficiency is a predictor of the intention to make use of an IVR system to report public safety matters.

H2. Effectiveness is a predictor of the intention to make use of an IVR system to report public safety matters.

H3. Satisfaction of the system is a predictor of the intention to make use of an IVR system to report public safety matters.

Based on the preceding literature review and associated hypotheses, the following model was formulated in order to determine what factors will influence citizens' intention to make use of the IVR system to report public safety matters to the city. Selected possible relationships are described graphically in Figure 3.

 

7 METHODOLOGY

The project website registered 485 citizens. These citizens were sent a questionnaire to complete at the end of the project regarding their experience with the IVR system. The questionnaire was developed through previously published material in the area of usability. For the purpose of this study, the ISO/DIS 9241-11:2015 standard was used which makes use of three variables: effectiveness, efficiency and perceived user satisfaction. Ethical approval for the study was obtained from the University of Fort Hare's Research Ethics Committee.

7.1 Result instrument

The ISO/DIS 9241-11:2015 was used as a theoretical framework for this study. The data collection instrument used was based on the standard and adopted for the study's purposes. The instrument used to collect data consisted of two sections.

Section A (3 questions) solicited demographic information from respondents and section B (8 questions) investigated the different usability characteristics important for an IVR system. The questionnaire comprised tried and tested scales, which were previously published and included three variables (efficiency, effectiveness and perceived satisfaction) to determine the usability of the IVR system.

All of these variables were measured on a five-point Likert scale (1= Strongly Disagree to 5 = Strongly Agree) and displayed sufficient reliability and validity scores in previous studies, as per the recommended threshold for reliability and validity (Nunnally & Bernstein, 1978).

A pilot study was conducted prior to the main study with a sample of citizens fitting the same profile and status as with the population that was to be used in the research study. Based on the pilot study's feedback, some questions were modified to make them more relevant to the aims of the research.

7.2 Research participants and procedure

Data was collected from the citizens that were registered on the project's website (n = 485). Citizens were sent an e-mail with a link to the questionnaire. A total of 361 questionnaires were completed and returned, thus the response rate was 81.2%. Participation in the research was voluntary and confidentiality was maintained. The demographic information is displayed in Table 1.

7.3 Statistical analyses

The statistical analyses were carried out with the IBM SPSS V24. Cronbach alpha coefficients (a) were used to assess the internal consistency of the measuring instrument and factors identified with exploratory factor analysis (Nunnally & Bernstein, 1978). A Cronbach's alpha values of 0.70 and above denote a good level of internal reliability while values between 0.50 and 0.69 denote an acceptable level of reliability, and values below 0.50 denote poor and unacceptable levels of reliability (Pallant, 2013). This analysis was also used to provide measures of reliability and mean inter-item correlations to test for internal construct consistencies. Table 2 provides the results of the empirical reliability testing for the variables used in this research: user satisfaction, or, system quality, interface quality and the usefulness of the system.

As can be noted from Table 2, the empirical reliability for the majority of the variables proved to be acceptable except for the perceived satisfaction of the system which reported a reliability reading just below 0.50. Attempts to improve the reliability of this scale did not yield visibly improved results. However, due to the fact that 'the perceived satisfaction the system' occupies a pivotal role to determine the usability of any system, a decision was made to retain the scale in its current form and continue with factor analysis to test for validity.

Summary statistics provided counts, percentages and distribution of measures at factor level. Exploratory and confirmatory factor analyses were used to test for uni-dimensionality and content validity of constructs. Linearity and directionality of relationships were established firstly via correlation and secondly by regression analysis.

Based on the results of the Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy (0.815) and Bartlett's test of sphericity (a < 0.05), the data was deemed suitable for factor analysis. Thus, an exploratory factor analysis and confirmatory factor analysis were conducted for the purpose of determining validity of the constructs and relevant scales. Table 3 shows the rotated component matrix with Varimax adopted as a rotation method. Empirical reliability was then rechecked in order to ensure that satisfactory Cronbach alpha was achieved. Overall reliability and validity for these variables can be considered to be satisfactory as per the recommended Cronbach alpha coefficient threshold of 0.76 (Nunnally & Bernstein, 1978).

 

 

8 RESULTS

Suitably high correlations (Table 4) between the independent and dependent variables were achieved. Table 4 also shows that all the correlations were deemed statistically significant. It was ensured that an excessive degree of multi-collinearity between the independent and dependent variables did not exist. Tolerance values were significantly large enough (.851, .920 and .816) to demonstrate lack of multi-collinearity and VIF values (1.175, 1.087 and 1.225) were also deemed acceptable (Pallant, 2013), thus demonstrating that the assumption of multi-collinearity would not be violated in the resulting model. The other assumptions pertaining to outliers, normality, linearity, homoscedasticity and independence of residuals were considered by examining the normal probability plot of the regression standardised residual and the scatterplot. These assumptions appeared to also have been met.

In terms of the analysis of variance results, the R Square value indicates that the proposed model explains .353 (35%) of the variance in behavioural intention to make use of the IVR system to report public safety matters. This result can be deemed to be acceptable. The proposed model can be considered to be statistically significant at 0.000, thus indicating that multiple R in the population does not equal 0.

Finally, all three of the variables in the model contributed to the prediction of the intention to use an IVR to report public safety matters. The Beta value for perceived satisfaction of the system was .353 making it the largest contributor to the model, thus it is the largest predictor of to use an IVR to report public safety matters.

Efficiency of the IVR system can be considered to be the second highest predictor of intention to use with a Beta coefficient of .349.

Effectiveness, with only one factor, was as expected the smallest predictor of intention to make use of the IVR system with a Beta value of .040. With respect to significance of these relationships, only the first two variables made a statistically significant contribution to the equation (a < 0.05).

With respect to the hypotheses, Hypothesis 1 ('Efficiency is a predictor of the intention to make use of an IVR system to report public safety matters') and Hypothesis 3 ('Satisfaction of the system is a predictor of the intention to make use of an IVR system to report public safety matters') were accepted. Hypothesis 2 ('Effectiveness is a predictor of the intention to make use of an IVR system to report public safety matters') was rejected.

 

9 DISCUSSION

Smart cities make use of crowdsourcing to collect data from citizens in order to improve their resource management. In order to collect information from citizens, an IVR system has been recommended as a useful technology as it can be used to overcome many of the problems in developing countries. IVR systems allow a user to access or provide information via the voice channel of their mobile phone (Botha et al., 2012). Recent advances in technology have made speech recognition technology with voice interfaces and indexing mechanisms an efficient method to collect data. For this reason, the project decided to incorporate IVR systems to collect data from citizens.

The usability of the IVR systems was measured making use of the ISO/DIS 9241-11:2015 standard which is widely used in industry. The standard proposes three factors that will determine the usability of the IVR system: efficiency, effectiveness, and satisfaction with the IVR system. The first factor, efficiency, refers to the extent to which citizens expend their resources, including time, effort and costs, in accomplishing their goals of using the IVR system (Brooke, 2013). If the citizen is familiar with the crowdsourcing system and is able to navigate it quickly in order to report their public safety information, the system will be considered an efficient system. This means that the system is efficient as the expected result, alerting the emergency services, will be achieved. However, if the citizen is not familiar with the IVR system, navigating the system by making use of the voice prompts may take a long time. On the other hand, the system is not efficient as the citizen had to waste time navigating an unfamiliar system. Efficiency was found to be significant in order to predict the citizen's intentions to make use of the IVR system to report public safety information.

Interface quality was used in order determine the efficiency of the IVR system. Contingency of the IVR system refers to the extent to which one person's queries, responses and comments are dependent on prior queries, responses and comments (Joinson, Reips, Buchanan, & Schofield, 2010). In the IVR system, contingency is important as the response of the user will determine the next voice prompt to be cued. It is essential that the system understand the root of a problem - whether the user has provided a wrong answer, or the user is not speaking clearly - and respond accordingly. The problem regarding the responses not being understood by the IVR system was solved by testing the prototype extensively before it went operational. More than 20 iterations were necessary in order to make sure that all user accents would be recognised by the speech software. Alternatively, callers were given the option to input their responses by making use of the touchpad if the system did not recognise their vocal responses.

The second factor, effectiveness of the IVR system, refers to if citizens can complete their tasks and achieve their goals by using the IVR system. This factor did not impact on the citizens' behavioural intention to make use of the IVR system. Citizens do make use of IVR systems to provide or receive information for other tasks, e.g. mobile banking or mobile service provider's administrative tasks, and are therefore familiar with the technology. Citizens know what the purpose of the IVR system is and therefore report the right information to the system.

The third factor, perceived satisfaction, refers to if the citizen feels that the crowdsourcing system was useful and would yield positive results. This in turn will improve the citizen's confidence in the system. This factor was found to be significant to predict citizen's intention to make use of the IVR system, with most of the citizen's reporting that they were satisfied with the system and thought that the system would assist them to report public safety matters. This is an important finding as this paves the way for developing countries to make use of mobile technology and IVR systems in order to overcome the expensive infrastructure that is associated with data collection from sensors in a smart city. Citizens can therefore be used as sensors in the city to report public safety issues which results in the city being able to improve their resource management.

 

10 CONCLUSION

Smart cities are making use of continuous data collection in order to manage city resources more effectively. Predictive analytics are used to determine where existing resources can be used for the maximum impact. This study made use of an IVR system that citizens could use to report public safety issues that could be used by the city to plan for such events.

This paper found that from a usability perspective, the IVR system can be considered an efficient crowdsourcing communication channel for citizens to report public safety matters. The citizens of East London indicated that they are satisfied with the system which means that they will make use of it in future to report their public safety issues. Therefore, when designing a public safety crowdsourcing system, the usability of the system must be considered in order to improve citizen participation.

 

ACKNOWLEDGEMENTS

This work is based on the research supported in part by IBM, the National Research Foundation (NRF) of South Africa, and the citizens of East London. The authors acknowledge that the opinions, findings, and conclusions or recommendations expressed in this paper are those of the authors and that IBM and the NRF accept no liability whatsoever in this regard.

 

References

Azkuna, I. (2012). Smart cities study: International study on the situation of ICT, innovation and knowledge in cities. The Committee of Digital and Knowledge-based Cities of UCLG, Bilbao.

Bevan, N., Carter, J., & Harker, S. (2015). ISO 9241-11 revised: What have we learnt about usability since 1998? In International Conference on Human-Computer Interaction (pp. 143-151). Springer.

Botha, A., Calteaux, K., Herselman, M., Grover, A. S., & Barnard, E. (2012). Mobile user experience for voice services: A theoretical framework. Proceedings ofM4D 2012 28-29 February 2012 New Delhi, India, 28(29), 335.         [ Links ]

Breetzke, T. & Flowerday, S. V (2016). The usability of IVRs for smart city crowdsourcing in developing cities. The Electronic Journal of Information Systems in Developing Countries, 73.         [ Links ]

Brooke, J. et al. (1996). SUS-A quick and dirty usability scale. Usability evaluation in industry, 189(194), 4-7.         [ Links ]

Brooke, J. (2013). SUS: A retrospective. Journal of usability studies, 8(2), 29-40.         [ Links ]

Cardone, G., Foschini, L., Bellavista, P, Corradi, A., Borcea, C., Talasila, M., & Curtmola, R. (2013). Fostering participaction in smart cities: A geo-social crowdsensing platform. IEEE Communications Magazine, 51(6), 112-119. https://doi.org/10.1109/MCOM.2013.6525603        [ Links ]

Christin, D., Reinhardt, A., Kanhere, S. S., & Hollick, M. (2011). A survey on privacy in mobile participatory sensing applications. Journal of systems and software, 84(11), 1928-1946. https://doi.org/10.1016/j.jss.2011.06.073        [ Links ]

Coursaris, C. K. & Kim, D. J. (2011). A meta-analytical review of empirical mobile usability studies. Journal of usability studies, 6(3), 117-171.         [ Links ]

Doan, A., Franklin, M. J., Kossmann, D., & Kraska, T. (2011). Crowdsourcing applications and platforms: A data management perspective. Proceedings of the VLDB Endowment, 4(12), 15081509.         [ Links ]

Harrison, C. & Donnelly, I. A. (2011). A theory of smart cities. In Proceedings of the 55th Annual Meeting of the ISSS-2011, Hull, UK (Vol. 55, 1).

Hassenzahl, M. & Tractinsky, N. (2006). User experience-A research agenda. Behaviour & information technology, 25(2), 91-97. https://doi.org/10.1080/01449290500330331        [ Links ]

Hussain, A., Mkpojiogu, E., & Hussain, Z. (2015). Usability evaluation of a web-based health awareness portal on smartphone devices using ISO 9241-11 model. Jurnal Teknologi, 77(4), 15.         [ Links ]

International Organisation for Standardisation. (2015). ISO/DIS 9241-11.2 Ergonomics of human-system interaction - Part 11: Usability: Definitions and concepts. Retrieved from http://www.iso.org/iso/catalogue_detail.htm?csnumber=63500

Joinson, A. N., Reips, U.-D., Buchanan, T., & Schofield, C. B. P (2010). Privacy, trust, and self-disclosure online. Human-Computer Interaction, 25(1), 1-24. https://doi.org/10.1080/07370020903586662        [ Links ]

Karat, C., Lai, J., Stewart, O., & Yankelovich, N. (2012). Speech and language interfaces, applications, and technologies. In The human-computer interaction handbook: Fundamentals, evolving technologies, and emerging applications. (3rd, pp. 367-386). Lawrence Erlbaum Associates Inc, New York, USA.

Kumar, A., Agarwal, S. K., & Manwani, P (2010). The spoken web application framework: User generated content and service creation through low-end mobiles. In Proceedings of the 2010 International Cross Disciplinary Conference on Web Accessibility (W4A) (p. 2). ACM. https://doi.org/10.1145/1805986.1805990

Kuniavsky, M. (2007). User experience and HCI. In The Human-Computer Interaction handbook: Fundamentals, evolving technologies, and emerging applications. (2nd). Lawrence Erlbaum Associates Inc, New York, USA. https://doi.org/10.1201/9781410615862.ch46

McConnachie, K. (2012, November). Smart cities: Data analytics transform urban living. Retrieved from www.itweb.co.za/index.php?option=com_content&view=article&id=59869

Nunnally, J. & Bernstein, I. (1978). Psychometric theory.

Pallant, J. (2013). SPSS survival manual. McGraw-Hill Education (UK).

Park, J., Han, S. H., Kim, H. K., Oh, S., & Moon, H. (2013). Modeling user experience: A case study on a mobile device. International Journal of Industrial Ergonomics, 43(2), 187-196. https://doi.org/10.1016/j.ergon.2013.01.005        [ Links ]

Patel, N., Chittamuru, D., Jain, A., Dave, P, & Parikh, T. S. (2010). Avaajotalo: A field study of an interactive voice forum for small farmers in rural India. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 733-742). ACM. https://doi.org/10.1145/1753326.1753434

Sherwani, J., Ali, N., Mirza, S., Fatma, A., Memon, Y., Karim, M.,... Rosenfeld, R. (2007). Healthline: Speech-based access to health information by low-literate users. In Information and Communication Technologies and Development, 2007. ICTD 2007. International Conference on (pp. 1-9). IEEE. https://doi.org/10.1109/ICTD.2007.4937399

Speicher, M. (2015). What is usability? A characterization based on ISO 9241-11 and ISO/IEC 25010. Retrieved from http://arxiv.org/pdf/1502.06792v1.pdf

Washburn, D., Sindhu, U., Balaouras, S., Dines, R. A., Hayes, N., & Nelson, L. E. (2009). Helping CIOs understand "smart city" initiatives. Growth, 17(2), 1-17.         [ Links ]

 

 

Received: 2 Oct 2016
Accepted: 22 Jun 2017
Available online: 8 Dec 2017