RESEARCH ARTICLE

 

Educator perceptions of virtual learning system quality characteristics

 

 

Indira Padayachee

University of KwaZulu-Natal, Durban, South Africa. padayacheei@ukzn.ac.za

 

 

---

ABSTRACT

In higher education institutions, digital learning environments, referred to as virtual learning systems in this article, have been adopted and are becoming increasingly popular among academics. A virtual learning system (VLS) has a suite of tools with associated functions and non-functional system characteristics. Higher education institutions (HEIs) implement a VLS with the intent to assimilate e-Learning with face-to-face instruction and thereby derive associated benefits from its usage. Currently there is limited information on educators' perceptions on the usefulness of VLS tool functionality, and the importance assigned to non-functional characteristics. This article adapted the generic framework of the ISO 9126 external software quality model to ascertain the perceptions of educators with regards to VLS functionality and non-functional quality characteristics. A case study research strategy was adopted, using two South African higher education institutions. The main contribution of this article is a proposed model representing VLS quality in use characteristics, measured by the constructs of perceived usefulness and perceived importance. In addition to the theoretical contribution, this article makes a practical contribution by providing educators' recommendations on the improvement of VLS quality characteristics. This article has potentially useful implications for the use of a VLS by educators and future VLS design iterations.

Keywords: perceived usefulness, perceived importance, course management systems, educators, learning management systems, ISO 9126 software quality, virtual learning system functions, non-functional characteristics

---

 

 

1 INTRODUCTION

New educational technology services and features are driven by advances in information technology and a growing market. Virtual learning systems have been formally adopted by higher education institutions to stay abreast of the latest educational technologies, to be competitive in the higher education domain and to afford their stakeholders new innovative ways of teaching and learning. A VLS refers to a class of software that is known by a number of names such as course management systems (CMSs), learning management systems (LMSs), virtual learning environments (VLEs), online learning platforms, and e-Learning systems. Examples of a VLS in use are Blackboard, ATutor, and Moodle (WCET, n.d.).

Studies have indicated that despite increasing adoption rates, the full potential of a VLS for online teaching and learning has not been achieved, with lecturers using a limited set of functions (Hueh & Hsu, 2008; Mlitwa, 2006; Dahlstrom, Walker, & Dziuban, 2013). VLS usage amongst academics in higher education is influenced in some measure by a number of factors, which includes system functionality (Egert, Jacobs, & Barnes, 2009), and non-functional characteristics (McGill & Klobas, 2009; Nanayakkara, 2007). The term non-functional refers to desired characteristics that a system must possess that may not be related to a specific function such as security or usability. The application of the technology acceptance model (TAM) in previous studies has shown that information system usage is preceded by behavioural intention to use, and the determinants of behavioural intention are perceived usefulness and perceived ease of use (Davis, 1989). While previous studies confirm the influence of perceived usefulness on adoption and usage of information systems, this construct has been applied to systems or to information technology applications as a whole and rarely to individual function/feature sets (Jasperson, Carter, & Zmud, 2005). Hence, there is limited research on educators' perceptions on the usefulness and importance of VLS functionality and non-functional characteristics, which has implications for system usage behaviour. This article attempts to address this gap by focussing on two key research questions:

1. What are the perceptions of educators with regard to the usefulness of the VLS functionality quality characteristic?

2. What the perceptions of educators with regard to the importance of VLS non-functional quality characteristics?

The main argument of the paper is that ISO9126 quality model (Behkamal, Kahani, & Akbari, 2009) can be applied to assess the quality in use of a VLS by studying the perception of end users on the usefulness and importance of system quality characteristics. It was towards this goal that the first four characteristics of the ISO 9126 external software quality model were selected, namely functionality, reliability, usability and efficiency, and corresponding sub-characteristics were defined to fit the context of a VLS. These four external software quality characteristics were selected as they represent the main software quality in use characteristics, which are easily observable by educators whilst interacting with the software (Chua & Dyson, 2004). The functionality characteristic of ISO 9126 external software quality model was applied to the VLS tool functionality. In the ISO 9126 model, security was considered as a sub-characteristic under the functionality characteristic in order to measure the security of individual functions within a product. However, for the purpose of this study, security is classified under the category of non-functional characteristics together with reliability, usability and efficiency in order to assess the software product characteristics that are not related to a specific function (Behkamal et al., 2009; ElementK, 2003).

1.1 Literature review

According to Bevan (1999), there is an increasing need and expectation for quality software products that demonstrate technical excellence, usability and compatibility with work practices. This article addresses this gap by reviewing and measuring VLS quality in use characteristics in the context of higher education.

The sections of the literature review provide an overview of VLS functionality and non-functional software quality characteristics, educators' perceptions of VLS in use and the conceptual model adopted for the study.

1.1.1 VLS functionality

This section defines the functionality characteristic of the ISO/ IEC 9126-1 quality model in terms of VLS tool functionality. Table 1 identifies and defines VLS communication, student productivity and involvement, course administration and management, assessment, student tracking and content tool sets. The classification of the tool sets presented in Table 1 was largely adapted from an EduTools matrix outlining VLS features and criteria (WCET, n.d.). The EduTools website was selected as it provides an independent review of VLS products, as well as a matrix of the products' features and criteria, which are used by many universities and colleges for selection purposes. A literature review of the functions of the VLS tool sets are summarised in Tables 4 to 9 in the Appendix. The VLS tool functions were used as a framework for understanding educators' perceptions on the usefulness of VLS functionality for online teaching.

1.1.2 VLS non-functional quality characteristics

This section describes the non-functional characteristics of the ISO 9126 quality model. According to Trice and Treacy (1988), system characteristics affect the efficiency and effectiveness of the user's interaction with a computer system, and as such should also be considered in conjunction with system functions. Mueller and Strohmeier (2010) undertook a literature review and an initial expert study to arrive at the following system and information-related design characteristics of VLEs: reliable, secure, interactive, appealing, transparent, structured, standard-supportive, accessible, and platform-independent. These characteristics are aligned to the non-functional characteristics of usability (interactive, appealing, transparent, and structured), reliability and security listed in Table 2. Table 2 provides a representation of the ISO 9126 quality characteristics of usability, security, reliability, and efficiency. The sub-characteristics associated with each of the non-functional quality characteristics described in Table 2, provided the baseline for measuring educators' perceptions.

1.1.3 Educators' perceptions of VLS in use

This section presents a summary of studies conducted on the educators' perceptions of a VLS in use. Chua and Dyson (2004) evaluated educators' perceptions of Blackboard version 6.1 tools using the ISO 9126 model characteristics of functionality, reliability, usability and efficiency with the objective of detecting design flaws. The Chua and Dyson (2004) study tested for specific sub-characteristics of e-Learning systems as originally defined by the ISO9126 model. The current study, however, focussed on integrating the concepts of perceived usefulness and perceived importance that are linked to system usage with ISO 9126 quality in use characteristics.

A study was conducted by Mlitwa and Van Belle (2011) into mediators of lecturers' perceptions on LMS at universities in the Western Cape in South Africa. The key findings reported were six mediators that influence C/LMS usage, namely perceived usefulness, perceived ease of use, accessibility, functional expectations, support and use mediation. The key constructs of perceived usefulness and perceived ease of use from the Mlitwa and Van Belle (2011) study are related to the current study. The construct of perceived usefulness is applied to VLS functionality and the construct of perceived ease of use is included within the broader category of usability, which is measured for perceived importance.

An empirical study was conducted by Borwarnginn and Tate (2014) into the perceptions of lecturers and students in the Faculty of Information and Communication Technology at the Mahidol University in Thailand focussing on the uses, issues and satisfaction levels associated with Moodle. The findings highlighted the need for more customisation of class pages, the addition of more interactive features, and support for collaborative tasks. These findings are related to the usability characteristic of VLS and required VLS functionality, which are included in the scope of the current study.

Gamage, Tretiakov, and Crump (2011) conducted a study on teacher perceptions of learner affordances in multi-user learning environments (MUVE), where interaction takes place in a shared virtual 3D space. The study findings suggest perceptions of MUVE affordances for learning by educators are generally positive. The current study did not focus on interaction in a shared 3D space, but rather general interaction functions offered by a VLS.

A study conducted by Goh, Hong, and Gunawan (2014) on the perception of lecturers using LMS revealed that Moodle was treated as a content repository and lecturers did not fully utilise the interactive features in Moodle. This continues to be a major problem as the the full potential of a VLS to support teaching and learning has not yet been fully realized, despite the benefits afforded. Furthermore issues of usability, interaction and communication were negatively perceived by lecturers. Perceptions of VLS functions and the usability attribute was covered in the current study.

Chang et al. (2015) proposed a new framework conceptualising technology-supported learning environments (TSLEs), comprising six dimensions: technical, content, cognitive, metacognitive, social, and affective. The current study focused on educators' perceptions of the technical dimension of VLS functionality and non-functional software characteristics. The findings of a study conducted by Asiri, Mahmud, Abu Bakar, Mohd Ayub, and Fauzi (2012), revealed that beliefs toward e-Learning can be examined from the viewpoint of the importance of web technology to instructors. A study conducted by Pritchett, Wohleb, and Pritchett (2013) revealed that certified educators cited virtual learning networks, video sharing and online event scheduling to be the most important educational Web 2.0 applications, and social bookmarks, social networks and music to be the least important.

1.1.4 Conceptual model

Quality in use is the user's view of the quality of software, and is measured in terms of using the software, rather than properties of the software itself. Quality in use is the combined effect of the software quality characteristics for the user (Bevan, 1999).The ISO 9126 ISO/IEC 9126 is an international standard, which categorises quality as six characteristics, namely functionality, reliability, usability, efficiency, maintainability and portability. The characteristics of maintainability and portability were excluded as they require the expertise of trained IT professionals. The functionality characteristic represents the functions that satisfy stated or implied needs in the system (Bevan, 1999). The reliability quality characteristic refers to the capability of the software to retain its performance level under specified conditions (Bevan, 1999). Efficiency refers to the capability of the software to provide required performance in relation to the amount of resources used, under specified conditions (Bevan, 1999). Usability refers to capability of the software to be understood, learnt, used and liked by users, under specified conditions (Bevan, 1999). Security is included as a non-functional characteristic, which refers to the ability to prevent unauthorised access, whether accidental or deliberate, to programs and data (Behkamal et al., 2009, p. 602). This article focuses on educators' perceptions on the quality characteristics of a class of online education technologies that Gonella and Panto (2008) reported as being widely adopted in universities.

The constructs used to measure VLS quality characteristics were perceived usefulness and perceived importance. According to Davis (1989), the construct perceived usefulness, which is adopted from the TAM model is a determinant of behavioural intention and usage behaviour. The construct perceived ease of use from the TAM model is encompassed in the usability quality characteristic. The construct perceived importance has been found to influence ethical judgement (attitude) in behavioural intention studies (Robin, Reidenbach, & Forrest, 1996), as well as measure information system success factors (Li, 1997). The system quality characteristics, factors, and usage are depicted in Figure 1 together with their relationships.

Users' perceptions of the usefulness of the functions of VLS was measured to gain an understanding of whether implied needs were met in the context of VLS usage in education, while users' perceptions of importance was measured to get an understanding of the importance attached to non-functional attributes of a system in the context of VLS usage as these characteristics have an influence on user satisfaction with a software product. These allied concepts provide insights into whether the functions are useful to the educator with regards to technology support for teaching, learning and administrative tasks and whether the system as a whole satisfies quality in use characteristics of reliability, security, usability and efficiency.

The literature shows there is limited research on educators' perceptions of VLS quality characteristics that are deployed in higher education. Hence the aim of this article is to understand educators' perceptions of VLS functionality and non-functional quality characteristics using the ISO 9126 external software quality model, and respectively measuring these characteristics using the constructs of perceived usefulness and perceived importance.

 

2 RESEARCH METHODOLOGY

This section discusses research design, data collection, population and sampling, and data analysis used for this research.

A mixed methods research design was used for the broader study combining qualitative and quantitative approaches. This paper reports primarily on the qualitative research approach used to conduct the study (Venkatesh, Brown, & Bala, 2013). The justification for a qualitative approach was to obtain rich and deep insights into the perceptions of educators with regards to VLS quality characteristics, and suggestions for potential improvements.

This research was conducted at two higher education institutions in South Africa, which are hereafter referred to as University A and University B. At the time the study was conducted, Blackboard version 6 was the VLS deployed at University A, and Moodle version 1.5.2 was adopted at University B.

In the first phase of the study, a literature review was conducted of VLS tool functionality, nonfunctional quality characteristics, studies on educators' perceptions of VLS in use and potential quality and usage models. The literature on VLS tool functionality and non-functional quality characteristics were used to frame the study and design the research instruments.

In the second phase of the research, a qualitative approach was adopted where data was collected through structured in-depth interviews. Interviews were used as the main method of data collection to confirm categories/themes and sub-themes relating to the two main research questions namely perceived usefulness of VLS tool functionality, and perceived importance of non-functional software characteristics. The interview questions concerned: VLS functions and services needed or deemed useful for online teaching; types of tool support found useful when performing online assessments and tracking student progress with a VLS; VLS facilities needed or regarded useful for student involvement and productivity in an online course; quality characteristics deemed important to be integrated into a VLS to support the online courses. A purposive sampling technique was used to select potential participants for the semi-structured in-depth interviews from the two universities. At the time of the study, the interviewees were using a VLS, and had a minimum of one year's experience using a VLS.

Thematic analysis, which is a commonly used method of analysis in qualitative research was used to reduce texts to codes that represented themes or concepts (Guest, MacQueen, & Namey, 2012). Tagging was used to analyse qualitative data with the aid of the NVivo tool. The process followed for thematic analysis was generating initial codes based on the categories from the two research questions, namely perceptions of VLS functionality and perceptions of non-functional characteristics as summarised in Tables 1 and 2. Additional themes and subthemes relating to individual toolsets and non-functional quality characteristics were identified and refined during the coding process.

In the third phase of the study, surveys were used as a follow-up technique to interviews with a broader sample of the target population and served to confirm the results of the qualitative study. A total of 107 completed questionnaires were returned from both universities. A five point Likert scale was used in the design of the survey questions ranging from strongly disagree (1), disagree (2), neither agree or disagree (3), agree (4) and strongly agree (5). Survey questions measured educators' perceptions of the usefulness of the VLS functionality for online teaching and the importance of VLS non-functional characteristics as depicted in Figure 2. The Cronbach's Alpha technique was used to measure the reliability or internal consistency of the VLS functionality and non-functional characteristic groups, which were all above 0.7, thereby permitting analysis. Descriptive statistics, namely the mean and standard deviation was computed for each of the groups. A quantitative analysis of the perceived usefulness of VLS functionality groups (namely communication, student tracking, course administration, course content, assessment, and student productivity and involvement) and the perceived importance of non-functional characteristic groups (namely usability, reliability, efficiency, security, flexibility and standards) revealed that the mean scores were all significantly above a neutral score of 3. Hence, there was significant agreement amongst the educators on the quality characteristics of Blackboard and Moodle formally adopted at the two universities.

 

3 FINDINGS AND ANALYSIS

This section covers the qualitative results of data collected from University A and University B pertaining to educator perceptions on quality characteristics of the VLS formally adopted for use at these institutions. The qualitative findings presented cover the demographic profile of interviewees, the perceived usefulness of Blackboard and Moodle functionality, as well as the perceived importance attached to their non-functional characteristics.

3.1 Demographic profile of interviewees

The number of interviewees who participated in this study per university is summarised in Table 3. All participants interviewed were educators who were teaching at both undergraduate and postgraduate levels of study from the two selected universities.

 

 

A total of ten educators were interviewed from various disciplines at University A, namely, three from fine arts, one from jewellery design, one from hotel and catering management, one from radiography, two from management studies and two from engineering. The interviewees comprised five females and five males. A total of sixteen educators were interviewed from several disciplines at University B, namely, one from tele-health, one from IT Ed, two from nursing, two from information systems and technology, one from pharmacology, one from education, one from internet studies, one from computer science, two from genetics, one from engineering, one from mathematics, one from law, and one from dietetics and human nutrition. Seven of the interviewees were males and nine were females.

All of the interviewees at University A had been using the Blackboard VLS in their teaching practice for over a year. In addition, educators employed at University A had completed the Pioneers Online training programme, which is a short certificated course. Certificated courses were run by the e-Learning unit within the Centre for Excellence in Learning and Teaching (CELT) at University A, which provided training in Blackboard together with instructional design and teaching philosophies. Some of the educators had also completed the intermediate and advanced training courses in Blackboard. University A had a dedicated technician to support Blackboard users.

The minimum length of usage of VLS Moodle by interviewees at University B in their teaching practice was a year. Some of the interviewees had attended introductory workshops on Moodle run by the Academic Computing Department, while others had learnt to use the VLS through self-experimentation. University B had a dedicated academic computing department providing adequate support for Moodle users.

All the educators interviewed at both universities reported that they were familiar with and comfortable using office applications such as Word, Excel and PowerPoint presentation with some reporting that they were competent with programming, and other specialised application software.

3.2 Perceived usefulness of Blackboard and Moodle functionality characteristics

The qualitative findings and analysis of educator perceptions on the usefulness of the following suite of VLS tools, namely communication, student productivity and student involvement, course administration and management, student tracking and content tool features is presented in this section. A summary of themes and subthemes were coded from all the interview transcripts, covering the perceived usefulness of VLS tool functions for online teaching.

3.2.1 Theme 1: Communication

The 'usefulness of communication tools' theme for the two universities covered a number of subthemes, which are described below. Educators at both institutions agreed on the 'usefulness of communication tools' theme, which confirmed the need for the corresponding communication tools and features described in Table 4 in the Appendix.

Online threaded discussion forum Posting in threaded forum discussions was the main tool reported to be used for learning, discussion and debate in the online environment. Some of the data extracts pertaining to the usefulness of this subtheme were:

Able to upload images to posts; automatic notification of new posts; keep a record of the conversation; Free fold in the conversation, and conversation central where you can upload and tag files

Educators reported mixed experiences with online discussion forums. Some educators at University B found that students were less inhibited and more frank in online discussions than they were in face-to-face discussions whilst others reported that students' uptake of online discussion forums was poor and generally one-way communication from lecturer to students where no mark was awarded for online participation. The latter finding on online discussion forums supports the view of Wyles (n.d.) who argues that the transmission model of information is common in a VLS because it was easier than the interactivity implicit in a conversation. Another viewpoint was that online discussion forums were very cumbersome and involved much reading and writing and was time-consuming. The findings of a study conducted by McGill and Klobas (2009) supports this finding that a VLS enhances communication and coordination, but creates more work. Educators reported that there were scheduled classes where face-to-face discussion took place, as both universities were residential, and therefore there was a lesser need for online discussion. Discussion forums were not used for the mathematics discipline as typing of mathematical equations was reported to be cumbersome.

Some of the additional features that educators recommended for Blackboard were the need for threaded discussion forums to be modelled around social media systems; tracking of student participation in threaded forums and chats, and notification mechanisms for discussion tools. Data extracts supporting the need for an improved design in Blackboard were:

This tool should resemble Web 2.0 type of functionality; It needs to be less structured and more spontaneous; Blackboard does not send a notification to students when a notice is posted to them; I would like an activity report on how many times they commented on a post

Data extracts supporting the need for an improved design in Moodle were:

Communicating in mathematics using forums is cumbersome; Need for separate channel of communication or forum for co-teachers on a course or module; should resemble a face book [sic] type of environment.

Blogging Responses on the usefulness of this tool varied as only a select few educators used this tool in their courses. The educators that used blogging found it to be useful for student participation in discussions, and contributing to the learning of a topic or subject area. Educators that had no prior experience of using the tool agreed that it was potentially useful. Some of the data extracts pertaining to the usefulness of this subtheme were:

Need indicators to see how many people have read your blog I quite like the idea of blogging; it should automatically track students blogs; blogging supports communication.

This research supports the finding of Halaseh (2014) who reported that comments in blogs help students' in their written forms of communication.

Internal email The built-in email facility was found to be very useful in communicating with students and maintaining a record of that communication. Some of the data extracts pertaining to the usefulness of this subtheme were:

Used to communicate new deadlines or new arrangements and there is a record of that; you have got 24/7 access so you can post and I see the post via my e-mail.

The finding on the usefulness of the course email tool supports the finding by Barron (2003) that email messages for each course can be stored in a separate account, which allows for easier tracking of all messages sent and received.

Online real-time chat This tool was less used at both institutions. Educators did not perceive this tool to be highly useful as both institutions offer face to face contact with the students. They believed that this tool was useful for supporting student communication, particularly in cases of part-time programme offerings. Data extracts pertaining to the usefulness of this subtheme were:

Students use it to chat among themselves; We conducted our session via the live chat and kept a record of our session.

Fidalgo (2012) maintained that the online real-time chat is a collaborative learning tool, which is supported in this study. The finding on the chat tool did not support the view of Kanthawongs and Kanthawongs (2013), who argued the online chat was one of the web components of a blended model. The argument presented by educators was that there was no real need for this tool as they had face to face contact with their students in scheduled classes.

Electronic whiteboard This tool was rarely used and reported to be less useful than the other communication tools. An educator at University A was cited as having attempted to use the tool but could not get it to work. This tool was used to a limited extent at University B and reported to be useful for drawings and annotations for distance learners. Another view expressed was that it would be more useful for lessons rather than lectures. Some of the data extracts pertaining to the usefulness of this subtheme were:

Presentation on whiteboards are used; we annotate on the presentation with extra drawings, extra explanations.

This finding of limited usefulness for this tool is not in keeping with the view of Sun, Cheng, and Finger (2009) that a key functional requirement of an eLearning system is an electronic whiteboard. A possible reason for this finding is that the majority of the educators were unaware of this tool or did not know how to operate this tool.

News forum or announcements The notice board or news forum was also reported to be very useful to make announcements and convey messages to students. It was also mentioned that student can receive immediate notification of announcements when they log on to the system. Some of the data extracts pertaining to the usefulness of this subtheme were:

News forums would be useful; a mobile application for Moodle where students can access get notices on their mobiles; see what readings are there.

The findings on the announcement tool is supported by Unal, Z. and Unal, A. (2011), who reported that the announcements (in Blackboard) or the news (in Moodle) is mostly used to help educators keep students updated on course information.

Wikis This tool was used by a select few educators to promote collaborative or group work in their courses. The findings indicated that wikis had not taken off. Possible reasons for this are that undergraduate levels of study have large student numbers or that educators have not adopted a constructionist approach. One of the views expressed was that it could be useful for higher levels of study. Some of the data extracts pertaining to the usefulness of this subtheme were:

Might be more useful for higher levels of study; Supports collaborative work.

This study shows limited support for the finding of Fidalgo (2012) that wikis are one of the Moodle modules used by students to create collaborative learning communities on topics of their interest.

3.2.2 Theme 2: Student involvement and productivity

The 'usefulness of student involvement and productivity tools' theme covered a number of subthemes, which are described below. Educators at both institutions agreed on the 'usefulness of student involvement and productivity tools' theme, which validates the need for the corresponding tools and features of described in Table 5 in the Appendix.

Online calendar The course calendar, which is both a planning and communication tool, was reported to be useful as it had an updated record of course events such as test and examination dates and submission deadlines for projects and/or assignments. Some of the data extracts pertaining to the usefulness of this subtheme were:

Student can use course calendar with deadlines to manage his/her own learning; One central calendar where you can put all the key dates so its reminding them whats coming up.

This finding was supported in the literature where students reported that they would use the online calendar for scheduling tasks and setting up alerts for upcoming assignments (Dahlstrom et al., 2013).

Online journal The online student journal was reported to allow students to reflect on their learning process. Some of the data extracts pertaining to the usefulness of this subtheme were:

Useful to incorporate portfolios through a journal; In Moodle there is an online journal and I like people to be reflective about their learning.

This finding supports the result of a previous study where educators reported creating a journal topic to give students a place for their own writing, which can be kept private between the student and the lecturer or shared with the class (Borwarnginn & Tate, 2014).

Student profiles The educators believed that student profiles were useful for students, as it allowed them to upload their pictures, edit their information, and conduct online discussions. Some of the data extracts pertaining to the usefulness of this subtheme were:

The students can edit their profiles; Profiles are more conducive to discussions.

This finding was supported the work of Halaseh (2014) who further argued that users' profiles together with participation and academic data can be analysed to get a understanding of course dynamics.

3.2.3 Theme 3: Course administration and management

The 'usefulness of course administration tools' theme for both the universities comprised two sub-themes, namely user management and course design and management, which are described below.

User management The user management function was performed largely by an administrator at University A and was, therefore, regarded as less useful than University B. Some properties of the user management function were reported useful and additional properties were recommended that would enhance usage. A positive comment from University B was that the system automatically uploaded registered students into the VLS and allowed guest access. Some of the data extracts pertaining to this subtheme were:

Should give students more privileges, their own classroom; lecturers should be given rights to reset password automatically; limit access to registered users; Most of our courses have guest access.

The usefulness of user management functions in this study confirmed the work of Botturi (2004), who reported that access privileges in terms of multiple level access privileges, and registration options such as Lightweight Directory Access Protocol (LDAP), batch and self-registration were considered important for adoption of an open source system.

Course design and management Course design was regarded as useful by interviewees at University A and University B. Educators expressed satisfaction in that the VLS allowed courses to be organised, and edited and components to be moved around. Some of the data extracts pertaining to the usefulness of this subtheme were:

Copying courses and rolling them over is supported; supports hiding of courses/documents being developed; Educators need to structure or organise their courses using a pre-defined format such as weekly or topics.

Course management was regarded as useful by interviewees at University A and University B. Educators found the use of authorised courses, course backups, password protected courses and selective release of course content to be particularly useful. Some of the data extracts pertaining to the usefulness of this subtheme were:

Supports changes, archives and back up of courses; records marks; supports online surveys; Authorised courses are very useful; all our courses have passwords on it; can selectively release course content.

An educator at University B recommended the addition of a welcome page to support programme-level communication with all students belonging to a discipline.

These findings confirm the usefulness of the VLS tools and features corresponding to course administration and management described in Table 6 in the Appendix.

3.2.4 Theme 4: Online assessment

The usefulness of online assessment tools comprised a number of subthemes, which are described below.

Online tests and quizzes Educators reported on the usefulness of creating and administering objective-type online assessments. Objective-type questions in the form of multiple-choice questions, true or false, matching and filling in the blank spaces, etc., were popular choices for online testing as they can be automatically scored. The availability of immediate feedback for online quizzes was reported to be the most helpful feature. Some of the data extracts pertaining to the usefulness of this subtheme were:

Allows quizzes only for self-assessment purposes with no grading; allows you to randomise your test questions and answers; sends an e-mail notification to the instructor upon completion of test; Importing questions is a useful feature; online tests opens and closes at a specific time; you can give back feedback immediately or delay feedback.

These findings confirm the usefulness of corresponding online assessment functionality described in Table 7 in the Appendix. Some of the challenges reported with this Blackboard tool were:

You cannot go into Blackboard and change anything like from MCQ to short answers it doesn't do that because it keeps a statistic of the test over time; Doesn't allow one to upload more than one image for each online test question.

Online marking and grading The online marking and grading sub-functions were welcomed by educators at both institutions. Some of the data extracts pertaining to the usefulness of this subtheme were:

The system can automatically mark multiple-choice questions; allows assessment feedback with comments and grades; exporting is there; I often use the grading forms; I find the criteria and indicators quite useful; Set up marking criteria on online marking forms; I like the integration of Turnitin into Moodle.

This finding supports the argument made by Bagarukayo and Kalema (2015), that computer-assisted marking technologies save lecturers time due to online marking and the ability to capture results electronically.

Some of the challenges reported with this tool were as follows:

System should allow for peer comments and not necessarily for grading purposes as students are not knowledgeable enough to grade other students work; should be able to set custom scales for grading; set up marking criteria on online marking forms where final grades are awarded.

Online grade book The grade book was met with mixed reactions. While many of the educators interviewed at both institutions acknowledged the usefulness and value of having a grade book in a VLS, it was the least used component of Blackboard and Moodle. A possible reason for this finding is that many educators are familiar/competent with the use of a spreadsheet tool for recording marks/grades, performing statistical analyses, and calculating class and final marks. Educators who have not been trained to use the grade book tool may therefore be less confident to use an unfamiliar tool for performing the same functions. Some of the data extracts pertaining to the usefulness of this subtheme were:

Grade book organises, records, and publishes marks; allows students to view/monitor their progress; allows tracking of students grades in a course over time; generates statistics per question; allows overriding marks for blog posts; assigns group assignment mark to all group members; has graphing ability to depict results.

This finding confirmed the proposal by Barron (2003) that online grade books should be able to assign weights to the assignments; apply formulas to determine final grades; and be uploaded and downloaded in common formats such as Excel. The perceived usefulness of these properties was confirmed in this study, and was consistent with grade book properties described in Table 7. Some of the challenges reported with this tool were as follows:

Grade book could be more effective/flexible with regards to in-depth feedback; Offline assessment grades have to be manually entered into Moodle grade book; It won't import from Excel. It does export very nicely.

Online assignment The online assignment sub-functions were welcomed by educators at both institutions. Peer reviews of assignments were reported to be of particular importance to educators involved in postgraduate programmes where fellow students were required to provide constructive criticism of their peers' work. In addition, educators assigned grades to peer reviewing of assignments so that students took this activity seriously. Some of the data extracts pertaining to the usefulness of this subtheme were:

Allows group assignments; peer review of assignments at postgraduate levels is useful; controls deadlines for assignments; return assignments with comments to students; gives confirmation of online submissions; has ability to turn the assignment back for further work; has digital box for assignment submission; Uploading assignment question file(s) is a nice-to-have feature; useful to have an online repository of all student assignment submissions with comments and final grades; allow single or multiple files assignment submission.

This finding supports the view of Barron (2003), who reports that the assignment-specific digital drop box is a valuable tool as it provides a designated area for assignment submissions as opposed to sending assignments as e-mail attachments. This finding also confirms the finding of Dlalisa and van Niekerk (2015), who reported that Blackboard permitted students to submit assignments at their own pace and in their own time.

Summary This finding on the theme of the 'usefulness of online assessments' supports the work of Martin (2008), who reported that assignment features of the Blackboard VLS were most useful. These findings by Barron (2003) and Martin (2008) were to some extent confirmed by the present research in that online submission of assignments, availability of immediate feedback for online quizzes, and online marking were reported to be useful. These findings confirm the need for automated testing management functions in a VLS, as summarised in Table 7.

Some of the additional features educators recommended were: a need for embedding of multiple pictures into quizzes; custom scales for grading; a client tool to create multiple choice questions on a PC and the subsequent upload of these questions into Moodle; a student presentation area providing server space to upload student projects; and the use of Adobe Acrobat and PDF files for online marking of electronic assignments where instructors can overlay in-depth freehand comments in coloured text on PDF documents while preserving the original submission.

3.2.5 Theme 5: Online tracking of student participation and progress

The 'usefulness of student tracking and progress tools' theme comprised two subthemes, which are described below. Educators at both institutions agreed on the 'usefulness of online tracking of student participation and progress tools' theme, which confirmed the need for the corresponding tools and features described in Table 8 in the Appendix.

Student participation Some educators required tracking of student participation in threaded forums and chats in an online environment. It was suggested that a VLS should provide activity reports, usage statistics and frequency of student participation in various forums to assist the lecturer in assigning final student participation marks. Some of the data extracts pertaining to the usefulness of this subtheme were:

Tracking student participation in online activities; track students contributions to discussions supported; Moodle gives you activity reports; graphs that Moodle generates that tells you the hours they spend on Moodle per week; when they logged in and logged out.

This finding supports the study of Botturi (2004), who reported that student tracking, where teachers can see student data and activities, was one of the key functions considered for adoption of a VLS.

Student progress Educators reported that they and the students can see the marks obtained for assessments and the progress made. Educators can view students' submissions. Some of the data extracts pertaining to the usefulness of this subtheme were:

Tracks student progress; keep track of student submissions; Tracking students marks is useful as they are able to see all their marks and how they are doing.

These findings supported the study of Bagarukayo and Kalema (2015) who reported that lecturers had access to learners' performance and learners found immediate feedback of online assessments useful.

3.2.6 Theme 6: Course content

The 'usefulness of course content tools' theme comprised two subthemes, namely user management and course design and management, which are described below. Educators at both institutions agreed on the 'usefulness of course content tools' theme, which confirmed the need for the corresponding tools and features described in Table 9 in the Appendix.

Course content creation Educators reported that this facility was useful in that it allowed them to create lessons, quizzes, tests, learning materials, learning portfolios, definition of terms in the glossary, and create content using wikis. Some of the data extracts pertaining to the usefulness of this subtheme were:

Create content viz. lessons, quizzes, tests etc.; support hyper linking where you can link to a page or bulk of pages; you can create lessons and link to the content of a particular topic, and to a quiz to test their knowledge of the topic; Create definition of terms or concepts in a course glossary, which can be linked to learning activities and tests; attach meta data on learning materials; create learning portfolios and home pages is important.

This finding support the work of Cavus and Alhih (2014) who maintained that a VLS must be able to assemble and deliver learning content rapidly, personalise content and enable knowledge reuse.

Course content delivery and management Educators reported that this facility allowed the use of multi-media content namely audio, video, podcast, and animation to scaffold the learning process. In addition, this facility allowed the use of links to external sources. Educators at both institutions agreed on the usefulness of the content creation and delivery/management subthemes, which confirmed the need for the content authoring, delivery and management tools and features described in Table 6. The 'course content delivery and management' subtheme was clearly found to more useful than the 'content creation' subtheme at both institutions. Some of the data extracts pertaining to the usefulness of this subtheme were:

Support for a variety of data formats such as Word and Excel documents; use of videos in a PDF; Post links to external resources; The use of multi-media content namely audio, video, podcast, and animation to scaffold the learning process; support embedding of code from providers of multimedia content; play PDF slide shows; use video and or podcasts to explain mathematical concepts.

This need for the course content management supported the work of Barron (2003), who argues that a VLS should allow instructors to upload files, organise the files in folders and subfolders, and edit them remotely. According to Barron (2003), an optimal VLS would provide content creation within the program in a WYSIWYG format, and allow the upload of HTML files. A VLS has brought the possibility of delivering learning content to a huge number of learners as quickly as possible.

3.3 Importance attached to non-functional characteristics of Blackboard and Moodle

The theme and subthemes relating to VLS non-functional characteristics deemed important by educators at the two universities were coded from all the interview transcripts.

Interview findings confirm the importance attached to the following non-functional characteristics of Blackboard and Moodle by educators at University A and University B: usability, security, reliability and efficiency. In addition educators confirmed the importance of flexibility, interoperability, extensibility, and standards. It should be noted while educators have confirmed the importance of VLS non-functional characteristics, several recommendations for improvement were suggested in the light of challenges experienced. These findings supported the work of Al-Busaidi and Al-Shihi (2010) who reported that reliability, system's functionality, interactivity, and response time were found to be significant on e-Learning acceptance and use. The findings of the present research confirmed the theory on required system non-functional characteristics as summarised in Table 2.

3.3.1 Usability

According to Ardito et al. (2005), the usability properties of a VLS should allow both teachers and learners to capably operate the system, and should be suitable for the planned learning activities.

All the interviewees from both universities agreed that usability is an important characteristic of a VLS. Some educators at University A commented that Blackboard was user friendly whilst others commented that Blackboard usability needed improvement. Data extracts confirming the importance of this characteristics were:

Blackboard is very user friendly; Tools should not be cumbersome to use; there should be a better flow in the way the environment works.

The educators at University B expressed the view that the user interface of Moodle needed improvement. Data extracts confirming the importance of the usability characteristic were:

User interface needs improvement; the functionality in Moodle is not readily discernible; navigation can be more intuitive; the design should be similar to familiar applications like Facebook.

This finding is consistent with that of McGill and Klobas (2009), who reported that ease of use leads to increased use. These findings concur with those of Egert et al. (2009), who reported the need to improve the user interface of a VLS. These findings confirm the design guidelines that a VLS should have an aesthetically pleasing/attractive interface and should incorporate appropriate icons to represent system functionality. Furthermore there should be clear, unambiguous, and intuitive navigation within the VLS as summarised in Table 2.

3.3.2 Security

According to Behkamal et al. (2009), security is the ability to prevent unauthorised access, whether accidental or deliberate, to programs and data. Interviewees at both universities viewed security as an important characteristic for both Blackboard and Moodle, as illustrated by the following data extracts:

Students should not be able to view other students marks or change marks; the system should not crash through malicious activity; access should be restricted to registered students; only registered students should take tests; Moodle is fairly secure.

These findings confirm the need for identification and authentication of users (Behkamal et al., 2009, p. 602), as well as ensuring the security of assessments (Robin et al., 1996). The issue of restricted access was raised by Robin et al. (1996) who argued that there should be restricted access to data based on permission levels. The finding on security supports the need for security measures such as passwords and encryption (ElementK, 2003) and ensuring that information is secure, accessible and accurate (Egert et al., 2009).

3.3.3 Reliability

Interviewees at both universities regarded reliability as an important characteristic, supported by the following data extracts:

reliable (twenty four hours seven days a week) availability, lecturers work in off-peak hours setting quizzes; reliability is a very important issue as you do not expect the system to be continually down; report downtime on a weekly basis

The findings with regard to reliability confirmed the need for little or no disruption/downtime as advocated by Horton and Horton (2003), as well the need for software fault tolerance argued by Behkamal et al. (2009).

3.3.4 Efficiency

Educators at both universities agreed that efficiency was an important characteristic in a VLS. Data extracts supporting the need for this characteristic were:

Uploading and downloading of assignments should not be time consuming; minimum number of steps to perform tasks; shouldn't freeze when too many people are logged in; It was a problem when we were just using our server and if eighty people logged on it just froze, because of the LaTeXcommands in between and so on; I want my computer systems to be slick and responsive; it is very efficient.

The findings with regard to efficient response time reiterates the need for the efficiency attribute to be incorporated into a VLS as reported by Kalinga (2008). These findings concur with the system efficiency sub-characteristics cited, namely keyboard shortcuts, importing registered students into the system, and performing operations on multiple objects (Kalinga, 2008); and maintaining performance level (Zeist & Hendriks, 1996).

3.3.5 Other non-functional characteristics deemed important

Flexibility Interviewees expressed the need for flexibility in a VLS, as illustrated by the following data extracts:

Access tools directly or access via links; seamless integration of tools rather than being separate entities; open architecture is needed; students should be able to choose a text only website, integration between the VLS and the University's website, the library, the publishers, online modules.

This finding supports the view of Meerts (2003), that there should be seamless integration or links to internal university systems such as the student management systems. The finding is also consistent with the works of Kritikou and Demestichas (2008); ElementK (2003) and Meerts (2003) who highlighted the need to import content from and supply content to digital libraries and other e-Learning systems.

Interoperability Interoperability was viewed as an important characteristic in a VLS. Data extracts verifying this view were:

importing and exporting data like class lists would be excellent to automatically populate students; Interoperability, yeah, that would be great, linking it to your library system; Interoperable with SMS [Student Management System]for exporting marks; bring in cvs [sic] files.

This finding supports the importance of interoperability standards and modular, extendable architectures in a technology environment (Wyles, n.d.).

Extensibility Data extracts confirming the usefulness of extensibility were:

Extensible so if I see third party tools that I want to integrate with the system I would be able to; With Moodle it's open-source there is thousands [sic] of plug-ins.

This finding confirms the work of Molebatsi (2011) who reported that Moodle allows for extensibility through addition of new functions into the site's directory, which stores all the plug-in modules.

Standards Standards compliance implies that the system conforms to "standards, conventions or regulations in laws and similar prescriptions" (Behkamal et al., 2009). SCORM is a "set of technical specifications that enables sharable, durable, and reusable Web-based learning content" (ElementK, 2003). The OpenID standard allows to access many websites without having to create new passwords for each website (Sakimura, Bradley, Jones, de Medeiros, & Jay, n.d.). Data extracts confirming support for VLS standards were:

Assistive technologies should be supported; There should be a standard where you log in once and access many different websites; Students developed some packages outside of Moodle and uploaded using SCORM.

 

4 DISCUSSION

The literature review on VLS tool functionality and non-functional characteristics provided the reference point for eliciting educator perceptions on the VLS quality characteristics using the broad framework of the ISO9126 software quality model. VLS tools deemed useful from the qualitative findings were communication, student productivity and involvement, course administration, assessment, student tracking and course content. The VLS non-functional characteristics deemed important from qualitative findings were usability, security, reliability, efficiency, flexibility, interoperability, extensibility and standards. The qualitative findings also revealed challenges associated with VLS functions and characteristics. The findings of the study were integrated into a proposed model of VLS Quality in Use Characteristics depicted in Figure 2 with relationships to perceived usefulness and perceived importance constructs, which is linked to system usage.

It is important to note that the present study targeted educators in higher education who were computer-literate and had the experience of using a VLS in their teaching practice. Another important factor worth mentioning was that a technical and training support system was available to keep the adopted VLS functional at both universities. In addition, the influence of demographic variables such as age, computer literacy, academic rank and discipline expertise of non-adoptees on the perceived usefulness of VLS tool functions was not covered in this study. These relationships can be explored in future studies.

 

5 CONCLUSION

The research makes a theoretical contribution to the field of VLS usage in the form of a proposed model representing the VLS quality in use characteristics. This model extends the discussion of quality characteristics derived from the ISO quality model by identifying VLS functions and non-functional characteristics and measuring educators' perceptions about the usefulness and importance of the quality characteristics of Blackboard and Moodle. The results demonstrate the validity of the ISO 9126 external quality model for assessing VLS quality in use characteristics. The research also demonstrates that the constructs of perceived usefulness and perceived importance, generally linked to technology usage can be used to measure educators' beliefs on software quality characteristics. The perceived usefulness and perceived importance constructs are related to implied user needs and user satisfaction, which are linked to VLS usage in higher education.

This knowledge can be used by managers/directors of e-Learning or educational technology departments to bridge the gap between features deemed useful and actual system usage via interventions such as training programmes and instructional design/development support. Managers/directors of e-Learning or educational technology departments can use the knowledge of non-functional characteristics deemed important to ensure that the implementation of a VLS in an organisation makes provision for characteristics such as security, reliability and performance when using a VLS. Furthermore, the constructs of perceived usefulness and perceived usability have been found to be determinants of usage, which in turn has implications for wider acceptance and usage amongst non-adopters of VLS in education. Finally, the findings and results of this research can be used by system designers to identify additional VLS tool features/functions required and non-compliant system quality characteristics and address these in future upgrades of systems. An improved quality in VLS design that supports the work of educators is likely to lead to an increase in VLS functionality usage.

 

References

Abran, A., Khelifi, A., Suryn, W., & Seffah, A. (2003). Usability meanings and interpretations in ISO standards. Software Quality Journal, 11(4), 325-338. Last accessed 19 Nov 2017. https://doi.org/10.1023/A:1025869312943        [ Links ]

Ardito, C., Costabile, M. F., de Marsico, M., Lanzilotti, R., Levialdi, S., Roselli, T., & Rossano, V. (2005). An approach to usability evaluation of e-learning applications. Universal Access in the Information Society, 4(3), 270-283. https://doi.org/10.1007/s10209-005-0008-6        [ Links ]

Asiri, M. J., Mahmud, R. B., Abu Bakar, K., Mohd Ayub, A., & Fauzi, B. (2012). Factors influencing the use of learning management system in Saudi Arabian higher education: A theoretical framework. Higher Education Studies, 2(2), 125-137. http://doi.org/10.5539/hes.v2n2p125        [ Links ]

Bagarukayo, E. & Kalema, B. (2015). Evaluation of elearning usage in South African universities: A critical review. International Journal ofEducation and Development Using Information and Communication Technology, 11(2), 168-183.         [ Links ]

Barron, A. (2003). Course management systems and online teaching. Journal of Computing in Higher Education, 15(1), 128-142. http://doi.org/10.1007/BF02940856        [ Links ]

Behkamal, B., Kahani, M., & Akbari, M. K. (2009). Customizing ISO 9126 quality model for evaluation of B2B applications. Information and Software Technology, 51(3), 599-609. http://doi.org/10.1016/j.infsof.2008.08.001        [ Links ]

Bevan, N. (1999). Quality in use: Meeting user needs for quality. Journal of Systems and Software, 49(1), 89-96. https://doi.org/10.1016/S0164-1212(99)00070-9        [ Links ]

Blackboard Inc. (n.d.). Blackboard academic suite learning system community system content system learn connect share. Last accessed 08 Dec 2017. Retrieved from https://www.blackboard.com/learning-management-system/blackboard-learn.html

Borwarnginn, P & Tate, A. (2014). An investigation into students' perceptions and lecturers' perceptions of a virtual learning environment. In Edulearn14 proceedings (pp. 1548-1554).

Botturi, L. (2004). Introducing a Moodle LMS in higher education: The e-Courses experience in Ticino (Switzerland). Last accessed 08 Dec 2017. Retrieved from https://www.learntechlib.org/p/43462/

Brooks, C. & Kettel, L. (2005). Building a learning object content management system. In Proceedings of World Conference on E-Learning in Corporate, Healthcare, & Higher Education (E-Learn 2005) (pp. 24-28).

Al-Busaidi, K. & Al-Shihi, H. (2010). Instructors' acceptance of learning management systems: A theoretical framework. Communications of the IBIMA, 2010, 1-10. https://doi.org/10.5171/2010.862128        [ Links ]

Cavus, N. & Alhih, M. (2014). Learning management systems use in science education. Procedia - Social and Behavioral Sciences, 143, 517-520. http://doi.org/10.1016/j.sbspro.2014.07.429        [ Links ]

Chang, H., Wang, C., Lee, M., Wu, H., Liang, J., Lee, S., ... Tsai, C. (2015). A review of features of technology-supported learning environments based on participants' perceptions. Computers in Human Behavior, 53, 223-237. http://doi.org/10.1016/j.chb.2015.06.042        [ Links ]

Chua, B. B. & Dyson, L. E. (2004). Applying the ISO 9126 model to the evaluation of an e-learning system. In R. Atkinson, C. McBeath, D. Jonas-Dwyer, & R. Phillips (Eds.), Proceedings of the 21st ASCILITE Conference (pp. 184-190). Last accessed 19 Nov 2017. Retrieved from http://www.ascilite.org/conferences/perth04/procs/chua.html

Cole, J. & Foster, H. (2007). Using moodle: Teaching with the popular open source course management system. O'Reilly Media, Inc.

Dahlstrom, E., Walker, J. D., & Dziuban, C. (2013). ECAR study of undergraduate students and information technology, 2013. Last accessed 03 Dec 2017. Retrieved from https://library.educause. edu/resources/2013/9 ecar study - of - undergraduate - students - and -information-technology-2013

Davis, F. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3), 319-340. https://doi.org/10.2307/249008        [ Links ]

Dlalisa, S. & van Niekerk, B. (2015). Factors affecting the post-implementation adoption and use of Blackboard amongst academics at DUT. In 9th Annual Teaching and Learning in Higher Education Conference Pre-Conference Proceedings (pp. 2-15). Last accessed 19 Nov 2017. Retrieved from http://tlhec.ukzn.ac.za/docs/default-source/default-document-library/conference-proceedings.pdf

Egert, C. A., Jacobs, S., & Barnes, S. B. (2009). When social worlds collide: Charting the intersection of social media and courseware / course management systems. In P Zaphiris & A. Ozok (Eds.), Online communities, LNCS 5621 (pp. 452-461). Spinger-Verlag Berlin. https://doi.org/10.1007/978-3-642-02774-1_49

ElementK. (2003). Learning management systems in the work environment: Practical considerations for the selection and implementation of an e-learning platform. Last accessed. Retrieved from http://elearnmap.ipgkti.edu.my/resource/gkb1053/sumber2/id_ip/lms_whitepaper.pdf

Fidalgo, P S. P (2012). Learning networks and moodle use in online courses: A social network analysis study (Doctoral dissertation, Universidade Nova de Lisboa, Lisbon, Portugal). Last accessed 19 Nov 2017. Retrieved from https://run.unl.pt/handle/10362/8862        [ Links ]

Gamage, V, Tretiakov, A., & Crump, B. (2011). Teacher perceptions of learning affordances of multi-user virtual environments. Computers and education, 57(4), 2406. http://doi.org/10.1016/j.compedu.2011.06.015        [ Links ]

Goh, W. W., Hong, J. L., & Gunawan, W. (2014). Exploring lecturers' perceptions of learning management system: An empirical study based on TAM. International Journal of Engineering Pedagogy, 4(3), 48-54. https://doi.org/10.3991/ijep.v4i3.3497        [ Links ]

Gonella, L. & Panto, E. (2008). Didactic architectures and organization models: A process of mutual adaptation. Last accessed 19 Nov 2017. Retrieved from https://www.openeducationeuropa.eu/sites/default/files/legacy_files/old/media15973.pdf

Guest, G., MacQueen, K. M., & Namey, E. E. (2012). Applied thematic analysis. Sage Publications. https://doi.org/10.4135/9781483384436

Halaseh, R. A. (2014). Studying learning networks within Moodle: A social network analysis approach (Doctoral dissertation, Technische Universität Berlin).         [ Links ]

Horton, W. & Horton, K. (2003). E-learning tools and technologies: A consumer's guide for trainers, teachers, educators, and instructional designers. Wiley.

Hueh, H.-P & Hsu, S. (2008). Designing a learning management system to support instruction. Communications of the ACM, 51(4), 59-63. https://doi.org/10.1145/1330311.1330324        [ Links ]

Inc., B. (n.d.). Blackboard learning system designer and instructor reference v. 6.2.0. Last accessed 08 Dec 2017. Retrieved from http://www.mhhe.com/mpss/support/HSSL/BbFormerWebCT_Resources/Instructor_ User_Guides/WebCTCE6_Designer&InstructorReference.pdf

Jasperson, J., Carter, P, & Zmud, R. (2005). A comprehensive conceptualization of post-adoptive behaviors associated with information technology enabled work systems. MIS Quarterly, 29(3), 525-557. https://doi.org/10.2307/25148694        [ Links ]

Kalinga, E. (2008). Development of an interactive e-learning management system (e-LMS) for Tanzanian secondary schools. Licentiate thesis. Last accessed 19 Nov 2017. Retrieved from http://bth.diva-portal.org/smash/get/diva2:836335/FULLTEXT01.pdf

Kanthawongs, P [Penjira] & Kanthawongs, P [Penjuree]. (2013). Individual and social factors affecting student's usage intention in using learning management system. Procedia-Social and behavioral sciences, 88, 89-95. https://doi.org/10.1016/j.sbspro.2013.08.484        [ Links ]

Kritikou, Y. & Demestichas, P (2008). User profile modeling in the context of web-based learning management systems. Journal of Network and Computer Applications, 31, 603-627. https://doi.org/10.1016/j.jnca.2007.11.006        [ Links ]

Li, Y. (1997). Perceived importance of information system success factors: A meta analysis of group differences. Information and Management, 32, 15-28. https://doi.org/10.1016/S0378-7206(97)00005-0        [ Links ]

Martin, F. (2008). Blackboard as the learning management system of a computer literacy course. MERLOT Journal of Online Learning and Teaching, 4(2), 138-145. Last accessed 19 Nov 2017. Retrieved from http://libres.uncg.edu/ir/uncw/f/martinf2008-1.pdf        [ Links ]

McGill, T. J. & Klobas, J. E. (2009). A task-technology fit view of learning management system impact. Computers and Education, 52(2), 496-508. http://doi.org/10.1016/j.compedu.2008.10.002        [ Links ]

Meerts, J. (2003). Course management systems (CMS) EDUCAUSE evolving technologies committee. Last accessed. Retrieved from https://library.educause.edu/resources/2003/1/course-management-systems-cms

Mlitwa, N. (2006). E-learning and learning management systems (LMS) in a changing higher education environment. In J.-P van Belle & I. Brown (Eds.), 36th Annual Conference of the Southern African Computer Lecturers Association: Electronic Conference Proceedings on CD-ROM.

Mlitwa, N. & Van Belle, J. (2011). Mediators for lecturer perspectives on learning management systems at universities in the Western Cape, South Africa. In Proceedings of the Pacific Asia Conference on Information Systems (PACIS 2011), 2011, Brisbane. Last accessed 19 Nov 2017. Retrieved from http://www.pacis-net.org/file/2011/PACIS2011-131.pdf

Molebatsi, M. M. (2011). Contextualized framework for learning management systems: The case ofa South African higher learning environment (M-Tech dissertation, Tshwane University of Technology).         [ Links ]

Mueller, D. & Strohmeier, S. (2010). Design characteristics of virtual learning environments: An expert study. In S. Strohmeier & A. Diederichsen (Eds.), Evidence-Based e-HRM? On the way to rigorous and relevant research, Proceedings ofthe Third European Academic Workshop on electronic Human Resource Management, Bamberg, Germany, May 20-21, 2010 (pp. 167-185). https://doi.org/10.1111/j.1468-2419.2010.00353.x

Nanayakkara, C. (2007). A model of user acceptance of learning management systems: A study within tertiary institutions in New Zealand. Last accessed 08 Dec 2017. Retrieved from http://connection.ebscohost.com/c/articles/28651965/model-user-acceptance-learning-management-systems-study-within-tertiary-institutions-new-zealand

Neal, L. & Miller, D. (2005). The basics of e-learning: An excerpt from Handbook of human factors in web design. Last accessed 08 Dec 2017. Retrieved from http://elearnmag.acm.org/featured.cfm?aid=1082219

Oliver, K. (2001). Recommendations for student tools in online course management systems. Journal of Computing in Higher Education, 13(1), 47-70. https://doi.org/10.1007/BF02940944

Paulsen, M. F. (2003). Online education, learning management systems, global e-learning in a Scandinavian perspective. NKI Forlaget.

Pritchett, C. C., Wohleb, E. C., & Pritchett, C. G. (2013). Educators' perceived importance of Web 2.0 technology applications. TechTrends, 57(2), 33-38. http://doi.org/10.1007/s11528-013-0643-3        [ Links ]

Robin, D. P, Reidenbach, R. E., & Forrest, P J. (1996). The perceived importance of an ethical issue as an influence on the ethical decision-making of ad managers. Journal of Business Research, 35(1), 17-28. http://doi.org/10.1016/0148-2963(94)00080-8        [ Links ]

Sakimura, N., Bradley, J., Jones, M. B., de Medeiros, B., & Jay, E. (n.d.). OpenID connect standard 1.0 - draft 21. Last accessed. Retrieved from http://openid.net/specs/openid-connect-standard-1_0.html

Sun, P-c., Cheng, H. K., & Finger, G. (2009). Critical functionalities of a successful e-learning system -An analysis from instructors' cognitive structure toward system usage. Decision Support Systems, 48(1), 293-302. https://doi.org/10.1016/j.dss.2009.08.007        [ Links ]

Trice, A. W. & Treacy, M. E. (1988). Utilization as a dependent variable in MIS research. ACM SIGMIS Database: The Database for advances in information systems, 19(3/4), 33-41. https://doi.org/10.1145/65766.65771        [ Links ]

Unal, Z. and Unal, A. (2011). Evaluating and comparing the usability of web-based course management systems. Last accessed 08 Dec 2017. Retrieved from https://www.informingscience.org/Publications/1358

Venkatesh, V, Brown, S., & Bala, H. (2013). Bridging the qualitative-quantitative divide: Guidelines for conducting mixed methods. MIS Quarterly, 37(1), 21-54.         [ Links ]

Wan, L., Zhao, C., Liu, Q., & Sun, J. (2005). Work in progress: An evaluation model for learning content management systems from a perspective of knowledge management. In Frontiers in Education, 2005. FIE' 05. Proceedings 35th Annual Conference. IEEE.

WCET. (n.d.). EduTools course management system features and criteria. Last accessed 19 Nov 2017. Retrieved from http://wcet.wiche.edu/sites/default/files/docs/resources/OldEdutoolsCMSFeatur esandCriteriaRev041513.docx

Wyles, R. (n.d.). Evaluation of Learning Management System software: Part ii of LMS evaluation. Last accessed 08 Dec 2017. Retrieved from https://static.aminer.org/pdf/PDF/000/269/656/idea_a_developing_tooLofJearning _resources_forJms_environments.pdf

Zeist, R. & Hendriks, P (1996). Specifying software quality with the extended ISO model. Software Quality Journal, 5(4), 273-284. http://doi.org/10.1007/BF00209185        [ Links ]

 

 

Received: 21 Sep 2016
Accepted: 13 Nov 2017
Available online: 8 Dec 2017

 

 

APPENDIX

 

Table 4 - Click to enlarge

 

 

Table 5 - Click to enlarge

 

 

Table 6 - Click to enlarge

 

 

Table 7 - Click to enlarge

 

 

Table 8 - Click to enlarge

 

 

Table 9 - Click to enlarge