ARTICLES

 

A Usability Evaluation of the Prototype Afrikaanse idiome-woordeboek^*

 

'n Bruikbaarheidsevaluering van die prototipe van die Afrikaanse idiome-woordeboek

 

 

Liezl H. Ball^I; Theo J.D. Bothma^II

^IDepartment of Information Science, University of Pretoria, South Africa (Uezl.baU@up.ac.za)
^IIDepartment of Information Science, University of Pretoria, South Africa (theo.bothma@up.ac.za)

 

 

---

ABSTRACT

The Afrikaanse idiome-woordeboek is a prototype e-dictionary of Afrikaans fixed expressions developed with the intention to test the functionalities of the e-dictionary. This dictionary is based on the function theory of lexicography. The e-dictionary makes use of various technologies. When digital tools are developed it is important to consider the usability of the tool. Usability evaluation was done on the Afrikaanse idiome-woordeboek to determine with what success it can be used. Discount usability methods, viz. heuristic evaluation and usability testing were used. This article reports on the findings from the usability tests which are discussed under the categories of content, information architecture, navigation, access (searching and browsing), help, customisation and the use of innovative technologies to manage data in e-dictionaries for search and display. The usability evaluation showed that the users did not always use the e-dictionary as the designers intended. Various recommendations are made to the designers of the Afrikaanse idiome-woordeboek, as well as for the design of e-dictionaries in general. Recommendations appropriate to e-dictionaries in general are made regarding usability evaluation, information architecture, searching in e-dic-tionaries, the data that can be included in e-dictionaries and training of users of e-dictionaries.

Keywords: e-dictionaries, online dictionaries, electronic dictionaries, usability evaluation, usability tests, discount usability, dictionary literacy

---

OPSOMMING

Die Afrikaanse idiome-woordeboek is 'n prototipe e-woordeboek van Afrikaanse vaste uitdrukkings wat ontwikkel is met die doel om die funksionaliteit van die ewoordeboek te toets. Die woordeboek is gebaseer op die funksieteorie van leksikografie. Die e-woordeboek maak gebruik van verskeie tegnologieë. Wanneer digitale werktuie ontwikkel word is dit belangrik om die bruikbaarheid van die tuig te oorweeg. Bruikbaarheidsevaluering is op die Afrikaanse idiome-woordeboek gedoen om te bepaal met watter mate van sukses dit gebruik kan word. Die metodes van afslag-bruikbaarheidsevaluering, naamlik heuristiese evaluering en bruikbaarheidstoetsing is gebruik. Hierdie artikel doen verslag oor die bevindinge van die bruikbaarheidstoetse en dit word bespreek onder die kategorieë inhoud, inligtingargitektuur, navigasie, toegang (soek en snuffel), hulp, aanpassing en die gebruik van innoverende tegnologieë om data in e-woordeboeke te bestuur vir soek- en vertoondoeleindes. Die bruikbaarheidsevaluering het gewys dat die gebruikers nie altyd die e-woordeboek gebruik het soos die ontwerpers bedoel het nie. Verskeie aanbevelings is gemaak vir die ontwerpers van die Afrikaanse idiome-woordeboek, so wel as vir die ontwerp van e-woordeboeke in die algemeen. Aanbevelings toepaslik vir e-woordeboeke in die algemeen is ook gemaak oor bruikbaarheidsevaluering, inligtingargitektuur, soek in e-woordeboeke, die data wat in e-woordeboeke ingesluit kan word en die opleiding van gebruikers van e-woordeboeke.

Sleutelwoorde: e-woordeboeke, aanlyn woordeboeke, elektroniese woordeboeke, bruikbaarheidsevaluering, bruikbaarheidstoetsing, afslag-bruikbaarheidsevaluering, woordeboekgeletterheid

---

 

 

1. Introduction

There are many exciting opportunities that technology brings to the field of lexicography. In the first place, much more data can be included in an e-dictionary. This has many advantages; for instance, e-dictionaries can include or link to more data (De Schryver 2003: 157), multimedia can be incorporated (Lew 2012: 344) or words can be written out and do not need to be abbreviated (De Schryver 2003: 157). Information technology also offers many advantages in terms of access to data; for instance, the speed with which data can be retrieved is a considerable advantage (Verlinde and Peeters 2012: 147) and various search features can be included to enable more efficient search (Lew 2012: 345, 351; Verlinde and Peeters 2012: 147). Bothma (2011) also suggests various modern technologies that could be used to enhance e-dictionaries, such as annotations, linked data, filtering and recommendations.

Technology can also allow e-dictionaries to be customised to the extent that it only gives information that is relevant in a specific situation. For example, a person who is writing a text needs different information than a person who is reading or a person who needs detailed background information about an item. This idea is formalised in the function theory of lexicography (see, for example, Bergenholtz 2011; Bergenholtz and Bergenholtz 2011; Bergenholtz, Bothma and Gouws 2015; Bergenholtz and Gouws 2007; Bergenholtz and Tarp 2003; Nielsen 2011; Tarp 2007, 2008, 2011).

when digital tools are developed, it is vital that these tools can be used effectively and efficiently by users; in other words, the usability of a tool is important. Usability can be seen as "the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use" (ISO 9241-11 1998). Usability becomes more important as products become more complex and can be critical to the success of a product (Tullis and Albert 2008: 5-7). Usability evaluation is the process where data about how users will use or do use a product are gathered and whether it is suitable and acceptable to users (Preece, Rogers and Sharp 2011: 433). Different methods can be used to do a usability evaluation, such as usability testing, inspection methods, inquiry methods, analytical modelling or simulations (Fernandez, Insfran and Abrahão 2011: 796-797; Shneiderman and Plaisant 2010: 134-135, 138-149). The usability evaluation method must be applicable to the product and the environment that the product will be used in; for example, products that will be employed in dangerous environments should ultimately be tested in real-life settings, not only in laboratories (Shneiderman and Plaisant 2010: 133).

Discount usability is an approach where simplified methods are used to evaluate a product (Nielsen 1993: 17; Nielsen 2009). Nielsen (2009) suggests that excellent results can be obtained if only a few participants are used in usability testing. About five participants are enough to point out the main usability problems (Nielsen 2000; 2009; 2012a). Furthermore, in discount usability, experts are used to conduct heuristic evaluations and prototypes, rather than complete systems are evaluated (Nielsen 2009). It has also been argued that sophisticated facilities are not needed (Preece, Rogers and Sharp 2011: 478; Nielsen 2012b).

There are many different aspects that a usability evaluation can assess. Some of the aspects that can be evaluated are navigation (e.g. Calisir et al. 2010), the structure or organisation of information (e.g. Hasan, Morris and Probets 2012), aesthetics and design problems (e.g. Hasan, Morris and Probets 2012; Neilson and Wilson 2011); search features (Hasan, Morris and Probets 2012); task completion (effectiveness) (e.g. Hamel 2012; Molich et al. 2010; Weir, Anderson and Jack 2006); time taken to complete a task (efficiency) (e.g. Lavie, Oron-Gilad and Meyer 2011; Molich et al. 2010) or user satisfaction (e.g. Lavie, Oron-Gilad and Meyer 2011; Roberts et al. 2013).

Though there are guidelines and criteria for evaluating websites, Ball (2016) has developed criteria specifically to evaluate e-dictionaries; see also Ball and Bothma (2018) for a summarised version. This set of criteria takes into consideration standard usability issues, but incorporates aspects from lexicography and developments in information technology. The main criteria are based on issues relating to content, information architecture, navigation, access (searching and browsing), help, customization and the use of innovative technologies to manage data in e-dictionaries for search and display.

 

2. Description of the Afrikaanse idiome-woordeboek

The Afrikaanse idiome-woordeboek is a prototype e-dictionary of Afrikaans fixed expressions and contains a subset of fixed expressions of the Afrikaans language. This e-dictionary was developed with the intention to test certain theories and the functionality of the e-dictionary and as such, visual design and aesthetics were purposefully ignored. The design of this dictionary is based on the function theory of lexicography and presents several dictionaries that are created from one large database (Bergenholtz, Bothma and Gouws 2011: 36). The different dictionaries are monofunctional and provide data that are relevant to specific situations. There are 15 fields for each expression in the database and different fields are presented in each dictionary. The five different dictionaries are based on different functions and are listed below.

- Function 1 - Ek wil basiese inligting oor die idioom hê - I want to have basic information about an expression

- Function 2 - Ek wil die betekenis van 'n idioom verstaan - I want to understand the meaning of an expression

- Function 3 - Ek wil 'n idioom in 'n teks gebruik - I want to use an expression in a text

- Function 4 - Ek wil alles oor 'n idioom weet - I want to know everything about an expression

- Function 5 - Ek wil 'n idioom na Engels vertaal - I want to translate an expression into English

Figure 1 shows the homepage of the Afrikaanse idiome-woordeboek and the five functions are given underneath the basic search field.

 

 

Figure 2 shows the basic data for the expression kabaal opskop (go through the roof) (function 1) which includes the following fields: expression, meaning, style and related expressions in English. Figure 3 shows all the data for this expression (function 4) and includes all the fields in the database that may be of interest to a user (fields for the editor's usage are not displayed).

 

 

 

 

The e-dictionary also makes use of various technologies such as advanced search and display options, browsing, multimedia in various articles, links to external sources that provide more data and customisation options. Each of these will be discussed below.

The advanced search and display options in the Afrikaanse idiome-woordeboek are shown in Figure 4. This allows a person to specify exactly in which fields in the database to search and exactly which fields to display in the article. A person can for example, choose to search for an expression with a certain meaning (e.g. dronk - drunk), by selecting to search in the meaning field (betekenis) and then to display the expression (idioom), its meaning (betekenis) as can be seen in Figure 5 as well as any other selected fields.

 

 

 

 

The ability to browse through the expressions in the Afrikaanse idiome-woordeboek is shown in Figure 6.

 

 

An example of an image (multimedia) that has been added to an article is shown in Figure 7.

 

 

A person can follow a link to an external source to see the context in which an example sentence has been used, see Figure 8.

 

 

The Afrikaanse idiome-woordeboek allows customisation in the sense that a user can save a selection of search and display fields and so create a custom dictionary specific to that user's requirements. Figure 9 shows where a selection has been saved.

 

 

Figure 10 shows how this saved selection can be loaded and a different search can be done with the options that have been saved.

 

 

The technologies in the Afrikaanse idiome-woordeboek discussed above enables a person to get only information that is relevant to a specific situation. Usability evaluation can show whether the technologies incorporated in this e-dictionary are indeed used successfully.

 

3. Methodology

The usability of the Afrikaanse idiome-woordeboek was evaluated through a heuristic evaluation and usability tests.

Heuristic evaluation is when experts evaluate a system according to a set of criteria (Nielsen 1993: 155; Schneiderman and Plaisant 2010: 134-135). Criteria to evaluate e-dictionaries were developed by Ball (2016) and were used to do an evaluation on the Afrikaanse idiome-woordeboek; a summary of the criteria is available in Ball and Bothma (2018). The heuristic evaluation of the Afrikaanse idiome-woordeboek involved one expert and was done during the week of 11 to 15 April 2016 and completed on 10 May 2016.

Usability testing is a method where participants are asked to complete several tasks to determine whether a product can be used successfully (Molich and Dumas 2008: 264; Preece, Rogers and Sharp 2011: 438). The participants are observed and can be recorded (Krug 2005: 135; Preece, Rogers and Sharp 2011: 478). One method that is often used in usability testing is the think-aloud method. Participants are encouraged to articulate their thoughts whilst performing the tasks to give the researchers insight into their actions (Shneiderman and Plaisant 2010: 143). At the end of a usability test the participant is often asked to complete a questionnaire to obtain subjective data (Fernandez, Insfran and Abrahão 2011: 802; Preece, Rogers and Sharp 2011: 477; Shneiderman and Plaisant 2010: 139).

Seven participants took part in the usability testing of the Afrikaanse idiome-woordeboek that was done from 16 April to 4 May 2016. The participants were selected by purposive sampling, and people with fairly similar demographics, backgrounds and experiences were chosen. Though the discount usability theory discussed earlier indicates that good results can be obtained from using few participants, the researchers recognise that a bigger sample would probably point out more issues. However, it was considered that seven participants would be sufficient for a first evaluation, also considering that a prototype was being evaluated. The evaluation of a final product could consider using more participants. The participants were met at locations that suited the participants. Though the tests were conducted in a laboratory type environment (office, study, etc.), as much as possible was done to reduce the artificial nature of the test. Each of the 16 tasks that the participants had to do contained a scenario that described a real information need. The participant could therefore imagine him-/herself in a certain situation and make an appropriate response. During the tests the participants were observed by a researcher and recorded. The participants themselves were recorded by a standalone camera and the computer screen was recorded using the screen capturing software, BB FlashBack Express. This software allowed for the participant to be recorded with the computer's webcam and the screen to be recorded at the same time. After completing the tasks, each participant was asked to complete a questionnaire with 35 questions, including both open- and close-ended questions. Both the tasks and questionnaire were designed to evaluate the e-dictionary according to the criteria for e-dictionaries.

The 16 tasks are listed in the table below. The tasks listed here are translated from the original that was given to the users in Afrikaans. The purpose of the task is given in an additional column. The tasks are listed here as the discussion that follows refers to specific tasks. The questionnaire that was given to participants is added as an appendix.

 

 

 

4. Findings

The findings from the usability testing will be discussed according to the main categories of the evaluation criteria for e-dictionaries (Ball 2016).

4.1 Content

When evaluating the content in an e-dictionary, the following should be considered: whether the e-dictionary can give relevant data, the level of detail and complexity of the data given, the credibility of the e-dictionary, the writing and editorial style used and the usage of multimedia.

To determine whether the Afrikaanse idiome-woordeboek gives relevant data to a user, the researchers firstly considered whether the tasks could be completed successfully, and secondly, whether data irrelevant to the task are withheld. (It is important to consider whether irrelevant data are withheld, because if too much data are given there is the possibility that the user is overwhelmed.) In terms of task completion, five of the seven participants could complete all the tasks successfully, but two participants completed all tasks except the fifteenth task correctly. At some points during the tests, the researcher had to provide some guidance if participants did not know how to proceed, but then the tasks could be completed. This was taken into consideration during the analysis of the tests. Due to this success rate of task completion, the researchers found that the e-dictionary is effective as it gives the data that are necessary to complete specific tasks. From the data collected in the questionnaire it was clear that most of the participants were positive that the dictionary provided relevant data for the task at hand and did not give too much data. However, during the tests participants reacted differently to the amount of data given. One participant specifically expressed that the perfect amount of data was given, whereas others expressed disappointment that not more data were given. One suggested that a brief summary of the expression is displayed with the option to see more data. In some cases, participants who had displayed more data found it easier to complete the tasks.

When looking at the level of detail and complexity of data that an e-dictionary provides, the use of external sources to provide more data to a user is evaluated. In the eighth task participants were required to follow an external link to see an example sentence in context (see Figure 8). Only one participant opened the link immediately, all other participants had to be prompted. Even on prompting, one of the participants did not open the link but commented that that is what (s)he would have done to see the sentence in context. Though few made comments about the use of the external sources to provide context for example sentences, most indicated in the questionnaire that the use of external sources is useful. Most participants agreed that the level of complexity and detail in the e-dictionary is appropriate.

Most participants agreed that the e-dictionary is credible, but two were not convinced. One of these participants indicated that the example sentences used in an e-dictionary should be grammatically correct and trustworthy. Though no task was designed specifically to determine the perceived credibility of the e-dictionary, one of the examples sentences that the participants encountered in doing task 3, is written in a mixture of Afrikaans and English (Waar my opvoeding te kort skiet moet my attitude maar opmaak" - "My attitude should make up for where I lack education") and three participants specifically commented about this during the tests. Another participant had noticed example sentences, taken from newspapers or other such sources, that express political or religious viewpoints and indicated that data in an e-dictionary should be neutral.

The writing and editorial style used in an e-dictionary can help a user to make sense of the data in the e-dictionary, particularly when words are not abbreviated. The Afrikaanse idiome-woordeboek included some abbreviations where references to other dictionaries were made. Two participants were particularly confused by the use of these abbreviations. In addition, the participants in general were negative about the labels in the Afrikaanse idiome-woordeboek. One commented that some of the labels were written in old-fashioned Afrikaans and another thought that the labels for the different dictionaries (functions) were too long.

The purpose of the sixth task was to determine whether the participants would find the inclusion of multimedia useful (see Figure 7). In this task the participants had to determine what the word loep ("lens") means and all the participants referred to the image included in the article to answer the question. One participant even enlarged the image to see it more clearly.

4.2 Information architecture

To make a judgement about the information architecture of an e-dictionary, the way in which the data are organised or structured to give access to it should be evaluated, as well as the way in which data are organised on page or article level.

The data in the Afrikaanse idiome-woordeboek are organised according to different functions (see Figure 1). Task 2 to 5 were specifically designed to evaluate the use of these functions in the Afrikaanse idiome-woordeboek. Participants did not struggle to find the functions. Initially, however, most participants would type in the search string and press the 'Enter' key without considering or selecting a function. The Afrikaanse idiome-woordeboek does not work with the 'Enter' key, which forced the participants to select an appropriate function and then click the search button. The five functions are not only on the homepage, but repeated on the subsequent pages so that a user can change options easily. Most participants changed from one function to another easily, but some participants went back to the homepage to redo the search to try to get to the relevant data if they were not successful at first. Only one participant specifically tried to work out what the difference between the different functions are. At first (s)he assumed they were the same, but later noticed that different fields were displayed in different options.

Tasks 2 to 5 will be discussed in more detail. For these tasks (where the function and the task matched clearly) most participants used the correct function and could complete the task easily. To complete task 2, the most appropriate function to consult was function 2 and four participants used this function. Two participants left the default function selected, but one of these participant changed to function 2 and then to function 4 once the article was displayed. Another participant selected function 4. The most appropriate function to do task 3 was function 3 and four participants did this. Two other participants tried the advanced search first, before also using the basic search and function 3. One participant tried various search strings and opened the articles with either function 3 or 4. To do task 4, a user has to select function 4 and six participants did this. Only one participant used the advanced search options, but did not select any display fields. Upon opening the article, (s)he saw no data and selected function 4 to display data. The purpose of task 5 was to see if the participants would use function 5 to get a translation for a specific expression. All participants expect one used function 5 to start the search. This participant chose function 4. Once the article was open, some participants immediately opened function 4 to see more data.

In other tasks (specifically tasks 7 and 15) that were designed to test other issues, some participants found the functions confusing. This happened specifically if a participant did not select the correct function or expected a function to contain data that it did not. For example, in the Afrikaanse idiome-woordeboek, synonyms are displayed under functions 1 and 4, but not 3. This caused some confusion.

In terms of the layout of a page, participants did not have trouble to find the correct data in the article and most indicated in the questionnaire that the data in the article are clearly organised. Most also indicated that they thought it useful to be able to filter data in an article. This could refer to the use of functions that filter pre-selected fields or to the advanced search options where display fields can be selected (see Figure 4). Though the responses from the questionnaire were predominantly positive, it was evident that the advanced search and displayed options were confusing to most participants. Tasks 9 and 12 tested the use of the display options specifically, but their use in other tasks was noted. The display options were sometimes ignored, all fields were selected, inappropriate fields to the task at hand were selected or the same fields as the search fields were selected. In the article of an expression, the Afrikaanse idiome-woordeboek does not display fields when there are no data for those fields in the database. For example, if no synonyms have been added in the database, this field is omitted in the article. This caused confusion for some participants; for example, if such a display field had been selected specifically and then did not display.

From these usability tests it was clear that most participants struggled to filter data on a page by using the advanced display options. However, during the tests it was clear that some participants wanted to filter out irrelevant data. For example, one participant wanted to filter out the grammar and reference data and another participant wanted to filter out the translations.

4.3 Navigation

Under the category of navigation, the ease with which a user moves through the e-dictionary is evaluated, as well as whether the user knows where (s)he is at all times. As links are the primary mechanisms through which a user navigates, they are also evaluated.

The ease with which a user can navigate will influence how quickly the user will find the relevant data for a specific task. This allows for a judgement about the efficiency to be made. The time it takes a user to get to data in an e-dictionary is very important. In this study, the users were asked to think-aloud during the tests which gave valuable data, but negatively influenced the time it took to do a task. Consequently, in this study it was not sensible to look at the time it took to complete tasks, but rather to look at the steps that participants took to get to the relevant data. The participants themselves indicated that they could find data easily, but felt that they had to scroll too much or click through too many levels. The scrolling was probably caused by the fact that the results were listed below the advanced search, and as such participants typically had to scroll down after a search to see the results. There were many comments about the fact that the search results were not prominent. Though the hierarchy of the site is fairly shallow (typically two levels), the amount of clicking was influenced by two factors. The search results themselves do not contain any additional descriptive data. A user first has to open the expression to see the article to find the data to answer the question in the task. Secondly, the search field is only on the homepage. The user then always has to navigate back from the article to start another search. Based on this it appears that participants did not find the Afrikaanse idiome-woordeboek efficient to use.

Participants indicated that they knew where they were in the e-dictionary, but were not as positive about knowing where to go next in the e-dictionary. This could be as a result of the fact that most participants did not know how to store the advanced search and display options in task 12 or use them in task 13. Very few participants found the help option in task 16. Some participants also felt that the link to the homepage could have been more prominent.

Though participants were negative about the labels as mentioned under the section on content, they indicated that they knew where the links were going to lead them and it did not lead them to unexpected places.

4.4 Access (searching and browsing)

The way in which a system provides access to the data in the e-dictionary should be evaluated. Access is typically given through searching or browsing options. The basic as well as advanced search options in an e-dictionary should be evaluated, as well as any browsing mechanisms. Filters can help users to narrow the content down to something specific. If a large number of results is retrieved, users should be able to manipulate the results. Lastly, the processing speed can be evaluated, though generally, systems are fast enough that this is becoming negligible.

In the usability tests, tasks 1 to 7 and task 15 could have been done by using the basic search. As has been mentioned before, the basic search field is only on the homepage (see Figure 1). As a result, most participants did not think that it is easy to find or that it is easy to change a search and search for something new. Two participants had made typing errors whilst searching and neither picked it up immediately when no results were retrieved.

Tasks 8-11 were designed to test the use of the advanced search features (see Figure 4). Although the data from the questionnaire made it clear that most participants think advanced search features can help a person to find relevant data, they were not convinced that the advanced search features in the Afrikaanse idiome-woordeboek are easy to use or easy to understand. From the usability tests it was also clear that the advanced search and display options were confusing and generally used incorrectly. In addition, some of the comments from the participants were that there are too many options, it is time consuming and the layout should be improved to communicate more clearly how the advanced search and display fields should be used. Most participants used the advanced search incorrectly, by indicating in the search field what data they wanted to display, instead of selecting where (i.e. in what field) they wanted to search. As the display fields are the same as the advanced search fields, these were often ignored or the same fields were selected. If the article then did not display any data or not the relevant data, it caused confusion.

One participant assumed that the basic and advanced search options were linked and used them together; for example, (s)he would select fields under the advanced search, but search with the basic search. Two participants specifically commented that the difference between the sections on the homepage were not clear.

Task 14 tested the use of the browse option, where all the expressions are listed in an alphabetical list (see Figure 6). Most participants did not see the browse option initially and tried to search for an expression that starts with an 'M' by using the basic or advanced search. Once the researcher had pointed out the browse option, they could use it and from the data from the questionnaire it is clear that the participants were positive about the ability to browse through the dictionary. One of the participants had used the browsing option in previous tasks when (s)he had not been successful in using the search options. One of the suggestions for improvement was that it would be helpful to only type in the first couple of letters and then jump to the expression(s) that contains those letters when browsing.

Another point of evaluation in terms of browsing is the ability to browse between items in the e-dictionary. The purpose of task 7 was to test whether users could browse between expressions, by asking users to find an appropriate synonym for an expression. Some participants did not follow the link for the synonym. Others followed the link successfully to confirm the meaning of the expression. Others followed various internal links (links for translations and synonyms) to confirm that they had selected a valid synonym. In a different task (task 5) one participant also relied heavily on internal links to confirm his/ her answer.

Filtering in the Afrikaanse idiome-woordeboek is made possible through functions, as well as the advanced search and display options. These have been discussed already under the category information architecture.

In some systems it is possible to search in the results that have been retrieved or to manipulate the results to some extent to refine them further; one participant commented that (s)he would have liked to be able to search in the results. However, most participants felt that the search results were logically arranged. Some participants were frustrated that the search results were not saved. This means that if a user retrieves more than one result, open one and go back to view the other, (s)he has to redo the search.

4.5 Help

The purpose of task 16 was to see whether participants could find the 'help' option easily and find the information in the documentation useful. Although most participants struggled to find the help option (an icon of a question mark), most thought that the information provided sufficient help. There were two incidents in tasks before task 16, where participants did not know how to proceed and consulted the 'help' for guidance. Neither found the information they were looking for in the documentation.

4.6 Customisation

The Afrikaanse idiome-woordeboek allows a user to store selections for the advanced search and display options (see Figures 9 and 10). Task 12 tested whether participants could save a selection and task 13 tested whether the participants could retrieve that selection and use it in a search.

Most users struggled to find where to store the selection and had to be shown. Some of the participants were also confused about what was saved and how it was being done. In order to save a search a person has to enter their email address, which is used to uniquely identify the user. This was also confusing, as some participants thought that something was emailed to them. Another participant was sceptical about putting in an email address in a system. Some participants thought that the results were being saved in task 12 and that in task 13 they had to search in the results of task 12. One participant was confused that upon loading a saved selection, the check boxes for that selection were not automatically checked. Another participant redid the search in task 12 and saved the selection a second time and noticed that the system does not validate the names of the saved selection as (s)he had used the same name in both instances.

4.7 Innovative technologies

No other technologies, apart from those already discussed, were tested in the Afrikaanse idiome-woordeboek.

 

5. Recommendations

5.1 Usability evaluation

The usability tests on the Afrikaanse idiome-woordeboek clearly showed that the users did not always use the functionality of the e-dictionary as the designers had intended. This underscores the importance of usability evaluation when an e-dictionary is designed that makes use of innovative technologies. Users do not necessarily have the same point of view as the designers of a system. Usability evaluation will help to show how users really use e-dictionaries and how they understand the options that are included.

This study also supports the idea that usability evaluations do not need to be elaborate or expensive exercises, but that simplified usability evaluation can be very effective. Only seven participants took part, the study was conducted in simple and accessible environments and various important usability issues were identified.

5.2 Information architecture

As was explained earlier, information architecture refers to the way in which the information in a product is organised or structured. In the case of the Afrikaanse idiome-woordeboek the information was organised in different functions and presented as different dictionaries to the user. From the usability tests it is clear that the difference between the functions were not always clear and the users did not always know what to expect from the functions. The researchers suggest that more is done in e-dictionaries to communicate the underlying structure or information architecture to the users. One way to communicate more clearly with the users is to write labels, headings or other micro-content that are easy to understand. For example, the participants in these usability tests were negative about the labels used in the Afrikaanse idiome-woordeboek.

A suggestion for the labels for the different functions on the homepage is suggested in Figure 11.

 

 

5.3 Searching in e-dictionaries

From the usability tests, it seems that users expect more from the search algorithm and the results themselves before opening a relevant article. For example, one user tried to determine the correct usage by using two different search strings and to make a judgement based on the number of retrieved results, trusting that the most results would be the correct option. The two users who had made typing errors and did not notice it at first should be considered. Implementing a fuzzy matching function to help users who mistype could be a valuable addition to an e-dictionary. Auto-completion of text could also be helpful, and might be something that users expect as it is not uncommon in search systems. There were also various suggestions that the results should give a little more data. This could, for example, help a user know that the correct item is selected to open. It is therefore recommend that more is done to improve the searching algorithms in e-dictionaries, as well as explore different ways to display the search results. In addition, more advanced search features can be added and different ways to present the advanced search options can be tested. For example, though the Afrikaanse idiome-woordeboek allows a user to search in different fields, it does not allow a user to define Boolean operators between fields explicitly, nor does it allow truncation. Even different layouts for the advanced search fields can be tested to see if other presentations make it easier to understand.

Apart from search features, it is recommended that some sort of browsing option is included. A user has to have a clearly defined information need and understand the information space for a search to be successful. If a user struggles with the search and a browsing option is available, then there is at least another access point to the content. This idea is supported by the case of the participant who had made a typing error and then tried the browse option to get to the relevant expression.

5.4 Data for e-dictionaries

At various points during the usability evaluation different participants had commented that they had expected or would have liked to see more data. The researchers therefore recommend that lexicographers should not hesitate to include more data in e-dictionaries. However, there were participants who felt the amount of data given was sufficient. As a result it is suggested that more data should always be accompanied with the ability to filter out unwanted data to avoid overload. Most participants also indicated that it is helpful to be able to filter data on a page. It can also be useful to display more data with a button or link that opens more data.

There are many sources available for lexicographers to include more data in e-dictionaries (Bothma 2011: 80), yet care should be taken that users know when something has been approved by lexicographers and when something is from and untested external source (Heid, Prinsloo and Bothma 2012: 285; Tarp 2012: 264). This is confirmed in these usability tests as the participants were sceptical about some of the example sentences that were included.

As some users seem to want more information, it is suggested that different ways to incorporate or give access to external sources are explored.

5.5 Education and training

Even if usability evaluation and proper design can produce e-dictionaries that are easy to use, the value of education and training should not be underestimated. Good user skills are imperative to successful dictionary consultation (Lew 2013: 16). Particularly if an e-dictionary makes use of advanced search features or other innovative technologies, users should be educated to be able to make use of these. In his paper, Lew (2013) discusses skills that are necessary for the successful consultation of e-dictionaries and how it is related to digital literacy. He discusses how some traditional skills necessary for dictionary use change in the digital environment; for example, e-dictionaries can have a wider variety of structures or data in an article can be hidden. Furthermore, he discusses various skills that are purely relevant for a digital environment, such as searching with wildcards or using Boolean operators. In his conclusion Lew (2013: 29) remarks that an "important concern is finding an appropriate context for teaching e-dictionary skills." Ball and Bothma (2017) suggest that information literacy programmes should be considered as a platform where dictionary use can be taught. Information literacy programmes are already used to teach users how to search in online environments such as databases and how to use search techniques such as Boolean operators and wildcards effectively. Dictionary literacy can become a subset of information literacy.

 

6. Conclusion

The Afrikaanse idiome-woordeboek incorporates various technologies to allow a user to get only relevant information for a specific situation. The usability test-ing of this e-dictionary showed that users do not necessarily use a system as the designers intend and various usability issues could be identified with discount usability methods. Various recommendations regarding the design of the Afri-kaanse idiome-woordeboek, as well as e-dictionaries in general, have been made. If the Afrikaanse idiome-woordeboek is modified according the suggestions in this article, further usability tests can be done to see if it can be used more effi-ciently to get only relevant information for a specific situation.

Usability evaluations do not need to be complicated and expensive exer-cises. This article has shown that using a few users in usability tests as sug-gested by the discount usability approach is effective in finding usability issues in a design. If lexicographers are serious about designing modern e-dictionaries that incorporate technology that is usable, they should embrace usability evaluation as practice. The discount usability methods of heuristic evaluation and usability testing are both appropriate for lexicographers. Lexicographers can perform heuristic (expert) evaluations by using the criteria explained in Ball and Bothma (2018) and also do their own usability tests by creating tasks and observing users as was explained in this article. The usability evaluations should be used at various stages in the development of e-dictionaries. Iterative evaluations after each design phase should be done. Discount usability meth-ods are purposefully simple and non-expensive so that a product can be evalu-ated repeatedly throughout the design process. In the initial stages, prototypes can be evaluated and then progressively more complete products. If prototypes are evaluated then changes can be made more easily. For example, in the case of the Afrikaanse idiome-woordeboek, a prototype was evaluated. As it is a proto-type, it will be easier to implement the necessary changes than if the evalua-tions were only done after the project has been completed.

The best type of prototype to use for usability evaluation can be a point of future research. The designers of the Afrikaanse idiome-woordeboek purposefully did not pay attention to the visual design of the system, as the purpose of the prototype was to be able to test the functionality of the e-dictionary. However, it is unclear if the lack of visual appeal or visual cues had an influence on the users' ability to use the e-dictionary effectively. For example, would improving the layout of the fields in the advanced search without changing the function-ality change users' use of the advanced search? Or would better visual cues improve their understanding of the information architecture? It is therefore still an open question as to what extent the functionality of an e-dictionary can be tested without taking visual design into consideration.

More usability evaluations of e-dictionaries and more research of best practices of usability evaluations of e-dictionaries can lead to advanced infor-mation tools that address users' needs effectively.

 

7. References

Ball, L.H. 2016. An Evaluative Study to Determine to What Extent Technology Can be Used in e-Dictionaries to Provide Relevant Information on Demand. Master's Thesis. Pretoria: University of Pretoria. https://repository.up.ac.za/handle/2263/61338.         [ Links ]

Ball, L.H. and T.J.D. Bothma. 2017. The Importance of Usability Evaluation when Developing Digital Tools for a Library - a Case Study. Proceedings of Bobcatsss 2017, 25-27 January 2017, Tampere, Finland: 137-142. http://tampub.uta.fi/handle/10024/101894.

Ball, L.H. and T.J.D. Bothma. 2018. Establishing Evaluation Criteria for e-Dictionaries. Library HiTech 36(1). (In press.         [ Links ])

Bergenholtz, H. 2011. Access to and Presentation of Needs-adapted Data in Monofunctional Internet Dictionaries. Fuertes-Olivera, P.A. and H. Bergenholtz (Eds.). 2011: 30-53..

Bergenholtz, H. and I. Bergenholtz. 2011. A Dictionary is a Tool, a Good Dictionary is a Mono-functional Tool. Fuertes-Olivera, Pedro A. and Henning Bergenholtz (Eds.). 2011: 187-207.

Bergenholtz, H., T.J.D. Bothma and R.H. Gouws. 2011. A Model for Integrated Dictionaries of Fixed Expressions. Kosem, I. and K. Kosem (Eds.) 2011. Electronic Lexicography in the 21st Century: New Applications for New Users: Proceedings of eLex 2011, Bled, Slovenia, 10-12 November 2011: 34-42. Ljubljana: Trojina, Institute for Applied Slovene Studies.         [ Links ]

Bergenholtz, H., T.J.D. Bothma and R.H. Gouws. 2015. Phases and Steps in the Access to Data in Information Tools. Lexikos 25: 1-30.         [ Links ]

Bergenholtz, H. and R.H. Gouws. 2007. Korrek, volledig, relevant. Dít is die vraag aan leksikografiese definisies. Tydskrif vir Geesteswetenskappe 47(4): 568-586.         [ Links ]

Bergenholtz, H. and S. Tarp. 2003. Two Opposing Theories: On H.E. Wiegand's Recent Discovery of Lexicographic Functions. Hermes, Journal of Linguistics 31: 171-196.         [ Links ]

Bothma, T.J.D. 2011. Filtering and Adapting Data and Information in the Online Environment in Response to User Needs. Fuertes-Olivera, P.A. and H. Bergenholtz (Eds.). 2011: 71-102.

Calisir, F., A.E. Bayraktaroğlu, C.A. Gumussoy, YI. Topcu and T. Mutlu. 2010. The Relative Importance of Usability and Functionality Factors for Online Auction and Shopping Web Sites. Online Information Review 34(3): 420-439.         [ Links ]

De Schryver, G.-M. 2003. Lexicographers' Dreams in the Electronic-Dictionary Age. International Journal of Lexicography 16(2): 143-199.         [ Links ]

Fernandez, A., E. Insfran and S. Abrahão. 2011. Usability Evaluation Methods for the Web: A Systematic Mapping Study. Information and Software Technology 53(8): 789-817.         [ Links ]

Fuertes-Olivera, Pedro A. and Henning Bergenholtz (Eds.). 2011. e-Lexicography: The Internet, Digital Initiatives and Lexicography. London/New York: Continuum.         [ Links ]

Hamel, M.-J. 2012. Testing Aspects of the Usability of an Online Learner Dictionary Prototype: A Product- and Process-Oriented Study. Computer Assisted Language Learning 25(4): 339-365.         [ Links ]

Hasan, L., A. Morris and S. Probets. 2012. A Comparison of Usability Evaluation Methods for Evaluating e-Commerce Websites. Behaviour & Information Technology 31(7): 707-737.         [ Links ]

Heid, U., D.J. Prinsloo and T.J.D. Bothma. 2012. Dictionary and Corpus Data in a Common Portal: State of the Art and Requirements for the Future. Lexicographica 28(1): 269-291.         [ Links ]

ISO 9241-11. 1998. ISO 9241-11:1998 Ergonomic Requirements for Office Work with Visual Display Terminals (VDTs) - Part 11: Guidance on Usability. http://www.iso.org/iso/catalogue_detail.htm?csnumber=16883.

Krug, S. 2005. Don't Make Me Think!: A Common Sense Approach to Web Usability. Second edtion. Berkeley: New Riders.         [ Links ]

Lavie, T., T. Oron-Gilad and J. Meyer. 2011. Aesthetics and Usability of In-Vehicle Navigation Displays. International Journal of Human-Computer Studies 69: 80-99.         [ Links ]

Lew, R. 2012. How Can We Make Electronic Dictionaries More Effective? Granger, S. and M. Paquot (Eds.). 2012. Electronic Lexicography: 343-361. Oxford: Oxford University Press.         [ Links ]

Lew, R. 2013. Online Dictionary Skills. Kosem, I., J. Kallas, P. Gantar, S. Krek, M. Langemets and M. Tuulik (Eds.). 2013. Electronic Lexicography in the list Century: Thinking Outside the Paper. Proceedings of the eLex 2013 Conference, 17-19 October 2013, Tallinn, Estonia: 16-31. Ljubljana/Tallinn: Trojina, Institute for Applied Slovene Studies/Eesti Keele Instituut.         [ Links ]

Lindgaard, G. and J. Chattratichart. 2007. Usability Testing: What Have We Overlooked? CHI 2007: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, San Jose, CA, USA, 28 April-3 May, 2007: 1415-1424. New York: ACM.         [ Links ]

Molich, R., J. Chattratichart, V. Hinkle, J.J. Jensen, J. Kirakowski, J. Suaro, T. Sharon and B. Traynor. 2010. Rent a Car in Just 0, 60, 240 or 1,217 Seconds? - Comparative Usability Measurement, CUE-8. Journal of Usability Studies 6(1): 8-24.         [ Links ]

Molich, R. and J.S. Dumas. 2008. Comparative Usability Evaluation (CUE-4). Behaviour & Information Technology 27(3): 263-281.         [ Links ]

Neilson, C.J. and V. Wilson. 2011. "We Want it Now and We Want it Easy": Usability Testing of an Online Health Library for Healthcare Practitioners. Journal of the Canadian Health Libraries Association 32(2): 51-59.         [ Links ]

Nielsen, J. 1993. Usability Engineering. Cambridge: Academic Press.         [ Links ]

Nielsen, J. 2000. Why You Only Need to Test with 5 Users. http://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/.

Nielsen, J. 2009. Discount Usability: 20 Years. http://www.nngroup.com/articles/discount-usability-20-years/.

Nielsen, J. 2012a. How Many Test Users in a Usability Study? http://www.nngroup.com/articles/how-many-test-users/.

Nielsen, J. 2012b. Traveling Usability Lab. http://www.nngroup.com/articles/traveling-usability-lab/.

Nielsen, S. 2011. Function- and User-related Definitions in Online Dictionaries. Kartashkova, F.I. (Ed.). 2011. Ivanovskaya lekskograficheskaya shkola: traditsii i innovatsii (Ivanovo School of Lexicography: Traditions and Innovations. A Festschrift in Honour of Professor Olga Karpova): 197-219. Ivanovo: Ivanovo State University.         [ Links ]

Preece, J., Y. Rogers and H. Sharp. 2011. Interaction Design. Beyond Human-Computer Interaction. Third edition. Chichester: John Wiley and Sons.         [ Links ]

Roberts, M.J., E.J. Newton, F.D. Lagattolla, S. Hughes and M.C. Hasler. 2013. Objective versus Subjective Measures of Paris Metro Map Usability: Investigating Traditional Octolinear versus All-Curves Schematics. International Journal of Human-Computer Studies 71: 363-386.         [ Links ]

Shneiderman, B. and C. Plaisant. 2010. Designing the User Interface. Strategies of Effective HumanComputer Interaction. Fifth edition. Boston: Pearson Education.         [ Links ]

Tarp, S. 2007. Lexicography in the Information Age. Lexikos 17: 170-179.         [ Links ]

Tarp, S. 2008. The Third Leg of Two-Legged Lexicography. Hermes - Journal of Language and Communication Studies 40: 117-132.         [ Links ]

Tarp, S. 2011. Lexicographical and Other e-Tools for Consultation Purposes: Towards the Indi-vidualization of Needs Satisfaction. Fuertes-Olivera, P.A. and H. Bergenholtz (Eds.). 2011: 54-70.

Tarp, S. 2012. Online Dictionaries: Today and Tomorrow. Lexicographica (28)1: 253-268.         [ Links ]

Tullis, T. and W. Albert. 2008. Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics. Burlington: Morgan Kaufmann.         [ Links ]

Verlinde, S. and G. Peeters. 2012. Data Access Revisited: The Interactive Language Toolbox. Granger, S. and M. Paquot. (Eds.). 2012. Electronic Lexicography: 147-162. Oxford: Oxford University Press.         [ Links ]

Weir, C.S., J.N. Anderson and M.A. Jack. 2006. On the Role of Metaphor and Language in Design of Third Party Payments in eBanking: Usability and Quality. International Journal of HumanComputer Studies 64: 770-784.         [ Links ]

 

 

* An shortened version of this article was presented at the Twenty-second Annual International Conference of the African Association for Lexicography (AFRILEX), hosted by the School of Languages and Literatures: African Language Studies Section, Rhodes University, Grahamstown, South Africa, 26-29 June 2017, while the original went through the editorial process for publication in Lexikos.

 

 

Appendix - Click to enlarge