Drawback of the pedagogical usability criteria

The first drawback of the pedagogical usability criteria presented here is their generalisability to other domains. In this study, generalisability is limited by various factors, for example, the small sample size, the narrow age range of respondents’ and the limited number of learning material evaluated. To address these issues, we are currently conducting empirical studies that aim at evaluating more representative set of learning material in different domains targeted for both adolescent and adult learners. The criteria of pedagogical usability will in the future be complemented by a segment evaluating “mobile usability” (Syvänen, Nokelainen, Ahonen & Turunen, 2003).

The second drawback, although in a more debatable sense, is the methodology we have applied to measure

usability, that is, a self-report questionnaire aiming to measure subjective end-user satisfaction. According to

researchers specialising in the measurement of usability (Rubin, 1994; Nielsen, 1993; Kirakowski, 2003), the use of questionnaires is normally justified in usability research as part of a test arrangement where a user answers a pre-test questionnaire, uses the application that is to be evaluated and then answers a post-test questionnaire evaluating the usability. However, Kirakowski (2003) sees self-evaluation questionnaires as particularly appropriate for measuring subjective matters (e.g., the enjoyability of use) in usability research. We all agree, that various other test arrangements (Rubin, 1994; Nielsen, 1993, 192-200) are better for gathering factual data (e.g., time spent in doing a particular task). On the other hand, the criteria developed here can also be used as the basis for heuristic evaluation (Nielsen & Molich, 1990).

The third drawback is that the criteria do not directly address the issue of cultural sensitivity (Reeves, 1994).

Authority (e.g., who is ‘permitted’ to ask questions or ‘obligated’ to lead group work on the basis of his/her

social status) and collaboration (e.g., are questions regarding the peer-learners presentation or evaluation of other learners’ work allowed) are examples of cultural issues that are strongly connected to pedagogics. The problem of describing cultural sensitivity via pedagogical criteria is two-fold. First, it is to some extent connected with the metadata (the definition of creator, ideal target group, religion, language and country) and technical usability (graphical user interface colours and symbols) issues. Second, cultural sensitivity is very strongly latent variable by nature, and hence it is not trivially operationalised into questions that end-user is able to answer. 

Finally, if we try to verbalise what the pedagogical usability criteria aim to address, we may ask: “Does the system, and/or learning material it contains, make it possible for the student and the teacher to achieve their goals?” The important thing to acknowledge is that the dimensions of technical and pedagogical usability (operationalised dimensions) might correlate and thus describe (i.e., in factor analysis language “load on”) both usability and utility (latent dimensions). From the perspective of the user/learner, the number of things to be memorised and the feedback required are examples of these kinds of common components between the user and the system, and between the user and the contents of the learning material. From the viewpoint of usability, the system works well in user terms when the user does not have to resort to any help and is able to control the program, and not the other way around. The number of things to be memorised is, in terms of utility, related to the functioning of the learning material because this will be minimized if the studied subject matter is presented in manageable sections that make sense to the users. There may be a certain degree of tension between some of the criteria. For example, usability is enhanced if the number of things to be remembered and errors are kept to a minimum, but for the utility of the learning material it is desirable for the user to remember the core set of functions so that they can be used readily without prompting. Another example is authenticity and usability. Using real representations from the outside world in the user interface may not be as efficient as a metaphoric design. In the design of learning materials, the aim is often to correspond closely with the real world so that it will be easier for users to link new knowledge to earlier knowledge structures. On the other hand, learning materials intended to inculcate a certain skill by repetition can be weak in their real-world correspondence but still produce good learning results, particularly if the skill being learned is not derived directly from the real world, either.

The criteria described here are not the first attempt to provide an analysis tool for the evaluation of the usability of digital learning material; both its structure and content are to great extend based on previous research. However, the criteria and research that led to its creation, have the following original features that bring added value for research on the pedagogical usability of digital learning material: 1) a multidisciplinary approach involving the fields of both computer technology and education, 2) a review of previous research, 3) the development of the criteria on the basis of systematic empirical research, performed with the help of real users; and 4) a multiple choice inventory that was developed on the basis of the criteria and can be used independently or as part of the computer application.