Quality is related to the roles involved in the process and is much related to the effectiveness of the process. On a tutorial note; the following explanation of the usage scenario is in reverse order, starting from the usage stage, as the actions are built upon another.

Use
The quality of use of a learning resource is related to what the learner(s) eventually learn. This is determined by the intrinsic quality of the learning content itself, the methodology (e.g. pedagogy, learning resource construction methods), and the people delivering and/or using the learning resource.

Aspects of the learning resource itself A learning resource will be used most effectively if it fits the desired learning situation as defined by the person (e.g. teacher) or system (e.g. LMS) that is supporting the learning process. Some typical quality characteristics are [Benigno 2001] :
— Accuracy; reliable, valid and authoritative.
— Clarity; a clear link between objectives and the content.
— Appropriateness; information for the intended learners’ level in appropriate vocabulary, language and concepts.
— Completeness; information related to self-contained activities, material required, prerequisites, information for obtaining related resources, assessment criteria, link to quality indicators.
— Motivation; engage the learner with interest and satisfaction.
— Composition and organisation; easy to use both for teacher and pupil.

Pedagogy
Different pedagogical and methodological teaching and learning scenarios should be considered, such as independent learning, group learning, or the learning resource used by a teacher in a classroom. The learning experience will not solely depend on the quality of the learning resource, but also on how effectively it is delivered and used.

The pedagogical affordance (i.e. the pedagogical clue to the use of an object) of a learning resource can depend upon the following:
— the metadata containing clues of how to use the learning resource
— added by the teacher in the integration and delivery processes
— built-in, for instance sequencing following IMS Learning Design
— within a “teaching tip” or “teaching resource”, i.e. other resources or comments may explain how other teachers have used it.

Tools and infrastructure
A minimum ICT infrastructure is required for elearning, for instance introduction of a video projector connected to a computer or the entry of personal hand-held and mobile devices contribute to the infrastructure set-up and its use [Curtis 2002].

Technical requirements of a learning resource are usually explained in the metadata description of the learning resources. Platform dependence or required software can cause major headaches to teachers who would like to use digital resources but lack the knowledge and skills. Novel ways to avoid these situations should be investigated, for example, before downloading, the resource system could make a check of relevant software (operating system) availability.

People
The learning experience is influenced by the way the learning resource is delivered, i.e. on the quality of delivery by the teacher. In this context, professional pre-service and continuous training on the pedagogical use of learning resources is key.

Integration, Repurpose & Reuse
Technical integration
A learning resource will have a specific format often expressed as a MIME type. Typically, one would be concerned with the hardware infrastructure, the players or viewers required, and the integration might involve some form of conversion to fit the technical environment.

Repurpose, adaptation and pedagogical integration
Pedagogical integration is effective to the extent that the learning resource can be fitted into the overall pedagogical approach. This may involve some adaptation of the learning resource, sometimes called ‘repurposing’. The repurposing may be for a different audience, different language, cultural, or pedagogical setting.

According to CELEBRATE case studies [Ilomäki 2004], teachers actually used learning resources quite independently of the intended pedagogical approach of the learning resource. Thus, possibilities for repurposing should be an integral part of learning resources’ design.

Reuse
Reuse is effective to the extent that a learning resource or any part of it can be fit into another learning resource or another context for learning. Allowing re-use of learning resources has economic consequences, but also adds to the quality, because often improvements are suggested and implemented in this way. The possibility to reand dis-aggregate the resource enhances the learner’s ability to construct her own knowledge and actively reflect upon the learning experience, as well as share it with other learners and users, who are natural parts of any learning experience.

Resolution & Obtaining
Obtaining
Obtaining a learning resource is effective to the extent that the learning resource has become available to the user. There are three basic models:
— The learning resource is played remotely.
— The learning resource is downloaded and played under the control of a shared system such as LMS, LCMS, local server.
— The learning resource is downloaded to a personal computer.

Resolution
Resolution can involve the identification of the user or user role by an identity management system, the generation of an agreement based on the rights to use these resources, and the enforcement of the agreement.

Currently, a number of Digital Rights Expression Languages(1) and schemes exist, as well as identity management schemes. The application of digital rights ranges from closed and commercial content to open and free content. For the latter, the Creative Commons Licence(2) has gained popularity as a simple means to ensure that the copyright is respected, as well as to send a clear message to others about the terms of re-use.

Discover & Evaluate
Discovery and evaluation is effective to the extent that it results in the identification of all relevant learning resources. It is often the case that a user (teacher, learner) is looking for a single learning resource that suits her best. This has been the reason for introducing better discovery and evaluation techniques. The more precise the discovery is, the less effort needs to be made on evaluation.

Discovery
Many techniques can be used, including
— Browsing
— Searching
— Recommendations and advisory models such as based on previous actions, user modelling and social networks
— Agent-based systems
— Selective dissemination of information

Advanced result presentation schemes may use two- (or three-) dimensional presentations or zoomable tree structures, clusters, of different colours or font size, etc. to indicate relevance or another criterion [Klerkx 2004].

Evaluation
The user may have many criteria for evaluation and typically they are related to the activities following evaluation (see figure 1). In order to come to a successful evaluation, the teacher or learner should have access to the learning resource itself, a learning resource preview, and metadata. Metadata in the broad sense include all annotations, comments, evaluation, and feedback from other users. More and more, the feedback, reviews and evaluations, either by boards of experts or other users, have become a valuable source of information on a resource’s usability.

One of the major issues in the area of discovery and evaluation is semantic interoperability. Does the user understand the information in the same
way as the person who provided this information?

=======
References
Curtis Michael, Kathleen Luchini, William Bobrowsky, Chris Quintana, Elliot Soloway (2002), Handheld Use in K-12: A Descriptive Account, Proceedings of the IEEE International Workshop on Wireless and Mobile Technologies in Education.

DCMI Metadata Terms, http://dublincore.org/documents/dcmi-terms/
IEEE Standard for Learning Object Metadata, 1484.12.1-2002, http://ltsc.ieee.org/wg12/par1484-12-1.html

European Quality Handbook (2006) CEDEFOP

Ilomäki et al (2004), Learning Objects in classroom settings: A report of 13 case studies conducted in Finland, France, Hungary, Ireland, and UK. CELEBRATE deliverable 7. case 5 and 8

Klerkx Joris, Duval Erik & Meire Michael (2004). Using Information Visualization for Accessing Learning Object Repositories, Information Visualisation, Eighth International Conference on (IV 04). London, England, pp. 465-470. http://csdl.computer.org/comp/proceedings/iv/2004/2177
/00/21770465abs.htm

Van Assche Frans and Nirhamo Lassi (2003), Celebrate LOM Application Profile, version 1.1. http://celebrate.eun.org/docs/

Vincenza Benigno, Günther Dippe, and Vuorikari Riina (2001), Recommendation on Quality Assurance and Selection of resources for the ETB network, deliverable D3.2 of the European Treasury Browser project.
=======

Semantic interoperability seeks to ensure that this is indeed the case; the challenges are in bridging of differences in language, culture, vocabularies, etc.

It is only when the evaluation is satisfactory that one can say that a user has found a learning resource. Since discovery and evaluation are usually subjective, more advanced techniques, such as agents, will also use information about the user and her preferences.

Approve and publish
There are explicit and implicit approval processes before publishing whose intention is to assure the high standards and quality of the content provided. Quality, conformance and other labels are used fairly extensively in e-learning products in general, and nowadays, to a certain extent, also within learning resources. Some countries, institutions and consortiums have labels approving, for example, technical compliance, the curriculum compliance or appropriateness of the resources. Many learning platforms and learning resource producers, for example, note their compliance to IMS specifications(3).

Describe
The aim of describing a learning resource is to facilitate the activities of the usage scenario (see figure 1). The description is effective to the extent that it indeed helps in the discovery, the evaluation, obtaining, modifying, integrating, and using of the resource. The description may be human readable and/or machine readable.
Description is more effective if it uses a model that is shared between the information providers, the readers of the information and the toolsmodels such as the LOM4 and DC5 exist. A further specification tailored to a user community is elaborated in a so-called application profile [Van Assche 2003].

Create
The production of learning resources, whether it was creation from scratch or aggregating from already existing components, is the crucial stage where aspects of interoperability, accessibility, transferability, repurposing and re-use are determined. The qualities that have the most effect on the overall quality are based on the usefulness of the content for the learning experience (desired learning outcomes), possibilities to reuse the content or its components (cost-effectiveness), and the pedagogical affordance of the resources.

Retract and Delete
Retract and delete are effective to the extent that all obsolete material is retracted/deleted and useful material is not retracted/deleted.

Notes:
1 IEEE, Learning Technology Standards Committee, WG4: Digital Rights Expression Language
http://ltsc.ieee.org/wg4/index.html.
2 Creative Commons, http://www.creativecommons.org
3 IMS http://www.imsglobal.org
4 Learning Object Metadata. http://ltsc.ieee.org/wg12/
5 Dublin Core Metadata Initiative. http://dublincore.org/