As of August 1st, 2008, the Edutech web site will no longer be updated. Edutech was funded by the Swiss Virtual Campus programme, which ended on July 31st, 2008. Some activities will be taken over by the e-Learning Services group at the SWITCH foundation.

About the Evaluation Method

Learning Management Systems (also Web Based Course Platforms) are complex systems that offer an overwhelming amount of functions. A platform has a broad range of users (students, authors, tutors, administrators) and each user group has its specific requirements. It is difficult, even impossible, to establish a formal list of requirements. An evaluation is therefore an extremely complex and expensive task.

The classical evaluation method uses large tables of scores with weighted criteria. Recent studies have discussed the controversies concerning this classical method.

Here is a list of edutech's methods, plus the methods of two other recent evaluations:

Baumgartner et al. Method (ref)
Qualitative weighting and summation

Schulmeister Method (ref)
Mixture of
  • practical tests
  • peer evaluation (gather opinions of learning management systems experts)
  • criteria based evaluation (based on white papers and other information supplied by the software producer)

edutech Method
  • Keep the number of criteria as small as possible (20 main criteria)
  • Consider only the most important criteria ("killer criteria")
  • Try to be as objective as possible, yet purposely include subjective ratings
  • Do not attempt to calculate one final overall score
  • Try out every feature - do not trust product whitepapers and other promises
  • What are the experiences of other users of the product? Interview experienced users who are in a similar context

About the Evaluated Products

This evaluation project has very limited resources and a tight timeframe. It is therefore impossible to consider any product for a detailed evaluation. edutech decided to make the evaluation process more efficient by strictly preselecting the products. Other large scale evaluations published in 2002 were a great help.

These were the conditions used to preselect products:

Not in Detail Evaluated Products

Ariadne
Ilias
Olat
LearningSpace
Medit

What Does Edutech Finally Recommend?

Based on the evaluation results, Edutech makes the following recommendations on March 29th '03:

1) Acquire a 2 years license for WebCT Vista now

WebCT has a large user base in Swiss higher education institutions. The redesigned Vista version allows to set up a large scale infrastructure. The modern application server architecture has the potential to serve thousands of students at an arbitrary number of institutions. Although its functional extensibility is limited, it is well accepted in development projects like the SVC due to its ease of use, efficient content authoring and large number of didactically useful tools.

Edutech suggests to set up a server until summer (fall) '03 and make it then available for projects of the SVC and other Swiss higher education institutions. Since the planned license would include a limited number of seats, it would be necessary to define rules specifying which projects can use the server.

2) Take a decision for an extensible platform later this year

None of the evaluated extensible platforms is fully convincing. However, considerable developments have been announced for Globalteach and IBT-Server, which should be available in summer '03. Also, further platforms should be considered and evaluated, that could not be taken into account until now: IBM/Lotus Learning Space, OLAT, Oracle iLearn and possibly others (e.g. open-source platforms).

References