What online work environment might best

suit you?

A brief summary of the project stages

• Starting point

– Tens, if not hundreds, of different programmes are available

– A quick overview resulted in the selection of the most potential

• Preliminary study

– The softwares’ basics investigated

– 23 different software selected for this stage

– The most potential of these were selected for a broader examination, based on opinions of the evaluation group and client

• A broader study

– A detailed account of the eight (8) most potential software

– Background study, testing and written report

– Applicability to various scenarios defined, thereby providing recommendations

Methodology

• Study based on:

– comparison of features based on evaluation criteria

– observations of evaluation group

– collaboration between partner network and suppliers of online meeting and conference software

– discussions with colleagues using various online meeting and conference programmes

– internet sources and discussions between client and evaluation group.

• The most important method was, however, practical testing individually and collectively, and recording observations

• Simulation of all situations was not completely successful (e.g. 100+ webinar)

Evaluation criteria

• Comparison of technical features formed the starting point for the evaluation

• Main categories of evaluation criteria – Technical evaluation– Administrative evaluation– Interactivity evaluation– Functionality evaluation– Cost evaluation

• The programmes were ranked under each feature• Summary table

– Lists all important features generally• Scenario tables

– Specific features emphasised in specific scenarios– Features nonessential to a scenario were not evaluated – Certain features were emphasised more than others using a weighting coefficient

Evaluation scenarios

• Pair work/small groups (2 – 3 individuals)– Two or three people usually in different locations physically engage in dialogue on,

for example, issues relating to work or study • Meeting/negotiation (4 – 10 individuals)

– There are usually several participants and various documents to deal with – The recording of, e.g., messages and observations written in the space is

emphasised– Ease of pacing and allocating participants’ turns to speak is also emphasised

• Teaching use, lecture, training (11-20 individuals)– Interaction and a clear user interface facilitate comprehension in teaching uses– Diverse tools enhance the learning process– Flexible and quick management of user rights is also of vital importance

• Meeting/conference (100-> individuals)– In large meetings the programme’s role diminishes as various technical

requirements are emphasised. However, it must be possible to follow the conference and elements of interaction are highlighted.

Comparison results

• A 44-page report on the results of the comparison was compiled. A detailed evaluation table is attached to the report

• The results of the general evaluation and scenario specific evaluations are summarised here

• Placings are based solely on comparison of features and their ranking

Average of overall ranking (general evaluation + scenarios)

General evaluation

Scenario specific evaluation

• Scenario 1 (small group/pair work, 2-3 individuals)

Scenario specific evaluation

• Scenario 2 (meetings, 4-10 individuals)

Scenario specific evaluation

• Scenario 3 (teaching situations, 11-20 individuals)

Scenario specific evaluation

• Scenario 4 (seminars, 100+)

Programme rankings

Testing

• In testing situations the evaluation group primarily made observations at a general level of the programmes for each scenario

• There were five testers altogether, from seven different machines

• Tests were conducted using the following operating system and browser groupings:– Windows XP Pro, Firefox– Windows XP Pro, Internet Explorer 6– Windows Vista Business, Firefox– Mac OSX, Safari– Ubuntu Linux, Firefox

• The aim was to test all essential functions and how well they work in different groupings

Collection of data and experience

• The evaluation group’s previous experiences of the various programmes formed to a large extent the basis of comparison

• We collected comments from as broad a group as possible through partner networks

• We familiarised ourselves with existing studies, test reports and material from suppliers

• In the case of some programmes we were in contact with programme suppliers (ACP/Humac, Elluminate) or active users (iLinc/Kemi-Tornion amk, WebEx/TKK, WebEx/Blackboard) in order to participate in guided presentations or authentic situations and to accrue experience

• In addition to the evaluation group’s views, scenario specific emphases also noted the emphases of nine other individuals

Result reliability

• For the sake of comparison and to ensure reliability of the results, the evaluation group also estimated rankings

• A comparison of the general and scenario specific evaluations with the evaluation group’s general estimations shows some discrepancies – In the small group scenario Dimdim and OpenMeetings– In meeting situations, the above and also GoToMeeting and

MS OfficeLive– In teaching situations Elluminate

• Perhaps the greatest divergence in the evaluation group’s placings was in the general evaluation. Everyone seemed to have their own favourite, and discrepancies with the feature comparison table were the greatest

Conclusions

• Generally, the basic features of all the programmes are similar and they are applicable to all user situations. Ranking the programmes was not easy.

• Adobe Connect Pro was the clear winner, with iLinc in second place.

• There were potential challengers in the group, especially in certain scenarios.

• It was interesting to note the open source Openmeeting’s high ranking in the comparison, despite its having just begun its development.

• Operating system dependency was a general problem in the majority of programmes.

• An interesting deficiency seemed to be evident in all, apart from ACP and Office Live: the evaluation group considered shared notes one of the most important features and the other programmes lacked this.

Further information:

This study is the Summer University of Häme’s subproject”Virtual meeting environments and Online ITK’’ in the Open Networks for Learning project.The study was sub-contracted to Mediamaisteri Group Oyj.Summer University of HämeHanne MurtoHanne.murto@hame.fi

The Association of Finnish eLearning Centreinfo@eoppimiskeskus.fi

Mediamaisteri Group Oyjmarko.makila@mediamaisteri.com