Methodology

ByChristian Friedl

Empirical foundation – Methodology and Evaluation design of the MOOC BOOK

Download Links:

  • (BizMOOC R4.2) Joint Evaluation Report on Three Pilot MOOCs [Link]
  • (BizMOOC R4.2a) Evaluation Report on Pilot MOOC1 – Learning to Learn [Link]
  • (BizMOOC R4.2b) Evaluation Report on Pilot MOOC2 – Innovation, creativity & problem-solving [Link]
  • (BizMOOC R4.2c) Evaluation Report on Pilot MOOC 3 – Intrapreneurship: Make Your Business Great Again [Link]
  • (BizMOOC R4.3) Lessons Learnt, Recommendations and Good Practice from Pilot MOOCs [Link]

In 2016, the BizMOOC project set out to explore the applicability of MOOCs for the world of business. To identify potential, barriers and key questions related to MOOCs in the business context, a first research phase was conducted summarizing the outcomes of a survey with 1,193 learners, interviews with 106 business and higher education experts and self-produced 14 discussion papers on MOOC hot topics. The developed guidelines and findings have been translated into the MOOC BOOK 1.0 – an online resource for a broader uptake of MOOCs for labour-market relevant use by institutions and individuals.

The next step was to build, test and evaluate three Pilot MOOCs catering a business audience. The aim of this endeavour was to test three different MOOC approaches for its uptake in the world of business and to apply the theoretical research outcomes of the MOOC BOOK 1.0 (published in February 2017) in a practical setting. Consequently, the pilot MOOCs differ in terms of didactics, resources, specific target groups, experience of the development team, platform etc. The three MOOC topics were chosen in line with two European Lifelong Learning key competences (Learning To Learn, and Sense of Initiative/Entrepreneurship) to underline labour-market relevance and contribute to the European agenda. This valuable exercise shall allow the collection of lessons learnt, recommendations and good practice to support a better exploitation of MOOCs for improving labour-market oriented skills. Thus, the findings based on the MOOC Evaluation report (R4.3) are fed into the MOOC BOOK 2.0 (this website).

The MOOC Evaluation report (R4.2) is based on the extensive evaluation scheme surrounding the three Pilot MOOCs. A sequential mixed-methods design with quantitative and qualitative aspects was applied. This included a pre- and post-course feedback by 55 experts and 838/553 participants, plus a course design review by 2 external evaluators and the MOOC production teams as a peer review and with the whole process being overseen by an external quality assurance board (see Figure below). The MOOC Evaluation report (R4.2) presents the meta-analysis of the individual reports delivered by the three MOOC teams on their courses (available as separate reports R4.2a, R4.2b, R4.2c; in total more than 300 pages) and serves as basis for the here presented lessons learnt, recommendations and good practice (Insights – R4.3). It therefore acts as pivotal element between individual analysis and overall recommendations developed as a core result of the project.

Findings suggest that the implementation and evaluation of the Pilot MOOCs was truly enriching. With more than 5.000 enrolments, the overall target KPIs were (over)-reached, the MOOC completion rates are above average, and the qualitative feedback is highly positive as well from learners and experts. The high impact of the Pilot MOOCs is especially remarkable, as no major global MOOC platforms have been used, but less known, second-tier EU-based MOOC platforms. In addition, the project consortium design included a diverse setting of MOOC experience and non-experienced business and industry partners to allow direct target group feedback and a comprehensive collection of lesson learnt from different perspectives.

On an individual MOOC level, the lower-budget MOOCs (MOOC1 and MOOC2) have resulted in a positive evaluation. MOOC1 was provided as a self-paced course on a low-budget open MOOC platform (OpenLearn Create) to pilot test if also institutions with limited resources can take part in the movement and run successful MOOCs. The project design allowed the direct inclusion of MOOC inexperienced institutions with limited budgets and resulted in valuable learnings what can be achieved, and where the limits are. Especially in terms of outreach, MOOC1 was by far the smallest MOOC of the three MOOCs, but received a lot of positive feedback from those that enrolled.

MOOC2 was carried out on a primarily Spanish-speaking platform (UniMOOC), opening up to an English-speaking audience for the first time. It therefore brought in many additional learners from Latin American countries, but also reached many learners from non-Spanish speaking countries. With this MOOC, we also wanted to test how existing open educational resources can be used and to what extent.

MOOC3 was carried out by the most experienced team, and relatively higher resource-investment (yet still low budget compared to leading MOOCs). This MOOC had the highest enrolment and completion rates and an outstanding positive evaluation. The huge interest might be partially explained by the niche topic (recent topic, but no MOOC on offer yet to this topic) touched by the course and by the more specific target group.

The applied quality and evaluation scheme is unique as it combines existing quality checklists and surveys with qualitative reviews. Input is collected during development phase and by pre- and post-course evaluation by both experts and MOOC-participants, next to statistics data during the MOOC. This quality and evaluation scheme has proven to be effective and applicable for other cross-sectorial/-institutional teams. The complete evaluation process including the tools and templates used are available in the section “Learn More“. The obtained results resulted in a long list of Lessons Learnt and enabled the project consortium to identify the important general recommendations, which can contribute to increasing the use of MOOCs and upgrading the quality of new MOOCs.

Through testing three different formats and designs of MOOCs this project was able to deliver important findings related to how can MOOCs better address the needs for the world of business. The most important factors include:

  • Clear identification of target audience(s) and its needs, motivations and limitations and learning outcomes;
  • The use of a reliable platform with a strong outreach and effective, proven to work learning design principles for course development in international collaboration;
  • Alignment of design, goals and content, as well as learning outcomes, to target group needs and expectation, as well as time preferences (ensuring an optimal „fit“ between these variables);
  • A validated quality assurance model for MOOC development and evaluation of MOOCs in cross-sectoral international teams;
  • Inclusion of local, context specific examples and content;
  • Considering multiple language versions of the course and assets;
  • Aligning promotion and course topics to business schedules and company competence frameworks.

The MOOC Evaluation Report (R4.2) concludes that MOOCs present a potentially valuable opportunity for the world of business to upgrade employee skills and develop organizational knowledge. However, the existing possibilities of MOOCs for the world of business are exploited only to a low extent. An important finding is that individuals do uptake MOOCs on their own, organization rest unaware or reserved. A vast majority of the participants of the three Pilot MOOCs found the course to be very highly or highly applicable in their everyday professional life and a strong majority was highly satisfied with the course admitting it enhanced their skills. These findings do confirm that MOOCs can effectively address the needs of the business world.

To this end, the multiple findings from the three Pilot MOOCs can be clustered in four domains: (1) Promotion; (2) Course Design; (3) Participation and Completion; and (4) MOOCs for the world of business. The extensive findings in those four domains have been furthered and used to elaborate 50 lessons learnt, 25 key recommendations and 20 good practices. They are presented in this chapter (use the links under “Insights”) and as well in a separate download-able Report in PDF Format (R4.3 Lessons Learnt, Recommendations and Good Practice from Pilot MOOCs).

 

Main Authors: Christian Friedl (FH JOANNEUM), Agnieszka Żur (Cracow University of Economics), Beck Pitt (Open University UK), Darco Jansen (EADTU) with support from the whole BizMOOC team.