Authors: Anastasia Sfiri (FH JOANNEUM), Karolina Pietkiewicz (The National Unions of Students in Europe) and Darco Jansen (EADTU)
Author’s Note: This chapter was written in 2016 and has been edited for minor updates, but does not feature analysis based on new data since 2016.
This paper aims to present and discuss MOOC initiatives in Higher Education and in the business sector, as well as the distribution of MOOC learners in EU28 on the basis of the analysis of existing secondary data (both literature and Internet sources).
The first section of the paper elaborates on MOOCs in Higher Education, establishes a definition of MOOCs for the purpose of the project and analyses the information gathered from the most commonly known Web portals on MOOCs with regard to i.a. the geographical and thematic distribution.
Subsequently it provides an overview of the MOOC platforms, providers and partnerships in Europe. The second section tackles the usage of MOOCs in the business sector, addressing the issues of upskilling and re-skilling for the purpose of pursuing professional careers from both the employees’ and employers’ perspective.
The third section describes the MOOC learners. Due to the limitations resulting from the lack of data, the researchers focused on learners’ behavioural patterns and statistical information, instead of presenting the complete learners’ distribution in EU countries.
The final section constitutes a summary of the findings and sets the topics to be addressed and considered in the future.
MOOCs show a spectacular growth over the last 5 years and there is a wide range of portals that were designed as search engines to help learners and companies in finding the right course for their specific needs. They provide an overview of the current state of MOOCs worldwide. A closer look makes apparent that reliable data is hard to find since none of the portals can be considered to be complete. Another issue to be addressed is what these portals define to be a MOOC.
For the purpose of this project a MOOC has been defined as follows: MOOCs are online courses designed for large numbers of participants that can be accessed by anyone anywhere as long as they have an internet connection, are open to everyone without entry qualifications and offer a full/complete course experience online for free. (http://www.openuped.eu/images/docs/Definition_Massive_Open_Online_Courses.pdf)
This section of the paper aims to present and discuss web resources that provide an overview of MOOC initiatives, i.e. MOOC platforms, providers and partnerships, in Higher Education. A clear distinction between platforms, providers and partnerships will be provided through a definition at the very beginning of respective sections in further parts of this paper.
Also, further in this paper lists of MOOC platforms, providers and partnerships are presented, giving an opportunity to view the vast variety of them and search for actual MOOC courses along with some of the data on MOOC participants that can be found in literature
As explained on the website “The European Commission launched Open Education Europa in September 2013 as part of the Opening up Education initiative to provide a single gateway” not only to European OER as they state in the “about this portal” page, but also to MOOCs and other Open Education initiatives. One of the main goals of the portal is to present this information to learners, teachers and researchers. Open Education Europa is funded through the Commission’s Erasmus+ programme.
In the beginning of 2016 the scoreboard listed 1712 MOOCs offered in EU28 countries by 305 institutions. Most MOOCs are offered in the United Kingdom (72), followed by France (39) and Spain (37). Since then, there has not been an updated map published – however, it is highly likely that the numbers have risen considerably.
Screenshot 2.1.: Distribution of MOOCs per European country (18.02.2016): http://www.openeducationeuropa.eu/en/european_scoreboard_moocs
Most popular area of study is science and technology (414), followed by the social sciences (317), applied sciences (254) and business (232).
Screenshot 2.2.: Distribution of MOOCs per subject (18.02.2016): http://www.openeducationeuropa.eu/en/european_scoreboard_moocs
This database, however, does not allow for a clear picture of MOOCs state of art in Europe, since it does not list eight European countries – Bulgaria, Czech Rep., Greece, Hungary, Latvia, Poland, Slovakia, even though, as our research has shown, some of those have MOOC initiatives (see: e.g. Screenshot 1.1 for Poland and Bulgaria or http://www.memooc.hu for Hungary) (Search engine on http://openeducationeuropa.eu/en/find accessed in January 2016). A second problematical aspect is that the scoreboard lists online courses and courses on demand along with the MOOC courses and there is no specific definition of the MOOC courses included. A third problem of the environment is that this platform has not been updated recently.
ClassCentral curates international MOOC listings and reviews from students who have taken MOOCs. Their goal is to help users of their system to find courses they want to take from reputable MOOC providers. They report that they “do not play favorites”, users will find MOOCs of all sorts of subjects and from many different reputable universities and MOOC providers, not just the “biggest” ones.
ClassCentral searches providers and educational institutions from all over the world that are offering MOOCs or gearing up to offer MOOCs and aggregates MOOCs from all these different sources. It offers two ways to find new MOOCs, browsing by subject or using a search. They also offer a MOOC Tracker, which is a tool that helps users track MOOCs that they are interested in. The MOOC tracker notifies users when a course in their catalog is about to start or if the course is offered again. It also searches for keywords and alerts users when new courses that match those keywords are announced or are about to start.
ClassCentral also provides a yearly analysis report of MOOCs listed at their portal (https://www.class-central.com/report/moocs-2015-stats/), (https://www.class-central.com/report/5-mooc-trends-of-2015/). According to the analysis report for 2015 ClassCentral shows that compared to last year, the growth rate for courses has slightly slowed down from 100% to 75%. Around 1,800 new courses were announced in 2015, taking the total number of courses announced since the inception of MOOCs to 4,200
Screenshot 2.3.: Growth of MOOCs, source: ClassCentral: https://www.class-central.com/report/moocs-2015-stats/, acessed on 25.02.2016.
Since 2015 there has been a distinct focus on monetization by MOOC providers. This focus has led to an increase in the percentage of courses focusing on the field of technology and business. The percentage of Computer Science and Programming courses grew more than 10%. This growth in technical and business courses has led to a decrease in the humanities and social science courses, but overall there is still a healthy balance of technical and non-technical courses.
Screenshot 2.4.: Course distribution of subjects, source: ClassCentral: https://www.class-central.com/report/moocs-2015-stats/, acessed on 25.02.2016.
Class Central also reports that MOOCs are currently being offered in 16 different languages, including Basque and Estonian. After English, Spanish and French are the biggest languages in which courses are offered.
The portal rates courses according to reviews written. Here are the 10 by ClassCentral highest ranked courses from 2015:
An advantage of Class central is that it separates self-paced courses from MOOCs that are defined as having a start and end date.
Screenshot 2.5.: Categories of the courses, source: ClassCentral: https://www.class-central.com/, acessed on 25.02.2016.
A problem with the course statistics is the length of the MOOCs. Class central reports that since creators are adapting their courses to better fit students’ schedules, which means that semester-long courses are being broken down into smaller courses. This makes it difficult to capture the growth of the MOOC space in a total number of courses. There are some courses that can be finished within a couple of hours (https://www.class-central.com/report/moocs-2015-stats/, https://www.class-central.com/report/5-mooc-trends-of-2015/) and they do not fit in our definition of MOOC that has to offer a full/complete course experience to the learners (at least 25 hours, which equal 1 ECTS).
The philosophy of the ClassCentral claims to focus on meeting the learners’ expectations by providing them with prestigious MOOCs, i.e. courses affiliated to prestigious institutions. The criterion ‘prestige’ is however vague and, more importantly, subjective. It has lead to a conclusion that the platform has not created a set of objective criteria that would facilitate selection of MOOCs to feed in the platforms offer in an unbiased manner. Consequently, there remains a question to answer about the quality of the courses and their learning outcomes.
MOOC List is an aggregator (directory) of MOOCs offered by different providers. It is a website where users can find MOOCs offered by universities and other entities around the world. The site works like a search engine allowing users to search by multiple criteria, providers, university/entity, thematic categories, length, estimated effort, language, subtitles, country, type of exam, type of certificate and tags.
As a requirement “MOOC List” only lists courses with open access (free). The course could have the option to pay for the certificate or exam, but the access to the course should be free. The directory includes self- paced courses as well as MOOCs that are very short in duration (one week long).
The MOOC List site offers a service called “My MOOC List”. The service is the way users can bookmark their favorite MOOCs and it requires registration. Registered users can also rate MOOCs they participate in.
At the time of this research “MOOC List” lists 83 platforms as providers of MOOCs, offered by more than 900 universities/entities. In the category of universities/entities there are also companies listed, for example a search of SAP shows 48 different MOOCs offered by the company’s own platform https://open.sap.com/.
Founded in 2006 as India’s first campus network, Minglebox.com has evolved into India’s leading education portal with a user base of 4mn+ students and young professionals across 250 cities in India. It is the preferred destination of the multitudes of students in India seeking information on higher education. Minglebox.com lists 29 MOOC providers like coursera and Edx, but it also includes online training companies that offer self-paced and/or commercial online courses.
A MOOC provider is an institution that creates and publishes a MOOC. In many cases these are Higher Education Institutions (HEI), but MOOCs are also offered by organisations like the World Bank, the International Olympic Committee or the software company SAP.
While some of the HEIs outsource their MOOCs to the more recognisable, with regard to this area of education and training, partners that offer MOOC platforms, other prefer to administer their own platforms. Particularly in Europe, HEIs are increasingly choosing to launch their own platforms.
Currently, several different scenarios are available for MOOC production and delivery:
In the first two scenarios, a central MOOC platform is available for the development and delivery of MOOCs. Additional design and implementation services are offered to the academic staff of educational institutions. An overview of these central MOOc platforms are provided in section 2.3.
In the collaborative and decentralised scenario, institutions of regional hub partners have their own MOOC platform, and those partners share different scalable services in the development of MOOCs and in their uptake by society. Some examples of this scenario are listed in section 2.4.
A MOOC Platform is the hardware and software needed to publish and run a MOOC. A MOOC platform can be run by the institution itself out outsourced to the MOOC platforms like EdX, Coursera, Udacity, Iversity and Futurelearn.
Screenshot 3.1.: ClassCentral, course distribution by providers from 2015: https://www.class-central.com/report/moocs-2015-stats/, acessed on 25.02.2016.
This subsection offers a brief overview of existing MOOC platforms in Europe from 2016.
AFTP is an open online course platform dedicated to the free online delivery of a full curriculum of undergraduate and graduate courses in economics and finance. The platform offers certification of completion upon successful completion of a course and passing of examinations.
Founded by Harvard University and MIT in 2012, edX is an online learning destination and MOOC provider, offering high-quality courses from the universities and institutions to learners everywhere. They are listing 90 global partners, many European ones among them. The platform is a nonprofit and open source MOOC provider.
The Open edX (https://open.edx.org/) platform on the other hand, is a free and open source course management system (CMS) that was originally developed by edX. Originally founded from the partners Harvard and MIT edX offers interactive online courses and MOOCs from universities and institutions. The Open edX platform is used to host Massive Open Online Courses (MOOCs) as well as smaller classes and training modules.
Canvas was founded in 2008 and by 2016 had more than 700 employees. It hosts more than 1,600 universities, school districts and institutions around the world. Canvas is composed as an LMS which is able to be used both with MOOCs and classic teaching concepts using blended learning.
Futurelearn is a platform that brings together courses offered by 144 partners from around the world among them some well known UK and international universities and a range of internationally renowned organisations. It also offers programmes, which basically consist of several courses provided by a HEI that lead to (after settling a fee) a certification of a particular university. The platform is owned by the Open University. According to the data on the website, more than 8 mln people have joined MOOCs of FutureLearn since 2013.
France Université Numerique (FUN) is a platform conducted only in French and operated by the Public Interest Group (GIP) FUN-MOOC. Class central reports that “FUN is a MOOC platform made available to institutions of French higher education and their academic partners worldwide. Launched by the Ministry of Higher Education and Research in October 2013, this initiative aims to bring together projects in French universities and schools to give international visibility, and enable all public access to various courses and quality anywhere in the world.” (https://www.class-central.com/provider/fun)
Open education is powered by Blackboard. They work together with around 70 different university all over the world and offer a very wide range of different courses. Using the blackboard technology all courses are held in responsive design, i.e. the environments is adaptable to teachers’ and learners’ needs and wishes.
Coursera works with top universities and organizations to make some of their courses available online, and offers courses in many subjects, including: physics, engineering, humanities, medicine, biology, social sciences, mathematics, business, computer science, digital marketing, data science, and other subjects.
Copernicus college launched in 2015 and advertises itself as the first polish MOOC platform. This site runs by Copernicus Center for Interdisciplinary Studies (Copernicus College) and offers a significant number of MOOCs in natural sciences, mostly astrophysics, and also philosophy.By 2016 they had created 12 MOOCs and they are all in open access (free of charge) and uncertified. The website is user friendly, complemented by free recorded guest lectures and the first e-book this institution has issued. This portal receives direct government support.
Other MOOC initiatives in Poland are scattered. Two Warsaw universities created 12 courses which were not free of charge and appear to be inactive. Another portal is being created by the Foundation of Polish Science, but still at incubation phase.
MeMOOC is a Hungarian MOOC portal built by the University of Miskolc using the openEdx environment. They offer their own courses that may lead to credits and certificates.
iversity, launched since 2011 is an European digital learning platform for higher education and professional development. iversity works together with 41 universities and provides at around 60 courses.
Another scenario for offering MOOCs includes a collaborative-decentralised approach, which means that each institution or hub-partner has its own MOOC platform. Each MOOC is by definition multilingual,offering various language versions and offers the possibility of contextualised learning through mobile technologies and gamification.
This subsection offers a brief overview of existing MOOC partnerships in Europe from 2016.
Elearning Communication Open-Data (ECO) is a European initiative that offers an open access to MOOCs developed in six different languages with aims to wider access to education and improve quality of learning and teaching. It has 22 partners from all over Europe that are educational organisations, MOOC providers or universities. Presently, ECO uses a collaborative-decentralised scenario — i.e., each institution or hub-partner has its own MOOC platform, and ECO supports multilingual (providing access in different languages) and offers the possibility of contextualised learning through mobile technologies and gamification.
Academic Earth was launched in 2009 and works together with various international universities such as Yale, MIT or Oxford. The website provides only an overview of MOOCs available at the partner universities platforms. Additionally, it offers an overview of possibilities to obtain online degrees at the partner universities.
The OpenupEd partnership is an alliance of institutional MOOC providers, brought together by the European Association of Distance Teaching Universities (EADTU), who agree to follow the quality principles and practices represented in the OpenupEd Quality Label. The partners in OpenupEd have a commitment to opening education through MOOCs to the benefit both of learners and of wider society. To this end, partners endorse the eight distinctive features described in as guiding principles for their MOOC offering.
GIP-FUN is a MOOC partnership responsible for the operation of the FUN platform PLUS other services. Its missions are:
EduOpen is a Italian network of academic institutions offering MOOCs free and open to everyone. Attendees can obtain various qualifications.
Many of the portals that have been analysed above target businesses. They assert that MOOCs are of interest to them due to the variety of the course offerings and the cost effectiveness of the MOOC as learning opportunities for their employees. ‘(…) MOOCs can offer professionals a plethora of opportunities to acquire more knowledge or develop skills and competences related to their current jobs. They can also help professionals to acquire the knowledge and skills they need to take a new professional direction.’(Castano-Munoz, Kalz, Kreijns & Punie, 2016, p.253) (CASTANO-MUNOZ, J., KALZ, M., KREIJNS, K., & PUNIE, Y. (2016). Influence of employer support for professional development on MOOCs enrolment and completion: Results from a cross-course survey. Proceedings of the European Stakeholder Summit on experiences and best practices in and around MOOCs (EMOOCS 2016), 251.)
For example Udacity (https://www.udacity.com/corporate-training) as a big MOOC provider is especially targeting business, having big companies like AT&T and General Electric use their courses as internal training or for recruitment. Furthermore several Companies (https://www.coursera.org/gsi) have partnered up with Coursera to make own courses or to make contributions to courses. For instance the UBS is involved as experts in a MOOC (https://www.coursera.org/learn/understanding-financial-markets/lecture/bWZos/money-markets-ubs-guest-lecturer). Since Coursera has abandoned the traditional MOOC model, we should bear in mind, that corporates are often involved in for pay services or for recruitment reasons.
iversity, one of the project partners of BizMOOC project, reports having employees from companies like L’Oréal, Fraport, Dräxlmeier, Deutsche Bahn, Valiant Bank, Shelton Flemming, Hermes, Merck Group and Universal Music Germany in their MOOCs or PRO courses. PRO courses have the same format as the open MOOCs but they are commercial (e.e include course fees) and usually run with smaller groups of learners (sometimes more than 100+).
iversity also reports that some businesses are also producing MOOCs and PRO courses (for pay MOOCs) for/with them. Some of these businesses are for example KMPG (MOOC), Homeuork (MOOC and PRO), Play Jugo (MOOC), Procurro in Business Analysis (MOOC).
There are dozens of companies in Europe that are using MOOC portals like edX/Coursera/Udacity/FutureLearn for employee training. Often it is not the official training, but that they “allow” employees to book MOOCs and lear for free or get reimbursed.
Companies also have now the opportunity to offer their own MOOC portal for approaching and training their clients to use their products. An example for such company is SAP (https://open.sap.com/) that provides its courses on openSAP, an environment developed and provided by SAP in cooperation with the Hasso Plattner Institute.
The Web research showed that data for the distribution of learners for EU countries are rather fragmented. They are based on selected types of MOOCs or are limited to selected MOOCs providers. Having studied the literature, the authors gathered qualitative and quantitative data on learners available and summed it up below focusing on learners’ behavioral patterns and statistical information, instead of presenting complete learners’ distribution in EU countries.
MOOCs are an advantage for opening higher education to a wider group of students and reducing the barriers of participating in learning (ESU, 2013), however different characteristics causes different behaviors and attitudes towards learning.
Research, covering 68 MOOCs, offered from fall 2012 to summer 2014 by Harvard and MIT, revealed the following characteristics of the learners (Ho et al, 2015):
In a global perspective, another study found out that about 16% of participants come from developing countries. These participants possess largely the same characteristics as those from developed countries (Christensen et al 2013).
It can be concluded that an average learner is a well-educated male from a developed country under 30 years old. If MOOCs are to be used as a mean of broadening access to education and for the re-skilling and upskilling purposes there should be a clear measure aiming at diverse participation of learners mirroring the diversity of the society and using MOOCs for a wide range of purposes: education, further education and training purposes.
The secondary data analysis shows that there are five types of MOOC learners’ behavior, based on their engagement (Hill, 2013):
No-Shows – register but never login to the course while it is active.
Observers – login and may read content or browse discussions, but do not take any form of assessment beyond pop-up quizzes embedded in videos.
Drop-Ins – perform some activity (watch videos, browse or participate in discussion forum) for a select topic within the course, but do not attempt to complete the entire course.
Passive Participants – view a course as content to consume. They may watch videos, take quizzes, read discussion forums, but generally do not engage with the assignments.
Active Participants – fully intend to participate in the MOOC and take part in discussion forums, most of assignments and all quizzes & assessments.
The learners’ behaviors in relation to the course’s duration is presented in the graphic below.
Picture 1. Learners’ behaviours’ patterns in selected type of MOOCs, source: http://mfeldstein.com/emerging-student-patterns-in-moocs-a-revised-graphical-view/
Kizilcec, Piech & Schneider (2013) point out that participants have very different objectives for their activities in a MOOC. The study identifies four prototypical types of learner engagement in MOOCs: learners completing, auditing, disengaging and sampling.
Another study carried out recently has shown that participants who expected to finish a MOOC were more likely to do so, compared to participants who did not think they would complete the course. This motivation in the category “Active participants” is a good predictor for completing the MOOC (Wang & Baker, 2015).
Monthly MOOCwatch report prepared by Class Central proves that MOOCs are becoming increasingly popular among the learners both in and beyond Europe. For instance, in February 2016 French MOOC platform France Université Numérique has reached 500,000 students in February 2016 and the University of London had reached 1 million enrollments on Coursera, openSAP had also reached 1 million enrollments (Lequerica, 2016).
Literature analysis also shows that learners have shown that motivation for learners to enroll in MOOCs can be different. In Christensen et al (2013) survey most of the respondents declared advancing their careers and curiosity are the main reason for taking up an online course.
This paper presents and discusses web resources that provide an overview of MOOC initiatives in Higher Education and in the business sector, with the most popular MOOC platforms.
As digitalisation has taken over practically all the branches of our everyday lives, ways of gaining educational experience, upskilling or re-skilling has faced the challenge of fitting in the open education landscape. Therefore, MOOCs have started playing and continued to play increasingly important role and the usage of those type of courses both in Higher Education and business has been growing over at least past five years. The United Kingdom, Spain and France are the leading countries in terms of MOOCs initiatives for Higher Education, followed by the Netherlands and Germany, with the rest of the countries rather far behind. However, educational policies and strategies on national levels have been increasingly focused on digitalisation, open educational resources (OER) and MOOCs. The analysis shows that MOOCs are most often designed in science and technology, business, applied sciences, social sciences and humanities.
There are certain limitations and specifically lack of one source of statistical data on MOOCs in EU countries, therefore for the purpose of this papers the authors used five different MOOC portals that gather information about MOOCs in Europe and worldwide.
The papers focuses on presentation of different MOOC platforms, providers and partnerships that differs in the course length, modules, offer and also the proportions of different academic areas. There is a great variety of them that offer courses in Europe and worldwide, as there exist relatively few barriers for the access to MOOCs. The most popular ones are offered on platforms provided by Coursera and edX. Additionally, in Europe, HEIs are increasingly offering MOOCs on their own platforms and through collaboration of cross-sectoral European partnerships (e.g. ECO, OpenupEd)
MOOCs in business serve for upskilling and re-skilling of the employees. The existing MOOC providers are used for that purpose, as well as the companies’ own platforms are created, such as KMPG (MOOC), Homeuork (MOOC and PRO), Play Jugo (MOOC), Procurro in Business Analysis (MOOC). Another example is SAP that provides its courses on openSAP, an environment developed and provided by SAP in cooperation with the Hasso Plattner Institute. It is worth noting, that it has been also used for the integration of refugees in Germany.
MOOCs are opening access to education, improving skills and gaining new ones. However, as ownerships of learning belongs to the learners themselves it is very important to apply a learner centered approach and meet the needs of those whom those courses are designed for. The digital age has swamped the whole world, but learners are yet to become familiar and used to new forms of education and for some it may never become a perfect method of learning. The analysis if the literature shows that there exist five different MOOC learner types – from the no-shows, through passive participants to active ones. Still, the level of drop-outs in those courses is relatively high but depends largely on their motivation and course length.
With the current developments described in this paper, there is still a long way to go to arrange the area of digital education. The persistent problems of lack of data, no clear guidelines to quality in digital education and training as well as commodification must be addressed in the immediate future. There must be an intensified data collection, including learners’ feedback, on MOOCs initiatives, taking into consideration and with clear distinction according to the providers, participants, subject, purpose, mode, length, entrance criteria etc. It is important for the quality assurance processes (to enhance the quality of learning and teaching) that should also embrace MOOCs and for the access purposes (wider access to education and training as well as reducing the drop-out rate). This is the only way to monitor and take upon the current state of art for further improvement.
Clear strategies should be drawn up – on European, national and local levels – to be able to address digital education and MOOCs in a holistic manner for the benefit of all parties involved and with regard to their expected outcomes (both in terms of learning itself, but education and training as a whole). They should address recognition of MOOCs, especially in Higher Education, access and attainment and quality of digital learning, teaching and training.
Castano-Munoz, J., Kalz, M., Kreijns, K., & Punie, Y. (2016). Influence of employer support for professional development on MOOCs enrolment and completion: Results from a cross-course survey. Proceedings of the European Stakeholder Summit on experiences and best practices in and around MOOCs (EMOOCS 2016), 251.
Christensen, G., Steinmetz, A., Alcorn, B., Bennett, A., Woods, D. & Emanuel, EJ. (2013). The MOOC
European Students’ Union (ESU) (2013). Policy paper on quality of higer edcuation (amended), available: http://www.esu-online.org/news/article/6064/Policy-paper-on-quality-of-higher-education/
Hill, P. (2013). Emerging Student Patterns in MOOCs: A (Revised) Graphical View. Retrieved from http://mfeldstein.com/emerging-student-patterns-in-moocs-a-revised-graphical-view/
Ho, A. D., Chuang, I., Reich, J., Coleman, C., Whitehill, J., Northcutt, C., Williams, J. J., Hansen, J., Lopez, G., & Petersen, R. (2015). HarvardX and MITx: Two years of open online courses (HarvardX Working Paper No. 10). doi:10.2139/ssrn.2586847
Kizilcec, R. F., Piech, C., & Schneider, E. (2013). Deconstructing disengagement: analyzing learner subpopulations in massive open online courses. In Proceedings of the third international conference on learning analytics and knowledge (pp. 170-179). ACM.
Wang, Y. & Baker, R. (2015). Content or platform: Why do students complete MOOCs? MERLOT Journal of Online Learning and Teaching, 11(1)
Lequerica A. (2016). MOOCwatch February 2016, avaialable: https://www.class-central.com/report/moocwatch-feb-2016/
Internet source http://www.elearnspace.org/Articles/MOOC_Final.pdf