As I mentioned in the previous post, I am doing some Gates funded research on MOOCs. My part was learning design analysis, while Katy Jordan has been looking at factors influencing completion rates. All this work is Katy's, I take no credit for it. She would blog it, but is about to have her first baby any day now, and strangely that has taken priority over blogging about MOOCs, so she said I could blog it on her behalf.
There will be a paper that details the full results and methodology, so I'm just giving some highlights here. Katy collected completion data from 221 different MOOCs. The range was limited to ones where we could get this data, so is biased by a couple of big data sets from Coursera and Open2Study. This last one in particular has good completion rates so may have biased the overall results somewhat.
If we start with enrolment figures first then, the picture looks like this:
(N = 220; Median = 18941; Minimum = 95; Maximum = 226,652). Although the really big numbers get the headlines, there are a lot of courses in the 0-10000 range.
Completion rates across this sample are:
(N = 129; Median = 12.6%; Minimum = 0.7%; Maximum = 52.1%). This bimodal distribution may be a result of those Open2Study courses. Those guys appear to be doing something right in terms of completion.
Does enrollment numbers relate to completers? The answer is yes, there is a significant negative correlation, ie the more people who enrol then the lower the percentage who complete. This single graph might well put paid to the idea that we can educate the world with massive courses:
Yeah, I hear you say, but it's not completion that's important but how many people are active. I'm afraid that shows exactly the same picture, ie there is a significant negative correlation between number of active users and the percentage who complete:
Does the reputation of the university matter? The answer is: possibly. There is a positive correlation between the Times Higher Ed score of a university and enrolments:
But this could just be because more elite universities have joined the elite clubs of MOOC providers. As with all correlation we can't imply causality, it may not be the university reputation that leads directly to the higher enrolments but a related factor. Anyway, it turns out that reputation has no effect on completion rates:
Have enrolments increased or decreased over time I hear you ask. Well, there is a negative correlation over time with enrolments, so numbers on individual MOOCs have declined. This may be that with more MOOCs available people can find the one that suits them better, instead of everyone doing AI, but if you wanted to be all sensationalist (I'm looking at you, journalists) you could say it indicates that the MOOC bubble has burst:
Have completion rates changed over time? Yes, in the opposite direction, there is a positive correlation with completion and date. This could mean people are getting better at MOOCs, and if the trend continued those completion rates would be less of an issue. But I suspect it is related to the point above, in that there are more MOOCs around so people are taking ones more closely related to their interests, so more are completing:
(see https://www.katyjordan.com/timeline for more detail)
Does the course length have an impact on completion? You bet, a negative correlation with course length, so the longer your course, the lower the percentage who see it through. This is probably obvious, but it shows that if you get to 12+ weeks it's probably just some bloke in a shack in Arkansas left:
This raises the question, should we create MOOCs of 2-3 weeks in length (many of Open2Study courses are this long, which may account for the good completion rates)? And if we're creating 2 week MOOCs, how are those different from OERs? UPDATE - see Doug's comments below. He makes the point that obviously the longer a course goes on, the more people will drop out. So we can't say length causes drop out, although even in this respect, you might be better off having 2 x 3week MOOCs than 1 x 6 week one, simply because you catch more people (if completion is important that is).
And a tantalising one to leave you with: does gender influence completion rates? We can't tell as only a few data sets included breakdown by gender, 13 in this case. So we couldn't draw any conclusions from such a small sample, and Katy will probably tell me off for including this, but I do so as a plea to others to release data. From the sample we had, there is a possible positive influence on the percentage of female students and completion, ie women tend to complete more than men, who no doubt wander off to watch football during week 3:
We looked at a number of other factors including assessment type, MOOC type and platform, but the variation in sample sizes makes conclusions from these difficult to form. This data raises a number of interesting questions, which being correlations it can't answer on its own. For me these are:
1) Can we improve those completion rates from around 10%?
2) Are learners getting better at MOOCs?
3) Should we limit ourselves to short MOOCs?
4) What demographic data may influence completion rates?
5) The big one is, does completion even matter? (If it does, then MOOCs probably aren't the solution to your problem)