Redefining MOOC completion rates

A few posts back I posted Katy Jordan's data on completion rates for MOOCs. It's set me thinking that we're probably being harsh in terms of how we define enrollment on MOOCs, which in turn makes completion rates look worse than they actually are. In formal education there are different ways of defining who has enrolled on a course. I'm pretty sure there is a cooling off period, so if a student drops out within the first week or so, they don't count as having enrolled in the first place (can anyone confirm this from their uni?).

So, taking MOOC enrollment figures to be the number who signed up for a MOOC even if they never come in to it is always going to give harsh figures. It's a bit like saying university courses enrollments are all those who come to an open day. Maybe a better figure would be the number of students active after 1 week. This is your baseline figure as those are the students who have actually started the course.

Another graph Katy plotted was the average number of students enrolled across weeks:

I stupidly didn't get the data from Katy before she went on maternity leave, so actual analysis will have to wait until she's back. But just using this graph as a rough guide we can do some calculations. At the end of week 1 it looks like to me that there are about 55% of students still active from the week 0 point. So, if we took this 55% as our starting figure, we know that the average completion rate was 12.6%. This would now be 22.9% (I have probably done something very wrong with the maths here, but bear with it for rough estimate). Now, approximately 23% completion with open learners isn't too bad.

When I ran T171 we had around 12,000 students and our completion rates were around 40-50% with open access learners. And that was with expensive and intensive tutorial support, so I know how hard it is to get open entry level 1 learners to complete (mind you it was a 28 week course, if I'd stopped it at 7 weeks the completion rate would have been nearer 70-80%).

But MOOCs can't have it both ways, they've been boasting about those massive enrollment figures, which take into account everyone who raised an eyebrow at their course. We know from Katy's data that the median enrollment (from our data) was 18941. So if we take my rough 55% number, this comes down to 10417. Which is certainly not 100K, but it's not too bad either.

So, as always, when you dig into it, the picture is more subtle:: MOOCs don't enrol as many as first thought but their completion rate isn't as bad either. That's not a very snappy headline though.

Comments

[Dougclow]

More good stuff, thanks Martin.

But ... and it's a minor point ... my power law spider sense is tingling! The black line with its equation looks a lot like a power law.

Obviously, when she gets back from maternity leave, Katy will be wanting to explore an exponential fit, and a two-population exponential fit (like in the Educause paper from Coursera folk: http://www.educause.edu/ero/article/retention-and-intention-massive-open-online-courses-depth-0 - though notably they do not discuss what other fits they explored) and all sorts of other things, following best practice (starting here: http://tuvalu.santafe.edu/~aaronc/powerlaws/) before claiming that these data are best fit by a power law.

Sorry - it's just red rag to a bull for me.

[plus.google.com/102100612251670625480]

My evidence is anecdotal, but I dimly recall that the OU doesn't 'count' students who've registered for a course until they submit their first TMA. Is that still the case? There doesn't seem to be any other way of knowing remote/distance students are physically studying a course until they complete an assignment.
Traditional and conventional campus universities - so I believe - use a different measure, which is something like counting students who have stayed until a third of the way through their first term, but HEFCE and other Councils will be the source for those data and that criterion.
It's a bit like signing up for a gym immediately after the excesses of the festivities we are approaching: many sign up, but fade away after one month...

[Techczech]

This is good. I can see that the MOOC going from few hundred to 40 participants I ran last year fits with this pattern without needing to be truly massive (just much bigger than usual for the subject).

But there is more to it than just the numbers. I wrote more about a year ago about how the drop out rates don't really matter given the investment and the experience and are not as drastic when compared with meat space universities in real terms: http://researchity.net/2012/08/18/mooc-motivations-and-magnitudes/