MOOC completion rates DO matter

It has become accepted practice amongst those who know about MOOCs to sniff at completion rates. Focusing on them (hell, even mentioning them) demonstrates just how constrained you are by the old ways of thinking daddio. I find this particularly from the cMOOC crowd, and I've stopped talking about them, because as David Kernohan suggests, to even talk about them is like saying you hate learning.

The commonly used argument against completion rates (or even worse 'drop-out rates'), is that they aren't relevant. Stephen Downes has a nice analogy, (which he blogged at my request, thankyou Stephen) in that it's like a newspaper, no-one drops out of a newspaper, they just take what they want. This has become repeated rather like a statement of fact now. I think Stephen's analogy is very powerful, but it is really a statement of intent. If you design MOOCs in a certain way, then the MOOC experience could be like reading a newspaper. The problem is 95% of MOOCs aren't designed that way. And even for the ones that are, completion rates are still an issue.

Here's why they're an issue. MOOCs are nearly always designed on a week by week basis (which would be like designing a newspaper where you had to read a certain section by a certain time). I've blogged about this before, but from Katy Jordan's data we reckon 45% of those who sign up, never turn up or do anything. It's hard to argue that they've had a meaningful learning experience in any way. If we register those who have done anything at all, eg just opened a page, then by the end of week 2 we're down to about 35% of initial registrations. And by week 3 or 4 it's plateauing near 10%. The data suggests that people are definitely not treating it like a newspaper. In Japan some research was done on what sections of newspapers people read. There is an interesting gender split but also the sections are quite evenly divided:

Men top 5 sections:

• Headlines (62.0%)
• Domestic News (55.4%)
• Sports (55.4%)
• Economy (53.3%)
• International News (47.8%)

Women top 5 sections:

• TV listings (71.4%)
• Headlines (65.3%)
• Domestic News (53.3%)
• International News (50.8%)
• Crimes and Accidents (39.2%)

For MOOCs to be like newspapers then you'd expect 65% to read the topics in week 1 and, say 54% the topics in week 7. This doesn't happen. Now, it could happen, if MOOCs were designed that way, and you thought that was appropriate for your subject matter. But to say it does happen is simply incorrect.

Now for any individual this may not matter, you've dropped out when you felt like it, and maybe that was a meaningful experience (or maybe it was a painful experience because you felt out of your depth, but we don't like to talk about that either). But for MOOCs in general as a learning approach it really does matter. Most MOOCs are about 6-7 weeks long, so 90% of your registered learners are never even looking at 50% of your content. That must raise the question of why are you including it in the first place? If a subject requires a longer take at it, beyond 3 weeks say, then MOOCs really may not be a very good approach to it. There is a hard, economic perspective here, it costs money to make and run MOOCs, and people will have to ask if the small completion rates are the most effective way to get people to learn that subject. You might be better off creating more stand alone OERs, or putting money into better supported outreach programmes where you can really help people stay with the course. Or maybe you will actually design your MOOC to be like a newspaper.

Kernohan raises the point that it is in the commercial interest of MOOC companies to dismiss drop out rates. A good question to ask yourself when someone says completion rates don't matter is "if they had 90% completion rates, would they still be telling me they don't matter?".

Comments

[Twitthaus]

Perhaps the more accurate analogy for MOOC providers with a commercial interest is that MOOCs are like advertisements. Readers glance at an advertisement and may skim read bits of it. All that's important to the advertiser though is brand visibility - and the possibility that someone somewhere might sooner or later buy their product.

[mweller]

Hi Gabi, I think that is true for many but would depend on what the purpose of the MOOC was. Some MOOCs might be run with the intention of helping people update skills, or be part of a community. There is no further product as such that they are trying to sell eg there are MOOCs on becoming a student. These are aimed at people who are already signed up to start at uni, and it's about helping them make that transition. So it's not really an advert, more an additional resource.

[Techczech]

I am one of those who don't think that MOOC completion rates are a valid source of criticism of MOOCs as compared to traditional education. When you take into account the investment in money and time even a 10% drop out rate for universities should quite alarming. http://researchity.net/2012/08/18/mooc-motivations-and-magnitudes

Sure, people don't generally approach MOOCs as if it were a newspaper or a textbook (although I have done this with two MOOCs of interest to me). However, that is not an argument against MOOCs in general, just against the way the materials are presented on the web. Here, the O for openness is more important than anything else.

But even though I accept your criticism of the way MOOCs (or at least their websites) are structured, the underlying issues apply equally to traditional education. Apart from large drop out rates in traditional education, there is the ignore and forget aspect. Most students forget most of what they've learned in class (unless it is reinforced by practical use - e.g. medicine, law, teaching). Why not just cut down what you present to 10%. Almost nobody reads most academic books cover to cover. Still I haven't perceived a noticeable monograph slimming. While we're at it, try quizzing a book author on all the facts in her own work.

Finally, while I'm not friend of commercialization of learning, I don't see the vested status quo interests of many MOOC opponents as any more superior than the commercial interests of which people are so suspicious.

I'm not saying this because I think MOOCs are completely new in every aspect or that they will transform education completely. I never thought that. But the majority of their pedagogical faults are shared by the incumbents.

[twitter.com/cogdog]

Perhaps worth considering is what mean ye by "completion"? Showing up? Passing an exam? These are completions as decided by the course provider. Where is the part of what a student determines they got out of a course?

I could not agree more (well maybe I can) about the blind allegiance we have to the weekly schedule, one that beats out like the rowers on the Roman slave ship where Ben Hur was pulling oars http://cogdogblog.com/2012/08/15/mooc-ramming-speed/

The weekly pace structure serves the course provider as much as the assembly line served Ford. Ok maybe extreme. But few question it- why must open course proceed at such a fixed pace?

A nice exception was the ETMOOC one run by Alec Couros in 2013- a two week pace gave more reasonable time to reflect and think about the concepts.

It feels like a factory floor.

[Juliette Culver]

Anecdotally, I've signed up for about six MOOCs, started about three of them (the timing just didn't work out for the others). Of those three I dropped out of two within the first couple of weeks. The other one I completed and got very engaged with - I participated in the forums, looked forward to the new videos coming out each week and even felt slightly sad when it finished. I carried on until the end for that one because it was a much better MOOC than the other two and I'm sure it wasn't a coincidence that it had a ridiculously high completion rate overall. It wasn't that I was only interested in the first couple of weeks content or had got everything I wanted out of the course, I just made a trade-off between my time and whether the course content was inspiring and interesting enough to continue.

[Twitthaus]

Another take on MOOC "completion rates": http://www.openeducationeuropa.eu/en/news/mooc-study-creates-new-taxonomy-engagement

[mweller]

@dominik - hi, I hadn't made that connection with knowledge retention, so thanks. I do think this is different though. Although you are right, lots of knowledge isn't retained, a lot of the more nebulous, generic skills are. I remember being told "long after you have forgotten all the papers we give you to read you will remember how to be a scientist". My point was more that if we have good stuff in the later parts of MOOCs, people simply aren't getting to see it, so we should reconsider their design if we think that stuff is important.
@Alan - actually I was talking about activity rates, which is just turning up, although people do have different definitions of completion. Sadly people just aren't turning up for those later weeks. And it would be ok if it was a different 10% in week 7 than it was in week 3 (as Stephen seems to suggest) but it isn't, so people aren't picking and choosing the bits they like best. I agree about Alec's course, and I think that illustrates that if you want the newspaper type model then you have to rethink design quite radically.

[MoocLab]

Of course it would matter if MOOC providers achieved 90% completion rates, because that would then become their selling point. But as it stands, course completion isn't what it's about. It's about what each individual can take from a course. For some, it will be just snippets of useful information. For others, it will be the desire to learn a new skill fully from start to end.

MOOCs are viable so long as they provide an offering on different levels - up to the learner to decide what he or she wants out of it.