The Ed Techie

Redefining MOOC completion rates

A few posts back I posted Katy Jordan's data on completion rates for MOOCs. It's set me thinking that we're probably being harsh in terms of how we define enrollment on MOOCs, which in turn makes completion rates look worse than they actually are. In formal education there are different ways of defining who has enrolled on a course. I'm pretty sure there is a cooling off period, so if a student drops out within the first week or so, they don't count as having enrolled in the first place (can anyone confirm this from their uni?).

So, taking MOOC enrollment figures to be the number who signed up for a MOOC even if they never come in to it is always going to give harsh figures. It's a bit like saying university courses enrollments are all those who come to an open day. Maybe a better figure would be the number of students active after 1 week. This is your baseline figure as those are the students who have actually started the course.

Another graph Katy plotted was the average number of students enrolled across weeks:

I stupidly didn't get the data from Katy before she went on maternity leave, so actual analysis will have to wait until she's back. But just using this graph as a rough guide we can do some calculations. At the end of week 1 it looks like to me that there are about 55% of students still active from the week 0 point. So, if we took this 55% as our starting figure, we know that the average completion rate was 12.6%. This would now be 22.9% (I have probably done something very wrong with the maths here, but bear with it for rough estimate). Now, approximately 23% completion with open learners isn't too bad.

When I ran T171 we had around 12,000 students and our completion rates were around 40-50% with open access learners. And that was with expensive and intensive tutorial support, so I know how hard it is to get open entry level 1 learners to complete (mind you it was a 28 week course, if I'd stopped it at 7 weeks the completion rate would have been nearer 70-80%).

But MOOCs can't have it both ways, they've been boasting about those massive enrollment figures, which take into account everyone who raised an eyebrow at their course. We know from Katy's data that the median enrollment (from our data) was 18941. So if we take my rough 55% number, this comes down to 10417. Which is certainly not 100K, but it's not too bad either.

So, as always, when you dig into it, the picture is more subtle:: MOOCs don't enrol as many as first thought but their completion rate isn't as bad either. That's not a very snappy headline though.

Find time for courage

At the MOOC research conference last week Amy Collier gave an impassioned call for courage in relation to MOOCs, and in the way I interpreted it, openness in general. As she put it "some things are too important not to have courage".

This quote came back to me last week, when I was in discussions about open access publishing at the OU. The responses from others in different departments were sensible and cautious, a wait and see strategy. I couldn't disagree with them, but part of me felt, 'remember when we used to take risks and do stuff because we thought it was right?' This isn't a moan about that decision, but rather that it made me consider the wider context in universities.

My concern is this - post-austerity, and particularly post-fees in the UK, all universities have become risk averse and cautious. There are lots of factors to consider. I don't blame an individual - it's not a great outcome to take a risk, get a bad NSS score and have to lay off staff the following year because of reduced enrolments. But I think this cautious approach seeps into the culture and takes years to overcome. Imagine you are a new academic who takes on a management role, sub-dean say. You are being enculturated into how a sub-dean behaves by observing others. If the appropriate sub-dean type behaviour is sensible caution, that is the mode you adopt. And then this gets passed down to the next incomer, and so on. Twenty years from now you find yourself asking "where did all our innovators go?". But if you reverse that and have people taking appropriate but occassional risky decisions, then this becomes the behaviour you adopt. 

So I make a plea to all of those in universities, especially myself as I lack it on a daily basis - have the occasional flash of courage.

Design responses to MOOC completion rates

Well, my previous post on data for MOOC completion rates caused a bit of a kerfuffle on Twitter. It was interpreted by some as saying "ONLY completion rates matter". And also of not taking into account other factors such as what learners who don't complete get from a MOOC. That seems rather like criticising Alien for not being a rom-com to my mind - they're doing different things. This research was showing one aspect with the quantitative data available. It is part of a bigger picture which ethnographic studies, surveys and more data analysis will complete. It wasn't attempting to be the full stop on MOOC research.

Anyway, here is another graph that Katy created, showing attrition rates of active users (those that come into the course and do something, not just those who complete assessments) across disciplines:

That's a pretty consistent pattern. If we saw it nature we'd give it some name like "The MOOC attrition law". My interest is as a course designer, so given that the drop-off pattern seems fairly robust, what does it mean for design? (Doug will have issues about the power-lawness of this) 

I think there are two responses (but maybe you can think of more).

Design for retention

The first is to say that completion is a desired metric. There may be courses where you really do want as many people as possible to complete. Imagine you were running a remedial maths course for instance, then it won't help your learners much if they only cover a third of the subject matter they need for whatever purpose (Bridge2 Success got learners through maths so they could get onto an employment program, so completion was very important here).

In this case you need to address the 'problem' of drop-out, because it is a problem for you. There might be a number of ways you do this: by adding in more feedback, using badges to motivate people, creating support structures, supplementing with face to face study groups, breaking your longer course into shorter ones, etc. The point is that you design in features that aim to improve completion.

Design for selection

The second design approach is to say that completion isn't an important metric. Here you accept the MOOC attrition law and design the experience with that in mind. I have some sympathy with Stephen Downes when he says no-one finishes a newspaper but we don't talk about people 'dropping out' of a newspaper (I've heard him say this but can't find a link - anyone?). So even to talk about drop-out is to map the wrong metaphor to MOOCs. His analogy breaks down a bit however because not all readers drop out at page 7 of a newspaper, they dip into different sections. People tend to drop out of MOOCs by week 3. It's not as if they're coming in and doing a bit from week 7 and a bit from week 5, and then leaving. They're simply not getting to those later weeks. And even if you are of the 'completion doesn't matter' camp, I'm sure most course designers don't think the content in week 5 is half the value of that in week 1. 

So, in this design approach you might break away from the linear course model, to allow people to do the 'newspaper' type selection. A course might be structured around themes for instance, and each one around largely independent activities (I tried to design H817open a bit like this). So in this case completion really doesn't matter, learners take the bits they want. 

In both cases I would suggest that the completion rate data is useful for you. In the first case you know what type of completion rate to expect, and in the second one it drives you to be more innovative in design approach. And that's the point about the research - it helps inform decisions.

By the way - this is my fifth blog post in 5 days. Just in case Jim Groom berates me for not blogging often enough...

Completion data for MOOCs

As I mentioned in the previous post, I am doing some Gates funded research on MOOCs. My part was learning design analysis, while Katy Jordan has been looking at factors influencing completion rates. All this work is Katy's, I take no credit for it. She would blog it, but is about to have her first baby any day now, and strangely that has taken priority over blogging about MOOCs, so she said I could blog it on her behalf.

There will be a paper that details the full results and methodology, so I'm just giving some highlights here. Katy collected completion data from 221 different MOOCs. The range was limited to ones where we could get this data, so is biased by a couple of big data sets from Coursera and Open2Study. This last one in particular has good completion rates so may have biased the overall results somewhat. 

If we start with enrolment figures first then, the picture looks like this:

(N = 220; Median = 18941; Minimum = 95; Maximum = 226,652). Although the really big numbers get the headlines, there are a lot of courses in the 0-10000 range.

Completion rates across this sample are:

(N =  129; Median = 12.6%; Minimum = 0.7%; Maximum = 52.1%). This bimodal distribution may be a result of those Open2Study courses. Those guys appear to be doing something right in terms of completion.

Does enrollment numbers relate to completers? The answer is yes, there is a significant negative correlation, ie the more people who enrol then the lower the percentage who complete. This single graph might well put paid to the idea that we can educate the world with massive courses:

Yeah, I hear you say, but it's not completion that's important but how many people are active. I'm afraid that shows exactly the same picture, ie there is a significant negative correlation between number of active users and the percentage who complete:

Does the reputation of the university matter? The answer is: possibly. There is a positive correlation between the Times Higher Ed score of a university and enrolments:

But this could just be because more elite universities have joined the elite clubs of MOOC providers. As with all correlation we can't imply causality, it may not be the university reputation that leads directly to the higher enrolments but a related factor. Anyway, it turns out that reputation has no effect on completion rates:

Have enrolments increased or decreased over time I hear you ask. Well, there is a negative correlation over time with enrolments, so numbers on individual MOOCs have declined. This may be that with more MOOCs available people can find the one that suits them better, instead of everyone doing AI, but if you wanted to be all sensationalist (I'm looking at you, journalists) you could say it indicates that the MOOC bubble has burst:

Have completion rates changed over time? Yes, in the opposite direction, there is a positive correlation with completion and date. This could mean people are getting better at MOOCs, and if the trend continued those completion rates would be less of an issue. But I suspect it is related to the point above, in that there are more MOOCs around so people are taking ones more closely related to their interests, so more are completing:

(see http://www.katyjordan.com/timeline for more detail)

Does the course length have an impact on completion? You bet, a negative correlation with course length, so the longer your course, the lower the percentage who see it through. This is probably obvious, but it shows that if you get to 12+ weeks it's probably just some bloke in a shack in Arkansas left:

This raises the question, should we create MOOCs of 2-3 weeks in length (many of Open2Study courses are this long, which may account for the good completion rates)? And if we're creating 2 week MOOCs, how are those different from OERs? UPDATE - see Doug's comments below. He makes the point that obviously the longer a course goes on, the more people will drop out. So we can't say length causes drop out, although even in this respect, you might be better off having 2 x 3week MOOCs than 1 x 6 week one, simply because you catch more people (if completion is important that is).

And a tantalising one to leave you with: does gender influence completion rates? We can't tell as only a few data sets included breakdown by gender, 13 in this case. So we couldn't draw any conclusions from such a small sample, and Katy will probably tell me off for including this, but I do so as a plea to others to release data. From the sample we had, there is a possible positive influence on the percentage of female students and completion, ie women tend to complete more than men, who no doubt wander off to watch football during week 3:

We looked at a number of other factors including assessment type, MOOC type and platform, but the variation in sample sizes makes conclusions from these difficult to form. This data raises a number of interesting questions, which being correlations it can't answer on its own. For me these are:

1) Can we improve those completion rates from around 10%?

2) Are learners getting better at MOOCs?

3) Should we limit ourselves to short MOOCs?

4) What demographic data may influence completion rates?

5) The big one is, does completion even matter? (If it does, then MOOCs probably aren't the solution to your problem)

The Learning Design of MOOCs

I got some Gates funding for the MOOC Research Initiative to look at two things: completion data, and learning design MOOCs. The first part allowed Katy Jordan to finish the work she had started in mapping various factors from over 200 MOOCs that influence completion. You can see more of her work here, and I'll blog on that later.

My part has been using the tools we've developed at the OU for learning design, building on Grainne Conole's work. We use two main tools: the Activity Planner and the Module Map. The first maps student activity across 6 categories, eg assimilative, productive, etc. This is a good way to think about how you want to teach a particular topic. The second one looks at the resources you will need in order to create those activities, and maps them under four headings: Content, Guidance, Demonstration & Communication.

So far I've only done the Module Map applied to 14 MOOCs, chosen from a range of providers and to give both x and cMOOC types. I have created two representations. The first is the Resource Type which shows the range of different media used (video, article, text, audio, etc). Here's an example:

The second representation is more interesting and maps the number of resources used under each of the four categories given above. Here is an example:

Now there are lots of caveats around this. For a start it's a subjective interpretation (Sheila MacNeill and I did the mapping). Secondly it counts number of resources, not length. So 3 x 10 min videos counts as 3 whereas 1 x 30 minute video is just 1. But nonetheless, I think it gives a good indication of the overall weighting in a course. Here is an example of what I think is a fairly typical MOOC, which actually came up for both x and cMOOCs:

Here is an example of a more extreme end of the xMOOC spectrum, with a heavy emphasis on content:

And this is what a more cMOOC looks like (this is DS106):

I was keen to emphasise that it's not the case that one design is 'better' than another, but rather they are doing different things. Sometimes you might want the content approach, and other times you might want a more activity/community centric approach.

In my talk someone suggested they are like food labelling, and that's how I'd like to develop them. You could imagine four or five 'types' so learners, and designers, could decide what type of MOOC this one was. 

In the next phase we will develop some more of the activity profiles for MOOCs to examine in more detail what learners are doing (or what the course designers think they should be doing). But anyway, it feels like a fruitful way of examining them. You can find all the maps on this wordpress blog I've created.

The Battle for Open

<This is how we do protest in Wales>

We have a new issue of JIME out, which is a special issue on the OER 13 conference. There are some excellent papers, so go read them. Then, when you've read all those, read my opinion piece, The Battle for Open.

I'm going to write a book about this over the next few months, so if you think it's an awful idea, please let me know. The article is effectively the first chapter of that book, which sets out the overall argument. My argument goes something like this: Openness has been victorious in many ways, but at this point of victory the real struggle begins. If you look at openness in research, teaching, publication scholarship, then it's hard to argue that openness hasn't been successful over the last few years in establishing itself as a core approach in higher education. It isn't something just a few oddballs bang on about now, it has moved to the centre of discourse (and, more importantly, funding).

And yet it doesn't feel like it should. We aren't seeing David Wiley leading triumphant processions down Wall Street. The MOOC invasion and backlash is the most visible part of all this, but I think that is just representative of a wider story. I try to articulate why I see it as a 'battle', giving three reasons:

1) There is real conflict between different visions

2) There are considerable spoils to be won

3) The victor writes history, so there is a battle for narrative taking place

What I want to do then in the book is explore this victory/battle dilemma for a number of areas, and then come to some conclusions. My main hypothesis is that open education isn't an interesting subset of education now, it is education. And so if you care about education you should care about the direction that openness takes. We have made mistakes in the past by outsourcing issues, such as selling off university presses or buying in LMSs. This meant education no longer controlled or had much of a say in core parts of its key functions. We are repeating this mistake with MOOCs and commercial companies. Openness cuts across all of these and so I see it as the key principle that educators and universities need to engage with.

As you can probably tell, it'll be a struggle to control the polemic, but I hope it will be a reasoned, well argued book with plenty of research and evidence, and I'll keep the spitty, table-thumping to a minimum.

The iceland of Dallas

 

<Dallas deathstar in the snow - this may, or may not, be a metaphor>

I was at the MOOC research initiative conference in Dallas, Texas last week. As Jim and others have reported, we got caught in icemageddon, but that's a whole other (war) story. I'll be doing a few posts about the conference. It was a fantastic meeting, well done George Siemens, Amy Collier and Tanya Joosten for putting it together. I got to have some great conversations, and meet people I've know online for years. Which is by way of apology for my first post being a bit negative.

This one concerns one aspect of the conference that I am having difficulty articulating, so I'm going to try and work it through in this post. There was a data strand to the presentations, and I went to a few of these. There is some fascinating stuff being done, particularly when you have analytics on so many learners. But I also had a vague sense of unease about some of these.

They were often presented by super-smart, young computer-science researchers from privileged universities: Carnegie-Mellon, Stanford, Harvard. I don't have anything against super-smart, young or computer science people, or anyone from those universities (some of my best friends are young people :). But there was something a bit, well, cold, about it all (and not just because of the wind chill factor of -10). People were nodes, and they could be manipulated to move from peripheries to the center by tweaking certain elements. It was easy to forget you were talking about learners, and not sales of baked beans.

But I think we need this research, it's useful and can tell us a lot about what's going on. Candace Thrall was stuck at the airport so couldn't give her presentation, but a colleague gave it on her behalf. She mentioned that one of the transitions we were going through was from a theory-led one to an evidence based one. Prior to this Jim Groom was telling me about Mike Caulfield suggesting we were in post-theory now, where only the big-data mattered (this was from a book I think, if anyone knows which one, let me know). 

UPDATE: Mike has done a great post elaborating on the post-theory debate here. It explains what I was trying to get at in this post.

I felt I had a glimpse of that post-theory world, and I wasn't sure I liked it. We may have been too theory-heavy before, where the evidence was inconsequential, because hey, we have a nice theory. But the pendulum swing to lack of theory where we only care about the evidence seems to lose sight of the people in the system. So I guess my plea to the super-smart, young computer science researchers at ivy league institutions who are now getting into to education is - don't ignore the bearded old guy with a bunch of theories in his back pocket, we need those too. 

Like Dallas, education should be warm and welcoming, and the danger is that data fetishisation will make it like the post-apocalyptic Dallas I experienced.