The Ed Techie

What quality measures apply to MOOCs?

<Image http://www.flickr.com/photos/wetwebwork/2402908982/ >

In case you haven't seen it, Ulf-Daniel Ehlers & Ebba Ossiannilsson & Alastair Creelman are running a project on Quality and MOOCs. It has a different blog post from an invited author every week for 12 weeks. They've all been of a very high standard (until this week, see below). I'd recommend Stephen Downes and Grainne Conole's posts in particular, but they're all worth reading.

So last week was my turn, and you can read my post here. The point I wanted to make was that we have created a set of quality criteria for education that is, obviously, matched to the context of education. So we are interested in things like student satisfaction ratings, student completion rates, benchmarking against agreed criteria and external review. Some of these are applicable to MOOCs, for instance you might still want external review of content before releasing it, and some courses may need to be linked to official criteria, for instance if they are related to specific vocations.

But MOOC audiences are very different, and the motivations for people studying MOOCs are different for those in higher education, or are at least of a different priority. So the same quality measures are not applicable. The most obvious of these is the much discussed drop-out rate. Course completion is just not as important for MOOC learners as it is for formal education students. And, as the (excellent) report on Edinburgh MOOCs highlights, the numbers of MOOC learners who complete surveys is a small fraction of the small fraction who complete, so student satisfaction is difficult to compare also.

My blog post ends with a plea though - and that is not to constrain MOOCs with the same quality demands we have placed on higher education. Those constraints are understandable when people are parting with lots of money, but the point of MOOCs is that they are liberated from these considerations. They allow experimentation, because the relationship between learner and educators is fundamentally different when money hasn't changed hands. We don't want to make MOOC providers fearful of experimentation because they'll fall short of quality measures that have been imposed from a different context. 

What university rankings really tell us

There's a law with league tables that goes something like, your criticism of them is indirectly proportional to your placing in them. If you're in the top 10, it's a marvellous piece of research, but as you go down, you become more suspicious. So, when I saw that the OU was ranked 99th in the Times Higher top 100 universities under 50 years old, my critical faculties were engaged. So you can probably dismiss all of this as sour grapes, but the table raised a couple of issues for me which reveal quite a lot about the state of higher education (and to be honest, I'm desperate to blog about something other than bloody MOOCs).

The first issue is that, despite all the talk to the contrary, it's research that counts still. The league rankings were based on six criteria. Of these 5 are related to research:

• Research: volume, income and reputation (30 per cent)
• Citations: research influence (30 per cent)
• International outlook: people 
• and research (7.5 per cent)
• Industry income: innovation (2.5 per cent).

This may favour smaller institutions, where a couple of big hitters can have an impact. But more significantly, it means only one criteria relates to teaching. That tells you a lot about what's valued in higher education by people in higher education, but I'm not sure the same is true of students. The one remaining factor then is:

• Teaching: the learning environment (30 per cent)

The OU would fair well here I though, after all, we usually come in the top 3 in terms of student satisfaction (yes, I'm strangely supportive of that league table). But the OU only scored 18.1% on this. How come? Well, closer examination of the methodology for rating the learning environment reveals that it isn't based on student satisfaction but is instead influenced mainly by, you've guessed it, research. So two of the six factors were:

"the ratio of PhD to bachelor’s degrees awarded by each institution. We believe that institutions with a high density of research students are more knowledge-intensive, and that the ­presence of an active postgraduate community is a marker of a research-led teaching environment valued by undergraduates and postgraduates alike.
The PhD-to-bachelor’s ratio is worth 3 per cent of the 100 Under 50 scores (up from 2.25 per cent).
This category also uses data on the number of PhDs awarded by an institution, scaled against its size as measured by the number of academic staff."

If you have a lot of undergraduate students, like the OU does, then no matter how good their learning experience is, you won't score highly because the ratio to postgrad will be low.

I do value research, but for 5 of the 6 criteria to be based around it and the remaining 1 be heavily influenced by it seems to raise issues. For potential students I wonder what use this is. I'd want to know what the teaching environment is like, as that's what I'm paying my money for. Quite often those researchers who contribute to the ranking view teaching as an inconvenience, so it doesn't bear much relationship to the quality of teaching. 

Choosing a university on the basis of its research is a bit like moving to an area and using the quality of schools as your sole criterion, when you don't have any children. It may be a proxy for good quality but it could be that everyone lives outside the area and busses their kids in (Harrow for example has a very posh school but not many of its pupils live in the surrounding area). For you, the quality of doctors may be a more important criteria.

I wonder what message this sends to all, but particularly new, academics. The message I would take away is "research is still king, do the minimum you have to in order to fulfil your teaching duty." And with students paying high fees for their education, this is setting up a conflict between the 'customers' and the context within which staff operate. And that can't be good for the long term future of education (I'm not going to mention MOOCs here, I'm not!).

One other thought was the research obsession left no room for any digital presence. These rankings could have been created 20 years ago. There is nothing in here about online engagement, digital profile, use of social media, outreach via new channels, etc. This would be difficult to measure I grant you, but given the significance of online presence for students, academics and institutions alike, it seems like an oversight.

The cost of support

Two factors are making universities (in the UK in particular) consider the costs of their courses like never before. The first is the withdrawal of state funding and reliance on student fees. I guess this was always the intention behind the shift to a pure market driven approach, it certainly makes universities focus on their course costs. Can they do it for cheaper? Can they justify their fees? Can they lower fees?

The second is our new old friend, MOOCs. It seems rather arrogant but many MOOC providers think they've just invented the idea of considering elearning costs. But all the same, the fact they are gaining attention, and that they are providing alternative models of support (largely through automatic assessment) makes universities take a look at their costs and assumptions too. 

None of this is new, I wrote this paper about economic models of large-scale elearning back in 2004, building on the foundational work of Tony Bates (MOOC providers may care to read it, as it may save them a few years of coming to the same conclusions). But that doesn't mean it isn't valid to reconsider it from the perspective of MOOCs. As I conclude in one of my MOOC presentations, one of the really good things about them are the questions they make us ask ourselves.

MOOCs are about no-support teaching, that's their basic model. In the cMOOCs this support is replaced by a peer network, in the xMOOC by automatic feedback. The student fees makes us consider how much learners are paying for this support, how much they are willing to pay and how good this service is over unsupported.

Below is an idealised chart showing typical ratios for course production and presentation costs over 5 presentations (you need 5 to get a clear picture as all the production costs are in year one). The big cream chunk in the second column is tuition costs. The green bit on top is student support costs (generic and specific student support services, eg support for students with disabilities, pastoral support, running regional centres, etc). The other bits are things like IT services. I've removed the actual figures, it's the relative amounts I want to focus on. This shows that by far the biggest cost is that of tuition. Paying people to support learners is where the money goes. 

The other key element is that, of course, production is a fixed cost. So once we've paid it, we've paid it (more or less, we may need to produce new items). Whereas, most of the presentation costs are variable - they increase as student numbers increase. Of course, countering this is the income side of the graph, where your income increases as you get more students too. The two should balance each other out.

Now, of course if you remove those big chunks in presentation you can offer courses cheaply. We've always known that. MOOCs are essentially doing away with the second bar, it's just the fixed costs of production. The good thing with this model is that there are no variable costs, and each presentation doesn't cost you, but you do still get the income side of the bar (although how that income is realise isn't clear). So in theory, you should move from debit to profit quickly as you recoup those production costs, without the hindrance of additional presentation costs.

The issue is that these services are key to long term success for learners. It isn't evenly distributed though. Some learners hardly ever avail themselves of these, don't care about tuition and do very well studying on their own. Other learners require a lot of support for various reasons and probably have more than their 'fair' share of these services (ie more than they've actually paid for). And most are in the middle, they make use of them sometimes, depending on circumstances.

For distance education in particular, this first group, the confident, independent learners probably do okay by MOOCs. They probably represent the 10% or so who complete. Then there are some for whom no amount of support can help through, either study isn't for them, or this is the wrong time. But sitting in the middle is a big group who need varying levels of support to 'survive' a protracted course of study. If MOOC dropout over 7 weeks is 90%, imagine what it'd be like over 3 or 4 years of degree study?

But that doesn't mean universities shouldn't look at ways of reducing the cost of that cream bar, if we can do so without losing students. Maybe not all of the support does need to be done by a human. But I'm pretty sure some of it does.

This highlights the dilemma for universities - many students may not think they need that cream bar. It's like a universal credit, such as a state pension. Some need it more than others, but if you remove the principle of all paying into it, then it becomes prohibitively expensive for those who do need it. So the question that fees and MOOCs make both universities and students address is - how much do I value support? It's a profound question for the future direction of education, but I agree with Christopher Newfield when he argues that dreams of zero marginal costs in MOOCs are fanciful.

It all rests in that second column..