The Ed Techie

Year of epic fail & minor win (running)

<Even puppy power wasn't enough to salvage my year>

For the past few years (2009, 2010, 2011, 2012), I end them with a post reviewing my running over the previous 12 months, and make some tenuous connection to education. No-one cares, but hey, it's my blog, and I like to plan this post when I'm out running, as it gives me something to fix on. So here goes.

This year started out with big plans. I signed up for a marathon in January, a 10K February, half marathons in March and April, just a 5k in May, and marathons in June and July. That was the plan. It started badly and got worse, in a year of woe on the running front. 

I was training ok for the January marathon and did a 20 mile long run on December 20th 2012. I didn't know it at the time but I was coming down with flu that very day. The next day I was feverish, shaking and wiped out. The whole family came down with it. Christmas day 2012 was cancelled, our holiday was postponed. Full-on, miserable, definitely not just a cold, type flu. This lasted for about 3 weeks, before I could get out running again. I could barely do 5 miles then, so the marathon was out of the question. Epic fail.

I regrouped, and concentrated on the 10K in February. I forgot my Garmin when it came to this, and so had no idea of pace. I went out too fast (I was like Simon Pegg in Run Fatboy Run), and slowed considerably but actually finished in a decent time (for me) of 49.50. This was a race PB for 10K. Minor win.

Then I had a note saying the March half-marathon had been cancelled due to insufficient numbers. Never mind, I'll concentrate on the April New Forest half-marathon. Come the race date, it was postponed due to ice and rescheduled for a day when I was in Bali for the OCWC conference. Epic Fail.

I did discover the wonderful parkrun though. Every Saturday you can go down and run a 5K free of charge, and get a time posted on their website. In a world of evil, the parkrun is an unalloyed thing of goodness. I went a few times in May and got my time down to about 24 mins. Minor win.

I abandoned the June marathon as I hadn't maintained enough fitness and decided to concentrate all my efforts now on the July Tenby marathon. If I could do one marathon this year, it would still be all worthwhile, I could salvage the mess of the year with this one race. I trained dilligently, if not enthusiastically. I wasn't going to get a good time, but I should be able to get round. Come the day before the marathon I was planning my arrival time when I had a sudden thought: "That's strange, I haven't received anything from the organisers". I checked emails with increasing panic. I rang the organisers. No, it seems I had registered for all the other races back in December, but this one I had forgotten to actually put my name down for. It was full, there was no way of running the race. Epic, epic fail.

I entered a slough of despondency then. I hadn't been enjoying the running, but at least it had some purpose. I picked myself up for a late entry into the September 10k, and managed another PB of 49.43. A minor win. My Cardiff Half time was average for me at 1.55.

And then I lost all running mojo. I ended the year with a desultory total of 880 miles, the lowest in 4 years. I could have reached the 1000 mark easily if I'd kept up any decent running schedule in November and December, but I really couldn't be bothered.

Why? Well, I'm doing an MA in history and as they got nearer the end it demanded more time. And I'm writing a book, and managing research projects, etc. I found that a day would be stressful if I had to fit running in as well as these things, but if I took it out, then suddenly the day fitted together more neatly. Sarah, my wife, has a theory that you can only keep three plates spining effectively, for example work, family and one serious hobby. You can have more on the go, but the others suffer. You have plenty of other minor interests, for example I like reading literature, but if I was to go at it seriously, one of the others would have to suffer. Completing MA and maintaining running has proven difficult. 

But ultimately, that is an excuse, one can always find the time. It's only an hour here and there with the occassional long run. So, I need a plan, and it is this: next year I am going to be less ambitious in the knowledge that once I started failing this year, it gave me permission to carry on doing so. I've signed up for the Windermere marathon in July, and I'm not going to stress about time - a 5 hour marathon is better than no marathon. This way I hope to get back to enjoying running, as any run I do, regardless of time, is a success. 

So, now for the tenuous link to learning. For a start, the title could stand for any edtech review of the year, but I won't go over that again. Instead I think the analogy is that in learning you can coast for a while or you can overstretch yourself, and then you need to reflect and plan for some success to make it seem wortwhile. This may be completing a Masters, or finishing a MOOC, or just reading that novel or learning that programme language you always promised yourself. Failure is part of the pattern, but now that learning, like running, is so diverse, there's always one more way to tackle it.

Redefining MOOC completion rates

A few posts back I posted Katy Jordan's data on completion rates for MOOCs. It's set me thinking that we're probably being harsh in terms of how we define enrollment on MOOCs, which in turn makes completion rates look worse than they actually are. In formal education there are different ways of defining who has enrolled on a course. I'm pretty sure there is a cooling off period, so if a student drops out within the first week or so, they don't count as having enrolled in the first place (can anyone confirm this from their uni?).

So, taking MOOC enrollment figures to be the number who signed up for a MOOC even if they never come in to it is always going to give harsh figures. It's a bit like saying university courses enrollments are all those who come to an open day. Maybe a better figure would be the number of students active after 1 week. This is your baseline figure as those are the students who have actually started the course.

Another graph Katy plotted was the average number of students enrolled across weeks:

I stupidly didn't get the data from Katy before she went on maternity leave, so actual analysis will have to wait until she's back. But just using this graph as a rough guide we can do some calculations. At the end of week 1 it looks like to me that there are about 55% of students still active from the week 0 point. So, if we took this 55% as our starting figure, we know that the average completion rate was 12.6%. This would now be 22.9% (I have probably done something very wrong with the maths here, but bear with it for rough estimate). Now, approximately 23% completion with open learners isn't too bad.

When I ran T171 we had around 12,000 students and our completion rates were around 40-50% with open access learners. And that was with expensive and intensive tutorial support, so I know how hard it is to get open entry level 1 learners to complete (mind you it was a 28 week course, if I'd stopped it at 7 weeks the completion rate would have been nearer 70-80%).

But MOOCs can't have it both ways, they've been boasting about those massive enrollment figures, which take into account everyone who raised an eyebrow at their course. We know from Katy's data that the median enrollment (from our data) was 18941. So if we take my rough 55% number, this comes down to 10417. Which is certainly not 100K, but it's not too bad either.

So, as always, when you dig into it, the picture is more subtle:: MOOCs don't enrol as many as first thought but their completion rate isn't as bad either. That's not a very snappy headline though.

The year of no s**t Sherlock

With apologies for the potty mouth title. If you want a really good review of the year from an ed tech perspective then I suggest reading Audrey Watters' series of posts. One thing that I found myself doing repeatedly this year was staring open-mouthed at my screen as I read yet another 'discovery' or 'innovation' from US based silicon valley start-ups relating to education. It seems that if you want to be noticed then having no shame in pretending you have invented something is a real advantage. And journalists really do love a 'new and shiny' story. Here are some of my faves:

The SPOC - don't want your MOOC to be massive and open, then try a Small Private one. Or as we've called them for the past 15 years "an online course".

Hybrid Pedagogy - Clayton Christensen discovered “a fundamentally new concept [in] the world of disruptive innovation.” Or as we call it "blended learning"

MOOC based learning and innovation on campus - Coursera announced that it was working with universities to deliver MOOCs on campus. Or as we call it "elearning".

Micro-MOOCs - a micro-MOOC is either a platform to host video and content (or as we call it "the web") or "short e-courses" (or as we call them "OERs")

Creating quality courses is hard and expensive - who knew?

Retention with open access is difficult - again, why did no-one tell them this?

Massive open online content - again, as we call them "OERs"

I hope next year isn't as dumb.

The best edtech book you've never read was published last week

And indeed is published every week. 

I'm referring to Stephen Downes' OLWeekly, where he gives a round up of all the material he's commented on that week. Like many of you I subscribe to the email, and when it came through on Friday, I was struck by how much great stuff there was in there this week. I thought "this could be an edited book". So I decided to see just what it would be like as a book.

To be clear, I'm not suggesting it should be a book, in many ways the book is an inferior format since you lose the comments and the media. But it's an interesting comparison. We know what editing a book takes, and if someone said they were going to edit a book every week, we'd think they were insane. So, I went through OLWeekly, and copied Stephen's comments and the article into a PDF. I missed some out because they were too long or they were newspaper/magazine articles who would probably send drones after me for copying their material. If I have included anyone's content here and they want it removed, let me kow. I'm not really interested in distributing it, just the point of comparison. The resultant PDF was 252 pages long. You can have a look for yourself: Download Downes_weekly. I didn't bother with any formatting, and some images are missing, I only had an hour. 

There are two interesting things that the exercise raised for me. The first is to recognise the great work Stephen does every week. What I admire about Stephen most is his endurance. I might have had an idea to do this, would have done about three and then given up. Stephen reminds me of an artist who goes in and out of fashion, produces albums of varying interest and quality, but what you come to admire most is their durability and their resolution to keep doing interesting things. Think Neil Young. With not much more effort in terms of copyediting and layout this could be a proper book, and the range of material in there is truly impressive. And he does this every week. We recognise book editing, but web curation isn't normally acknowledged, there is no yardstick for measuring it. So comparison with book editorship is useful from that perspective at least.

The second point is that if you were to try and create a book that covered these topics it would probably take you a year or so. You'd need to organise a publishing deal, cajole the writers, sort out rights, get it through copyediting, etc. When blogging was all fresh we used to comment a lot about the freedom it gave, and make these sort of comparisons, but we take it for granted now. It's worth noting it occassionally though - this is still pretty amazing stuff, and it calls to mind Scott Leslie's classic Just Share post. It really is the most efficient way to operate.

And if all that wasn't interesting, here is a picture of Stephen on a steer:

 

Find time for courage

At the MOOC research conference last week Amy Collier gave an impassioned call for courage in relation to MOOCs, and in the way I interpreted it, openness in general. As she put it "some things are too important not to have courage".

This quote came back to me last week, when I was in discussions about open access publishing at the OU. The responses from others in different departments were sensible and cautious, a wait and see strategy. I couldn't disagree with them, but part of me felt, 'remember when we used to take risks and do stuff because we thought it was right?' This isn't a moan about that decision, but rather that it made me consider the wider context in universities.

My concern is this - post-austerity, and particularly post-fees in the UK, all universities have become risk averse and cautious. There are lots of factors to consider. I don't blame an individual - it's not a great outcome to take a risk, get a bad NSS score and have to lay off staff the following year because of reduced enrolments. But I think this cautious approach seeps into the culture and takes years to overcome. Imagine you are a new academic who takes on a management role, sub-dean say. You are being enculturated into how a sub-dean behaves by observing others. If the appropriate sub-dean type behaviour is sensible caution, that is the mode you adopt. And then this gets passed down to the next incomer, and so on. Twenty years from now you find yourself asking "where did all our innovators go?". But if you reverse that and have people taking appropriate but occassional risky decisions, then this becomes the behaviour you adopt. 

So I make a plea to all of those in universities, especially myself as I lack it on a daily basis - have the occasional flash of courage.

Design responses to MOOC completion rates

Well, my previous post on data for MOOC completion rates caused a bit of a kerfuffle on Twitter. It was interpreted by some as saying "ONLY completion rates matter". And also of not taking into account other factors such as what learners who don't complete get from a MOOC. That seems rather like criticising Alien for not being a rom-com to my mind - they're doing different things. This research was showing one aspect with the quantitative data available. It is part of a bigger picture which ethnographic studies, surveys and more data analysis will complete. It wasn't attempting to be the full stop on MOOC research.

Anyway, here is another graph that Katy created, showing attrition rates of active users (those that come into the course and do something, not just those who complete assessments) across disciplines:

That's a pretty consistent pattern. If we saw it nature we'd give it some name like "The MOOC attrition law". My interest is as a course designer, so given that the drop-off pattern seems fairly robust, what does it mean for design? (Doug will have issues about the power-lawness of this) 

I think there are two responses (but maybe you can think of more).

Design for retention

The first is to say that completion is a desired metric. There may be courses where you really do want as many people as possible to complete. Imagine you were running a remedial maths course for instance, then it won't help your learners much if they only cover a third of the subject matter they need for whatever purpose (Bridge2 Success got learners through maths so they could get onto an employment program, so completion was very important here).

In this case you need to address the 'problem' of drop-out, because it is a problem for you. There might be a number of ways you do this: by adding in more feedback, using badges to motivate people, creating support structures, supplementing with face to face study groups, breaking your longer course into shorter ones, etc. The point is that you design in features that aim to improve completion.

Design for selection

The second design approach is to say that completion isn't an important metric. Here you accept the MOOC attrition law and design the experience with that in mind. I have some sympathy with Stephen Downes when he says no-one finishes a newspaper but we don't talk about people 'dropping out' of a newspaper (I've heard him say this but can't find a link - anyone?). So even to talk about drop-out is to map the wrong metaphor to MOOCs. His analogy breaks down a bit however because not all readers drop out at page 7 of a newspaper, they dip into different sections. People tend to drop out of MOOCs by week 3. It's not as if they're coming in and doing a bit from week 7 and a bit from week 5, and then leaving. They're simply not getting to those later weeks. And even if you are of the 'completion doesn't matter' camp, I'm sure most course designers don't think the content in week 5 is half the value of that in week 1. 

So, in this design approach you might break away from the linear course model, to allow people to do the 'newspaper' type selection. A course might be structured around themes for instance, and each one around largely independent activities (I tried to design H817open a bit like this). So in this case completion really doesn't matter, learners take the bits they want. 

In both cases I would suggest that the completion rate data is useful for you. In the first case you know what type of completion rate to expect, and in the second one it drives you to be more innovative in design approach. And that's the point about the research - it helps inform decisions.

By the way - this is my fifth blog post in 5 days. Just in case Jim Groom berates me for not blogging often enough...

Completion data for MOOCs

As I mentioned in the previous post, I am doing some Gates funded research on MOOCs. My part was learning design analysis, while Katy Jordan has been looking at factors influencing completion rates. All this work is Katy's, I take no credit for it. She would blog it, but is about to have her first baby any day now, and strangely that has taken priority over blogging about MOOCs, so she said I could blog it on her behalf.

There will be a paper that details the full results and methodology, so I'm just giving some highlights here. Katy collected completion data from 221 different MOOCs. The range was limited to ones where we could get this data, so is biased by a couple of big data sets from Coursera and Open2Study. This last one in particular has good completion rates so may have biased the overall results somewhat. 

If we start with enrolment figures first then, the picture looks like this:

(N = 220; Median = 18941; Minimum = 95; Maximum = 226,652). Although the really big numbers get the headlines, there are a lot of courses in the 0-10000 range.

Completion rates across this sample are:

(N =  129; Median = 12.6%; Minimum = 0.7%; Maximum = 52.1%). This bimodal distribution may be a result of those Open2Study courses. Those guys appear to be doing something right in terms of completion.

Does enrollment numbers relate to completers? The answer is yes, there is a significant negative correlation, ie the more people who enrol then the lower the percentage who complete. This single graph might well put paid to the idea that we can educate the world with massive courses:

Yeah, I hear you say, but it's not completion that's important but how many people are active. I'm afraid that shows exactly the same picture, ie there is a significant negative correlation between number of active users and the percentage who complete:

Does the reputation of the university matter? The answer is: possibly. There is a positive correlation between the Times Higher Ed score of a university and enrolments:

But this could just be because more elite universities have joined the elite clubs of MOOC providers. As with all correlation we can't imply causality, it may not be the university reputation that leads directly to the higher enrolments but a related factor. Anyway, it turns out that reputation has no effect on completion rates:

Have enrolments increased or decreased over time I hear you ask. Well, there is a negative correlation over time with enrolments, so numbers on individual MOOCs have declined. This may be that with more MOOCs available people can find the one that suits them better, instead of everyone doing AI, but if you wanted to be all sensationalist (I'm looking at you, journalists) you could say it indicates that the MOOC bubble has burst:

Have completion rates changed over time? Yes, in the opposite direction, there is a positive correlation with completion and date. This could mean people are getting better at MOOCs, and if the trend continued those completion rates would be less of an issue. But I suspect it is related to the point above, in that there are more MOOCs around so people are taking ones more closely related to their interests, so more are completing:

(see http://www.katyjordan.com/timeline for more detail)

Does the course length have an impact on completion? You bet, a negative correlation with course length, so the longer your course, the lower the percentage who see it through. This is probably obvious, but it shows that if you get to 12+ weeks it's probably just some bloke in a shack in Arkansas left:

This raises the question, should we create MOOCs of 2-3 weeks in length (many of Open2Study courses are this long, which may account for the good completion rates)? And if we're creating 2 week MOOCs, how are those different from OERs? UPDATE - see Doug's comments below. He makes the point that obviously the longer a course goes on, the more people will drop out. So we can't say length causes drop out, although even in this respect, you might be better off having 2 x 3week MOOCs than 1 x 6 week one, simply because you catch more people (if completion is important that is).

And a tantalising one to leave you with: does gender influence completion rates? We can't tell as only a few data sets included breakdown by gender, 13 in this case. So we couldn't draw any conclusions from such a small sample, and Katy will probably tell me off for including this, but I do so as a plea to others to release data. From the sample we had, there is a possible positive influence on the percentage of female students and completion, ie women tend to complete more than men, who no doubt wander off to watch football during week 3:

We looked at a number of other factors including assessment type, MOOC type and platform, but the variation in sample sizes makes conclusions from these difficult to form. This data raises a number of interesting questions, which being correlations it can't answer on its own. For me these are:

1) Can we improve those completion rates from around 10%?

2) Are learners getting better at MOOCs?

3) Should we limit ourselves to short MOOCs?

4) What demographic data may influence completion rates?

5) The big one is, does completion even matter? (If it does, then MOOCs probably aren't the solution to your problem)

The Learning Design of MOOCs

I got some Gates funding for the MOOC Research Initiative to look at two things: completion data, and learning design MOOCs. The first part allowed Katy Jordan to finish the work she had started in mapping various factors from over 200 MOOCs that influence completion. You can see more of her work here, and I'll blog on that later.

My part has been using the tools we've developed at the OU for learning design, building on Grainne Conole's work. We use two main tools: the Activity Planner and the Module Map. The first maps student activity across 6 categories, eg assimilative, productive, etc. This is a good way to think about how you want to teach a particular topic. The second one looks at the resources you will need in order to create those activities, and maps them under four headings: Content, Guidance, Demonstration & Communication.

So far I've only done the Module Map applied to 14 MOOCs, chosen from a range of providers and to give both x and cMOOC types. I have created two representations. The first is the Resource Type which shows the range of different media used (video, article, text, audio, etc). Here's an example:

The second representation is more interesting and maps the number of resources used under each of the four categories given above. Here is an example:

Now there are lots of caveats around this. For a start it's a subjective interpretation (Sheila MacNeill and I did the mapping). Secondly it counts number of resources, not length. So 3 x 10 min videos counts as 3 whereas 1 x 30 minute video is just 1. But nonetheless, I think it gives a good indication of the overall weighting in a course. Here is an example of what I think is a fairly typical MOOC, which actually came up for both x and cMOOCs:

Here is an example of a more extreme end of the xMOOC spectrum, with a heavy emphasis on content:

And this is what a more cMOOC looks like (this is DS106):

I was keen to emphasise that it's not the case that one design is 'better' than another, but rather they are doing different things. Sometimes you might want the content approach, and other times you might want a more activity/community centric approach.

In my talk someone suggested they are like food labelling, and that's how I'd like to develop them. You could imagine four or five 'types' so learners, and designers, could decide what type of MOOC this one was. 

In the next phase we will develop some more of the activity profiles for MOOCs to examine in more detail what learners are doing (or what the course designers think they should be doing). But anyway, it feels like a fruitful way of examining them. You can find all the maps on this wordpress blog I've created.

The Battle for Open

<This is how we do protest in Wales>

We have a new issue of JIME out, which is a special issue on the OER 13 conference. There are some excellent papers, so go read them. Then, when you've read all those, read my opinion piece, The Battle for Open.

I'm going to write a book about this over the next few months, so if you think it's an awful idea, please let me know. The article is effectively the first chapter of that book, which sets out the overall argument. My argument goes something like this: Openness has been victorious in many ways, but at this point of victory the real struggle begins. If you look at openness in research, teaching, publication scholarship, then it's hard to argue that openness hasn't been successful over the last few years in establishing itself as a core approach in higher education. It isn't something just a few oddballs bang on about now, it has moved to the centre of discourse (and, more importantly, funding).

And yet it doesn't feel like it should. We aren't seeing David Wiley leading triumphant processions down Wall Street. The MOOC invasion and backlash is the most visible part of all this, but I think that is just representative of a wider story. I try to articulate why I see it as a 'battle', giving three reasons:

1) There is real conflict between different visions

2) There are considerable spoils to be won

3) The victor writes history, so there is a battle for narrative taking place

What I want to do then in the book is explore this victory/battle dilemma for a number of areas, and then come to some conclusions. My main hypothesis is that open education isn't an interesting subset of education now, it is education. And so if you care about education you should care about the direction that openness takes. We have made mistakes in the past by outsourcing issues, such as selling off university presses or buying in LMSs. This meant education no longer controlled or had much of a say in core parts of its key functions. We are repeating this mistake with MOOCs and commercial companies. Openness cuts across all of these and so I see it as the key principle that educators and universities need to engage with.

As you can probably tell, it'll be a struggle to control the polemic, but I hope it will be a reasoned, well argued book with plenty of research and evidence, and I'll keep the spitty, table-thumping to a minimum.

The iceland of Dallas

 

<Dallas deathstar in the snow - this may, or may not, be a metaphor>

I was at the MOOC research initiative conference in Dallas, Texas last week. As Jim and others have reported, we got caught in icemageddon, but that's a whole other (war) story. I'll be doing a few posts about the conference. It was a fantastic meeting, well done George Siemens, Amy Collier and Tanya Joosten for putting it together. I got to have some great conversations, and meet people I've know online for years. Which is by way of apology for my first post being a bit negative.

This one concerns one aspect of the conference that I am having difficulty articulating, so I'm going to try and work it through in this post. There was a data strand to the presentations, and I went to a few of these. There is some fascinating stuff being done, particularly when you have analytics on so many learners. But I also had a vague sense of unease about some of these.

They were often presented by super-smart, young computer-science researchers from privileged universities: Carnegie-Mellon, Stanford, Harvard. I don't have anything against super-smart, young or computer science people, or anyone from those universities (some of my best friends are young people :). But there was something a bit, well, cold, about it all (and not just because of the wind chill factor of -10). People were nodes, and they could be manipulated to move from peripheries to the center by tweaking certain elements. It was easy to forget you were talking about learners, and not sales of baked beans.

But I think we need this research, it's useful and can tell us a lot about what's going on. Candace Thrall was stuck at the airport so couldn't give her presentation, but a colleague gave it on her behalf. She mentioned that one of the transitions we were going through was from a theory-led one to an evidence based one. Prior to this Jim Groom was telling me about Mike Caulfield suggesting we were in post-theory now, where only the big-data mattered (this was from a book I think, if anyone knows which one, let me know). 

UPDATE: Mike has done a great post elaborating on the post-theory debate here. It explains what I was trying to get at in this post.

I felt I had a glimpse of that post-theory world, and I wasn't sure I liked it. We may have been too theory-heavy before, where the evidence was inconsequential, because hey, we have a nice theory. But the pendulum swing to lack of theory where we only care about the evidence seems to lose sight of the people in the system. So I guess my plea to the super-smart, young computer science researchers at ivy league institutions who are now getting into to education is - don't ignore the bearded old guy with a bunch of theories in his back pocket, we need those too. 

Like Dallas, education should be warm and welcoming, and the danger is that data fetishisation will make it like the post-apocalyptic Dallas I experienced.