The Ed Techie

What do all these numbers mean?

<image - http://www.flickr.com/photos/mmorgan8186/5946796450/>

Bloggers, or anyone who maintains an online profile, have an ambiguous relationship with visitor stats and data. On the one hand we like to dismiss them as meaningless, but then secretly feel chuffed when we can outscore someone. I've tried to promote them as one way of measuring impact, but with the caveat that context is important. For instance, if you're a blogger in a relatively obscure area, such as Barry Town football club, then your range is limited and unlikely to compare in absolute numbers with, say, a blog reviewing Apple products.

I recently passed 300,000 views on this blog, over about 700 posts - that's not as exciting as it might sound as I've been going since about 2006. My friend Liam says he gets about 200K a year on his mobile tech blog. Here are some more stats from tools I use:

• Slideshare - 220,000 views over 6 years on 59 presentations
• Blipfoto - 92,000 views over 420 entries
• Flickr - 49,000 views over 1,140 photos
• Citations on academic articles - 1,620 (according to Google)

The first thing to ask is how reliable is this data? Tony can probably answer that better than me, but I think the blog traffic doesn't take into people who subscribe via RSS, so in that case underestimates. At the same time it does include my own obsessive self-clicking, so that may even out. It may also include bot traffic which doesn't really mean anything at all.

But if we take the data as accurate for now, the more interesting question is what does it mean? I would like to make the case that 300,000 visits to my blog equals 300,000 careful, considered reading of my posts, but the data doesn't back that up. Google analytics tells me that my average visit duration is 57 seconds. I suppose that is just about the time it takes to read a blog post, but it probably doesn't compare well with time taken on reading academic articles. My bounce rate is high too, at around 85%, so people come in, read a post and then go elsewhere, but I think that's fairly typical for blogs, I'm not providing a newspaper where you want people to go from one feature to another.

Blipfoto probably has the highest return on audience per post, and does better than Flickr. This provides a nice comparison as I have linked my blipfoto account to twitter, so it automatically tweets when I post, but I haven't done this for Flickr (which I think of more as a repository). I get 219 views per post on blipfoto versus 42 on Flickr, so we could say that the effect of linking to twitter is a 5-fold increase in traffic (if that's what you want). But I would say that Blipfoto is the most personal of these tools and the least connected to my professional life, so I wouldn't argue for any professional impact here.

What I think we're struggling with here is the newness of all this. What is the value of a slideshare view versus a download of a paper? Is there a conversion rate? I've tried to keep a foot in both the traditional camp, through publications, and the new forms of dissemination, so these figures might offer a reasonable comparison between the two. Looking at my citations you could crudely say that one citation is worth about 200 blog views (or vice versa).

However, it's probably futile to try and measure these things. It's not as if citations are a particularly reliable metric anyway, so we shouldn't use those as a currency to convert everything else into. It will also vary widely depending on discipline, and individual. And more importantly, we already have plenty of systems that our career path requires us to game, so why invent more?

Make no mistake, turning altmetrics into performance measures linked to real things like promotion, job security, money would have negative effects. For example, there is a correlation between number of posts and views, so if I was purely chasing the numbers I should post more often, regardless of quality (what quality?), or I might think I could get better numbers from splitting this post into three separate ones. This already happens in traditional outputs, I remember someone advising me not to write books because for the same effort I could have 5 REFable publications. This return on investment view ignores the internal validity of the form. I also tend to think of blogging as a social function, connecting with friends and peers, but if I'm linking it to direct returns then it's another example of monetising relationships that Brian is so uneasy about.

But when I've tried to make the case for digital scholarship and tenure, it is something along these lines that I want to argue - that the impact you see here is as valid as the impact we have chosen to recognise and measure. So we need to start thinking about what these numbers do mean for academics - we just need to make sure we don't turn them into the thing itself, with all of us chasing and parading numbers. But maybe that's inevitable as soon as you shine a light on them, in which case, ignore this post.

Quality is the best gaming device

[There are some games you shouldn't play - image http://www.flickr.com/photos/whatwhat/8457309/]

Gamification comes up a lot in higher education. It's a by-product of a metrics driven economy, as soon as you meaure something, and add value to that measurement, then people will find ways to gamify it. The measurers often know this, and the trick is if you can get people to do the desired behaviour through gaming, then it's worthwhile, for instance good assessment will mean that learners end up acquiring desired skills and knowledge, even if they game the system. But too often, gaming itself becomes the focus.

The following are all aspects of gaming for higher education:

• Assessment gaming - are there ways of getting good scores in assessment with minimal effort?
• Teaching - are there ways of getting good scores in the National Student Survey? Can we find ways around contact hours?
• Learner analytics - if you start measuring performance through 'data exhaust', and attaching value to it, then it stops being exhaust, a by-product of natural behaviour and becomes a deliberate action. For instance if time spent in the VLE is a measure used to automatically monitor students, they will soon log on for long periods, while they go to the pub.
• Research metrics - the REF is essentially a very expensive exercise in gaming. Universities game it by buying in good REF researchers just before the REF deadline, sacrificing some staff for others (eg put your name first because that person isn't going to be refable) or moving non-eligible staff to teaching contracts. Individuals game it by getting their papers cited (self-citation, mutual citation circles), getting students to publish papers with their names, and publishing in targeted journals.
• Online identity - methods of increasing blog traffic, gaining reputation, getting attention.

It's quite a depressing list, you could spend all of your time being concerned with gaming and not with the actual work it is meant to represent. All these was brought into focus for me by this great blog post from author Paul Raven. A wannabe ebook author writes to him asking for advice on how to get improve sales. The potential author asks:

"Why not start a Facebook Group: “The EBook Likers?” Join the group, and you pretty much agree to go around and LIKE all the other Member’s eBooks which are on Amazon. The Power of LIKE!"

And he goes on, excitedly:

"Something like this goes on every day at Twitter... The I’ll-star-your-tweets-if-you-star-my-tweets factor. Most tweeps on Twitter rarely, if ever, favorite any tweets at all. But there is an in-bred niche of super-favoriters who go to Favstar to track exactly how many stars and retweets each of their tweets get.

By analyzing the data, it becomes clear that the Favstar Superstars don’t achieve their status with superior content, but with superior networking. Take any Favstar Superstar and examine several of their tweets in detail, and you will find the exact same avatars always at the beginning thirty spots, with just a few odd avatars; the further up the number of stars a tweet gets, the more variety in avatars."

He continues with lots of such possibilities and questions. As Raven points out, he never mentions his writing once, and asks nothing about the craft of writing. It's a sad example of how gamification becomes the focus itself.

Raven offers this sage advice:

"There’s one thing that might make your book sell better, though — and that’s making it a better book. Or making a new book that’s better than the last, and another one that’s even better than that, and then another and another. And sending them out, whether to agents or editors or straight into the whirlpool of the Kindle store, and letting them speak for themselves, while you wait at home patiently, writing the next one."

That is the key for me. Whether it's blogging, teaching or writing research papers, the thing to focus on is the quality of the activity itself. Sure, there are some things that are easy and worth doing - I use images in blog posts because I think it may attract attention, and I tweet a link to my posts when I publish them, I tend to blog during weekdays as no-one reads at weekends, etc. I want people to read my posts, I just don't want to fundamentally alter the nature of what I do to achieve that end. So, for instance, I don't play games such as trying to create controversy to get hits, or breaking posts up into shorter ones on the basis that more posts create more traffic, or spend time promoting posts in other forums. 

But it's an easy trap to fall in to - the next time I find myself thinking, or sitting in a meeting discussing, methods of gamification, I'll remember that needy writer in Raven's post. That way lies madness.

 

5 things I think about Learning Analytics

<Ed Techs go in search of data in the wild>

I am at the 1st Learning Analytics conference in Banff, which has been interesting. I came not sure of what it was, or what my take on it was. The conference has been good, very interdisciplinary in nature (for which you can read 'I didn't understand some of it'). I'm still not sure about a lot of it, but here are five things that have occurred to me over the course of the past few days. I don't believe they're strong enough to say I've learnt them, but rather they are things I now have come to a tentative viewpoing on.

1) Analytics for learners is distinct from analytics of learners. Erik Duval gave a good talk which listed some approaches to empowering learners by providing data and visualisations to them. I think this is potentially very powerful, both in formal and informal settings. Showing learners where they are in respect to others, or giving them recommendations in terms of resources or people, or providing an overview of their activity that helps identify aspects they need to work on, etc. All of these seem like useful things to do. What I am less sure about is the analysis of learners in institutional systems because it will allow them to make efficiency savings.

2) There is a range of intentionality and effort. Tony was talking about the types of things you can build by simply taking one piece of data and then plugging this into various tools. The example that stuck for me was the way by simply picking up on a hashtag he could create a representation of a community, construct a blog roll, mine delicious links for of the people in that community to find resources, etc. So from one simple action we can at least infer a good representation of a community. And to join it all the individual needs to do is tweet with that hashtag. This is distinct from the development of very specific tools, such as a virtual machine that sits on a student's machine and they do their course work within it, or a tool like Cohere. Here the proposition is that we will give you a tool which is useful, and from it we will be able to gain richer data. But the effort required is much greater. I'm not suggesting one approach is better than another, merely pointing out that they are quite distinct. I think our instinct is to usually go for the latter for rich data but the former gets at a wider network.

3) Analytics may reveal some uncomfortable truths. A couple of presentations suggested that when you look at the data then you see that students either don't spend very long in your carefully created content and hop off to Google to find their own, or that doing some activities had no impact on exam performance. This prompted David Wiley to ask 'why do we do them then?', and I suspect analytics may reveal many such uncomfortable moments for course designers, in a way that end of course surveys don't currently. You know that resource you spent lots of money on and thought was great? Students spend 15 seconds on it. That big activity we devised? Has no impact on performance. The real question then becomes, what do you do with this knowledge? It may be that you still wish to persist with some elements because we believe in them pedagogically, or because they develop certain skills. But it may also mean that we find the way we think students study our courses and the way they actually do are quite separate.

4) A little knowledge may be a dangerous thing. Related to the previous point, I also worry that interpreting this data into appropriate action will be a difficult task. And at some point the data may fall into the hands of a senior manager or accountant who will look at it and make a simplistic conclusion. One can imagine decisions such as halving the course staff only drops five percent points in average, or reducing course content by one third results in only 10 more students dropping out, so let's implement the cuts. But there is a bigger picture beyond the pure data, and so viewing it in a holistic and contextual manner is essential.

5) We shouldn't put off ethics issues. There are obviously some rather Orwellian overtones to learning analytics. I don't think this need be the case, but there is a strong commercial interest in this and there is also an element of 'let's record everything because we can'. But I think there are also powerful benefits for the learners too, and whenever we are online we are casting off data anyway. For the academics and researchers we should at least draw up a set of principles from the start that we adhere to. If others want to use data for nefarious means, that's up to them, but here are the things we believe in. For example this could include the premise that analytics is performed with the benefit of learners as a priority, that we are always transparent, that the learner owns their data, etc (Erik pointed to the 4 principles of the now defunct Attention Trust which are a good starting point).

So that's the thoughts I have for now. I also learnt that Banff is a) beautiful and b) flipping cold.