The Ed Techie

The RAE - Packt Like Sardines

"After years of waiting
After years of waiting nothing came
And you realize you're looking,
Looking in the wrong place
I'm a reasonable man
Get off my case"

	Packt Like Sardines In A Crush
	Adblock
	Found at skreemr.com

The results of the Research Assessment Exercise (RAE) were announced last week. As readers may know, I'm not keen on it, so this isn't an objective view, but I thought I'd explore the motivation behind it, and the problems with it.

I have three main objections to the RAE:

1. It is overly complex
2. It is expensive
3. It is fundamentally flawed

Complexity

What is the justification for the RAE? There are two main reasons put forward:

1) It allows transparent and objective allocation of research funding to universities

2) It provides UK academics with a recognised standard for their research, which is transferrable between universities.

Let's take this second argument first - I guess the idea is that academics can easily move between jobs and have their research recognised. Except, this year the results are so confusing that no-one knows what they mean, and there isn't an individual rating. So from an individual's perspective it's not much use - except maybe to say 'I work in a 4* unit', or 'I want to join a 4* unit'. But as this year's results can be interpreted in a variety of ways, that doesn't mean much anyway. For instance, my own unit is either 3rd, 9th or 24th in the UK, depending on which way you tweak the figures. That's quite some margin of interpretation.

Expense

The main justification is so that the government can easily allocate research funding. Now there is probably something in this, but the system they have produced is so complex that it defeats the object. I asked on Twitter the other day if anyone had done a return on investment analysis for the RAE. For the unit I was involved with (Education), there has been full time staff appointed to it for over a year, a number of high level staff working on it part-time, an administrative and database system created, and plus each academic has to work on their own individual submission.

When we submit bids for research grants we are required to give a full costing of time, plus there is usually 40% overheads added in, and increasingly we are asked to estimate 'opportunity costs', ie what we lose by doing this when we could have been doing something else. For the massive, distributed work of the RAE, which requires often your best researchers to help coordinate, these costs must be considerable, and yet no such return on investment is required. And that is just one unit of assessment - there are many others across the university.

Why? Because as Simon Caulkin in the Guardian puts it:

"the RAE is a potent symbol and vehicle for the bullying top-down managerial culture that has steadily eroded both the quality of working life and results in much of the public sector."

The RAE is an agreed con - it has value because we say it has value. A university has to participate because it would not be able to attract some funding, it wouldn't be able to get new academics if it didn't, and it needs to allow current academics to get a rating. Academics need to participate because it forms part of the agreed career and promotion profile. A few academics might be able to say no, but it's only if universities stop playing the game that threatens the system, and as money is involved, that won't happen. Added into the mix is the academic publishing business – this exists largely because of formal assessment exercises like the RAE. Because the RAE recognises this type of output, academics are forced to publish through this means

But, because it's so expensive to participate in fully, any notion that it creates a level playing field is a nonsense - the better universities can afford to put the administrative effort into it, to release people from teaching to concentrate full time on preparing the narrative, and so on.

Fundamentally flawed

The RAE won’t be the RAE anymore, partly in recognition of the problems above, but there will be some assessment exercise. These first two issues can potentially be addressed by modifications. More damning though is that the whole attempt to quantify research outputs and link these to funding is at its heart, flawed. Here are the main flaws:

The experimenter effect

It is ironic, to say the least, that a research assessment exercise fails to understand a basic of research, namely the experimenter effect. The very act of measuring changes behaviour, and doubly so when it is linked to money. Academics have to play the RAE game – and this inevitably means a focus away from teaching, or even doing research that won’t directly or obviously link into the RAE. John Naughton relates this anecdote:

“In one major academic department I know, the most creative and original member of the department was excluded from the RAE by his colleagues because his pathbreaking work “didn’t fit the narrative”

The categorisation error

I work in educational technology, which was grouped (or lumped might better describe it) in with more general education. Actually there is little in common between the two in terms of what they value, what they deem serious outputs, what are the major research questions, etc. Educational Technology felt very much like a poor cousin and often one had the feeling of trying to twist your research to fit ‘their’ criteria. And this is repeated across many domains. The point is not that they may have the categorisations wrong, but that no categorisation can be correct. This is particularly true of the highly innovative research, which by its very nature won’t fit into a pre-existing category.

Measuring the unmeasurable

The RAE is very New Labour, with their almost Stalinesque obsession with quotas and direct measurement. The problem is that research is rarely like that. One fantastic paper is not worth two good papers. And the more the system tries to accommodate these various factors (e.g. with factors of esteem as it did this time), then the more convoluted and cumbersome it becomes. And if we add in blogging, online activity, creating software, youtube videos, etc then it becomes even more complex. How would we measure Michael Wesch’s output? By his research publications or his YouTube views? Again, there is talk of addressing this next time round, but it will always be chasing the game.

And this leaves academics with an unenviable choice – do they play the game and concentrate on RAE type outputs or do they work on creating new forms of identity and communication, which may be more relevant (not to say interesting), but run the risk of being ‘invisible’ to any official view?

What alternative is there?

So maybe the RAE is inevitable. We need to allocate money to universities for research and as soon as we do this, a complex, unsatisfying system will always follow.

 Here are my alternatives:

1. Don’t allocate research money – instead make it very easy and quick to bid for.
2. Allocate a set amount to all universities, based on their agreed research rank (eg from the last RAE), and then have another set which can be ‘won’. This has the advantage of at least allowing universities to know how much they will be receiving, whilst still allowing them some ability to develop.
3. Allocate an amount based on some very easy to measure unit – eg number of publications drawn from a database. It’s flawed, but we all know what it is and can get on with the rest of it easily.
4. Abandon categories and allow these to flow from the actual tags writers use.
5. Universities ignore whatever comes next and instead promote their own research and concentrate on getting funding.

None of these solutions is perfect, but they have the virtue of being cheaper and less pointless than the current system. After all, I’m a reasonable man…

And here is AJ Cann's more succinct take on it:

The beautiful use of statistics

If ever you wanted to enthuse someone about data and statistics then show them Hans Rosling's TED presentation 'New insights on poverty and life around the world.' The stories you can tell with good data and visualisation tools are beautifully illustrated in this talk. And I love the phrase 'grandma verified statistics.'

-->

(Hat tip: D'Arcy Norman)

The RAE - time to make a stand?

I had to do some admin today for my RAE submission (Research Assessment Exercise - the formal process for judging an individuals and institutions research output in the UK). As well as being tedious, the sheer futility of it depressed me.  In some areas I'm sure the traditional academic journal is useful, but it is almost entirely irrelevant in educational technology (see my previous rants on this: here and here).  I do read academic research articles (if they're available online - I never trawl through print ones), but they represent only one part of what constitutes my scholarly network, and an ever-decreasing one at that. The debate, articles, opinions, experiments, developments and people I want to engage with are found predominantly in internet native fora now, primarily the blogosphere, but also social networking sites, wikis, podcasts, video clips, open educational resources, etc.

I've mentioned before that blogging has seen a decline in my academic publishing output. This is not really due to time pressures, but is more related to motivation. There were three main reasons to publish in academic journals previously: the formal recognition required for promotion, CVs and the RAE; as an outlet for creativity; to gain recognition and reputation amongst your peers.

Now blogging satisfies these last two, and to a much greater extent than the very narrow confines of academic articles. I used to wade through the tedium of the proper format just in order to write the one paragraph that was of interest, because I had no other outlet. Now I do, so why would I put myself through that very poor fit of writing style when I can find a more natural outlet through this blog?

It's not just that the RAE is irrelevant that irritates me (although I would like to know what return on investment it provides given the enormous amount of time that has been spent by my university and others in preparing for it, with committees, reviewers, systems, processes all being put in place and then the distributed time from each individual), but more that it might be actually harmful. There are two arguments you can make against it. The first is that it is measuring the wrong thing, or at least not recognising lots of other activities (and yes they did say they were expanding the criteria this time, but I doubt 'my blog' would carry much weight, they key being that the people behind the process don't understand the digital world). One could therefore lobby for the recognition of blogs as a formally recognised publication. For RAE rating read Technorati rating. The second argument you can make against it is that it is just the wrong thing to do. In classic Heisenberg mode, the very act of measurement changes what they are measuring. You are not recognising good research or scholarly activity but rather good RAE game playing. And life's too short for that.

The academic publishing racket is a good example of this last complaint. By measuring academic publications, the RAE was getting a proxy for research ability and output, or so it would hope. But by the very act of measuring it, the RAE has legitimised the existence of journals in a time when they are finding their editorial and business model increasingly irrelevant. Many journals exist now not because of the dissemination of research, but because of the RAE.

There are some arguments for why you need something like the RAE, such as how do you standardise so that someone can move between jobs and how do you flow research money to universities? I'm less and less convinced by them, for the first one I would argue that in a connected world you should be able to find out much more about an individual than the RAE will tell you. If I ever apply for another job (not very likely anyone would want me) then I would like to just tell them to look at my blog. If they like that, then they'll like me.  The money is the real issue of course, and the finances of higher education have always possessed the easy logic of quantum physics for me, but I'm sure there are other, more efficient, methods than this one.

As a Prof I wonder if I should make a stand on this - I think it's very damaging for younger academics, who feel they have to engage in it to gain promotion. This is at the detriment of doing interesting things. Developing software, blogging, exploring web 2.0 applications (which don't fit the funding model well - they're more suck it and see, rather than write a 50 page proposal, get ten partners, then see in a few years time), innovating in teaching - all these would be better things to do for academics in my area (maybe the same isn't true in Classical studies, say, although the last one should be).

It all feels very last century, or last 5 centuries for that. The lack of recognition for the large socio-technological change we are going through (it's really a paper exercise despite attempts to dress it up - at heart what they want are traditional, printed journals that sit on dusty shelves), doesn't reflect well on academia. My friend is a graphic designer, and when he started in the 1980s he was always coming home with the top of his finger cut off. He explained that this was caused by the scalpel when he was cutting out photographs in order to create montages and images, which would be glued together and then photographed again. His skill at this was well recognised in his company. Within 5 years all those desks, darkrooms, scalpels and glue pots were gone, and the Mac reigned supreme. The RAE feels like we're still being judged on our scalpel skill.

Academic publishing - what's the point?

I had a message today saying that an article (The distance from isolation) I submitted to the journal Computers and Education has been published. Good news one might have thought, except check the date it was submitted - November 2004. That's three years to publish an article (admittedly it was available online a year after submission). Now, it's a good article and all that, but it's hardly current. For a start I address three technologies, but web 2.0 doesn't get a mention since the phrase hadn't been coined then. If I was writing it now, this would certainly form the crux of my argument (which thankfully has held up reasonably well).

Not all journals take this long to publish, but it does make you wonder what is the point of journal publication? In a classic example of the Heisenberg uncertainty principle, it seems that the function of journals is now to feed the Research Assessment Exercise. Initially the RAE used publications as an indication of the quality of research. Now it is the justification for the journals themselves. I learn a lot more from blogs now than I do journals, and I have also noticed that since keeping a blog I have published fewer 'proper' articles. Surely journals can't survive when they a) cost a lot of money, b) are out of date and c) don't facilitate the type of engaging debate you find in the blogosphere.

I know people will put down the marker of quality here, but I think a blog posting does undergo a form of peer review - if you post something stupid you can bet someone will tell you. And there are a host of metrics that indicate the quality of a blog, such as your technorati authority (mine remains doggedly low).

I've had some discussions with people at the OU about whether we should start encouraging and recognising blogging as an academic activity. I would like to see this,  but I don't want the RAE or the OU to start formalising the process, since this will stifle it. The moral here seems to be that formal metrics kill creativity. From a career perspective it is best to have a high ranking in the former, but for your own devlopment (and ultimately the health of the organisation), you want to be in the creativity camp.

OpenLearn conference

Patrick McAndrew (keep blogging Patrick!) sent me a call for papers for an open learn conference here at the OU, which might be of interest to some readers:

The call for papers is open for the openlearn 2007 conference: researching open content in education. Deadline for submission by short papers (~1000words) is 31st May 2007. The conference will be held 30-31 October 2007 near The Open University in Milton Keynes, UK. There will be no charge for attendance with priority for registration given to those responding to the call for papers. Selected papers will be developed for publication in a special issue of the Journal of Interactive Media in Education.

The four main themes are:

• Research agenda,
• Sustainability,
• User experience,
• Software and tools

Further information can be found at: http://www.open.ac.uk/openlearn/openlearn2007

Forms for submission of papers can be found at http://www.open.ac.uk/openlearn/openlearn2007/papers.php and should be emailed to [email protected] by 31st May 2007.

Why I'm not putting in an FP7 bid

A few of us have been working on a proposal for the new framework 7 funding from the European Commission. Our idea was around the central question of how do learners work in a world where content is free, can be reused and remixed. What type of learning environment is needed and how does this affect the learning process? After some discussions with people at the commission it became apparent that they weren't keen on proposals that focused on, or even mentioned, content. The feeling is that this was addressed in frameworks 5 and 6, which looked at learning objects. For us this meant we couldn't make a virtue of the openlearn work, which was central for us.

But more significantly I think the division between content and environment is not a valid one to make any more (if ever it was). In an RSS/AJAX world I find the distinction between tools and content is increasingly blurred. As educators are fond of pointing out 'content isn't everything', and I am fond of responding 'it may not be everything, but that doesn't mean it's nothing'. The interesting research question for us is the interplay between the two in a web 2.0 world. How do learners create, share, remix, subscribe, blend content in an environment that they create, share, remix, and blend? This is the key difference with e-learning 2.0 - what you do with content you do with the environment too. Initially you had the content and the environment fixed and packaged. Then there has been some loosening up in terms of content with more exploratory pedagogies. Now the central control over the environment is liberated also, so the key thing to explore is the interaction between the two.

Maybe the next round of funding will be more amenable to exploring this, but given the enormous effort it takes to put together a European bid, I won't bother this time around when it's clear that they are after something different.