The Ed Techie

The random REF

<Image from gringer http://www.flickr.com/photos/gringer/5096129532/>

As a senior member of staff (I know, how did that happen?), part of my duties currently include reading papers for the UK research assessment exercise, the REF. I've moaned about how this reinforces traditional publication models previously. This is a more general moan.

I think the REF has muddled objectives. It aims to make academics accountable to some extent, but primarily acts as a means of allocating research funds. It may have some secondary aims, or you could classify these as indirect consequences, such as reinforcing the position of the main publishers and maintaining the status of Russell Group universities.

When you have multiple objectives the resultant process is often unsatisfactory. So if we ignore the secondary aims (although I wish the Finch report had been brave enough to make clear reference to the REF in undermining the move to open access), then how might the first two be best served?

Let's assume for now that you want to go along with accountability drive. There are of course plenty of reasons why this may not be a good idea, but that's for a different post. As someone who has to read papers and try and judge them, the real problem with the REF is that it brings in quality judgements. I thought that educational technology was a special case in that it didn't fit well into a broader education category and so the people making assessments weren't sufficiently well acquainted with the subject area to make decent judgements. But I've spoken to people in other disciplines and everyone feels the same. The specialisation of research means that no categorisation can be fine enough to really capture it. And it's a fruitless task anyway, who can judge the impact or quality of a paper? I think you can spot a poor paper easily, and maybe a really excellent one, but 90% of the rest sit in the middle so judging whether something is a 2* or 3* is essentially a coin toss.

So, my suggestion (if you want to do the accountability thing) is to simplify it and just apply a simple gateway threshold. So, for example you could take the REF guidelines - in four years someone who has a research part to their contract should be expected to publish four refereed articles, have got one piece of research funding in and supervised one student. Against this simple criteria you can then have accepted trade-offs: running large university projects, digital scholarship equivalents, personal consultancy as funding, etc. But you remove the quality judgement, it's a simple measure that this is what a research active should be expected to deliver. I don't necessarily agree with it, but we'd all know what the rules were.

The next element is to allocate research funding. You could simply do away with direct funding and have it all come through research councils, as they do in some countries. Or you could distribute it evenly, or according to number of staff who've passed the threshold above. But I've got a more interesting suggestion - there was some maths research (from those crazy Italians) recently that argued that random selection of some, but not all, politicians would benefit democracy:

"the introduction of a variable percentage of randomly selected independent legislators can increase the global efficiency of a Legislature, in terms of both the number of laws passed and the average social welfare obtained."

I wonder if the same might apply to research funding? So some is allocated according to worth (but using a simple system), but a percentage is allocated randomly, on the proviso it is used for research. This might well generate innovative research we wouldn't see otherwise. And it prevents a self-reinforcing group who get caught in version of group think. And it'd be more fun.

I know what you're thinking - why isn't he in charge of the next REF? We can only shake our heads.

The research Hussites

[This is another from my catalogue of strained metaphors, and my grasp of religious history is rather tenuous, so I'm sure people who are better acquainted with the subtleties of Hussite history can point out lots of flaws with it. But take the surface points as of interest.]

I've been giving a talk recently called "Digital Scholarship: 10 Lessons in 10 Videos". I'll blog these in detail later, but one of my lessons is that we should rethink research. By this I mean we have a certain attitude towards how research is conducted, which was shaped prior to the arrival of digital, networked and open technologies. Some of that attitude still works, but there are also a host of possibilities that remaining wedded solely to that view prohibits.

An aspect of this that interests me is the Do It Yourself (or Do It Ourselves as Tim O'Reilly has it), do it now approach. In my talk I give three examples from my own experience:

• Start your own journal - for example my MetaEdtech Journal
• Build an app - eg our Facebook apps
• Interrogate data - such as Tony's playing around with tools and social data

But you can think of many other examples: Jim Groom's DS106 mega experiment or the MOOC approach from George Siemens and co can be seen as research and experimentation in the open.

What I find fascinating about these is that they didn't need permission. As Larry Lessig points out in his review of The Social Network, it is this removal of the permission filter that is really significant:

"what’s important here is that Zuckerberg’s genius could be embraced by half-a-billion people within six years of its first being launched, without (and here is the critical bit) asking permission of anyone. The real story is not the invention. It is the platform that makes the invention sing."

Back to research then, and our ingrained attitude goes something like this:

1. Come up with an idea
2. Write a proposal
3. Get funding
4. Do research
5. Publish

Obtaining the funding is often an absolutely necessary step - you can't build a Large Hadron Collider in your back yard after all. But it also performs another function - it validates research as something worthwhile. So ingrained is this approach, that if you don't get funding, it doesn't count as research. We don't question it anymore. 

But with the arrival of new easy to use tools, open data, a network of peers and cheap or free storage, you can do a lot of research now without funding. The video I show for this lesson is this one from Derek Sivers. He's talking about starting a business, but a lot of it could apply to research also:

So what's all this got to do with a 14th Century Czech Priest? Well, Jan Hus argued that we didn't need priests, that everyone formed part of the church and the Papal hierarchy should be undermined. I'm not making any comment on his religious argument here (he says, trying to avoid offending anyone). There is an analogy with this approach to research though. For the Catholic churches read research councils. We are all the research councils, we don't need permission to conduct our practice and we don't need their approved channels to reach the audience. 

My argument is not to overthrow research councils, we still need them, but to propose that often there are low to zero cost alternatives available which might get at some of your research question. We should consider these at least without always defaulting to the funding = research model.

Mind you, poor Jan didn't have a happy ending, so maybe I should stay quiet.

Bridge 2 Success

This is the third in the what I'm up to at the OU posts. A while back we were part of a consortium that won a bid in the Next Generation Learning Challenges funded by the Bill and Melinda Gates Foundation, Educause, & The Hewlett Foundation. The partners are us (IET at the OU), MIT, the Anne Arundel Community College and University of Maryland University College (UMUC) - the project is Bridge to Success.

It's aim is to use OERs to help community college students who struggle with study skills, and math(s) in particular. So we've taken some OU courses and these have been adapted in the US, and made available through our OpenLearn labspace. But this is about more than just creating content, so these materials are used in classroom instruction, and we are researching how students and instructors find the process.

My role is to head up the research work programme (Patrick McAndrew is the OU lead and does all the real work). One of the things I'm interested here is in the process of adaptation. We have given our materials to US partners to adapt, and what we are looking at is the type and degree of adaptation that occurs. There are at least four types of adaptation that take place:

1. Simple context - changing £ for $ is a simple way of thinking about this
2. Language adaptation - changing the nature of the instruction that is given in terms of the style used
3. Content adaptation - reworking, deleting or adding content so that it meets the needs of a US maths curriculum
4. Pedagogic adaptation - altering the nature of activities and instruction based on knowledge of the types of students

Now, you probably want to get away with just doing type 1 adaptation as that is fairly straightforward and takes the least time. But you still need subject experts to do this, and it quickly becomes apparent that it bleeds across to the other types of adaptation too. It is all context. 

So what will be interesting to see is just how much adaptation takes place, whether some instructors adapt more than others, and if this has any impact on student performance. There undoubtedly comes a point when you have adapted something so much, it would have taken as much time to create it from scratch. The argument here then is whether the OER approach is worthwhile, if it takes as long to adapt as to create? Even if this is the case (and we haven't found that to be so yet, but it's still early), the use of OER may lead to better material than creating from scratch - this would be a tough thing to test, but maybe for another project.

Those are the kinds of things I'm interested in, beyond the more practical (and worthwhile) outcomes of hopefully improving student performance, confidence and understanding.

Here is a presentation from MIT's Brandon Muramatsu on the project if you want to know more:

Demystifying OER and Bridge to Success

View more presentations from Brandon Muramatsu

Ten digital scholarship questions for researchers

<Image http://www.flickr.com/photos/elycefeliz/3262326159/>

In my talk the other day for the Change MOOC, and in my book, I argue that digital scholarship has the potential to alter the way we conduct research, or at least to add some new tools to the toolbox.

Now checklists should be treated with even more caution than a finding that 3D improves learning, so take this as some things to think about rather than a definite checklist you can submit to your research funder. I'd be interested to hear of other suggestions, and also whether this is all stuff people are doing anyway. My feeling is that when it comes to writing research proposals we're all locked into a set of existing behaviours, and more significantly, the process is so arduous that you focus on surviving the process itself and not on thinking beyond it. In other words if the research funders aren't asking for it, we don't consider it. But if we (and by we, I mean researchers) want to take ownership of our own practice, then maybe we should.

So here's my list of digital scholarship type stuff to consider for your project:

1. Do you have an open access publishing policy for the outputs of the project?
2. Will you make the data openly available, or is some of it not appropriate to do so? If so, where will you put it?
3. Will there be one main individual in charge of social media communication or is it just distributed across the group?
4. Will you have specific twitter/blog/youtube accounts for the project or do individuals in the project have a high online reputation it would be better to utilise?
5. How will you incorporate analytics into the project? Do you expect to get a certain number of views, dwell time, global distribution, etc on a main site?
6. What media will you use? For example, will you create a 'trailer' video for the project and an overview of the findings?
7. How will you archive discussion around the research, eg twitter hashtag conversations?
8. Will you provide a curation service, eg a Scoop-It page of relevant resources as you go along?
9. Are there new methodologies you will employ, eg crowdsourcing?
10. Is there a planned release for findings throughout the project? Will any aspects be not open for dissemination eg via twitter or blogs?

I'll stop there because ten is a nice number.

A quick and dirty REF

I am currently engaged with working on the REF submission for my Institute. I fully understand why we do this, in cash strapped time in particular, research money will make a big difference. It is also almost impossible not to play the game if you want to be recognised as a research institution and attract staff.

But it is a horrible, pernicious process that seems deliberately designed to kill innovation, perpetuate existing models, waste time, focus energy on gaming the system and keep publishers happy. If you wanted to design a process with the sole aim of restricting the adoption of new methods and technology, then you could do a lot worse that the REF.

Not every country has an equivalent of the REF, so we shouldn't view it as a necessary evil. But, let's say that for whatever reason, something like it is going to persist. Two questions then arise: can it be better and can it be easier?

I'm not going to answer the first of these questions. It would take too long, and I've covered it lots of times before. But in a conversation with Tony the other day we pondered how the REF might be made easier and a hell of a lot cheaper. The total (estimated) cost of the 2008 RAE in England was £56.7 million, which compares with the £1.5 billion of QR research funding. They estimate costs for the REF will be similar. My guess is this figure is low, and may not take into account all of the meeting and individual time associated with the process, and probably not opportunity costs also. In a review of the 2001 RAE, for more teaching oriented institutions it was found that the "RAE is over 16 times less efficient than the norm". I don't think the REF can be argued to be more efficient as it largely follows the same methodology.

So here is my (well Tony's mainly) proposal for a quick and dirty REF:

• Research councils monitor where money goes and evaluate research projects
• Publishers have databases of authors and citations

The REF then centrally datawarehouses these sources, adds in various weighting factors, creating a score for individuals, which is aggregated for the institution to give their REF evaluation (maybe by subject or maybe across a whole institution).

It uses proxies for research quality, and is focused on research funding and formal publications, but hey, so is the current REF. If we're going to be forced to play that game, then couldn't we at least do it in an efficient manner? It would at least free up a lot of time to you know, actually do research.

It would at least be worth doing a pilot based on this to see how far it differs from the very labour intensive system we have currently, don't you think?

 

More brains! The perpetuation of the zombie scholar

With Jim Groom I'm writing a chapter for the Zombies in the Academy book. It's at an early draft stage, but thought I would share it in this wiki.

My argument is that the context we have set up in education for all aspects of scholarship is like a contagion that works against innovation. This was an idea I explored in my researchers and new technology post and which forms a theme of my upcoming book (but with no zombies). There is an analogy here with the manner in which zombies turn all-comers into one of their own, which is exemplified by this clip from Dan O'Bannon's Return of the Living Dead:

 

The scholarly discipline is in ever more need of fresh brains to feed itself.

But I propose that, unlike zombification, there may be an antidote: the adoption of the fast, cheap and out of control technologies espoused by Brian Lamb.

Anyway, take a look at the paper and if you feel like contributing let me know.

The lack of uptake of new technology by researchers

Yesterday I gave a presentation for George and Stephen's open course PLENK. You can see the recording of the session here. The slidedeck is below:

Research, technology & networks

View more presentations from Martin Weller.

I have been writing a chapter on research and how researchers are (or rather aren't) using new technologies for my digital scholarship book. Several good surveys and reviews have been published recently looking at this, and the overall picture is a rather depressing one (see some references in my Mendeley list). While there are islands of innovation, generally researchers are making little use of new technologies and are very cautious and conservative in their adoption.

I talked about why this might be so and suggested the following reasons:

• The stranglehold of publishing - tenure and reward are still linked to publishing in academic journals, so spending time on other forms of communication is not directly encouraged. This is reinforced through research quality assessment exercises which are closely allied to publishing, such as the REF.
• Wasting new blood - related to the above, in other sectors new recruits are seen as the people who bring in new ideas. Yet in education because tenure is linked so closely to traditional outputs, new researchers are positively discouraged from spending time on other endeavours. So the very people who might engage in transforming practice are discouraged from doing so. This leaves it to tenured academics to engage in innovation, but these have mostly been successful through more traditional means so their history tells them to carry on this way. The pool of potential adopters then is decreased by the research context we have in place. I suggested this was quite a harmful environment to create.
• Antithetical to research ethos - I also put forward the suggestion that in science in particular, research is about predicting outcomes, and much of web 2.0 (or whatever you want to call it) is about unpredictability - you don't know what will happen with your outputs, or through a network.
• Research is the core practice - despite how much academics might say they value teaching, or public engagement, it is still research that is most highly valued. This means it is the area people are least willing to risk their reputation in and want to control most tightly.

I then suggested some themes that I thought were emerging, but that is for another post.

Of course one might argue it doesn't matter. Researchers will adopt new technologies if they are effective, and there is an underlying assumption in my talk that adoption is a good thing, which isn't necessarily true. This may well be the case, but there are three responses to this which at least bear consideration:

• The impact of the negative context - if research adoption of new technologies (and associated practices eg open publishing) is being impaired by the financial/reward structure we have in place, then we need to address whether that structure is doing research a disservice in the long term
• Powerful tools - I know through personal experience that even when people have been dismissive, or cautious of a particular tool, when it is demonstrated in the right context, or they appreciate its possibilities, then they will find uses for it in their academic work that I had never anticipated. This isn't true for everyone or for all uses, but it is worth chipping away at misconceptions and demonstrating potential.
• The role of the researcher - as higher education seeks to demonstrate its value to society, it strikes me that it is potentially dangerous not to be leading on the use and development of new tools, and particularly on what they mean for knowledge construction and sharing, rather than leaving it to others.

OU conference - evaluation

Following on from the previous two posts looking at the OU conference, this final one in the trilogy looks at some evaluation. I am particularly indebted to Karen Cropper, Rebecca Ferguson and Juliette Culver for doing much of the analysis for this post.

Evaluation took four main forms:

1. A questionnaire of attendees in surveymonkey
2. Statistics from cloudworks
3. Analysis of twitter users adopting the #OUConf10 hashtag
4. Analysis of the elluminate sessions

Questionnaire

There were 102 responses to the questionnaire. Below are some charts representing salient issues:

 

Most attendees were central staff, but there was a significant audience that had no connection to the OU, and a mix across other categories.

 

Attendance was split evenly across most sessions.

 

Most people would not have attended had it been face to face.

 

People usually combined it with an element of work.

 

The content, technology and organisation were all rated good to excellent, with discussion a bit below these.

 

In general participants thought that the conference allowed a community to form, and thought that the open nature of the conference worked well. Opinion was divided as to whether interaction was less than with a face to face conference.

 

There was strong agreement that they would attend another online conference.

We asked attendees to give us three words to describe the conference and the resultant word cloud is below:

 

Some issues raised by participants included:

• Separating out time to attend an online conference
• Perception an online conference is ‘techie’
• Possibly having shorter events but spread out over more time
• Allowing more open sessions and not just presentation

Some representative quotes from respondees are:

• "liked the accessibility factor - it saved a lot of money having to travel to Milton Keynes”
• "The moderators were excellent; very engaging, kept things going well, did a sterling job. The discussions in the text box during sessions was very good indeed. Generally the speakers and the topics covered were of great interest and engaging.”
• “This was an exceptional way to involve people outside the University and particuarly useful for CPD for those who can't afford to travel or whose institutions won't pay for them to go to conferences. A real credit to the OU's open learning ethos.”
• “thanks for the opportunity to take part. Not being part of the OU myself I welcomed the 'open invitation' and got loads out of it. Am already spreading the word!"
• “Most conferences should be presented like this in future. As well as Elluminate the various social network backchannels can be used for even more interaction”
• “Just wanted to thank all concerned for their hard work and the effort they put in to make this such a rewarding experience. I've developed some very important new contacts from the conference. I interacted much more than I would at a face-to-face conference.”
• “Excellent - from this distance I made better contact than I ever have at a conference before.”
• “It worked! - That's quite exciting! - To know that you can participate without too much stress is a big achievement for me - it takes some of the "fear factor" away”

Cloudworks Statistics

The following data was taken from the cloudworks site and are for the conference ‘cloudscape’ which is comprised of different elements, or ‘clouds’.

• No. clouds - 46
• No. comments - 168
• No. embeds - 59
• No. link - 80
• No. views of cloudscape - 2352
• No. registered users viewing - 392
• No. guests - 2852

Twitter analysis

The conference used the #OUConf10 hashtag, so anyone tweeting with this would have their tweet stored in an archive (using TwapperKeeper).

• No. twitterers using hashtag - 141
• No. tweets using hashtag - 766
• Av. no tweets per user - 5.43
• No. followers added over conference - 150

Elluminate analysis

The following data was gathered from analysing the elluminate sessions over the two days. Overall 287 people participated in at least one session.

The number of people who participated in each session are given below.

• Session 1 - 202
• Session 2 - 141
• Session 3 - 108
• Session 4 - 97

Conclusion

Overall the technology worked well, and most participants found the conference a useful and interesting experience. The number of participants was in line (if not a bit higher) than previous year's face to face conference, but the costs were considerably lower. In terms of benefit from the conference most participants gained as much, if not more, from this form.

A few small points it would be interesting to explore:

• I would like to perform more detailed analysis of the discussions in cloudworks, elluminate and twitter, but haven't had time yet.
• Some people commented that they felt more at liberty to ask questions in text chat than they do in a face to face conference.
• The cost and green comparison with previous face to face versions of the conference.

Project report as coffee table book

I've mentioned before I was part of the Sidecap project, which looked at OER use in Fiji, West Indies and Mauritius. The project has now finished and instead of producing the standard EU report (which, let's face it, no-one ever reads) Frank Rennie from UHI decided to create it as a book in Blurb. We all contributed sections and gathered photos from the duration of the project. My book turned up today:

 

It's a small, and undoubtedly obvious point, but the glossiness of it and the approachability of the layout made me want to read it. In almost exactly opposite to the way I'd approach a normal project report. With almost no shame I am going to mention affordances - a nice glossy book affords you picking it up and looking through it, it extends an invite to you. A project report printed on A4 paper and stuffed in a binder says the opposite, it says 'move on, nothing to see here.'

You can order the book if you want online (no profit goes to us, it just covers the cost of printing). I think all publicly funded projects should mandate this as a deliverable. On my output as collateral damage riff, this is exactly the sort of thing that means a project report destined to be tucked away in the database equivalent of the final scene of Raiders of the Last Ark, becomes a findable, accessible resource.

Or was everyone doing this anyway and I've just caught up?

Do you have to do social media to get social media?

Christian, aka Documentally has a post entitled "Understand Social Technology Through Participation", in which he says:

"How much can you understand from just watching and not participating?
Last night I met a Professor who although used Facebook and other platforms seemed proud to state he did not use twitter.
His job was to study social Technology and although obviously a very smart chap It made me wonder.. Is it possible to comprehend something as complex as social Technology by not participating in a platform like twitter. Can you glean just as much insight from only using sites like Facebook?..
All of my insights into Social Technology have come from my uses of it. "

I agree with him, it's been a bit of a bugbear of mine, the way for example, when Facebook was a'happening it was suddenly inundated with education, psychology, media, business and social science researchers who all wanted to have a look at this phenomenon. They did their research, wrote it up in traditional journal papers or media outlets and then disappeared. And I kept thinking 'they don't get it.'

But first I ought to put the contrary case:

It is perfectly possible to research any subject or group of individuals without using an active participation method. For example I can research the behaviour of vets, football hooligans or stockmarket traders without becoming a vet/hooligan/trader. Indeed not only is it possible, but it is often desirable to have the objectivity that research requires.

It is also the case that not everyone has to use every technology to 'qualify'. I may not use Second Life but that doesn't mean I can't comment on social media in general. And we don't want to have some kind of qualification to start with where people starting wearing a McDonalds type star badge to demonstrate their social-media expertise.

But having said all of that I think that academics and journalists who don't use the new media should be treated with caution. Here's why:

1. It's a fast changing world - the way technologies are used changes, so the survey you did last year won't apply this year
2. The use and benefits are very subtle, you have to spend some time to understand them and you only really do that by participating
3. There is an interconnection between many different tools, so you cannot cherry pick one in isolation without understanding how it might exist within an ecosystem of tools for an individual
4. It is only through participation that you begin to see the types of interesting things to ask. By using the tools you will formulate more meaningful research questions
5. If you simply research it then you see it as a distinct entity or community eg 'it's a teen thing'
6. If you want people to engage with your research then you have to earn some points in the reciprocity economy

So, in general I already dismiss any journalist article that begins "I've never used [technology X] but I know this...". But I am also suspicious of any academic article on social media where the author doesn't have a blog, or twitter id.