The Ed Techie

The Keynote Equivalent?

I've been thinking a lot about recognition of digital output in formal systems, in short, can you get promotion for doing all this online stuff? One of the great things about being online is that as soon as you start to openly think these things through, people start to point you at stuff (mainly Tony). It seems a lot of people have been doing the same (and getting further than me).

For example Eric Schnell says that

The New Media Department and The University of Maine amended their promotion and tenure guidelines (all the way) back in 2007 with redefined criteria in the form of alternative recognition measures. Their documents identify nine alternatives to the standard 'article in a peer-review journal' model

He then lists the nine alternatives: Invited/edited publications; Live conferences, Citations; Download/visitor stats; Impact in online discussions; Impact in the real world; Net-native recognition metrics; and Reference letters.

I think the inclusion of download/visitor stats and net-native recognition metrics is significant here. Similarly, Geoffrey Rockwell has a wiki for a project on evaluating digital work in the humanities. The types of digital output they list as relevant are:
Online peer reviewed publication; Scholarly Electronic Editions; Specifications; Research Tools; Hypermedia/Hyperfiction; Instructional Technology/CALL; and Research blogs.
They also provide a list of questions for those evaluating such work (I'm not sure I agree with the list, it seems focused on a specific product, rather than, say overall significance to a community).

Gerry McKiernan blogs about a study which examined 39 different impact factors for science. The authors of the study (Bollen J, Van de Sompel H, Hagberg A, Chute R, 2009 A Principal Component Analysis of 39 Scientific Impact Measures. PLoS ONE 4(6)) state:

To better capture scientific impact in the digital era, a variety of new impact measures has been proposed on the basis of social network analysis and usage log data. Here we investigate how these new measures relate to each other, and how accurately and completely they express scientific impact.

After reviewing a number of impact factors, including social network measures, they conclude that:

the notion of scientific impact is a multi-dimensional construct that can not be adequately measured by any single indicator, although some measures are more suitable than others. The commonly used citation Impact Factor is not positioned at the core of this construct, but at its periphery, and should thus be used with caution.

So what we currently have is too simplistic in an online world.

Which brings me on to the idea of equivalence. I wonder if we could follow Eric Schnell's line of creating equivalents for existing, and well understood, criteria. Let's take the Keynote speech as an example. Being invited, and giving, keynotes is often listed as one of the marks of esteem if you are seeking promotion. The reasons are twofold I believe:

1. Reputation - it demonstrates that you have gained significant standing in your field to be asked regularly to give a keynote talk at a conference.
2. Impact - if you are giving the keynote then everyone at the conference hears it, and you can therefore claim a significant impact in your subject.

The important element then is not the keynote itself, but what it signifies. If we start with this basis, then we can think of online equivalents. For example, if I give a talk and then put up a slidecast of that presentation, a certain number of views might equate to impact (how many people would hear a live presentation?). If the presentation is retweeted, linked to, embedded, then this might give an indication of reputation.

I don't think we can provide simple translations (500 views + 5 embeds = 1 keynote), which means there needs to be both a case made by the individual, and an interpretation by the panel, but by focusing on the existing criteria and considering what it is they are meant to demonstrate fundamentally, we can then talk about online equivalents.

Arguments for social media engagement

In a comment on a previous post my colleague Chris Jones argued that establishing an online identity wasn't suitable for everyone:

What of the bulk of other academics interested in research and publication? They may move to open access routes for publication but will they want to spend their time in developing a public persona? I am not sure they or I will. I haven't yet developed a blog, though I follow others, including yours. I am not sure that digital scholarship covers all or even the main aspects of intellectual endeavour. Sometimes it is a lone academic quarrying away obscurely on a narrow point that makes a difference. Some of the dynamics of intellectual life require a position outside of the public gaze.

I've pulled Chris' quote out because it is worth discussing in detail I think. Chris is not someone who objects just for the sake of, or from a resistance to technology, but from a more considered standpoint. His argument is that he has a very well established, and effective, way of engaging in scholarly discourse and activity, and in order to develop the type of online identity and reputation I was suggesting he would need to give up doing something. I think it is easy for those of us who spend a significant amount of time in the online, participatory media world to think its attractions are obvious, and undeniable. But Chris raises some valid objections here that bear consideration. I think there are four responses one might make, which range in levels of evangelism.

The argument of recognition - the softest response is to say that I am not proposing that everyone should, or needs, to develop an online identity. What I am arguing is that there is a monopoly on the types of activity and outputs that are currently recognised in the scholarly system, and that we should be able to expand upon this to include the more digital scholarship work, alongside the normal work.

The argument of simplicity - this takes the proposal a touch further and says that not everyone needs to spend a great deal of time and effort in becoming a 'podstar' or blogging celebrity, but there are sufficient number of easy to use tools available such that everyone can easily create an online presence. This doesn't really require you to give up any other activity, it is a byproduct of what you do anyway. Creating a presence using Tumblr for example is a quick way to have an online identity.

The argument of benefit - in his autobiography Martin Amis asks of the humourless 'how do they raise children? How does it get done without a sense of humour?' I sometimes feel the same about social network/web 2.0/participatory media (call them what you like) and being an academic. How does it get done without using these tools every day? I can't imagine finding resources as readily without the links and filtering my network provides, or the analysis they provide, or the feedback I get from them, or the responses to help they give me. I set out a few benefits in the talk on online identity. For me this is one of the strongest arguments - it makes things better.

The argument of imperative - this is the most forceful argument, and an extension of the argument of benefit. Not only does online activity provide you with benefits as an academic, but it is fundamentally key to what you should be doing. The argument here goes something like 'how can you be relevant to students, operate as a scholar if you're not engaged with the most powerful research/communication medium ever?' I'm not sure I'd go this far, because you can use various online services effectively within your research without it necessitating establishing an online identity, but, if one thinks this social media stuff has any relevance, then I maintain that the only way to really understand it is to do it.

Have I missed any, and are any of these really compelling enough? And do they apply across all domains eg. could I argue the same for, Art History, say?

We're all studio bands now

Marieke, over at RemoteWorker, asked me to do a guest blog post around this issue of being a remote worker and establishing a profile at your work place, which you can read in full on her blog.

I mixed up several issues, the first being the effectiveness of being an online worker, where I made a (tongue-in-cheek) comparison with the power-breakfasters of yore:

In the 1980s in the heyday of the Yuppie, there was talk of the ‘power breakfast’ when Masters of the Universe would meet at breakfast to do business to show how tough they were. This is nothing – nowadays I get up and reply to some tweets, put a comment on someone’s blog, respond to comments on my own blog that have come in overnight and maybe even produce a quick publication in the form of a blog post. I have done global networking before I’ve changed out of my pyjamas. And I’m pretty normal in this.

I also wanted to explore the notion that our online presence is somehow an improvement on our real identity, ie the online me is superior to the 'real' (or at least the real-time) me in many senses:

I know that some people now don’t bother attending my ‘live’ presentations (or are unconcerned if they miss them) because they know I will put them up later. And more than that, I feel that I have time to correct the presentations (although they still remain a fine example of amateur hour) so that the recorded version may well be superior to the live one. I think this may pervade across all of my online identity: my blog has more interesting things to say than I do, and my twitter stream is wittier than I am. I have become like one of those bands of whom people bemoan ‘they’re not as good live’. And as we perfect our online skills, maybe this is the fate that will befall us all. In which case, don’t ask me to give a keynote, I’ll just record you one instead.

I was just exploring some ideas here, so pop over to Marieke's blog to give it a read.

Riffability and MPO

Couple of quick terms for you to play with, and no apologies for butchering the English language, it likes it really.

Riffability - the potential for a resource or idea to generate quick modifications, for it to be riffed upon by others.

Riffability is one (but not the only) factor for how memes, ideas or social objects work. If something has a high riffability rating, then it means others will modify it, suggest alterations, or explore ideas around it. Not everything that spreads is riffable - some ideas/outputs are just perfect as they are and you just want to share. Others promote debate and discussion, but that's not the same as riffing.
A few examples: one of my Cardiff twitter contacts, Sianz, produces food/object based visual puns in twitpic. These are very riffable I think, and thus the photos themselves act as social objects around which others congregate.
The discussion I had (particularly with Andy) on metrics is an example of something that isn't riffing, it's debating (or just plain arguing). But had I couched the original post in the sense of 'here's some metrics I'd use', then it might have been more riffable, in that others could have pitched in with their suggestions. When Scott Leslie was preparing his Educator as DJ talk, it was a riff which some of us could work with.
So I think some ideas have more riffability than others, and you can pose your idea in a manner that has greater riffability. If your intention is to get input from others, then learning how to inject a bit of riffability may be a developing skill (when HR consultants offer courses on 'Improving Riffability' I'll know the term has made it).

MPO - Multiple Personality Order (as contrasted with Disorder). Following on from my last post, comments from Pete J and Jim G, questioned whether we only had one online personality and whether we weren't all a bit schizophrenic. I think they're right, my final conclusion that 'your academic identity = your online identity' was too simplistic. Your online academic identity will be a subset of your online identities. I follow a few people on twitter for instance who have a professional and a personal id (you know who you are). Sadly, their professional one is, well, a bit boring compared with their personal one. But I think this reiterates what I was trying to say in my talk on identity - we're still at the beginning of all this.
Increasingly people (particularly teenagers) will develop and cultivate multiple personalities online. This is an astute, and dexterous, thing to do. It allows them to have a pseudonym which might be the identity where they can mess about, swear, talk rubbish and connect only with trusted friends and like-minded people. At the same time they can have at least one other personality which is a more public facing one, which is closely allied to the real identity.
Of course, many people do this very well at the moment, and some environments (virtual worlds in particular) actively encourage a separation of 'real' identity and online one. My conjecture is that it will become the norm, and take place in more publicly social spaces. And it is likely people won't stop at two identities, but have many. When you add into this that people find you in different spaces and so may have one facet of your personality exaggerated (eg if you follow someone in LastFM but not twitter, you would have a different impression of them), then defining what exactly is 'your identity' becomes increasingly difficult.
But this is a good thing and just a natural adaptation (I'm not doing a Greenfield and arguing social networks will cause MPD). I've always found the sort of person who prides themselves on only having one aspect to their personality rather odd, those people who declare 'I'm me, take me or leave me, I call a spade a spade' etc. Being able to nuance your behaviour to a given environment (without being completely false) is a skill. And online, when stuff hangs around forever and can be spread globally in an instant, being adept at doing this will be increasingly significant.

Blogging and academic identity

Blogging and academic identity

View more OpenOffice presentations from mweller.

I gave a talk recently about blogging (twittering, etc) and how we create online identities for ourselves. The slidecast is above. I wanted to explore some ideas, the main ones being:

• It is about identity, not technology X
• Your identity will be constituted from several different tools/services
• Your configuration and emphasis of those tools is part of what makes the identity (as well as what you put in them)
• An online identity is becoming default for academics now
• All this is driven by really easy and diverse ways of sharing
• There are numerous benefits to you as an academic

I concluded with two propositions, which you might like to disagree with:

1. Soon, your online identity will be your academic identity
2. There is an online identity of some form out there for everyone

I hope you find the presentation useful - I have to say I've come to loathe the sound of my own voice and my incoherent rambling so much, I think I may desist from doing slidecasts. Whatever my online identity is, it's not as a raconteur.

Some more thoughts on metrics

Following on from my last post on the possible use of metrics to measure online digital reputation, here are some more thoughts.

Andy Powell took me to task in the comments, arguing eloquently that the metric is so obviously flawed that it is positively harmful. I've been pondering this, and here are some reflections, both for and against. As the methodology is the property of the consultants, I don't know exactly how the algorithm works, so am guessing from the results.

I can see at least three possible problems with the methodology:

1) A deadly attractor - given the search term used ('distance learning'), the OU unsurprisingly dominates the space. I am guessing that close association with the OU then boosts other sites. So, for example, if I had this blog, but was a Professor at, the University of Glamorgan, say, then my ranking wouldn't be as high.
2) An echo chamber effect - a lot of the sites tend to reference each other (me, Tony, Brian, Grainne, for example). You then get a positive reinforcement effect. They claim to have adjusted for this, but I'm not convinced.
3) Bias in initial setup - the blurb seems to suggest that they find the influential sites by searching and analysis, but there must be some priming. The list is heavily UK-centric for example. This may be a result of the term used (see below), but until we know how each search is initiated I think we have to suspect some initial influence depending on the starting parameters.

Search term - as I mentioned in the original post, the search term is significant. 'Distance learning' is very niche - it's not 'online learning' or 'elearning'. Had it been then the list would have been very different. This (plus the things above) may account for some notable absences from the list - why no Stephen Downes for instance, who we would surely think of as the hub par excellence?

Results may be revealing - Andy argues that the presence of Brian Kelly demonstrates that the list is nonsense as Brian doesn't really blog about distance learning. But I think it may be telling us something interesting. It can't just be random, and why is Brian higher than Grainne, for example? It could be telling us that people who write about distance learning tend to reference Brian, even if he doesn't write about it directly. In this sense Brian does have 'influence'.  It could also show us that people who are writing about distance learning online are writing more about IT than pedagogy. This in itself is revealing isn't it?

It's not just popularity - popularity is a factor, and in some respects is a proxy for influence. But we all know that popularity can be gained for all sorts of unacademic things. So popularity is a factor, but only within a given context - it's not about the overall number of subscribers say, but the number of links relating to a given term, and its semantic cousins. So, if I was the expert in modern interpretation of Macbeth say, then I would expect to have the leading amount of links relating to this topic, even though the number of links would be small overall, because it is a niche subject. I wouldn't be a popular blogger relative to every other subject, but within this very specialised subject then I would be 'popular' relative to other subject sites.

Gaming - any algorithm is subject to gaming, as is any system. Exams and the REF are subject to gaming, but the key is to put in enough checks to make gaming difficult, detectable, and ultimately not worth the effort. Any such algorithm would need to be sophisticated enough to avoid obvious gaming.

A metric would only be a partial solution - I think we'd probably always want an element of peer-review in any analysis, and wouldn't rely solely on an automatic measure. But rather we could view an algorithm as part of a portfolio of evidence an individual might present.

We've got to start somewhere - my take on this is that the output may have problems, but it's a start. We could potentially develop a system focused on higher education, which is more nuanced and sophisticated than this. By analysing existing methodologies and determining problems with them (such as the three I've listed above) we could develop a better approach. I hold out hope that we can get interesting results from data analysis that reveals something about online scholarly activity.

Connections versus Outputs

<image No Trumpets Please! by Mr Jaded http://www.flickr.com/photos/jaded/1907907/>

Warning! This post may contain own trumpet blowing!

Laura Dewis sent me a report the lovely OU Online Services people had prepared in collaboration with consultants (MarketSentinel). They were interested in examining the broader influence of various web sites and looking at sentiment mining. The idea from an official communications perspective being you can see how well regarded your institution is in different sectors, and maybe influence that perception.

But from my perspective the analysis they performed could be tweaked to provide a measure of an individual's influence or prominence in the online community of their particular topic. As you will know, I am interested in the concept of 'digital scholarship' and in trying to get the type of online activity many of us engage in recognised as scholarly work.

The problem has been a mismatch between what has traditionally been measured to indicate academic standing in a particular field and the type of activity that takes place online. In short, the formal systems such as the RAE or the REF in the UK, are focused on outputs. They make a nod towards impact, or reputation, but it's really outputs that they want - and we largely know what these look like in this world: Refereed  papers in respected journals; Keynote speeches; Gaining large research bids; etc. These don't map well onto the digital world - is 1000 blog posts good? What about 1000 subscribers? Is having a large twitter network a sign or standing? All too easily such measures fall down. So I have been looking for something more robust which might act as a metric for measuring an individual's reputation in their subject area.

The report the Comms team provided me gets some way towards this. They chose the subject area of 'distance learning' really as a test to see how it worked - one would expect the Open University to come out well in this. Here is the blurb from the company on how they determine influence:

"a stakeholder of a topic is “an entity (individual or organisation) who is sufficiently  referenced in the context of the topic” .
When performing a MarketInfluence study our computer systems initially collect any documents on the Internet (web pages, Word, pdf or PowerPoint documents) which match a defined search phrase.
These documents (often hundreds of thousands) are then analysed. The analysis is especially focused on identifying who references whom.
Based on these references it is possible to calculate influence."

Influencers

They then apply a number of filters to prevent self-citation. From this they drew up the following list of top 100 influencers in 'distance learning':

I come out 4th, Brian Kelly 6th and Tony 10th. I'm sure that Tony and I both gain some credit benefit from being associated with the OU, which drags up our distance learning status, but even so the mix of individuals in with large professional output such as the Guardian, JISC and BBC is interesting - the space has become democratised to an extent.

Betweeners
They then have a measure of friends in common, what they term 'betweenness':

"Stakeholders with high Betweenness are “stations” where information (on the issue in focus) is passed via in order to reach the constituency of said stakeholder. "

Here is the table for those with high betweenness:

So Brian, Tony, Alan, Grainne and I are all good conduits for information (in this narrow domain). Individuals seem to work better as betweeners than organisations it seems.

Hubness
They reference Gladwell's Tipping Point and his notion of 'connectors', to suggest some sites/people have a high level of 'hubness': "Hubness is the characteristic of disproportionately linking to those who are authoritative on a given topic."

Again, people tend to be better hubs than organisations. Oh, and from now on you MUST address me as 'your hubness'.

It is still problematic, and could be gamed. I don't know enough about the algorithms used to assess this. One would also need to be careful about the search term used - 'distance learning' is quite a niche, UK term I think (had it been 'educational technology' we would have had a very different list). But, all this isn't (just) about ego massage - it strikes me that if we could develop such an algorithm so that we could easily enter any subject domain, this would provide a useful tool for measuring an individual's online influence/reputation/status etc in their field. This would then provide evidence for justifying this type of work and in seeking promotion. It could offer us the Alt-REF I was after. At the moment this work belongs to the consultants, and we would want to tweak it for academic use, but it does suggest that such an analysis is possible.  A JISC project to develop the service for all academics?

What this gets at is that online activity is different - it is less focused around outputs and more around overall activity and reputation. And it does begin to back up what I've always felt - that this stuff isn't just peripheral, playing around, but increasingly is significant to individuals and organisations. I wouldn't want to try, but one could think about it in monetary terms - how much is this influence worth if you were to try and buy it (through advertising or other means?).

Robin Mason RIP

(I hope Robin's family and friends don't mind me posting this).

I was deeply saddened to hear that my colleague, Robin Mason, passed away today. It was Robin who brought me to the Institute of Educational Technology (from the Technology Faculty). Robin was a pioneer in e-learning, and fabulously well connected in the educational world - everyone knew, and liked, Robin.  Like John Naughton, Robin was one of the big influences in my early academic career at the OU - I learnt from her how to tread the right balance between scholarly activity and practical application, and just plain have fun with new ideas.
Interestingly I was warned against working with Robin (and with John for that matter) because she was 'maverick' (yes, that was the term used). This seems to me now the highest accolade I could ever hope to achieve.

Bridging the gap - book chapter

A while ago, I wrote a chapter with James Dalziel on bridging the gap between web 2.0 and higher education. It was published in the Handbook of Research on Social Software and Developing Community Ontologies, edited by Stylianos Hatzipanagos, & Steven Warburton, and published by IGI. There are some good chapters in this book, so it's definitely worth a read (despite the rather uninspiring title). I've had news that they are using it again in another book: E-Collaboration: Concepts, Methodologies, Tools, and Applications.

I wrote most of the chapter at the end of 2007, so it already feels a bit outdated, but I forgot to put the PDF up here earlier in the year, and I know some of you like those proper references so here it is. (I think I am breaking all sort of copyright law in publishing it, but am I bovvered?)

Always-right-ism

A curmudgeon returns

Part 2 in my curmudgeon series (don't worry, after this, it's all utopia again). A couple of posts back I mused that "we have seen the demise of any sense of compulsion or quality. There is no more 'should' anymore. Everything is okay." I pondered whether this started with post-modernism and was reinforced by the internet as echo-chamber.

Ain't no echo chamber

I've been thinking more about it, largely thanks to the excellent comments on that post. On reflection I feel that the echo-chamber effect is probably overplayed. Certainly we tend to talk to like-minded people - this is human nature. Sometimes it is a result of preference: I don't need to talk to fascists to know I disagree with them. But it is also a fundamental aspect of what we term community: in academic circles, theoretical physicists don't advance their science by hanging around with people who they have to explain Newton's second law of motion to - they need to assume a base level of knowledge to be able to have the discussions they need.

Even so, my experience of being online is that my network is far more diverse than my face to face one. There is a great deal of variation in the people in my twitter network, without going to extremes. In this sense it is the opposite of an echo-chamber (a diversity chamber?) and almost every day something will come across my path that I wouldn't have encountered otherwise.

Always-right-ism

But I still feel that, particularly in relation to artistic tastes, the lack of compulsion I mentioned is prevalent in society. I could be wrong about this, and I have no empirical evidence (indeed, what form would such evidence take?). But humour me, and let's pretend I'm not just a grumpy old man, and there's something in it. If it isn't a result of the echo-chamber, maybe it is a result of what I shall, with a frightening lack of imagination, term 'always-right-ism'. We have had two mantras in recent years: The customer is always right, and more recently, the user is always right.

So, us internet users in particular are immersed in an environment where we are repeatedly told we are always right. If we don't get how to use a site immediately, give up, because it's the designer's fault. This is largely true, but coming on the back of the customer is always right attitude, it may start to pervade other areas of our lives.

In education we have seen the customer attitude tainting the student-staff relationship. The student cannot be always right (they wouldn't be a student then) and so the customer approach only works to an extent.

So my hypothesis is that a similar stance has begun to influence people's artistic choices. You struggled to read a book? It's the author's fault. Didn't like a piece of music? They should try and be more accessible. Didn't understand a piece of art? They are being wilfully obtuse. Think Lord of the Rings is the best thing written? That's fine, the rest of the country seems to agree.

But...

Brian Lamb posted a comment on my last piece, quoting Nick Hornby who was bemoaning the notion that reading should be hard:

"One of the problems, it seems to me, is that we have got it into our heads that books should be hard work, and that unless they’re hard work, they’re not doing us any good... If reading is to survive as a leisure activity…then we have to promote the joys of reading rather than the (dubious) benefits…please, if you’re reading a book that’s killing you, put it down and read something else"

I have a lot of sympathy with this view. As Nathaniel Hawthorne said 'easy reading is damn hard writing.' The curse of making things difficult to read runs through academia - I have supervised PhD students who feel that if it isn't verbose and almost impenetrable, then it isn't academic. I have spent a lot of my time trying to do the opposite (hence this blog) - trying to say interesting things in an accessible manner.

So I don't think books should necessarily be hard to read (often that's just a sign of poor writing), but you can stretch yourself in terms of the narrative style, subject matter, characterisation, structure, etc. Not every book you read has to be written simplistically, with a straightforward narrative. I recently read Jonathan Coe's excellent biography of the experimental novelist BS Johnson, and I have a lot of sympathy with Coe's stance: he likes books with plot and narrative (the Dickens tradition) but also admires those who experiment with the form. He says of Johnson, he wrote as though it mattered, and I think by extension, if we occasionally subject ourselves to art work that is created by people who thinks it matters, then maybe it matters to us a bit more too.