Leaderboards and measurement
Randy Farmer and Bryce Glass have a book coming out called Building Web 2.0 Reputation Systems. O’Reilly is generously letting them blog sections from the book.
Their recent post Leaderboards Considered Harmful has some interesting things to say on the oft-seen web 2.0 “pattern” of leaderboards — quantitative displays of something that is related to the activity that the site encourages.
Many leaderboards make the mistake of basing standings on only what is easy to measure. Unfortunately, what’s easy to measure oftentimes tells you nothing at all about what is good. Leaderboards tend to fare well in very competitive contexts, because there’s a convenient correlation between measurability and quality. (It’s called “performance”—number of wins versus losses within overall attempts.)
Their post is illustrated with a shot of the twitter stats section with its count of Friends, Followers and Updates.
Farmer and Glass say:
But how do you measure quality in a user-generated video community? Or a site for ratings and reviews? It should have very little to do with the quantities of simple activity that a person generates (the number of times an action is repeated, a comment given or a review posted.) But these types of things—discrete, countable and objective—are exactly what leaderboards excel at.
To pick on Facebook, because it’s such a big target, this is what happens with “friending” people.
Leaderboards have this amazing ‘Code of Hammurabi’ effect on community values: what’s written becomes the law of the land.
Obviously you can reject the interaction that the leaderboard is pushing you towards. Twitter allows private updates and the people who chose to limit their followers also tend to limit their friends. If you check, you’ll see people with private updates tend to settle around the 50-150 friends mark, a number that is said to be around the “ideal” size for a community.
But the majority of people will take the leaderboard as an indicator of the purpose of the site: Facebook is for amassing friends, Twitter for amassing friends and followers.
And you’ll likely notice this effect in the things that people do—and won’t do—on your site. So tread carefully—are you really that much smarter than your community, that you alone should dictate the makeup of its character?
Buzz Andersen blogged about Farmer and Glass’s post and said:
it’s become apparent to me that social software is a medium turns all communication into a self-representation game whose ultimate goal is popularity. It doesn’t matter whether you’re posting photos, keeping track of website links, writing an essay, or any of the other things people do with social software: in the end the instant feedback of the community combined with your innate desire to be accepted can’t help but shape your actions.
As with television, this can be both a good thing and a bad thing (it has both driven a democratization of creativity and given rise to monstrous, egotistical wastes of bandwitdh like Scoble, Arrington and their acolytes). I think the negative effects can never completely be mitigated, but hopefully thoughtful social software designers will start to consider exactly what sorts of behavior their software is encouraging and discouraging.
Picking on Scoble is easy (and fun) but perhaps Twools would exist even if there was no quantification of interaction. If people with tens of thousands of “friends” are the exception, rather than the rule, perhaps people are smart enough to use a service for their own ends, rather than the ends implied by the leaderboard. Scoble and Guy Kawasaki are up the front, but the majority of people are down in the Long Tail, happy with their community of 50-150 friends.
Do leaderboards matter, or not? I think they might do, but perhaps not as much as Farmer and Glass and Buzz think they do.
You can't research the future -- or can you?
At OZCHI today, people report that Fiona Ingram said in her keynote that you “can’t research the future”.
Viveka noted William Gibson’s quip that “the future is here, it’s just not evenly distributed”.
The thing about what Gibson said is that it means that you can research the future, you just need to go to where it is.
For example, if you want to see what it looks like to live with a CBD congestion charge, you can go to London. If you want to find out what happens when everyone has high-speed broadband to their hand-held device, you can go to Japan.
The trick is to realise that when you go to visit the future, like the past, they do things differently there. If you go to Japan to visit the future you have to take out all the bits that are Japan so that what you’re left with isn’t coloured by the Japan-ness of what you see. This may or may not be possible.
On the one hand, perhaps the future just there, ready to be taken up. And on the other, perhaps the future that you go to visit can’t be reached from here. Instead, there’s a different future, coloured by the past we live in.
Fake it till you make it
A while ago I spoke with a guy who had a business building and selling speech-recognition based telephony systems — the sort of thing that your bank might have when you ring up and you ask for your “account balance” or tell it you want to “transfer money”.
He told me that his company put their own product in place on their switchboard so that when you rang up and asked to speak with someone it could route you through. However, the problem with this is that speech recognition systems suck are really bad at real names.
Problem: How can you teach a speech recognition system what people’s names sound like?
Solution: Fake it, at least at first, with what’s called the Wizard of Oz technique.
Instead of putting the system in place and forgetting it, they put the system in place and trained by having the receptionist route people through normally _but invisibly. People who rang up thought they were dealing with the system but the receptionist was listening to their requests and putting them through manually. When it was determined that the system had learnt enough the receptionist was removed from the system and people’s calls were routed effectively.
Of course, Arthur C Clarke knew this ages ago.
ninakix:
I’ve never been much of a morning person. My snooze button and I have had a love/hate relationship for years. Sure, I get a few minutes of extra sleep, but I’ve often abused the snooze and ended up being late. The Minute Glass has an ingenious solution for all you snooze abusers. It’s powered by magnetic induction which occurs when you shake the clock, requiring no batteries or external electricity source in order to function. When the alarm goes off, you have to shake it and generate enough electricity for the device to function for another full day before the alarm will turn off. All that shakin’ is sure to wake you up. Plus, the clock also has a built in LED flashlight for all your lighting needs. (via The Minute Glass - Josh Spear, Trendspotting)
Love it, but I’m pretty sure I checked my email in my sleep this morning.
That sure is pretty.
The alarm clock with some kind of exertion interface for the snooze button is a remarkably common trope among industrial designers. Perhaps they have trouble getting up in the morning?
Perhaps that’s the problem. An alarm clock with a better snooze button is treating the symptom, not the cause. What I’d like to see is an alarm clock that teaches you to get a better night’s sleep, rather than forcing you awake each morning by making you do a puzzle to get the alarm to stop or chase the alarm around the room, or as a friend of mine told me, find the alarm that his wife has hidden somewhere in the room. Something that implements this advice, in other words.
You’d need to be able to tell it when you went to bed and when you got up. It’d have to be engaging, something that you’d want to use and something that you’d trust. My initial instinct is that it should have an alarm, or series of alarms, that tell you when it’s time to go to bed based on, say, a week’s worth of it noticing (or being told) when you got up. And you’d probably have to be able to tell it how good you felt in the morning when you got up.
Of course, something like a go-to-bed-clock would, if it worked, eventually become redundant. Once it had trained you, you wouldn’t need to rely on it any more.
cartographer:
<emotion>
<category set="basicEmotions" name="Disgust"/>
<intensity value="0.82"/>
</emotion>
An XML scheme for marking up emotions? Seriously?
Until recently I would have had the same reaction.
Then I installed this software in our research lab at work. The system can attemtps to classify people’s faces, cross-culturally, on the seven basic emotions (pdf) described by Prof. Paul Ekman, formerly of UCSF.
Given my background in figuring out how people use speech recognition interfaces, I’m leery of claiming anything approaching accuracy for any system, like this one, that uses statistical modelling to perform what people do “naturally”. It’s sophisticated guessing, embedded in software, which gives it gravitas that it probably doesn’t deserve. Turning something fuzzy like emotional classification schemes into something concrete, like software, has the danger of making the scheme seem significantly less contingent than it might otherwise be.
On the other hand, this sort of thing could be a useful way to provide an extra dimension to research were you’re making some sort of statement about the enjoyment of using something or doing something. (Assuming that said using or doing requires that you sit quite still, looking mostly straight ahead.)
Does the software work? More or less. In my 15 minutes playing with it I found it didn’t like the way my glasses reflected the additional lighting the system required to work. It also seemed to think that my face was consistently “angry” as the initial state, though there should be way of calibrating the model so that each new person sits down at a “clean” zero state.
I’m not convinced of the utility of this sort of thing for commercial applications such as airport screening. But we’re certainly going to see what we can do with it in our research.
cartographer:
The word we use to describe this is “Context”, and we feel strongly that mobile devices will play a central role in establishing a context to the places and people in our lives. But what does this mean?
Oh, Nokia. Context is not (just) information. As Adam Greenfield puts it in the comments:
context is an achievement - a state of being that arises out of the shared performance and understanding of two or more parties. That is, there’s a very deep question about whether “context” in the sense that the technology community understands it can be captured in an object at all.
Someone very smart once pointed out to me that machine-context is different to human-context. Machine-context is sensable. What is connected to what; the data travelling between them, the “tasks” being performed.
Human-context, on the other hand, is only partially sensable. At any degree of particularity, that is down at the 1:1 level, I don’t believe that context as it is currently able to be sensed is very useful to people. The specifics of what an individual is doing at any one time are too multifacted and composed of too many things that aren’t sensable to be useful.
At the small group level, things are slightly more sensable, though the possible outcomes are more general. Groups, I imagine, tend to have (more) specific needs which require (more) predictable support.
At the large sizes, crowds, for example, context becomes a bit easier because now you’re in the realm of modelling aggregate behaviour which is, somewhat depressingly, generally predicable.
It’s at the group and crowd level where interesting things are possible with sensors and “context”. Down at the individual level, I’m of the strongly held opinion that only annoying things are possible.
