Flickr Commons

The Library of Congress Pilot Project

The Library of Congress has a Prints and Photographs Online Catalog comprised of over 1 million images (and growing) that have been available online for over 10 years.

Back in June of 2007, we began our first collaboration with a civic institution to facilitate giving people a voice in describing the content of a publicly-held photography collection.

The key goals of this pilot project are to firstly give you a taste of the hidden treasures in the huge Library of Congress collection, and secondly to how your input of a tag or two can make the collection even richer.

You’re invited to help describe photographs in the Library of Congress’ collection on Flickr, by adding tags or leaving comments.*

*Any Flickr member is able to add tags or comment on these collections. If you’re a dork about it, shame on you. This is for the good of humanity, dude!!

I’m very excited about this and will be participating. Just look at all that good old-fashioned well-formed library data in each photo’s description…

However, I think there’s a missed opportunity here to leverage some of the extra power in having many people tag.

At Flickr’s sister site, del.icio.us, we’ve seen wonderful growth and understanding around how communities of users tag collectively. They’re not necessarily collaborating, which is why del.icio.us holds some special properties we do not see in the tagging at Flickr. However, I think Flickr should expose the identities/usernames along with the tags associated with a photo. Most photos are only tagged by the owner – it’s a safe assumption that this will continue to occur into the future. However, when the tagger is NOT the owner/uploader of the photo, this information is currently lost and not passed along in the Flickr interface.

Please, Flickr, expose the ‘who’ part of the tagging triumvirate (see last paragraph of Vander Wal’s definition). Especially now that we’ll have such rich data around our collective history.

This is still a strong belief the three tenets of a folksonomy: 1) tag; 2) object being tagged; and 3) identity, are core to disambiguation of tag terms and provide for a rich understanding of the object being tagged.

Another interesting note about this pilot – this is the first time we’ve seen a distinction of ‘no known copyright’:

Can anyone use “no known copyright restrictions?”
For the time being on Flickr this new usage is being contained to the Library of Congress account. If the pilot works – or, when it works! – we’ll look to allow other interested cultural institutions the opportunity to extend the application of “no known restrictions” to their catalogues.

Hooray, Library of Congress + Flickr!

The poster is up – Tag Decay: A View Into Aging Folksonomies (PDF 1.7MB)

It was a great problem to have people standing, listening, and asking questions for four hours. I just wish my throat had been warned ahead of time – I didn’t talk much the next day.

Thanks everyone.

I was in charge of the big schedule board again. We had it up much faster this year with less tape failures. Technique is very important. And having 12 hands.

I learned how to play Bughouse in the first session. Two chess boards, four players, two chess clocks – and it turns you a bit nuts in less than 10 minutes – which proved just enough time for me to recover before the next hour.

I hosted the next session in the Bughouse room on Expertise Location and had a very engaging discussion around the problems of figuring out “who knows what” and how to keep track of that when you’re trying to hire or place people on teams.

I lured them in with an explanation of my thesis work around Contextual Authority Tagging and asked for input from the “real world”. I heard lots of encouraging comments about how my work meshes nicely with the movement in today’s knowledge management circles away from documenting our knowledge into files (separating the knowledge from the person who knows it) to documenting the people, their work, and simply keeping track of who knows what.

The group agreed that my ideas around tagging others’ knowledge is related to the 360° interview process and the Johari window and its concept of a “blind spot”.

“Everything is pointers.” The overwhelming consensus was that the real way people figure things out is by asking other people, and moving up the chain of expertise until the answer is uncovered. If Bill (who knows about X) doesn’t know the answers himself, he’ll point you to Dave. If Dave doesn’t know, he points you to the next person. This is how we solve problems and if I can help companies do that in a more efficient, documented, trackable way – then everyone agreed I’ve got a very marketable project – as soon as I write it all down, show that it works, and then defend it and get out of school.

The most interesting comment to come from the day’s talk was about a “persistent gap” that may prove itself to exist between what a person thinks they know about and what the group around them thinks the person knows about. Identifying if and when that happens would be a very interesting application of this technique and something I hadn’t really considered before. I’ve been working under the very straightforward assumption that there will be convergence between the three “lists” of terms/tags in my experiment:
- What I think I know
- What they think I know
- What I think they think I know

We’ll see.

The current plan gives me a year to write down what those who have come before me have already done (called the Literature Review) and a year to prove and then write down my own work (called the Dissertation).

Then of course, I’ll have to be a part of that “real world”. Hmmm…

The main reason I feel they’ve got the best system is how they handle their ‘raw’ tags and their ‘clean’ tags.

When a photo is tagged at Flickr, the tag itself is saved in two different formats – raw and clean. If you were to tag a photo with “St. Patrick’s Day”, that’s what would remain in your list of tags, visible on screen. But how Flickr encodes and cleans up that tag results in “stpatricksday”. This is a subtle, but powerful model.

It keeps the original tagger happy (“I know how I want to tag things, darn it”) and it makes the tags more functional in terms of finding things later (both for the tagger, and everyone else). The clean tag is what is used in URLs, in the tagclouds, and wherever aggregation is important for statistics. The tags “N.Y.C.” and “NYC” and “nyc” are all ‘cleaned’ down to the same thing (“nyc”) so when a query comes in for nyc, all three original photos would be presented in the results.

I wanted that cleaning function for myself. I looked everywhere today, and couldn’t find it detailed in any one place any more than on the Flickr API pages themselves.

raw: The ‘raw’ version of the tag – as entered by the user. This version can contain spaces and punctuation.

tag-body: The ‘clean’ version of the tag – as processed by Flickr. This version is used for constructing urls.

Flickr-Like Tag Cleaning Regular Expression

Please let me know if you find any errors, or if this gets out of date. I’ll try to keep it current over time.

The function can be described fairly simply – “remove spaces and punctuation, then lowercase”. I wrote a one line ruby regular expression to do this:

clean_tag = raw_tag.gsub(/[\\s"!@#\\$\\%^&*():\\-_+=\\'\\/.;`<>\\[\\]?\\\\]/,"").downcase

The rest of my test code is included below. As far as I could test today, these are the ‘punctuation’ that Flickr is scrubbing from your raw tags:

#!/usr/local/bin/ruby -w

require 'cgi'

# removes whitespace
# downcases A-Z
# removes 27 different punctuation characters
  # quotation marks
  # exclamation point
  # at symbol
  # pound sign
  # dollar sign
  # percent sign
  # carat
  # ampersand
  # asterisk
  # open parenthesis
  # close parenthesis
  # colon
  # hyphen
  # underscore
  # plus sign
  # equals sign
  # apostrophe
  # forward slash
  # period
  # semicolon
  # backtick
  # open angle bracket
  # close angle bracket
  # open square bracket
  # close square bracket
  # question mark
  # backslash
# does not affect other characters (you should safely CGI.escape these)
  # curly brackets
  # tilda
  # pipe
  # british pound
  # euro symbol
  # chinese characters
def clean_tag(raw_tag)
  clean_tag = raw_tag.gsub(/[\\s"!@#\\$\\%^&*():\\-_+=\\'\\/.;`<>\\[\\]?\\\\]/,"").downcase
end

tags = [
  # should remove the offending characters
  "\\"double\\" quotes",          # quotation marks                     doublequotes
  "!excited!iam!",              # exclamation point                   excitediam
  "test@example.com",           # at symbol                           testexamplecom
  "pound#it",                   # pound sign                          poundit
  "$ave on everyThing",         # dollar sign                         aveoneverything
  "i feel 30% better",          # percent sign                        ifeel30better
  "carats^aretasty",            # carat                               caratsaretasty
  "and&this&and&that",          # ampersand                           andthisandthat
  "maris*61",                   # asterisk                            maris61
  "i think (maybe)",            # open and close parentheses          ithinkmaybe
  "F:ooBar",                    # colon                               foobar
  "hyphen-ated",                # hyphen                              hyphenated
  "under_my_score",             # underscore                          undermyscore
  "1+1=2",                      # plus and equals                     112
  "Saint Patrick's Day",        # apostrophe                          saintpatricksday
  "/leaning/forward/ish",       # forward slash                       leaningforwardish
  "Mrs. Jones",                 # period                              mrsjones
  "semi;automatic;parsing",     # semicolon                           semiautomaticparsing
  "back`tick`here",             # backtick                            backtickhere
  "open<and>close",             # open and close angle brackets       openandclose
  "don't[be]square",            # open and close square brackets      dontbesquare
  "you?sure",                   # question mark                       yousure
  "back\\\\slash",                # backslash                           backslash
  # should only encode the rest of these
  "crab|vs|pipe",               # pipe                                crab%7Cvs%7Cpipe
  "東京",                         # chinese characters                  %E6%9D%B1%E4%BA%AC
  "£",                          # british pound                       %C2%A3
  "nice {curly} brackets",      # curly brackets                      nice%7Bcurly%7Dbrackets
  "Mötley Crüe",                # umlauts                             m%C3%B6tleycr%C3%BCe
  "Tōkyō"                       # long o                              t%C5%8Dky%C5%8D
].each do |t|
  print t
  print "\\n\\tcleaned  -->  "
  print clean_tag(t)
  print "\\n\\tescaped  -->  "
  print CGI.escape(clean_tag(t))
  print "\\n"
end

Both short papers I submitted to ASIST were accepted.

Fred and I submitted a claimID write-up with the title “Self-Representation of Online Identity in Collected Hyperlinks”.

Additionally, my first attempt at writing down my thoughts about the use of social tags over time was accepted with the current title “Tag Decay: A View Into Aging Folksonomies”.

I’m very excited about both of them – and look forward to feedback. These two topics, in their own way, are presenting themselves as the structure beneath my upcoming dissertation research – Contextual Authority Tagging.

The second bit of news this week was concerning my talk at the VCU Technology Days next week in Richmond, Va. I’ve been given the keynote slot at 12:30pm on Wednesday to speak about “Online Identity Management”. Additionally, I’ll be there on Thursday to field questions about claimID specifically. Please drop in if you’re nearby.

It’s funny how weeks go by without a feeling of tangible progress. And then there are weeks like this one.

What: EXPRESSIVITY VS. UNIFORMITY: Are controlled vocabularies dead, and if not, should they be?
When: 1:00 to 2:00pm April 2nd, 2007
Where: Pleasants Family Room in Wilson Library at UNC-CH
Who: Led by Dr. Stephanie Haas, with panelists Dr. Gary Marchionini, Terrell Russell, Tim Shearer, Christiane Voisin, and Lynn Whitener
Presented by: ASIS&T-UNC

Controlled vocabularies, nomenclatures, LC or MeSH subject headings have a long history in LIS. They make classification, categorization, aggregation, sorting, and other operations easier. But with the rise of folksonomy, recommendors, improved natural language processing techniques and other technologies, are they needed any more, or are they just stifling the creativity of our expression?

This panel was on Monday and the week’s flown by since. I wanted to post my comments and see if any extra controversy could be kicked up after the fact.

For the most part – we didn’t disagree very much. The two (social tagging and controlled vocabularies) seem like different ends of a spectrum and should be able to work together… we’re only at the beginning and we need better tools.

Here’s what I said:

I think nuance and a spectra of understanding are too hard for most people for most things. People want clean lines – they want black and white. If it’s beyond my area of interest or expertise – just give me the answer already! So I think there will always be a place for controlled vocabularies wrought by experts and combed over time. People want the ‘right’ answer.

It’s a simple (read: impossibly complex) question of how high the bar of ‘good enough’ needs to be. And it’s different for every information problem. Each person looking for information has their own biases, their own history, their own level of expertise and will use different words/queries accordingly. As they continue their search, they will, themselves, become more sophisticated and use more in-group or official terminology. That doesn’t make any of the words they used to get that far, incorrect. It just means that all valid paths to the ‘right’ information are valuable.

Likewise, we know that there’s value in having a fixed set of words – for aggregation and analysis, as well as the sense that you’re getting everything the database has to offer.

However, I think we’re entering a new time where many more voices are being heard and recorded – and through all this noise and messiness, we’ll still be able to extract a remarkable order.

What’s up for debate is how dumb that order will look and how much information it will actually provide… Will it truly be the lowest common denominator? I think Gary’s right in that the vast majority of information objects are not worthy of our human attention/time. We’ve got automatic classification, fulltext retrieval, etc. As we move forward, there’s just too much of it. We need to focus our attention on the things that deserve our human attention.

These computers, they’re very good at counting things, you know…