Flickr Commons

The Library of Congress Pilot Project

The Library of Congress has a Prints and Photographs Online Catalog comprised of over 1 million images (and growing) that have been available online for over 10 years.

Back in June of 2007, we began our first collaboration with a civic institution to facilitate giving people a voice in describing the content of a publicly-held photography collection.

The key goals of this pilot project are to firstly give you a taste of the hidden treasures in the huge Library of Congress collection, and secondly to how your input of a tag or two can make the collection even richer.

You’re invited to help describe photographs in the Library of Congress’ collection on Flickr, by adding tags or leaving comments.*

*Any Flickr member is able to add tags or comment on these collections. If you’re a dork about it, shame on you. This is for the good of humanity, dude!!

I’m very excited about this and will be participating. Just look at all that good old-fashioned well-formed library data in each photo’s description…

However, I think there’s a missed opportunity here to leverage some of the extra power in having many people tag.

At Flickr’s sister site, del.icio.us, we’ve seen wonderful growth and understanding around how communities of users tag collectively. They’re not necessarily collaborating, which is why del.icio.us holds some special properties we do not see in the tagging at Flickr. However, I think Flickr should expose the identities/usernames along with the tags associated with a photo. Most photos are only tagged by the owner – it’s a safe assumption that this will continue to occur into the future. However, when the tagger is NOT the owner/uploader of the photo, this information is currently lost and not passed along in the Flickr interface.

Please, Flickr, expose the ‘who’ part of the tagging triumvirate (see last paragraph of Vander Wal’s definition). Especially now that we’ll have such rich data around our collective history.

This is still a strong belief the three tenets of a folksonomy: 1) tag; 2) object being tagged; and 3) identity, are core to disambiguation of tag terms and provide for a rich understanding of the object being tagged.

Another interesting note about this pilot – this is the first time we’ve seen a distinction of ‘no known copyright’:

Can anyone use “no known copyright restrictions?”
For the time being on Flickr this new usage is being contained to the Library of Congress account. If the pilot works – or, when it works! – we’ll look to allow other interested cultural institutions the opportunity to extend the application of “no known restrictions” to their catalogues.

Hooray, Library of Congress + Flickr!

The main reason I feel they’ve got the best system is how they handle their ‘raw’ tags and their ‘clean’ tags.

When a photo is tagged at Flickr, the tag itself is saved in two different formats – raw and clean. If you were to tag a photo with “St. Patrick’s Day”, that’s what would remain in your list of tags, visible on screen. But how Flickr encodes and cleans up that tag results in “stpatricksday”. This is a subtle, but powerful model.

It keeps the original tagger happy (“I know how I want to tag things, darn it”) and it makes the tags more functional in terms of finding things later (both for the tagger, and everyone else). The clean tag is what is used in URLs, in the tagclouds, and wherever aggregation is important for statistics. The tags “N.Y.C.” and “NYC” and “nyc” are all ‘cleaned’ down to the same thing (“nyc”) so when a query comes in for nyc, all three original photos would be presented in the results.

I wanted that cleaning function for myself. I looked everywhere today, and couldn’t find it detailed in any one place any more than on the Flickr API pages themselves.

raw: The ‘raw’ version of the tag – as entered by the user. This version can contain spaces and punctuation.

tag-body: The ‘clean’ version of the tag – as processed by Flickr. This version is used for constructing urls.

Flickr-Like Tag Cleaning Regular Expression

Please let me know if you find any errors, or if this gets out of date. I’ll try to keep it current over time.

The function can be described fairly simply – “remove spaces and punctuation, then lowercase”. I wrote a one line ruby regular expression to do this:

clean_tag = raw_tag.gsub(/[\\s"!@#\\$\\%^&*():\\-_+=\\'\\/.;`<>\\[\\]?\\\\]/,"").downcase

The rest of my test code is included below. As far as I could test today, these are the ‘punctuation’ that Flickr is scrubbing from your raw tags:

#!/usr/local/bin/ruby -w

require 'cgi'

# removes whitespace
# downcases A-Z
# removes 27 different punctuation characters
  # quotation marks
  # exclamation point
  # at symbol
  # pound sign
  # dollar sign
  # percent sign
  # carat
  # ampersand
  # asterisk
  # open parenthesis
  # close parenthesis
  # colon
  # hyphen
  # underscore
  # plus sign
  # equals sign
  # apostrophe
  # forward slash
  # period
  # semicolon
  # backtick
  # open angle bracket
  # close angle bracket
  # open square bracket
  # close square bracket
  # question mark
  # backslash
# does not affect other characters (you should safely CGI.escape these)
  # curly brackets
  # tilda
  # pipe
  # british pound
  # euro symbol
  # chinese characters
def clean_tag(raw_tag)
  clean_tag = raw_tag.gsub(/[\\s"!@#\\$\\%^&*():\\-_+=\\'\\/.;`<>\\[\\]?\\\\]/,"").downcase
end

tags = [
  # should remove the offending characters
  "\\"double\\" quotes",          # quotation marks                     doublequotes
  "!excited!iam!",              # exclamation point                   excitediam
  "test@example.com",           # at symbol                           testexamplecom
  "pound#it",                   # pound sign                          poundit
  "$ave on everyThing",         # dollar sign                         aveoneverything
  "i feel 30% better",          # percent sign                        ifeel30better
  "carats^aretasty",            # carat                               caratsaretasty
  "and&this&and&that",          # ampersand                           andthisandthat
  "maris*61",                   # asterisk                            maris61
  "i think (maybe)",            # open and close parentheses          ithinkmaybe
  "F:ooBar",                    # colon                               foobar
  "hyphen-ated",                # hyphen                              hyphenated
  "under_my_score",             # underscore                          undermyscore
  "1+1=2",                      # plus and equals                     112
  "Saint Patrick's Day",        # apostrophe                          saintpatricksday
  "/leaning/forward/ish",       # forward slash                       leaningforwardish
  "Mrs. Jones",                 # period                              mrsjones
  "semi;automatic;parsing",     # semicolon                           semiautomaticparsing
  "back`tick`here",             # backtick                            backtickhere
  "open<and>close",             # open and close angle brackets       openandclose
  "don't[be]square",            # open and close square brackets      dontbesquare
  "you?sure",                   # question mark                       yousure
  "back\\\\slash",                # backslash                           backslash
  # should only encode the rest of these
  "crab|vs|pipe",               # pipe                                crab%7Cvs%7Cpipe
  "東京",                         # chinese characters                  %E6%9D%B1%E4%BA%AC
  "£",                          # british pound                       %C2%A3
  "nice {curly} brackets",      # curly brackets                      nice%7Bcurly%7Dbrackets
  "Mötley Crüe",                # umlauts                             m%C3%B6tleycr%C3%BCe
  "Tōkyō"                       # long o                              t%C5%8Dky%C5%8D
].each do |t|
  print t
  print "\\n\\tcleaned  -->  "
  print clean_tag(t)
  print "\\n\\tescaped  -->  "
  print CGI.escape(clean_tag(t))
  print "\\n"
end