Terrell Russell: This Old Network

Today I was working through how a new application would be handling tags and realized that I strongly believe Flickr has the most robust method of storing and querying tags. I think they do it well and wanted to copy their lead.

The main reason I feel they’ve got the best system is how they handle their ‘raw’ tags and their ‘clean’ tags.

When a photo is tagged at Flickr, the tag itself is saved in two different formats – raw and clean. If you were to tag a photo with “St. Patrick’s Day”, that’s what would remain in your list of tags, visible on screen. But how Flickr encodes and cleans up that tag results in “stpatricksday”. This is a subtle, but powerful model.

It keeps the original tagger happy (“I know how I want to tag things, darn it”) and it makes the tags more functional in terms of finding things later (both for the tagger, and everyone else). The clean tag is what is used in URLs, in the tagclouds, and wherever aggregation is important for statistics. The tags “N.Y.C.” and “NYC” and “nyc” are all ‘cleaned’ down to the same thing (“nyc”) so when a query comes in for nyc, all three original photos would be presented in the results.

I wanted that cleaning function for myself. I looked everywhere today, and couldn’t find it detailed in any one place any more than on the Flickr API pages themselves.

raw: The ‘raw’ version of the tag – as entered by the user. This version can contain spaces and punctuation.

tag-body: The ‘clean’ version of the tag – as processed by Flickr. This version is used for constructing urls.

Flickr-Like Tag Cleaning Regular Expression

Please let me know if you find any errors, or if this gets out of date. I’ll try to keep it current over time.

The function can be described fairly simply – “remove spaces and punctuation, then lowercase”. I wrote a one line ruby regular expression to do this:

clean_tag = raw_tag.gsub(/[\\s"!@#\\$\\%^&*():\\-_+=\\'\\/.;`<>\\[\\]?\\\\]/,"").downcase

The rest of my test code is included below. As far as I could test today, these are the ‘punctuation’ that Flickr is scrubbing from your raw tags:

#!/usr/local/bin/ruby -w

require 'cgi'

# removes whitespace
# downcases A-Z
# removes 27 different punctuation characters
  # quotation marks
  # exclamation point
  # at symbol
  # pound sign
  # dollar sign
  # percent sign
  # carat
  # ampersand
  # asterisk
  # open parenthesis
  # close parenthesis
  # colon
  # hyphen
  # underscore
  # plus sign
  # equals sign
  # apostrophe
  # forward slash
  # period
  # semicolon
  # backtick
  # open angle bracket
  # close angle bracket
  # open square bracket
  # close square bracket
  # question mark
  # backslash
# does not affect other characters (you should safely CGI.escape these)
  # curly brackets
  # tilda
  # pipe
  # british pound
  # euro symbol
  # chinese characters
def clean_tag(raw_tag)
  clean_tag = raw_tag.gsub(/[\\s"!@#\\$\\%^&*():\\-_+=\\'\\/.;`<>\\[\\]?\\\\]/,"").downcase
end

tags = [
  # should remove the offending characters
  "\\"double\\" quotes",          # quotation marks                     doublequotes
  "!excited!iam!",              # exclamation point                   excitediam
  "test@example.com",           # at symbol                           testexamplecom
  "pound#it",                   # pound sign                          poundit
  "$ave on everyThing",         # dollar sign                         aveoneverything
  "i feel 30% better",          # percent sign                        ifeel30better
  "carats^aretasty",            # carat                               caratsaretasty
  "and&this&and&that",          # ampersand                           andthisandthat
  "maris*61",                   # asterisk                            maris61
  "i think (maybe)",            # open and close parentheses          ithinkmaybe
  "F:ooBar",                    # colon                               foobar
  "hyphen-ated",                # hyphen                              hyphenated
  "under_my_score",             # underscore                          undermyscore
  "1+1=2",                      # plus and equals                     112
  "Saint Patrick's Day",        # apostrophe                          saintpatricksday
  "/leaning/forward/ish",       # forward slash                       leaningforwardish
  "Mrs. Jones",                 # period                              mrsjones
  "semi;automatic;parsing",     # semicolon                           semiautomaticparsing
  "back`tick`here",             # backtick                            backtickhere
  "open<and>close",             # open and close angle brackets       openandclose
  "don't[be]square",            # open and close square brackets      dontbesquare
  "you?sure",                   # question mark                       yousure
  "back\\\\slash",                # backslash                           backslash
  # should only encode the rest of these
  "crab|vs|pipe",               # pipe                                crab%7Cvs%7Cpipe
  "東京",                         # chinese characters                  %E6%9D%B1%E4%BA%AC
  "£",                          # british pound                       %C2%A3
  "nice {curly} brackets",      # curly brackets                      nice%7Bcurly%7Dbrackets
  "Mötley Crüe",                # umlauts                             m%C3%B6tleycr%C3%BCe
  "Tōkyō"                       # long o                              t%C5%8Dky%C5%8D
].each do |t|
  print t
  print "\\n\\tcleaned  -->  "
  print clean_tag(t)
  print "\\n\\tescaped  -->  "
  print CGI.escape(clean_tag(t))
  print "\\n"
end

Tags: flickr - SocialTagging