Quite inspiring, as someone who’s selected that line of work, if I do say so myself.

A few samples:

In fact, because the library is universal, library science now needs to be a universal skill set, more broadly taught than at any time previous to this. We have become a data-centric culture, and are presently drowning in data.

I could go on and on, but the basic point is this: wherever data is being created, that’s the opportunity for library science in the 21st century. Since data is being created almost absolutely everywhere, the opportunities for library science are similarly broad. It’s up to you to show us how it’s done, lest we drown in our own creations.

The dilemma that confronts us is that for the next several years, people will be questioning the value of libraries; if books are available everywhere, why pay the upkeep on a building? Yet the value of a library is not the books inside, but the expertise in managing data. That can happen inside of a library; it has to happen somewhere. Libraries could well evolve into the resource the public uses to help manage their digital existence. Librarians will become partners in information management, indispensable and highly valued.

In a time of such radical and rapid change, it’s difficult to know exactly where things are headed. We know that books are headed online, and that libraries will follow. But we still don’t know the fate of librarians. I believe that the transition to a digital civilization will founder without a lot of fundamental input from librarians. We are each becoming archivists of our lives, but few of us have training in how to manage an archive.

When you announce yourselves to the broader public as the individuals empowered to help us manage our digital lives, you’ll doubtless find yourselves overwhelmed with individuals who are seeking to benefit from your expertise. What’s more, to deal with the demand, I expect Library Science to become one of the hot subjects of university curricula of the 21st century.

One interesting note – and somewhere I think Mark misses the boat:

It’s interesting to note that books.google.com uses Google’s text search-based interface. Based on my own investigations, you can’t type in a Library of Congress catalog number and get a list of books under that subject area. Google seems to have abandoned – or ignored – library science in its own book project. I can’t tell you why this is, I can only tell you that it looks very foolish and naïve. It may be that Google’s army of PhDs do not include many library scientists. Otherwise why would you have made such a beginner’s mistake? It smells of an amateur effort from a firm which is not known for amateurism.

This isn’t a shortcoming of Google – this is a liberation from the shortcoming of the historical reality of shelf space as the limiting factor in a physical library. These numbers have all been subjectively applied by the local librarian and rarely agree across libraries. Additionally, these subjective assignments reflect more about the culture making the assignment than the content of the work.

I’m surprised Mark swung so wildly on this point. If we’re all sharers now, and we bring our own opinions to the table. What we need more than ‘search by catalog number’ is a means to sift/sort the multiple readers’ opinions of what a book/work is about. This may include expert and non-expert opinion – the point is that the work can be filed in multiple places at the same time.

As Clay Shirky has said, “there is no shelf”:

People have been freaking out about the virtuality of data for decades, and you’d think we’d have internalized the obvious truth: there is no shelf. In the digital world, there is no physical constraint that’s forcing this kind of organization on us any longer. We can do without it, and you’d think we’d have learned that lesson by now.

I have been watching and reading about social network portability and data portability and OpenID and facebook beacon and doc searls’ vendor relationship management and Obama’s call for open formats and Google Drive and Jon Udell’s hosted lifebits scenarios (another post just today).

And Chris Messina has been hanging around this space for a while as well and posted two solid write-ups this past week – on data portability and data brokers.

All of these things seem to scream for some integration, for a system that plays by all the rules and ‘just works’ for the simplest of use cases today, and is ready to scale up and handle the use cases of tomorrow.

I’m envisioning a wrapper – a specification that defines how data should be held and managed for an individual. At first, this should be a single human – later, perhaps organizations or groups of people.

It should be the bucket where our digital stuff lives and the vehicle through which we interact with vendors and each other.

I see three parts – at least for now.

1. Data Repository

This is the heart of the matter. A solid datastore on which to build. This can be a collection of approved document formats – ones that are found to be open and/or well understood. Archival quality stuff here. These need to last for a long time and not be rendered incompatible or unreadable in the future. Really, this is nothing more than a well-defined filesystem or collection of files.

I think it’s most useful at this point to consider the different types of data and simply list the types of formats that meet these criteria. If we don’t have such a format at this time, document the gap and hope that the next few years provide standards that match up.

• ICS/iCal – Calendar/Event Data
• MPEG – Video Data
• Ogg – Video and Audio Data
• TXT – Text, preferably UTF-8
• PDF – Portable Document Format
• ODF – Open Document Format (Word Processing, Charts, Spreadsheets, Presentations)
• vCard – People Listings, AddressBooks
• JPEG – Photographs / Images
• SVG – Scalable Vector Graphics / Images
• PNG – Portable Network Graphics / Images
• KML – Keyhole Markup Language – Mapping Data
• FOAF/XFN – Relationships between people
• OPML – Subscriptions
• GEDCOM – Geneology Data (has limitations)
• PHR/EHR – Personal/Electronic Health Record – complicated, lots of standardization attempts
• APML – Attention / Interest Data
• Financial Data – records, transactions, balances, gets complicated quickly, OFX, GnuCash

2. Data Channels

This is the second piece – getting data in and out of the repository. Open protocols are the key here and it seems we have quite a number of them already being pushed around the live web. Let’s name some and find some gaps…

• XMPP – Messaging – does voice, text, images, this is the Jabber protocol
• HTTP – The web protocol – very handy
• RSS/Atom – Syndication
• OAuth – Authentication between applications
• OpenID – Authentication of Users
• Yadis – Service Discovery
• SMTP/IMAP – Mail protocols

3. Data Management

The third part of this specification would be focused on the management of the data that in the repository and keeping things secure and logged. This is the most complicated part and what makes OpenLifeBits the most different from anything we’ve already got today. Encryption should be at the heart of keeping things well secured (having a brokered encryption market (managing access to secret keys) is another task altogether). Additionally, the dataset should have the capability to be split/merged at will. If you don’t want your medical history stored near your financials, so be it.

• Metadata – describe the data in the repository – using open standard METS
• Permissions – access control – does an open standard exist, Unix permissions?
• Encryption – variety of standards, definitely should have good, strong defaults
• Backup – needs to be atomic, automatic, and recoverable (versioned, even)
• Logging – a full record of what has happened to the life of the dataset

Interfaces

A fourth piece that is really beyond the scope of the definition of any spec is the interface(s) into this data and how a person actually interacts with the data and the outside world via the dataset. A comprehensive list will not be possible to create today, but a solid look at what is available today should force a flexibility of thinking to allow future innovators to do what they do best.

• Mac
• Windows
• Linux
• API
• smartphone
• star trek communicator

Brokered Digital Identity Management

The entire infrastructure/dataset should be portable. Most people will not want to worry about the intracacies of managing this kind of data. We already do it for banking – we get a broker. We outsource and allow experts to manage our stuff for us. We let them worry about the details and there is a marketplace to encourage them to behave well. If they do not, we can move our stuff. This should be the case with our lifebits as well.

Big Picture

There is a large amount of momentum (and cash) behind today’s corporate model (companies own data about you). Inverting the system to be beholden to me and my permission model is not something that will happen overnight. Additionally, the legal questions around ownership of data and the contractual obligations of those you share your information with remain unanswered questions. I have a hunch though that a lot of these types of questions have precedent – just not with the specifics of personal data archives.

As we move into a more digital existence, we will need tools that begin to manage this type of stuff on the personal level. Do you think we can start small and simple and grow into the more complicated models later? Is anyone going to see any value in this OpenLifeBits model besides the geeks among us?

Next week’s IIW in Mountain View should have quite a few people willing to talk about an OpenLifeBits. Please come and find me if you want to hash some of this out further.

I want to be able to specify where in time I think my known item search should look. I think it could be done fairly simply with a well-designed normal distribution curve.

I want to see a timeline (aka a landscape-oriented rectangle) with a distribution curve that I can drag around. I would be centering the curve on *when* I wanted to focus my search.

The search itself would still do fulltext and weight like before, but now, would scale that prior weighting by how well it fit under my specified curve.

I have not done any due-diligence in looking through the information retrieval literature, but I have not seen this interface before and it seems like it would be very helpful for certain types of known-item, time-based queries.

Things that are not within my “window of interest” would be punished with a reduced relevance score in my overall search results. Things that matched my curve, in time, would receive a boost. Otherwise, the search behaves as it always has. This would simply be an addition parameter that gives more power to the searcher who knows *when* they’re looking for.

Has anyone seen anything like this before? Where?

Recently, James Thomas decided to not use Google’s products at all for two weeks and quickly realized it made the Internet quite hard to use. They’re everywhere – and he had to go out of his way to force his computer to not lookup or visit google.com. Not exactly an option for the vast majority of users. (He did also note that the Internet was faster…)

Google is moving into the ISP space (over 100,000 organizations already), the photo hosting space, the email space, the calendar space, the higher education space, the analytics space, more banner ads, and now, into the RSS space itself with yesterday’s purchase of FeedBurner, the premiere RSS serving tool. Most high powered RSS sites I see are pushed out over FeedBurner’s network – their statistics and republishing in multiple formats of your serialized datastream are first rate. For $100M, and a promise of a couple years future employment for the owners, Google now has insight into that side of our collective data behavior as well. They’ve got the readership side figured out with Google Reader, and now the serving side is known to them via this deal. How tidy.

I’m starting to sense a shift in my own dealings with the King of Search. I go out of my way to avoid Google Groups and Gmail. I don’t use Google Docs or Google Apps for my domain, even though they’re arguably easier and more functional than most other setups available today. I avoid having them know all the feeds I’m reading (Reader) and things I’m searching for (log out of google account before searching). I can only assume since these services all sport a unified login now, that Google could not plausibly deny that aggregation is possible across all their (growing) properties.

And I trust Google. I do.

But who I don’t trust is everyone else. I don’t trust that Google will not be driven by the government to hand over certain records or prevent themselves from a data breach forever. They are a very high value target.

With regards to Google knowing too much, Fred has a paragraph that’s worth quoting in his post from yesterday afternoon…

Anonymity is the ultimate irony of the internet. The medium is so clouded in the perception of anonymity, it can fundamentally change human behavior. Of course, the reality is that the internet is the most sophisticated data mining tool ever invented. Compared to any offline action, you are less anonymous when you are using the internet. The nature of our revelations in this false anonymous context could lead a CEO to believe that they really could uncover the “true” persona of an individual, hence being able to accurately answer these very personal questions. In fact, this may be partially true; however, what we’d have to give up to get this benefit is almost always too much.

All that said – I truly want my stuff to be online and available to me. I want global access to what is mine – and to be secure in the fact that it’s redundantly backed up and ‘safe’ from the bad guys. I think that is the way of the future. A personal repository of my stuff with nuanced access given to those who need it when they need it. Jon Udell posted something along the lines of what I want earlier this week… Hosted Lifebits…

Grade 11

You’re applying to colleges. You publish your essay into your space, then syndicate it to the common application service. The essay points to supporting evidence — your e-portfolio, recommendations — which are also (to a reasonable degree of assurance) permanently recorded in your space.

College sophomore

You visit the clinic and are diagnosed with mononucleosis. You’ve authorized the clinic to store your medical records in your space. This comes in handy a couple of years later, when you’ve transferred to another school, and their clinic needs to refer to your health history.

Working professional

You use your blog to narrate the key events and accomplishments in your professional life, and to articulate your public agenda. All this is, of course, published in your space where you are confident (to the level of assurance you can reasonably afford) that it will be reliably available for your whole life, and even beyond.

I think we are well on our way to giving up too much. There will always be a wide spectrum that defines how we live our lives, but more and more, we are choosing to give up our personal information for the sake of convenience in the very short term. This is a dangerous precedent and, I’m certainly not the first to say it, but, I’d rather not be the one who jumps first. Have we really gotten to the point where giving up all our privacy is the right answer? Posting everything online?

So it dawned on him: If being candid about his flights could clear his name, why not be open about everything? “I’ve discovered that the best way to protect your privacy is to give it away,” he says, grinning as he sips his venti Black Eye. Elahi relishes upending the received wisdom about surveillance. The government monitors your movements, but it gets things wrong. You can monitor yourself much more accurately. Plus, no ambitious agent is going to score a big intelligence triumph by snooping into your movements when there’s a Web page broadcasting the Big Mac you ate four minutes ago in Boise, Idaho. “It’s economics,” he says. “I flood the market.”

It seems so wrong… Is this just paranoia on my part?

Update: Fred did it again today – went and posted something relevant – Your Private Twitters Aren’t:

If you’ve been Twittering privately for the past few months, I’ve got some bad news. As reported by Meish, the Twitter API does not enforce privacy ACL’s, meaning all of your private Twitters are available to the public. To check this out for yourself, visit http://twittervision.com/username, and you’ll be able to see private Twitter streams.

I must note that it appears that not all accounts are affected by this problem. It’s impossible to calculate the breadth of this breach, or what it will do to Twitter as a company, but it illustrates a greater problem with the internet. What if your Gmail, or Google History, or Facebook/Myspace account leaked? Or what if the government swept up your information in a national security letter, only to have your information posted in court documents? Think it can’t happen, or that these well-meaning companies can even control it? Just ask people at Enron how they feel.

The fact that this was a pipe dream is fodder for another discussion at another time. I want to talk about how I am managing the documents I do have in electronic form.

One of the ways I devised to attack the problem of the piles was to locate and continue to use a reference manager for my electronic files (mostly PDFs). There were a few to choose from so I needed to make a decision. I decided I’d choose based on 1) open formats, 2) documentation, and an 3) open development model (open source). After looking through RefWorks, vanilla BibTeX files, ProCite, EndNote, Reference Manager (the application) and BibDesk, I chose BibDesk.

BibDesk (screenshots) wins because of these:

• Open data format: BibDesk is a Mac app built to manage BibTeX files. BibTeX is an open format designed to be easily edited and easily moved from platform to platform (it’s plaintext). It has been developed by many eyes over many years and seems to be one of the most robust and flexible formats available for keeping track of bibliographic records.
• Documentation: Being as old as it is, BibTeX has a great number of well documented use cases. Many of the people who use BibTeX have documented their different situations online, where I can find them through basic web search. This is a very big deal – as I can be confident that whatever it is I’m trying to do with the software, I’m probably not the first. And BibDesk itself is documented pretty well too.
• Open source code: The codebase for BibDesk is open and available for download, inspection, and editing (if you are so inclined). I can trust that the format and application with which I’m choosing to manage my papers and references will not go away tomorrow. The company that owns the application cannot decide tomorrow that I need to pay to upgrade or that a particularly favorite feature is no longer necessary (and therefore no longer available). The fact that BibDesk also happens to be under quite active development is an added bonus. It is not uncommon when, if I’ve noticed something with BibDesk that I find annoying or less-than-polished, upon upgrading, it is improved or fixed outright. This continual improvement and attention to detail is a good thing and cannot be overstated as a matter of instilling confidence in a software selection.

As an added piece of the BibTeX/BibDesk solution I chose Subversion for version control. Version control protects me from errors/deletion of my own doing as well as those caused by “accidents” like power-failures or bad disk drives, etc. Subversion works well with the rest of this system because the entire BibTeX file is simple text. It also allows me seamless syncronization between different computers (home and school) and I can be sure I always have the most recent updates (read: PDFs I’ve already found once) available to me, no matter where I happen to be studying/working.

Additionally, by having my subversion repository “in the sky” (my hosting provider) I’ve got yet a third place where the data lives, making it even less likely that I’ll lose it in a fat-fingered accident or because someone has stolen my laptop.

I’ve collected the steps for my personal SVN/BibDesk/BibTeX solution below. I share them here since, when researching this on my own, nobody seemed to have this setup documented in one place.

Downloading and Installing BibDesk

This is one of the most straightforward steps. You can get BibDesk from the project’s homepage. Once the dmg is mounted, simply drag BibDesk to your Applications directory.

Configuring BibDesk

I made a couple choices as to how I wanted BibDesk to work – and I share them here because the choices work well with the plans I had for Subversion.

I want BibDesk to manage my PDFs like iTunes manages music files. I do not want to have to name all the PDFs and make sure they’re in the right place and constantly worry about whether I’ve saved it correctly. I want to tell the reference manager that I’ve got a reference, and I’ve got the electronic version of that reference, and please link them together. But I also want to be able to dig into that file tree later and browse to a well-named PDF so I can send it to a friend or colleague.

BibDesk does this through two features – the CiteKey, and AutoFile. Both are found in the BibDesk preferences panel.

CiteKey is where you define your unique identifier for each entry across your library of references.

I have chosen “First Author, Year, First 20 Characters of Title” as my convention. This gives the “Format String” the value of “%a1%Y%t20″. Many others before me have chosen different format strings, but this combination seems to be a good balance between regular citation convention and the need to remain useful when simply browsing the file tree.

The CiteKey determines how the references are kept unique within the BibTeX file. I take this an extra step and keep that same format as the unique key for naming the attached PDFs as well. This is where the AutoFile preferences come into play.

I tell BibDesk to store my files in a set directory. I tell BibDesk to file my papers automatically (like iTunes). And I tell BibDesk to use the same format I had just defined for the CiteKey. The format string for this is “%f{Cite Key}%e”. The %e on the end tells BibDesk to use the file extension usually associated with that type of file (almost always .pdf for me).

Update: Additionally, consider checking the second box (relative path for local-urls) if you’re using subversion to manage your BibTeX file from multiple machines with different full paths. I missed this myself, as my machines have consistent user accounts (and therefore, full paths).

And that’s it for the BibDesk setup. BibDesk should now store my references automatically as I find them and save them in the GUI. When I want to get back to a linked PDF from within BibDesk, I can simply double-click on it, and the linked file will open in my PDF viewer. No more digging around for lost files. I could also go browse the file tree where I told BibDesk to store my files – and they’ll be there named well and easy to scroll through.

Of course, BibDesk itself has keywords (tagging), annotation, and search – so it’s easy to find the reference you’re looking for from within the app in the first place as well.

Subversion

Subversion is a version control system that allows you to track changes across a file system. Read more about it here and here.

The system works as follows… There is a repository that holds your files. This repository could live on your computer or somewhere else. Mine lives “in the sky” at my hosting provider and is a full copy of all my SCHOOL files. When you work with subversion, you’re actually always interacting with a “working copy” on your local machine. When you are ready to save your changes to the version control system, you “commit” your changes back to the repository. If you are on a different machine with a different working copy of the same repository (this can be used by multiple people as well…), you simply have to resync that working copy, and it will download all the changes made to the repository in the sky. Magical stuff, no? Multiple computers, in sync, and no hard thinking necessary.

My BibDesk files are only a part of a greater collection of documents that I have under version control – but you can see how they live under the SCHOOL folder here.

My BibTeX file (“BibFileTGR.bib”) lives right alongside the “BibDeskDocs” directory that I specified above in the AutoFile preferences panel. All my collected PDFs live in that BibDeskDocs directory. When things change in the BibTeX file or in the Docs directory, I commit the changes to the subversion repository and never think about it again. They’re safe. I’ve got a remote backup, and a full history of changes, built-in.

Configuring Subversion

Installing Subversion and getting it functional on your system is beyond the scope of this document, but please take the time to consider using it. It’s an integral part of what makes this system so powerful.

Quite a few hosting companies provide svn space. My own stuff is hosted at TextDrive.

Here are some of the Knowledge Base articles at TextDrive about getting your subversion up and running. If they’re not exactly the same as your setup, they shouldn’t be too far off. Again, the power of plaintext configuration and open source documentation.

• Getting Started with Subversion
• Managing Subversion users

Feedback

I’d love to hear from others who are using something like this for their own reference management. Do you have anything to add? Anything I left out?

I had used version control for my software projects for a while, but definitely had a moment when everything clicked for me when I realized my BibTeX was just plaintext as well and would fit very nicely within my existing setup.

I won’t be locked into a vendor – or an upgrade cycle – or an opaque “corruption” of my database file. I’ll even be protected against my own fat fingers, since I have the ability to go “back in time” through my versions after a bad save or deleted reference (oops).

Presumably, I can continue to cultivate and build my reference library for many years and never have to worry about it disappearing or getting corrupted.