Instead of the long contemplative post on how this move is the next logical step in a career of working in Identity Management, I’ll keep it short. I start work next Monday, February 1st, and I’m a bit pre-occupied this week with getting moved from Portland, Oregon down to Half Moon Bay, California.

I expect great things from our team in 2010, and so should you.

I don’t honestly recall why I decided to go by a different name. Name changes are common throughout history to mark a new beginning in one’s life. In the Bible, Abram is given the name Abraham, Jacob is renamed Israel, and Saul of Tarsus becomes Paul the Apostle. Converts to Islam will often take on a new Islamic name, and it is common for monarchs and newly elected popes to take a regnal name when they inherit the throne. It is customary in many cultures to take the surname of a spouse, or a blending of the two surnames, when one is married. Perhaps at some level I wanted to mark this new beginning in my life. I was leaving behind everyone I knew, and would be starting fresh with school, with friends… with everything. Maybe I wanted a new name to represent this new part of my life. Or perhaps I was simply emulating my older brother Steven, who at the time had chosen to go by “Steve”.

My immediate family was pretty good about respecting my decision. My mom later told me that when they were picking names for my brother and me, they went through all the possible nicknames and made sure they would be okay with them. Occasionally my mom would slip and call me William, and I remember that I used to get really mad about that. I don’t think my grandparents ever stopped calling me William, but after a while I got over it.

Identity Online

I think that I really started giving thought to my online identity when I was in college at George Tech. When I was a student, we were all given a “GT Number”, which was simply an opaque username and email address. Mine was gte739u, and so my email address was gte739u@prism.gatech.edu. Everyone had these numbers, and we all got used to them. Papers and tests might have a place to put your name, but they always had a place to put your GT Number. We weren’t names, we were numbers… we were simply $student++. I’ve never been one for pseudonyms, maybe because I didn’t have any real issues with my name. Up until this point, I had always used variations of my name for accounts: wnorris, wjnorris, or wjn730 if nothing else was available. It was only when I no longer had that freedom to identify myself how I chose that I became aware of how important it was to me.

It wasn’t until my second or third semester that I was eligible to get an account in the College of Computing, which you got to choose yourself. I was quite happy when I could finally give out a decent school email address to people — wnorris@cc.gatech.edu. In a small way, I felt like Equality 7-2521 asserting his individuality, taking the name Prometheus.

When I began to realize the benefit of a personal homepage, I found that the domain willnorris.com was already registered, so I settled on wirewater.org instead. I thought it sounded cool and I liked the definition in the Jargon File. I used that as my personal homepage as well as my main email address for several years, until I was able to buy willnorris.com a few years later and switch everything over to that. I had used wirewater.org so much during those years that I decided to just keep it indefinitely. I don’t think I ever receive legitimate email on that account anymore, but it costs so little that I don’t really worry about it. There is a competitive market for registering “.org” domains, so I can be assured that the price will always remain at a reasonable rate. If I want to change my registrar for whatever reason, I can easily do so.

A(nother) New Identity

In 2007, a new service called FreeYourID was launched by GNR and Janrain. For $11 a year, you could get a third-level .name domain of the form firstname.lastname.name. They would also forward email sent to firstname@lastname.name, and later added a few other identity related services like XFN links and redirects to your social network profiles. The most exciting part of all this was that every FreeYourID domain was automatically an OpenID, backed by MyOpenID. It was a great example of putting individuals in control of their identity online, and how OpenID delegation fit into that picture. Seeing the potential for this, I grabbed will.norris.name on the very first day. It wasn’t long before I started using this new URL as my primary identifier online. I still had willnorris.com and continued to use it as a blog, but will.norris.name became my “identity site”. It was a simple landing page that had contact information and links to my profiles on various services. Later I added an activity stream, and XFN links to friends and colleagues. More importantly though, I used it as my primary OpenID on any services that supported it.

About a year and half (and hundreds of OpenID logins) later, I decided that I didn’t want to maintain two sites. I polled my friends, and decided to migrate away from will.norris.name. It was a very manual process of updating my various online profiles, and presented even more challenges with OpenID. But like my transition from wirewater.org I had done several years earlier, I didn’t worry too much about because the extra domain wasn’t really costing me that much.

That all changed this year when it was announced that FreeYourID was shutting down after just two years of operation, and that all accounts would be transitioned over to Key-Systems GmbH. Never mind the fact that the new site to manage your registration is absolutely terrible, the cost for renewal was also raised to 23.39 € (about $35). And unlike my previous .org registration, hours of searching and phone calls have not revealed any way to transfer a third-level .name to a different registrar (in fact, most registrars won’t even transfer second-level .name domains). My domain was scheduled to expire in a few weeks, and I would have liked to just let it go so I don’t have to spend the $35, but there’s a little problem…

OpenID and Reusable Identifiers

I started the process of updating my OpenID on sites a year ago, but I’ve still identified three relying parties that do not support changing your OpenID (at least not that I can find): Disqus, Clickpass, and Pibb. I’m certain there are many more, but these are the only ones that I know I have accounts with, and are currently set to use will.norris.name. So if I let my domain expire, and someone else buys it, they can immediately login to my account at these three services. This is the way OpenID is designed to work… whoever controls the domain is able to authenticate as that URL. So what does this mean for me? Quite simply, it means that if I want to make sure that no one else is able to access my account on any of these three services, I’m forced to pay $35 to renew a domain I don’t use and don’t want.

Who’s to blame for this? Well, I could blame Key-Systems for tripling the price of .name accounts when they took over the FreeYourId service. I could blame myself for having bought the domain in the first place, instead of just sticking with the .com I already had. I could blame the services listed above for not supporting OpenID changes on accounts. And I could blame the OpenID protocol itself for keying on reusable identifiers, instead of using those as aliases to unique, non-reusable identifiers like XRI has been architected to do from the very beginning. All of these would be fair parties to place the blame on, but this post isn’t about placing blame. Instead, this post is about getting the technologists developing and deploying this stuff to start thinking through the entire account lifecycle.

Identifiers Change

We’re living in a world where the identifiers we use to refer to people online are more important than ever. From IRC nicks to email addresses to Twitter handles. These monikers are typically all that identifies us within a particular service context, and sometimes between contexts. This is particularly true of Twitter handles, which in recent years have come to be seen by some as the de facto namespace for people. I was more than a little upset when my former employer (a company focused on OpenID, no less) linked to my Twitter profile instead of my personal homepage when they announced my hiring. And again this year at WordCamp Portland, it was disheartening to discover that the attendee name badges had a place for your Twitter handle, but not for your blog URL. At a WordCamp! The emphasis on our identifiers on these services makes it increasingly difficult to change your identifier without breaking things. But the fact is, identifiers do change. As our online and offline worlds collide, more and more people are moving away from pseudonyms toward using real identities online (something Facebook had the forethought to require from the very beginning). While this is of course a personal decision, it’s one that Chris Messina recently undertook. Similarly, Tom Coates and Ben Metcalfe, two individuals who understand online identity and social media very well, have considered doing the same.

I guess my point is just this. Identity is important. And identifiers change. So we need to be ready for that as we continue to build the “social web”.

Some of the best software libraries I’ve used address both ends of the spectrum. There’s a common adage in software development (and I’m sure it goes back farther than that): “make the common things easy, and the hard things possible”. First, you don’t want to make things any harder than necessary for the majority of users that are just using the basic functionality of a library. If they don’t care about customizing and tweaking every little aspect of it, then the way they interact with the library should be relatively simple and straightforward. But for those users that have unique needs, the library should allow them to configure it in such a way to accommodate that. It is certainly my goal to address both extremes in the Shibboleth OpenID library, but it will happen in phases.

The first phase will address the edge-cases, those users of the library that tend to have unique needs and requirements. That may seem backwards, but I assure you it isn’t. First of all, the really practical reason for starting here is that Shibboleth is itself an edge case. The reason I chose not to use the existing Java OpenID libraries in the first place was that they didn’t adequately conform to the way Shibboleth needed them to work. But from a design perspective, I’ve found that this approach tends to yield better results anyway.

Small Pieces Loosely Coupled

I’ve learned that in order to make a system really flexible and modular, it’s best to architect it that way from the very beginning. You have to decide where the logical divisions of labor are within the system, and then translate that into the code itself. Each component should be relatively self-contained, it’s purpose should be clear, and its interface (the way it interacts with other components) should be separated from its actual implementation. Sometimes these components are obvious, and there are clear places where the code should be divided. But more often than not, its a judgement call. Architecture is often harder than actual construction, whether you’re talking about software or brick and mortar. It requires a lot of creativity because you’re often working from a blank canvas, but it also requires that your plans are grounded in what is actually possible. Architectural plans are worthless if they can’t actually be implemented in the real world in a practical way. By no means do I think that I’ve found the best architecture for this library, because it’s always subjective. Fortunately, I’ve had similar libraries that I’ve borrowed from heavily for inspiration, as well as much smarter developers that I work with to bounce ideas off of.

At its core, this library is an OpenID messaging library. It is capable of converting between generic HTTP messages and strongly typed OpenID objects that developers can work with. My last two posts have talked about this in detail. The library also provides the additional logic for implementing the OpenID specification, things like Diffie Hellman key exchange and OpenID message signing. What the library does not do is tell you how you must compose these pieces into a working system. That’s because there isn’t just one way to do it. It greatly depends on the application that is wanting to add OpenID support. If you’re using a Java framework like Tapestry or JSF, perhaps you have other processing that happens to the message before the OpenID library gets involved. How does the user get authenticated and where do the user attributes come from? I have no idea… that’s up to your application to decide. At this level, the library makes no assumptions (or at least as few as possible) about how all of these small pieces should be coupled together.

If this sounds like a lot of work left up to the user just for something as simple as OpenID, you’re right… it is a lot of work. But when you really need that level of control, it’s important that the library support that.

Addressing the Common Case

So what about everyone else, all the “mere mortals” who don’t need that much control, and just want to add OpenID support to some application using the default configuration? At a high level, I’d love to have a Servlet Filter that you can drop in front of your application, configure a few small things, and have it just work as an OpenID relying party. I’ve always been a huge fan of the Tapestry framework, so I’d love to have a Tapestry component that can be dropped in just as easily. All of these things are possible by building a layer that sits on top of those individual components in the library, and arranges them in a prescribed way.

Now, I don’t anticipate that a drop-in Servlet Filter will ever be a part of the core library, because I don’t think it belongs there. It would be a separate deliverable unto itself that simply relies on the library to do all of OpenID work. This also means that the Filter wouldn’t necessarily need to come from me, anyone could write it and make it available. I don’t imagine that the core library itself will ever have everything that the “common case” users will need. And I’m okay with that, because I’m not building an OpenID product, I’m building an OpenID library.

Current Status

This is by no means a complete picture of the Shibboleth OpenID library, but it should give you a rough idea. It identifies some of the larger components of the libraries, and some of the interdependencies.

The orange blocks are pieces that are basically complete and present in the current library. All of the message handling is complete for OpenID 2.0 message, as well as three of the most popular message extensions (SReg, AX, and PAPE). Additionally, association management is done, and a very simple AssociationStore is provided (though it needs a little improvement). The security layer is complete insofar as signing and verifying signed messages. The yellow blocks represent pieces that are not yet complete, but will be included in the core library in the future. The discovery component is still up in the air a little bit because it’s not completely clear if we’ll be using XRD, XRDS, or both. The portions of the security layer that depend on discovery are, of course, waiting on the completion of the discovery stack. Those pieces include everything that an application would need to construct an OpenID provider or relying party. They implement the full OpenID protocol.

But those components alone leave a lot to be filled in by the application using the library. It says nothing about how an incoming HttpServletRequest object is converted into an OpenID Message. The application would be responsible for instantiating the specific objects and wiring them together to actually get a working AssociationManager. And for applications that wish to have control over these specific aspects of an OpenID flow, this is a good thing. The next layer up on the stack however, the yellow “Managers” block, will provide simple Manager objects that wire things together in a prescribed way. Most users of the library will deal with the Manager layer, and probably nothing else. Only when they have specific needs will it be necessary to dig any deeper.

Now this last layer is actually nothing special… it’s a very common pattern, and both OpenID4Java and Joid work in very similar ways. I point it out only to note that while this layer will be part of the core library in a future release, it isn’t right now. Much of the code that will likely make up these components has already been written, but in the form of the Shibboleth IdP extension. For the last few months I’ve been simultaneously building both a generic OpenID library, as well as an actual product that makes use of the library. One of the tougher ongoing challenges while doing this is in deciding which of the two projects a particular bit of code goes into. Much of the time it’s clear, but when in doubt, I’ll put something into the Shibboleth extension rather than the library. If anything, I want to err on the side of keeping the library “pure” so to speak. To not accidentally bake any assumptions into the library itself that might limit its flexibility. One of my focuses after the holidays will be in identifying which pieces need to be refactored from the Shibboleth extension back into the core library in order to build out that management layer.

(And in case it’s not clear, the final layer in grey in the stack above are other pieces that will make use of the OpenID library, but will likely not be part of the library itself.)

Central Registry

As we mentioned, every OpenID message type has a number of supporting classes. Let’s take the authentication request message as an example. You have:

• AuthenticationRequest — interface that defines this specific OpenID message type
• AuthenticationRequestImpl — default implementation of this message type
• AuthenticationRequestBuilder — builder of AuthenticationRequest instances
• AuthenticationRequestMarshaller — converts AuthenticationRequest message into a ParameterMap
• AuthenticationRequestUnmarshaller — converts ParameterMap into an AuthenticationRequest message
• AuthenticationRequestValidator — this wasn’t talked about last time, but is responsible for validating that an AuthenticationRequest message contains all of the requisite parameters

All classes except for the actual message implementation must be thread-safe, as only a single instance is maintained by the library (technically they don’t follow a singleton pattern, but only one instance is typically used). All of these are stored in central registries, so that they can be retrieved to marshall or unmarshall a message as needed. Each one has it’s own factory that allows registering and looking up of specific implementations:

• MessageBuilderFactory
• MessageMarshallerFactory
• MessageUnmarshallerFactory
• MessageValidatorFactory (not yet implemented as of this writing)

MessageValidator implementations are registered based on the message class that it validates. For the other three factories, implementations are registered based on a QName which consists of the OpenID protocol namespace URI, and the value of the mode parameter. Yes, there are three OpenID message types that don’t actually have a ‘mode’ parameter, but I’ll save that discussion for another post. Also, the QName here doesn’t exactly represent a namespaced parameter name like it does in the ParameterMap, instead it is just a container for a namespace URI and a string value. Perhaps this is technically a misuse of the QName object, but it’s working fine for now. A static instance of each factory is available from the Configuration class.

Message Flow (redux)

So now let’s go through a message flow like we did last time, and look at how each of the factories are used. (At the time of this writing, I’m still working on hooking in the MessageValidators, so I won’t be talking much about that).

Remember that when a message comes in, it is in some kind of transport specific encoding. Depending on how the message was received and the format it is in, an appropriate MessageDecoder is used to convert it into a ParameterMap. The next step is to find an appropriate MessageUnmarshaller to convert this ParameterMap into an actual Message object. The MessageUnmarshallerFactory has a getUnmarshaller(ParameterMap) method that will lookup exactly what we need. Once we have an unmarshaller, we can call its unmarshall(ParameterMap). This method is responsible for building an appropriate Message object, and then populating it based on the data provided in the ParameterMap. Internally, the unmarshaller uses the MessageBuilderFactory to find an appropriate MessageBuilder using the getBuilder(ParameterMap) method. Once the correct builder is obtained, its buildObject() method is called to get an instance of the Message object. This instance is then populated using data from the ParameterMap and returned. (If anyone wants to volunteer a flow chart that illustrates this, I’d be greatly appreciative!)

When it comes time to send a message back out, the MessageMarshallerFactory’s getMarshaller(Message) method is called to get the correct MessageMarshaller for a given message. The marshaller’s marshall(Message) method is called and returns a ParameterMap, and that is passed through an appropriate MessageEncoder to send it out on the wire.

Custom Implementations

The library comes with default implementations for all of this, so a user can simply choose to ignore all of this plumbing and be just fine. But just in case you do want to customize part of this, how would you go about doing so? Simply by registering them with the appropriate factory. Let’s say you want to provide your own AssociationRequest implementation for whatever reason. But maybe you don’t necessarily care to customize the way the data is unmarshalled into and marshalled out of the object… the default implementations for those are fine. You would of course have your custom AssociationRequest:

public class MyAssociationRequest implements AssociationRequest {
    /* implementation here */
}

Then to make sure that your custom implementation is built instead of the default implementation provided by the library, you would also need to provide a MessageBuilder:

public class MyAssociationRequestBuilder implements 
             MessageBuilder<AssociationRequest> {

    public AssociationRequest buildObject() {
        /* build and return an instance of MyAssociationRequest */
    }
}

Then register your message builder:

MessageBuilder myBuilder = new MyAssociationRequestBuilder();
QName qname = new QName(OpenIDConstants.OPENID_20_NS, AssociationRequest.MODE);
Configuration.getMessageBuilders().registerBuilder(qname, myBuilder);

Once your builder is registered, it will be used to build AssociationRequest objects for all incoming messages of that type. However, the default marshaller and unmarshaller for that type will continue to be used… you don’t need to worry about that. And once I get the validators hooked in, that will just work as well with your custom class. Or, you could provide your own Validators if you like. You can customize as much or as little of the library as you want.

I don’t imagine that anyone will want to provide custom message implementations very often, but the option is most certainly there. What is far more likely is providing a custom message extension like Attribute Exchange or PAPE. That works in very much the same way, which I’ll explain next.

Let me first preface this by clarifying that I’m not saying the existing OpenID libraries are not usable. Quite the contrary, I know that the OpenID4Java library is used for AOL’s OpenID provider, on Google’s Blogger, as part of Sun’s OpenSSO, and countless other projects. Additionally, JOID powers Verisign’s very usable PIP. There is no question that they work for many use cases. However, they lack the clean architecture I was looking for, which can really only be corrected by starting from a blank canvas.

(I’m not sure how many posts this will take, or how sensical the order of things will be, but better to go ahead and get it written down in some form.)

Message Handling Flow

One of the most immediate differences you’ll see in the Internet2 library is the very clear separation of logic in the message handling code. I wanted the core message objects to be simple Java beans that provide access to strongly typed properties, and nothing more. When I’m processing an OpenID message, I don’t want to be thinking about how that message was encoded during transit. Additionally, I don’t want to duplicate code if at all possible, so there needs to be one very clear place where any particular process is implemented. To achieve this, messages are transformed into three distinct formats as they are being processed.

When a message comes in to an OpenID provider, it is in some kind of transport specific format. Typically that will be a URL-encoded string that is taken either from an HTTP POST request body, or from an HTTP GET request query string. Alternately, it may be a Map retrieved by calling ServletRequest.getParameterMap. This transport specific format needs to first be converted into some kind of common intermediary format so that the next step in the process can deal with all messages in the same way, regardless of transport method. In the Internet2 library, this common format is a ParameterMap.

ParameterMap

A ParameterMap is simply a LinkedHashMap with QName keys, String values, and a little additional logic. Why QNames for keys? Aren’t those for XML? Yes they are, but they actually work beautifully for OpenID message parameters as well. You see, an OpenID message is really just a collection of namespace qualified parameters, and can be quite easily represented in XML. (Yes, this is a little bit of a rabbit trail, but it’s interesting nonetheless). Let’s start with a really simple KVF encoded OpenID message:

openid.ns:http://specs.openid.net/auth/2.0
openid.mode:checkid_setup
openid.claimed_id:http://example.com/
openid.identity:http://example.com/
openid.ns.sreg:http://openid.net/extensions/sreg/1.1
openid.sreg.required:email,fullname

Yeah it has no signature, etc, but that’s not the point. What might this look like in XML?

<message xmlns="http://specs.openid.net/auth/2.0" 
         xmlns:sreg="http://openid.net/extensions/sreg/1.1">
    <mode>checkid_setup</mode>
    <claimed_id>http://example.com/</claimed_id>
    <identity>http://example.com/</identity>
    <sreg:required>email,fullname</sreg:required>
</message>

See how cleanly it maps? This is no accident. This is a very common pattern for handling namespace qualified parameters. First you assign your namespace to an alias, then you use that alias as a prefix for any parameters that are part of that namespace. The simple registration ‘required’ parameter name has three parts: there’s the base parameter name (“required”), the namespace alias (“sreg”), and the actual namespace URI which is declared separately (“http://openid.net/extensions/sreg/1.1”). A Java QName object consists of three parts: a namespace URI, a local part, and a namespace prefix. Slightly different terms, but exactly the same concepts.

Okay, so back to our OpenID library. We’ve taken our transport specific encoding, passed it through an appropriate MessageDecoder, and ended up with a ParameterMap. Before we move on, I want to point out one more thing about the parameters in a ParameterMap. None of the parameter names contain the “openid.” prefix. This prefix is specific to messages that are encoded using URL Form encoding, since that’s the only way to identify which parameters are part of the OpenID message. One of the jobs of the URLFormCodec is to strip this prefix as messages come in, and add the prefix as messages go out. The message encoder and decoder is the only part of the entire library that knows anything about this prefix, and quite frankly it’s the only part that should.

Okay, so now that we have our ParameterMap, it needs to be converted into an actual message object, which is the job for a MessageUnmarshaller.

Unmarshalling messages

Message unmarshallers are responsible for taking a ParameterMap and using it to populate a specific kind of message object. Remember the desire for message objects to have strongly typed properties? The corresponding unmarshaller for that message type is the one and only place that needs to worry with how the parameter passed on the wire gets converted into that strong type. For example, AssociationRequest messages may include the Diffie-Hellman public key of the OpenID relying party. Java provides a very specific object just for that called DHPublicKey, so that’s what we want our AssociationRequest object to use. Parameters can only be passed as strings during transit, so the AssociationRequestUnmarshaller (and nothing else) is responsible for knowing how to convert that string into a DHPublicKey.

Similarly, Attribute Exchange fetch requests may include a list of required attributes it wants for a user. These attributes are identified by URIs, so Attribute Exchange does it’s own aliasing similar to the namespace declarations we saw above. This way, the “ax.required” message parameter need only contain a comma-separated list of attribute aliases rather than the full namespace URIs. But when you get right down to it, these aliases are just an optimization that is used during transport. Really all that’s being represented is a list of attributes URIs. This is why the FetchRequest object in the Internet2 library exposes this particular message parameter simply as a List of attribute URIs. It’s the FetchRequestUnmarshaller that is responsible for taking the AX message parameters, dereferencing the attribute aliases, and populating the FetchRequest object appropriately.

Reversing the process

What about returning OpenID response messages? We just do the same process in reverse. The message object is passed through an appropriate MessageMarshaller which populates a ParameterMap. And the ParamerMap is in turn passed through a MessageEncoder that produces some kind of transport specific format. That may be a Key-Value form encoded string, as is the case with direct responses, it may be a URL suitable for redirecting the user to, or it may be an HTML response to use for HTML form submission.

Uniformity over brevity

Depending on how you separate them, there are roughly nine different message types in the core OpenID 2.0 spec, and for each of these message types, the Internet2 library has five files that handle the processing. There’s the message interface, the concrete implementation, the message builder (which I didn’t actually talk about in this post), the message marshaller, and the message unmarshaller. At times all these files may seem needlessly verbose, especially when you see that some of them are only a few lines long. It turns out that this separation doesn’t necessarily result in more lines of code, just that the code is broken up into smaller chunks. Besides, the goal here is not conciseness. The goal is uniformity and predictability in how messages are processed, as well as clean, logical separation of duties. When every message is processed in exactly the same way, bugs tend to expose themselves much earlier in the process, and strange edge cases are far rarer. When things are logically separated, it makes the overall architecture much easier to understand. And perhaps more importantly, it makes it possible to fully understand one part of the library without needing to be concerned with others. You can go in and look at the code for signing messages, and not have to wade through code dealing with transport encodings.

I’ve talked with core team about this numerous times… in fact, I spoke at WordCamp Portland and Seattle these last two weeks and talked with Matt about it. For the most part, I actually agree with him that OpenID doesn’t necessarily belong in core, at least not yet.

There’s a lot of thought being given to how WordPress can serve as your “digital hub” on the web. Right now, Automattic is playing in that space in the form of BuddyPress. Now right now, BP allows you to create another social network silo. BP installations don’t talk to each other, and there’s no way to use your account on one BP network to login to a different BP network. I talked with Mark Jaquith this weekend about my desire to see this outward facing functionality. For that, I think OpenID becomes painfully obvious.

I would also like to see this OpenID plugin deployed on WordPress.com to replace the existing plugin. Currently, all WP.com blogs are OpenIDs, but you can’t login or leave comments using an external OpenID. And currently, almost no one uses the existing OpenID provider. Of course, I would argue that this is because they haven’t done a good job of promoting it or adding any new features like SReg or AX. Using my OpenID plugin would greatly enhance the OpenID provider functionality on WP.com, and it would allow people to use OpenID when leaving comments. Some of the changes that are included in 3.3 are actually steps toward cleaning up the plugin so that it is more suitable for deploying on WordPress.com. There’s still more work to be done on this front, but it’s something I intend to continue pursuing.

As for inclusion in WordPress core, I just don’t think we’re there yet. The OpenID plugin is pretty popular, but it is far from having the critical mass that would justify inclusion in core. I am a firm believer that WordPress should by no means try and include every cool feature under the sun in core. It would quickly grow out of control. I do believe, however, that the appropriate hooks should be provided in core to allow any cool feature under the sun to be added as a plugin. The core dev team agrees with me on this, and they’ve been very good about making whatever changes were necessary to allow plugins to provide that functionality. In fact, I overhauled how the authentication system is extended in WordPress 2.8 simply to make things like OpenID and OAuth much easier to implement.

A few other things I’d want to see fixed before considering inclusion in core… the OpenID plugin weighs in at what? almost 900K? Remove the screenshots and readme.txt and you’ve got 700K left. Over 500K of that is the JanRain OpenID library. So size is an issue. Also, the biggest problem that people have with getting the plugin to work is related to their environment. WordPress is known for having a very minimal set of requirements to get it running. I’d really want to track down and fix a lot of these weird environment issues that continue to plague the plugin. Finally, we need a really solid UI, both comment form integration and the admin side. I’m pretty happy with the new comment form integration, but the current admin screens need work. More than anything, there is just a lot of functionality in the plugin and it’s hard to boil it down. Especially when you consider both the OpenID consumer and provider options, both site-wide and per-user.

Second, this release features a new user interface for the integrating OpenID into the WordPress comment form. Instead of simply advertising OpenID support on the “Website” field, and always attempting OpenID authentication, the plugin now detects OpenID support for a URL, and gives the user the option to authenticate the comment. This provides a cleaner, less obtrusive interface that should work on most all themes. It also gives the user the option to not authentication that particular comment if they don’t want (particularly useful if you’re on a mobile device or in a hurry and don’t want to mess with OpenID). Feel free to try it out on this post if want. You really don’t even have to submit the comment to see it in action… just enter a valid OpenID URL for the website field, and move focus somewhere else (ie, click in the comment box like you’re going to type a comment). There is currently no option to revert to the old style of comment form integration, so hopefully folks will like this new UI. If you really don’t like it, you always have the option of turning off comment form integration and modifying your theme to your heart’s content.

Finally, this release includes a lot of minor bug fixes that people have been complaining about (sorry it took so long). I’m sure I didn’t get to all of them, so please let me know what I missed, and I’ll try to do more regular minor releases with these smaller fixes.

I’ll additionally note that working on WordPress plugins is no longer part of my day job, so I currently work on them rather sporadically as I have time. The changes in this release have been developed a few hours at a time over the last couple of months. I’ve been running trunk here on my site for quite some time and haven’t had problems, but you never know. Please use the DiSo issue tracker to report any new bugs, or to remind me of existing tickets that are still not fixed in this release.

X-Apple-Store-Front: 143441-1,5

For example, the following simple curl command will retrieve this Rock page:

curl -H "X-Apple-Store-Front: 143441-1,5" \

http://ax.itunes.apple.com/WebObjects/MZStore.woa/wa/viewGrouping?id=24

To further verify that iTunes was doing little more than rendering the result with WebKit, I went through the response HTML, fixed a couple of URLs, and got this result — exactly what you see in iTunes 9. Of course, the “Quick View” for each album doesn’t work, but you can see how it is all pieced together.

So what am I to do? The most obvious answer is to simply use one of the larger hosted OpenID providers that support such a feature, like Google. But I have some real problems with that, both philosophically and practically. Fortunately, I think I may also have a better solution.

Maintaining Control

My first knee-jerk reaction to being forced to use an OpenID provider like Google is a purely philosophical argument. If the whole point of OpenID was to create a completely distributed identity ecosystem, then isn’t this a huge step backwards if we’re no longer allowing for self-hosted OpenIDs in certain cases? While I think it’s an unfortunate situation, it is a completely understandable one. There really is no good way to maintain the privacy of the user, as I demonstrated above. Remember that OpenID, unlike SAML, was never designed to mask the identity of the user… quite the opposite, in fact! When you take any technology and try to apply it in situations it was not originally intended for, you have to be prepared for the reality that it’s not going to fit quite as nicely as one might like.

So philosophical arguments aside (as valid as they may be), what other reasons might one have for not wanting to use a third-party OpenID provider? One of the biggest in my mind is privacy. Have you ever looked at the “Visited Sites” page on MyOpenID or a similar provider? It shows which sites you’ve logged into, how many times, when the last login was, etc. Pretty neat data to have at your disposal as a user, right? But also potentially pretty scary data for JanRain to have for every user. Now, I’m not suggesting that JanRain or any other OpenID provider purge this data… it’s actually a really valuable feature and I think it’s in the user’s best interest to have access to it. But let’s remember what started this conversation… directed identity.

While directed identity can be used for everyday logins, it really shines when maintaining user privacy becomes very important. What if I’m a whistleblower who wants to report unsafe business practices to the government? What if I participate in some taboo subject matter online that I want to keep hidden from family and friends. What if I am a political dissident, and revelation of my identity could result in imprisonment or even death? What if I simply want to exercise my constitutional right to privacy? While some of these are rather exotic examples, they illustrate my point pretty well. There are cases where privacy is truly important. As I demonstrated in my last post on directed identity implementations, there are algorithms that can adequately protect the identity of the user from the site they are logging in to. But remember, those algorithms do absolutely nothing for protecting my identity from my identity provider. If privacy is my goal, then why would I trust Google or JanRain with my daily activities any more than anyone else?

In Andy Oram’s post on this topic “Shortening cookies: Using OpenID to improve government privacy online” and his follow-up “Privacy and open government: conversations with EPIC and others about OpenID”, he talks about the possibility of the federal government running an OpenID provider which is used to authenticate users to other government websites and services. First of all, I don’t think the government needs to stand up yet another OpenID provider. The private sector has done a pretty good job so far of making sure people have portable identifiers, whether they actually realize it or not. But more importantly, I simply don’t trust the government to maintain and respect my privacy. In discussing whether the government is actually the right party to run this service, Oram mentions:

The problem is whether visitors can trust any particular server 1) to stay up, 2) not to go out of business, 3) not to leak information, 4) not to abuse the information for private gain, and 5) not to cave in to government pressure and release information outside of the scope of the law.

It’s really the last point that bothers me. This is a system that, by design, holds the links between the private identities of citizens, and the anonymous identifiers they are using to communicate with the government. Not only are these individuals potentially screwed if the system is broken into, but also if some judge decides there is justifiable cause for revealing these identities. Privacy is a matter of policy, rather than technical design. This is not sufficient. I think we need a system that protects user privacy as a technical feature, not simply as a policy decision. We need a system that couldn’t reveal the identity of a user even if the Chief Justice himself were to order it.

OpenID Proxy

My idea for such a system is actually very simple in its design. At a high level, it’s not unlike Emailtoid, JanRain’s RPX, or the never really launched Vidoop Connect. It is an OpenID proxy that stands between the actual relying party and OpenID provider, so that the two never actually communicate directly to one another. The proxy faces both parties and therefore implements both sides of OpenID — it provides an OpenID Provider implementing directed identity which communicates with the actual relying party, and it also implements an OpenID relying party which communicates with the actual OpenID provider. Let’s look at the basic user flow…

Our user, Alice, goes to OSHA.gov to report the unsafe work environments at her job. Fearing retribution from her employer if they were to find out she reported them, Alice wants to remain anonymous in her communication with OSHA. When logging in to OSHA’s whistleblower site, Alice enters the URL of an OpenID proxy, “proxy.example.net”. This proxy could be run by the government itself, a private citizen’s rights organization, it doesn’t really matter. OSHA’s OpenID relying party communicates with the OpenID Proxy’s server, and begins an identifier select OpenID flow, which results in Alice being sent over to the proxy. Now, instead of having to create a new account at the proxy, Alice uses her real OpenID URL, “alice.example.org” to login. After successfully authenticating, the OpenID proxy uses a hashing algorithm to generate an opaque URL for Alice, and returns that identifier back to OSHA. So Alice was able to use her own self-hosted OpenID URL in a privacy preserving manner — OSHA has no way of identifying her based on the resulting directed identity issued by the proxy.

But what about the proxy itself? As we’ve already mentioned, the OpenID provider which generates the directed identity is able to determine the user the identifier belongs to, even if a secret salt was used as part of the hashing algorithm. In order to protect Alice’s identity from even the OpenID proxy, we need to do two more things. Let’s look again at our hashing algorithm for generating directed identities:

md5( username + openid_provider + relying_party + secret_salt )

For OpenID, that username would simply be Alice’s OpenID, “alice.example.org”. If subpoenaed, the OpenID proxy would be able to determine whether a given directed identity indeed belonged to “alice.example.org” simply be running it through the above algorithm. Remember that the secret salt is what prevents the relying party from using this same technique to identify the user? Well what if we add a little secret salt of our own to the username portion of the algorithm? What if we replaced “username” with “username + user_salt”? That way, even the proxy itself wouldn’t be able to replay the hashing algorithm without actually knowing the correct salt value to plug in there.

Now, before I talk about the user salt itself, there is one caveat that must be pointed out… this is the second of the “two more things” I mentioned above. The user salt is not exactly secret, because it must be provided to the OpenID Proxy so that it can be used to generate the directed identifier. The thing that protects user privacy is that the proxy must not record the value of the user salt or the user OpenID. It can’t log the values anywhere and it can’t store them in a database. It must use the OpenID and salt to generate the directed identifier, and then get rid of them. Otherwise, it would still be technically possible to trace the identifier back to the actual user. This would be one reason why it might make sense for such a system to be run by a citizen’s rights organization that is generally more trusted to do everything possible to protect user privacy. It’s also worth noting that nothing in this architecture suggests that there would be only one OpenID proxy… there can, and should, be several proxies for user’s to choose from, all using these same (or similar) techniques.

Adding user salt

There are two methods I can think of to add the user salt mentioned above. The first is certainly the most straight-forward, but also a little more difficult for some users. If Alice’s OpenID provider were to itself return a directed identity to the OpenID proxy, that would be sufficient for salting the proxy’s hashing algorithm. So long as that directed identity is not being stored by the proxy, there would be no way to trace back any identifiers the proxy generates. It’s worth noting that in this case, a self-hosted directed identity actually would be sufficient. Remember earlier when I mentioned that the URL “http://willnorris.com/openid/9eb4d59c1d488a4” wouldn’t do much for protecting my identity? That is true, but for the purposes of salting the proxy’s hashing algorithm, it would work just fine. But this of course still requires that the user have access to an OpenID provider that supports directed identity. The WordPress OpenID plugin which I use does not currently, so this wouldn’t work for me.

The second method of providing a user salt requires a slight modification to the OpenID provider, but should be a bit easier to do. The salt could be provided separately by the OpenID provider as an extension on the OpenID transaction itself. To make things easier, it could simply be a new Attribute Exchange attribute. No need for new OpenID extensions, just a new attribute which represents a “user salt”. The OpenID proxy would then combine the user’s OpenID together with the salt value, and use that to generate the final directed identity that is returned to the relying party. If the proxy were subpoenaed to verify if a given directed identifier belonged to “alice.example.org”, it would be unable to do so without also knowing the user salt. And as long as the salt was not being recorded anywhere, the proxy would be completely incapable of verifying the user that an opaque identifier belonged to.

So what about using some other things as the user salt? How about the association used in the OpenID transaction? Remember that the point is still for Alice to be able to login to OSHA.gov over time and be recognized as the same (possibly anonymous) user. In order for this to work, she must return with the same directed identifier from the OpenID proxy each time. And in order for the proxy to ensure that it generates the same directed identifier, it must use the same input values. While an OpenID association is intended to be relatively long-lived, it can, and often does, change over time. When this happens, the proxy would end up generating a new directed identifier for Alice, and she would no longer appear as the same user at OSHA.gov.

Some practical matters

So there are some additional practical matters that I think must be considered for this solution to truly work. First of all, what happens if the proxy’s secret salt is compromised? This would certainly be a bad thing, but not quite as bad as it would be in a standard directed identity situation. Remember in our final algorithm from last time, the secret salt was the one and only key that protected the privacy of the user in a generated directed identifier. If the salt is compromised by a party, they could then brute-force the hashing algorithm and identify the true identity of a user associated with a directed identifier. But remember that one of the goals of the proxy is to protect the true identity of the user from even the proxy itself… that’s why we added the user salt. So even if the secret salt of the proxy were compromised, an attacker would be unable to identify the user associated with a given directed identifier. Additionally, if the proxy were aware of the compromised salt, they could transition over to a new secret salt using the same method I mentioned last time that USC uses to update relying parties of new USCID values. The proxy would simply inform the relying party that user X, identified by this opaque identifier, was previously identified by this other opaque identifier, and their records should be updated. Nothing about the user herself is revealed, only the transition from one opaque identifier to another.

What if the user still wants to maintain different personas for different relying parties? For example, what if Alice wants to reveal the name of her employer via an AX attribute when communicating with OSHA.gov, but not when communicating with WhiteHouse.gov? Technically, the solution is pretty simple, but it’s not as user friendly as I would like. The OpenID proxy would simply need to embed the trust root of the actual relying party inside its own trust root that it presents to Alice’s OpenID Provider. Part of the OpenID flow is that a provider prompts the user if they trust the site that is asking them to authenticate, and additionally if they want to release any additional attributes to the party. The relying party is identified using a URL known as the “trust root”. So a user may see a prompt (borrowing from MyOpenID’s language) along the lines of:

You are signing in to proxy.example.net/ as http://alice.example.org/.

Traditionally, the decision that Alice makes here is recorded so that she is not prompted each time she logs in. So if she wants to make a different decision depending on which site she is logging in to, then the OpenID proxy would need to change its trust root. One example might result in a prompt that looks like:

You are signing in to proxy.example.net/site/osha.gov as http://alice.example.org/.

This would record her decision specifically for “OSHA.gov via proxy.example.net”. Using this method, she could easily release different attributes for different relying parties, without any changes to her OpenID provider whatsoever. While I don’t like the idea of the user having to parse apart that URL to figure out what it means, it’s at least workable. As a long term solution, a new OpenID extension could be defined which provides a more human friendly version of the trust root which explains to the user more clearly where her data is going. In fact, I think that has already been proposed and being worked on.

Proof of concept

Though the explanation of the concept ends up being a little wordy, the design itself is actually quite simple. Because of the strict requirement that identifiers not be stored, there is actually not much to it. The only database storage would be for OpenID associations and nonces… everything else is calculated in real time. I’ll be working with Michael Richardson this week to be build a working implementation to show how it would work. I’m also beginning conversations with EPIC to help advise them on their recommendations to OMB regarding federal website cookie policies. I welcome any questions or comments on this approach to privacy with OpenID. Some of the specifics I just really thought about this last weekend, so I may be overlooking a few things. But the overall architecture is nothing that hasn’t been done before, and has a strong, proven track record.

Shibboleth and SAML

Shibboleth is an open source web single sign-on product which is very popular with universities around the world. It is primarily an implementation of the Security Assertion Markup Language (SAML), but supports other identity protocols as well. SAML v2 defines a specific type of identifier that may be used to identify a user within a transaction known as a Persistent Identifier. The name itself does not convey it’s full meaning, but it is defined as “a persistent opaque identifier for a principal that is specific to an identity provider and a service provider or affiliation of service providers.” This is SAML’s directed identity.

Shibboleth’s implementation of persistent identifiers has matured a bit over the years, but the main algorithm remains the same. At it’s simplest, the identifier is simply a hash of three things: an identifier for the user, an identifier for the identity provider, and an identifier for the relying party. Using md5 as our hashing algorithm, and using a few made up values, this would look like:

md5( "jsmith" + "http://idp.example.com/" + "http://sp.example.com/" ) = 
    2f6bc52c7527747eff263f4183c7f402

When a relying party receives the string “2f6bc52c7527747eff263f4183c7f402”, it is completely opaque and reveals nothing about the identity of the user.

Working with known algorithms

As I mentioned before, Shibboleth is open source software. All of our code is publicly available, including the way that we calculate persistent identifiers. So how could a relying party use this knowledge to take the above opaque identifier and decipher the identity of the user it belongs to? Pretty simply, if they have a list of possible usernames, or can reasonably guess them. A list of usernames is far easier to get than you might imagine from unprotected university LDAP directories. Then you simply iterate through the list and run the same algorithm until you find the matching hash value. And even given a population of 30,000 usernames (the approximate number of students at USC where I worked), the following shell script churns through them in about 90 seconds:

~$ date && for i in `head -n 30000 /usr/share/dict/web2`; 
   do echo $i | md5 >/dev/null; done && date
Sun Aug  2 16:25:53 PDT 2009
Sun Aug  2 16:27:35 PDT 2009

To protect against such a brute-force attach to reveal the identity of a user, we added a secret salt to the hashing algorithm. That way, you can’t regenerate the hash without also knowing the salt value:

md5( "jsmith" + "http://idp.example.com/" + "http://sp.example.com/" 
    + "my-secret-salt" ) = cf79282c587897fb733d8338fe7bc9c2

It’s worth pointing out that, while use of a secret salt prevents a relying party from brute forcing the hash, it in no way prevents the identity provider from doing so, since they do of course know the secret salt. Though we never ended up needing to do this at USC, we built the tools that would enable us to identify the user a persistent identifier belonged to. In the event that a relying party reported that a particular user was abusing their system, we wanted to make sure we could identify who it was.

The realities of deployment

The above algorithm actually works pretty well for generating unique and secure opaque values for tuples of user, identity provider, and service provider. But what happens when one of those values change? At USC, users could request that their username be changed for a variety of reasons, and approximately 300 such requests were made each year. So using the above algorithm, a username change would change every one of the user’s generated persistent identifiers. And the reality is, businesses are often bought out by other businesses. What happens when “http://sp.example.com/” needs to change to “http://sp.new-company.com/”? That will effect the generated persistent identifier for every user. How do you deal with these realities? There are a number of ways, and I’ll outline just a few. There is no one right solution, as the policy and practices of a particular institution will greatly impact their decision of how to address these situations.

Perhaps the simplest option in some respects would be to simply not change which identifiers are used for generating the hash and instead map the new identifiers to the old values. Even if a user’s username has changed to “jjones”, you map it back to “jsmith” for the purposes of generated persistent identifiers. While this allows a deployment to continue using the same hashes, it does introduce the burden of maintaining this map of identifiers. And what is the institution’s policy with regards to re-issuing identifiers? If “jsmith” is allowed to be re-issued to a different user at some point in the future, that is going to create problems.

An alternate solution would be to simply use better identifiers. At USC, we had another user attribute called the “uscPVID” which we used instead of username. This was itself an opaque identifier that effectively served as the primary key within the enterprise directory. Even if all the other data for a user changed like their name and username, the uscPVID would remain the same. The same kind of persistent key can be obtained for relying parties by creating a lookup table that maps the relying party’s public ID to some more persistent internal ID. If and when a relying party changes their public ID, you simply modify, or add a new entry in your lookup table.

Finally, an identity provider could migrate from old to new identifiers, either with a one time update, or gradually. For a one time update, the identity provider would generate the old and new identifier for a user or set of users, and provide that information to the relying part out of band. The relying party would then be responsible for updating their database accordingly. Alternately, the new and old mapping could be provided gradually at the time of user login by using attributes. This was the technique used at USC for updating relying parties of changes to a different user attribute known as the USCID. For reasons I won’t bother explaining here, this 10 digit ID could change for a user when certain events occurred. To alert relying parties, we would include two attributes — the current USCID for the user, along with a list of “historical” USCIDs for that user. Relying parties were then responsible for updating any records they had for one of the historical USCIDs to the current USCID of the user.

Application to OpenID

The above hashing method could be used with little or no modification to create directed identity URLs for OpenID users. You could of course simply generate completely random IDs and store them in the database… that works too. In my next post however, I’ll be talking about a new kind of OpenID service I’ve been doing a lot of thinking about, an OpenID proxy which works to protect the privacy of OpenIDs that don’t support directed identity themselves. We’ll be using the above hashing algorithm, but doing some interesting things with the user identifier.