PHP

First a quick note, to make sure this discussion does not get derailed. There is a time and a place to talk about these topics in the abstract. That is incredibly important work, especially in the development of these specifications, but that’s not what I’m currently interested in. I’m focused on developing solid PHP libraries to implement these technologies. Why PHP? Because that’s what WordPress uses, which is the current platform I’m targeting with the work I’m doing in DiSo. I know that PHP isn’t as sexy as Python or Ruby, but it’s what we’re using. I agree that we need solid libraries written in these other languages as well, but that’s not my focus. PHP is widely deployed and used, including companies very involved in implementing the Open Stack like Facebook and Plaxo (Luke, Joseph — I’m expecting some help from you guys :) ).

I’ll also note that I’m specifically targeting PHP 5. PHP 4 is no longer supported, and maintaining backwards compatibility (especially when talking about XML parsing) is a complete pain. This creates a problem with getting code into WordPress core, but I’m okay with that… they’ll move to PHP 5 eventually.

OpenID

Let’s start with the most mature library we’ve got. JanRain made a huge name for themselves in the OpenID community a couple of years ago by providing open source libraries in a number of different languages, including of course PHP. Like any library, there are a few weird things here and there, but by and large it is an excellent implementation that has served the community (including this developer) very well. Last week, JanRain announced that they were restructuring the development process of the PHP library to make it more open to developers. The code itself has moved from their internal darcs repository to github, they’ve added Luke Shepard of Facebook and myself as committers, and releases, bug tracking, etc will eventually be moved to the Google Code project. Going forward, we’ll be looking at trimming down the library a bit, removing support in core for older protocol versions and edge cases that weren’t really used, and overall making it easier for developers to use.

OAuth

There are two OAuth PHP libraries that I’m aware of, the “official” library stored in the OAuth Google Code project, and the Mediamatic library from Marc Worrell. The former library seems to have more users because of it’s exposure from the OAuth website, and is much lighter weight than the Mediamatic library (too much so for my taste). I initially chose the Mediamatic library for my work in getting OAuth working with WordPress, but eventually found some problems with the general library architecture. After some discussion with developers of both libraries, I’ve begun work on a new OAuth library. I re-architected the library from scratch, and then used a combination of the two libraries for much of the actual implementations. It’s probably about 80+ percent done, and should hopefully provide something both communities can work with.

Metadata Discovery

Discovery has certainly received the least amount of love from the development community, which is a bit ironic given that it’s a foundational part of almost every application of the Open Stack. There’s no shortage of metadata discovery and parsing libraries: Joseph Smarr contributed one to the xrds-simple Google Code repository, the OpenID library has its own, and the Mediamatic OAuth library has its own. Yet amazingly, none of these help you at all if you’re wanting to manipulate or publish a metadata document. They’re all half-baked, each written for a very specific use-case. What we need is a full implementation of the discovery protocols. And that, of course, is where it gets a little more complicated…

Disclaimer: If you really want everything there is to know about this subject, go read the writings of Eran Hammer-Lahav… I’m just going to gloss over it a bit.

Metadata discovery includes two steps: you need to know how to get the metadata about a resource, and you need to know what format that metadata is in so that you can parse it and make sense of it. OpenID uses a technology known as Yadis to retrieve the metadata document, which is in an XML language known as XRDS (Extensible Resource Descriptor Sequence). OAuth Discovery uses a combined and simplified version of these two known as XRDS-Simple. Discovery for OpenID and OAuth is more-or-less compatible.

Now, there is also work being done in the OASIS XRI TC (of which I’m a member) to develop the simpler, and more uniform successor to these protocols. Retrieval of the metadata will use a collection of methods known as LRDD (pronounced “lard”), while the metadata itself will be in a much simpler format known as XRD. While identical in spirit, these are complete rewrites of the previous specs. The new specs are not compatible with the old, but they are also designed so that they do not conflict either, so that both may be used simultaneously. Shifting to these new discovery protocols will certainly not be easy, but believe me when I tell you that it will be worth it. In fact, it’s absolutely essential for players like Google to implement OP-driven identifier selection (allowing users to login with OpenID by simply entering “gmail.com”).

So as I said earlier, we don’t have any real good discovery libraries for PHP. As part of my work on WordPress, I started development on a XRDS-Simple library in PHP. More recently, I created a separate branch of the code which implements LRDD+XRD exclusively. Realistically, we’ll probably need a library which handles both the old and new protocols for a while. The idea would be that none of the higher level libraries like OpenID or OAuth need worry about metadata discovery, except for maybe a lightweight wrapper around the discovery library. The new OAuth library I’m working on will do this from day one; the existing OpenID library will take a little while, but I think we’ll eventually see it rely on a separate library for discovery.

Feedback and Help

First of all, I welcome any feedback on the implementations that currently exist, especially the OAuth and discovery libraries I’m working on. They are not complete and most certainly not production ready, but they’re getting close. I’d also like to solicit development help, especially from people with larger deployments and/or a vested interest in this technology. All the new development is happening on github, so creating a clone to hack on is incredibly simple. Even if you don’t have development cycles you can put into this, I’ve already got at least one technical decision I need to make that I’d love feedback on, which I’ll be covering in my next post: “Why Does HTTP Suck So Much in PHP”.

I’ve spent a lot of time working with the WordPress authentication system. I took over the OpenID plugin for WordPress two years ago, and was hired by Vidoop last May to work on the DiSo Project full time. Last summer, Matt Mullenweg invited me to talk at WordCamp SF 2008 about OAuth. As you can see in my slidedeck, it was a lot of smoke and mirrors at that point… we didn’t have OAuth in WordPress, although it was on the roadmap for 2.7.

We’ve had an OAuth plugin for a little while that Stephen Paul Weber wrote, but it wasn’t until a couple of months ago that I finally sat down to polish it up. The first use-case we wanted to tackled was XML-RPC, so I got to work with Joseph Scott. Having OAuth authentication with XML-RPC would allow for blog clients like MarsEdit or the WordPress iPhone app to communicate with your blog without having to share your WordPress password.

Problem

It wasn’t very long before we butted up against my biggest complaint about the WordPress authentication system — it is very “username/password” centric. There are places in the authentication code where it bails out prematurely if the username or password are missing. This isn’t a problem if your plugin simply wants to authenticate the user against a different password store like LDAP; in fact that works quite well.

The problem is that there are a number of authentication systems widely deployed in the wild (SAML, OpenID, OAuth, etc) that do not fit the standard model of username and password. You can look at the OpenID plugin to see some of the more interesting things that need to be done in order to make it work on the various versions of WordPress. However, when it came to hooking OAuth into the WordPress XML-RPC endpoint, there simply was no way to hack around it… we had to change some of the underlying assumptions.

It’s worth noting one additional requirement we had. Because the wp_authenticate() function, which does most of the heavy lifting for WordPress authentication, resides in pluggable.php it is possible for a plugin to replace the function entirely and authenticate the user however they want. The problem with this solution is that many authentication mechanisms don’t know if they should be invoked without examining the request. If wp_authenticate is replaced, and then the plugin determines it shouldn’t intervene, then it’s already too late. There is no way to pass the function call back to the standard version of wp_authenticate(). This is actually the case for all functions in pluggable.php. One possible solution is to create wrapper functions for everything, which I initially advocated. Instead, Peter Westwood came up with a better solution using a well-established model, which we ended up using for the new authentication system.

Solution

It took far more planning than actual coding, but we finally developed a solution that breaks the dependence on a username and password, but maintains backward compatibility with existing plugins that hook into the authentication code. WordPress 2.8 includes a new filter called authenticate which is passed three parameters: a mixed value (either a WP_User object, a WP_Error object, or null), along with the username and password (either or both of which may be null). All of the standard WordPress authentication logic has been moved into two functions that implement this filter, both with relatively low priority.

• wp_authenticate_username_password() (priority 20) includes the standard logic for authenticating using a standard username and password. It still calls the wp_authenticate_user filter, so plugins that rely on that should be fine. This function also performs the check for an empty username or password.

• wp_authenticate_cookie() (priority 30) is only added into the filter chain when the user is authenticating via wp-login.php and does the normal checking for the WordPress authentication cookie.

Both of these functions first check to see if the first parameter passed in is a valid WP_User object, and immediately stop if it is. This allows plugins to add their own functions into the filter chain which populate the WP_User object using whatever means they see fit. WordPress still takes care of setting the authentication cookie when appropriate, so plugins need only worry with authenticating the user and returning a valid WP_User object.

So what will this look like for plugins? Well, the OAuth plugin for WordPress isn’t finished yet, but the function below should be pretty close to the final version. While there is of course a lot more code for actually implementing OAuth, this is the only hook into the WordPress authentication system needed to make it work. Note that this function doesn’t care about the username and password parameters that are available from the authenticate hook… other plugins may use them.

add_filter('authenticate', 'oauth_authenticate');

/**
 * If the current request was signed using a valid OAuth access token, verify 
 * the request and return the associated user.
 *
 * @param WP_User|WP_Error|null $user authenticated user
 * @return WP_User|WP_Error|null OAuth authenticated user, if request was signed
 */
function oauth_authenticate($user) {
    if (Auth_OAuth_Signer::requestIsSigned()) {
        $oauth_server = oauth_server();
        $user_id = $oauth_server->verify();
        if ($user_id !== false) {
            $user = new WP_User($user_id);
        }
    }   

    return $user;
}

For Plugin Authors

If you’re currently hooking into the WordPress authentication system, especially if you’re providing a custom implementation of wp_authenticate(), take a look at the new authenticate hook. If you are relying on the wp_authenticate action hook, you should also look closely to see if the new hook will do what you need. We left the wp_authenticate hook in place for now, but I’m pretty sure it’s no longer necessary and will likely be removed in future releases. If you are using the wp_authenticate_user hook exclusively, then you’re probably fine, but it’s probably still a good idea to take a look at the new stuff.

So, OAuth in WordPress?

We made additional changes to the WordPress XML-RPC code to make it use the new authentication system appropriately, so it is now possible to hook OAuth into WordPress without any core modifications. We do in fact have a basic OAuth plugin that works with the trunk version of WordPress. However, I don’t think I’m going to push to have the OAuth code included in WordPress 2.8 for two reasons:

• the OAuth libraries are in flux right now. There have been two main PHP libraries that people have used for OAuth, both with their own strengths and weaknesses. I’m currently working with the oauth-php community to combine these libraries using the best parts from each, and a new clean architecture. This effort can be found on github. (This library also requires PHP5 which is a deal breaker for WordPress… not sure how we’ll manage that.)

• Because OAuth has the potential to be such an important part of how third party clients interact with a WordPress blog, I want to make sure we get this right. Personally, I’d feel much more comfortable getting some real world experience with this code in a slightly more constrained environment by releasing it as a plugin first. Once we’ve done that and are comfortable with how it integrates into WordPress (plugin API, admin interface, database schema, etc), I’m all for making it a core part of WordPress.

Early in the project, it didn’t seem like too many people were talking openly about the state of social networking. We all commonly referred to Facebook and MySpace as “silos” and “walled gardens”, but few people were talking about what the alternative would actually look like. Now, I’m sure there were far more people working on this than I realized… I have no doubt about that. But for me, it seemed like we were somewhat alone out there trying to figure out what protocols and technologies we could piece together to build a truly distributed social network. Because of this, I often described DiSo almost as a think-tank for developing the model and the technologies. We were writing code as well because we had to have a reference implementation that proved what we were talking about was possible, but I always de-emphasized that. Code comes and goes, dozens of implementations of these protocols will be written in different languages for different platforms, and that’s fine. What’s more important then, is the protocol itself. That’s what DiSo was all about… gathering (and creating when necessary) the collection of protocols necessary to make this stuff work. At least that’s how I viewed it, and explained it.

Today, the Open Stack is becoming very real. We have agreed upon standards for service and metadata discovery, authentication, API access, and contact information. We have commitments and production deployments from key players in this space including Google, Yahoo!, MySpace, AOL, Microsoft, and many others. What seemed like a small effort between a few individuals is now a full-scale shift of how we think about social interaction online. Just to be clear, I’m not trying to take credit for any of this stuff happening. The DiSo Project has played a small role in helping to shape and direct a few of the individual discussions, but this truly is a concerted effort between a lot of people who see the real potential for what we’re trying to do.

So where does this all leave DiSo today? Well, it’s obvious now that DiSo is not the think-tank for these technologies… they’re being developed all over the web, inside and between dozens of companies. Instead, I now put more emphasis on the code that we’re writing, because I think we represent a key principal of this entire Open Stack model.

The easiest example to give a layman for distributed social networking is “being able to interact with your Facebook friends using your MySpace account.” In the future, most people will likely have accounts with one or two of the large social networks or identity providers, and participate in the open web from there. With these large networks and providers at the table now, we can ensure that we develop a solution that will give users choice and the freedom to participate from anywhere. If this is truly going to be “user-centric”, then in fact, a user shouldn’t even have to join one of these large social networks in order to participate. Just like you can setup a server to host your own email instead of using a provider like GMail, you should be able to run your own server which provides the various pieces of the Open Stack. And that’s precisely where DiSo fits in — we’re working to provide the software that lets you participate in the open web from the comfort of your own domain. While there may be advantages to using one of the large providers, in order for this system to be truly open, then users must always have the option of maintaining complete control and running everything themselves. As soon as DiSo users become second-class citizens because they are not in one of the major social networks, then we’ve failed to achieve the level of openness we sought.

Today, all of my development for DiSo is being done in PHP, and most of it specifically for WordPress. Between myself, Steve Ivy, and Stephen Weber, we have basic WordPress plugins for OpenID, OAuth, XRDS-Simple, rich profiles (hCard), Activity Streams, contacts, and permissions. Many of them have work yet to be done before they are really stable and ready for mass consumption, but many can be seen right here on willnorris.com. Six Apart is also developing implementations on top of Movable Type, including the recently announced Motion, which provides some real cool functionality around activity streams.

Posting comments

OpenID is integrated into comment posting by intercepting a comment submission to see if it includes a valid OpenID. If it does, the user is sent to their OpenID provider to authenticate, and upon their return the comment is submitted. Previously, the wp-openid plugin itself performed the comment submission, basically by copying the logic found in wp-comments-post.php. This introduced a number of problems, especially when using any other plugins that modify the comment submission process such as reCaptcha. Violating DRY is bad, but necessary at times. Breaking other plugins is really bad and had to be fixed.

The current solution I’m using is to capture the comment submission POST, do the OpenID dance, and then replay the POST (modified if necessary). If the OpenID dance results in the commenter being authenticated as a valid WordPress user, then the comment POST is modified to look like they were logged in all along. If the OpenID dance results in user attributes (via attribute exchange, sreg, hcard, foaf, whatever), then those values override what was included in the original comment form. If OpenID authentication fails for whatever reason, the idea is to give the user the option to submit the post without OpenID. This part isn’t finish yet, but will be before the release. Currently, if OpenID authentication fails, then the comment is very likely lost unless you use other means to save the comment. And of course, if any other plugins include additional data in the original comment POST, it will be included in the replayed POST.

Still left to do

Because all of the OpenID responses are being sent to /openid_consumer, it’s not quite as simple to display friendly messages to the end user. I’m may try to find a way to display error messages similar to how they look today (for example, login errors are displayed on the wp-login.php page, etc). Otherwise, I’ll just have a somewhat generic error pages that is specific to OpenID errors, and then include links back to whatever the user was doing.

Need Testers

Right now, I’m in need of people to test this new version of the plugin to find any cases I may have overlooked. Like I said, the message display is in need of work, but everything is at least functional as best as I can tell. If you’re interested in testing, checkout a copy of the latest code from the Diso Repository and give it a shot. If you have an older version installed, you will most certainly need to disable it first, then re-enable after installing the new version. Otherwise, WordPress won’t handle the /openid_consumer endpoint properly. If you have any questions or comments, you can leave a comment here or on the Diso Mailing List. As always, I would strongly discourage you from using this on your production WordPress installation (notice I’m not using it here).

I’ve been working in the Identity Management space for a few years now. I started getting involved with the Shibboleth project while at the University of Memphis. After a year and a half, I moved to California and took a job at USC working in their middleware group. I’ve spent the last two years there helping to develop and manage various parts of the Identity Management cloud including the LDAP directories, meta-directory processes, and their Shibboleth environment. In October 2006 I formally joined the core Shibboleth development team, focusing on the Shibboleth 2.0 Identity Provider.

Meanwhile, I have also been toying with OpenID for a couple of years. In early 2007 or so, I sort of took over development of Alan Castonguay’s OpenID plugin for WordPress. I started with a couple of new features, then worked to add support for the latest OpenID protocol, lots of code refactoring, etc. I got to know characters like Chris Messina, Scott Kveton, and a host of others. I continued making updates to the WordPress plugin as I had time, but it never felt like enough. Don’t get me wrong, I certainly enjoyed the work I was doing at USC and with Shibboleth… I just would have liked to have had more time for everything else as well. Every now and then Chris or Scott would prod me about going to work at Google or somewhere to spend more time on OpenID and related technologies, but I wasn’t ready to leave my work at USC.

Late last year, Chris Messina and Steve Ivy announced the DiSo Project, initially based on my updated wp-openid plugin. Within the first week after it was announced, I sat down with Chris and Steve and we decided it would be best to officially move the wp-openid plugin under the DiSo umbrella to allow for tighter integration with the other planned work. Then a lot happened this last February in the social networking space — Google announced the Social Graph API and SGFoo really got people talking more about enriching the OpenID endpoint (among other things). Things were beginning to move pretty fast, and I felt like if I didn’t jump in now then I’d end up watching all the great new developments from the sidelines. I spent the next few months interviewing with a number of companies active in this space and made a couple of trips to San Francisco to talk with them in person.

In the end, a dinner conversation with Luke Sontag had me sold. I was quite familiar with Vidoop and their OpenID provider, knew they had a great development team, but had always been a little skeptical of the company. After Luke gave me a better picture of their overall vision and where technologies like DiSo fit into that picture, I knew that these guys really “get it”. They understand the importance of what DiSo is trying to do, and more importantly they are willing to do their part in making it a reality. I love Vidoop’s OpenID implementation and have been using it since before I took this job, but that’s not why I did. I took the job because the team at Vidoop know their shit, they know the kinds of problems we’re up against, and they are ready to take a shot at developing some real solutions. Well that and I really can’t wait to get started working with Chris a lot more. :)

Testers should be able to leave an authenticated comment on this page using any OpenID 1.1 or 2.0 provider. We are aware of a bug that prevents interop with Vox OpenIDs in certain cases. Please do limit OSIS testing to this blog post. If you run into trouble, you can contact me directly, or the DiSo Project List. Happy testing! :)

Over the next week or two, I hope to get the wp-openid codebase migrated into the DiSo subversion repository. The plan is to synchronize changes made there back over to wp-plugins.org so that the wordpress.org project page as well as plugin update notifications will continue to work as they do now. Any existing bug reports at wp-plugins will also be moved over to Google Code, which should provide some better features and flexibility. If you’re interested in following the progress of wp-openid or the other DiSo projects, feel free to join the DiSo Google Group.