You can call it congestion or throttling. I call it unacceptable.

Posted on June 12, 2014 by Andrew Nacin

I pay for 105 Mbps of downstream Internet from Comcast, and I get just 6 Mbps every evening.

I’ve been running speed tests for weeks now. It looks like this every time:

: Every day.

: Every night.

I put a sizable hole in my wallet every month for “Blast” internet. Comcast even upgraded “Blast” from 50 Mbps to 105 Mbps recently, but each evening, I might as well be using their 6 Mbps “Performance Starter.” When I actually want to use anything close to the speeds they charge me for, I can’t.

Please, Comcast, tell me how you’re going to blame Netflix for this one.

Please, Comcast, tell me again you can send a technician but only during business hours, when my Internet works fine.

Please, Comcast, tell me how you’re going to charge me more.

Please, Comcast, tell me how net neutrality would stifle “innovation” and discourage you from “investing” in infrastructure.

And please, Comcast, tell me how that would be any different from the status quo.

Security is nuanced

Posted on May 30, 2014 by Andrew Nacin

As a software vendor, there are many reasons to prefer responsible disclosure of security issues. But the most important reason is also less obvious: vulnerability reports need to be correct.

I’ve seen countless “full disclosure” reports that are wrong and invalid. Most of these could have been prevented by privately disclosing it to the vendor and allowing them the opportunity to respond.

Everyone sees WordPress in the headlines, over and over again, but no one ever notices the “this report is invalid” response. We’ve prevented countless invalid reports from being published simply because they were disclosed to us first. This is better for the software’s users, who are otherwise left to scramble every time they see a report. They are not an expert, and often, neither is the reporter.

There’s another angle here, though: sometimes, the vulnerability report is correct, but incomplete. This too can send everyone scrambling, starting with the vendor who was not given the opportunity to straighten things out.

When responsible disclosure works, it works really well. I offer Jetpack’s recent vulnerability as a recent case study. While this was discovered internally at Automattic, the effect is the same: they were able to quickly and thoroughly investigate the issues and follow through with a plan of action. During their investigation they learned the problem was far more severe than they originally identified, and that their initial proposed fix was incomplete.

There will always be individuals who want everything to be fully disclosed, and there are some great arguments for that. I’m not trying to sway you one way or the other. But if you’re trying to do the right thing — you’re doing full disclosure in the interest of users, possibly even providing a patch or steps to mitigate — working with the vendor is a good way to ensure you haven’t missed anything.

I’m not sure how many times now I’ve responded to a mailing list saying, “well, this isn’t a vulnerability,” or “your proposed patch is incomplete,” or “your patch makes it worse.” Or when I don’t respond to the mailing list because my thought is This is worse than the reporter realizes.

Security is nuanced.

Last week, a security researcher disclosed some issues with WordPress. They stated that WordPress issues a cookie in plain HTTP that identifies the user. Correct. They also stated that this cookie does not expire when the user logs out. Also correct.

She went on to state that the cookie has a lifetime of 3 years, that it can be used to mess with the account’s two-factor authentication settings, and change their email and other settings. She followed that up with “have notified them and waited 24hrs for response, none yet. guessing it’s a wontfix for now, since their SSL support is patchy.”

Yes, absolutely, SSL support in WordPress is patchy. (Coincidentally, fixing all of this was already slated for the next major release.) While you can force SSL for the dashboard, we don’t have the concept of forcing SSL for the front-end of the site. So, that “logged in” cookie is issued over plain HTTP. It’s trivial in a plugin to force that cookie to be secure, but of course most users aren’t going to do that. Of course, most users also aren’t using SSL (sadly).

WordPress also doesn’t have any concept of session management. The user is authenticated and a cookie is issued, but there’s no way for it to be automatically invalidated upon logout (without you changing your password). Changing this is a major architectural change we’ve been planning for some time. But this is also why cookies are designed with a limited expiration: just 48 hours, or 14 days if you click “Remember me.”

But the report was that it didn’t expire for three years, right? This is where it gets a bit weird. I’d love it if “Remember me” remembered you for 30, 90, or even 365 days, but WordPress will wait to make any changes until cookies can be invalidated. Some time ago, though, WordPress.com configured this cookie to last for three years.

Normally, this wouldn’t be that bad. You see, the “logged in” cookie is relatively harmless. It allows presentational things like the toolbar, edit links in your theme, and such. But you can’t use it to manage your account, change your email, or do anything particularly crazy. In fact, WordPress issues a separate, secure “auth” cookie used for the “wp-admin” dashboard. It’s like Amazon knowing who you are after six months of not being on the site, but asking you to log in when you start to check out.

On WordPress.com, however, a lot of settings can be managed outside of the dashboard, on their “new dashboard,” at wordpress.com/settings/. As you might have guessed by now, they were not requiring the “auth” cookie on this page, only the “logged in” cookie. That is the true critical vulnerability here. A number of decisions came together to make a latent issue a very real one. The issue wasn’t reported like this because the reporter isn’t familiar with the intricacies of user authentication internals in WordPress, nor should anyone expect them to. Rather, an assumption was made it was a design decision that would probably just be a “wontfix” due to “patchy” SSL support.

The disclosure was well-intentioned, but because it happened within just 24 hours, Automattic wasn’t able to react quickly enough to identify the actual issue and request that it not be disclosed until they got a fix in place.

I’m just trying to set the record straight on what happened, since I keep seeing confused tweets, blog posts, comments, and Facebook chatter. (Favorite Facebook comment, after I posted a brief explanation: “Did we just open all those nesting doll layers to discover the belly was empty?”) This situation probably could have been handled better by everyone involved, but that’s not really the point. The primary issue is now handled, and other issues are also being addressed. I don’t think this requires post-mortem blog posts with a side of FUD. I’m not looking for a debate on full disclosure versus responsible disclosure. And I certainly don’t think anyone should be questioning the researcher, who has spent years dedicating herself to making the Internet more secure.

In this case, WordPress.com the service was able to react quickly to mitigate this issue for all of its users. Of course, with software, holes can’t be closed so easily, and in this case, the situation was probably exacerbated by confusing WordPress.com the service with WordPress the software. That confusion is easy to understand, especially since some of these issues have roots in the core software.

Security is nuanced.

I’ve not linked to any previous posts because I don’t want this to be a criticism of any researcher or writer, but if you’re looking for background: 1 2 3. Also, related work for WordPress 4.0 is happening here.

My Talk Proposals for Open Source Bridge

Posted on April 15, 2014 by Andrew Nacin

Update, April 29: I’m excited to share that both of these proposals were accepted. See you soon, Portland!

I submitted two proposals to Open Source Bridge, an annual conference in Portland, Oregon, for “open source citizens.” The call for papers is now closed, but they let anyone leave comments on proposals that are private to the conference organizers. If you have any feedback on these, or have seen me speak before, it would be awesome if you could leave a comment. Here they are:

Extreme Software Portability as an Art Form

Writing portable software is hard. Throw in thousands of bad and worse shared hosting configurations, a decade of technical debt, the need to cater to a sprawling ecosystem, and PHP — and you have WordPress. We’ve found breaking changes harm our community and unfairly punish our users, so we don’t make them. But that doesn’t mean we don’t innovate or evolve — we’re just forced to get really clever. And it works, with adoption continuing to soar.

Trust, Community, and Automatic Updates

WordPress shipped in October what is perhaps its most polarizing feature ever — automatic updates in the background of self-hosted web software, on by default and no easy way to turn it off. In most open source communities, this would be cause for open revolt. Learn how through trust, communication, and a steadfast commitment to its philosophies, the WordPress core team convinced a skeptical community to go along, even if it meant users giving up some control.

WordPress contributors Mel Choyce and Aaron Jorbin also both submitted proposals: My Journey into Open Source Design and Modernizing a Stagnant Toolbox.

An hour to make government better

Posted on March 24, 2014 by Andrew Nacin

I had the pleasure last week of spending not quite an hour of my time at the U.S. General Services Administration reviewing the public APIs of three federal agencies. We all sat around a table and they let me rant while I worked to understand their documentation, tested their APIs, and discussed what could improve the developer experience. They took copious notes, had great questions, and were even deploying adjustments on the fly.

In talking to GSA’s Gray Brooks, I learned dozens of federal agencies are lining up for these review sessions. It’s an exciting time to be a consumer of government data, and it makes me really happy that they’re encouraging citizens to actively shape this process. It was obvious how helpful it was for these government technologists to hear an outside perspective. If your local government is on the path of “Gov 2.0” see if you can offer even just a little bit of your time to share your expertise.

Unlike someone like Eric Mill of Sunlight (who also gave a round of feedback yesterday), I’m not a regular consumer of government APIs. But as a lead developer of WordPress, a citizen, and a taxpayer, I feel very strongly about an open, transparent, and efficient government; open government data; and an open web enriched by accessible APIs.

You can follow the GSA’s efforts at 18f.gsa.gov, @18f, and GitHub. For the record, the U.S. agencies were ForeignAssistance.gov (State/USAID), FEMA, and the FDA.

Bonus: About three years ago, when Ben Balter was at the FCC, he wrote an API Terms of Service that was heavily inspired by WordPress.com’s own TOS. Ben is now a government evangelist at GitHub — where you can now find WordPress.com’s TOS. Anyway: it turns out that the TOS he wrote for the FCC is now used by more than 20 other federal agencies, and the GSA is pitching it as a common template for all agencies. That’s quite the body of derivative work.

The qualities of a great WordPress contributor

Posted on February 7, 2014 by Andrew Nacin

“A few years ago, there were only a handful of people with commit access. Now there are 15 or so with permanent commit. How has this benefited the development of WordPress? In your opinion, is it possible to have too many people with core commit access? What advice can you give for those looking to either gain temporary or permanent access?”

Jeff at WP Tavern asked me these questions about a month ago, and while writing a reply, I realized it’d make a great blog post. I’m going to answer these questions, but I also strongly believe you don’t need commit access (or even desire it) to be a great contributor who commands respect and influences the project. So perhaps the most important question I hope to answer here is this: What are qualities of a great open source contributor?

A committer is a contributor with the ability to modify the main WordPress repository. If you’re used to a decentralized model, this would be those with push/write access. There have been around three dozen committers to WordPress over the last decade; almost twenty currently have access. But in 2009, there were just five.

The lead developers all got together for the first time at WordCamp Orlando in December 2009, right around the time I started contributing. (So no, I was not there.) At the time, there weren’t any committers who weren’t lead developers. They set a goal to expand the number of people. When they added Dion Hulse that January, Matt wrote:

One of the goals for the team in 2010 is to greatly expand the number of people with direct commit access, so the emphasis is more on review and collaboration. Right now commit access is tied up with being a “lead developer,” of which we’ve always found a small group of 3-5 works best, but now we want commit to be more a recognition of trust, quality, and most importantly activity, and something that can dynamically flow in and out as their level of commitment (har har) changes and decoupled from the “lead dev” role.

I think it’s great to empower and reward contributors with commit access. When Matt and I are together, he likes to challenge me with one particular thought experiment — he’ll ask what would happen if we just gave everyone the ability to push a change to WordPress. Obviously, that wouldn’t work. But at what point does it not work? 10? 50? 100? 1000? I don’t think it’s a raw minimum or maximum number. I think it has a lot to do with our philosophies and how we organize ourselves, for example.

How to be a great contributor

I’ve been involved in identifying and mentoring new committers for a few years now. Certainly, quality of contributions is important, but it’s only one small piece of the puzzle. Especially as contributors start to start review contributions by others, you start to see a number of other qualities. For example, what’s their judgment and temperament like? Also, you can have the most amazing judgment out there, but without strong communication skills, we’ll never know.

Attention to detail is extremely important. As a committer taking on the shared responsibility of maintaining a large, sprawling codebase, you need to always be thinking about edge cases, big ramifications of seemingly small changes, backwards compatibility, etc. We’re building something pretty amazing, but at a fifth of the internet, it’s also being run at such an incredible scale that all changes are risky. We’re very cognizant of this: once we hit a release candidate, every commit needs to first be reviewed by two lead developers, even if it’s a lead developer who authored it. It helps to be honest when, for example, you know you are not confident. It’s important to be humble, which includes knowing when to ask for help, when you’re wrong, and when to change your mind. Along these lines, I like citing software developer Havoc Pennington:

In the presence of good rationale, maintainers should be willing to change their mind often.

It’s not about making mistakes, because we all make them. Being thorough and acutely perceptive just goes an incredibly long way to writing and reviewing code. (Word of the day: perspicacious.)

We also take a lot of pride in building user-centric software. You don’t need to have amazing UX skills, but strongly believing in our core philosophies goes a long way. These philosophies have roots in another of Pennington’s essays on creating good user interfaces.

There is a lot of history embedded in the code, and we don’t make changes without a deep understanding of the code around it. So it’s important to want to learn it. You need to be curious and love to learn. At the same time, the more you work with a codebase, the more frustrated you’re bound to get with it, so you need to have a lot of patience. For years I’ve not only been building a mental map of WordPress in my head, but I’ve been overlaying it with a potential roadmap.

Patience when communicating with others can reveal a lot about temperament. Thousands of people represent the WordPress project in some way, so it’s important for contributors to lead by example. If a person feels he was treated poorly when he reported a bug, he’s not likely to contribute again. Karl Fogel covers an aspect of this in Producing Open Source Software:

Sometimes someone shows technical skill and an ability to work within the project’s formal guidelines, yet is also consistently belligerent or uncooperative in public forums. That’s a serious concern; if the person doesn’t seem to shape up over time, even in response to hints, then we won’t add him as a committer no matter how skilled he is. In a volunteer group, social skills, or the ability to “play well in the sandbox”, are as important as raw technical ability.

Committers are essentially ambassadors of the WordPress project, so it’s important that they lead by example. It doesn’t serve the project to give a leadership role to someone who is often rude, cantankerous, or overly critical. Along the same lines, knowing how to actually handle a “poisonous person” is helpful. It’s good to reach out to contributors to talk about this kind of behavior.

How project priorities affect contributions

So what advice can I give to those looking to become a committer? It probably isn’t something I’d make a goal. I’d just work hard to make WordPress better. Regardless of your “access” you are sure to make an impact and receive respect from others.

I got my start doing a lot of thankless tasks during the 2.9 and 3.0 cycles. I tested the heck out of some of the newer features like trash, and weighed in and patched bugs. When multisite started to be merged in 3.0, I would sit there for hours dissecting it and seeing what I could do to understand it (and clean it up). I volunteered for tasks no one else wanted to work on because I knew it was important to the project. I’d look out to see which tickets were being flagged for the current release, then tackle them, one by one. It was definitely noticed that the committers could reliably turn to me during crunch times to get things done. Being responsive to the needs of the project is key.

I’m not suggesting you need to work on things you don’t enjoy. You can totally make the project better while sticking to an area of interest. You could even spend time in an area that overlaps significantly with what you’re employed to do, if contributing isn’t just a hobby for you. For example, Scott Taylor initially invested a lot of his time on making audio and video support better while he was at emusic (though music is certainly also passion of his). A lot of contributors have stepped up to improve an API after digging deep into it for a project.

Worth mentioning: Discussing with your employer about contributing is not an easy conversation. While hiring based solely on open source contributions can be fraught, companies that encourage their employees to contribute are helping themselves, too. Contributing not only gives back or helps to establish a voice in the community, but it’s also an enriched learning environment. I learned more in my first three months contributing than I did in three previous years of web development. It never hurts to have a subject matter expert on staff. And if you’re a consultant, you’ll be worth more to your clients.

Many committers are “generalists” with a wide knowledge of WordPress, who have contributed extensively to all sorts of areas. Some are “specialists,” who work hard on a particular feature (perhaps a feature plugin), component (like multisite), or have a particular focus (like JavaScript). Becoming an expert in an area is hugely beneficial to the project. Some of our more recent process changes mean we are trying to empower more people to help “maintain” these different areas. This is great because it means more contributors will be able to make bigger differences, whether it’s triaging new or old tickets, providing feedback, helping to build out a roadmap for a particular component, or mentoring newer contributors. Not everything is exciting (like digging through old tickets on Trac, as if looking for diamonds in the rough) but your contributions will definitely be noticed.

You may not see it right away when you start out as a contributor, but it’s important to place the project’s goals ahead of your own. You might have reported a dozen tickets and it’s possible you disagree with how ten of them were handled. You can and should work to fix tickets you reported, but at some point, you’ll hopefully run out of those and need to look elsewhere to get your fix. Many contributors — including every committer — have “wish lists” or “pet projects” or “pet bugs” that they still haven’t gotten to, because they realize the project isn’t ready for them, or because their solution isn’t ready for the project, or because it simply isn’t a priority.

Some have said contributing to WordPress is more politics than code. I don’t think that’s true, but it can be easy to think that when you wonder why some things are moving faster than some individual ticket (of thousands). Scott Taylor recently wrote:

For a lot of people, I know the wait can be frustrating, but I would try viewing it in a larger context: we all have things we would love to go in immediately, but as responsible committers, we have to weigh the pros and cons of tossing code onto 20% of internet. I have tickets that have been open for 7 releases, but for each I either:

don’t think the idea is fully-baked

don’t have absolute confidence that the feature/code is necessary

haven’t made a good enough case for it

haven’t provided bulletproof evidence that it’s going to make WP better

How the WordPress lead developers choose committers

In the interest of transparency and avoiding mystery, let me explain the specifics of the decision-making process. For a few years now, we’ve been granting commit access for individual release cycles at first, on a temporary or “guest” basis. It’s often because they’re working on a particular feature, though it of course can double as a trial period. Sometimes a contributor is given commit access to a particular area or focus. (We use a relaxed approach for partial or “component” commit.)

The lead developers keep pretty close tabs on what’s going on, and we always have people in mind for commit access. Typically, someone catches our eye and we’ll privately mention it to each other. This can happen very early on — sometimes, someone’s first bug report or patch just knocks your socks off. If you do good work, you will get noticed. A potential committer will often end up working on projects one-on-one with an existing committer. These projects end up being great opportunities to mentor that contributor, see what they’re interested in, get a feel for how they think, and so on. We make it a point to request feedback from other core developers and guest committers, what they think of so-and-so.

The lead developers discuss commit access over email before each release cycle. We’ll review current guest committers and proposed new committers. We mention contributors we’re keeping an eye on. Nothing is rigid about this process. There’s no formal “voting” procedures; there’s never been anything short of a consensus. A proposal can come at any time, not just at the start of a cycle (that’s just when we check in on guest committers). A few times we’ve essentially agreed ahead of time to grant commit access once conditions are met. Some examples: they’ll be ready in another month or two; we want a documentation committer, so let’s see who steps up after we start this initiative; or let’s try to get the main developer of this feature plugin commit access if the merge goes well. (Building features as plugins first has perhaps lessened the need for granting someone commit access for a cycle, but I think it’s too early to tell.)

Once a committer accepts, I usually go over a few guidelines. They go something like this:

For your first half-dozen commits or so, run them by me or another lead (either in IRC or privately) just while you get the hang of things.
The first sentence of your commit message gets used as the subject for the wp-svn mailing list. Add more details after that and don’t forget props or tickets.
Try to be on IRC when you’re committing; for the occasional instant feedback.
Aim to wait for some kind of consensus on the big or controversial stuff. Even Ryan and I, for example, rarely move on anything hefty without the other glancing at it. On the other hand, don’t let us slow you down — we are known bottlenecks.
Ryan and I miss things weekly and we have 12,000 commits between us. Code can always be reverted. Peter will revert you at least once; wear it as a badge of honor.

Often, the final trigger for commit access is I’m getting tired of committing all of your patches, because I never need to do anything to them. That’s what Ryan Boren told me on February 8, 2010, when he asked if I wanted commit, four years ago tomorrow. I’ve been really lucky to have had the pleasure of passing that sentiment on.