The Zen of Twitter Support

The Twitter support team is lead by Crystal along with two other full-time employees including one dedicated solely to spam. We also have an additional part-time employee helping out. As you might imagine, this little team handles a lot of requests and does an amazing job with precious few resources. To further improve efficiency we've just migrated to new support software called Zendesk.

We looked at several options and found Zendesk to be a perfect fit—it works seamlessly with other tools we love such as Campfire, Highrise, and they've even created a built-in Twitter integration via SMS called Targets. Plus, the Support Dropbox allows us to place one-click feedback on any web page. Check out the Zendesk blog for more info about our partnership.

What Does It Mean for You?

We expect an increase in productivity with this move which should translate to faster solutions to any issues you might experience while using Twitter. Ticket specificity means requests get sent to the right department and help resources such as tips, troubleshooting, and policies will be easier to find. Visit our new Zendesk powered site at help.twitter.com.

A significant part of support queries are in regard to policies and rules of engagement on Twitter. As part of this new support improvement, we're previewing a document called Twitter Rules which will provide more clarity around some of the questions people have when it comes to issues of content and usage boundaries.

Building on Open Source

Kestral photo by mugley

When we plan new engineering projects at Twitter, we measure our requirements against the capabilities of open source offerings, and prefer to use open source whenever it makes sense. By this approach, much of Twitter is now built on open source software.

In some cases, our requirements—in particular, the scalability requirements of our service—lead us to develop projects from the ground up. We develop these projects with an eye toward open source, and are pleased to contribute our projects back to the open source community when there is a clear benefit. Below are two such projects, Kestrel and Cache-Money. Every tweet touches one or both of these key components of the Twitter architecture.

Kestrel's Wonderful Plumage

Kestrel is a message queue server we use to asynchronously connect many of the services and functions underlying the Twitter product. For example, when users update, any tweets destined for SMS delivery are queued in a Kestrel; our SMS service then reads tweets from this queue and communicates with the SMS carriers for delivery to phones. This implementation isolates the behavior of SMS delivery from the behavior of the rest of our system, making SMS delivery easier to operate, maintain, and scale independently.

Users of the Starling message queue server will find Kestrel familiar, as Kestrel is a port of Starling from Ruby to Scala. In addition to being generally more efficient, Kestrel adds several new features, such as a facility for handling significantly bursty queues.

Robey is the lead developer of Kestrel. You can read his lively journal entry on Kestrel's latest features. Kestrel is available on github.

As Good as Cache-Money

Cache-Money is an elegant write-through caching plugin for Ruby on Rails. In write-through caching, new or updated data is first written to an efficient cache (such as memcached) and then stored in a database; subsequent requests for this data are then likely to read the data from the faster cache, rather than from the slower database. In addition to the efficiency gains associated with caching, this technique also addresses the risk of short-term replication lag between master and slave databases since data written during the lag time will likely be present in the cache. Cache-Money plugs directly into Rails's ActiveRecord to transparently provide this functionality.

Nick is the lead developer of Cache-Money. Check out his blog for an excellent introduction. Cache-Money is available on github.

It's Business Time!

Twitter receives a crushing amount of partnership opportunities on a regular basis—it's a good problem to have yet until now there has been nobody on staff dedicated solely to business development. Things are changing. We hired Kevin Thau as our Director of Mobile Business Development late last month. Although his title includes the word "mobile" Kevin is digging in on several fronts since he's our first official business development guru.

For now, Kevin is assessing all opportunities, picking up ongoing threads, and also actively working on our mobile business strategy. If you send email to our partner address or to kevin (at) twitter.com then you will be corresponding with the intrepid Mr. Thau. Kevin joins us from Buzzwire and was at Openwave introducing the world to the mobile web at the very beginning. At Openwave, Kevin worked with carriers, device manufacturers, and content providers to develop an ecosystem integral to today's global wireless data business.

Welcome, Kevin!

PS—We still have job openings for a business product manager and a director of strategic partnerships as well as systems and software engineers, operations engineers, a designer, and a founder associate. Check out our jobs page for more information.

We Got The Crunchies

Photo by Laughing Squid

Jack, Ev, and myself spent Friday evening at San Francisco's Herbst Theater for the second annual Crunchies awards produced by GigaOm, VentureBeat, TechCrunch, and Silicon Alley Insider. The venue was at capacity when near the end of the ceremonies they read our names aloud for "Best Startup Founders." We jumped onstage to accept the recognition on behalf of everyone who works at Twitter and everyone who uses Twitter—let's all share the little gorilla statue together.

Don't Blame the Geeks!

Photo by Thomas Hawk.

This week Facebook announced on their company blog, "If Facebook were a country, it would be the eighth most populated in the world, just ahead of Japan, Russia and Nigeria." While Twitter is nowhere near that scale, we no longer see significant increases in overall traffic during events like Macworld or CES. Yes, those events generate huge amounts of tweets but now that we've got a diverse set of folks from all over the world using Twitter it takes global events like the Presidential Election, the Mumbai Attacks, or massively shared events like the upcoming Inauguration and Superbowl to create dramatic peaks in our charts. For example, we saw tweets-per-second (yes, we have internal TPS reports at Twitter) jump to 10x their normal amount during the 2008 Presidential Election and we sustained 5x normal throughout the day.

So, What's with the Delays?

Over the last few days, Twitter has been experiencing intermittent delivery delays—we updated the status blog but were fairly quiet here on our company blog because we only knew the symptom, not the problem. Today, we've made some progress. We've discovered a problem in our system's interaction with memcached which intermittently effects the throughput of our timeline processing. Now that we've got visibility into this issue, we are actively working to correct it. To sum up, despite our friends at Mashable clucking "Shame on you" for not anticipating Macworld, we are in fact prepared for exuberant techno-tweeting. However, we are clearly not immune to making mistakes and finding bugs—both of which we do quite publicly on a regular basis.

Monday Morning Madness

This morning we discovered 33 Twitter accounts had been "hacked" including prominent Twitter-ers like Rick Sanchez and Barack Obama (who has not been Twittering since becoming the president elect due to transition issues). We immediately locked down the accounts and investigated the issue. Rick, Barack, and others are now back in control of their accounts.

What Happened?

The issue with these 33 accounts is different from the Phishing scam aimed at Twitter users this weekend. These accounts were compromised by an individual who hacked into some of the tools our support team uses to help people do things like edit the email address associated with their Twitter account when they can't remember or get stuck. We considered this a very serious breach of security and immediately took the support tools offline. We'll put them back only when they're safe and secure.

Reacting Quickly and Fixing the Problems

In addition to this Monday morning madness we're coming off a wacky weekend where lots of folks were tricked into participating in a Phishing scam aimed at Twitter users. In both cases, our on-call team was able to attend to the matter quickly and prevent too many people from being affected. Our support team is definitely going to have a busy week because we reset a bunch of passwords just to be on the safe side.

Could OAuth Have Helped?

We plan to release a closed beta of the open authentication protocol, OAuth this month but it's important to note that this would not have prevented a Phishing scam nor would it have prevented these accounts from being compromised. OAuth is something we can provide so that folks who use third party applications built on the Twitter API can access their data while protecting their account credentials.

What Are We Doing?

We are engaged in a full security review of all access points to Twitter. In the meantime, we are taking immediate action. First, we are increasing the security of our sign-in mechanism. For added security, we are further restricting access to our support tools. Events like this will happen from time to time to services like ours and its important how we conduct ourselves and that we take this as an opportunity to make Twitter stronger.

Gone Phishing

If you receive a direct message or a direct message email notification that redirects to what looks like Twitter.com—don't sign in. Look closely at the URL because it could be a scam.

What Is Phishing?

Wikipedia defines phishing as "the criminally fraudulent process of attempting to acquire sensitive information such as usernames, passwords and credit card details by masquerading as a trustworthy entity in an electronic communication." We've identified a phishing scam directed at Twitter users and we don't want you to get tricked into giving your password to a scammer.

How Does It Work?

This particular scam sent out emails resembling those you might receive from Twitter if you get email notifications of your Direct Messages. The email says something like, "hey! check out this funny blog about you..." and provides a link. That link redirects to a site masquerading as the Twitter front page. Look closely at the URL field, if it has another domain besides Twitter but looks exactly like our page then it's a fraud and you should not sign in. Here are some basic tips on how to avoid Phishing scams.

What If I Get Tricked?

Some folks may have clicked the link and given their Twitter password to the phishing site. In those cases it would be possible for the phisher to send out direct messages on your behalf which could trick your followers. In those cases, we proactively reset the passwords of the accounts.

So, if you find yourself unable to login to your account with your username and password, please use the reset password link to regain access. This will send an email to the address associated with your account and you'll be able to create a new password.

If you don't receive the reset password email, please check your junk or spam email folder as it may be accidentally delivered there. If you are still having trouble logging in, please contact our support team and we'll help you out.

Book About Twitter

Check it out, @jojeda has published a book called Twitter Means Business about how companies are "harnessing Twitter to engage their customers, promote their products and monitor what is being said about their brands" among other things.