Writing a twitter service on App Engine

Posted by Nick Johnson | Filed under app-engine, python, twitter, coding, tech, oauth

Services that consume or produce Twitter updates are popular apps these days, and there are more than a few on App Engine, too. Twitter provide an extensive API, which provides most of the features you might want to access.

Broadly, Twitter's API is divided into two distinct parts: The streaming API, and everything else. The streaming API is their recommended way to consume large volumes of updates in real-time; unfortunately, for a couple of reasons, using it on App Engine is not practical at the moment. The rest of their API, however, is well suited to use via App Engine, and covers things such as retrieving users' timelines, mentions, retweets, etc, sending new status updates (and deleting them, and retweeting them), and getting user information.

Authentication

Most of Twitter's API calls require authentication. Currently, Twitter support two different authentication methods: Basic, and OAuth. Basic authentication, as the name suggests uses HTTP Basic authentication, which requires prompting the user for their username and password. We won't be using this, since it's deprecated, and asking users for their credentials is a bad idea. The OAuth API makes it possible to call Twitter APIs on behalf of a user without knowing their password, and that's what we'll focus on today.

The key features of OAuth is that every consumer - that's you - needs a 'consumer key' and a 'consumer secret'. As the names imply, the first one is a key that identifies you, while the second is a secret value, known only to you and Twitter. Together, these allow you to prove to Twitter that you are who you say you are. In addition, for each user you authenticate, you'll need to store a user token and a user secret. These operate in the same way as the client token and secret, and allow you to prove to Twitter that you're making legitimate requests on behalf of that client.

In order to get permission to act on behalf of a user, you need to go through an authentication process. This consists of sending the user to a URL on twitter.com, where Twitter asks them if they want to authorize you to use their account. If the user agrees, Twitter redirects the user back to your site, embedding in the URL the details required. After that, you have all the necessary keys and secrets to use the API as that user (until they revoke your access!)

OAuth is a relatively straightforward protocol, but it'd be nice if we didn't have to implement it ourselves. Fortunately, Mike Knapp has already done all the hard work for us, in the form of the AppEngine-OAuth library (that link goes to my own fork of it, which has a few improvements that haven't yet made it into the mainline). This library takes care of all the nitty-gritty of OAuth authentication and making OAuth requests, making things Just Work.

Using AppEngine-OAuth

The first thing you should do is go to Twitter and create an OAuth consumer. Take the key and secret you're given, and store it somewhere - as a configuration variable in your code, or in the datastore, wherever suits. Next, download the library from the link above - you only need the file 'oauth.py', and place it in your app's root.

There are three major components you need to integrate: Sending the user to Twitter to be authenticated, handling the redirect back to complete authentication, and making API calls using the credentials. We'll tackle these in order.

Starting the Authentication process

Initiating authentication is easy. First, construct an oauth.TwitterClient:

consumer_key = "LKlkj83kaio2fjiudjd9...etc"
consumer_secret = "58kdujslkfojkjsjsdk...etc"
callback_url = "http://www.myurl.com/callback/twitter"
 
  client = oauth.TwitterClient(consumer_key, consumer_secret, callback_url)

Here, callback_url should be the complete URL of the callback handler in your app. Next, you can generate a URL and redirect the user to it as follows:

self.redirect(client.get_authorization_url())

Completing Authentication

When the user is done at Twitter, they will be redirected back to the callback URL you provided. Handling this requires constructing another instance of the TwitterClient, as above, and calling get_user_info on it:

class CallbackHandler(webapp.RequestHandler):
  def get(self):
    client = oauth.TwitterClient(consumer_key, consumer_secret, callback_url)
    auth_token = self.request.get("oauth_token")
    auth_verifier = self.request.get("oauth_verifier")
    user_info = client.get_user_info(auth_token, auth_verifier=auth_verifier)

The 'user_info' variable here is a dict containing the relevant information about the authenticated user. Of particular interest is the "token" and "secret" keys, which we need to store to authenticate future requests, and the "username" key, which identifies the user. Services like Twitter also return additional keys - for instance, "name", which is the user's display name, and "picture", a URL to their avatar. If you want to see a real working example of a callback handler, here's Tweet Engine's one.

Making Authenticated Requests

Now that you've authenticated the user and stored their credentials, you can make authenticated requests as them. Again, the OAuth library makes this easy. We once again construct a TwitterClient instance, and call .make_request on it, passing in the URL, the user's OAuth token and secret, as well as any additional parameters:

client = oauth.TwitterClient(consumer_key, consumer_secret, callback_url)

additional_params = {
  status: "Testing Twitter OAuth",
}

result = client.make_request(
    "http://twitter.com/statuses/update.json",
    token=client_token,
    secret=client_secret,
    additional_params=additional_params,
    method=urlfetch.POST)

The 'result' variable, here, is a urlfetch Response object, which you can treat as you would the result of any other urlfetch call. Again, for a real example, you can check out how Tweet Engine does it, though in this case it's slightly complicated by some extra layers of abstraction we haven't covered here.

Rate Limiting

One concern a lot of users have about using Twitter via App Engine is rate limiting. Twitter has two different rate limiting systems: per-IP, and per-app/per-user. This comes up a lot with users of the search API via App Engine, because all App Engine apps make external requests via the same pool of IPs. Authenticated requests, such as those we're making, are all rate limited per app and/or user, so we don't have to be worried about the per IP ratelimits.

That's all there is to using Twitter's OAuth API on App Engine. Have a use-case in mind? Let us know in the comments!

17 February, 2010

Comments