Using ReaTiWe like a SuperFeedr subscriber

Like maybe you know, SuperFeedr have also PubSubHubbub implementation. ...With a few specific things:

• you need to authenticate your HTTP requests - HTTP Basic Authentication
• do not support hub.secret
• only allow port 80 for your callback urls
• only supports "sync" verification

To support their implementation (and future basic authentication protected PuSH implementations), I first added setting PuSH hub per subscription . Adding new field in the subscription form on the Settings page and additional parameter to the sub bot command was easy.

More difficult problem was the authentication...Until now I avoided keeping sensitive information in the system - Twitter, XMPP or other passwords etc. But for using SuperFeedr as a hub, you need to add your SuperFeedr account username and password to the hub URL in the form:

http://username:password@superfeedr.com/hubbub

The python code, I'm using to convert URLs with username and password to authentication header is:

import hashlib, urllib, urllib2, base64
from urlparse import urlparse
...
url = self.request.POST['hub']
req = urllib2.Request(url, urllib.urlencode(form_fields))
o = urlparse.urlparse(url)
if o.username and o.password:
    base64string = base64.encodestring('%s:%s' % (o.username, o.password))[:-1]
    authheader =  "Basic %s" % base64string
    # recreate the hub URL without the username:password part
    new_url = "%s://%s%s" % (o.scheme, o.hostname, o.path)
    req = urllib2.Request(new_url, urllib.urlencode(form_fields))
    req.add_header("Authorization", authheader)
urllib2.urlopen(req)

Other problem with the SuperFeedr PubSubHubbub implementation was their requirement for a valid PuSH callback even on unsubscribe action. Until now, I removed the subscription in the POST action and used a deferred task only for sending the unsubscription request to the hub. In that case however, SuperFeedr system refused to remove my subscription, because the callback do not exists, when they receive the request. No such problems with the Google PuSH reference hub . So to make the story shorter, I moved the removing of the subscription from the database in the deferred task.

Still some problems...Mostly because of the (in)famous DownloadError: ApplicationError: 5 Google AppEngine problem - timeout error on the requests, taking longer than 10 sec.