leŭksman

For the last few years, desktop Linux and Mac OS X alike have used CUPS (Common Unix Printing System) as their native printing subsystem. In an ideal world, this would mean it’s really easy to set them up to talk to each other.

In the world we live in, however, things are not so easy. Configuring CUPS remains a black art, compounded by the absolutely abysmal reporting of errors to the printer UI on both OSes. To have any clue what’s going on you have to seek out and find the log files…

I can understand that on Linux, but really, how did Steve Jobs let this out the door on his precious Macintosh? Heck, Apple even bought the company that developed CUPS a few years back. Stop making our iPods smaller for a couple minutes and fix your printing error messages!

The situation:

I have a relatively straightforward setup: an Ubuntu Linux desktop PC (stormcloud.local) with a well-supported USB printer hooked up, and a Mac laptop (nimbus.local) which roams the world. When at home, it’s nice to be able to print directly from the Mac rather than print to PDF, copy the file, and then print.

The cheat sheet:

First the basics — make sure printer sharing is enabled on Linux; this much you should be able to do through the regular GUI:

Now the voodoo! Add to /etc/cups/cupsd.conf on Linux:

    # Allow remote access
    ServerAlias *
    Port 631

And restart cupsd:

    sudo /etc/init.d/cupsd restart

Now, you can add the printer on the Mac; be sure to fill everything out!

Several gotchas I discovered:

Listening isn’t enough

Very early in my journey I made sure that the Linux box’s cupsd.conf was set to listen on the network as well as to itself:

    BAD: Listen localhost:631

    GOOD: Port 631

But when I’d try to hit the CUPS web administration pages I’d just get a “400 Bad Request”. After some experimentation, I found that it actually responds just fine… as long as in the HTTP headers I call it “localhost” instead of by its proper local network name.

To get it working (so eg http://stormcloud.local:631/ would actually pull something up!) I had to add this to cupsd.conf:

    GOOD: ServerAlias *

No, setting the name I wanted in ServerName mysteriously wasn’t enough.

Pick a queue, any queue

The Mac’s IPP printer setup dialog says you can leave the “Queue” field “blank for default queue”. This is a lie! Despite having only one printer available, I could only get printing working if I listed the queue explicitly.

To add insult to injury, you need to include the “printers/” prefix. This is easiest if you find the printer on the web interface and copy-paste the path from the URL…

Now I can print my dang Fandango tickets, which I’m pretty happy about!

Drivers

I’ve been using the native driver for the printer on the Mac side. It should also work to just leave it at Generic PostScript as long as the Linux box has a driver, but I feel safer with it there.

I’ve written up the little introduction to StatusNet’s new inter-site federation protocol; for now you can also find this article on the StatusNet wiki. I’ll follow up in the next couple days with demos of the subscription UI and subscribing to Google Buzz feeds as native OStatus data sources. — brion

What is OStatus?

You’re about to learn what OStatus is and how it’s going to make communications between StatusNet sites work better!

Some of you may remember the bad old days when you couldn’t send e-mail between services like AOL and CompuServe. StatusNet/Laconica sites have used a custom protocol OMB to set up subscriptions between sites, but OMB requires special support to be used with non-StatusNet services and is hard to extend.

By building on top of shared technologies already in use and being rolled out, we can go beyond that… our new effort, with better compatibility between sites, is named OStatus.

The basics

If we step back a bit, we find that Atom and RSS feeds are already a lingua franca between sites. Virtually every service provides such feeds, giving a way for other tools and services to read someone’s outgoing posts.

At its simplest, a feed is a list of recent posts with some author attribution.

But the fun of social networks is in real-time interaction — if your updates don’t get sent to your friends right away, it’s just not the same.

Luckily, that exact problem has been solved recently, with PubSubHubbub (PuSH). In the PuSH system, a site can subscribe to updates for a feed from a hub server associated with that feed. Whenever a new post is created, the publishing site pings the hub, and the hub sends out just the new posts to all the subscribers.

OStatus is built on that simple base: each participating site produces Atom feeds of updates and uses PuSH subscriptions to send relevant updates to other sites:

Alice posts a message on identi.ca; a fragment of her Atom feed is PuSHed out to the other StatusNet sites that have users following her posts. Each of those sites is responsible for routing the message to individual subscribers.

The real beauty of it is that at this point we already have something useful, without anything StatusNet-specific. In fact you can already subscribe to someone’s public Google Buzz feed as an OStatus remote user, and they haven’t done anything special for us!

The good stuff

PuSH is a low-level piece of the puzzle; it’s just a way for services to send updates to each other. For a fully interactive social network, you need that human touch… you want to know who’s subscribing to you, which of your posts they like, and you want to get their replies to you.

For this we’re making use of more extensions that build on top of Atom feeds: Activity Streams, Portable Contacts, and Salmon.

Activity Streams extends Atom feeds to describe social “activities” more specifically than just “ok here’s a post”. Entries are marked with a verb (post, follow, favorite, etc) and more detail on who or what’s being acted on. We’ve extended this further using elements from Portable Contacts to provide profile information:

Extended author information beyond what Atom’s native <author> element provides.

Salmon was created as a way for blog aggregators to send comments back to the original blog server, “like salmon swimming upstream”. We’re using this channel to send replies as well as other user-to-user events:

Bob replies to Alice from his home site; since she’s not subscribed to him, the reply is sent back to identi.ca as a Salmon ping.

When your friend on Example.Net subscribes to your Identi.ca account, Example.Net sends an Atom entry with an Activity Streams “follow” verb to your Salmon reply channel on Identi.ca. This lets Identi.ca know who subscribed to you, so their profile gets added to your subscribers list and you get a notification that they’re following you.

If your Example.Net buddy gets social network overload and decides to unsubscribe from you, another notification comes through about the unsubscribe, and we can drop them from the list.

Other events can be sent over this channel as well, such as when your friends on other services mark your posts as a favorite.

Groups

One of the most annoying limitations with OMB 0.1 was that there’s no support for joining a group on another service. You might have noticed that groups have Atom feeds of posts just like users do… if you were wondering if that means groups could be run over OStatus too, you’re correct!

As a user on Identi.ca you can join a group on Example.Net without setting up an Example.Net account. Group posts from Example.Net will appear right in your inbox on Identi.ca just like any local group, and you can even post back to it.

Hey everybody!

Things may have looked pretty quiet the last couple weeks, but here at SNI we’ve actually been rushing to get some really awesome things ready for you all…

In addition to finalizing StatusNet 0.9 with scalability and bug fixes to make it release-ready, we’ve been implementing our new federation protocol OStatus, successor to our older OMB 0.1.

I know the scattered notes we’ve got up on the wiki right now are a little confusing, so over the next few days I’m going to start consolidating those into a clearer picture what OStatus is and how it’ll let StatusNet sites connect better to each other and to other systems.

In the meantime, active work on OStatus is currently proceeding in the ‘testing’ branch in our gitorious mainline repository…

Stay tuned!

– brion vibber (brion @ status.net)
Software Architect
StatusNet, Inc
San Francisco

Yay, it’s big commit time!

I’ve landed my XMPP output queuing work in 0.9.x, running on my public test site.  Do let me know if XMPP subscription or i/o is flaky on it!

Should be ready to merge to testing & master and deploy when we’re content to do so… there is a database change necessary for the DB-based queueing system, so I want to confirm that’s not a problem before pushing out.

Big thanks to Craig Andrews for his work on generalizing the DB queues which I’ve started integrating; we should be able to land the rest including the IM pluginization for 1.0.

Commit summary:

XMPP queued output & initial retooling of DB queue manager to support non-Notice objects.

Queue handlers for XMPP individual & firehose output now send their XML stanzas to another output queue instead of connecting directly to the chat server. This lets us have as many general processing threads as we need, while all actual XMPP input and output go through a single daemon with a single connection open.

This avoids problems with multiple connected resources:

• multiple windows shown in some chat clients (psi, gajim, kopete)
• extra load on server
• incoming message delivery forwarding issues

Database changes:

• queue_item drops ‘notice_id’ in favor of a ‘frame’ blob. This is based on Craig Andrews’ work branch to generalize queues to take any object, but conservatively leaving out the serialization for now. Table updater (preserves any existing queued items) in db/rc3to09.sql

Code changes to watch out for:

• Queue handlers should now define a handle() method instead of handle_notice()
• QueueDaemon and XmppDaemon now share common i/o (IoMaster) and respawning thread management (RespawningDaemon) infrastructure.
• The polling XmppConfirmManager has been dropped, as the message is queued directly when saving IM settings.
• Enable $config['queue']['debug_memory'] to output current memory usage at each run through the event loop to watch for memory leaks

To do:

• Adapt XMPP i/o to component connection mode for multi-site support.
• XMPP input can also be broken out to a queue, which would allow the actual notice save etc to be handled by general queue threads.
• Make sure there are no problems with simply pushing serialized Notice objects to queues.
• Find a way to improve interactive performance of the database-backed queue handler; polling is pretty painful to XMPP.
• Possibly redo the way QueueHandlers are injected into a QueueManager. The grouping used to split out the XMPP output queue is a bit awkward.

There’s a few tricks to properly supporting right-to-left languages like Arabic and Hebrew in web applications, which we currently have a few issues with in StatusNet.

I’ve written up some notes on the wiki… link on for the fun!

Woohoo! After a couple months off and on adjusting the architecture to something that seems to meet our needs, I’ve merged my refactoring of StatusNet’s background queue processing to 0.9.x.

Some of my design notes are up on the wiki, with a couple updates based on tweaks I made from my original plans.

Key items from the commit summary:

Major refactoring of queue handlers to support running multiple sites in one daemon.

Key changes:

• Initialization code moved from common.php to StatusNet class; can now switch configurations during runtime.
• As a consequence, configuration files must now be idempotent… Be careful with constant, function or class definitions.
• Control structure for daemons/QueueManager/QueueHandler has been refactored; the run loop is now managed by IoMaster run via scripts/queuedaemon.php IoManager subclasses are woken to handle socket input or polling, and may cover multiple sites.
• Plugins can implement notice queue handlers more easily by registering a QueueHandler class; no more need to add a daemon.

The new QueueDaemon runs from scripts/queuedaemon.php:

• This replaces most of the old *handler.php scripts; they’ve been refactored to the bare handler classes.
• Spawns multiple child processes to spread load; defaults to CPU count on Linux and Mac OS X systems, or override with –threads=N
• When multithreaded, child processes are automatically respawned on failure.
• Threads gracefully shut down and restart when passing a soft memory limit (defaults to 90% of memory_limit), limiting damage from memory leaks.
• Support for UDP-based monitoring: http://www.gitorious.org/snqmon

Rough control flow diagram:

Todo:

• Respawning features not currently available running single-threaded.
• When running single-site, configuration changes aren’t picked up.
• New sites or config changes affecting queue subscriptions are not yet handled without a daemon restart.
• SNMP monitoring output to integrate with general tools (nagios, ganglia)
• Convert XMPP confirmation message sends to use stomp queue instead of polling
• Convert xmppdaemon.php to IoManager?
• Convert Twitter status, friends import polling daemons to IoManager
• Clean up some error reporting and failure modes
• May need to adjust queue priorities for best perf in backlog/flood cases

Detailed code history available in my daemon-work branch.

While looking at StatusNet’s preferences dialog, I noticed we have a fairly generic but unfriendly timezone selector, done as a drop-down box with a raw list of zone names — this makes you find and select something like “America/Los_Angeles” or “Asia/Tokyo”:

Not only is the list very long and confusingly sorted, but the names are all either English or acronyms, making it hard to internationalize.

It occurs to me that we very rarely actually output formatted dates in the web interface anyway… most of the time we output relative times like “a few seconds ago” or “3 months ago”.

In fact, about the only place I can see that we’re outputting a fully formatted date is on an individual notice like this:

“Brion Vibber (brionv) ‘s status on Friday, 27-Nov-09 19:19:11 UTC”
“Brion Vibber (brionv) ‘s status on Friday, 27-Nov-09 11:19:11 PST”
etc

though we also have tooltips on the approximation eg “about a month ago”: “2009-11-27T11:19:11-08:00″

My own inclination would be to drop the timezone preferences entirely; in the rare cases where we output a formatted local date we can let client-side JavaScript do the actual date formatting with the client system’s actual time zone and language settings, with a UTC/GMT fallback for clients with JS disabled.

This drops an unnecessary and hard-to-select option field from preferences and site administration panels, and avoids inconsistencies when you travel or move and forget to update the timezone.

Any objections or suggestions for further refinement?

I had a chat today w/ Craig Andrews & Chris Vollick about StatusNet’s geolocation support and how we might be able to better present how location info will be used, to avoid freaking people out with unexpected maps to their homes.

Some notes and this mockup for expanded location setup in preferences are on the wiki.

The main points are:

• provide context for the browser’s location lookup opt-in (don’t trigger it until you’re poking at your location options)
• show a sample map in the prefs so you know what other people will see
• provide a one-click way to disable browser location lookups entirely
• provide a one-click way to reduce the accuracy to city-level, probably by default

Any thoughts?

I did a little testing on the plane yesterday to see what PHP 5.3 issues are still remaining in StatusNet 0.9; most of the problems are with upstream libraries, but they break a number of things.