leŭksman

Now that we’ve got UI translations going pretty well for StatusNet core via TranslateWiki, I’d like to see if we can get things working for plugins as well before the 0.9 final release.

I’d like some quick feedback from folks before merging into 0.9.x; you can see my work branch here.

In core code, to make a string translatable we wrap it in one of the gettext functions like this, most often the _() shortcut:

 $this->text(_("You have a new message."));

If you’re a plugin though, you need to add your own translations into the mix, and gettext makes you tell it which “domain” you’re pulling from each time…

 // at init time
 bindtextdomain("AwesomePlugin",
                dirname(__FILE__) . "/locale");
 ...
 $this->text(dgettext("AwesomePlugin",
                         "You have a new message."));

Repeating your plugin name for every string gets old REAL fast!

In my work branch I’ve added a new gettext wrapper function _m() which knows if it’s been called from a plugin, and picks out the right domain for you based on the plugin directory.

So rather than playing around with bindtextdomain() and dgettext() manually, you can just add one character and not worry about the rest:

 $this->text(_m("You have a new message."));

It also knows how many parameters you passed to it, so instead of whipping out ngettext() or dngettext() (or god forbid dnpgettext!) you can just keep using the nice compact _m() when your needs get fancier:

Plurals:

 $this->text(sprintf(_m("You have a new message.",
                        "You have %d new messages.",
                        $messageCount),
                     $messageCount);

Contexts for disambiguation:

 // read vs delete
 $this->text(_m("message-action", "Read"));

 // read vs unread
 $this->text(_m("message-state", "Read"));

Plurals with contexts!

 $this->text(_m("message-state",
                "Read",
                "Read",
                $messageCount));

Plugins maintained in the main StatusNet repo shouldn’t need to worry about anything else — the .po file templates and updates via TranslateWiki will be handled through the same workflow as the core. (I’m working with Siebrand at TranslateWiki to make sure we can automate this as much as possible, including adding new plugins.)

scripts/update_po_translations.php can regenerate all (or just core, or just plugin) .po templates for those who want to push them in immediately, though.

As with core, the binary .mo files won’t be included in the git repository to simplify code maintenance. I’ve added a Makefile at the base level which’ll build all the .mo files for folks to test localization in their working copies (or to build a release!)

I’ve been getting feedback for some time that this checkbox in StatusNet’s Twitter connect settings is confusing:

People tend to think this should mean that this’ll pull all your Twitter friends’ updates into your timeline on the SN site… which is actually a separate checkbox, when enabled (sorry, it’s still disabled on identi.ca):

I suspect we could come up with clearer wording for both of these… let’s start the bidding!

  [x] Subscribe to my Twitter friends’ accounts on %%site.name%% automatically.

  [x] Show my friends’ tweets here on %%site.name%%.

Any other ideas?

If you’ve been having trouble compiling your own PHP installations on Mac OS X 10.6, here’s the secret to making it not suck! After running the configure script, edit the generated Makefile and make these fixes:

• Find the EXTRA_LIBS definition and add -lresolv to the end
• Find the EXE_EXT definition and remove .dSYM

Standard make and make install should work from here…

For reference, here’s the whole configure line I currently use; MySQL is installed from the downloadable installer; other deps from MacPorts:

‘./configure’ ‘–prefix=/opt/php52′ ‘–with-mysql=/usr/local/mysql’ ‘–with-zlib’ ‘–with-bz2′ ‘–enable-mbstring’ ‘–enable-exif’ ‘–enable-fastcgi’ ‘–with-xmlrpc’ ‘–with-xsl’ ‘–with-readline=/opt/local’ –without-iconv –with-gd –with-png-dir=/opt/local –with-jpeg-dir=/opt/local –with-curl –with-gettext=/opt/local –with-mysqli=/usr/local/mysql/bin/mysql_config –with-tidy=/opt/local –enable-pcntl –with-openssl

Some of StatusNet’s awesomer features like the XMPP and Twitter bridges require running background daemons to watch event queues or keep connections to XMPP servers open.

Alas, this just isn’t going to scale to the future StatusNet 1.0 world where we’re going to be running thousands of instances for our hosted services. We see a lot of problems with memory leaks and instability in those daemons with even just a few live sites now…

I’ve worked out what I think is a feasible architecture for a more scalable queue/daemon system for big sites to use; details and some diagrams I whipped up to help me keep my head straight are up on the wiki:

I’m planning to use a single lightweight master daemon which will maintain long-running connections to the ActiveMQ queue and XMPP daemons. Actual event handling will still be done at the PHP/StatusNet level, but now as short-running processes which will handle a single event and exit.

This reduces the surface area for memory leaks and other oddities we encounter in long-running PHP scripts, and most importantly means we only have to run as many processes are there actually are events being handled!

Please feel free to give some feedback before I jump into implementation.

– brion vibber (brion @ status.net)
Senior Software Architect, StatusNet
San Francisco

Adium 1.4b10 has just been released, with fixes for StatusNet support:

Menu items are now labeled as StatusNet rather than the old Laconica name, and more importantly you can now select whether to connect to the server over SSL or not — in previous Adium betas it always tried to use SSL, which worked with identi.ca but failed for many third-party sites.

I’m poking about at the Realtime Web Summit, just got out of the PubHubSubbub session.

PuSH is a relatively lightweight protocol for pushing feed updates in more or less real-time using standard web protocols (eg, HTTP!). As currently spec’d, PuSH covers the server-to-server replication space pretty well — publishers send their updates to hubs, which send them on to the callback URLs given by subscribers.

For StatusNet, we’re really interested in two possible extensions to this, which would be outside the scope of the current PuSH spec:

Microblogging metadata extensions. PuSH deals with RSS and Atom feeds, but doesn’t really care what’s inside them. Microblogging and other social-type services will have various metadata — profile name & avatar, friend relations, ratings, comment id links, etc — which could be embedded into activity stream feeds, allowing different services to handle remote subscriptions interoperably.

Of course, we could just push everybody to support OMB… but the PuSH model may be more flexible, allowing subscriptions to blogs etc to be aggregated into your notice stream.

“Last mile” push to clients. There’s still no standardization for real-time communications from an aggregator or social service to end-user web, desktop, or mobile clients. PuSH as spec’d can’t handle this since it needs a URL to post updates to subscribers, which a NAT’d or mobile client obviously isn’t going to have.

A more or less standard way to attach XMPP or long-polling to pull updates from an aggregating hub to client end-points would be very nice; just like common use of the Twitter API has allowed interop between client apps and services (many Twitter clients will happily speak to identi.ca if you just change the API url to https://identi.ca/api!). That third-party ecosystem is mostly restricted to polling, though, and could really benefit from interoperable methods for pushing updates to an open client.

I’ve been poking around on StatusNet’s i18n to see if we can get localizations working with less fidgeting; all languages should “just work” when you drop in your StatusNet install.

We’re loading translations using gettext, which unfortunately can only  load translations for languages which are set up as locales system-wide. This is massively problematic and has lead to localization just plain not working for most people — it’s even broken on identi.ca!

The good news is that we already have a compatibility layer that doesn’t have this limitation: php-gettext, which provides source-compatible drop-in gettext interface in pure PHP.

The bad news is that if the native gettext module *is* present, there doesn’t seem to be any way to override calls to _() — PHP has no facility for monkeypatching to replace existing functions. So, to force use of the compat functions for unsupported locales we’d need to change all calls to a new name.

Any preferences between something short and cryptic like __() or something clearer and StatusNet-y like common_msg()?

This is also a good time to think about handling localization for plugins; currently the plugins that ship with StatusNet’s source just have their localizations lumped into StatusNet’s main file — that obviously won’t do for plugins that are maintained and distributed separately.

In the gettext model it looks like the way to handle this is to use multiple “domains”; StatusNet’s core domain is “statusnet” (duh) and set as the default domain for gettext calls. A plugin can bind its own locale subdirectory to another domain (say “ldapplugin”) and instead of calling _(“Some text”) can call dgettext(“ldapplugin”, “Some text”).

This could perhaps be simplified by adding helper methods onto the Plugin base class…

$this->_("blah")

Thoughts?

Updated 2009-10-16: I seem to be able to work around the locale setup problem by setting a valid locale before setting the invalid one. That should hold us for a while before we try larger changes.

Slides for my talk at SVG Open available for download as PDF or Keynote source. (I can make my test corpus available as well — let me know if interested!)

– brion

post mirrored from Wikimedia Tech Blog

I’d like to share some exciting news with you all… After four awesome years working for the Wikimedia Foundation full-time, next month I’m going to be starting a new position at StatusNet, leading development on the open-source microblogging system which powers identi.ca and other sites.

I’ve been contributing to StatusNet (formerly Laconica) as a user, bug reporter, and patch submitter since 2008, and I’m really excited at the opportunity to get more involved in the project at this key time as we gear up for a 1.0 release, hosted services, and support offerings.

StatusNet was born in the same free-culture and free-software community that brought me to Wikipedia; many of you probably already know founder Evan Prodromou from his longtime work in the wiki community, launching the awesome Wikitravel and helping out with MediaWiki development on various fronts. The “big idea” driving StatusNet is rebalancing power in the modern social web — pushing data portability and open protocols to protect your autonomy from siloed proprietary services… People need the ability to control their own presence on the web instead of hoping Facebook or Twitter always treat you the way you want.

This does unfortunately mean that I’ll have less time for MediaWiki as I’ll be leaving my position as Wikimedia CTO sooner than originally anticipated, but that doesn’t mean I’m leaving the Wikimedia community or MediaWiki development!

Just as I was in the MediaWiki development community before Wikimedia hired me, you’ll all see me in the same IRC channels and on the same mailing lists… I know this is also a busy time with our fundraiser coming up and lots of cool ongoing developments, so to help ease the transition I’ve worked out a commitment to come into the WMF office one day a week through the end of December to make sure all our tech staff has a chance to pick my brain as we smooth out the code review processes and make sure things are as well documented as I like to think they are.

We’ve got a great tech team here at Wikimedia, and we’ve done so much with so little over the last few years. A lot of really good work is going on now, modernizing both our infrastructure and our user interface… I have every confidence that Wikipedia and friends will continue to thrive!

I’ll start full-time at StatusNet on October 12. My key priorities until then are getting some of our key software rollouts going, supporting the Usability Initiative’s next scheduled update and getting a useful but minimally-disruptive Flagged Revisions configuration going on English Wikipedia. I’m also hoping to make further improvements to our code review process, based on my experience with our recent big updates as well as the git-based workflow we’re using at StatusNet — I’ve got a lot of great ideas for improving the CodeReview extension…

Erik Moeller will be the primary point of contact for WMF tech management issues starting October 12, until the new CTO is hired. I’ll support the hiring process as much as I can, and we’re hoping to have a candidate in the door by the end of the year.

– brion vibber (brion @ wikimedia.org)
CTO, Wikimedia Foundation
San Francisco

Update: Evan’s announce is up on the StatusNet blog.

I’m generally appalled at the state of compound documents…

In the Apple world, Mac apps like Keynote love to use bundled directories which look like flat files at the UI level. Cute, but Thunderbird gleefully destroys them as attachments… Apple’s Mail.app transparently packages them into .zip archives for you, but Thunderbird just gives you a file with a directory listing, which naturally enough fails to open when you download it. Nice!

OpenOffice and the latest greatest MS Office stick their XML documents into a .zip archive and package image files, etc into that archive. These actually do act like flat files, so attachments and uploads work. But it makes file type detection and validation a little harder; verifying which file type your zip thingy is and whether it contains extra files slipped in…

And then you get fun packages like Scribus, which just give you an XML file referencing all your external image files by path, leaving no way to transfer your entire document without manually managing a directory structure and sending around or archiving multiple files manually.

We had a request for allowing Scribus uploads to Wikimedia sites for things like PR materials… sounds great, except for how any actually relevant document will need image files packed into the same directory which you can’t do. D’oh!