Memory leaks and DB_DataObject

December 14th, 2009

As part of my work restructuring the background daemons, I’m starting to do memory profiling on StatusNet… I’d like to get it to the point where we can enable a debug mode that just snapshots out how much memory each class and global takes up while you’re running.

I’ve already found some low-hanging fruit in DB_DataObject. This is a “clever” library that likes to stash a lot of stuff into a global $_DB_DATAOBJECT array, including all your query result data.

Sure you’ve got a free() method, but nobody calls those consistently…

I found I could quite easily drop a destructor onto our Memcached_DataObject intermediate class which calls the free() method when the object is destroyed (explicitly with an unset(), or implicitly by falling out of scope and no longer being referenced).

My queue-handler daemon was able to get through an average of 595 notices on my stress test before croaking on a 32M memory_limit, but jumped up to get through 761 notices simply by adding this one-line destructor. (Average of 5 runs.)

Not bad for a start!

I’ve committed the destructor onto 0.9.x branch.

L10n for StatusNet plugins

December 7th, 2009

Now that we’ve got UI translations going pretty well for StatusNet core via TranslateWiki, I’d like to see if we can get things working for plugins as well before the 0.9 final release.

I’d like some quick feedback from folks before merging into 0.9.x; you can see my work branch here.

In core code, to make a string translatable we wrap it in one of the gettext functions like this, most often the _() shortcut:

 $this->text(_("You have a new message."));

If you’re a plugin though, you need to add your own translations into the mix, and gettext makes you tell it which “domain” you’re pulling from each time…

 // at init time
 bindtextdomain("AwesomePlugin",
                dirname(__FILE__) . "/locale");
 ...
 $this->text(dgettext("AwesomePlugin",
                         "You have a new message."));

Repeating your plugin name for every string gets old REAL fast!

In my work branch I’ve added a new gettext wrapper function _m() which knows if it’s been called from a plugin, and picks out the right domain for you based on the plugin directory.

So rather than playing around with bindtextdomain() and dgettext() manually, you can just add one character and not worry about the rest:

 $this->text(_m("You have a new message."));

It also knows how many parameters you passed to it, so instead of whipping out ngettext() or dngettext() (or god forbid dnpgettext!) you can just keep using the nice compact _m() when your needs get fancier:

Plurals:

 $this->text(sprintf(_m("You have a new message.",
                        "You have %d new messages.",
                        $messageCount),
                     $messageCount);

Contexts for disambiguation:

 // read vs delete
 $this->text(_m("message-action", "Read"));

 // read vs unread
 $this->text(_m("message-state", "Read"));

Plurals with contexts!

 $this->text(_m("message-state",
                "Read",
                "Read",
                $messageCount));

Plugins maintained in the main StatusNet repo shouldn’t need to worry about anything else — the .po file templates and updates via TranslateWiki will be handled through the same workflow as the core. (I’m working with Siebrand at TranslateWiki to make sure we can automate this as much as possible, including adding new plugins.)

scripts/update_po_translations.php can regenerate all (or just core, or just plugin) .po templates for those who want to push them in immediately, though.

As with core, the binary .mo files won’t be included in the git repository to simplify code maintenance. I’ve added a Makefile at the base level which’ll build all the .mo files for folks to test localization in their working copies (or to build a release!)

Confusing Twitter settings

December 3rd, 2009

I’ve been getting feedback for some time that this checkbox in StatusNet’s Twitter connect settings is confusing:

Subscribe to my Twitter friends here.

People tend to think this should mean that this’ll pull all your Twitter friends’ updates into your timeline on the SN site… which is actually a separate checkbox, when enabled (sorry, it’s still disabled on identi.ca):

Import my Friends Timeline.

I suspect we could come up with clearer wording for both of these… let’s start the bidding!

  [x] Subscribe to my Twitter friends’ accounts on %%site.name%% automatically.

  [x] Show my friends’ tweets here on %%site.name%%.

Any other ideas?

Compiling PHP on Snow Leopard

November 12th, 2009

If you’ve been having trouble compiling your own PHP installations on Mac OS X 10.6, here’s the secret to making it not suck! After running the configure script, edit the generated Makefile and make these fixes:

  • Find the EXTRA_LIBS definition and add -lresolv to the end
  • Find the EXE_EXT definition and remove .dSYM

Standard make and make install should work from here…

For reference, here’s the whole configure line I currently use; MySQL is installed from the downloadable installer; other deps from MacPorts:

‘./configure’ ‘–prefix=/opt/php52′ ‘–with-mysql=/usr/local/mysql’ ‘–with-zlib’ ‘–with-bz2′ ‘–enable-mbstring’ ‘–enable-exif’ ‘–enable-fastcgi’ ‘–with-xmlrpc’ ‘–with-xsl’ ‘–with-readline=/opt/local’ –without-iconv –with-gd –with-png-dir=/opt/local –with-jpeg-dir=/opt/local –with-curl –with-gettext=/opt/local –with-mysqli=/usr/local/mysql/bin/mysql_config –with-tidy=/opt/local –enable-pcntl –with-openssl

Putting the media in Wikimedia!

November 7th, 2009

I’m here in the city of lights for Wikimedia’s big Multimedia Usability meeting. We’ve got a fair chunk of our MediaWiki devs and folks with more of the media & communications organization end in one place to hash out some of the key issues and see what we can really accomplish in the short and medium term, including long-needed reworking of the upload interface and the workflow of manually tidying up metadata for newly uploaded files — sometimes coming in batches of many thousands!

Since my free time’s pretty low these next couple months I’m trying to keep my own commitments to where I can pack the most punch…

Things to do proof of concept coding to confirm our implementation theory:

  • Metadata!
    • Proof of concept for template/field/subtemplate extraction and mapping to RDF
    • Try to organize w/ Robert about how to store and search the attached RDF fields in Lucene
    • [Note I've been pulling existing exif data to get some stats about what can be pre-extracted; only about 7% of Commons files with EXIF data have a 'Copyright' field.]

Things to ponder specs on:

Other things to peek at and give some directional advice on:

  • Check out what it’d take to integrate Geohack tools better (via Magnus)
  • Take a peek at Unicode encoding & keyboard input problems for some languages requiring funky script support such as Malayalam (via Gerard)

Update: Also want to poke XMPP RC test setup per Duesentrieb. :D

StatusNet Sphinx search work in progress

November 5th, 2009

I’m retooling StatusNet’s Sphinx search engine support to pluginize it and better support multiple sites… Putting some notes up on the wiki.

Would love to see any additional issues or suggestions from folks using, or wanting to use, Sphinx for their SN instances!

StatusNet on TranslateWiki

November 3rd, 2009

Ok, we’ve got localization for StatusNet set up on TranslateWiki — setup was pretty straightforward and they’re excited about supporting us. Big thanks to Siebrand Mazeland for getting a test setup in yesterday and the live setup today! [And of course thanks to GerardM for pimping out TranslateWiki every chance he gets -- lots of cool stuff going on there now!]

Portal page for our project:
http://translatewiki.net/wiki/Translating:StatusNet

There’s still some rough edges in the display for gettext-sourced translations, and I want to make sure we’ve got the update process going cleanly, but it seems to be working pretty well. We’ve already had some fixes to the German translation, and I’ve confirmed that I can pull and commit updates from TranslateWiki into git — yay!

To get set up, create yourself a login and hit up the requests page to get your edit permissions confirmed.

If you need to grab TranslateWiki folks live to poke something, right now the best place is the #mediawiki-i18n channel on irc.freenode.net; Siebrand and Nikerabbit are our main contacts there. There’s also a web gateway to the channel.

I’m also collecting some ideas for improvements to the interface and process on my user page.

– brion vibber (brion @ status.net)

StatusNet hiring customer support in Montreal

November 3rd, 2009

In Canada and want to get involved in an awesome open-source messaging company?

StatusNet’s gearing up for our hosted services launch later this year, and that means we’re hiring for customer and admin support positions at the head office in Montreal, Quebec.

Sorry fellow remote web app developers — I’ll let y’all know when we’ve got more openings on the tech end!

Don’t spam me, bro

October 29th, 2009

I got a pretty HTML spam email from Microsoft about the opening of their Mission Viejo, CA retail store today. This was probably not a smart marketing decision…

  1. Spam is bad; I don’t remember ever asking for this
  2. I haven’t lived in the area for 3 years. What old product reg info were they pulling from?
  3. All links in the mail run through the email marketing partner’s domain, so I’m never going to click them… haven’t they heard of phishing?

Although I have to admit, I’m tempted to swing by when I’m down that way visiting the folks for Christmas, just to see a Microsoft store… :)

WikiSym talk: Community Performance Optimization

October 27th, 2009

Got my presentation slides up for my WikiSym 2009 closing keynote. Had a lot of fun — I missed the last couple, but definitely need to get back to WikiSyms in the future!

Good questions from audience after the talk, hopefully video will be available online shortly.


I love Wikipedia!