Archive for the 'devel' Category

StatusNet updates

Thursday, August 12th, 2010

A few quick StatusNet-related updates:

  • 0.9.4 is about to be released (beta2 tag running on live servers now)
  • StatusNet 0.9.4 is the first release to include an Esperanto translation of the UI — big thanks to all our translators at TranslateWiki.net who’ve helped with EO and so many other languages!
  • We’re still chugging away at a StatusNet mobile client (open source, based on the Appcelerator Titanium cross-platform runtime) Probably betas for iPhone users next week? We’ll also want Android testers, but you don’t need to sign up. :)
  • Among other things, I’m poking at our experimental TinyMCE-based wysiwyg edit plugin. More work is needed before it’s ready for general usage, but it’s kinda neat! (So much easier than adding wysiwig to a wiki since there’s no need to round-trip markup through edits over and over. ;)

Beta testers for StatusNet Mobile wanted soon…

Monday, August 2nd, 2010

A few weeks ago, we released the first public beta of StatusNet’s dedicated desktop client for Linux, Mac, and Windows.

We’re still working on bug fixes and improvements, but we’ve also been working on a mobile version, which runs on Android and iPhone/iOS platforms (Blackberry support isn’t ready yet in the Titanium cross-platform runtime we’re using, but it should come along in a few months). If you’re really brave you can dive into the source code, but we’re in the middle of major restructurings and UI design so it’s not really usable yet. :)

Once we get the UI polished up in the next week or so, we are going to need some testers to help make sure things work on different devices, different OS versions, at different screen sizes, with different servers, etc…

If you have an iPhone or iPod Touch and want to test without being a developer yourself, you’ll need to find your device’s UDID number and send it to us; due to the way Apple’s code signing works you won’t be able to install the beta distribution unless your device’s ID was included when building the signing key.

Android users will only need to ensure that “Unknown sources” is checked in Application settings, which allows installing apps from the web as well as from the Android Market. (Some models unfortunately don’t allow changing this setting, in which case you may need to root your phone to get the beta running. Sorry!)

Particular things we’re looking for…

  • older iPhone 2G, 3G devices still running iPhoneOS 3
  • Android phones running older OS versions (1.6 or later; 1.5 does not work with our runtime)
  • Android devices with unusual screen sizes:
    • small low-res screens (less than 320×480)
    • larger tablet devices

Gitorious ain’t glorious today

Friday, July 30th, 2010

Gitorious’s ssh-authenticated access has been down all day, preventing us from pushing our commits on StatusNet and the StatusNet client.

Until it’s fixed, I’ve put up my StatusNet & StatusNet Desktop/Mobile client dev branches up on github, to be merged back onto the Gitorious mainline later:

* StatusNet – http://github.com/brion/statusnet
* client – http://github.com/brion/statusnet-client

Hopefully this inconvenience won’t last too long; I can add folks as collaborators on those repositories to make direct commits in the meantime.

Yet another iOS mulititasking explanation post

Sunday, July 11th, 2010

There’s been a lot of confusion about just how multitasking works in the iPhone’s latest iOS 4.0, and just what the limitations on background processes are. Most of the articles I’ve seen attempting to clarify it have concentrated on the addition of the new suspend state and how apps being background vs suspended vs terminated relates to the task list interface. That helps with the end-user confusion, but to me has just made it even more confusing from the developer’s perspective — I want to know what my app will be able to do in this brave new multitasking world, and what its limitations are going to be.

I’ve gone ahead and actually looked at the documentation (RTFM); here’s some notes… (Impatient readers may wish to skip to the summary at the end!)

Application state lifecycle

First, let’s go ahead and put those “background” and “suspend” states into perspective…

Not running

On screen? no
Running? no
In memory? no
Resources none

In the beginning, there was nothing… Before your application is started, it just doesn’t exist in the system yet.

An app that’s not running has no way to execute code, but popup notifications may be shown on its behalf by a registered server application or from earlier scheduling.

When your app gets launched, your code gets loaded into memory and you transition into:
Active state

On screen? yes
Running? yes
In memory? yes
Resources as you like

Your app is large and in charge! Your code and data are in memory, code is being executed, and you’ve got free control over the user interface, audio, network, etc.

There’s also an inactive state when the system takes over the UI and event loops for stuff like showing the incoming phone call dialog; your app is temporarily paused from the UI, but all your resources stay intact and you’ll get them back soon.

When it comes time to switch apps (through the home menu, task list, or programatically), your app loses control of the screen and enters the…
Background state

On screen? no
Running? yes
In memory? yes
Resources restricted*

Your code is still running, but you’ve got no access to the screen, and various resources start getting cut off. Usually this is a temporary state giving an application a chance to save data, close out unneeded resources, and generally tidy up before being suspended completely.

There are some special exceptions which can allow an app to run in background state for prolonged time, which is where the really interesting stuff comes in. We’ll get to that soon!

When we’re done with background state, the OS can put your app to bed; now it’s in…
Suspend state

On screen? no
Running? no
In memory? yes
Resources mostly freed

This is the biggest change in iOS 4: after your post-switchaway cleanup, the app remains in memory so it can be continued at a moment’s notice.

Previously, after your app did a little cleanup on the way out it would be terminated and all its memory and network resources freed. Instead, the app is now simply stopped at this point, but with the explicit warning that it may or may not ever be continued.

If the app is reactivated from suspend mode, anything you kept in memory is still there — you have a lot less work to do to reestablish your application’s running state than when relaunching the process.

But you may die before you wake, in which case you’re back to…

Not running state.

If the system needs more memory to assign to another application, or gets shut down, your suspended app will be terminated without being woken to inform it.

You need to be prepared for termination before entering suspend state… but really, you’re already writing code that assumes it could crash at any time and saves state at intervals and key points so it won’t lose user data, right? Um, right?

Background mode limitations

So just what are the limits of what you can do while running in background state? The docs mix together a lot of strict limits along with recommendations for being a good citizen; I’ve tried to split them out here:

YOU CANNOT (technical restrictions on what you can do):

  • Can’t make OpenGL calls; they will terminate your app.
  • Can’t accept new connections on a listening socket.
  • Can’t use shared system resources like the Address Book (it sounds like they might sorta work if still open, but you could get terminated if there’s a conflict.)
  • Can’t use external accessories — you must register for and handle disconnection events.

YOU SHOULD (recommendations for behaving well):

  • Should be prepared for loss of connectivity — open network connections could be torn down at any time.
  • Should save your state, since your app could be terminated due to memory pressure.
    (You should be saving state during regular operation anyway to protect against application or system crashes, power failures, etc. Programs that assume orderly shutdown are asking for trouble!)
  • Should avoid updating your windows and views; it’ll work but since your UI is hidden it’s a waste of time & battery.
  • Should normalize your UI state — cancel modal alerts, hide temporarily shown passwords, etc.
  • Should “do minimal work” while in background.

Reaching the user when not on screen

iPhone OS 2 introduced networked push notifications, where — through the magic of the internet — your app’s web services can trigger a notification dialog on the phone, even if your application is no longer running. iOS 4 extends this to local background tasks; if your app is in the background state, it can pop up a notification immediately without needing to go out to the network.

You can also schedule a future notification at any time (up to 128 scheduled per app), which will trigger even if your app has been terminated — obviously handy for alarms, calendars, and timer apps.

Notifications are limited in that they alert the user, not the app. If you were suspended or not running when the notification came, you won’t be woken unless the user pushes the button that opens your app.

When in background…

Any app can start up background task threads, which will block the background->suspended state transition.

This is primarily intended for orderly shutdown tasks, like completing that photo uploading to Twitbook or syncing mailbox state to a server after reading a bunch of messages. The system actually gives you a time limit, and will terminate your app if you don’t declare your tasks complete when the time limit’s up! Once you’re done, you’re forced to suspend… absent other triggers, your app is going to stay that way until the user switches back to it or it’s terminated.

You can also register to receive an event for “significant location updates“, which will wake or even relaunch your application when the cell network has noticed that you’ve moved a non-trivial distance. This avoids running down the battery with the GPS if you want updates but don’t really need to be watching it continuously.

Special backgrounding modes

An app can declare itself to have certain types of backgrounding characteristics, which can allow some additional behaviors in background state. Since these are pre-declared in the code-signed app bundle, you need to be aware of what affect they’ll have on your app’s runtime behavior, and will have to run the App Store approval gauntlet with an extra bulls-eye on your forehead. ;)

Background audio mode

Normally, the system audio frameworks cut you off when transitioning from active to background state. If your app is marked as a background audio app, you get two perks:

  • Audio in/out continues to work in background state.
  • Suspend is blocked while playing audio, so you can keep running in the background arbitrarily long.

If audio is not active, your app will still be able to suspend — so a music player that’s paused, or reaches the end of its playlist in the background, can free its resources.

Articles I’ve seen have had a lot of vague language seeming to indicate that apps in this mode can “only” play audio and do nothing else, which might imply that there’s some kind of wacky alternate API for bg audio — this is not true. The docs recommend avoiding unnecessary work while playing background audio to keep resource usage down, but there’s no artificial restriction beyond the general limitations on backgrounded apps.

You’ll still use the same old network interfaces, the same old audio APIs, etc; reportedly it only took an hour to port Pandora’s iPhone player to use background audio.

Background VOIP mode

The VOIP mode is really about management of long-running network clients — an actual VOIP app will probably need to also mark itself as needing background audio. Your app will still get suspended if the user switches away with no active call, but gets a few special abilities:

  • Sockets you register as VOIP control channels will stay live when your app is suspended. If data comes in, you’ll be woken up — this lets you handle an incoming call.
  • You can register a timeout to be woken at intervals, so you to send keepalive pings if needed.
  • The app is automatically launched in the background on boot, so you can connect to the server.
  • The app is automatically relaunched on non-zero exit code, so a one-off app crash won’t break the VOIP service.

Note that while this mode sounds ideal for IM/chat apps, connections to real-time update streams for social networking clients, etc, I suspect that Apple would not actually approve such apps.

Continuous location mode

Navigation apps, GPS tracers, etc may need a more direct way to monitor the GPS for location changes while backgrounded. This is similar to the background audio mode:

  • Continue to use the regular location services APIs…
  • …while you’ve got it active, suspend will be blocked and you can remain in background mode arbitrarily long.

GPS has a particularly bad reputation for running down the battery, so if you’re just looking to ping 4square or something you should probably use the “significant location updates” event registration instead.

Summary

Now that we’ve seen something about how it all works, let’s take some sample cases and ask whether they’ll actually do what we need… So what do we need?

I’m a media player (Pandora, Airfoil Speakers, etc)

  • Can I continue playing audio after switching away?
    • Yes — mark your app as requiring background audio, and it’ll stay backgrounded on switch.
  • Can I keep communicating with the network while playing audio?
    • Yes! But if you’re doing other stuff not needed for your audio and Apple notices, they may not approve your app.
  • Can I start playing audio later on after having been backgrounded, like an alarm clock?
    • No — if you’re not playing audio at switch time, you’ll still get suspended. You could schedule a local notification to alert the user and they could push a button to launch your app and then you could play the music. Ewww!
  • Can I keep a socket open to listen for other computers to connect and send me audio to play?
    • No — your listening sockets will be closed, and you’ll have been suspended anyway as above.

I’m a VOIP client (SIP clients, Skype etc)

  • Can I keep an active call going after switching away from the app?
    • Yes — mark your app as requiring background audio, and a running call will be able to keep on going.
  • Can I maintain a connection to my server to listen for incoming calls?
    • Yes — mark your app as needing VOIP mode, and set the special flag on your control channel after setting up the connection.
  • Can I be automatically launched on boot, so I can open that connection?
    • Yes — mark your app as needing VOIP mode, you’ll be automatically launched if you
  • Can I maintain a listening socket to receive direct SIP calls?
    • No — all listening sockets will be closed in the background. You need an existing connection to a server which’ll send a packet down when there’s an event.
  • Can I auto-answer calls?
    • I’m not 100% sure on this one; if the system fully foregrounds you to handle incoming events so you can show an “incoming” screen then yes, otherwise I don’t think so.

I’m an IM or social networking client (AIM, Meebo etc; StatusNet, Twitter, Facebook, etc)

  • Can I finish uploading a post in the background if the user switches apps before it’s done?
    • Yes — do it from a background task thread, and notify the system when you’re done and ready to be suspended.
  • Can I poll my server in the background to check for updates?
    • No — you’ll need to pair with a server component and use networked notifications to alert the user.
  • Can I keep a socket open to listen for real-time updates from my server?
    • No — in theory the VOIP mode would allow this, but Apple would have to approve your app’s using it for non-VOIP use.
  • Can I be woken to check status when the physical location has changed?
    • Yes — you can register for significant location updates and be woken or launched to check if you need to perform any actions.

I’m any kind of server:

  • Can I listen for clients while in the background?
    • No. Your listening sockets will be closed, and any Bonjour service stuff will be torn down.
  • Can I finish up an existing client connection after switching away?
    • In theory this ought to work, if the operation can complete in a background task thread before you’re forced to suspend.

I’m any other bit of software:

  • Can my app be woken at a specified time?
    • No. You can set a notification to display at a given time, but user interaction will be needed to wake or launch your app.

Whee!

Nexus 1 + Froyo notes & iPhone 4

Wednesday, July 7th, 2010

The Android 2.2 “Froyo” update finally came through over the last few days for Nexus 1 owners. After a few days of on-and-off usage, some notes to add to my initial review of the N1 running 2.1:

What’s new:

  • Speed: Things definitely feel snappier than they used to, but not really in a firmly quantifiable way. I’ll try another head-to-head scrolling test after a bit, but I can still expect to see the N1 way behind on that — scrolling still feels jerkier, and usually slower, than on an iPhone.
  • Tethering: For me, this was one of the main the killer features that pushed me to actually buy the N1, and I’m very happy to see it working! AT&T might finally have gotten around to enabling tethering for the iPhone, but they’ve shot themselves in the foot by making it cheaper to buy a new Android phone instead of the $20/month to not get a bandwidth limit increase on your iPhone. Over your 2-year contract, that comes to $480 wasted on AT&T… and it still wouldn’t power your Wifi iPad while the Android will! Sorry, guys. I know which features I want.
  • Screen: my background image is still pretty badly banded, but gradients in the web browser look smoother. There may be piecemeal improvements in how images get rendered and dithered for fullcolor output, but it’s still a bit inconsistent.

Otherwise, the OS isn’t mind-blowingly different, but definitely has a lot of nice little bumps. Ars Technica has a general review of Froyo on the N1 if you want to peek at a few other under-the-hood changes.

Update: There’s also a notification system that looks like a very flexible superset of what the iPhone platform has, which might be very nice for things like sending realtime updates to our upcoming mobile client without it having to poll in the background. That ain’t much useful to users yet, but we’re sure gonna use it in future!

Compared to the iPhone 4

Of course, Apple’s been moving as well. iOS 4 is out for the existing iPhone 3G and 3Gs, and the new iPhone 4 is available and busy fighting a reception issue scandal.

iOS 4 on my iPhone 3G feels like a very nice incremental improvement. Things aren’t radically different, but it’s definitely a bit nicer: folders have helped organize home screens by moving out rarely-used apps, background processing is a big help for a few apps (like Pandora!) and there are other niceties like threading in the mail reader.

I haven’t picked up an iPhone 4 for personal use yet, but I did swing through an Apple store the other day (when the crowds had died down a bit!) to check it out. There are only a couple of interesting user-visible hardware changes beyond the case change:

My favorite is the awesome, awesome high-resolution display. I am really looking forward to this pixel density being available on desktop-size screens… some day we can stop worrying about pixels and just have text and graphics that look good.

There’s some talk that HDTV has actually set display technology back for large formats; I’ve seen only a handful of commercially-available monitors that venture much beyond 1920×1080, and those are all to gain extra desktop space not to improve density/sharpness.

The screen on the Nexus 1 is visibly sharper than the iPhone 3Gs, but even with my slightly blurry vision is visible pixelated at smartphone-usage distances from my eye. The iPhone 4 really, literally, truly moves it into the realm where pixels no longer matter; as this level of display technology makes it out into the broader market, I think it’s going to make a big difference in what we’re comfortable reading on a small screen.

The front-facing camera & video calling support is the primary selling point in Apple’s current ad campaign; the nice saleslady demoed it for me, and the quality’s pretty good for what it is. But honestly I don’t see myself ever using it as more than a gimmick; I’ve had a webcam on my laptop for 5 years and have never been on a video chat that’s not about trying out the video chat feature. Perhaps Apple will prove me wrong — and like with video chats on computers, some people get a lot more mileage out of it than others. I can certainly see if I had a small child we’d probably be on with my parents a lot more often — my mom doesn’t need too many real-time updates on the cats. ;)

There’s also an improved main camera, which may be a nice extra but isn’t a killer feature for me — the current phone cameras are adequate (though not great) and aren’t main selling points for me.

We are all dual-booters

Wednesday, June 9th, 2010

Today’s personal computers basically run two distinct operating systems: the native host OS (Windows/Mac/Linux or iPhone/Android/etc) and the web.

Web apps have solved all kinds of problems that are still poorly handled by most native systems: apps automatically update every time you use them, they manage their own library dependencies, there’s a security sandbox that lets you run pretty much anything without concern that it’s going to eat your system (unless your browser is buggy!)

Let’s face it: most of us probably spend a lot of our time in the web, and even if they’re not doing everything that’s where a lot of action is. Some folks have used this as a sort of excuse for the extreme control some platforms exercise over software publishers – “don’t like the rules? Make a web app, you can do anything!” 

But web apps are still much more limited in some areas. Access to hardware is rare (cameras, audio recording, scanners, attached storage). Communication between apps is greatly complicated by that sandbox, and shared data on the host machine like contact lists and photo archives may be completely inaccessible without a host-specific shim. (Most impressive thing I’ve seen is a bank web site that did deposit via scanned check image, using a signed Java applet to hook into native scanner support. It only worked on Windows, alas.) Background processing is very limited, and most web apps give up on directly notifying you of new activity and just send you email, hoping you’ve got something else that’ll tell you there’s new mail.

There’s a lot of great activity going on in and around HTML5 these days that’s getting better graphics support, faster code execution, etc. But the things that really bring the web native are going to be about access to shared hardware and data resources.

Some good things have been coming in such as touch and orientation events in Mobile Safari, but there’s a long way to go. My pet peeve: I find it pretty surprising that HTML file upload controls don’t trigger something useful like the camera roll on the iPhone/iPad or the Android browser. I can’t believe nobody has thought of this, so I’ll assume for now that the various browser folks just ain’t gotten to it yet… Anybody feel like starting on patches for Android’s Browser and the mobile branch of Firefox? :)

Apple Doesn’t Want Developers?

Thursday, April 8th, 2010

Ok, now this is just confusing:

By now, most of you have probably seen or heard about Apple iPhone SDK 4.0 and a little hidden gem in their freshly minted 4.0 Terms of Services, notably 3.3.1:

“Applications that link to Documented APIs through an intermediary translation or compatibility layer or tool are prohibited.”

It’s clear that products like Titanium, Unity3D, Ansca, MonoTouch and others are now a bit in question for iPhone 4.0+ with this language. We’re all trying to get our heads around what this means and trying to reach out to Apple to get clarification.

I called up Apple to ask for details about how this affects Appcelerator & Unity3d as particular examples that I’d been planning to use for games and utilities, and whether I can get a refund for my program fees if I were to not accept the updated agreement.

The rep was very pleasant and polite and took my information to escalate the question; the CSRs of course don’t have immediate answers for this. I was told someone should get back to me within 24 hours.

If y’all also call, please above all be polite — customer service reps are real people too, and they didn’t write the agreement! Be polite, accept being on hold gracefully, let them know how this would affect your future development plans, and be specific about what questions you’re asking and what answers you need.

Introduction to OStatus: Atom-based federated messaging

Thursday, February 25th, 2010

I’ve written up the little introduction to StatusNet’s new inter-site federation protocol; for now you can also find this article on the StatusNet wiki. I’ll follow up in the next couple days with demos of the subscription UI and subscribing to Google Buzz feeds as native OStatus data sources. — brion

What is OStatus?

You’re about to learn what OStatus is and how it’s going to make communications between StatusNet sites work better!

Some of you may remember the bad old days when you couldn’t send e-mail between services like AOL and CompuServe. StatusNet/Laconica sites have used a custom protocol OMB to set up subscriptions between sites, but OMB requires special support to be used with non-StatusNet services and is hard to extend.

By building on top of shared technologies already in use and being rolled out, we can go beyond that… our new effort, with better compatibility between sites, is named OStatus.

The basics

If we step back a bit, we find that Atom and RSS feeds are already a lingua franca between sites. Virtually every service provides such feeds, giving a way for other tools and services to read someone’s outgoing posts.

File:Basic-feed.pngAt its simplest, a feed is a list of recent posts with some author attribution.

But the fun of social networks is in real-time interaction — if your updates don’t get sent to your friends right away, it’s just not the same.

Luckily, that exact problem has been solved recently, with PubSubHubbub (PuSH). In the PuSH system, a site can subscribe to updates for a feed from a hub server associated with that feed. Whenever a new post is created, the publishing site pings the hub, and the hub sends out just the new posts to all the subscribers.

OStatus is built on that simple base: each participating site produces Atom feeds of updates and uses PuSH subscriptions to send relevant updates to other sites:

File:Push-delivery.png

Alice posts a message on identi.ca; a fragment of her Atom feed is PuSHed out to the other StatusNet sites that have users following her posts. Each of those sites is responsible for routing the message to individual subscribers.

The real beauty of it is that at this point we already have something useful, without anything StatusNet-specific. In fact you can already subscribe to someone’s public Google Buzz feed as an OStatus remote user, and they haven’t done anything special for us!

The good stuff

PuSH is a low-level piece of the puzzle; it’s just a way for services to send updates to each other. For a fully interactive social network, you need that human touch… you want to know who’s subscribing to you, which of your posts they like, and you want to get their replies to you.

For this we’re making use of more extensions that build on top of Atom feeds: Activity Streams, Portable Contacts, and Salmon.

Activity Streams extends Atom feeds to describe social “activities” more specifically than just “ok here’s a post”. Entries are marked with a verb (post, follow, favorite, etc) and more detail on who or what’s being acted on. We’ve extended this further using elements from Portable Contacts to provide profile information:

Extended author information beyond what Atom’s native <author> element provides.

Salmon was created as a way for blog aggregators to send comments back to the original blog server, “like salmon swimming upstream”. We’re using this channel to send replies as well as other user-to-user events:

File:Salmon-delivery.pngBob replies to Alice from his home site; since she’s not subscribed to him, the reply is sent back to identi.ca as a Salmon ping.

When your friend on Example.Net subscribes to your Identi.ca account, Example.Net sends an Atom entry with an Activity Streams “follow” verb to your Salmon reply channel on Identi.ca. This lets Identi.ca know who subscribed to you, so their profile gets added to your subscribers list and you get a notification that they’re following you.

If your Example.Net buddy gets social network overload and decides to unsubscribe from you, another notification comes through about the unsubscribe, and we can drop them from the list.

Other events can be sent over this channel as well, such as when your friends on other services mark your posts as a favorite.

Groups

One of the most annoying limitations with OMB 0.1 was that there’s no support for joining a group on another service. You might have noticed that groups have Atom feeds of posts just like users do… if you were wondering if that means groups could be run over OStatus too, you’re correct!

As a user on Identi.ca you can join a group on Example.Net without setting up an Example.Net account. Group posts from Example.Net will appear right in your inbox on Identi.ca just like any local group, and you can even post back to it.

Why OStatus will rock your world

Monday, February 22nd, 2010

Hey everybody!

Things may have looked pretty quiet the last couple weeks, but here at SNI we’ve actually been rushing to get some really awesome things ready for you all…

In addition to finalizing StatusNet 0.9 with scalability and bug fixes to make it release-ready, we’ve been implementing our new federation protocol OStatus, successor to our older OMB 0.1.

I know the scattered notes we’ve got up on the wiki right now are a little confusing, so over the next few days I’m going to start consolidating those into a clearer picture what OStatus is and how it’ll let StatusNet sites connect better to each other and to other systems.

In the meantime, active work on OStatus is currently proceeding in the ‘testing’ branch in our gitorious mainline repository…

Stay tuned!

– brion vibber (brion @ status.net)
Software Architect
StatusNet, Inc
San Francisco

XMPP output fix for StatusNet

Thursday, January 21st, 2010

Yay, it’s big commit time!

I’ve landed my XMPP output queuing work in 0.9.x, running on my public test site.  Do let me know if XMPP subscription or i/o is flaky on it!

Should be ready to merge to testing & master and deploy when we’re content to do so… there is a database change necessary for the DB-based queueing system, so I want to confirm that’s not a problem before pushing out.

Big thanks to Craig Andrews for his work on generalizing the DB queues which I’ve started integrating; we should be able to land the rest including the IM pluginization for 1.0.

Commit summary:

XMPP queued output & initial retooling of DB queue manager to support non-Notice objects.

Queue handlers for XMPP individual & firehose output now send their XML stanzas to another output queue instead of connecting directly to the chat server. This lets us have as many general processing threads as we need, while all actual XMPP input and output go through a single daemon with a single connection open.

This avoids problems with multiple connected resources:

  • multiple windows shown in some chat clients (psi, gajim, kopete)
  • extra load on server
  • incoming message delivery forwarding issues

Database changes:

  • queue_item drops ‘notice_id’ in favor of a ‘frame’ blob. This is based on Craig Andrews’ work branch to generalize queues to take any object, but conservatively leaving out the serialization for now. Table updater (preserves any existing queued items) in db/rc3to09.sql

Code changes to watch out for:

  • Queue handlers should now define a handle() method instead of handle_notice()
  • QueueDaemon and XmppDaemon now share common i/o (IoMaster) and respawning thread management (RespawningDaemon) infrastructure.
  • The polling XmppConfirmManager has been dropped, as the message is queued directly when saving IM settings.
  • Enable $config['queue']['debug_memory'] to output current memory usage at each run through the event loop to watch for memory leaks

To do:

  • Adapt XMPP i/o to component connection mode for multi-site support.
  • XMPP input can also be broken out to a queue, which would allow the actual notice save etc to be handled by general queue threads.
  • Make sure there are no problems with simply pushing serialized Notice objects to queues.
  • Find a way to improve interactive performance of the database-backed queue handler; polling is pretty painful to XMPP.
  • Possibly redo the way QueueHandlers are injected into a QueueManager. The grouping used to split out the XMPP output queue is a bit awkward.

I love Wikipedia!