Planet IM

Artur Hefczyc: Migration on the new server completed

Wed, 06 Aug 2008 10:26:06 +0000

Tigase.org website, e-mail, SVN, XMPP and all other services have been successfully migrated to a new server at the external service provider.

Let me know if you find any problems with the website or any other services.

I hope this solves for good problems with availability and accessibility to the Tigase.org services.

Christian Hammond: Django Development with Djblets: Dynamic Site Configuration

Tue, 05 Aug 2008 11:50:36 +0000

Table of contents for Django Development with Djblets

Django Development with Djblets
Django Development with Djblets: Custom Tag Helpers
Django Development with Djblets: Unrooting your URLs
Django Development with Djblets: Data Grids
Django Development with Djblets: Dynamic Site Configuration

Django’s a powerful toolkit and offers many things to ease the creation of websites and web applications. Features such as the automatically built Administration UI are just awesome and save developers from having to reinvent the wheel on every project. It does fall short in a few areas, however, and one of them is site configuration.

Now, Django’s settings support isn’t bad. You get a nice little settings.py file to put settings in for your site/webapp, and since it’s just a Python module, your settings can really hold whatever data you want and can be well documented with comments. That’s all great, but when you want to actually change a setting, you must SSH in, edit the file, save it, and restart the server.

This is fine for small websites with a few visitors, or sites with almost no custom settings, but for large sites it can be a problem. Bringing down the site however briefly could interrupt users, generate errors, causing worries during a checkout process or when editing a post on a site. Doing anything more complex and preventing downtime really means rolling your own thing.

So we rolled our own thing. Review Board has operated with the standard Django settings module since day 1, but it’s become obvious over time that it wasn’t good enough for us. So we wrote the dynamic Site Configuration app for Djblets.

djblets.siteconfig

siteconfig is a relatively small Django app that stores setting data in the database, along with project versioning data in case you need it to handle migrations of some sort down the road. There’s a lot of useful things in this app, so I’ll summarize what it provides before I jump into details:

Saving/Retrieving settings.

Setting values can be simple strings, integers, booleans, or something more complex like an array of dictionaries of key/value pairs.

Default values for settings.

These default values could be provided by your application directory or could be based off the contents in your settings.py file.

Syncing to/from settings.py.

Just because you’re using djblets.siteconfig doesn’t mean you can’t still support settings.py. Settings can be pulled from your old settings and saved back there when modified.

Compatibility with standard Django settings.

Django offers a lot of useful settings that you may want to customize. We do the hard work of mapping these and keeping them in sync so you can customize them dynamically.

Auto-generated settings pages.

Much like the existing admin UI, we offer an easy way to provide settings pages for your application. Go ahead and stick these in the admin UI if you like.

Getting Started

To start out, you’ll want to add some code in your app that creates a SiteConfiguration model, perhaps as a post_syncdb management hook. This is project-specific, since you’ll be storing version information in here and linking it with your existing Site. Here’s one way to do it. We’ll use this file as a template in later examples:

# myapp/siteconfig.py
# NOTE: Import this file in your urls.py or some place before
#       any code relying on settings is imported.
from django.contrib.sites.models import Site
 
from djblets.siteconfig.models import SiteConfiguration
 
 
def load_site_config():
    """Sets up the SiteConfiguration, provides defaults and syncs settings."""
    try:
        siteconfig = SiteConfiguration.objects.get_current()
    except SiteConfiguration.DoesNotExist:
        # Either warn or just create the thing. Depends on your app
        siteconfig = SiteConfiguration(site=Site.objects.get_current(),
                                       version="1.0")
 
    # Code will go here for settings work in later examples.
 
 
load_site_config()

Saving/Retrieving Settings

Settings exist in a SiteConfiguration model inside a JSONField. Anything that Django’s native JSON serializer/deserializer code can handle, you can store. Usually you’ll want to store primitive values (strings, booleans, integers, etc.), but you’re free to do something more complex.

from djblets.siteconfig.models import SiteConfiguration
 
 
siteconfig = SiteConfiguration.objects.get_current()
siteconfig.set("mystring", "foobar")
siteconfig.set("mybool", True)
siteconfig.set("myarray", [1, 2, 3, 4])
 
mybool = siteconfig.get("mybool")

It’s pretty straightforward. The set/get functions are just simple accessors for the SiteConfiguration.settings JSON field, but you’ll want to use them because they’ll handle the registered defaults for settings, which I will get to in a moment.

Since this is just a Djblets JSONField, which is a glorified dictionary, you can do a lot of things, such as iterate through the keys and values:

for key, value in siteconfig.settings:
    print "%s: %s" % (key, value)

And so on. Let’s go a step further.

Default Values

Say you’re starting fresh and just created your SiteConfiguration instance, or you’ve introduced a new setting into an existing site. The setting won’t be in the saved settings, so what value is returned when you do a get call? Well, that depends on your setup a bit.

In the first case, with a fresh new setting:

>>> print siteconfig.get("mynewsetting")
None

Any setting without a default value and not in the saved settings will return None. Good ol’ reliable None.. If you want to make sure you return something sane at this specific callpoint, you can specify a default value in the call to get, like so:

>>> print siteconfig.get("mynewsetting", default=123)
123

This can be really useful at times, but it’s a pain to have to do this in every call and keep the values in sync. So we provide a third option: Registered default values.

# myapp/siteconfig.py
 
 
defaults = {
    'mystring':     'Foobar',
    'mybool':       False,
    'mynewsetting': 123,
}
 
 
def load_site_config():
    ...
 
    if not siteconfig.get_defaults():
        siteconfig.add_defaults(defaults)

That’s all it takes. Now calling get with just the setting name will return the default value, if the setting hasn’t been saved in the database.

Down the road we’re going to have support for automatic querying of apps to grab their settings, giving third party apps an easier way to integrate into codebases using siteconfig.

Syncing Settings with settings.py

Django’s settings.py is and will continue to be an important place for some settings to go. If you’re developing a reusable application for other projects, you may not want to fully ditch settings.py. Or maybe you’re using a third party application that uses settings.py and want to make the settings dynamic.

siteconfig was designed from the beginning to handle syncing settings in both places. Now, when I say syncing, I don’t mean we write out to settings.py, since that would require enforcing certain permissions on the file. What we do instead is to load in the values from settings.py if not set in the siteconfig settings, and to write out to the settings object, an in-memory version of settings.py.

Let’s revisit our example above, but in this case we want to be able to dynamically control Django’s EMAIL_HOST setting.

# myapp/siteconfig.py
from django.conf import settings
 
from djblets.siteconfig.django_settings import apply_django_settings, \
                                               generate_defaults
 
 
settings_map = {
    # siteconfig key    settings.py key
    'mail_host':        'EMAIL_HOST',
}
 
defaults = {
    ...
}
defaults.update(generate_defaults(settings_map))
 
 
def load_site_config():
    ...
 
    apply_django_settings(siteconfig, settings_map)

What we did here was to generate a mapping table of our custom settings to Django settings.py settings. We can then generate a set of defaults using the generate_defaults function. This goes through the Django settings keys, pulls out the values if set in settings.py, and returns a dictionary similar to the one we wrote above. This guarantees that our default values will be what you have configured in settings.py, handling the first part of our synchronization.

The second part is handled through use of the apply_django_settings function. This will take a siteconfig and a settings map and write out all values that have actually changed. So if “mail_host” above is never modified, apply_django_settings won’t write it out to the database, allowing the default value to change without having to do anything too fancy.

Compatibility with Django’s Settings

The above example dealt with the settings.EMAIL_HOST setting, but in reality we won’t have to cover this at all. There’s a few other goodies in the django_settings module that handle all the Django settings for you.

First off is get_django_settings_map. You’ll rarely need to call this directly, but it returns a settings map for all the Django settings sites are likely to want to change.

Then there’s get_django_defaults, which you’ll want to either merge into your existing defaults table or add directly. We don’t do it for you because you may very well not care about these settings.

The third is one you just saw, apply_django_settings. In the above example, we passed in our settings map, but if you don’t specify one it will use get_django_settings_map.

Let’s take a look at how this works.

# myapp/siteconfig.py
 
from djblets.siteconfig.django_settings import apply_django_settings, \
                                               get_django_defaults
 
 
def load_site_config():
    ...
 
    if not siteconfig.get_defaults():
        siteconfig.add_defaults(defaults)
        siteconfig.add_defaults(get_django_defaults())
 
    apply_django_settings(siteconfig, settings_map)
    apply_django_settings(siteconfig)

This is all it takes.

If you take a look at django_settings.py, you’ll notice that we don’t reuse the existing Django settings names and instead provide our own. This is an attempt to cleanly namespace the settings used. You’ll also notice that there’s several settings maps and defaults functions. This is so you can be more picky if you really want to.

Auto-generated Settings Pages

Oh, this is where it gets fun.

Dynamic settings are all well and good, but if you can’t set them through your browser, why bother?

We wanted to make sure that not only could these be modified through a browser, but that it was dead simple to put up pages for changing settings. All you have to do is add a URL mapping and one form per page.

The form doesn’t even need much more than fields.

Let’s start out with some code. It should be pretty self-explanatory.

# urls.py
from myapp.forms import GeneralSettingsForm
 
urlpatterns = patterns('',
    (r'^admin/settings/general/$', 'djblets.siteconfig.views.site_settings',
     {'form_class': GeneralSettingsForm}),
    (r'^admin/(.*)', admin.site.root),
)

# myapp/forms.py
from django import forms
from djblets.siteconfig.forms import SiteSettingsForm
 
 
class GeneralSettingsForm(SiteSettingsForm):
    mystring = forms.CharField(
        label="My String",
        required=True)
 
    mybool = forms.BooleanField(
        label="My Boolean",
        required=False)
 
    mail_host = forms.CharField(
        label="Mail Server",
        required=False)
 
 
    class Meta:
        title = "General Settings"

That’s it. Now just go to /admin/settings/general/ and you’ll see your brand new settings form. It will auto-load existing or default settings and save modified settings. Like any standard form, it will handle help text, field validation and error generation.

Proxy Fields

For a lot of sites, this will be sufficient. For more complicated settings, we have a few more things we can do.

Let’s take the array above. Django’s forms doesn’t have any concept of array values, but we can fake it with custom load and save methods and a proxy field.

# myapp/forms.py
import re
 
class GeneralSettingsForm(SiteSettingsForm):
    ...
 
    my_array = forms.CharField(
        label="My Array",
        required=False,
        help_text="A comma-separated list of anything!")
 
    def load(self):
        self.fields['my_array'].initial = \
            ', '.join(self.siteconfig.get('myarray'))
 
        super(GeneralSettingsForm, self).load()
 
    def save(self):
        self.siteconfig.set('myarray',
            re.split(r',\s*', self.cleaned_data['my_array']))
 
        super(GeneralSettingsForm, self).save()
 
    class Meta:
        save_blacklist = ('my_array',)

What we’re essentially doing here is to come up with a new field not in the settings database (notice “my_array” versus “myarray”) and to use this as a proxy for the real setting. We then handle the serialization/deserialization in the save and load methods.

When doing this, it’s very important to add the proxy field to the save_blacklist tuple in the Meta class so that the form doesn’t save your proxy field in the database!

Fieldsets

Much like Django’s own administration UI, the SiteSettingsForm allows for placing fields into custom fieldsets, allowing settings to be grouped together under a title and optional description. This is done by filling out the fieldsets variable in the Meta class, like so:

class GeneralSettingsForm(SiteSettingsForm):
    class Meta:
        fieldsets = (
            {
                'title':   "General",
                'classes': ('wide',),
                'fields':  ('mystring', 'mybool', 'my_array',),
            },
            {
                'title':       "E-Mail",
                'description': "Some description of e-mail settings.",
                'classes':     ('wide',),
                'fields':      ('mail_host',),
            },
        )

This will result in two groups of settings, each with a title, and one with a description. The description can span multiple paragraphs by inserting newlines (”\n“). All fields in the dictionary except for ‘fields’ are optional.

Re-applying Django settings

When you modify a field that corresponds to a setting in settings.py, you want to make sure you re-apply the setting or you could break something. This is pretty simple. Again override save as follows:

from myapp.siteconfig import load_site_config
 
 
class GeneralSettingsForm(SiteSettingsForm):
    def save(self):
        super(GeneralSettingsForm, self).save()
        load_site_config()

This will re-synchronize the settings, making sure everything is correct. Note that down the road, this will likely be automatic.

There’s one last thing to show you…

Disabled Fields

Sometimes it’s handy to disable fields. Maybe a particular setting requires a third party module that isn’t installed. The SiteSettingsForm has two special dictionaries, disabled_fields and disabled_reasons, that handle this. Just override your load function again.

class GeneralSettingsForm(SiteSettingsForm):
    def load(self):
        if not myarray_supported():
            self.disabled_fields['my_array'] = True
            self.disabled_reasons['my_array'] = \
                "This cannot be modified because we don't support arrays today."
 
        super(GeneralSettingsForm, self).load()

The resulting fields in the HTML will be disabled, and the error message will be displayed below the field.

Depending on whether or not you have a custom administration UI, you may want to tweak the CSS file or completely override it. Note that this all assumes that the djblets/media/ directory exists in your MEDIA_URL as the “djblets” directory. If you have something custom, you can always override siteconfig/settings.html and pass the new template to the view along with your settings form.

More Coming Soon!

There’s some new stuff in the works for the siteconfig app. Auto-detection of defaults and settings maps is the big one. I also have a few more useful tricks for doing advanced things with settings pages that I’ll demonstrate.

If you’re a user of Djblets, let us know, and feel free to blog about it! I’ll link periodically to other blogs talking about Djblets and projects using Djblets.

And as always, suggestions, constructive criticism and patches are always welcome :)

Previous in series

Ross Burton: Sound Juicer "We're Singing In Tune But Now It's Over" 2.23.1

Mon, 04 Aug 2008 19:33:17 +0000

Sound Juicer "We're Singing In Tune But Now It's Over" 2.23.1 has been released. Tarballs are available on burtonini.com, or from the GNOME FTP servers. Nothing that amazing here, sorry:

Fix play+pause+play (#523182, thanks Matthew Martin)
Add %ay, album year (#522909, Juan F. Giménez Silva)

Jabber Filaments Blog: XMPP and (not versus) REST

Fri, 01 Aug 2008 19:30:36 +0000

If you’ve been on vacation or haven’t yet added the feeds at Redmonk to your reader, you may have missed an interesting discussion about the OSCON presentation Beyond REST? Building Data Services with XMPP. As is often the case with discussions about software architecture, there is controversy swirling around the presentation, bringing some relief to the midsummer torpor.

Jabber, Inc.’s CTO, Joe Hildebrand (who rarely minces words) writes:

Many people are coming to the conclusion that HTTP polling doesn’t work. It doesn’t scale, and it can’t deal with high-flow feeds without missing data that scrolls off the end of your feed during the interval between polls.

These are bold words, but are supported by more than a little experience around the Jabber, Inc. software shop.

This discussion is not about dumping on HTTP based development, it is a discussion about using the right tools for the job. As Stephen O’Grady writes:

But also because we’re generally different-tools-for-different-jobs kind of folks. And XMPP is most certainly a different tool, as Kellan and Rabble’s presentation reminds us. Consider the proposition, for instance, of AMQP internally to AtomPub/XMPP externally.

No, my interest in XMPP is not a binary, mutually exlusive REST and/or, but rather a recognition that the technology that is more than instant messaging might be poised for a role in more than just instant messaging.

Part of the discussion is about how to make it easier to work with XMPP, in which Joe Hildebrand points out the value of XMPP client libraries for simplifying and accelerating application development. Jabber, Inc. has new JabberWerx Libraries that are designed with these goals in mind.

If you have interest, experience or opinions about this topic, have a look and consider joining the thread.

Christopher Zorn: Palmetto Open Source Software Conference

Fri, 01 Aug 2008 16:23:13 +0000

As some of you know I live in Charleston, SC. Likewise, you may know that South Carolina is not necessarily known as a technical state. The amount of software/engineering jobs are not the same as other states. Similar with technical students and people. That said, I went to the Palmetto Open Source Software Conference on Weds July 30th 2008. It was a one day conference at the University of South Carolina and I was pleasantly surprised.

The conference was well done, and apparently the concept and idea came about 3 months ago. So, with 3 months planning and a first year conference, it was very well done! The turn out was better than I imagined too. Being a developer/contributor of open source projects, I was in the minority, but there were plenty of people excited about open source and using it. All of the speakers were great and presented well. You can get a list at the website. http://posscon.org/ This post is not about the specifics of the conference but the themes that seemed to be throughout each of the presentations and informal discussions.

One theme in the conference seemed to be pragmatic approaches to solving problems, be it business problems or just software problems. People wanted to use and used open source because it solved their problem and solved it well.

Another theme was "openness", being open in the way you do business, working with businesses and the open source community, and "open", in giving your employees the ability to improve your business, and the way they do work, by encouraging them to be curious and contribute or use open source.

And last, how does South Carolina find these kinds of employees. Ones that can solve problems, have a curiosity and can do most anything you ask of them (jack of all trades). The last theme or topic that was discussed brought questions on how to help students and others in South Carolina to be curious and excited about technology, solving technical problems, and using open source, to enrich and encourage that.

These are all old topics and nothing new, but pragmatic and open technical solutions are definitely the way to go, and seeing companies and programmers start to think this way is very very encouraging for South Carolina! I just wish more of them knew what XMPP was. :)

Thanks to everyone involved and for starting this important topic of conversation in South Carolina!

I definitely recommend everyone in South Carolina and the surrounding areas to go back next year!

Artur Hefczyc: Public accounts in tigase.org domain

Fri, 01 Aug 2008 08:26:48 +0000

Registering new accounts on the website has been disabled. Also all existing public accounts will be removed when the service is moved onto the new server.

If you have an account on the Tigase.org website which you use as your XMPP/Jabber account please move to any public service available as in about week all those accounts will be removed.

Tigase.org will only keep accounts for people involved in Tigase projects. Creating a new account will be possible by the administrator only.

If you use an account on the Tigase website for interoperability testing or any other tests please contact me to preserve your account.

Artur Hefczyc: Back from holiday

Fri, 01 Aug 2008 07:15:19 +0000

I am back from holiday and all Tigase services are back too.

It's been a long holiday, maybe too long but very enjoyable and interesting. Part of the trip was participation in the XMPP Summit in Portland.

Unfortunately during the trip our main server which has been working fine for last 6 months has stopped accepting network connections for unknown reason. There is no apparent trace in log files so I am uncertain whether this is a hardware problem or misconfiguration. After kicking the power button restarting the machine everything came back to normal.

To avoid such a problem in the future I am now in process of moving all Tigase services to the external service provider - Flosoft.biz on to dedicated Eco08 XXL machine. This should solve for good problem with the hardware as well as with accessing the machine while I am away.

John Bailey: The Holy War of Tool Choice

Fri, 01 Aug 2008 04:14:40 +0000

Tools are vital to any open-source project. Because of that, tool choice is critical. You really don't want to be switching tools frequently, because doing so is more work than you'll ever be able to put into the project. By that same token, you don't want to pick tools no one in the project wants to use. This leads to the problem--tool choice is often a "Holy War", so to speak, because everyone has their pet tool and thinks everything else to be inferior.

Let's use Pidgin as a practical example. Prior to our most recent legal problems (which prompted our rename), we had been evaluating new version control software for some time. At that time, we were using CVS. Almost everyone who has developed on a project as large as Pidgin will agree that CVS sucks. You can't rename files unless you rename them in the repository itself, branching and merging are a pain, etc. I wasn't involved with the project at this point in time, but I do know that every distributed VCS in existence at the time was evaluated. After the evaluation, Monotone was the preferred choice. Keep in mind that this was over two years ago, more likely closer to three.

We continued to use CVS, and later made an ill-advised switch to Subversion. Sticking with these tools was mainly due to our complete reliance on SourceForge, who offered only CVS and SVN. Finally, when it came time to rename our project to Pidgin, we had a donated virtual server on which we could run (essentially) whatever we wanted. Ethan Blanton used tailor to convert our SVN repository into a monotone database.

We also set up Trac for bug, patch, and feature request tracking. The built-in wiki was a bonus. We ended up installing a number of plugins for Trac to customize it and provide some additional features, such as restricting some ticket fields from those who we feel should not be able to modify them.

All was well initially. We had a few complaints because we chose to use monotone instead of sticking with SVN, but these were users who had no intention of contributing patches, so we were (mostly) happy to see them angrily go back to using our "blessed" releases. We did, however, have a lot of confusion initially about what monotone was and how it related to Pidgin. A few hundred explanations later and those questions faded too.

Trac proved to have some scalability and performance issues for us on numerous occasions. Eventually we were able to narrow the biggest issues down to their causes and implement fixes. Some of this involved a LOT of poking and prodding, as well as some assistance from Trac developers, who we thank profusely for their time in resolving the issues. I am quite surprised, overall, that we haven't had any real complaints about Trac. For a while it was frequently completely unusable, and of course we saw a number of complaints related to that, but since the issues have been resolved we've seen no real complaints.

As for monotone, fast-forward a year from the announcement of the rename. We consistently get complaints about having to download a "huge" database just to get the latest development source for Pidgin. We do also get a few complaints that it's hard to use, but almost all of these are solved simply by pointing people at the UsingPidginMonotone wiki page. As for the huge database, we do acknowledge that it is inconvenient for some people, but "shallow-pull" support is coming to monotone. A shallow pull would be similar to an svn checkout.

I don't mind the database, myself. I have 11 working copies (checkouts) from my single pidgin database (8 distinct branches, plus duplicates of the last three branches I worked on or tested with). Each clean checkout (that is, a checkout prior to running autogen.sh and building) is approximately 61 MB. If this were SVN, each working copy would be approximately 122 MB due to svn keeping a pristine copy of every file to facilitate 'svn diff' and 'svn revert' without needing to contact the server the working copy was pulled from. Now, let's add that up. For SVN, I would have 11 times 122 MB, or 1342 MB, just in working copies. For monotone, I have 11 times 61 MB for the working copies (671 MB), plus 229 MB for the database, for a grand total of 900 MB. For me, this is an excellent bargain, as I save 442 MB of disk space thanks to the monotone model. For another compelling comparison that's sure to ruffle a few feathers, let's compare to git. If I clone the git mirror of our monotone repository, I find a checkout size of 148 MB after git-repack--running git-gc also increased the size by 2 MB, but I'll stick with the initial checkout size for fairness. If I multiply this by my 11 checkouts, I will have 1628 MB. This is even more compelling for me, as I now save 728 MB of disk space with monotone.

Richard Laager also took the time to point out an interesting, but as yet unexploited feature of monotone--the use of certs allow us to do a number of useful things while being backed by the cryptographic signing of certs. For one, we could implement something similar to the kernel's Signed-Off-By markers. We could also have an automated test suite run compiles of each revision and add a cert indicating whether or not the test succeeded. We could also include a test suite using check (we do already have some tests) and use more certs to indicate success or failure. The extensibility of monotone using lua and the certificates is truly mind-boggling.

I think I've picked on version control enough here, but before I move on, I'd like to say that I don't specifically hate any one version control system. Ok, maybe I hate CVS and Subversion, but I'm certainly not alone in that. I don't, however, hate git, bzr, hg, etc. In fact, I think hg looks quite promising--the Adium folks have decided to switch to it, and are currently in the process of trying to convert from SVN. However, given that for us monotone was the best choice when we made our decision, and given the extensibility of monotone, I think it would be foolish of us to choose another tool at this point. Perhaps in a few more years when all the DVCSes have had time to mature further it will make sense to revisit this decision.

Another tool choice involves communication between developers and users. Specifically, forums. We have elected not to have forums for Pidgin, although as a legacy of our continued involvement with Sourceforge for download hosting we are stuck with the forums they provided us. We are quite frequently belittled for our choice to forego widely visible forums. At least four of us that I am aware of have a strong distaste for forums. I find them to be worse than the ticket system for attracting duplicated questions which most users won't search beforehand. All the useless extra features, such as emoticons, formatting, etc., are a waste, as well.

It's also been my experience with forums that one or two people will start answering others' questions with misinformation that will sound legitimate to those receiving the bad answers. I've seen this numerous times, but it comes to mind so quickly as I've seen it quite recently on Ubuntu's Launchpad "Questions" forum or tracker or whatever you call it for Pidgin.

Instead of forums, we have chosen to use mailing lists. We receive complaints about this too. These complaints run the gamut from "mailing lists are hard to use" to "but you can't search the mailing list!" In reality, the mailing lists are searchable--Google indexes our mailing lists, as do a number of other search engines. Perhaps we could provide a nice search box or something that does the necessary google magic (make the search query "terms site:pidgin.im") to make it a bit friendlier, but searchability exists. Not that it matters much in my experience, but hey, I'm just a developer; what do I know?

I'd also argue that mailing lists are easier to use than forums, as all you have to do is, y'know, send an e-mail. With a forum, you have to register, log in, find the right forum if there is more than one, then post your question, and check back for answers. With the mailing list, those genuinely interested in providing assistance will helpfully use the "Reply All" feature of their mail client to provide the answer both to the user requesting help and the mailing list's archive, thus providing an answer that is both helpful and searchable.

Even so, all the choices made during the evolution of a project will result in people trying to start a crusade, holy war, revolt, ..., over the chosen tool. Unfortunately, it's a fact of life. The best we can do is take it in stride, justify our choices, and move on.

Note: It's come to my attention that I had missed the ability to share a git database across multiple working copies. In that scenario, the total size of the database and 11 working copies is slightly under 750 MB, and thus a space savings in the neighborhood of 150 MB over monotone. It had been my understanding that I needed a copy of the database per working copy. I stand corrected. I don't use git on a daily basis, as the projects I work with currently use CVS, SVN, or monotone, so I am bound to miss finer details of git here and there. There are other reasons I prefer to stick with monotone, but I won't get into them here, as they're not important to the point of this post.

Hal Rottenberg: Growing Popularity for the Psi Jabber Client

Thu, 31 Jul 2008 13:00:22 +0000

I had an email in my mailbox this morning which I’d not seen before:

The domain psi-im.org (psi) is about to exceed their bandwidth limit

Now bear in mind this limit is somewhat artificial as I host the website myself and the quota was just a random number I pulled out of my hat. But it was a pretty big number. A year ago, we would usually run through about 30-35 GB of bandwidth in a month. (Bear in mind, this does not include actual software downloads—those are hosted on Sourceforge. I’ll get to that.)

We used over one-hundred-and-nineteen gigabytes of bandwidth in the month of July, 2008. And that’s just for the website itself! We do not host a single tarball, zip archive or Mac disk image on the site.

For the benefit of those on the Psi groupchat this morning (available at psi@conference.psi-im.org), I broke the stats down a bit, and I was asked to share them here as well.

Traffic by sub-domain:

blog.psi-im.org 5.91MB / planet.psi-im.org 1.68GB (both point to the same place)
flyspray.psi-im.org 8.75GB (our bug tracker)
forum.psi-im.org 34.21GB (not too surprising, given our strong community)
psi-im.org 74.97GB

Daily average number of unique visitors:

July 2008: 4102

Top Countries (by number of hits):

(Top-level domains .net and .com excluded.)

Russia
Poland
Czech Republic
Germany
Brazil
Ukraine
Slovakia
Italy
Australia
France
Bulgaria
Netherlands
United Kingdom

And last but not least…

Download Statistics

Unfortunately, I can’t get very specific on these as Sourceforge’s stats engine is down for upgrades right now (has been for a few weeks…grr). What I can say is we generally average around 1,500 downloads of Psi—every single day. That was as of a month ago or so but is very typical.

There is one number that I can get specific about.

Total Downloads

Now this number is actually low because it doesn’t include the release candidates and it may not include downloads in the last few weeks because of the stats overhaul. (SF claims they are keeping track.). But anyway, here goes.

Total downloads of Psi official releases (going all the way back to 0.86, released in 2002), only including the Windows zip archive and executable installer, the source tarball, and the Mac disk image. Not including any of our unofficial contributed builds or the Linux distribution packages……drum roll please…

1,997,919

Wow. All I can say is thanks to the loyal users of Psi!

Source Code Statistics

And by request, source code stats.

Ralph Meijer: XMPP and Social Networks, part 2: Nodes

Wed, 30 Jul 2008 15:25:02 +0000

In part 1 I wrote about what you can subscribe to and how a social network service will send out notifications. I often used node as the thing you subscribe to, a term comes directly from the XMPP Publish Subscribe specification. In other publish-subscribe implementations this is often referred to as topic. Nodes are kept by a publish-subscribe service, and, among other things, this service is responsible for keeping the list of subscribers and sending out notifications.

Publish-subscribe services currently come in two forms: dedicated publish-subscribe services with their own domain (e.g. pubsub.ik.nu) and publish-subscribe services tied to a user account (often mentioned in combination with the Personal Eventing Protocol, also known as PEP). In the latter case, nodes are kept at the bare JID of a user's account (e.g. ralphm@ik.nu. Personal pubsub-nodes have nice properties, like the ability to directly associate a particular node with a person, and the possibility of doing access control on the user's contact list (roster).

Node organization

In the context of federating social networks, a service needs to decide where to put the nodes it wants to allow other entities to subscribe to and send out notifications from. In some cases it makes sense to keep nodes at user accounts, though in some other cases it is better to provide the nodes at the domain of the service itself. This depends on the nature of the social objects and the subscribable unit you provide. Let's explore some use cases.

Jaiku

In Jaiku, social objects (microblog posts and aggregated items like photos, bookmarks, etc), are organized in streams. Streams are tied to either a user, or a channel, and don't change ownership. The social objects themselves are static, once created, they cannot be edited. They can have comments associated with them, but those also cannot be edited. The only thing that can happen to streams, stream items, and comments is deletion.

Here, it makes sense to have a node for each stream, and possibly a stream for the comments to each stream item. Those can be tied to the owner's JID (e.g. ralphm@jaiku.com or #jabber@jaiku.com). Another possible node could be: all comments by a person. Another node an entity might want to subscribe to is: all public microblog posts. Such a node would be associated with the domain of the service rather than any particular user's JID.

anyMeta

The company I work for, Mediamatic Lab has a (proprietary) CMS called anyMeta. Instead of 'content', the C in CMS here stands for Community, to highlight the social network properties it provides. anyMeta is a highly semantic system that deals in things (a person, an article, an event, a blog), and edges (the relations between things, each with a predicate like friend-of, author-of, etc). I mainly work on federating instances of anyMeta.

Things in anyMeta are usually editable, so it makes sense to want to keep informed about changes. For example, an article can have a large number of edits, and a person might move, change employers or have other changes to his profile. Thus, we chose to at least provide each thing as a subscribable unit. Upon creating a thing, a new node is created, and a representation of the thing is published to the node. Editing a thing, results in subsequent publishes. Subscribers will receive notifications as the node gets published to.

We organized the nodes in a flat namespace, tied to a domain, rather than a user. One reason is that the owner of any particular thing might change. Tying a node to the first owner, and then needing to move it when the owner changes, is cumbersome.

Node naming

Each node has an identifier that is unique within the publish-subscribe service holding them. So you could have two nodes named updates tied to two different users. Node identifiers are opaque; one should not derive meaning from how the node identifier looks. Embedded slashes might suggest some hierarchy, for example, but an application should not assume that such a hierarchy actually exists.

That said, it makes perfect sense to use logical, human readable identifiers for nodes. They might even be very similar to the URI layout of the service's web site. Let's check what one could do for the examples given above.

Jaiku

It makes sense to have the node identifier for the regular posts (called presence) be presence and the nodes for the individual posts (with comments) presence/123456, where the number is the same as used in the web page for that post. Those two examples could be tied to a JID representing me at Jaiku: ralphm@jaiku.com.

The node for all public posts could be called explore and located at the JID of the whole service: jaiku.com. This would be similar to the web site, where all public posts can be viewed at http://jaiku.com/explore.

It might also make sense to have a dedicated node for a user's profile information, that can be retrieved and presented at a service or application that consumes the social object updates. At least a (full) name and some icon or headshot would be nice to have there. Obviously, subscribing to such a node would mean that future profile changes will also propagate to the consuming entities. An example identifier would be profile, to be kept at the user's JID.

anyMeta

In anyMeta, each thing has an identifier, that could be used for the node identifier as well. However, in the current implementation, all nodes are held by a loosely coupled, generic publish-subscribe service that caters multiple anyMeta instances. We chose to use unique identifiers as generated by the publish-subscribe service, which don't have any relation with the thing identifier.

As you might have guessed, some of the stuff being discussed here has already been implemented in anyMeta. The publish-subscribe service used is Idavoll. It has grown an HTTP interface that is used (internally) to create new nodes, publish items that represent things, and subscribe to, and receive notifications from, remote publish-subscribe nodes. The thing that holds my Mediamatic profile is represented by the node generic/4efe2253-2242-4e01-bfdf-957cc2a9481d at pubsub.mediamatic.nl. All things in this site, but also the PICNIC site, have nodes like this. In a future post I will explore what we do with these nodes.

In this part, we explored how one could organize the nodes that entities can subscribe to to get updates. Some might be tied to the (virtual) JID of the user's account, or associated with the JID of the service itself. Node identifiers might be human guessable, and like the web URIs, or could be seemingly random opaque strings. Implementations that consume subscribe to, and consume notifications from, the nodes at social networking services, should not assume anything about the organization and naming of the providing service. This presents a challenge for the next episode: how does one know which nodes are there and what they are called? So, up next: discovery. Homework assignment: look carefully at the HTML of my Mediamatic profile page.

Ross Burton: GUADEC

Tue, 29 Jul 2008 20:40:00 +0000

Hmm, so I never did blog a GUADEC roundup. In two words: it rocked. Congratulations to Baris and everyone else who organised it!

In other late GUADEC news I finally reviewed the rest of my GUADEC photos and uploaded them to Flickr. I'll try and not take a month to upload next time, honest!

Ralph Meijer: XMPP and Social Networks, part 1: Notifications

Sat, 26 Jul 2008 14:12:11 +0000

The use of XMPP publish-subscribe in federation and third-party applications deviates a bit from the standard use-case. Usually publishing, subscribing and receiving notifications happen through the same protocol on specific (leaf) nodes. Entities subscribe to a node that represents a particular thing they are interested in getting updates for, and when an item is published to that node, these subscribers will receive a notification for that item.

For federating social networks, the focus is on the exchange of updates on social objects or comments between services. For third-party applications, the most important thing is getting updates, preferably as soon as possible. So, for both of those use cases, receiving notifications through XMPP gives it an edge over HTTP: no polling, lower latency, less connections.

How these items are published, does not really matter that much. What you will typically see is that services somehow have a new item available (submission via the web, SMS, e-mail or a web-based API) and want to expose that through XMPP. Posting a new update through XMPP from a third-party client usually does not provide an advantage over existing web-based APIs.

For a service like Jaiku, Twitter or Identi.ca to provide XMPP publish-subscribe support, it is important to define the subscribable unit and provide that as a node. Such a node will usually not be published to directly, but is more of an aggregate node. Examples would be: all updates by a particular user, all updates in particular channel, all updates by a user and his contacts, all public updates. An other example could be: all comments on a particular social object.

Conceptually, all such aggregate nodes are internally subscribed to a particular subset of new and updated social objects and comments. You might even implement it exactly like that. Think of a prospective search that is captured by a node: every time a new item comes into the service, it is determined which of the provided nodes would be a match for this item, based on author, contact lists and permissions. Subsequently, for all of those nodes, a notification will be sent out to its subscribers. Telling items apart in this scenario is then likely not done using the service JID, node identifier of item identifier, but using some identifier in the payload, like Atom's id element, although those other identifiers might provide a context.

For those familiar with the concept of XMPP publish-subscribe collection nodes: those would be a special form of aggregate nodes that make it explicit what their relationship to the nodes they aggregate items for is.

Ralph Meijer: XMPP Summit #5: XMPP and Social Networks

Sat, 26 Jul 2008 14:12:05 +0000

The major topics on the 5th XMPP Summit were Jingle, and XMPP as a complementary protocol next to HTTP for building social networking services, as stpeter briefly mentioned. While I think that the consensus on OAuth over XMPP, was very important, I think we also settled on a good set of best practices for federating social networks using XMPP Publish Subscribe.

This particular topic has had my full attention over the last year or so, and it is about time that I start writing about that, explaining the afore mentioned best practices in their context. As this covers a lot of ground, I'd like to make a series out of it, each detailing a particular aspect.

Topics that will come by include: the subscribable unit and how notifications are generated, payload formats, discovery, local representation and implementation strategies.

Augie Fackler: Best iTunes Message Ever

Sat, 26 Jul 2008 00:45:01 +0000

"Purchasing updateAllSoftwareButton" just made me chuckle. Couldn't they have at least named the button something like "All Software Updates" so that a purchase would result in seeing "Purchasing All Software Updates" or something else sane?

Christian Hammond: Django Development with Djblets: Data Grids

Fri, 25 Jul 2008 12:16:59 +0000

Table of contents for Django Development with Djblets

Django Development with Djblets
Django Development with Djblets: Custom Tag Helpers
Django Development with Djblets: Unrooting your URLs
Django Development with Djblets: Data Grids
Django Development with Djblets: Dynamic Site Configuration

It’s been a while since my last post in this series, but this one’s a good one.

A common task in many web applications is to display a grid of data, such as rows from a database. Think the Inbox from GMail, the model lists from the Django administration interface, or the Dashboard from Review Board. It’s not too hard to write something that displays a grid by doing something like:

<table>
 <tr>
  <th>Username</th>
  <th>First Name</th>
  <th>Last Name</th>
 </tr>
{% for user in users %}
 <tr>
  <td>{{user.username}}</td>
  <td>{{user.first_name}}</td>
  <td>{{user.last_name}}</td>
 </tr>
{% endfor %}
</table>

This works fine, so long as you don’t want anything fancy, like sortable columns, reorderable columns, or the ability to let users specify which columns they want to see. This requires something a bit more complex.

djblets.datagrids

We wrote a nifty little set of classes for making data grids easy. Let’s take the above example and convert it over.

# myapp/datagrids.py
from django.contrib.auth.models import User
from djblets.datagrid.grids import Column, DataGrid
 
class UserDataGrid(DataGrid):
    username = Column("Username", sortable=True)
    first_name = Column("First Name", sortable=True)
    last_name = Column("Last Name", sortable=True)
 
    def __init__(self, request):
        DataGrid.__init__(self, request, User.objects.filter(is_active=True), "Users")
        self.default_sort = ['username']
        self.default_columns = ['username', 'first_name', 'last_name']

# myapp/views.py
from myapp.datagrids import UserDataGrid
 
def user_list(request, template_name='myapp/datagrid.html'):
    return UserDataGrid(request).render_to_response(template_name)

Now while this may look a bit more verbose, it offers many benefits in return. First off, users will be able to reorder the columns how they like and choose which columns to see. You could extend the above DataGrid to add some new column but leave it out of default_columns so that only users who really care about it would see it.

Custom columns

Let’s take this a step further. Say we want our users datagrid to optionally show staff members with a special badge. This might be useful to some users, but not all, so we’ll leave it out by default. We can implement this by creating a custom column with image data:

class StaffBadgeColumn(Column):
    def __init__(self, *args, **kwargs):
        Column.__init__(self, *args, **kwargs)
 
        # These define what will appear in the column list menu.
        self.image_url = settings.MEDIA_URL + "images/staff_badge.png"
        self.image_width = 16
        self.image_height = 16
        self.image_alt = "Staff"
 
        # Give the entry in the columns list menu a name.
        self.detailed_label = "Staff"
 
        # Take up as little space as possible.
        self.shrink = True
 
    def render_data(self, user):
        if user.is_staff:
            return '<img src="%s" width="%s" height="%s" alt="%s" />' % \
                (self.image_url, self.image_width, self.image_height, self.image_alt)
 
        return ""
 
class UserDataGrid(DataGrid):
    ...
    staff_badge = StaffBadgeColumn()
    ...

This will add a new entry to the datagrid’s column customization menu showing the staff badge with a label saying “Staff.” If users enable this column, their datagrid will update to show an staff icon for any users who are marked as staff.

Custom columns can render data in a couple of different ways.

The first is to bind the column to a field in the model. By default, a Column instance added to a DataGrid will use its own name (such as “first_name” above) as a lookup in the object. If you want to use a custom field, set the db_field attribute on the column. This field can span relationships as well. For example:

class UserDataGrid(DataGrid):
    ...
    # Uses the field name as the lookup.
    username = Column("Username")
 
    # Spans relationships, looking up the age in the profile.
    age = Column("profile__age")

The second is the way shown above, by overriding the render_data function. This takes an object, which is the object the datagrid represents, and outputs HTML. The logic in this can be quite complicated, depending on what’s needed.

Linking to objects

A datagrid is pretty worthless if you can’t link entries for the object to a URL. We have a couple of ways to do this.

To link a column on a datagrid to a URL, set the link parameter on the DataGrid to True. By default, the URL used will be the result of calling get_absolute_url() on the object represented by the datagrid, but you can override this:

# myapp/datagrids.py
 
class UserDataGrid(DataGrid):
    username = Column("Username", sortable=True, link=True)
    ...
    @staticmethod
    def link_to_object(user, value):
        return "/users/%s/" % user

Then

link_to_object takes an object and a value. The object is the object being represented by the datagrid, and the value is the rendered data in the cell. The result is a valid path or URL. In the above example, we’re explicitly defining a path, but we could use Django’s reverse function.

Usually the value is ignored, but it can be useful. Imagine that your Column represents a field that is a ForeignKey of an object. The contents of the cell is the string version of that object (implemented by the __str__ or __unicode__ function), but it’s just an object and you want to link to it. This is where value comes in handy, as instead of being HTML output, it’s actually an object you can link to.

If you intend to link a particular column to value.get_absolute_url(), we provide a handy utility function called link_to_value which exists as a static method on the DataGrid. You can pass it (or any function) to the Column’s constructor using the link_func parameter.

class UserDataGrid(DataGrid):
    ...
    myobj = Column(link=True, link_func=link_to_value)

Custom columns can hard-code their own link function by explicitly setting these values in the constructor.

Saving column settings

If a user customizes his column settings by adding/removing columns or rearranging them, he probably expects to see his customizations the next time he visits the page. DataGrids can automatically save these settings in the user’s profile, if the application supports it. Handling this is as simple as adding some fields to the app’s Profile model and setting a couple options in the DataGrid. For example:

# myapp/models.py
 
class Profile(models.Model):
    user = models.ForeignKey(User, unique=True)
 
    # Sort settings. One per datagrid type.
    sort_user_columns = models.CharField(max_length=256, blank=True)
 
    # Column settings. One per datagrid type.
    user_columns = models.CharField(max_length=256, blank=True)

# myapp/datagrids.py
 
class UserDataGrid(DataGrid):
    ...
    def __init__(self, request):
        self.profile_sort_field = "sort_user_columns"
        self.profile_columns_field = "user_columns"

The DataGrid code will handle all the loading and saving of customizations in the Profile. Easy, isn’t it?

Requirements and compatibility

Right now, datagrids require both the Yahoo! UI Library and either yui-ext (which is hard to find now) or its replacement, ExtJS.

Review Board, which Djblets was mainly developed for, still relies on yui-ext instead of ExtJS, so datagrids today by default are built with that in mind. However, the JavaScript portion of datagrids should work with ExtJS today. The default template for rendering the datagrids (located in djblets/datagrids/templates/datagrids/) assumes both YUI and yui-ext are present in the media path, but sites using datagrids are encouraged to provide their own template to reflect the actual locations and libraries used.

We would like to work with more JavaScript libraries, so if anyone wants to provide an implementation for their favorite library or help to dumb down our implementation to work with any and all, we’d welcome patches.

See it in action

We make extensive use of datagrids on Review Board. You can get a sense of it and play around by looking at the users list and the dashboard (which requires an account).

That’s it for now!

There’s more to datagrids than what I’ve covered here, but this should give you a good understanding of how they work. Later on I’d like to do a more in-depth article on how datagrids work and some more advanced use cases for them.

For now, take a look at the source code and Review Board’s datagrids for some real-world examples.

Previous in series Next in series

Christian Hammond: Django Tips: PIL, ImageField and Unit Tests

Fri, 25 Jul 2008 10:25:40 +0000

Table of contents for Django Tips

Django Tips: PIL, ImageField and Unit Tests

I recently spent a few days trying to track down a bug in Review Board with our unit tests. Basically, unit tests involving uploading a file to a models.ImageField would fail with a validation error specifying that the file wasn’t an image. This worked fine when actually using the application, just not during unit tests.

The following code would cause this problem. (Note that this isn’t actual code, just a demonstration.)

# myapp/models.py
from django.db import models
 
class MyModel(models.Model):
    image = models.ImageField()

# myapp/forms.py
from django import forms
from myapp.models import MyModel
 
class UploadImageForm(forms.Form):
    image_path = forms.ImageField(required=True)
 
    def create(self, file):
        mymodel = MyModel()
        mymodel.save_image_file(file.name, file, save=True)
        return mymodel

# myapp/views.py
from myapp.forms import UploadImageForm
 
def upload(request):
    form = UploadImageForm(request.POST, request.FILES)
 
    if not form.is_valid():
        # Return some error
        pass
 
    # Return success

# myapp/tests.py
from django.test import TestCase
 
class MyTests(TestCase):
    def testImageUpload(self):
        f = open("testdata/image.png", "r")
        resp = self.client.post("/upload/", {
            'image_path': f,
        })
        f.close()

After a good many hours going through the Django code, expecting there to be a problem with file handles becoming invalidated or the read position not being reset to the beginning of the file, I finally decided to look at PIL, the Python Imaging Library. It turns out that at this point, due to some weirdness with unit tests, PIL hadn’t populated its known list of file formats. All it knew was BMP, and so my perfectly fine PNG file didn’t work.

The solution? Be sure to import PIL and call init() before the unit tests begin. If you have a custom runner (as we do), then putting it in here is appropriate. Just add the following:

from PIL import Image
Image.init()

Hopefully this helps someone else with the problems I’ve dealt with the past few days.

Mark Doliner: Poor Planning

Wed, 23 Jul 2008 16:41:40 +0000

Why do ssh, scp and sftp all have different ways of specifying the port number? I run ssh on my home computer on a non-standard port, and it's a pain in the calf to remember which one to use when.

> ssh -p 1234 example.com
> scp -P 1234 example.com
> sftp -oPort=24 example.com (I thought this one was a joke at first)

Mark Doliner: Error Handling

Wed, 23 Jul 2008 16:35:41 +0000

"The general rule of thumb is that it takes $10 to fix the bug during development; $100 to fix the bug in QA; a $1,000 to fix the bug during beta testing; and $10,000 or more to fix the bug post-deployment."

I pretty much agree with that. Not to those specific numbers, just to the fact that it's a pain to fix stuff after the fact. So when developing you should take your time and double-check your code when you're finished. Then have a peer review it and make sure it's perfect.

But there's no way you're going to catch all the bugs. So to make debugging easier you should always have really good error handling and error logging. Check for and log every error possible. That means looking at the return value for every system call. If an error can happen then assume it will happen.

Kevin Smith: I’m not at the XMPP Summit

Mon, 21 Jul 2008 20:16:29 +0000

I’m not at the XMPP Summit, sadly, but I’m being my usual nuisance self remotely, thanks to the summit MUC, and the live feed (thanks bear) at http://www.ustream.tv/channel/xmpp-summit–5. I’m sure someone who’s there can write a summary later, but at least this means those of us not fortunate enough to make it out there can enjoy [...]

Matt Rogers: Sunday - 07/20/2008

Mon, 21 Jul 2008 04:04:41 +0000

Went to see Hancock today. Enjoyable movie. The Dark Knight is up next. Tons of people waiting in line for the next showing (1.5 hours from when we got to the theater). Completely crazy. I want to see it, but I’ll wait until the theaters are a bit more empty.
Completely skipped out on the Kopete bugday today. I figured I was going to have to miss one of them, and it was just too busy today for me to make it.
Started setting up development stuff on the OS X side of the laptop today. Fink has gotten pretty nice since the last time i tried it.
GTD reboot is the reason for going back to the OS X side of things. A few nice applications there that I need that there aren’t open source (or even Linux-based) replacements for. If I can stay better organized and contribute to KDE more by using OS X rather than Linux, then that’s what I’m doing.

Johann Prieur: People in GNOME

Sun, 20 Jul 2008 15:20:44 +0000

Right after our GUADEC presentation, Ali Sabil and I met with Owen Taylor, Marina Zhurakhinskaya (both from Red Hat’s Online Desktop project), Robert McQueen (Telepathy) and Travis Reitter (Soylent). The goal of that quick meeting was to define how to integrate all those pieces of technologies together. Even if People aims to define a data model and a D-Bus API to be used on all the free desktops, we are also part of the GNOME community, thus particularly eager to see People integration with GNOME technologies happening.

It’s been always difficult to make people understand what People is. Moreover, our project is quite unknown. We are not present on GNOME’s main communication channel (Planet GNOME) and that’s something you feel also at GUADEC when the conference room isn’t very populated. In that blog post, I’ll try to explain what is People and what it will be for GNOME.

The best explanation I can give to begin with is maybe that People is actually like “libsoylent”, which has been announced few time ago. The main goal is the same: provide first-class people objects to applications. Our approach is different: we designed People from the backend to application level, which allows us to consider whatever contact source, while libsoylent, based on what is done in Soylent, only uses Evolution Data Server and Telepathy as sources (for now). I think that our development is more advanced and our solution wider, hence even Soylent should use People.

On the next diagram, you can find several components that are part of the GNOME ecosystem. It is based on the GUADEC after-talk meeting result and is not meant to be exhaustive.

Green boxes are what we call contact sources. Any application, service or data source handling and exposing objects referred as contacts, friends, persons, people is a potential contact source. Those contact sources are plugged into the People framework through a backend system. That interaction is represented as dashed arrows on the diagram. Each contact source provides partial information about some contacts to People.
The blue box is the framework. It contains the backend system, a mechanism to gather partial contact informations into high level contact object (often called meta-contacts) and a D-Bus daemon to expose those high level contacts to applications. At some point, we’d also like to expose backend directly to allow synchronization solutions to use them.
Yellow boxes are a few possible applications that could use the framework.

People in GNOME

Concerning Empathy, the current contact list could be turned into a people list, where each item wouldn’t be an instant messaging contact anymore but just a known person, that you may reach using instant messaging. That list would contain some friends you have in Facebook and you could actually subscribe to them on Jabber directly from Empathy, as People would provide to Empathy the information saying that your friend has a jabber account.

This is what People does, it provides contact information to application, and then applications should know how to use that information. People won’t tell you to launch Evolution to write an e-mail, it’ll just tell you that you can reach a certain contact through e-mail. We then need sort of an “activity launcher” and I think that’s what Soylent should be. Using the people list provided by the Empathy widget set, Soylent should allow you to start activities with the people you know. At some point, maybe Empathy’s client and Soylent are meant to be the same application.

Our demonstration at GUADEC was an application containing a list of high level contacts built from local (sqlite database) and distant sources (lastfm and friendfeed web services). The list exposed contact information such as the full name of the contact, a picture, an e-mail address, a phone number, an icon per social network he’s susbscribed to, all those information being provided by People. At some point, Ali brought his mobile phone next to the laptop and his phone’s addressbook automatically became a contact source (over bluetooth) for the application, adding contacts from the phone in the list. This was supposed to show how flexible the backend system is and how cool are the things that People enables :).

Yesterday we set up www.people-project.org which is currently a wiki that will be useful to collaborate on stuff that are not code. We are also releasing 0.0.5 “Smelly hotel lobby” (tribute to our hotel in Istanbul, where people still smoke in public places and People don’t) today, as dictated by our sprint schedule. We are currently developing more backends (Online Desktop, Telepathy, EDS, …) and trying to improve what we have for now. The backlog is huge but considering all the possible cool applications, will is around.

Johann Prieur: jprieur

Fri, 18 Jul 2008 21:42:37 +0000

I finished my internship at Nokia around mid-June. I got to learn a lot of stuff over there, mainly corporate world related and it helped me a lot to define my professional expectations better. A lot.

I graduated at the end of June and I’m now an engineer, for real. A bit sad to leave university and friends but hey, what’s next seems to be shiny as well.

Last week, Ali and I presented the People framework at GUADEC in Istanbul. Few attendees but it seems we managed to reach the right persons and I have the hope that we’ll be able to add value to the desktop with that project. More on that later.

I’m now back home for some time, trying to not get bored.

Process One: ProcessOne at XMPP summit

Fri, 18 Jul 2008 08:50:23 +0000

ProcessOne will be at the XMPP summit from 20th to 22th of july 2008.

We are eager to meet XMPP servers and clients developers, but also our users around Portland.

If you would like us to meet and will not be at the XMPP summit itself, please drop me a mail

Process One: Usage estimation of public XMPP servers per domain

Fri, 18 Jul 2008 08:40:45 +0000

Based on data gathered by our IMtrends search engine, here is our very first "market shares" estimation on XMPP servers, for domains known by IMtrends.

Please, first note that study relies on a panel of 7292 XMPP domains as discovered by the IMtrends engine. This should be still only a partial analysis of the existing XMPP servers: we cannot take into account totally private servers. However, we have received feedback that a lot of the known XMPP public deployments are already gathered in the IMtrends database. You can check by yourself on our search engine and add more public servers in the database that we could be missing: See IMtrends.

Please, also note that those statistics do not take into account the size of the domains (the number of total users and simultaneous users peak).

Now, let's get to the data. Here is a breakdown on server usage per type on the 18 july 2008:

Software Family	Market Share
ejabberd	37.0%
jabberd14	22.4%
openfire	18.4%
jabberd2	11.3%
Google XMPP	6.2%
wpjabber	0.9%
XCP	0.2%
djabberd	0.1%
SoapBox	0.1%
psyc	0.1%
Isode M-Link	0.1%
tigase	0.1%
timp	0.1%
JCS	0.0%
MOO-XMPP	0.0%
Unknown*	3.0%

We will publish more statistics analysis in the coming months, so stay tuned.

Kevin Smith: Thanks, Psi

Thu, 17 Jul 2008 21:09:05 +0000

I’ve been meaning to make this post for an age… A decent number of people thank the Psi project, and that’s great; I get thanked more than most, I think, because I’m the one who gets to make the release announcements, and it looks like I do all the work because of this. These days, I [...]