Ma petite entreprise

I just demo'ed a port of Revealicious to a 2 x 42" multi-touch surface built by Christophe Viau at ETS Montréal. It was a lot of fun, and a lot of work too !

I wrote the port in OpenGL and Scala, using TUIO as the interaction communication protocol. Here are some technical notes on the implementation:

• Implementing interactions is hard: bye bye hover/click/drag, you have to deal with multiple cursors, gestures, time(out) and unreliable event sources.
• You need muscle: interfaces are not supposed to hog your CPU, yet the users expect fluid interactions and transitions. Using a fast language and library helps a lot, but you still need to optimise so that you don't eat CPU when the UI is idle (this can be tricky !)
• Tweening will make you happy: as multi-touch, and especially surface interfaces usually require animations and even sometimes 3D, you'll get much better results if you use tweening (like exponential ease out). This small change will make you interface shine at a small cost !
• OpenGL libraries are not designed for UI: almost every single OpenGL library is about games or creating real 3D environments. When you do an interface in 3D, you need a scene graph, but you also really need good picking/selection/events functions. You also need something simple, and an ability to deal with text. While there is tons of OpenGL libraries, I've found none of them to meet these requirements (and I've looked in Python and Java libraries)

Overall, my experience with Scala for this was really great: the performance is there, using immutable data structures is reassuring when you know you have at least 3 concurrent threads in your app (AWT events, TUIO events and OpenGL loop), and the memory footprint is not that bad (you need to constrain the heap consumption). On a language level, the DSL-ability of Scala makes it really convenient to write event handlers, even thought the type systems is sometimes more a burden than a boon. Being able to re-use existing Java libraries is also a big plus.

If you're interested in some design-related notes on this topic, here is the design notes on creating a multi-touch interface on FFunction's blog.

So last Friday, updating my Ubuntu 9.10 resulted in my inability to log in -- apparently something broke tcsh, which has been my shell of choice for some time now. It's an awkward situation to be unable to login to your computer, but (thank God) I had a "demo" account which allowed me to get things right. I wonder what would have happened if tcsh broken on one of my servers... I remember being unable to find a way to start an ssh session without starting the user's default shell... ouch !

Anyway, I've been contemplating the idea of switching to zsh for some time, and I found it was the right time to do the transition. So far, so good ! Here are the things I really loved in zsh:

• Proper right prompt support: tcsh was the only shell to support a proper right prompt -- when I mean proper, I mean it works even when you resize the terminal or when what you write on the command line reaches the right prompt. It's a small detail, but this kept me hooked to tcsh
• Amazing completion: I was always jealous of bash to allow completion of $VAR even within expressions (like /some/path/$VAR). Zsh does this and much, much more. Try typing "man <something>" and it will tell you if there is a man page or not
• Good programmability: tcsh does not support functions, which pleased my sense of simplicity, but which starts to be painful when you want to do advanced prompt configuration (like tell if you're in a screen session or within a git repository). I know bash does this too, but zsh just seems better
• Good community support and documentation: it's not too difficult to find good zsh configuration examples, and the community is very responsive, which makes the transition much easier !

I've also tried the FISH shell, but its relative slowness and worse, lack of vi bindings made me prefer zsh. It's interesting to try though, if you don't despeartly need a right prompt and vi bindings

A friend of mine is working on a project where his Python app has to work on Jython (Python on the JVM). As he had the bad surprise of seeing a noticable performance decrease, I wanted to check this for myself.

I know the JVM is not very good for dynamic languages (yet), but I wanted to test this with a real application. So I took a collection of Pamela files from a real project I finished a couple of months ago, and made Pamela parse the whole files 100 times (Pamela is a markup to HTML processor, similar to HAML).

The numbers are not very suprising, and a bit disappointing for the JVM as a runtime for dynamic languages:

• On CPython, Pamela processed 60 KB/s, and the test run for 30.27s
• On Jython 2.5r0, Pamela processed 30.25 kb/s, and the test run for 63.03s

The tests do not take into account the VM startup, and all the data is pre-loaded to memory. If you compare this very simple benchmark to results of Ruby vs JRuby it seems like having a 50% performance hit is not so bad when switching from native to JVM runtime.

I did another comparison, which is to start a simple Retro web service and run Apache Benchmark (ab -n1000) on it. The result are not really better:

• On CPython, Retro serves 426.52 req/s
• On Jython, Retro serves 53.79 req/s (and if you set concurrency to -C100 you'll likely get a connection reset by peer...)

My bet is that if you use the JDK library (java.util.collections, java.io, etc) you'll likely get better performance -- but the utility of Jython seems to be either running existing Python libraries on the JVM or using Python as a "glueing language" for your native JVM objects. In this latter case, you might as well look into other scripting languages available for the JVM like JavaScript or Lisp !

Anyway, things are likely to change when dynamic method dispatch gets implemented in Java 7 !

I recently had to ship an application which was primarily made of a couple of standalone Retro-based web servers, all mounted on a Pound HTTP proxy. Pound is much better than Apache for proxying, first because it's faster, easier to configure, but also because Apache will often block and give 503 - Service Not Available when you back-end web-service crashes while serving a request. Pound goes away with that nicely, and has the added bonus of periodically checking the health of your back-end web servers.

More recently, I've been working on a project using the fashionable 'redis' and 'mongodb' databases. In both cases, I was faced with the problem of automatically starting and monitoring non-forking processes, which did not come with any /etc/init.d/ management script.

I've tried Monit, but really spent two days trying to get it to automatically start and monitor processes that I was "daemonizing" with the daemonize command. The main issue I had was that Monit would not restart the daemon if the pid was still present, and was apparently unable to check if the pid actually matched a running process -- so it was just assuming that the daemon was running if there was a pid file. It's maybe me using Monit wrongly, but I gave up after two frustrating days.

I then found supervisord which despite a lack of buzz, is actually a pretty decent solution for this particular problem: it expects your command to be non-forking, and will manage all the daemonizing and monitoring aspects itself. The documentation is not so great, but writing a rule to automatically start and monitor a service is as simple as this:

[program:services-redis]
priority=10
directory=/opt/services/redis
command=/opt/services/redis/redis-server /opt/services/redis/redis.conf
user=root
redirect_stderr=true

if you want to try it, here is how to install it:

sudo aptitude install python python-setuptools
sudo easy_install supervisord

then vi /etc/supervisord.conf

[supervisord]
logfile=/var/log/supervisord
loglevel=info
pidfile=/var/run/supervisord.pid

[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
http://localhost:9001

[inet_http_server]
port=9001

and just append whatever service configuration you'd like. Here's what you'd do for Apache:

[program:apache2]
command=/usr/sbin/apache2 -c "ErrorLog /dev/stdout" -DFOREGROUND
redirect_stderr=true

Again, note that supervisord asks your programs to not fork (ie. run in foreground), and will do the daemonization for you. As such, it's a good replacement to init.d as well as Monit, and because it offers command-line, Web, XML-RPC and Python interop, it makes it a very good candidate for managing your services across a various range of server nodes.

The only thing I'd like to have is to be able to retrieve is per-process metrics, such as cpu, memory, number of threads -- but these are already available in the /proc filesystem on Linux.

So to sum this up, if you have to make sure a set of programs are always running, and if you want a simple way to configure which of these programs have to run, supervisord is your friend !

PS: If you want to automatically start supervisor, here how to set it up with init.d on Ubuntu 9.10

I realized I did not share the slides of my Montréal Python 8 Pamela presentation. Pamela is language/tool that allows developers and designers to write HTML easier by providing a CSS-savvy, Python-inspired syntax to HTML.

Pamela comes with the following features:

• Indentation is used to express structure
• Shorthands for CSS classes and ids
• Syntax that looks very much like HTML, except simpler
• Command-line program to convert your .paml files to HTML
• Optional embedded web server to test your Pamela files

Pamela is the perfect companion to CleverCSS, so if you're interested, have a look at the following presentation:

After presenting Retro and Pamela, I will be giving a short talk on Sink (not yet released) at Montréal Python 9 (September 30th at UQAM).

Sink is a simple but handy tool that allows you to compare and take snapshots of multiple directories, going a little bit further that what you could do with "diff -r". I will be presenting how to use sink to :

• see what has changed on your filesystem after a ./configure && make && sudo make install
• do a post-install and post-configuration backup on a new box
• merge two or more directories

Sink is written in Python and has been used for years now, so I think it's high time that I present it to the Montreal Pythonistas ! See you Wednesday at Montréal Python 9

I've been quite busy working on dataviz-related project, but also web-related projects which are making it to production, and on that topic I has some thoughts I'd like to share

Something like 8 years ago I was still studying at the engineering school, working on the theory and practice of agent systems. Agents were a much-hyped term at that time, but they appeared to me as an evolution of OOP towards more scalable and more maintainable architectures.

The essence of agents (or actors, to a lesser degree) is autonomy (concurrency) and communication (message-passing) -- concepts which people having played with any serious massively concurrent, distributed platform (like Erlang) will be familiar with. In practice, agents are really just "objects with an inbox" with the ability to have access to an execution thread to schedule tasks.

What makes agents more interesting though is that using them will impact the way you architecture your application. Instead of having a monolithic application that process everything "vertically", you have an application composed of "horizontal" services delivered by agents.

To put this in context, you might have a Web application that uses XMPP, SMTP and database services which are available in the application environment -- the XMPP, SMTP and database services might be offered by one (or more) agents. You then don't need to have a heavyweight Web application importing a sh**load of dependencies, but simply benefit from the high-level by using a simple communication protocol.

There is nothing new here, the concepts were already there in CORBA 15 years ago, but with today's focus on "cloud computing" the agent paradigm seems to me as more interesting than ever -- because we now have a context to put it to good use. People using GNOME may know of D-BUS, which provides a similar architecture to enable easy "exchange of sevices" between applications.

So even in the modest web applications I'm working on, I already see the advantages of using an agent-based approach to designing the apps. However, I also feel some pain due to the lack of a consistent "agent platform" -- I make do with assembling selected components to form a "poor's man agent platform".

Here are the pieces you'd need to make your own:

• a message bus: something that allows inter-process (and ideally over-network) communication of messages. It's up to you to choose your message format, (I've personnaly chosen JSON with a standard type for messages) and your communication channel (spread or beanstalkd work quite well). I'm thinking about using D-BUS as well, as it (apparently) has a good support for managing your services.

• processes, threads or coroutines: agents need a way to poll their inbox and schedule arbitrary bits of code (either in reaction to messages, or in a proactive way). Implementing services with agents means that you'll use other lower-level services, and that communication will be made over the message bus. As a consequence you need to have an efficient way of parallelizing and scheduling the polling of inbox, polling of response status and execution of scheduled code. Nowadays having an event-based reactor and a simple promises/futures API is what seems to be works best -- but it's still difficult because of so-so language support. The alternative is to limit the use of the message bus and ensure that a request never fails... which requires a leap of faith

• administrative tools: because we're talking about a platform, you need things to ensure that your services are running, monitor the load, log problems, etc. Ideally, this could be build on the agent platform itself, by implementing watchdog services so that the necessary "bootstrapping tools" would be kept to a minimum. So far, I make do with crontabs, daemons and probably monit.

Anyway, I'm quite happy to see that the agent paradigm finds relevant application in the domain of web applications, even if there is still some work to be done until we can get a usable platform -- but it seems like most of the pieces are here !

I just gave a short talk on Retro at Montréal Python 7 and was happy to hear that the presentation was clear and showed the interest of using Retro to quickly implement data-oriented Web Services.

While we're hearing a lot of buzz on the "Web 3.0" and all the linked-data thing, I think we still need to have more simple RESTful exposition of data, so here's something to help:

This presentation should help you get started with Retro, and you can also grab the code from Github or clone the repository to get started:

git clone git://github.com/sebastien/retro.git

There is much more to discover about Retro, so I'll let you have a look and play with it !

As you may know, one of my "new year resolutions" is to do more presentations of the different open-source projects I've been working on. I'll be presenting "Retro" (used to be called Railways, but I switched because the name was so 2000 at Montréal Python 7 this Wednesday at La Banque.

Retro is a lightweight declarative web toolkit that I've been using for many web-projects, and that I think is worth sharing ! I'll do a flash presentation (~5min) presenting how it can be used to quickly create web services and why it puts you in a good mindset to design web APIs.

Presenting technologies is always an interesting challenge, especially when there are many similar projects (in this case, there are dozens of web frameworks/toolkits/libs) -- I'll do my best to communicate clearly what are the design decisions that motivated Retro's creation and why it can be better fit for specific applications.

You can check out retro on my GitHub page

Some of you may have noticed that some articles disappeared during the last week... that's because I migrated to a new server (hosted at iWeb.ca) and put the wrong database backup online. So I'm sorry for the inconvenience, but nothing was lost... pfeeeew !

In the quest for finding the proper "market positionning for Sugar", I just found a new candidate : "javascript meta-language frameworks".

It's actually from a presentation (starting at slide 51 here) that is featured in todays John Resig's blog post.

After the last presentation of Sugar (as you'll have understood, one I'm not really proud of), I realized I had to decouple Sugar and LambdaFactory, and position Sugar as a meta-language targetting JavaScript and LambdaFactory as a generic meta-programming framework. Meta-programming is still a hairy topic that few people are able to understand, and some of the deeper design decisions in LambdaFactory cannot be well communicated with only one example of front-end language -- I think I've been too excited by the possibilities offered by LambdaFactory and neglected presenting Sugar as a viable language, and focusing on it from a feature perspective.

My goal is to finish the PEG-based parsing library, so that I can ship Sugar without depending on C-based dparser, and then increase my guerilla-marketing efforts to promote Sugar as a better-JavaScript. Then maybe, if Sugar becomes more widely used, we'll see people starting to be interested in the underlying meta-programming framework and start writing and sharing transformation passes (like the dead-code removal pass I presented at YULHackers).

Lately I've been doing some back and forth between JavaScript and Sugar source code. In some places I rewrote the JavaScript in Sugar as I was expanding the code, and found myself removing syntactic elements -- the result is quite interesing. Here is the original JavaScript code

$(".do-save").click(function(){var l=;$("li").each(function(){l.push($(this).text())});save(l)})

And then the Sugar version

$ ".do-save" click { var l= ; $ "li" each {l push ($(target) text ())} ; save (l) }

The exercise of removing unnecessary syntax (like function()) and replacing parens by spaces was quite satisfying, and I think the result speaks for itself.

However, there is still some room for syntax simplification to be done on this part of the expression

$ "li" each {l push ($(target) text ())}

where I would like to see something along the lines of

$ "li" each {l push ( $ `target text ! )}

The additions made to the syntax would be

• a single argument invocation with a variable is denoted by functor 'argument
• a zero-argument invocation is denoted by functor !

Although these two syntactic forms are not common, the backquote is used in Lisp to escape interpretation, and the exclamation denotes a command (an invocation).

I guess some people will say that we can't simplify something by adding new elements, but for lack of a better word this is still how I would discribe my goal in evolving Sugar's syntax. Being a visual person, I try to use a "visual" approach to designing Sugar's syntax, and make use of whitespace and simple punctuation elements to add more rhythm to reading the code. I often think of programming languages as textual interfaces, and in this respect, syntax deserves as much design as user interfaces do nowadays.

Then again, there are the questions of style (some people like indentation, some people like braces) and references (some people know Java, some people know Smalltalk). For me, what's important despite the contextual elements is consistency -- the first users of vi may have not liked its style or its references, but making the small effort of using it for some time, even if it looks and feel weird is sufficient to give you a feel that there is an underlying consistency build by careful design.

So I'm just back from the FreeHackers' Montréal YULhackers unconference where I presented Sugar in front of a dozen of people (including some fellow Pythonistas from The Navarra Group).

I decided to use a three sections presentation of Sugar:

1. The origins of Sugar, coming from my experience with common programming languages
2. A practical overview of Sugar, from its design to its actual syntax
3. Why Sugar is a good replacement to JavaScript for front-end web development

Overall, the feedback I got seemed to be a little bit mixed: some people were asking about the practical interest of using Sugar (which makes me think I did not make my point clearly), and there was little reaction to my demonstration of the meta-programming features of Sugar (which seem to me as the most interesting part).

I must say I expected a little bit more enthusiasm from the people there, but I then realized that I wasn't presenting to the LtU crowd, so it's kind of normal that I did not get too much feedback on such a hairy topic as meta-programming languages

Interestingly, Riadh suggested to put more efforts into "marketing Sugar" so that its benefits would be more easily understandable by everybody. So far, my personal attitude towards "marketing my projects" was something along the lines of "it's working for me, I've open-sourced it so people can see if they can make us of it" -- which for some projects like Sugar does not really help their visibility.

More recently, I've started to publish projects like Sugar on github.com and to post announcements on programming.reddit, which is a great way to generate some buzz (it worked quite well for the trendy K7 project), so maybe I'll use this blog to post more "marketing material" for Sugar.

Anyway, the experience was interesting and it gave me a chance to create a presentation that will probably interest some of the regular readers here

UPDATE: The presentation got promoted on SlideShare and is now visible on site front page (for a limited period of time I guess) !

The current credit crisis is already a problem for a lot of small and medium businesses that need more funding to get things going -- and it's already a problem for a couple of startups, which were forced to shut down in spite of having a great product.

Read Write Web's published today an article on Netvibes that feels pretty much like "Netvibes bashing" (the actual title is "Netvibes appears to be dying"). As a Netvibes user I was intrigued by such a news, and reading the article I was quite puzzled by its rather subjective tone.

RWW's article feels like the author is trying to create "bad vibes" for Netvibes, and I really don't get why there's any need for that kind of attitude in the current context. Is there any reason for RWW to be that harsh toward a start-up, especially considering RWW's large audience ?

Anyway, Netvibes is with delicious.com, last.fm and github.com one the few services I use and continue to appreciate -- it's also one of the few French start-ups that have an international success... so go Netvibes, go !

So it's confirmed, I will be presenting Sugar this Saturday at Montréal Free Hacker's Union (see the announce here).

I'm still hesitating on the approach I will use for the presentation: so far I've advertised Sugar as a replacement to JavaScript for front-end web development (which is where it's most useful), but its actual goal is a bit different (Sugar is meant to be a practical meta-programming language). As a result, the presentation draft focuses as much on the conceptual background of Sugar as on its syntax and usage.

My intuition is that it's more interesting to do a presentation on things that people won't find in the Sugar source code, like its inception and design. The presentation being followed by a workshop, I have plenty of time to present Sugar in practice.

However, I know we're in North America here and that people would normally expect a "hands-on tutorial" rather than a theoretical overview followed by practical examples (which is more a French/European way of presenting). So this will be an experiment... I'll let you know how this went