Ma petite entreprise

As you may know I've been very interested in the open-data initatives, and have personally contributed to it, presenting my work at conferences such as BitNorth 2009 and OpenData Ottawa 2010.

Last month, Mike Lenczner (Ile sans fil) invited me to co-found MontrealOuvert along with Jonathan Brun and Jean-Noé Landry. I'm very excited about this as we all have complementary skills and a strong will to make things move forward.

At FFunction, I've found it rather hard to find clients willing to start data visualization projects, mostly because they are still hindered by the rigidity of their information systems. Cities and government are no exception to this: fragmented information, incompatible database technologies, no standard format or repository of data are common problems there as well.

Open data is a key concept in addressing the problems of managing and making use of today's vast quantities of data: by creating repositories of available data, by publishing this data under an open license, by using a standardized format supported by open-source technologies, and by exposing it via Web APIs we can already solve a lot of problems organizations are facing today... and lay the foundation for powerful, interactive, visual tools that will help us reveal the potential of these terabytes of data we're already collecting.

Speaking of which, I should create a visualization of my blogging activity: I'm pretty sure there are interesting patterns to discover !

So last Friday, updating my Ubuntu 9.10 resulted in my inability to log in -- apparently something broke tcsh, which has been my shell of choice for some time now. It's an awkward situation to be unable to login to your computer, but (thank God) I had a "demo" account which allowed me to get things right. I wonder what would have happened if tcsh broken on one of my servers... I remember being unable to find a way to start an ssh session without starting the user's default shell... ouch !

Anyway, I've been contemplating the idea of switching to zsh for some time, and I found it was the right time to do the transition. So far, so good ! Here are the things I really loved in zsh:

• Proper right prompt support: tcsh was the only shell to support a proper right prompt -- when I mean proper, I mean it works even when you resize the terminal or when what you write on the command line reaches the right prompt. It's a small detail, but this kept me hooked to tcsh
• Amazing completion: I was always jealous of bash to allow completion of $VAR even within expressions (like /some/path/$VAR). Zsh does this and much, much more. Try typing "man <something>" and it will tell you if there is a man page or not
• Good programmability: tcsh does not support functions, which pleased my sense of simplicity, but which starts to be painful when you want to do advanced prompt configuration (like tell if you're in a screen session or within a git repository). I know bash does this too, but zsh just seems better
• Good community support and documentation: it's not too difficult to find good zsh configuration examples, and the community is very responsive, which makes the transition much easier !

I've also tried the FISH shell, but its relative slowness and worse, lack of vi bindings made me prefer zsh. It's interesting to try though, if you don't despeartly need a right prompt and vi bindings

I recently had to ship an application which was primarily made of a couple of standalone Retro-based web servers, all mounted on a Pound HTTP proxy. Pound is much better than Apache for proxying, first because it's faster, easier to configure, but also because Apache will often block and give 503 - Service Not Available when you back-end web-service crashes while serving a request. Pound goes away with that nicely, and has the added bonus of periodically checking the health of your back-end web servers.

More recently, I've been working on a project using the fashionable 'redis' and 'mongodb' databases. In both cases, I was faced with the problem of automatically starting and monitoring non-forking processes, which did not come with any /etc/init.d/ management script.

I've tried Monit, but really spent two days trying to get it to automatically start and monitor processes that I was "daemonizing" with the daemonize command. The main issue I had was that Monit would not restart the daemon if the pid was still present, and was apparently unable to check if the pid actually matched a running process -- so it was just assuming that the daemon was running if there was a pid file. It's maybe me using Monit wrongly, but I gave up after two frustrating days.

I then found supervisord which despite a lack of buzz, is actually a pretty decent solution for this particular problem: it expects your command to be non-forking, and will manage all the daemonizing and monitoring aspects itself. The documentation is not so great, but writing a rule to automatically start and monitor a service is as simple as this:

[program:services-redis]
priority=10
directory=/opt/services/redis
command=/opt/services/redis/redis-server /opt/services/redis/redis.conf
user=root
redirect_stderr=true

if you want to try it, here is how to install it:

sudo aptitude install python python-setuptools
sudo easy_install supervisord

then vi /etc/supervisord.conf

[supervisord]
logfile=/var/log/supervisord
loglevel=info
pidfile=/var/run/supervisord.pid

[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
http://localhost:9001

[inet_http_server]
port=9001

and just append whatever service configuration you'd like. Here's what you'd do for Apache:

[program:apache2]
command=/usr/sbin/apache2 -c "ErrorLog /dev/stdout" -DFOREGROUND
redirect_stderr=true

Again, note that supervisord asks your programs to not fork (ie. run in foreground), and will do the daemonization for you. As such, it's a good replacement to init.d as well as Monit, and because it offers command-line, Web, XML-RPC and Python interop, it makes it a very good candidate for managing your services across a various range of server nodes.

The only thing I'd like to have is to be able to retrieve is per-process metrics, such as cpu, memory, number of threads -- but these are already available in the /proc filesystem on Linux.

So to sum this up, if you have to make sure a set of programs are always running, and if you want a simple way to configure which of these programs have to run, supervisord is your friend !

PS: If you want to automatically start supervisor, here how to set it up with init.d on Ubuntu 9.10

I realized I did not share the slides of my Montréal Python 8 Pamela presentation. Pamela is language/tool that allows developers and designers to write HTML easier by providing a CSS-savvy, Python-inspired syntax to HTML.

Pamela comes with the following features:

• Indentation is used to express structure
• Shorthands for CSS classes and ids
• Syntax that looks very much like HTML, except simpler
• Command-line program to convert your .paml files to HTML
• Optional embedded web server to test your Pamela files

Pamela is the perfect companion to CleverCSS, so if you're interested, have a look at the following presentation:

After presenting Retro and Pamela, I will be giving a short talk on Sink (not yet released) at Montréal Python 9 (September 30th at UQAM).

Sink is a simple but handy tool that allows you to compare and take snapshots of multiple directories, going a little bit further that what you could do with "diff -r". I will be presenting how to use sink to :

• see what has changed on your filesystem after a ./configure && make && sudo make install
• do a post-install and post-configuration backup on a new box
• merge two or more directories

Sink is written in Python and has been used for years now, so I think it's high time that I present it to the Montreal Pythonistas ! See you Wednesday at Montréal Python 9

I've been quite busy working on dataviz-related project, but also web-related projects which are making it to production, and on that topic I has some thoughts I'd like to share

Something like 8 years ago I was still studying at the engineering school, working on the theory and practice of agent systems. Agents were a much-hyped term at that time, but they appeared to me as an evolution of OOP towards more scalable and more maintainable architectures.

The essence of agents (or actors, to a lesser degree) is autonomy (concurrency) and communication (message-passing) -- concepts which people having played with any serious massively concurrent, distributed platform (like Erlang) will be familiar with. In practice, agents are really just "objects with an inbox" with the ability to have access to an execution thread to schedule tasks.

What makes agents more interesting though is that using them will impact the way you architecture your application. Instead of having a monolithic application that process everything "vertically", you have an application composed of "horizontal" services delivered by agents.

To put this in context, you might have a Web application that uses XMPP, SMTP and database services which are available in the application environment -- the XMPP, SMTP and database services might be offered by one (or more) agents. You then don't need to have a heavyweight Web application importing a sh**load of dependencies, but simply benefit from the high-level by using a simple communication protocol.

There is nothing new here, the concepts were already there in CORBA 15 years ago, but with today's focus on "cloud computing" the agent paradigm seems to me as more interesting than ever -- because we now have a context to put it to good use. People using GNOME may know of D-BUS, which provides a similar architecture to enable easy "exchange of sevices" between applications.

So even in the modest web applications I'm working on, I already see the advantages of using an agent-based approach to designing the apps. However, I also feel some pain due to the lack of a consistent "agent platform" -- I make do with assembling selected components to form a "poor's man agent platform".

Here are the pieces you'd need to make your own:

• a message bus: something that allows inter-process (and ideally over-network) communication of messages. It's up to you to choose your message format, (I've personnaly chosen JSON with a standard type for messages) and your communication channel (spread or beanstalkd work quite well). I'm thinking about using D-BUS as well, as it (apparently) has a good support for managing your services.

• processes, threads or coroutines: agents need a way to poll their inbox and schedule arbitrary bits of code (either in reaction to messages, or in a proactive way). Implementing services with agents means that you'll use other lower-level services, and that communication will be made over the message bus. As a consequence you need to have an efficient way of parallelizing and scheduling the polling of inbox, polling of response status and execution of scheduled code. Nowadays having an event-based reactor and a simple promises/futures API is what seems to be works best -- but it's still difficult because of so-so language support. The alternative is to limit the use of the message bus and ensure that a request never fails... which requires a leap of faith

• administrative tools: because we're talking about a platform, you need things to ensure that your services are running, monitor the load, log problems, etc. Ideally, this could be build on the agent platform itself, by implementing watchdog services so that the necessary "bootstrapping tools" would be kept to a minimum. So far, I make do with crontabs, daemons and probably monit.

Anyway, I'm quite happy to see that the agent paradigm finds relevant application in the domain of web applications, even if there is still some work to be done until we can get a usable platform -- but it seems like most of the pieces are here !

I just gave a short talk on Retro at Montréal Python 7 and was happy to hear that the presentation was clear and showed the interest of using Retro to quickly implement data-oriented Web Services.

While we're hearing a lot of buzz on the "Web 3.0" and all the linked-data thing, I think we still need to have more simple RESTful exposition of data, so here's something to help:

This presentation should help you get started with Retro, and you can also grab the code from Github or clone the repository to get started:

git clone git://github.com/sebastien/retro.git

There is much more to discover about Retro, so I'll let you have a look and play with it !

As you may know, one of my "new year resolutions" is to do more presentations of the different open-source projects I've been working on. I'll be presenting "Retro" (used to be called Railways, but I switched because the name was so 2000 at Montréal Python 7 this Wednesday at La Banque.

Retro is a lightweight declarative web toolkit that I've been using for many web-projects, and that I think is worth sharing ! I'll do a flash presentation (~5min) presenting how it can be used to quickly create web services and why it puts you in a good mindset to design web APIs.

Presenting technologies is always an interesting challenge, especially when there are many similar projects (in this case, there are dozens of web frameworks/toolkits/libs) -- I'll do my best to communicate clearly what are the design decisions that motivated Retro's creation and why it can be better fit for specific applications.

You can check out retro on my GitHub page

Some of you may have noticed that some articles disappeared during the last week... that's because I migrated to a new server (hosted at iWeb.ca) and put the wrong database backup online. So I'm sorry for the inconvenience, but nothing was lost... pfeeeew !

The current credit crisis is already a problem for a lot of small and medium businesses that need more funding to get things going -- and it's already a problem for a couple of startups, which were forced to shut down in spite of having a great product.

Read Write Web's published today an article on Netvibes that feels pretty much like "Netvibes bashing" (the actual title is "Netvibes appears to be dying"). As a Netvibes user I was intrigued by such a news, and reading the article I was quite puzzled by its rather subjective tone.

RWW's article feels like the author is trying to create "bad vibes" for Netvibes, and I really don't get why there's any need for that kind of attitude in the current context. Is there any reason for RWW to be that harsh toward a start-up, especially considering RWW's large audience ?

Anyway, Netvibes is with delicious.com, last.fm and github.com one the few services I use and continue to appreciate -- it's also one of the few French start-ups that have an international success... so go Netvibes, go !

I've been really silent lately, which is a shame because a lot of cool things happened. I've migrated a subset of my research and pet projects to the wunderschön GitHub, where the newest one is K7 a "standard library" for the V8 JavaScript Engine.

I've also been working on a PEG-style parser for Sugar (so it will run for Python, JavaScript and ActionScript) but I'm still fixing the last issues with left recursion support. I have a Sugar grammar almost ready for it, so I am quite excited to see the possibility of having Sugar fully self-hosted.

Not to mention that Datalicious, which I started last summer is going pretty well, with a lot of interesting UI and visualization-related projects.

Anyway, the initial purpose of this post was to announce that I'll be presenting Sugar at the Free Hacker's Union Montréal, which will happen the second Saturday of February (the 14th, probably at Bolidea's HQ).

Sugar is a new programming language that I started designing and implementing some years ago, which features a syntax inspired by Python and Smalltalk. Sugar main use case is as a replacement to JavaScript and ActionScript (or Python), but its best feature is actually that the language implementation is fully transparent: you can have access to a whole program object model and manipulate it as you like (not at runtime, though).

Although I have not written the presentation yet, it will gravitate around the following points:

• Designing a syntax to solve syntactic, semantic and organizational problems (w/r Sugar and JavaScript)
• Presentation of the different traits of Sugar and how they are expressed (in concrete syntax)
• Presentation of Sugar architecture (including parts of LambdaFactory)
• The idea that your program is not your source code, but that your program is what is expressed by your code
• The wonders of understanding the differences between syntax, semantics and standard library

After this presentation, I will animate a workshop where participants will have the pleasure to hack on the client-side (with Sugar JavaScript backend) and on the server-side (with Sugar Python backend). I'm also thinking of having an optional (for the motivated) session with LambdaFactory showing how people can manipulate Sugar programs and mix in their own languages -- but I don't know if people will be interested.

I'll keep you posted when I've got this presentation ready !

UPDATE: The FreeHacker's Mtl is postponed until (probably) next week. Stay tuned !

I don't usually blog about news in the computer industry, but I feel I have to express my point of view on today's CNet article about Apple filling a lawsuit against PsyStar (website down as of right now).

There used to be a time in the early 90s when Apple tried to follow the PC-path and open the Apple architecture (PowerPC-based at that time). Apple was already in a not-so-good financial position and this experiment did not have the expected result of expanding the Mac market, but rather have the other vendors cannibalize Apple's market share... that was a hard blow at the time.

However, things have changed since Jobs returned to Apple, and although the Mac market did not grow to a 50% share it's still pretty solid -- especially since OSX who won the hearts of many developers because of its UNIX lineage. Now Apple is making tons of cash with iPods, iPhones, the MusicStore... its hardware and software are ahead of the curve, inspiring both other hardware manufacturers and software developers (proprietary or open-source) around the globe.

To me, Apple is a hardware manufacturer, who also happens to have a passion for software development, and had the guts to carry on with a different vision with a strong will. Thing were harsh in the 90s, but things have changed since them. Our perspective on software (and OSes in particular) has evolved: we are getting away from the proprietary, Microsoft-monopoly-style perspective, and moving to a shared collaborative software where everybody can participate and interact.

Apple has made much progress towards open-sourcing technologies (Darwin, RendezVous, WebKit, Foundation, etc), but is still strongly attached to keeping OSX proprietary. Even more than that, now that Macs are "just" very well designed PCs, they want to have the monopoly on OSX. This is an old-school, conservative behavior that shows if not lack of understanding, elements of fear.

So Apple, let some competitors in, because now they won't eat your market shares but rather let your market grow. You should be more worried about Linux getting more shared in the desktop side -- it's still small now, but look at the progress made since the end of the 90s.

I think Apple's control-freak behavior does not fit anymore in today's context : they're present in too many sectors, and try to control everything end-to-end in every one of them. That's a clear lack of understanding of the profound mental (and social) changes that happened thanks to the Internet and open-source software. So Apple, please focus on making really cool hardware, providing paying services around them, but open source OSX ! Let your OS live its life and help your market share and technological leadership grow.

I don't know how many of you are familiar with NeXTSTEP/OpenStep and the GNUstep project, but these "desktops/environments/frameworks/systems" were one of the hottest stuff that happened during the 90s (along with OpenDoc) in trying to change the desktop as we know it today.

The "*step" systems were/are all based on Objective-C, which is a Smalltalk-like C dialect. It's actually based on a simple message-driven OO runtime written in C, decorated with a set of extensions to the C syntax that mimic Smalltalk message-passing style (object say:"Hello" to:thisPerson). Smalltalk being (in practice) not only a language but a whole environment, the language itself had an influence on how the "*step" systems were designed from a user perspective.

A couple of years ago some people coming from the GNUstep community started the Étoilé project, which is based on the same technology as the "*step" systems. While GNUstep seems to be quite busy trying to catch up with the Cocoa API constantly being updated by Apple (Cocoa is the successor of OpenStep), Etoilé took the path of innovation, trying to modernize and evolve the concepts into something new.

Here is Étoilé mission-statement (from their website):

Étoilé intends to be an innovative GNUstep based user environment built from the ground up on highly modular and light components with project and document orientation in mind, in order to allow users to create their own workflow by reshaping or recombining provided Services (aka Applications), Components etc. Flexibility and modularity on both User Interface and code level should allow us to scale from PDA to computer environment.

So far, I'm still waiting for a clear example on how all this "fits together", namely an illustrated explanation of how (we) users can reshape our worflow by recombining services (which is very much like OpenDoc when you add the document-centric perspective).

In the meantime, I've been quite impressed by some posts by David Chrisnall related to how he's trying to improve the Objective-C runtime (the GCC implementation being... well... look for yourself !) and to bridge Smalltalk and Objective-C.

I really like the double approach that the Étoilé people have : they design both at the language and at the interface level. As clearly stated in Ian Piumarta S3 presentation "Late-bound object lambda architectures", most of the current desktop systems/framework fail re-use (aka. re-invent the wheel) because the underlying language does not offer proper support for it. There is clearly a need to design both the API/architecture as much as the UI: APIs being "just another interface" for human-machine interaction.

Anyway, from a technical standpoint the approach is very seducing: Objective-C is a good base, it's well tested and now in use in many systems (Mac, iPhone,...). Now if you improve the runtime, plug LLVM in and allow to switch the syntax you have a pretty flexible and fast system, akin to the original Smalltalk vision.

I'd be personally really pleased to use a better Objective-C runtime library, take advantage of the (quite cool) Objective-C class library, have great performance (LLVM and C), and be able to plug the syntax I want. It seems like this is not too far away !

This sunday I took some time to watch Linus' Google Tech Talk video on Git. Like others, I've been interested in SCM for a long time (started with CVS, then used PRCS for a couple of years, tried Arch, Monotone used Darcs for a while, got poison-patched, tried Git and eventually settled on Mercurial) and seeing Git getting better documentation and easier CLI, I decided it was time to give it another chance (this being encouraged by the stated of Mercurial API).

Judging from Linus' reputation, I expected the talk to be fairly technical and to go directly to the "guts of git" . To my surprise, I found that instead of detailing the architecture and implementation of git, Linus was focusing much more on the use and the social impact of git and its fundamental concepts...

In the beginning, Linus talks about the concept of decentralized SCM, explaining briefly what it is, and then explaining in detail why he thinks every open-source projects should use a decentralized SCM model: because in the decentralized model it is so easy to create branches (each repo is a branch), everybody can contribute, even if the maintainers don't trust them. In centralized models, developers have to give credentialx to other developers, and as you usually don't want somebody to mess with the code in your project, you end up being conservative about contribution, spend a lot of time doing politics and only give commit access to a few people.

Because the decentralized model makes it easy to create your own branch and to keep track of changes coming from the "main line", it encourages a healthy evolution of the codebase: you can start working on a new feature on your own, without spending tons of time arguing about it with the lead devs, and once you're satisfied with it, simply advertise your repository URL. If you did a good job (and propaganda people will start to use it, and maybe use your repo as the new "main line" -- or maybe the maintainers will pull your changes from your repo.

Also, according to Linus, the notion of trust in a decentralized models follows very closely our "hard-wired" social patterns: we only have so many people that we can trust. In centralized model, you have to trust everyone with commit access. With decentralized model, you can trust a few people and pull from them. Each of these people may in turn trust a few people and pull from them too, resulting in cascading propagation of changes. For large projects like the kernel, this seems like a very democratic way of managing contributions.

One last interesting point that Linus makes is about performance. We may tend to think that Linus cares about performance because he's a geek and he just wants to brag about how fast his C-based implementation is compared to others -- it is absolutely not the case. Although Linus is proud (and confident) of Git's performance, he explains how having a fast SCM software makes operations cheap: it's easy to commit, branch and most importantly easy to merge.

According to Linus, merging is the most crucial feature of SCM systems -- in most existing systems, merging is a major pain, it is a slow, complicated process. By making it simpler (automatic resolution) and most importantly, fast, git encourages merging, and thus supports the decentralized model (you have to merge often to keep in sync with other "branches").

So why did I wrote "Linus is a designer" in the title ? Well, because in this talk, Linus does not really talks like the regular hacker: he does not only creates something to solve a technical problem (a tool), but he also tries to have an impact on people and change their perception of things. Linus' perspective on the decentralized model, and how it benefits open-source projects, or his perspective on performance and how it favors a specific type of usage (frequent merge) do not (only) stem from technical problems, but rather are the result of trying to create or evolve habits.

This is quite unusual in the world of software, and seeing that coming from Linus, one the world most respected hacker, is even more surprising. In the end, it shows us that there is a bright future for the term of "software design", where both the concepts and interface of a software not only serve their original purpose (help a user in a task) but also manage to make things change, and to make people look at things in a slightly different way... I guess this will probably remind some of you of previous discussions on programming language design

Voilà, that's all I had to say about that ! Now you can just watch the whole thing or read a pretty good summary here.

Dans la foulée, voilà un message pasté "as-is"

For personal reasons, I do not browse the web from my computer. (I also have not net connection much of the time.) To look at page I send mail to a demon which runs wget and mails the page back to me. It is very efficient use of my time, but it is slow in real time.

Et voici le mail en question de... RMS !! Le meilleur étant pour la fin : il faut jeter un coup d'oeil au fil de discussion en entier... bref, si c'est vrai, les bras m'en tombent !