After spending some more time trying to unbreak the repository for the third time I decided it might be a good time for a clean up. I created a new repository, imported all upstream versions for series 1.2.x (which is in squeeze), 1.6.x (unstable), and 1.7.x (experimental). I also imported packaging-related commis for those versions using git format-patch and black magic.

One of the good things about doing this move, and which should make hacking the WebKitGTK+ debian package more pleasant and accessible can be seen here:


kov@goiaba ~/s/debian-webkit> du -sh webkit/.git webkit.old/.git
27M webkit/.git
1.6G webkit.old/.git


If you care about the old repository, it’s on git.debian.org still, named old-webkit.git. Enjoy!

Hackfest black board, photo by Mario

The biggest chunk of time, though, ended up being devoted to a very uninteresting (to outsiders, at least), but very important task: making it possible to more easily reproduce our test results. TL;DR? We made our bots’ and development builds use jhbuild to automatically install dependencies; if you’re using tarballs, don’t worry, your usual autogen/configure/make/make install have not been touched. Now to the more verbose version!

The need

Our three build slaves reporting a few failures

In addition to those, we have another bot called the EWS, or Early Warning System. There are two of those at this moment: one VM provided by Collabora and my desktop, provided by myself. These bots build every patch uploaded to the bugzilla, and report build failures or passes (you can see the green bubbles). They are very important to our development process because if the patch causes a build failure for our port people can often know that before landing, and try fixes by uploading them to bugzilla instead of doing additional commits. And people are usually very receptive to waiting for EWS output and acting on it, except when they take way too long. You can have an idea of what the life of an EWS bot looks like by looking at the recent status for the WebKitGTK+ bots.

Maintaining all of those bots is at times a rather daunting task. The tests require a very specific set of packages, fonts, themes and icons to always report the same size for objects in a render. Upgrades, for instance, had to be synchronized, and usually involve generating new baselines for a large number of tests. You can see in these instructions, for instance, how strict the environment requirements are – yes, we need specific versions of fonts, because they often cause layouts to change in size! At one point we had tests fail after a compiler upgrade, which made rounding act a bit different!

So stability was a very important aspect of maintaining these bots. All of them have the same version of Debian, and most of the packages are pinned to the same version. On the other hand, and in direct contradition to the stability requirement, we often require bleeding edge versions of some libraries we rely on, such as libsoup. Since we started pushing WebKitGTK+ to be libsoup-only, its own progress has been pretty much driven by WebKitGTK+’s requirements, and Dan Winship has made it possible to make our soup backend much, much simpler and way more featureful. That meant, though, requiring very recent versions of soup.

To top it off, for anyone not running Debian testing and tracking the exact same versions of packages as the bots it was virtually impossible to get the tests to pass, which made it very difficult for even ourselves to make sure all patches were still passing before committing something. Wow, what a mess.

The explosion^Wsolution

So a few weeks back Martin Robinson came up with a proposed solution, which, as he says, is the “nuclear bomb” solution. We would have a jhbuild environment which would build and install all of the dependencies necessary for reproducing the test expectations the bots have. So over the first three days of the hackfest Martin and myself hacked away in building scripts, buildmaster integration, a jhbuild configuration, a jhbuild modules file, setting up tarballs, and wiring it all in a way that makes it convenient for the contributors to get along with. You’ll notice that our buildslaves now have a step just before compiling called “updated gtk dependencies” (gtk is the name we use for our port in the context of WebKit), which runs jhbuild to install any new dependencies or version bumps we added. You can also see that those instructions I mentioned above became a tad simpler.

It took us way more time than we thought for the dust to settle, but it eventually began to. The great thing of doing it during the hackfest was that we could find and fix issues with weird configurations on the spot! Oh, you build with AR_FLAGS=cruT and something doesn’t like it? OK, we fix it so that the jhbuild modules are not affected by that variable. Oh, turns out we missed a dependency, no problem, we add it to the modules file or install them on the bots, and then document the dependency. I set up a very clean chroot which we could use for trying out changes so as to not disrupt the tree too much for the other hackfest participants, and I think overall we did good.

The aftermath

By the time we were done our colleagues who ran other distributions such as Fedora were already being able to get a substantial improvements to the number of tests passing, and so did we! Also, the ability to seamlessly upgrade all the bots with a simple commit made it possible for us to very easily land a change that required a very recent (as in unreleased) version of soup which made our networking backend way simpler. All that red looks great, doesn’t it? And we aren’t done yet, we’ll certainly be making more tweaks to this infrastructure to make it more transparent and more helpful to the users (contributors and other people interested in running the tests).

If you’ve been hit by the instability we caused, sorry about that, poke mrobinson or myself in the #webkitgtk+ IRC channel on FreeNode, and we’ll help you out or fix any issues. If you haven’t, we hope you enjoy all the goodness that a reproducible testing suite has to offer! That’s it for now, folks, I’ll have more to report on follow-up work started at the hackfest soon enough, hopefully =).

The way animations usually work in WebKit is by repainting parts of the page every few milliseconds. What that means in technical terms is that an area of the page gets invalidated, and since the whole page is one big image, all of the pieces that are in that part of the page have to be repainted: the background, any divs, images, text that are at that part of the page.

What the accelerated compositing code paths allow is the creation of separate pieces to represent some of the layers, allowing the composition to happen on the GPU, removing the need to perform lots of cairo paint operations per second in many cases. So if we have a semi-transparent video moving around the page, we can have that video be a separate texture that is layered on top of the page, made transparent and animated by the GPU. In webkit-clutter’s case this is done by having separate actors for each of the layers.

I have been looking at this code on and off, and recently joined Joone in the implementation of some of the pieces. The accelerated compositing infrastructure was originally built by Apple and is, for that reason, works in a way that is very similar to Core Animation. The code is still a bit over the place as we work on figuring out how to best translate the concepts into clutter concepts and there are several bugs, but some cool demos are already possible! Bellow you have one of the CSS3 demos that were made by Apple to demo this new functionality running on our MxLauncher test browser.

You can also see that the non-Accelerated version is unable to represent the 3D space correctly. Also, can you guess which of the two MxLauncher instances is spending less CPU? In this second video I show the debug borders being painted around the actors that were created to represent layers.

The code, should you like to peek or test is available in the ac2 branch of our webkit-clutter repository: http://gitorious.org/webkit-clutter/webkit-clutter/commits/ac2

We still have plenty of work to do, so expect to hear more about it. During our annual hackfest in A Coruña we plan to discuss how this work could be integrated also in the WebKitGTK+ port, perhaps by taking advantage of clutter-gtk, which would benefit both ports, by sharing code and maintenance, and providing this great functionality to Epiphany users. Stay tuned!

If you’d like to give it a try, you can clone the repository from gitorious:

$ git clone git://gitorious.org/webkit-clutter/webkit-clutter.git

Then to build it you use cmake. From inside the source code directory do this:

$ mkdir build
$ cd build
$ cmake .. -DPORT=Clutter -DSHARED_CORE=1 -DBUILD_MX_LIB=1
$ make


The BUILD_MX_LIB option is optional – it will build what we call the “Mx toolkit library” in addition to the vanilla one. Then you can test that the library is built and working by running the programs inside the “Programs” directory. Enjoy!

Unlike last year we didn’t find any big design issues hurting our work (page cache, I’m looking at you!) on new futures. I also didn’t have any huge plans for new API, although we did manage to get some new stuff in there, like the plugins management API Xan created, and the further work done by Dan in soup. This meant, from my point of view the hackfest has been a great oportunity to look at refactorings that we could do to further simplify understanding the code, changing it, and even sharing it =). Besides pushing the debian packaging of the 1.3.x series a bit further.

Coruña was great as always, and I enjoyed going around, eating and hacking there, although I got a cold on the last days which kinda hindered my ability to stare at the screen for too long, some times. Now that I’m at the hot brazilian summer again I’ll hopefully get better soon =)

Cheers \o/

Wouldn’t it be nice if Google practiced what they preach?

Update: so it looks like my message went through to the people who needed to see it, and they found a filtering error in the User Agent sniffing code that made it think Epiphany was a too old Safari – I’m told the change will land in Youtube soon, thanks for those paying attention, and working on this! User Agent sniffing keeps being a problem, of course, and there are other stuff to fix, so I will probably still push my patch to spoof the user agent to google services which are still mishandling Epiphany, but it’s good to see some progress being made!

Update2: I started shipping a patch to send the Chrome user agent string to google domains in the Debian package for WebKitGTK+, when the “enable-site-specific-quirks” setting is enabled (which is the case for Epiphany); I already found something we were missing out on =D Google Images seems to have been greatly improved, and now faking being Chrome we are also able to enjoy it:

It should come as no surprise, then, that one of my first patches to WebKitGTK+ was actually making the awesome Web Inspector work in our port. After the initial support, though, not a lot has been done to further improve it, partly because it was already good enough for many uses, partly because I somehow started doing non-web development again .

These last weeks, through my R&D efforts in Collabora, I have been able to push Web Inspector features and integration a bit further. A simple change that boosts the Inspector’s usability quite a bit is having the nodes that are being hovered highlighted. Along with that, the ability to attach the inspector to Epiphany’s window should make it easier to use for poking the DOM.

The Web Inspector has a number of settings that control its behaviour. Since, for instance, enabling javascript debugging may slow down javascript performance, the inspector usually has it disabled by default, and provides a button to enable it. It also provides an option for always enabling that feature, but that does not work right now, because we are not saving/restoring the relevant settings. A solution to that is in the works using the GSettings infrastructure that was recently merged into glib.

Here’s a simple screencast, showing these improvements in action (click the video to check it out in full size):

Here’s some information regarding 1.2.2:

1.2.2 is an update to the 1.2.x stable series; along with a lot of crash, and misc fixes the biggest changes are: 1) the inclusion of a new API from the development branch (webkit_back_forward_list_clear()),
because it’s simple and will help with fixing a problem in Epiphany stable, and 2) lots of drag and drop, and clipboard related work by Martin Robinson.

Despite not being strictly fixes, we believe the stable series has a lot to gain from this work; a couple examples should illustrate this better: the changes included fix both a crash when dragging links from WebKit into other browsers, and the annoying bug that made the cursor get stuck in a grab when dragging, sometimes.

http://webkitgtk.org/webkit-1.2.2.tar.gz
MD5: 40338001324a38b977c163291e8816d3

Here’s some information regarding 1.2.3:

To some such a quick succession in releases may look like a brown paper bag was in order. Not strictly, but indeed 1.2.3 aims to fix some oversights with easy fixes. First of all, WebKit was not buildable with ICU 4.4.1, but thankfully a fix had already been checked in to trunk, so 1.2.3 includes that fix. Secondly, Debian’s Michael Gilbert has done a great job going through all CVEs released about WebKit, and including patches in the Debian package. 1.2.3 includes all of the commits from trunk to fix those, too.

http://webkitgtk.org/webkit-1.2.3.tar.gz
MD5: 0ab5c478a6f5b74a1ae96bf13a456662

You can read some more details, including the list of CVEs that were addressed, in the NEWS file:

http://gitorious.org/webkitgtk/stable/blobs/master/WebKit/gtk/NEWS

Enjoy!

If you make Epiphany or Midori identify themselves as Chrome or Firefox, the game will work. I really don’t get this User Agent sniffing bullshit coming from Google. If you go to gconf-editor, and under epiphany->general set the user_agent key to “Mozilla/5.0 (X11; U; Linux; en-gb; rv:1.9.0.2) Gecko/2008092313 Firefox/3.8″ it works. I’m starting to seriously consider User Agent spoofing for *.google.com as a quirk on WebKitGTK+. Lame.

Update: as a protest, I’m making blog.kov.eti.br and kov.eti.br say Chrome/Chromium are not supported, by doing User Agent sniffing.
Update2: it’s been pointed out to me that the game is not HTML5 – it’s actually smart usage of divs, and flash *urgh* for the audio
Update3: thanks to a friend who works at Google getting in touch with pacman’s designer, it looks like it now works without faking U-A – I’m happy for this, thank you! Despite this good step forward, google is still denying us the nice fade in effect, and still sees us as ‘unsupported’ in Wave and similar products, so I’ll keep my protest for now.

It turns out that if you are a user or developer of applications that use the GStreamer framework, you can start taking advantage of all that freedom right away! Collabora Multimedia has developed, along with Entropy Wave GStreamer support for the new format, and the code has already landed in the public repositories, and is already being packaged for some distributions.

I just couldn’t wait the few days it will take for the support to be properly landed in Debian unstable, so I went ahead and downloaded all of the current packages from the pkg-gstreamer svn repository, built everything after having the libvpx-dev package installed, and went straight to a rather unknown, small video site called Youtube with my GStreamer-powered WebKitGTK+-based browser, Epiphany!:

If you’re running Debian unstable, or any of the other distributions which will be lucky to get the new codecs, and support packages soon, you should be able to get this working out of the box real soon now. Check the tips on WebM’s web site on how to find WebM videos on youtube.