Gustavo Noronha (kov)

Most times when I blog about WebKitGTK+ it’s to talk about new features, and their usage in Epiphany. This time I’d like to tell those who care more about our test infrastructure. Like I said in my previous post, testing is something we take very seriously in WebKit land. It would be hard to get such a complex project, with such diversity of platforms moving forward without automated testing.

Apple hosts a buildbot master that controls a whole lot of build slaves, for many platforms. Today we added the fourth WebKitGTK+ build slave to the family: 64bits release. This makes up for 4 build slaves in total, 2 release bots sponsored by Collabora, and 2 debug bots sponsored by Igalia. These bots build WebKitGTK+, and then run what we call the “layout tests”. I use quotes there because the name is a bit misleading. Despite being HTML/JavaScript tests, they cover a whole lot of functionality, and tests for regressions in many areas, including security, crashes, animations, media playing, DOM behaviour, and javascript API behaviour. WebKitGTK+ bots currently run 6397 tests, which represent about half the available tests.

Our bots are also, as of today, the first ones to run platform-specific API tests. Almost a year ago we started writing small tests based on glib’s GTestSuite, and they have been very valuable in helping us make sure our API expectations aren’t breaking (at least unknowingly), and to be able to test things that would be very hard to have Layout Tests for. So, yay! Thanks to everyone involved.

Back to layout tests, now: the other half is currently skipped because of one of three main reasons:

• We suck, and the test fails for real, either because we are missing implementation of something it uses (say, JavaScript isolated worlds, which has been recently added), or because our implementation is wrong
• The test is a render dump, and we did not generate results for it yet
• We lack functionality to run the test in our DumpRenderTree implementation

The first one is the worst of all. It means we have broken functionality, or lack web compatibility. The second one is less bad, we can usually trust layout, and rendering to be good because most of the rendering code is shared (thought there are exceptions, of course). Render dumps are a special way of representing the render tree as text, and we need to generate our own results because of differences in things such as font sizes. The third one is also pretty bad - it means we cannot test some features; DumpRenderTree is an application that uses the port’s API to run the tests, and provide additional JavaScript API through JavaScriptCore.

If you feel like helping WebKitGTK+, choosing a bunch of these (specially non-render-dump) skipped tests to make them pass could likely be a good first step =).

So, one of the things I get to do during work hours for Collabora is to contribute code, and do maintenance tasks for WebKitGTK+, and have been doing so since early last year, working on all kinds of things, from improving the network backend to handle the real-world web, to fixing scrolling problems, while reviewing patches from the many awesome developers who have been joining us (more on that later =D).

One of the big features I have worked on this past month or so, along with Xan Lopez is the Page Cache. The page cache is a feature of web browsers that makes going back, and forward between pages in the same view very fast. It’s better explained in this post, but to summarize, the idea is that instead of destroying all the work you have done since downloading the resources, and having to reparse/rebuild the structures the view uses to display the page from the cached resources, you hit pause on the page, and store the whole thing as is, and when coming back to it, you just hit play. You can see in the video two instances of Epiphany, one with the page cache enabled, one with it disabled. Easy to see which was has it enabled. Thanks to KiBi for the suggestion regarding a page that shows this easily =D.

We initially thought we had this feature enabled, since our initialization functions (that exists since before the current maintainers were involved) did setup the number of desired pages in the cache, but during the hackfest we held in December we found out we were fooled all this time. Enabling the page cache does make going back faster, but also made lots of things become unstable and crash.

Since then, we have been working on figuring out all the problems, and fixing them, using help from adventurous users of in-development software ;D. I believe we’re now at a point in which I can happily declare the GTK+ port has a working page cache in trunk! If you’re interested in the nasty details, bear with me!

Let me go back in time a bit, and show you what problems we had. First, some background: the GTK+ port deviates a lot from the other ports when it comes to scrolling. This is because, when designing this part of the port, Holger Freyther had a very nice idea in mind: that the WebView should be a first-class citizen GTK+ scrollable widget. Meaning it would use GTK+’s adjustments for scrolling, and be able to interact with any parent scrolling widget, be it a GtkScrolledWindow, or a MokoFingerScroll.

We cannot just throw away all the rest of the scrolling code in WebCore, though, that deals with all the details related to interacting with the DOM, and JavaScript code. This means our WebView contains adjustments that need to be set, and unset on our port’s version of WebKit’s own representation of the view, called the FrameView, to interact with it, and to get updates on the bounds of content, and such. For every load, in the non-page-cache case, a new FrameView is created, the previous one is destroyed - this means we need to set the adjustments on every load.

The problem starts when you have the page cache enabled, because the code path used to do what is called “commit” the load of a cached page (that is, start replacing the content that is currently being displayed by the one that should now be displayed) is completely different, and we were not setting the adjustments on this new view, so we started with that.

But all was not well. We were still having weird behaviour with scrollbars disappearing, and becoming the wrong size, and worse, crashes when “back” was hit. We then started investigating in more detail how it is that the page cache does its magic, to try and figure out the source of all evil.

It turns out that when you leave a page that can be cached, the existing FrameView is no longer destroyed - it is stored as is in a CachedFrame to be restored if you go back, and a new one is created for the new page. This was having the undesired effect of having the adjustment be set in more than one FrameView at once, causing all kinds of (predictable, after we knew for real what was going on) unwanted effects. Thus, we reworked the code to make sure the adjustments are only ever set in one FrameView at once, making sure they are unset when the FrameView is being frozen, and reset when it’s being restored from the page cache.

Last, but not least, it was discovered that going back to a page that contained resources with data: URIs (such as Google results pages which contain a small number of image hits) also caused a crash. This was because our network backend was not storing the data: URI in the ResourceResponse objects it fed into WebCore. The page cache relies on those responses to recreate the requests it uses to artificially replay the load when restoring the page from the page cache, so we fixed that as well.

What can be taken from all this? Building browsers is a lot of hard work! I can’t think how we could deal with this level of complexity without the awesome testing suite of WebKit. The good news is all of those issues I talked about in this post are now covered by the automate tests that run as part of the normal buildbot cycle in our bots, so we’re covered for the future, at least for these specific problems =D.

Isso era pra ser comentário de um post no blog do Eriberto, mas como ele exige login para fazer comentários e reiniciar a senha que eu obviamente esqueci demorou muito, vai um post mesmo.

O post dele trata da ida embora da combinação Control-Alt-Backspace, que matava o X. Essa combinação ir embora é uma coisa boa, na minha opinião por vários motivos, o principal deles sendo que eu já derrubei meu X várias vezes sem querer fazendo um comando no Emacs ou no Bash ;D. Se uma combinação desse tipo era tão importante e precisava ser intuitiva é melhor a gente parar de zoar a Microsoft fazendo camisas com C-A-del e começar a zoar a nós mesmos =P.

Depois ele diz que “Novidade: agora é AltGR PrintScreen K.”. Não é bem assim. Essa combinação existe desde sempre e é um dos comandos do chamado Magic SysRq do Linux. Essa combinação específica serve para matar todos os processos do virtual terminal atual, que acaba por ser o suficiente para conseguir algo semelhante ao C-A-B. Outras combinações são AltGr+SysRq+e, que manda um SIGTERM pra todo mundo, AltGr+SysRq+i, que manda um SIGKILL pra moçada, AltGr+SysRq+s que faz um sync de emergência (manda pro disco tudo que tá em memória esperando pra ir pro disco), AltGr+SysRq+u, que remonta os sistemas de arquivo em modo leitura, e AltGr+SysRq+b que dá reboot.

As combinações mágicas precisam estar habilitadas no Linux (estão na maioria das distribuições, por padrão) e podem te ajudar a sair de um “travamento”, mesmo que seja reiniciando o sistema de forma limpa, sem risco de perder dados.

Did I mention I hate flash? I do. It crashes a lot, and is overall a bad thing for the web, in my opinion. But I do enjoy watching videos on the web, and unfortunately, up to this day, flash is what most sites use to show videos. Months ago I read a couple of blog posts with nice hacks to make Firefox able to play youtube videos without using the flash player. Some recent discussions with colleagues at work got me itching to try my hand at something similar for Epiphany.

So I went ahead, and wrote a new extension that does just that - in youtube video pages, it finds the flash player element, and replaces it with an HTML5 video tag pointing to the actual movie file. This causes the internal HTML5 media player built into WebKitGTK+, that is based on GStreamer, to play the movie. That means you only need to have the necessary GStreamer magic, and the extension enabled, to enjoy the movie.

There are some caveats - in-video text messages are gone (though I’m pretty sure we could get them added somehow), playlists, and other places which display videos other than the ‘normal’ video watching page are not handled, youtube needs to think you have the flash plugin installed, at least, so the only way to make it work right now is to actually have a flash plugin installed. I think we could probably get away with the last problem somehow, by looking at what the totem youtube plugin code does, for instance, and replicating it.

One thing everyone forgot to talk about the WebKitGTK+ hackfest was that master Dan Winship added basic Content-Encoding support to libsoup, and patched WebKitGTK+ to use it. If you are using a recent enough version of those you will finally be able to visit web sites that send gzipped content despite the browser saying it could not handle it, like the Internet Archive.

This was one of those cases in which the web shows all of its potential to behave weirdly. The HTTP/1.1 RFC says that if an Accept-Encoding header is not present, the server MAY assume the client accepts any encoding, so we were having many sites send us gzip content even though we did not support it. We then started sending a header saying “we support identity, and nothing else!”.

It turns out the web sucks, so many servers were not happy with a full header, and started giving us angry looks (slashdot, for instance, would not render correctly because it started sending encoded CSS files!). We then simplified the header we were sending, which made those servers happy again. Some sites, though, completely ignored our saying we didn’t support anything except identity, and sent us gzipped content anyway. Most of these were misbehaving caches (this was the case for Wikipedia), so would work after you asked for a forced reload, which would ignore the cache, but some servers, such as the Internet Archive’s didn’t really want to talk about encodings - they only wanted to send gzip-encoded content.

So, in the end, our only way out was implementing the damn encoding support, which finally happened during the hackfest. Take that, web!

The WebKit hackfest is now over, and I think it was a very productive week. Thank you very much to all who attended, to Igalia for organizing the hackfest, and hosting us so well, to Collabora for having sponsored the event, and allowed me to spend the week working on it, and to the GNOME foundation for having payed all of my costs!

Xan blogged about day 0, and also a summary of all that was done, so I’ll focus on the stuff he forgot to mention ;D. The hackfest, for me, started on day -1 with me not allowing Xan to go sleep before he had reviewed a couple patches of mine to fix DOM context menu handling. It always bothered me that Epiphany failed to open right-click menus in some pages, or let pages handle the right click. Well, this is fixed now, and Zimbra users can now have their right click menus, and WoW players can remove talent points from their calculators =P.

It turns out that many of the attendees don’t like pages messing with their context menus, though, and they had some good points to back up their positions (like pages making it hard to save images, for instance), so I implemented a way to force openning the custom menu: Ctrl-rightclick.

We wanted to use a GtkInfoBar to display questions regarding the form saving - our initial implementation always saved all credentials, but that didn’t sound good enough. Xan and I thought it would be very complicated to make this work, because there were assumptions in the code regarding which widget contains which, but it turned out to be quite trivial - making EphyEmbed a descendant of GtkVBox instead of GtkScrolledWindow, fixing a small number of assumptions, and that was it.

The passwords are saved in the GNOME Keyring. It’s interesting to point out that GNOME Keyring seems to be unhappy with the number of passwords a browser stores - Xan’s daemon was hanging, crashing, and spawning a large number of threads. My daemon decided to take up some 300MB of RAM at one point. It’s somewhat funny to see how much a browser pushes the limits of our platform. We are hoping this will improve with the new keyring APIs, and the rewrite that is ongoing. It’s nice to see my browser form passwords in seahorse, though, and be able to manage them like any other.

One more thing worth of notice, although this post is already a bit too big: one of the main concerns people had during the Hackfest was on making build time smaller. Touching a single file in WebCore causes a debug build of 10 minutes on my laptop. Evan Martin and Benjamin Otte made a push at removing unnecessary includes from WebCore, and WebKitGTK+ files, which brough the build time down a bit. They end up inspiring Aroben, from Apple, to go even further into this, and remove many includes from files all over WebKit.

Evan was also able to bring linking time down by making it possible to link libwebkit without having to build all the intermediate libraries, which brought build time down to 1 minute, when touching a single file in WebCore. Behdad and I also started looking into breaking WebCore up into lots of shared libraries for Debug builds, since we don’t care too much about speed penalties in those. None of these experiments got committed yet, but I am hoping we will be having a better time hacking on WebKitGTK+ in the near future.

It was awesome meeting everyone, by the way! See you around =).

There are few things I really hate. One of them is regressions. Regressions are bad because they usually take away things we are used to rely on, and leave us with the idea that perhaps the technical improvements didn’t really improve our lifes as a user, despite putting less burden on the developers. Software is made for users, after all.

As part of my work on WebKitGTK+, I always keep an eye on regressions, both from previous WebKitGTK+ releases, and those imposed on embedding applications on their migration away from Gecko, and try to focus some of my efforts into lowering their numbers, whenever I can.

In recent times I have worked on removing a few very user-visible regressions in Epiphany, which I see as the most demanding WebKitGTK+ user in GNOME, such as save page not working, missing
favicon support, failing to
perform server-pushed downloads (such as GMail attachments), and not being able to view source. An example of a regression from a previous version of WebKit also exists: in 1.1.17 we started advertising more than we should as supported by the HTML5 media player, causing download to be almost completely broken.

All of these are working if you are using WebKit and Epiphany from trunk/master, so should be on the next development versions of WebKitGTK+ and Epiphany. Other people have also fixed many other regressions; a few examples: Xan has reimplemented the Epiphany customization of the context menu, Frederic Peters provided a work-around for mailto: links while we don’t have SoupURILoader yet, and Joanmarie Diggs keeps rocking on the accessibility front!

If you find regressions, keep them coming! If you have a patch, even better! =)

Next week WebKitGTK+ team gets together to work furiously on improving WebKitGTK+ in a hackfest sponsored by Collabora, and Igalia, and hosted/organized by Igalia. While there I should also get my hands on one of these. Can’t wait! =)

Eu tô há algum tempo ficando cada dia mais chateado com o Banco do Brasil. Apesar de usar GNU/Linux internamente, e aparecer na mídia como uma instituição aliada do software livre, o banco sempre foi um pé no saco pra quem quer usar softwares e protocolos abertos. A idéia de girico da equipe do BB de usar java para uma “solução de segurança” sempre me deixou sem entender. Por muito tempo isso me obrigava a usar software proprietário se quisesse usar o banco online. Resultado: eu não usava banco pela Internet até a Sun lançar o java como software livre.

Acontece que o navegador que eu uso atualmente não tem suporte a plugins java ainda (porque, claro, o plugin do Java não é um plugin normal, como todos os outros que já funcionam), e eu uso atualmente amd64, e, caso você não saiba, a ’solução de segurança’ do BB só funciona com 32 bits. Como todos sabem, java é portável.

Algumas coisas, como essa, eu consegui passar por cima. Por exemplo: eu estava um dia olhando uma propaganda do banco sobre recursos para o iPhone. Na hora veio na minha cabeça: mas, peraí, iPhone não tem Java! Como, então, se usa iPhone para acessar o banco? Óbvio: retira-se a solução de ’segurança’, que sempre foi uma estupidez inútil, e se permite fazer algumas operações pelo iPhone. Não entendo muito bem por que a cisma com o iPhone, hype é foda, mas de qualquer forma, eu obviamente pensei em fazer meu próprio iPhone! Aqui vai, como!

Instale o Midori, no Debian basta instalar o pacote ‘midori’. Abra o Midori e vá no menu ‘Edit’, opção ‘Preferences’, e na aba ‘Network’ mande o Midori se identificar como ‘Custom’. Adicione a seguinte linha no campo de texto:


Mozilla/5.0 (iPhone; U; CPU like Mac OS X; en) AppleWebKit/420+ (KHTML, like Gecko) Version/3.0 Mobile/1C25 Safari/419.3


Pronto, você já pode logar no BB online pelo Midori, como se fosse um iPhone. Ḿas pra eu acessar pelo meu navegador normal não pode, né, BB?

Nos últimos tempos, minha relação com o banco ficou mais complexa: eu recebo meu pagamento pelo trabalho que faço para a Collabora por transferência internacional, e isso faz de mim um exportador de serviços (que chique, hein?), o que me obriga a usar uma estrutura similar a de empresas no trato com o banco, incluindo usar o horrível, péssimo, mal feito e bugado “Gerenciador Financeiro aplicativo”. Feito na maravilhosa linguagem Java™, que como todos sabem é Write once, debug everywhere™.

Esse pedaço de fail é extremamente chato de mexer, muito complicado. A única coisa fácil que dá pra fazer nele é errar. Mas até funcionava no meu amd64 (a solução de segurança não tava nem aí que ela falhava). Acontece que no último mês eles mudaram a forma como isso é feito, e agora não funciona mais com amd64, então eu precisei usar uma vm 32bits pra fazer o que eu preciso fazer.

Como isso já está beirando o ridículo, e eu sei que a única coisa que me causa esses aborrecimentos é a incompetência técnica do Banco do Brasil, acho que chegou a hora de repensar definitivamente qual banco usar para minhas necessidades bancárias. Sugestões?

… de saber onde foi parar o código que o Katatudo prometeu por volta de 2004.

Pelo jeito vamos ficar sem, mesmo. Hoje em dia o Leonardo Cardoso, que nos tinha prometido que o trabalho estava sendo feito já saiu da empresa sem cumprir a promessa, e até a minha única esperança e que algo fosse sair, o Cristiano Anderson, parece já ter ido para pastagens mais verdes, também sem cumprir a promessa que fez =(.

Mas agora acho que não precisa mais, né? A propaganda já foi feita, a empresa já foi comprada, todo mundo já pôs o que queria no bolso, uma outra geração de gente pra ser enganada já entrou no mercado. Se esse fosse o único caso de cara de pau no software livre brasileiro eu ficava até feliz. Nossa vontade de querer acreditar em tudo, junto com nossa capacidade de esquecer de tudo bem rápido torna esse tipo de coisa possível. Também torna possíveis alguns discursos políticos muito engraçados, btw.

Over the last few days I have worked for some hours on packaging the last dependency of GNOME Shell that was missing in Debian: gjs. That work has been sponsored by Collabora, along with the work required to package GNOME Shell itself. I was very impressed by the time it took for FTP Masters and Assistants to deal with the new packages - it only took a couple days for each; kudos!

What this all means is that you can now easily test GNOME Shell, and even make it your default environment on a Debian unstable system, by just installing the ‘gnome-shell’ package. As I expected, some people have been having trouble with the new package, but the general feedback has been fairly positive. If the README.Debian package doesn’t help you with your specific issue, do take the time to file a bug report!

By the way, thanks go to Joss, for his work on dh_girepository, and for him, and Sebastian Dröge for their work on creating the new GObject Introspection policy.