My journey of mastering procrastination has led me to an interesting article on Hacker News today:
RightJS 1.5: 6-8 times faster than jQuery
(The title has been updated to "RightJS Version 1.5.0 Is Out" since I started writing this.)
Wow, I thought! This sounds like an excellent example of cargo cult science, one of my favourite subjects.
I mean I love innovation in this field just like everybody else. But seriously, jQuery is not exactly know for being slow & heavy. So anybody claiming a 6-8x speed improvement must have achieved an unbelievable breakthrough. Either that, or he must be using using the cargo cult method.
Applying the cargo cult method to performance testing is rather simple, which probably explains its popularity. You pick a random series of tests that can be run against the various implementations you want to compete with. Then you spend a few hours hacking away at your implementation until it is the clear winner. Don't give up if it becomes too hard, just tweak the test cases to slightly favor your implementation. It's really as easy as that.
I can totally understand why people are doing that. The opposite would mean that you have to apply the scientific method, which is really cumbersome. First you have to collect data, lots of it. In our case that means performing very detailed analysis and profiling of a large enough set of real world JavaScript applications. Using this data set, you should be able to answer questions like: What are the most common selectors people use? What DOM operations are popular? Which of those are actually relevant to the performance of the analyzed applications? With those answers you can attempt to come up with a series of tests that will rank the various implementations according to their performance. But actually writing those tests will be very hard. Should you use the most distinct and sexy way in each implementation? Or should you use the most effective techniques people have come up with?
Luckily there is a third option. It is called specialized benchmarking. You start by admitting that the things you are going to test are purely based on your curiosity about them, possibly because they are related to the particular problem YOU care about. Make it very clear that the outcome of those tests should in no way be seen as an indicator for overall performance and try to hide them from people who don't know what that means.
Specialized benchmarking will possibly not answer anybodies questions other than your own, but it beats the hell out of the cargo cult method.
Let's examine why the RightJS performance tests get it wrong and what they could do about it. From this point on I will only refer to material on their page, the post on Hacker News was just how I heard about them.
First of all, they claim that their performance benchmarks are there "To give you some ideas about the project quality and abilities". I think that should be changed to: "We are especially fast for the following operations (...), those however are not proofen to be good indicators for general performance in JS projects.". It's kind of like weight loss advertisement. You can show pictures of people who lost 50 pound, but you gotta put that "* Results are not typical" note there. This way people can pause for a second, and remember that there are no magic bullets to weight loss and consider their purchase with that in mind.
After that, they could start to decide whether some of their tests are worth keeping, and if so, make sure that they are as scientific as possible. I'll just use their test #1 as an example, but check the test suite for yourself, to see that this pattern is repeated throughout the entire thing.
Testing jQuery DOM building (343ms*):
"make": function(){
for(var i = 0; i<250; i++){
$("<ul id='setid" + i + "' class='fromcode'></ul>")
.append("<li>one</li>")
.append("<li>two</li>")
.append("<li>three</li>")
.appendTo("body");
}
return $("ul.fromcode").length;
}
Testing RightJS DOM building (80ms*):
"make" : function(){
for (var i = 0; i < 250; i++) {
document.body.appendChild(
new Element('ul', {
'class': 'fromcode', id: 'setid'+i
}).insert([
new Element('li', {html: 'one'}),
new Element('li', {html: 'two'}),
new Element('li', {html: 'three'})
])
);
}
return $$('ul.fromcode').length;
}
I smell cargo! First of all, why is RightJS using a native DOM method, document.body.appendChild, and jQuery has to use .appendTo('body')? Those are two radically different operations, and just to see how radical lets make the following change:
Optimized jQuery DOM building I (194ms*):
"make": function(){
for(var i = 0; i<250; i++){
document.body.appendChild(
$("<ul id='setid" + i + "' class='fromcode'></ul>")
.append("<li>one</li>")
.append("<li>two</li>")
.append("<li>three</li>")[0]
);
}
return $("ul.fromcode").length;
}
Ouch, an error rate of 43% against jQuery. Let's try harder:
Optimized jQuery DOM building II (72ms*):
"make": function(){
for(var i = 0; i<250; i++){
document.body.appendChild(
$(
"<ul id='setid" + i + "' class='fromcode'>"+
"<li>one</li>"+
"<li>two</li>"+
"<li>three</li>"+
"</ul>"
)[0]
);
}
return $("ul.fromcode").length;
}
If this was a presentation I would have an LOLCat saying "jQuery rulez" right now. But luckily this isn't and I'll try to reason scientifically about this.
jQuery is NOT faster in this example. Don't believe the numbers you see. They have been meaningless all along. The reason for that is simple: While initially it looked like we were performing the same test with jQuery as we were with RightJS, we never actually did! The jQuery example, from the beginning, was creating DOM elements from HTML strings, while RightJS was wrapping the document.createElement API. This is not the same thing and you cannot learn anything from comparing apples to oranges.
The truth as far as this test case is concerned? Well, jQuery simply does not have a document.createElement wrapper. Thus you cannot compare it to implementations that do. And why should you? DOM building like this is largely useless, given excellent alternatives such as John' Micro -Templating engine.
Just to show how useless this test was from the beginning, here is my not so paradox implementation that outperforms the pure DOM test:
Testing Pure DOM building (37ms*):
"make": function(){
for(var
body = document.body,
ul = document.createElement("ul"),
li = document.createElement("li"),
i = 0,
fromcode;
i < 250; ++i
){
fromcode = ul.cloneNode(true);
fromcode.id = "setid" + i;
fromcode.className = "fromcode";
fromcode.appendChild(li.cloneNode(true)).appendChild(document.createTextNode("one"));
fromcode.appendChild(li.cloneNode(true)).appendChild(document.createTextNode("two"));
fromcode.appendChild(li.cloneNode(true)).appendChild(document.createTextNode("three"));
body.appendChild(fromcode);
};
return utility.getSimple.call(body, "ul.fromcode").length;
}
Optimized jQuery DOM building III (36ms*):
"make": function(){
var elements = '<div>';
for(var i = 0; i<250; i++){
elements = elements+
"<ul id='setid" + i + "' class='fromcode'>"+
"<li>one</li>"+
"<li>two</li>"+
"<li>three</li>"+
"</ul>";
}
$(elements+'</div>')
.children()
.each(function() {
document.body.appendChild(this);
});
return $("ul.fromcode").length;
}
As you can see, the cargo cult method is quite powerful : ).
Anyway, I don't want to discourage the development of RightJS in any way. I think it's awesome that there are libraries that are trying to compete with jQuery.
It is really hard to do meaningful performance testing and infinitely easy for some random punk like me to come along and point out all the flaws. To me, even trying to do a general purpose performance test against 6 (!) implementations, that is pure bravery. So in case you decide to do something similar, just admit the odds you are up against and people will be very forgiving and engaged.
Comments, hate mail & suggestions are welcome!
-- Felix Geisendörfer aka the_undefined
* Results not typical - Some recent version of Firefox on my Laptop, picking random samples from runs that looked good!
William: thanks, fixed ..
Hi Felix,
The Upgra ..
Maybe you can write so ..
This is very nice plug ..
Nice article!
Somet ..
That's fine ognian, th ..
Really nice work! I wa ..
Thanks didn't quite un ..