AltovaXML claims to be 3 times faster than SaxonJ 8.7 ???

This weekend I read about a claim made by Altova that their XSLT 2.0 processor was 3 times faster than SaxonJ 8.7.

In particular, what they claim here is the following:

AltovaXML is 3X faster than the competition.

The AltovaXML processor is more than three times faster than Saxon. In a test of a selection of 1,127 test files from the official W3C test suite for XSLT 2.0, the same files AltovaXML processed in 6 minutes 42 seconds took Saxon a full 22 minutes to process. (Further details about test methods and metrics can be found below.) This makes an unbelievable difference in developer productivity as well as application optimization.

Michael Kay responds to these statements in his blog entry "Altova Mudslinging":

At least they had the decency to describe how they measured it – by running a thousand or so conformance tests from the W3C test suite using a batch script in which each transformation was run individually from the command line. Any reasonably competent user knows that that’s a hopelessly inefficient way of running Java programs, because the initialisation costs for loading the Java VM a thousand times far exceed the time spent actually doing useful work. But of course, Altova know that a lot of people will only read the headlines.

On the Questions and Answers page of Saxon Michael Kay provides his results comparing the performance of AltovaXML to Saxon:

With a properly written Java test driver, Saxon takes 2 min 50 sec to run the full suite of 4864 tests, compared with the 6 min 42 sec that Altova took to run a subset of 1127 tests using their own processor. That’s 28.6 tests per second for Saxon, 2.8 tests per second for Altova. The Saxon timing includes the expensive operation of comparing the results with reference results, I don’t know if that’s included in Altova’s figures. Either way, if they’re choosing to compete on performance, then they’ve got a long way to go.

Anyway, if they were running the W3C conformance test suite, then where are their results? And why were they only running a subset of the tests? Saxon’s results are here

So, it turns out that Saxon is about 10 times faster than Altova on the full W3C conformance tests.

Here comes the fun part.

Given the times reported the 4000 or so W3C conformance tests must be all small to medium-sized.

As the beginning of wisdom is to call things by their right names, I ran some of my favourite transformations (from my FXSL library and  Sudoku solver) on my 3-year old home (DELL Pentium IV 3GHz, 2MB RAM Windows XP SP2) PC.

The results are below:

The "25min +" and "35min +" timings above actually mean that this was the time I was waiting for the AltovaXML2007-run – transformation to finish, before I decided to kill the process.

While everyone is encouraged to perform their own tests and the results of the tests may vary, it seems that the above results are quite significant.

To summarize the results:

For my favourite tasks AltovaXML2007 performs about 10 to hundreds of times slower  than SaxonJ 8.7.3

UPDATE: Another person has run his performance tests. The results:

"Saxon finishes this stylesheet in less then 3 minutes and consumes ~130 MB of RAM. For the most of the time Saxon utilizes only 70% of CPU — disc is probably slower then Saxon ;-)
I tried the same stylesheet and data with Altova2007. After 30 minutes only 400 files from 5000 vere generated and 950 MB of RAM was vasted, CPU utilization for Altova process was over 90%. I decided to interrupt test here. For this stylesheet Altova was at least 100 times slower than Saxon. Moreover Altova eat more and more memory with each outputed file, so there is probably some memory leak in this product."

As some proverbs go

Cows might fly, but they are most unlikely birds
.

Haste makes waste.

and

There is plenty of sound in an empty barrel.

Advertisements
This entry was posted in Computers and Internet. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s