Tuesday, November 30, 2010

The King is dead, long live the King

In MySQL 5.5 we introduced a possibility to use alternative malloc implementations for MySQL. In Solaris we have found mtmalloc to be the optimal malloc implementation. For Linux we've previously found tcmalloc to be the optimal malloc implementation. However recently when working on a new MySQL feature I discovered a case where tcmalloc had a performance regression after running a very tough benchmark for about an hour. Actually I found a similar issue with the standard libc malloc implementation. So it seems that many malloc implementations gets into fragmentation issues when running for an extended period at very high load.

So I decided to contact Mark Callaghan to see if he had seen similar issues. He hadn't, but he pointed me towards an alternative malloc implementation which is jemalloc. It turns out that jemalloc is the malloc implementation used in FreeBSD among other things. I found a downloadable tarball of jemalloc, downloaded it and installed it on my benchmark computers. Given that MySQL already supports any generic malloc implementation it was a simple matter of pointing LD_PRELOAD towards jemalloc instead of towards tcmalloc to make this experiment.

The background is that tcmalloc gave about +5-10% better performance than libc malloc on Linux. Both libc malloc and tcmalloc have had performance regressions in certain situations. So the new results for jemalloc was very exciting. I got +15% compared to libc malloc and so far after using it for about a month I have found no performance regressions using jemalloc.

3 comments:

Didier Spezia said...

Hi,

the drizzle people had a similar issue with tcmalloc last year and they have reverted to libc malloc. See this post from Eric Day (http://oddments.org/?p=67). Otherwise you may be interested by testing nedmalloc available at http://www.nedprod.com/programs/portable/nedmalloc/

Regards,
Didier.

Larry Lowe said...

So how does this work on MySQL 5.5 on Mac OS 10.6.7?

I've compiled jmalloc 2.2.1 but it's not clear how you tell MySQL to utilize it.

Mikael Ronstrom said...

You tell MySQL to use by simply setting a reference to the jemalloc library in LD_PRELOAD before starting mysqld.

LD_PRELOAD will make mysqld bind the malloc automagically to the jemalloc library in this manner.