IssueDb4UnderXenDomU

Symptoms

Summary: program fails with error: "unable to initialize mutex: Function not implemented"

A program (rpm, OpenLDAP's slapd) aborts with an error message like:

    rpmdb: unable to initialize mutex: Function not implemented
    ...
    error: db4 error(38) from dbenv->open: Function not implemented

The exact db4 function in which the problem occurs may vary, but the error code 38 and the message unable to initialize mutex: Function not implemented are constant.

The problem we are facing (and its variations, depending on the workarounds that one can try) is quite accurately described in http://www.openldap.org/lists/openldap-software/200603/msg00199.html

Occurrences

Reproduceable on any Xen DomU running RHEL4-derived systems (CentOS4, SL4, SLC4).

Observations

On the site-BDII, the problem was fixed by using the "ldbm" backend and avoiding Berkeley DB4 completely, see IssueBdiiNotStartingOnXenDomU

We cannot do this for Phedex, since it uses a CMS-built version of apt+rpm+db4, so we cannot rebuild that with different flags.

Solution or Workaround

I wasn't able to find much information by googling: all the related pages date back to 2005/2006 (when CentOS4/RHEL4 was current), and the links to software packages or patches are often broken. Some suggest fixing the problem by recompiling the libc with a special gcc option; finally I found the exact patch here:

http://lists.centos.org/pipermail/centos-devel/2005-December/000175.html

Rebuilding the libc with the patch applied solved the issue, apparently. (Patch and new glibc RPM attached.) Caveat: installation of the glibc RPM directly from the DomU command-line prompt led to a failed upgrade and an unusable system. I had to:

1. stop the DomU
2. mount its root filesystem in the Dom0:
3. chroot into it (bind-mount /dev and /proc for RPM to work properly):

           mount /dev/vm-root-filesys /mnt
           mount --bind /dev /mnt/dev
           mount --bind /proc /mnt/proc
           chroot /mnt rpm -Uvh glibc-2.3.4-2.41.xenu.rpm
           umount /mnt/proc
           umount /mnt/dev
           umount /mnt
         

Note that upgrading the glibc RPM can lead to problems when bulk upgrades of the system RPMs are performed (yum update), as that can replace the DomU-specific glibc with a new version. A workaround to this is described in: http://marc.info/?l=xen-users&m=111761896306738&w=2

Also note that the same issue is known to the gLite/WLCG upstream, and tracked in bug #42475 at CERN Savannah where other workarounds are suggested, none of which worked here.

-- RiccardoMurri - 05 Nov 2008