IssueDb4UnderXenDomU < LCGTier2

Tags: view all tags

<!-- keep this as a security measure:
   * Set ALLOWTOPICCHANGE = Main.TWikiAdminGroup,Main.LCGAdminGroup
   * Set ALLOWTOPICRENAME = Main.TWikiAdminGroup,Main.LCGAdminGroup
   #uncomment this if you want the page only be viewable by the internal people
   #* Set ALLOWTOPICVIEW = Main.TWikiAdminGroup,Main.LCGAdminGroup
-->

---+!! %TOPIC%

%TOC%

---++ Symptoms
Summary: %FORMFIELD{"Symptom summary"}%

A program (==rpm==, OpenLDAP's ==slapd==) aborts with an error message like:
<verbatim>
    rpmdb: unable to initialize mutex: Function not implemented
    ...
    error: db4 error(38) from dbenv->open: Function not implemented
</verbatim>
The exact db4 function in which the problem occurs may vary, but the
error code 38 and the message 
=unable to initialize mutex: Function not implemented= are constant.

The problem we are facing (and its variations, depending on the
workarounds that one can try) is quite accurately described in
http://www.openldap.org/lists/openldap-software/200603/msg00199.html


---++ Occurrences

Reproduceable on any Xen DomU running RHEL4-derived systems (CentOS4, SL4, SLC4).


---++ Observations
<!--
   #collect here the information which may help to better understand the state of the system or services, e.g.
   #log excerpts, strace output, etc.
   #this also may help to identify the problem if similar conditions arise again
-->

On the site-BDII, the problem was fixed by using the "ldbm" backend
and avoiding Berkeley DB4 completely, see IssueBdiiNotStartingOnXenDomU

We cannot do this for Phedex, since it uses a CMS-built version of
apt+rpm+db4, so we cannot rebuild that with different flags.  


---++ Solution or Workaround

I wasn't able to find much information by googling: all the related
pages date back to 2005/2006 (when CentOS4/RHEL4 was current), and the
links to software packages or patches are often broken.  Some suggest
fixing the problem by recompiling the libc with a special gcc option;
finally I found the exact patch here:

  http://lists.centos.org/pipermail/centos-devel/2005-December/000175.html

Rebuilding the libc with the patch applied solved the issue,
apparently. (Patch and new glibc RPM attached.)  Caveat: installation
of the glibc RPM directly from the !DomU command-line prompt led to a
failed upgrade and an unusable system.  I had to:
   1 stop the !DomU
   2 mount its root filesystem in the !Dom0:
   3 =chroot= into it (bind-mount =/dev= and 
      =/proc= for RPM to work properly):
      <verbatim>
        mount /dev/vm-root-filesys /mnt
        mount --bind /dev /mnt/dev
        mount --bind /proc /mnt/proc
        chroot /mnt rpm -Uvh glibc-2.3.4-2.41.xenu.rpm
        umount /mnt/proc
        umount /mnt/dev
        umount /mnt
      </verbatim>

Note that upgrading the glibc RPM can lead to problems when bulk
upgrades of the system RPMs are performed (=yum update=), as that can
replace the !DomU-specific =glibc= with a new version.  A workaround
to this is described in:
  http://marc.info/?l=xen-users&m=111761896306738&w=2

Also note that the same issue is known to the gLite/WLCG upstream, and tracked in 
[[https://savannah.cern.ch/bugs/?func=detailitem&item_id=42475 bug #42475 at CERN Savannah]]
where other workarounds are suggested, none of which worked here.

-- Main.RiccardoMurri - 05 Nov 2008

IssueForm
Affected Service	BDII, Phedex/cmsvobox
Symptom summary	program fails with error: "unable to initialize mutex: Function not implemented"
Reason Understood	yes
Solution Exists	yes

Attachments

Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
rpm	glibc-2.3.4-2.41.xenU.i386.rpm	r1	manage	4376.3 K	2009-01-28 - 09:26	RiccardoMurri	glibc RPM rebuilt with Xen DomU compatibility options
diff	glibc.spec.diff	r1	manage	0.7 K	2009-01-28 - 09:21	RiccardoMurri	patch to glibc.spec to enable the DomU-compatibility option