Error While Loading Shared Libraries, Cannot Open Shared Object File

In the “I wish the Internet had an actual correct answer” category comes a question from a Windows colleague trying to build software on Linux. He asks “I’m trying to do some web performance testing and I compiled weighttp and the libev libraries, which worked fine, but when I try to run the program it gives me the following error.”

weighttp: error while loading shared libraries: libev.so.4: cannot open shared object file: No such file or directory

“I checked /usr/local/lib and the files are there. Do you have a suggestion?”

Ah yes, a classic problem when building software. The problem here is that libev installed itself into /usr/local/lib:

$ ls -l /usr/local/lib/libev*
-rw-r--r--. 1 root root 435770 Feb 22 15:20 /usr/local/lib/libev.a
-rwxr-xr-x. 1 root root 926 Feb 22 15:20 /usr/local/lib/libev.la
lrwxrwxrwx. 1 root root 14 Feb 22 15:20 /usr/local/lib/libev.so -> libev.so.4.0.0
lrwxrwxrwx. 1 root root 14 Feb 22 15:20 /usr/local/lib/libev.so.4 -> libev.so.4.0.0
-rwxr-xr-x. 1 root root 174059 Feb 22 15:20 /usr/local/lib/libev.so.4.0.0

…but the dynamic linker doesn’t know where they are, because it’s never heard of /usr/local/lib. /usr/local is a traditional place for add-on software to install itself, so it doesn’t interfere with the system libraries. If you’re coming from a Windows background the .so files are essentially equal to DLLs, and load when you execute a program that depends on them. Programs that use dynamic libraries have several advantages, in that they’re smaller, and the libraries can be updated without having to recompile all the programs that depend on them. So if there’s a security problem with libev you can just patch libev, and not have to rebuild everything that uses that library.

You can see what libraries a program is dynamically linked to with the ‘ldd’ command:

$ ldd /usr/local/bin/weighttp
        linux-vdso.so.1 =>  (0x00007fff251ff000)
        libev.so.4 => not found
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f8f1cc1e000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f8f1c88b000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f8f1ce49000)

That confirms we’re just dealing with the new library, and not some other problem. Cool.

Anyhow, there are five fixes that come to mind, and I’ll group them into “terrible ideas” and “decent ideas.” Of course, terrible and decent are my opinion, and your situation may dictate a different conclusion, so I’ll add some commentary. If you’re looking for the quickest way out skip to #5.

Suboptimal Fixes/Terrible Ideas[0]

1. Install the libraries to /usr/lib instead of /usr/local/lib. I really don’t like this because you’re installing add-on software into a system directory, and you might introduce a conflict, overwrite a pre-existing library, and/or destabilize other parts of the system. People complain about Windows being unstable, and this sort of thing is exactly why it gets unstable – installers overwrite system DLLs with their own all the time. Recent versions of Windows go to a lot of trouble to keep installers from doing this. If you’re bored look up the WinSxS subsystem, which is what Microsoft built to deal with the problem. Linux doesn’t have something that deals with the problem, it just assumes that you know what you’re doing. Such is the power of UNIX. Here’s a rope, try not to hang yourself.

When you’re building open-source software under a UNIX OS you often use the “configure” command. You can change where the software installs by using the “prefix” flag:

$ ./configure –prefix=/usr

This will also install all the binaries, header files, etc. into the system directories. If you were one of my system administrators I’d get on your case for doing this and make you go back and do it right. This isn’t the right way. Don’t do it this way. The right ways are simple, do those instead.

2. Make a symbolic link from /lib to the files in /usr/local/lib. This is less intrusive than installing everything to /usr, but still a bad idea, for most of the same reasons as #1. If you’re the kind of person that likes to smoke while refueling vehicles go ahead and try:

$ sudo ln -s /usr/local/lib/libev.so.4 /usr/lib/libev.so.4
$ sudo ldconfig

Obviously substitute the correct file name for libev.so.4. The ‘ldconfig’ command updates the system’s dynamic library cache so it sees the changed libraries.

Still, don’t do this. Bad sysadmin.

3. Copy the files from /usr/local/lib into /usr/lib. This is even more of a bad idea than #2 because now you’ve got two sets of libraries, and if you ever have to patch or upgrade it you’ll probably forget about one set. Oops. I hope you weren’t patching because of a security problem!

$ sudo cp /usr/local/lib/libev.* /usr/lib
$ sudo ldconfig

With all these ideas you’re also very likely to run afoul of the 32-bit vs. 64-bit thing with newer OSes, where 32-bit libraries go in /usr/lib and 64-bit libraries go in /usr/lib64. It might work, it might not work, but the better ways are a heck of a lot simpler.

Better Idea

4. Set the environment variable LD_LIBRARY_PATH to point to /usr/local/lib. If you are new to UNIX/Linux and don’t know what shell you’re running use the ‘ps’ command to see the processes you’re running:

$ ps
  PID TTY          TIME CMD
 7796 pts/4    00:00:00 tcsh
10048 pts/4    00:00:00 ps

Hey look, I’m running tcsh, so my fix would be:

setenv LD_LIBRARY_PATH /usr/local/lib

and I can put that command into ~/.cshrc so it executes when I log in.

If ps tells you you’re running bash:

$ ps
  PID TTY          TIME CMD
 7796 pts/4    00:00:00 bash
10048 pts/4    00:00:00 ps

the command is:

export LD_LIBRARY_PATH="/usr/local/lib"

and you can put that into your ~/.bashrc so it’s always there.

Changing the environment is a per-user thing, so other users on the system will need to do this, or you’d need to put the fixes in /etc/bashrc or /etc/csh.cshrc, the system-wide login scripts. This fix is nice, though, if you don’t have root-level privileges on a system, and/or want to install things into your own home directory.

Absolutely Simple, Best Fix Ever[2]

5. If you have root privileges on this Linux box, why not just ask the dynamic linker to check /usr/local/lib, too? Edit the file /etc/ld.so.conf and add “/usr/local/lib” on its own line at the bottom[3]. DO NOT REMOVE THINGS FROM THIS FILE. When you’re done it might look something like:

$ cat /etc/ld.so.conf
include ld.so.conf.d/*.conf
/usr/local/lib

or it might look completely different, with the exception of the last line. Run ldconfig to tell it to update the cache:

$ sudo ldconfig

You can check your work with:

$ ldconfig -p | grep local

You should see the stuff in /usr/local/lib show up now, and your binary works. Boy, that was easy, and you didn’t make more work for yourself down the road, destabilize the system, overwrite system files, or leave security vulnerabilities lying around. Cool.

Good luck. As always, if you see a problem here let me know in the comments. If you have other questions you’d like me to answer in a blog post you should email them or tweet them at me! My contact information is on the top-right at lonesysadmin.net. While you’re there why don’t you subscribe to my feed, too?

——–

[0] So why did I list the bad ideas? Because you’ll see every one suggested as a fix by people pretending to be experts[1]. It is my hope that you’ll understand why these are bad ideas, why they might cause you more work in the long run, and can make an informed decision.

[1] I am not an expert, I’m a guy that’s done a lot of this stuff, and might be able to answer your question. If someone calls themselves an expert run away.

[2] Not guaranteed to be simple or best. I like it, though.

[3] If you’re stymied by editing on the command line try “nano.” It’s a simple editor and is often installed. Another option is ‘vi’ but it’s cryptic, and you can use Google to find a tutorial on it if you need one. The nice thing about vi is that it’s always installed.

Comments on this entry are closed.

  • This works great for a lot of things, but after a while you’ll start to see things hitting the new area first “(or last, depending on where you put it in LD_LIBRARY_PATH) and seeing a different version of the library than it was expecting. Instead, always compile yourself and use “–prefix = /usr” in your configure line ;-) More realistically, make sure the package installs in the right area before you install it, something like “rpm -qlp package.rpm”. Lastly, if it’s some vendor thing that *has* to go in the wrong place, shoehorn it in any way possible.

  • Two things:

    1. It should be noted that shared libraries can also have dependencies on other shared libraries.

    2. If you’re dealing with code you compile and manage yourself, you can specify your own custom library search path by passing the -R flag to the linker, and the program will always search it.

    • Ooh, I forgot about -R. Thanks Chris!

  • While ultimately I agree your solution is the best given that it’s /usr/local/lib we’re talking about, but an alternate solution would be to simply pass in the LD_LIBRARY_PATH via the command line.

    % LD_LIBRARY_PATH=/usr/local/lib /usr/local/bin/weighttp

    This will get this command liked up with the right library and won’t impose it system wide nor be applied for other commands the user might run.

  • You left out the “do-it-the-right-way”-fix for this kind of problem which is: Make a friggin’ package.

    It’s generally a bad idea to circumvent your distributions package manager in any way if not absolutely necessary. Creating a proper package for your distribution is usually not that hard for relatively simple software like weighttp. (I am aware that at least the .deb ways are a bit arcane, but that’s what Google and HOWTOs are for.) Libev should already be packaged for any reasonable distribution, so it can be added as a simple dependency.

    Having a proper package, you gain dependency tracking, file tracking, a proper uninstall procedure, repeatable installs, version tracking and conflict resolution, all for the cost of 10 to 15 minutes upfront work. There is no excuse for not doing a software installation properly. (I admit, it’s easy for me to talk like that since my devbox runs Arch Linux which makes package building trivial and all productions needs are already taken care of by a local debian repository.)

    • I love it — yes, actually building a package fixes the whole conflict problem and everything. You’re absolutely right.