Oracle Linux Boot Failure: A GLIBC Version Mismatch / GLIBC_2.33' not found

In this blog post I share a real production incident and its resolution. While the issue was severe, proper troubleshooting methodology and rescue media made recovery possible without data loss.

An Oracle Linux 8.10 production server suddenly became unresponsive. The system would boot but freeze indefinitely at the graphical login screen, showing only the Oracle Linux logo with a loading spinner that never completed.

No amount of waiting helped. The system was completely inaccessible through normal means. SSH connections timed out, and the console remained locked at the authentication screen.

Our initial discovery was trough the emergency shell access. I mean, to diagnose the issue, we bypassed the normal boot process using GRUB emergency parameters:

# At GRUB menu, press 'e' on the kernel line

# Add to the linux/linuxefi line:

rw init=/bin/bash

 Once we gained shell access, the true nature of the problem became apparent immediately:

[root@myprodddb01 /]# rpm --version

rpm: /lib64/libc.so.6: version 'GLIBC_2.33' not found (required by /lib64/libcrypto.so.1.1)

[root@myprodddb01 /]# yum

yum: /lib64/libc.so.6: version 'GLIBC_2.33' not found (required by /lib64/libcrypto.so.1.1)

[root@myprodddb01 /]# dnf

dnf: /lib64/libc.so.6: version 'GLIBC_2.33' not found (required by /lib64/libcrypto.so.1.1)

Every foundational system command was broken. This was not a simple misconfiguration, this was a fundamental system library corruption.

GLIBC_2.33' not found was the error. Well, lets first take a look at the GLIBC. 

GLIBC (GNU C Library) is the core system library that nearly every Linux program depends on. It provides essential functions for:

• Memory management
• System calls
• String operations
• File I/O
• Network operations

Without a working GLIBC, the system cannot function.

That's enough for the giving the background..

So, Oracle Linux 8.10 ships with GLIBC 2.28. However, our system's binaries were looking for GLIBC 2.33 and 2.34, which are part of Oracle Linux 9 (based on RHEL 9).

[root@myprodddb01 /]# /lib64/libc.so.6 --version GNU C Library (GNU libc) stable release version 2.28

The library version was correct (2.28), but the programs themselves (rpm, yum, ping, dnf) were looking for libraries from Oracle Linux 9.

How did this happen? In our case this is not certain yet, but we have some clues and here is the list of possible causes for such a catastrophic situation:

• Mixed package repositories - OL9 repos were accidentally added to an OL8 system
• Manual binary replacement - Someone copied binaries from an OL9 system
• Failed upgrade attempt - An attempt to upgrade from OL8 to OL9 went wrong
• Incorrect package installation - Installing .el9 RPMs on an .el8 system

Anyways; to fix the system, we needed rpm to reinstall packages. But! rpm itself was broken because it needed GLIBC 2.33. We couldn't use yum or dnf for the same reason. Even basic networking tools like ping were non-functional. The broken system could not fix itself.

The solution was in rescue mode recovery.

We booted from Oracle Linux 8 ISO, and entered Rescue Environment. The rescue environment automatically detected and mounted our system to /mnt/sysimage. The rescue environment provided working tools with correct GLIBC 2.28.

sh-4.4# /lib64/libc.so.6 --version GNU C Library (GNU libc) stable release version 2.28 Copyright (C) 2018 Free Software Foundation, Inc.

sh-4.4# lsblk

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT

sda 8:0 0 80G 0 disk

├─sda1 8:1 0 1G 0 part

├─sda2 8:2 0 1G 0 part

└─sda3 8:3 0 70G 0 part

└─root--vg-root--lv 253:0 0 70G 0 lvm /

sh-4.4# mount /dev/mapper/root--vg-root--lv /mnt/sysroot 

We then identified the corrupted packages using the rescue environment's working rpm: ,

sh-4.4# rpm --root=/mnt/sysroot -qa | grep "\.el9"

This command listed all Oracle Linux 9 packages installed on our OL8 system.

And, we copied the GLIBC 23 libraries (and libcrypto) from rescue to our broken system.

cp -fv /lib64/libc-2.28.so /mnt/sysroot/lib64/

cp -fv /lib64/libc.so.6 /mnt/sysroot/lib64/

cp -fv /lib64/libm*.so* /mnt/sysroot/lib64/

cp -fv /lib64/libpthread*.so* /mnt/sysroot/lib64/

 cp /lib64/libcrypto.so.1.1.1k /mnt/sysroot/lib64/

After these actions, we chrooted into the system to verify and tested the foundational commands, they were all run successfully

chroot /mnt/sysroot

rpm --version        

ping -c 2 8.8.8.8   

yum --version 

We verified the GLIBC version.

rpm -q glibc

-- Expected: glibc-2.28-251.0.1.el8.x86_64

We rebooted and tested.

exit

sync

reboot

This fixed the issue but during the emergency shell access, we also reset the root password:

In emergency mode (init=/bin/bash);

mount -o remount,rw /

passwd root

# Enter new password

sync

reboot

Well, we fixed the issue and learned a few important things.

• Never install packages from OL9 (or any EL9) on an OL8 system, even if they seem compatible. ---Major version boundaries exist for a reason.
• Always verify yum repository configuration.
• Before installing RPMs, always check the version.
• Keep Rescue Media Ready ( to save time and be ready).
• Take snapshots before: System upgrades, Repository changes, Mass package installations.
• Monitor Package Origins : Set up alerts for package installations from unexpected repositories.