●●●●●●●●

●●●●●●

GRUB2 and Troubleshooting OL7 Boot Problems

Author: Bob W Wang (bob.w.wang@oracle.com) Updated on Sep 2017

Agenda1 GRUB2 New Features2 OL7 Booting Process 3 GRUB2 Configuration Files4 GRUB2 Environment Variables5 Explaining menuentry6 Demonstrating GRUB2 Related Commands7 Fixing GRUB2 Issues8 OL7 Troubleshooting Modes for Boot Problems

GRUB2 is the new Linux bootloader. GRUB2 stands for GRand Unified Bootloader version 2. As GRUB was not maintained for some time and lacked some critical features like GPT management needed to handle disks bigger than 2.4TB it was decided to start a new version from scratch with modularity in mind.

1 GRUB2 New Featuresability to boot on various file systems xfs ext4 ntfs hfs+ raid etcgzip files decompression on the flymanagement of all disk geometriessupport for GPT  GUID Partition Tables and MBR  Master Boot Recordportability with different architectures BIOS EFI Coreboot etcability to load modules at execution time.

2 OL7 Booting Process

●

3 GRUB2 Configuration Files

The GRUB2 configuration is spread over several files: /boot/grub2/grub.cfg: this file contains the final GRUB2 configuration do

●

●●

●●

●

●

not edit it directly!/etc/grub2.cfg: this is a symbolic link to the /boot/grub2/grub.cfg file/etc/default/grub: this file contains the list of the GRUB2 variables the values of the environment variables can be edited/etc/sysconfig/grub: this is a symbolic link to the /etc/default/grub file/etc/grub.d: this directory contains all the individual files internally used by GRUB2.

4 GRUB2 Environment Variables

GRUB_TIMEOUT: The time in seconds after the menu is displayed to boot the default entry unless a key is pressed. The default is 5. Set to 0 to boot immediately without displaying the menu or to -1 to wait indefinitely.

GRUB_DISTRIBUTOR: Set by distributors of GRUB and is used to generate more informative menu entry titles. The example evaluates to "Oracle Linux Server".

●

●

●

●

●

GRUB_DEFAULT: The default menu entry to boot. A value of 0 boots the first menuentry. A value of 1 boots the second menuentry. A value of saved instructs GRUB2 to load the last successfully loaded operating system.

A GRUB_DEFAULT value of saved also allows you to use the grub2-set-default and grub2-reboot commands to specify the default entry. These two commands are described as follows: grub2-set-default: Sets the default entry for all subsequent reboots grub2-reboot: Sets the default entry for the next reboot only

For example with GRUB_DEFAULT=saved the following command sets the default entry for all subsequent reboots to the second menuentry:

# grub2-set-default 1

GRUB_DISABLE_SUBMENU: By default the grub2-mkconfig command generates a top-level menu entry for the kernel with highest version number and puts all other found kernels or alternative menu entries for recovery mode in a submenu. Setting GRUB_DISABLE_SUBMENU=true disables this.

GRUB_TERMINAL_OUTPUT: The terminal output device. When specifying multiple devices separate the valid terminal output names with spaces.

GRUB_CMDLINE_LINUX: Kernel boot parameters. 

GRUB_DISABLE_RECOVERY: By default two menu entries are generated for each Linux kernel: one default entry and one entry for recovery mode. Setting GRUB_DISABLE_RECOVERY="true" disables this.

To better understand some of the environment variables here are the standard display with  GRUB_DISABLE_RECOVERY="true" and GRUB_DISABLE_SUBMENU=true :

If GRUB_DISABLE_RECOVERY=“false” here is the new display:

  Each kernel line gets an associated line in recovery mode.

If GRUB_DISABLE_RECOVERY is now set to “true” like in the initial standard display and GRUB_DISABLE_SUBMENU is set to false here is the new display:

 

If the first entry is selected Oracle Linux Server the system boots. If

the second option is chosen the standard menu is shown with an additional line at the bottom to go back to the first menu with the Esc key:

5 Explaining menuentry

The GRUB 2 configuration file /boot/grub2/grub.cfg contains menuentry stanzas which represent an installed Linux kernel. Each stanza begins with the menuentry keyword with options. Each menuentry is also a single boot menu entry in the GRUB 2 menu. The associated block of code is enclosed in curly brackets { }.

●○

●○

●○○

The stanza includes a linux16 directive followed by the path to the kernel and an initrd16 directive followed by the path to the initramfs image. The linux16 directive specifies the kernel version number to be booted as well as kernel boot parameters. A separate /boot partition was created therefore the path to the kernel as well as to the initramfs image are relative to /boot.

The initrd16 directive must point to the location of the initramfs file corresponding to the same kernel version. In other words the kernel as given on the linux16 /vmlinuz- line must match the version number of the initramfs image given on the initrd16 /initramfs-[kernel_version].img line of each stanza.

  6 Demonstrating GRUB2 Related Commands

Add/remove entriesUse yum or rpm to add or remove installed kernels - Don t go toying around manually on the exam - You will mess things up and cost yourself time.

   Show kernel that will load at boot: grub2-editenv list

Lists default kernel selection

    Show all kernel boot options

grep "^menuentry" /boot/grub2/grub.cfg | cut -d "'" -f2cat /etc/grub2.cfg | awk -F\' '$1=="menuentry " {print i++ " : " $2}'

●○

◆

○◆

●○

◆◆◆

○◆◆

◆

●○

◆

◆

◆◆◆

Change Default Kernel: grub2-set-defaultEX: grub2-set-default 'Oracle Linux Server 7.2, with Linux 3.10.0-514.10.2.el7.x86_64'

This will select that particular kernel - if it exists

EX: grub2-set-default 0This will make the default kernel the first entry in the bootloader arrayed numbering .

  GRUB2 PW Protection

Edit /etc/grub.d/01_users and add the following lines:set superusers='admin'password admin pwd123

Supplement admin with whatever username you want and pwd123 with their password

Edit /etc/grub.d/10_linux to tell GRUB what entries to PW protectBy default each entry will have a --unrestricted directiveReplace --unrestricted with --users user without the quote and specify which user by name you created. This will allow only those users to access that menu option.If you do not make user of the --unrestricted directive or a --user directive the system will assume ONLY SUPER USER.

Encrypting PWs in GRUB2grub2-mkpasswd-pbkdf2

it will prompt you for a password and it will output it in encrypted form that is compatible with GRUB2.Copy the hash and paste it in the template file where you configured users 01_users file found in the /etc/grub.d/ folder.The format is really messy but it looks something like:

set superusers='admin'password_pbkdf2 admin grub.pbkdf2.sha512.10000.19074739ED80F115963D984BDCB35AA671C24325755377C3E9B014D862DA6ACC77BC110EED41822800A87FD3700C037320E51E9326188D53247EC0722DDF15FC.C56EC0738911AD86CEA55546139FEBC366A393DF9785A8F44D3E51BF09DB980BAFEF85281CBB

◆

◆◆

◆

●○

◆◆◆◆

●○

◆

●○

◆◆◆

◆◆

◆

●○○

○○

◆

C56778D8B19DC94833EA8342F7D73E3A1AA30B205091F1015A85

Append that entry to the bottom and it will check for hashes.If you are in terminal you may install gpm # yum install gpm to enable mouse support in order to copy and paste the encrypted password.NOTE: IF YOU MESS THIS UP YOUR SYSTEM PROBABLY WONT BOOT.

Resettings GRUB2Sometimes you gotta restart shit

rm /etc/grub.d/* delete all grub.d scriptsrm /etc/sysconfig/grub remove all system configurationyum reinstall grub2-tools reinstall the entire packagegrub2-mkconfig -o /boot/grub2/grub.cfg reset the entire configuration

Reinstalling GRUB2grub2-install <device>

This will reinstall and restore any corrupted files in the /boot/grub2/ directory - If files are missing they will be recreated.

Finalizing changes:IF YOU CHANGE ANYTHING IN GRUB YOU NEED TO RUN grub2-mkconfig OR THE CHANGES WILL NOT STICK.

Two ways to do this:grub2-mkconfig -o /boot/grub2/grub.cfg  

grub2-mkconfig > /boot/grub2/grub.cfg # don t forget ">" otherwise it will go to the standard output

USE ONLY IF ON TRADITIONAL BIOSgrub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg

USE ONLY IF ON UEFI BIOS 

Telling GRUB to boot into different targetsMove into the append/edit mode with the e key at boot menuIF USING x86-64 -- append boot directive to the end of the linux16 lineIF USING UEFI -- append boot directive to the end of the linuxefi linePossible Appends

systemd.unit=[systemctl.target] 

So as you can see we can do a lot with GRUB2 - In many ways I think it's much more convoluted than GRUB1.

7 Fixing GRUB2 IssuesIf you install a second operating system or if your GRUB configuration file gets corrupted due to human error there are ways you can get your system back on

●

its feet and be able to boot again. In the initial screen press c to get a GRUB command line remember that you can also press e to edit the default boot options and use help to bring the available commands in the GRUB prompt:

Fix Grub Configuration Issues in LinuxWe will focus on ls which will list the installed devices and filesystems and we will examine what it finds. In the image below we can see that there are 4 hard drives hd0 through hd3 . Only hd0 seems to have been partitioned as evidenced by msdos1 and msdos2 where 1 and 2 are the partition numbers and msdos is the partitioning scheme . Let s now examine the first partition on hd0  msdos1 to see if we can find GRUB there. This approach will allow us to boot Linux and there use other high level tools to repair the configuration file or reinstall GRUB altogether if it is needed:

# ls (hd0,msdos1)/

As we can see in the highlighted area we found the grub2 directory in this partition:

●

●

●

Find Grub ConfigurationOnce we are sure that GRUB resides in hd0 msdos1 let s tell GRUB where to find its configuration file and then instruct it to attempt to launch its menu:

set prefix=(hd0,msdos1)/grub2set root=(hd0,msdos1)insmod normalnormal

Find and Launch Grub Menu

Then in the GRUB menu choose an entry and press Enter to boot using it. Once the system has booted you can issue the grub2-install /dev/sdX command change sdX with the device you want to install GRUB on . The boot information

will then be updated and all related files be restored.

# grub2-install /dev/sdX

More complex scenarios GRUB 2 boot problems can leave the system in several states. The user may see one of the following displays on the monitor when a boot fails. The display provides the first indication of what might be causing the failure to boot. Here are the failure prompts and displays and the possible cause of each:

e.

f.

g.

h.

i.

j.

k.

grub> prompt: GRUB 2 loaded modules but was unable to find the grub.cfg file.

grub rescue> prompt: GRUB 2 failed to find its grub folder or failed to load the normal module.

grub>: - The grub prompt on a blank screen. GRUB 2 has found the boot information but has been either unable to locate or unable to use an existing GRUB 2 configuration file usually grub.cfg .grub rescue>: - The rescue mode. GRUB 2 is unable to find the grub folder or its contents are missing/corrupted. The grub folder contains the GRUB 2 menu modules and stored environmental data.GRUB - a single word at the top left of the screen with no prompt and no cursor. GRUB has failed to find even the most basic information usually contained in the MBR or boot sector.

Busybox or Initramfs: GRUB 2 began the boot process but there was a problem passing control to the operating system. Possible causes include an incorrect UUID or root= designation in the 'linux' line or a corrupted kernel.Frozen splash screen blinking cursor with no grub> or grub rescue prompt. Possible video issues with the kernel. While these failures are not of GRUB 2's making it may still be able to help. GRUB 2 allows pre-boot editing of its menu and the user may restore functionality by adding and/or removing kernel options in a menuentry before booting.

Each of the GRUB 2 failure modes can normally be corrected either from the GRUB 2 terminal or by using an Oracle Linux Installation CD. Additionally there are compatible 3rd party bootable "rescue" CD/USB options which may also work. If using an Oracle Linux Installation CD it is recommended but not always necessary to use the same version CD as the system you are trying to repair. This ensures compatibility of any modules and configuration files that may be loaded while attempting to repair the system.

Other more complex scenarios are documented along with their suggested fixes in the "Ubuntu GRUB2 Troubleshooting guide: https://help.ubuntu.com/community/Grub2/Troubleshooting". The concepts explained there are valid for other distributions as well.

8 OL7 Troubleshooting Modes for Boot Problems

Booting into Rescue or Emergency Targets

To boot directly into rescue target add systemd.unit=rescue.target or just 1 to the kernel command line. This target is useful if the problem occurs somewhere after the basic system is brought up during the starting of "normal" services. If this is the case you should be able to disable the bad service from here. If the rescue target will not boot either the more minimal emergency target might.

To boot directly into emergency shell add systemd.unit=emergency.target or emergency to the kernel command line. Note that in the emergency shell you will have to remount the root filesystem read-write by yourself before editing any files:

mount -o remount,rw /

Common issues that can be resolved in the emergency shell are bad lines in /etc/fstab. After fixing /etc/fstab run systemctl daemon-reload to let systemd refresh its view of it.

If not even the emergency target works you can boot directly into a shell with init=/bin/sh. This may be necessary in case systemd itself or some libraries it depends on are damaged by filesystem corruption. You may need to reinstall working versions of the affected packages. If init=/bin/sh does not work you must boot from another medium.

Early Debug Shell

You can enable shell access to be available very early in the startup process to fall back on and diagnose systemd related boot up issues with various systemctl commands. Before enable debug shell you may need to edit file /etc/default/grub to remove "rhgb quiet" from kernel command line and then run 'grub2-mkconfig > /boot/grub2/grub.cfg' and then 'systemctl set-default  multi-user.target'

Enable it using:

systemctl enable debug-shell.service

or by specifying

systemd.debug-shell=1

on the kernel command line.

Once enabled the next time you boot you will be able to switch to tty9 using CTRL+ALT+F9 and have a root shell there available from an early point in the booting process. You can use the shell for checking the status of services reading logs looking for stuck jobs with systemctl list-jobs etc.

Warning: Use this shell only for debugging! Do not forget to disable systemd-debug-shell.service after you've finished debugging your boot problems. Leaving the root shell always available would be a security risk.

Recovering a Lost Root Password

rd.break :~ "Ramdisk break"

mount -o remount,rw /sysrootchroot /sysrootpasswdtouch /.autorelabel    #(or you can disable selinux)Then press Crtl-D, Ctrl-D  to exit and reboot.

Additional Resources:Thanks to the following internet sources which made this document more elaborated.

–

–

––

●––

●–

–

http://docs.oracle.com/cd/E52668_01/E53499/html/section-b4s-kzd-kn.htmlhttps://opensourceblogging.wordpress.com/2017/02/15/rhel-7-booting-process/https://help.ubuntu.com/community/Grub2/Troubleshooting/https://freedesktop.org/wiki/Software/systemd/Debugging/

Extract an initramfs Image on OL5 and OL6How to Extract an initramfs Image and Edit/View it Doc ID 1671276.1How to Rebuild the Initial Ramdisk Image in Oracle Linux 5 or Oracle Linux 6 Doc ID 1573625.1Extract an initramfs Image on OL7 via skipcpio a built-in tool from dracut/usr/lib/dracut/skipcpio initramfs-3.10.0-229.el7.x86_64.img | zcat | cpio -ivd

http://thegeekdiary.com/centos-rhel-7-how-to-extract-initramfs-image-and-editview-it/