Fixing Ubuntu “EXT4-fs: Detected aborted journal → Read-only filesystem” & Boot Loops
A complete, field-tested guide to recover a production Ubuntu server stuck in read-only mode, initramfs/emergency shell, or the “A start job is running for /dev/disk/by-uuid/…” hang. Includes quick commands, permanent hardening, FAQs, and a shareable infographic.
Symptoms
Kernel / Filesystem
EXT4-fs error (device dm-0): Detected aborted journal
EXT4-fs (dm-0): Remounting filesystem read-only
Boot hang
A start job is running for /dev/disk/by-uuid/445a60ab-... (no limit)
Emergency shell
You are in emergency mode.
Give root password for maintenance:
Read-only breakage
# touch /root/test
touch: Read-only file system
Pre‑flight checks (safe)
- Identify root device → findmnt -no SOURCE / • lsblk -f
- Confirm read‑only → mount | grep ‘ / ‘ shows
ro,flags - Skim last kernel messages → dmesg | tail -n 100
- Note LVM (common on Ubuntu Server) → device appears like
/dev/dm-0or/dev/mapper/ubuntu--vg-ubuntu--lv
Root causes
- Unclean shutdown / power cut → ext4 journal abort → root remounted
ro. - Heavy write load (containers, logs, backups) → stress + crashes.
/etc/fstabmounts to missing disks (e.g., backup volume) → boot waits forever.- Disk pressure (e.g.,
/var/local/enhance/containers> 100GB) → failures cascade. - Underlying VPS storage instability (SMART often unavailable on virtual disks).
Quick recovery (copy‑paste)
1) If you can SSH
# Check usage & mount state
uname -a; df -h; mount | grep ' / '
# If / is ro: force fsck on next boot then reboot
sudo touch /forcefsck && sudo reboot
2) In initramfs or emergency shell
# Activate LVM and inspect
lvm vgchange -ay && ls -l /dev/mapper
# Find your root (examples)
findmnt -no SOURCE /
lsblk -o NAME,FSTYPE,SIZE,TYPE,MOUNTPOINT
# Repair ext4 (common root is /dev/dm-0)
fsck.ext4 -fy /dev/dm-0
# Reboot
reboot
Tip: Run fsck only when the target filesystem is not mounted read‑write.
3) Boot hang: “A start job is running for … by-uuid/XXXX”
# Mount root rw, then comment the broken fstab entry
mount -o remount,rw /
# Example: mark the missing disk optional
# Replace the actual UUID in the sed pattern below
sed -i '/UUID-REPLACE-ME/s/^/#/' /etc/fstab
# Recommended fstab flags for removable/slow/backup disks:
# nofail,x-systemd.device-timeout=10s
# Example line:
# UUID=xxxx-xxxx /mnt/backup ext4 nofail,x-systemd.device-timeout=10s 0 2
reboot -f
4) Free space fast (after boot)
# Clear logs
find /var/log -type f -exec truncate -s 0 {} \;
# See biggest dirs
du -sh /var/* | sort -h | tail -n 20
# Inspect Enhance usage
du -sh /var/local/enhance/* | sort -h | tail -n 20
Permanent fixes (hardening)
- Auto‑clean logs weekly
(crontab -l; echo '0 3 * * 0 find /var/log -type f -exec truncate -s 0 {} \;') | crontab - - Monitor container growth daily
(crontab -l; echo '0 4 * * * du -sh /var/local/enhance/containers >> /var/log/enhance_containers_usage.log') | crontab - - Force fsck repairs on reboot
# One-time next boot sudo touch /forcefsck # Make it default (GRUB) sudo sed -i 's/GRUB_CMDLINE_LINUX_DEFAULT=.*/GRUB_CMDLINE_LINUX_DEFAULT="fsck.mode=force fsck.repair=yes quiet splash"/' /etc/default/grub && sudo update-grub - Mark non‑critical disks as optional in
/etc/fstab→ addnofail,x-systemd.device-timeout=10s. - Keep backups off the root disk (separate volume or object storage).
- Avoid hard resets — always use
rebootso the journal flushes.
Case study: from read‑only to healthy
- Kernel logged ext4 journal abort → root remounted
ro. - Remount failed → entered initramfs; activated LVM and found root at
/dev/dm-0. - Ran
fsck.ext4 -fy /dev/dm-0→ fixed orphaned inodes & bitmap differences. - Boot then hung on a missing backup‑disk UUID → commented its line in
/etc/fstaband addednofail. - System came up
rw; logs trimmed; container usage reviewed.
Infographic: Recovery flow
FAQ
Why does Ubuntu remount the filesystem read‑only?
ext4 detected an inconsistent journal (often after a crash). It flips rw → ro to protect data. Repair with fsck.ext4 -fy on the root device (usually from initramfs).
“/dev/mapper/ubuntu–vg-ubuntu–lv not found” during fsck.
Activate LVM inside initramfs: lvm vgchange -ay. The root LV often appears as /dev/dm-0; verify with findmnt -no SOURCE / or lsblk -f.
Boot blocks at “A start job is running for /dev/disk/by-uuid/…”.
An fstab entry points to a missing disk. Comment the UUID line and add nofail,x-systemd.device-timeout=10s so missing disks don’t stall boot.
How do I free space quickly?
Truncate logs: find /var/log -type f -exec truncate -s 0 {} \;. Inspect containers: du -sh /var/local/enhance/* | sort -h | tail -n 20. Move backups off the root disk.
How can I reduce repeat incidents?
Keep cron maintenance, avoid forced power‑offs, mark non‑critical mounts nofail, and run fsck after any crash. Consider monitoring write spikes from containers and logs.
Emergency Remote Fix (Production)
Stuck in initramfs or emergency mode? Live guidance or hands‑on repair to bring the site back safely.
Ongoing Prevention & Audits
Log rotation, container pruning, fsck policy, and disk alerts so you never hit read‑only surprises again.
Found this useful? Share it with your team.
Keywords: Ubuntu read-only filesystem, ext4 aborted journal, initramfs emergency mode, boot UUID hang, fstab nofail, LVM, fsck, containers, logs.
