You cannot connect to an Azure Linux virtual machine (VM) by using Secure Shell (SSH). When you run the Boot Diagnostics feature on Azure portal, you see log entries that resemble the following examples.
Checking all file systems.[/sbin/fsck.ext4 (1) — /] fsck.ext4 -a /dev/sda1/dev/sda1 contains a file system with errors, check forced./dev/sda1: Inodes that were part of a corrupted orphan linked list found./dev/sda1: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY
EXT4-fs (sda1): INFO: recovery required on readonly filesystemEXT4-fs (sda1): write access will be enabled during recoveryEXT4-fs warning (device sda1): ext4_clear_journal_err:4531: Filesystem error recorded from previous mount: IO failureEXT4-fs warning (device sda1): ext4_clear_journal_err:4532: Making fs in need of filesystem check.
[ 14.252404] EXT4-fs (sda1): Couldn’t remount RDWR because of unprocessed orphan inode list. Please unmount/remount insteadAn error occurred while mounting /.
This example is caused by a clean fsck. In this case, there are also additional data disks attached to the VM (/dev/sdc1 and /dev/sde1).
Checking all file systems. [/sbin/fsck.ext4 (1) — /] fsck.ext4 -a /dev/sda1/dev/sda1: clean, 65405/1905008 files, 732749/7608064 blocks[/sbin/fsck.ext4 (1) — /tmp] fsck.ext4 -a /dev/sdc1[/sbin/fsck.ext4 (2) — /backup] fsck.ext4 -a /dev/sde1/dev/sdc1: clean, 12/1048576 files, 109842/4192957 blocks/dev/sde1 : clean, 51/67043328 files, 4259482/268173037 blocks
This problem may occur if the file system was not shut down cleanly.
Note We highly recommend that you make a backup of the virtual hard disk (VHD) from the inaccessible VM before you follow the recovery steps. You can stop the VM and back up the VHD by using Azure Storage Explorer.
To recover the VM, use one of the following methods.
- Attach the system disk to a recovery VM. To do this, go to the following Azure article, and follow the steps from the beginning up to the “Fix issues on original virtual hard disk” section:
- After you mount the system disk as a data disk on the new VM, locate the drive name of the system disk that you mounted, and then check the relevant log file. Notice that the log entry for each Linux distribution varies slightly.
grep SCSI /var/log/kern.log
CentOS, SUSE, Oracle, Red Hat
grep SCSI /var/log/messages
- Identify the partition that's required in order to run the disk check.
sudo -i fdisk -l df -h
Sample outputs from both commands
Disk /dev/sdc: 32.2 GB, 32212254720 bytes<255 heads, 63 sectors/track, 3916 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytesSector size (logical/physical): 512 bytes / 512 bytesI/O size (minimum/optimal): 512 bytes / 512 bytesDisk identifier: 0x000c23d3Device Boot Start End Blocks Id System/dev/sdc1 * 1 3789 0432256 83 Linux/dev/sdc2 3789 3917 1024000 82 Linux swap/Solarisdf -hFilesystem Size Used Avail Use% Mounted on/dev/sda1 29G 2.2G 25G 9% /tmpfs 776M 0 776M 0% /dev/shm/dev/sdb1 69G 180M 66G 1% /mnt/resource
In the output, you can determine that sda1 and sdb1 are mounted as part of the local system. The newly attached disk may be present as sdc1; but if it is, it is not mounted. In this situation, we have to run fsck against /dev/sdc1.
Run fsck on the partition that cannot be mounted.
fsck -yM /dev/sdc1
The following is a sample of the result after you run the command:
fsck from util-linux-ng 2.17.2 e2fsck 1.41.12 (17-May-2010) /dev/sdc1: clean, 57029/1905008 files, 672768/7608064 blocks
Unmount and detach the original virtual hard disk, and then create a VM from the original system disk.
- Access the serial console of your VM [vmname].
- If the VM is configured, press M for manual recovery to enter Single User mode, or log on as root. If the VM is not configured, restart the VM by using the Azure portal while holding down the ESC key. If you are presented with a grub prompt, enter "C" for command prompt.
- Change to the /etc directory, and then back up your fstab file:
cp fstab fstab_orig
- View and verify the contents of the fstab file in the cat/etc/fstab folder.
- Run blkid, and then compare the names and UUIDs of the partitions on this VM with the entries in your fstab file.
- Edit the fstab file to remove or comment out using a # any incorrect entries using your favorite text editor, for example: nano /etc/fstab or vi /etc/fstab.
- Verify the updates, and test the syntax before you run the following commmand:
reboot $ sudo mount -a
- Restart the VM, and then test SSH access.