Getting started
Check disk health (smartctl) and replace a failed RAID disk
Two commands and five minutes tell you whether a disk is healthy and the RAID is intact. Below: when to ask for a replacement, what to send us and how to rebuild the array once the new disk is in.
In short
- Disk health is shown by
smartctl -a /dev/sdafor SATA and SAS andsmartctl -a /dev/nvme0for NVMe. Warning signs: a non-zeroCurrent_Pending_Sectorand a growingReallocated_Sector_Ct; on NVMe,Percentage Usednear 100% and non-zeroMedia and Data Integrity Errors. - RAID status is shown by
cat /proc/mdstat:[UU]means both disks are in service,[U_]means one has dropped out and the array is running with no redundancy. - For a replacement, send the serial number of the failed disk and the
smartctl -aoutput in a ticket. The server is usually shut down for the replacement. Then you rebuild the array yourself: copy the partition table from the healthy disk to the new one (not the other way round) and add the new disk withmdadm --add. - Make a backup before the replacement: under the offer, backups are your job, and an array with one live disk holds only until the next failure.
Contents
- Install smartmontools
- Write down the disks and their serial numbers
- SATA and SAS: which SMART values to check
- NVMe: wear and media errors
- Check software RAID: /proc/mdstat and mdadm
- Windows: what you can check
- How to request a disk replacement
- How to rebuild RAID after a disk replacement
- How to check that everything worked
- If something went wrong
- FAQ
You will need
- SSH access to the server as root or as a user with sudo: how to connect via SSH.
- A fresh backup of important data if you are going to replace a disk.
The commands below are for Linux. What you can check on Windows is described in a separate section: there are fewer options there.
Install smartmontools
The smartctl program from the smartmontools package reads SMART, the self-monitoring log that every disk keeps. Install the package with the first command on Ubuntu and Debian or with the second on AlmaLinux and Rocky Linux:
sudo apt install smartmontools
sudo dnf install smartmontools
The commands below are written with sudo. If you are logged in as root, as is usual on Hetzner servers, you can leave it out.
Write down the disks and their serial numbers
The command lists the physical disks without their partitions: the name in the system, size, model and serial number.
lsblk -d -o NAME,SIZE,TYPE,MODEL,SERIAL
NAME SIZE TYPE MODEL SERIAL
nvme0n1 953.9G disk SAMSUNG MZVL21T0HCLR-00B00 S6EXAMPLE00001
nvme1n1 953.9G disk SAMSUNG MZVL21T0HCLR-00B00 S6EXAMPLE00002
lsblk shows one disk although the server has several, and /proc/mdstat is empty
Most likely the server has a hardware RAID controller: the system sees the array the controller has built as a single disk. sudo smartctl --scan shows how to address the disks behind the controller. For MegaRAID controllers it is the -d option with the disk number, for example sudo smartctl -a -d megaraid,0 /dev/sda. The state of the array itself is shown by the controller vendor’s utility.
Save the serial numbers of all disks. Names such as sda or nvme1n1 can swap places after a replacement and a reboot, but the serial number stays the same: the technician uses it to find the right disk in the server.
SATA and SAS: which SMART values to check
The command prints everything the disk knows about itself: model, serial number, the overall verdict and the attribute table.
sudo smartctl -a /dev/sda
First find the line with the verdict:
SMART overall-health self-assessment test result: PASSED
FAILED means the disk itself admits a fault, and it is time to replace it. PASSED guarantees nothing: a disk passes this check even when it is already losing sectors. So next look at the attribute table, the RAW_VALUE column:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0
9 Power_On_Hours 0x0032 067 067 000 Old_age Always - 29041
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
| Attribute | What it counts | When to worry |
|---|---|---|
5 Reallocated_Sector_Ct | Bad sectors that the disk has already replaced with spare ones | Above zero and growing from one check to the next |
197 Current_Pending_Sector | Sectors the disk could not read and is waiting to reallocate | Any number above zero |
198 Offline_Uncorrectable | Sectors with errors that could not be corrected | Any number above zero |
199 UDMA_CRC_Error_Count | Transfer errors between the disk and the controller, usually caused by a cable or connector | Growing: report it in a ticket, the data center replaces the cable |
On SATA SSDs, wear is shown by an attribute that each manufacturer names differently: Wear_Leveling_Count, Media_Wearout_Indicator, Percent_Lifetime_Remain. Look at its VALUE column: it usually starts at 100 and goes down as the disk wears.
SAS disks report differently. The verdict is in the SMART Health Status line (normal is OK), reallocated sectors are in the Elements in grown defect list line, and uncorrected errors are in the Total uncorrected errors column of the Error counter log table.
If in doubt, run a short self-test. The disk runs it on its own in a couple of minutes, and the server keeps working meanwhile. The second command shows the result: Completed without error means all is well, Completed: read failure means the disk could not read part of its surface.
sudo smartctl -t short /dev/sda
sudo smartctl -l selftest /dev/sda
Self-test errors are a reason to ask for a replacement. Save the output: you will need it for the ticket.
NVMe: wear and media errors
NVMe disks have no attribute table; they have a SMART/Health Information block instead:
sudo smartctl -a /dev/nvme0
SMART overall-health self-assessment test result: PASSED
SMART/Health Information (NVMe Log 0x02)
Critical Warning: 0x00
Temperature: 41 Celsius
Available Spare: 100%
Available Spare Threshold: 10%
Percentage Used: 7%
Media and Data Integrity Errors: 0
Error Information Log Entries: 0
| Line | What it means | When to worry |
|---|---|---|
Critical Warning | Flags for serious problems: the spare capacity has run out, the disk is overheating, the disk has switched to read-only mode | Any value other than 0x00 |
Percentage Used | How much of the write endurance stated by the manufacturer has been used up. It can go above 100% | The closer to 100%, the higher the risk of failure. From 90%, make a backup and ask for a replacement without waiting for a failure |
Available Spare | How many spare blocks are left to replace worn-out ones | It has dropped to Available Spare Threshold |
Media and Data Integrity Errors | Data errors the disk could not correct | Any number above zero |
Error Information Log Entries | Entries in the controller’s error log | Not a reason to worry on its own: harmless command errors end up here too |
If you have doubts about one disk of a pair, run the same command for the other one: the difference in wear and errors shows which of them is weaker.
Check software RAID: /proc/mdstat and mdadm
On Hetzner and OVH servers with several disks, Linux is usually installed on software RAID 1. One command shows its state:
cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 nvme1n1p3[1] nvme0n1p3[0]
966188032 blocks super 1.2 [2/2] [UU]
bitmap: 3/8 pages [12KB], 65536KB chunk
md1 : active raid1 nvme1n1p2[1] nvme0n1p2[0]
1046528 blocks super 1.2 [2/2] [UU]
md0 : active raid1 nvme1n1p1[1] nvme0n1p1[0]
33520640 blocks super 1.2 [2/2] [UU]
unused devices: <none>
There are three arrays here, each with two partitions, one on each disk. [2/2] [UU] means the array has both of its two members and both are working. This is what an array that has lost a disk looks like:
md2 : active raid1 nvme0n1p3[0] nvme1n1p3[1](F)
966188032 blocks super 1.2 [2/1] [U_]
(F) means the partition is marked as failed, [2/1] [U_] means one member out of two is working. If the disk has disappeared completely, its partition will not be in the line at all, but [U_] will remain.
mdadm --detail shows the details of one array. Normal is State : clean; a dropped disk shows up as clean, degraded and faulty or removed lines at the bottom:
sudo mdadm --detail /dev/md2
State : clean, degraded
Active Devices : 1
Working Devices : 1
Failed Devices : 1
Number Major Minor RaidDevice State
0 259 3 0 active sync /dev/nvme0n1p3
- 0 0 1 removed
1 259 7 - faulty /dev/nvme1n1p3
RAID 0 does not protect against disk failure. It spreads every file across all the disks, so if one of them fails, all the data in the array is lost. If you have RAID 0, a backup is your only protection.
Even when the array is intact, a disk with worrying SMART values is better replaced now, while both disks are working.
Windows: what you can check
Windows has no smartctl of its own, but the overall state of the disks is visible in PowerShell run as administrator. The first command lists the disks with their serial numbers and status: a healthy disk has HealthStatus Healthy, a problem disk has Warning or Unhealthy.
Get-PhysicalDisk | Format-Table FriendlyName, SerialNumber, MediaType, HealthStatus, OperationalStatus
The second shows the reliability counters: Wear is wear in percent, ReadErrorsUncorrected is uncorrected read errors. Some disks and drivers do not report these counters, and then the columns are empty.
Get-PhysicalDisk | Get-StorageReliabilityCounter | Format-Table DeviceId, Wear, ReadErrorsUncorrected, Temperature, PowerOnHours
Detailed SMART data on Windows comes from third-party programs: smartmontools for Windows (the same smartctl) or CrystalDiskInfo.
How RAID is built on a Windows server depends on the installation. If it is Storage Spaces, Get-VirtualDisk shows the state: an intact array has HealthStatus Healthy. If it is a mirrored volume, open Disk Management (diskmgmt.msc): a healthy volume has the status “Healthy”. You can also rebuild the mirror there after a disk replacement. If you are not sure how your array is set up, ask in a ticket before the replacement.
How to request a disk replacement
Make a backup first. While the array is running on one disk, there is no second copy of the data on the server: if that disk fails too, the data is gone. The rebuild after a replacement puts load on the remaining disk, so make the backup before it. Under the offer, backups are the client’s responsibility, and we are not liable for data loss except in the cases of clause 7.2, as clause 5.5 says.
Copy important data off the server
Website files, databases, settings: to another server or to your own computer. Check that the copy opens.
Find the serial number of the failed disk
The command prints the
Serial Numberline. For SATA and SAS, give a disk such as/dev/sdb.sudo smartctl -i /dev/nvme1 | grep -i serialSave the SMART output to a file
The command writes the full disk report to a file in the current folder. Download the file to your computer (for example with WinSCP or
scp) or copy the output from the terminal straight into the ticket.sudo smartctl -a /dev/nvme1 > smart-nvme1.txtRemove the failed disk from the array
This way the array will not access the disk at the moment it is replaced. The command marks the partition as failed and removes it from the array. Repeat it for every array that has partitions on this disk:
cat /proc/mdstatshows which arrays they are. If the disk has already disappeared from the system, skip this step.Remove only the partitions of the failed disk. If you remove the healthy one by mistake, the array will run only on the failing disk, the technician will replace exactly that disk, and no working array will be left. Before running the command, match the disk name (
nvme1n1in the example) with the serial number from the previous step.sudo mdadm /dev/md0 --fail /dev/nvme1n1p1 --remove /dev/nvme1n1p1sudo mdadm /dev/md1 --fail /dev/nvme1n1p2 --remove /dev/nvme1n1p2sudo mdadm /dev/md2 --fail /dev/nvme1n1p3 --remove /dev/nvme1n1p3Open a ticket
Copy the template into a ticket and attach the file with the SMART output. The serial number of the healthy disk is a safeguard against mistakes: the technician will know which disk must not be touched.
Server: #1234567 Replace disk: serial number S6EXAMPLE00002 (currently /dev/nvme1n1) Healthy disk, do not touch: S6EXAMPLE00001 Reason: Media and Data Integrity Errors are growing, the disk dropped out of the RAID smartctl -a output: attached Backup done. The server can be shut down: at any timeWait for the replacement
The server is usually shut down for the replacement and is unavailable while the technician works. How long this takes depends on the data center. If downtime suits you better at a particular time, say so in the ticket, and we will pass it on to the data center. When the disk has been replaced, we will write in the same ticket.
If the server does not boot after the replacement, request a console: on Hetzner we connect it for you (Hetzner KVM console), on OVH it is in your panel (Rescue and IPMI on OVH).
How to rebuild RAID after a disk replacement
A new disk usually comes without a partition table and will not join the array by itself. In the example the healthy disk is nvme0n1 and the new one is nvme1n1. SATA disks will have names such as sda and sdb, and their partitions will be sdb1 instead of nvme1n1p1.
Do not mix up the disks. The partition table is copied from the healthy disk to the new one. If you swap them, the empty partition table of the new disk will be written over the healthy disk’s partition table, and the data will become inaccessible. Before copying, match the serial numbers and make sure the new disk has no partitions. Disk names may have changed after the replacement: use your own names in the commands, not the ones from the example.
Find the new disk
The new disk will have an unfamiliar serial number and no partitions under it. You will recognize the healthy disk by the serial number you wrote down before the replacement and by its partitions, which have
mdarrays of typeraid1under them.lsblk -o NAME,SIZE,TYPE,SERIALFind out the partition table type of the healthy disk
The command answers
gptordos. This decides which command you use to copy the partition table.lsblk -d -o NAME,PTTYPE /dev/nvme0n1Save the partition table of the healthy disk to a file
This is insurance: if something goes wrong, you can restore the partition table from the file. The first command is for GPT, the second for MBR (
dos). The file appears in the current folder. If there is nosgdiskcommand, install thegdiskpackage.sudo sgdisk --backup=nvme0n1-table.bak /dev/nvme0n1sudo sfdisk -d /dev/nvme0n1 > nvme0n1-table.txtCopy the partition table from the healthy disk to the new one
GPT. In
sgdiskthe order of the disks is the reverse of what you would expect: the new disk (where to copy to) comes right after-R, and the healthy disk (where to copy from) comes last. The second command gives the new disk its own identifiers so that they do not match those of the healthy disk.sudo sgdisk -R /dev/nvme1n1 /dev/nvme0n1sudo sgdisk -G /dev/nvme1n1MBR (dos). Here the order is the usual one: on the left is the healthy disk the partition table is read from, on the right the new disk it is written to.
sudo sfdisk -d /dev/nvme0n1 | sudo sfdisk /dev/nvme1n1Check the result with
lsblk: both disks should have the same partitions.Add the partitions of the new disk to the arrays
The partition number is the same as that of the healthy disk in the same array: if
md0containsnvme0n1p1, addnvme1n1p1. Go through all the arrays incat /proc/mdstat: you may have more or fewer than three.sudo mdadm /dev/md0 --add /dev/nvme1n1p1sudo mdadm /dev/md1 --add /dev/nvme1n1p2sudo mdadm /dev/md2 --add /dev/nvme1n1p3Wait for the sync to finish
The array copies the data to the new disk in the background; the server keeps working all the while, only slower than usual. The command refreshes the status every 10 seconds; exit with
Ctrl+C. Multi-terabyte HDDs take many hours to sync.watch -n 10 cat /proc/mdstatmd2 : active raid1 nvme1n1p3[2] nvme0n1p3[0] 966188032 blocks super 1.2 [2/1] [U_] [==>..................] recovery = 12.6% (121780224/966188032) finish=68.2min speed=206342K/secInstall the boot loader on the new disk
Otherwise the server will not boot from the new disk if the old one fails later. First find out the boot mode:
[ -d /sys/firmware/efi ] && echo UEFI || echo BIOSIf the answer is
BIOS, install the boot loader on the new disk. The command is for Ubuntu and Debian; on AlmaLinux and Rocky Linux it is calledgrub2-install:sudo grub-install /dev/nvme1n1If the answer is
UEFI, the boot loader is on a separate EFI partition, and how to replicate it on the new disk depends on how the system was installed.findmnt /boot/efishows where it is: if the answer is an array/dev/md…, the partition is already syncing along with the others. If the answer is a partition on a single disk, write in a ticket before you reboot the server and attach the output offindmnt /boot/efiandlsblk.
Once the sync has finished and the boot loader is on both disks, the array will again survive the failure of one of them.
How to check that everything worked
In cat /proc/mdstat, every array shows [2/2] [UU] again and there is no recovery line, and sudo mdadm --detail /dev/md2 shows State : clean. Check the SMART data of the new disk with the same smartctl -a command: it should have no errors.
If something went wrong
| What you see | Why | What to do |
|---|---|---|
mdadm: /dev/nvme1n1p3 not large enough to join array | The partition on the new disk is smaller than in the array: the partition table was copied wrongly, or the new disk is smaller than the old one | Compare the partitions in lsblk. If the disk itself is smaller, write in a ticket |
mdadm: Cannot open /dev/nvme1n1p1: Device or resource busy | The partition is already in use: the system has often assembled an extra array such as md127 from the old partition table of the new disk | Look at cat /proc/mdstat. If the extra array contains only partitions of the new disk, stop it with sudo mdadm --stop /dev/md127 and repeat --add |
The new disk does not show in lsblk | The disk was not detected, or a different disk was replaced | Match the serial numbers with the ones you wrote down before the replacement and write in a ticket |
| The server does not boot after the replacement | The boot loader was only on the removed disk, or the BIOS is looking for a boot device on the new one | At the console, choose to boot from the healthy disk, or boot into Rescue and install the boot loader: Hetzner Rescue System, Rescue on OVH |
If the array does not rebuild, stop and write in a ticket, attaching the output of cat /proc/mdstat and lsblk: further attempts on an array with one disk are more dangerous than a pause.
FAQ
SMART says PASSED. Does that mean the disk is fine?
Media and Data Integrity Errors and Percentage Used.My NVMe shows Percentage Used 93%. Will the disk fail soon?
Can I recover data from RAID 0 after a disk failure?
Can a disk be replaced without shutting down the server?
Did this guide help?
Thank you! This tells us which guides work.
What went wrong?
Tell support at which step you got stuck and include the link to this guide, so you don’t have to explain from scratch.