Changes between Version 18 and Version 19 of BadBlockHowto

Timestamp:

Apr 11, 2017, 2:53:40 PM (7 years ago)

Author:

Gabriele Pohl

Comment:

Move section about LVM repairs to chapter "case studies"

BadBlockHowto

@@ -499 +499 @@
 For creating, destroying, resizing, checking and copying partitions, and the file systems on them, GNU's `parted` is worth examining. The [http://www.tldp.org/HOWTO/Large-Disk-HOWTO.html Large Disk HOWTO] is also a useful resource. 
 
+=== Bad block reassignment ===
+
+The SCSI disk command set and associated disk architecture are assumed in this section. SCSI disks have their own logical to physical mapping allowing a damaged sector (usually carrying 512 bytes of data) to be remapped irrespective of the operating system, file system or software RAID being used.
+
+The terms ''block and sector'' are used interchangeably, although block tends to get used in higher level or more abstract contexts such as a ''logical block''.
+
+When a SCSI disk is formatted, defective sectors identified during the manufacturing process (the so called primary list: PLIST), those found during the format itself (the certification list: CLIST), those given explicitly to the format command (the DLIST) and optionally the previous grown list (GLIST) are not used in the logical block map. The number (and low level addresses) of the unmapped sectors can be found with the `READ DEFECT DATA SCSI` command.
+
+SCSI disks tend to be divided into zones which have spare sectors and perhaps spare tracks, to support the logical block address mapping process. The idea is that if a logical block is remapped, the heads do not have to move a long way to access the replacement sector. Note that spare sectors are a scarce resource.
+
+Once a SCSI disk format has completed successfully, other problems may appear over time. These fall into two categories:
+
+* recoverable: the Error Correction Codes (ECC) detect a problem but it is small enough to be corrected. Optionally other strategies such as retrying the access may retrieve the data.
+* unrecoverable: try as it may, the disk logic and ECC algorithms cannot recover the data. This is often reported as a ''medium error''.
+
+Other things can go wrong, typically associated with the transport and they will be reported using a term other than ''medium error''. For example a disk may decide a read operation was successful but a computer's host bus adapter (HBA) checking the incoming data detects a CRC error due to a bad cable or termination.
+
+Depending on the disk vendor, recoverable errors can be ignored. After all, some disks have up to 68 bytes of ECC above the payload size of 512 bytes so why use up spare sectors which are limited in number ^[#footnote8 [8]]^ ? If the disk can recover the data and does decide to re-allocate (reassign) a sector, then first it checks the settings of the `ARRE` and `AWRE` bits in the read-write error recovery mode page. Usually these bits are set ^[#footnote9 [9]]^ enabling automatic (read or write) re-allocation. The automatic re-allocation may also fail if the zone (or disk) has run out of spare sectors.
+
+Another consideration with RAIDs, and applications that require a high data rate without pauses, is that the controller logic may not want a disk to spend too long trying to recover an error.
+
+Unrecoverable errors will cause a ''medium error'' sense key, perhaps with some useful additional sense information. If the extended background self test includes a full disk read scan, one would expect the self test log to list the bad block, as shown in section [#Repairsinafilesystem Repairs in a file system]. Recent SCSI disks with a periodic background scan should also list unrecoverable read errors (and some recoverable errors as well). The advantage of the background scan is that it runs to completion while self tests will often terminate at the first serious error.
+
+SCSI disks expect unrecoverable errors to be fixed manually using the `REASSIGN BLOCKS SCSI` command since loss of data is involved. It is possible that an operating system or a file system could issue the `REASSIGN BLOCKS` command itself but the authors are unaware of any examples. The `REASSIGN BLOCKS` command will reassign one or more blocks, attempting to (partially ?) recover the data (a forlorn hope at this stage), fetch an unused spare sector from the current zone while adding the damaged old sector to the GLIST (hence the name ''grown'' list). The contents of the GLIST may not be that interesting but `smartctl` prints out the number of entries in the grown list and if that number grows quickly, the disk may be approaching the end of its useful life.
+
+Here is an alternate brute force technique to consider: if the data on the SCSI or ATA disk has all been backed up (e.g. is held on the other disks in a RAID 5 enclosure), then simply reformatting the disk may be the least cumbersome approach.
+
+==== Example ====
+
+Given a ''bad block'', it still may be useful to look at the `fdisk` command (if the disk has multiple partitions) to find out which partition is involved, then use `debugfs` (or a similar tool for the file system in question) to find out which, if any, file or other part of the file system may have been damaged. This is discussed in section [#Repairsinafilesystem Repairs in a file system].
+
+Then a program that can execute the `REASSIGN BLOCKS SCSI` command is required. In Linux (2.4 and 2.6 series), FreeBSD, Tru64(OSF) and Windows the author's `sg_reassign` utility in the `sg3_utils` package can be used. Also found in that package is `sg_verify` which can be used to check that a block is readable.
+
+Assume that `logical block address 1193046` (which is `123456` in hex) is corrupt ^[#footnote10 [10]]^ on the disk at `/dev/sdb`. A long selftest command like `smartctl -t long /dev/sdb` may result in log results like this:
+
+{{{
+# smartctl -l selftest /dev/sdb
+smartctl version 5.37 [i686-pc-linux-gnu] Copyright (C) 2002-6 Bruce Allen
+Home page is http://smartmontools.sourceforge.net/
+SMART Self-test log
+Num  Test              Status            segment  LifeTime  LBA_first_err [SK ASC ASQ]
+     Description                         number   (hours)
+# 1  Background long   Failed in segment      -     354           1193046 [0x3 0x11 0x0]
+# 2  Background short  Completed              -     323                 - [-   -    -]
+# 3  Background short  Completed              -     194                 - [-   -    -]
+}}}
+
+The `sg_verify` utility can be used to confirm that there is a problem at that address:
+
+{{{
+# sg_verify --lba=1193046 /dev/sdb
+verify (10):  Fixed format, current;  Sense key: Medium Error
+ Additional sense: Unrecovered read error
+  Info fld=0x123456 [1193046]
+  Field replaceable unit code: 228
+  Actual retry count: 0x008b
+medium or hardware error, reported lba=0x123456
+}}}
+
+Now the GLIST length is checked before the block reassignment:
+
+{{{
+# sg_reassign --grown /dev/sdb
+>> Elements in grown defect list: 0
+}}}
+
+And now for the actual reassignment followed by another check of the GLIST length:
+
+{{{
+# sg_reassign --address=1193046 /dev/sdb
+# sg_reassign --grown /dev/sdb
+>> Elements in grown defect list: 1
+}}}
+
+The GLIST length has grown by one as expected. If the disk was unable to recover any data, then the ''new'' block at lba `0x123456` has vendor specific data in it. The `sg_reassign` utility can also do bulk reassigns, see `man sg_reassign` for more information.
+
+The `dd` command could be used to read the contents of the ''new'' block:
+
+{{{
+# dd if=/dev/sdb iflag=direct skip=1193046 of=blk.img bs=512 count=1
+}}}
+
+and a hex editor ^[#footnote11 [11]]^ used to view and potentially change the `blk.img` file. An altered `blk.img` file (or `/dev/zero`) could be written back with:
+
+{{{
+# dd if=blk.img of=/dev/sdb seek=1193046 oflag=direct bs=512 count=1
+}}}
+
+More work may be needed at the file system level, especially if the reassigned block held critical file system information such as a superblock or a directory.
+
+Even if a full backup of the disk is available, or the disk has been ''ejected'' from a RAID, it may still be worthwhile to reassign the bad block(s) that caused the problem (or simply format the disk (see `sg_format` in the `sg3_utils package`)) and re-use the disk later (not unlike the way a replacement disk from a manufacturer might be used).
+
+
+== Case Studies ==
+
+This section is intended to collect step-by-step descriptions of some real-life use cases.
+
+=== Recovering a (mostly) unreadable sector of a Notebook HDD ===
+
+This was done in March 2016 under Windows 7 using ''Cygwin''^[#footnote12 12.]^ ports of ''GNU ddrescue''^[#footnote13 13.]^ and ''The Sleuth Kit (TSK)''^[#footnote14 14.]^. All commands shown should work similar on other platforms and with other filesystems.
+
+==== Determine Logical Block Address of unreadable sector ====
+
+Examine smartctl output:
+{{{
+root:~# smartctl -x /dev/sdb
+smartctl 6.5 2016-02-29 r4227 [x86_64-w64-mingw32-win7-sp1] (daily-20160229)
+...
+Model Family:     SAMSUNG SpinPoint MP5
+Device Model:     SAMSUNG HM640JJ
+...
+Firmware Version: 2AK10001
+User Capacity:    640.135.028.736 bytes [640 GB]
+Sector Size:      512 bytes logical/physical
+Rotation Rate:    7200 rpm
+Form Factor:      2.5 inches
+...
+ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
+...
+  5 Reallocated_Sector_Ct   PO--CK   252   252   010    -    0
+...
+  9 Power_On_Hours          -O--CK   100   100   000    -    251  <=== See Self-test Log below
+...
+197 Current_Pending_Sector  -O--CK   100   100   000    -    1    <=== At least 1 bad sector
+...
+SMART Extended Comprehensive Error Log Version: 1 (2 sectors)
+Device Error Count: 351 (device log contains only the most recent 8 errors)
+...
+Error 351 [6] occurred at disk power-on lifetime: 251 hours (10 days + 11 hours)
+  When the command that caused the error occurred, the device was active or idle.
+
+  After command completion occurred, registers were:
+  ER -- ST COUNT  LBA_48  LH LM LL DV DC
+  -- -- -- == -- == == == -- -- -- -- --
+  40 -- 51 00 01 00 00 33 3f d8 a6 40 00  Error: UNC 1 sectors at LBA = 0x333fd8a6 = 859822246  <=== Its LBA
+
+  Commands leading to the command that caused the error were:
+  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
+  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
+  25 00 00 00 01 00 00 33 3f d8 a6 40 00     00:00:06.924  READ DMA EXT
+...
+
+SMART Extended Self-test Log Version: 1 (2 sectors)
+Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
+# 1  Short offline       Completed: read failure       90%       176         859822246  <=== Detected 75 power on hours ago
+}}}
+
+A read scan helps to verify the LBA and checks for other possible bad sectors
+(alternatively replace `/dev/null` by a file path to create a disk image):
+
+{{{
+root:~# ddrescue --ask --verbose --binary-prefixes --idirect --force /dev/sdb /dev/null disk.map
+GNU ddrescue 1.21-rc2
+About to copy 610480 MiBytes from /dev/sdb [SAMSUNG HM640JJ::...] to /dev/null [0].
+Proceed (y/N)? y
+...
+non-tried:        0 B,     errsize:      512 B,      run time:          2h
+  rescued: 610480 MiB,      errors:        1,  remaining time:         n/a
+percent rescued:  99.99%      time since last successful read:         20s
+Finished
+}}}
+
+The `ddrescue` map file now shows byte ranges of good and bad disk areas:
+
+{{{
+root:~# cat disk.map
+...
+#      pos        size      status
+  0x00000000  0x667FB14C00  +
+0x667FB14C00    0x00000200  -  <=== 512 bytes unreadable
+0x667FB14E00  0x2E8B541200  +
+}}}
+
+Translate the byte position to the LBA:
+
+{{{
+root:~# echo $((0x667FB14C00/512))
+859822246
+}}}
+
+Or convert the map file to a `badblocks` like list with `ddrescuelog` (part of recent versions of ''ddrescue'' package):
+
+{{{
+root:~# ddrescuelog --list-blocks=- disk.map
+859822246
+}}}
+
+Both match the LBA reported by `smartctl`.
+
+==== Find affected file ====
+
+Get start offset of affected partition:
+
+{{{
+root:~# fdisk --list /dev/sdb
+...
+Device     Boot Start        End    Sectors   Size Id Type
+/dev/sdb1          63 1250258624 1250258562 596.2G  7 HPFS/NTFS/exFAT
+}}}
+
+Get filesystem block (cluster) size if unknown (4096 in many cases):
+
+{{{
+root:~# fsstat /dev/sdb1
+...
+File System Type: NTFS
+...
+Sector Size: 512
+Cluster Size: 4096
+...
+}}}
+
+Calculate number of bad cluster as `(BAD_LBA - START_LBA) / SECTORS_PER_CLUSTER`:
+
+{{{
+root:~# echo $(((859822246-63)/8))
+107477772
+}}}
+
+Find inode (here: MFT entry) used by this cluster:
+
+{{{
+root:~# ifind -d 107477772 /dev/sdb1
+663-128-2
+}}}
+
+Print some info about this inode:
+
+{{{
+root:~# istat /dev/sdb1 663-128-2
+...
+Name: Backup_2015-12-17.zip
+Parent MFT Entry: 30    Sequence: 1
+Allocated Size: 4660039680      Actual Size: 4660039516
+Created:        2015-12-17 13:43:30.460000000 (CET)
+File Modified:  2015-12-17 13:46:19.647000000 (CET)
+...
+Type: $DATA (128-2)   Name: N/A   Non-Resident   size: 4660039516  init_size: 4660039516
+106950180 106950181 ...
+...
+107477772  <=== The bad cluster
+...
+108087884
+}}}
+
+Find full path of affected file:
+
+{{{
+root:~# ffind /dev/sdb1 663-128-2
+/Backups/2015/Backup_2015-12-17.zip
+}}}
+
+If the file is no longer needed, it could be overwritten in place and removed then. This is easy with `shred` from ''GNU coreutils'': `shred --iterations=1 --remove /PATH/TO/FILE`. This should reallocate the bad sector in most cases.
+
+==== Try to recover the bad sector ====
+
+Start with 100 read retries of the bad sector, write to `recovered.bin` if successful:
+
+{{{
+root:~# ddrescue --ask --verbose --binary-prefixes --idirect --retry=100 \
+                 --input-position=859822246s --output-position=0 --size=1s \
+                 /dev/sdb recovered.bin recovered.map
+...
+Current status
+     ipos: 419835 MiB, non-trimmed:        0 B,  current rate:      32 B/s
+     opos:        0 B, non-scraped:        0 B,  average rate:       4 B/s
+non-tried:        0 B,     errsize:        0 B,      run time:      1m 49s
+  rescued:      512 B,      errors:        0,  remaining time:         n/a
+percent rescued: 100.00%      time since last successful read:          0s
+Finished
+}}}
+
+We were very lucky:
+
+{{{
+root:~# cat recovered.map
+...
+#      pos        size      status
+  0x00000000  0x667FB14C00  ?
+0x667FB14C00  0x00000200    +  <=== Now OK!
+0x667FB14E00  0x2E8B541200  ?
+}}}
+
+Check whether the disk firmware took the chance to reallocate the sector using the recovered data:
+
+{{{
+root:~# dd skip=859822246 count=1 iflag=direct if=/dev/sdb of=test.bin
+dd: error reading ‘/dev/sdb’: Input/output error
+0+0 records in
+0+0 records out
+0 bytes (0 B) copied, 23.5006 s, 0.0 kB/s
+}}}
+
+No luck in this case. So overwrite the sector manually:
+
+{{{
+root:~# dd seek=859822246 count=1 oflag=direct if=recovered.bin of=/dev/sdb
+1+0 records in
+1+0 records out
+512 bytes (512 B) copied, 1.05331 s, 0.5 kB/s
+}}}
+
+Read data back and check:
+
+{{{
+root:~# dd skip=859822246 count=1 iflag=direct if=/dev/sdb of=test.bin
+1+0 records in
+1+0 records out
+512 bytes (512 B) copied, 0.0211745 s, 24.2 kB/s
+
+root:~# diff -s recovered.bin test.bin
+Files recovered.bin and test.bin are identical
+}}}
+
+Finally, run a SMART self-test and check its result:
+
+{{{
+root:~# smartctl -t short /dev/sdb
+...
+Sending command: "Execute SMART Short self-test routine immediately in off-line mode".
+...
+Please wait 2 minutes for test to complete.
+
+root:~# sleep 120 # :-)
+
+root:~# smartctl -x /dev/sdb
+...
+ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
+...
+  5 Reallocated_Sector_Ct   PO--CK   252   252   010    -    0   <=== Interesting...
+...
+  9 Power_On_Hours          -O--CK   100   100   000    -    252
+...
+197 Current_Pending_Sector  -O--CK   100   100   000    -    0   <=== As expected
+...
+SMART Extended Self-test Log Version: 1 (2 sectors)
+Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
+# 1  Short offline       Completed without error       00%       252         -          <=== Works again!
+# 2  Short offline       Completed: read failure       90%       176         859822246
+}}}
+
+Interestingly the `Reallocated_Sector_Ct` did not increase. Either the firmware did not record the reallocation or decided to reuse the original sector.
+
+Done!
 
 === LVM repairs ===
@@ -504 +848 @@
 This section was written by Frederic BOITEUX. It was titled: "HOW TO LOCATE AND REPAIR BAD BLOCKS ON AN LVM VOLUME".
 
-Smartd reports an error in a short test :
+Smartd reports an error in a short test :
 
 {{{
@@ -516 +860 @@
 So the disk has a bad block located in `LBA block 37383668`
 
-In which physical partition is the bad block ?
+In which physical partition is the bad block ?
 
 {{{
@@ -538 +882 @@
 We have to find in which LVM2 logical partition the block belongs to.
 
-In which logical partition is the bad block ?
-
-''IMPORTANT'' : LVM2 can use different schemes dividing its physical partitions to logical ones : linear, striped, contiguous or not... The following example assumes that allocation is linear !
+In which logical partition is the bad block ?
+
+''IMPORTANT'' : LVM2 can use different schemes dividing its physical partitions to logical ones : linear, striped, contiguous or not... The following example assumes that allocation is linear !
 
 The physical partition used by LVM2 is divided in PE (Physical Extent) units of the same size, starting at `pe_start` 512 bytes blocks from the beginning of the physical partition.
 
-The `pvdisplay` command gives the size of the PE (in KB) of the LVM partition :
-
-{{{
-#  part=/dev/hdb3 ; pvdisplay -c $part | awk -F: '{print $8}'
+The `pvdisplay` command gives the size of the PE (in KB) of the LVM partition :
+
+{{{
+#  part=/dev/hdb3 ; pvdisplay -c $part | awk -F: '{print $8}'
 4096
 }}}
 
-To get its size in LBA block size (512 bytes or 0.5 KB), we multiply this number by 2 : `4096 * 2 = 8192 blocks` for each PE.
-
-To find the offset from the beginning of the physical partition is a bit more difficult : if you have a recent LVM2 version, try :
+To get its size in LBA block size (512 bytes or 0.5 KB), we multiply this number by 2 : `4096 * 2 = 8192 blocks` for each PE.
+
+To find the offset from the beginning of the physical partition is a bit more difficult : if you have a recent LVM2 version, try :
 
 {{{
@@ -559 +903 @@
 }}}
 
-Either, you can look in `/etc/lvm/backup` :
+Either, you can look in `/etc/lvm/backup` :
 {{{
 # grep pe_start $(grep -l $part /etc/lvm/backup/*)
@@ -565 +909 @@
 }}}
 
-Then, we search in which PE is the badblock, calculating the PE rank in which the faulty block of the partition is : `physical partition's bad block number / sizeof(PE)` =
+Then, we search in which PE is the badblock, calculating the PE rank in which the faulty block of the partition is : `physical partition's bad block number / sizeof(PE)` =
 {{{
 36194858 / 8192 = 4418.3176
 }}}
 
-So we have to find in which LVM2 logical partition is used the PE number 4418 (count starts from 0) :
+So we have to find in which LVM2 logical partition is used the PE number 4418 (count starts from 0) :
 {{{
 # lvdisplay --maps |egrep 'Physical|LV Name|Type'
@@ -605 +949 @@
 So the PE #4418 is in the `/dev/WDC80Go/ext1` LVM logical partition.
 
-Size of logical block of file system on `/dev/WDC80Go/ext1` :
-
-It's a ext3 fs, so I get it like this :
+Size of logical block of file system on `/dev/WDC80Go/ext1` :
+
+It's a ext3 fs, so I get it like this :
 {{{
 # dumpe2fs /dev/WDC80Go/ext1 | grep 'Block size'
@@ -614 +958 @@
 }}}
 
-bad block number for the file system :
-
-The logical partition begins on `PE 3072` :
+bad block number for the file system :
+
+The logical partition begins on `PE 3072` :
 {{{
  (# PE's start of partition * sizeof(PE)) + parttion offset[pe_start] =
@@ -622 +966 @@
 }}}
 
-512b block of the physical partition, so the bad block number for the file system  is :
+512b block of the physical partition, so the bad block number for the file system  is :
 {{{
 (36194858 - 25166208) / (sizeof(fs block) / 512)
@@ -628 +972 @@
 }}}
 
-Test of the fs bad block :
+Test of the fs bad block :
 {{{
 dd if=/dev/WDC80Go/ext1 of=block1378581 bs=4096 count=1 skip=1378581
 }}}
 
-If this `dd` command succeeds, without any error message in console or syslog, then the block number calculation is probably wrong ! *Don't* go further, re-check it and if you don't find the error, please renounce !
-
-Search / correction follows the same scheme as for simple partitions :
+If this `dd` command succeeds, without any error message in console or syslog, then the block number calculation is probably wrong ! *Don't* go further, re-check it and if you don't find the error, please renounce !
+
+Search / correction follows the same scheme as for simple partitions :
 * find possible impacted files with `debugfs` (`icheck <fs block nb>`, then `ncheck <icheck nb>`).
-* reallocate bad block writing zeros in it, ''using the fs block size'' :
+* reallocate bad block writing zeros in it, ''using the fs block size'' :
 
 {{{
@@ -643 +987 @@
 }}}
 
-Et voilà !
-
-=== Bad block reassignment ===
-
-The SCSI disk command set and associated disk architecture are assumed in this section. SCSI disks have their own logical to physical mapping allowing a damaged sector (usually carrying 512 bytes of data) to be remapped irrespective of the operating system, file system or software RAID being used.
-
-The terms ''block and sector'' are used interchangeably, although block tends to get used in higher level or more abstract contexts such as a ''logical block''.
-
-When a SCSI disk is formatted, defective sectors identified during the manufacturing process (the so called primary list: PLIST), those found during the format itself (the certification list: CLIST), those given explicitly to the format command (the DLIST) and optionally the previous grown list (GLIST) are not used in the logical block map. The number (and low level addresses) of the unmapped sectors can be found with the `READ DEFECT DATA SCSI` command.
-
-SCSI disks tend to be divided into zones which have spare sectors and perhaps spare tracks, to support the logical block address mapping process. The idea is that if a logical block is remapped, the heads do not have to move a long way to access the replacement sector. Note that spare sectors are a scarce resource.
-
-Once a SCSI disk format has completed successfully, other problems may appear over time. These fall into two categories:
-
-* recoverable: the Error Correction Codes (ECC) detect a problem but it is small enough to be corrected. Optionally other strategies such as retrying the access may retrieve the data.
-* unrecoverable: try as it may, the disk logic and ECC algorithms cannot recover the data. This is often reported as a ''medium error''.
-
-Other things can go wrong, typically associated with the transport and they will be reported using a term other than ''medium error''. For example a disk may decide a read operation was successful but a computer's host bus adapter (HBA) checking the incoming data detects a CRC error due to a bad cable or termination.
-
-Depending on the disk vendor, recoverable errors can be ignored. After all, some disks have up to 68 bytes of ECC above the payload size of 512 bytes so why use up spare sectors which are limited in number ^[#footnote8 [8]]^ ? If the disk can recover the data and does decide to re-allocate (reassign) a sector, then first it checks the settings of the `ARRE` and `AWRE` bits in the read-write error recovery mode page. Usually these bits are set ^[#footnote9 [9]]^ enabling automatic (read or write) re-allocation. The automatic re-allocation may also fail if the zone (or disk) has run out of spare sectors.
-
-Another consideration with RAIDs, and applications that require a high data rate without pauses, is that the controller logic may not want a disk to spend too long trying to recover an error.
-
-Unrecoverable errors will cause a ''medium error'' sense key, perhaps with some useful additional sense information. If the extended background self test includes a full disk read scan, one would expect the self test log to list the bad block, as shown in section [#Repairsinafilesystem Repairs in a file system]. Recent SCSI disks with a periodic background scan should also list unrecoverable read errors (and some recoverable errors as well). The advantage of the background scan is that it runs to completion while self tests will often terminate at the first serious error.
-
-SCSI disks expect unrecoverable errors to be fixed manually using the `REASSIGN BLOCKS SCSI` command since loss of data is involved. It is possible that an operating system or a file system could issue the `REASSIGN BLOCKS` command itself but the authors are unaware of any examples. The `REASSIGN BLOCKS` command will reassign one or more blocks, attempting to (partially ?) recover the data (a forlorn hope at this stage), fetch an unused spare sector from the current zone while adding the damaged old sector to the GLIST (hence the name ''grown'' list). The contents of the GLIST may not be that interesting but `smartctl` prints out the number of entries in the grown list and if that number grows quickly, the disk may be approaching the end of its useful life.
-
-Here is an alternate brute force technique to consider: if the data on the SCSI or ATA disk has all been backed up (e.g. is held on the other disks in a RAID 5 enclosure), then simply reformatting the disk may be the least cumbersome approach.
-
-==== Example ====
-
-Given a ''bad block'', it still may be useful to look at the `fdisk` command (if the disk has multiple partitions) to find out which partition is involved, then use `debugfs` (or a similar tool for the file system in question) to find out which, if any, file or other part of the file system may have been damaged. This is discussed in section [#Repairsinafilesystem Repairs in a file system].
-
-Then a program that can execute the `REASSIGN BLOCKS SCSI` command is required. In Linux (2.4 and 2.6 series), FreeBSD, Tru64(OSF) and Windows the author's `sg_reassign` utility in the `sg3_utils` package can be used. Also found in that package is `sg_verify` which can be used to check that a block is readable.
-
-Assume that `logical block address 1193046` (which is `123456` in hex) is corrupt ^[#footnote10 [10]]^ on the disk at `/dev/sdb`. A long selftest command like `smartctl -t long /dev/sdb` may result in log results like this:
-
-{{{
-# smartctl -l selftest /dev/sdb
-smartctl version 5.37 [i686-pc-linux-gnu] Copyright (C) 2002-6 Bruce Allen
-Home page is http://smartmontools.sourceforge.net/
-SMART Self-test log
-Num  Test              Status            segment  LifeTime  LBA_first_err [SK ASC ASQ]
-     Description                         number   (hours)
-# 1  Background long   Failed in segment      -     354           1193046 [0x3 0x11 0x0]
-# 2  Background short  Completed              -     323                 - [-   -    -]
-# 3  Background short  Completed              -     194                 - [-   -    -]
-}}}
-
-The `sg_verify` utility can be used to confirm that there is a problem at that address:
-
-{{{
-# sg_verify --lba=1193046 /dev/sdb
-verify (10):  Fixed format, current;  Sense key: Medium Error
- Additional sense: Unrecovered read error
-  Info fld=0x123456 [1193046]
-  Field replaceable unit code: 228
-  Actual retry count: 0x008b
-medium or hardware error, reported lba=0x123456
-}}}
-
-Now the GLIST length is checked before the block reassignment:
-
-{{{
-# sg_reassign --grown /dev/sdb
->> Elements in grown defect list: 0
-}}}
-
-And now for the actual reassignment followed by another check of the GLIST length:
-
-{{{
-# sg_reassign --address=1193046 /dev/sdb
-# sg_reassign --grown /dev/sdb
->> Elements in grown defect list: 1
-}}}
-
-The GLIST length has grown by one as expected. If the disk was unable to recover any data, then the ''new'' block at lba `0x123456` has vendor specific data in it. The `sg_reassign` utility can also do bulk reassigns, see `man sg_reassign` for more information.
-
-The `dd` command could be used to read the contents of the ''new'' block:
-
-{{{
-# dd if=/dev/sdb iflag=direct skip=1193046 of=blk.img bs=512 count=1
-}}}
-
-and a hex editor ^[#footnote11 [11]]^ used to view and potentially change the `blk.img` file. An altered `blk.img` file (or `/dev/zero`) could be written back with:
-
-{{{
-# dd if=blk.img of=/dev/sdb seek=1193046 oflag=direct bs=512 count=1
-}}}
-
-More work may be needed at the file system level, especially if the reassigned block held critical file system information such as a superblock or a directory.
-
-Even if a full backup of the disk is available, or the disk has been ''ejected'' from a RAID, it may still be worthwhile to reassign the bad block(s) that caused the problem (or simply format the disk (see `sg_format` in the `sg3_utils package`)) and re-use the disk later (not unlike the way a replacement disk from a manufacturer might be used).
-
-
-== Case Studies ==
-
-This section is intended to collect step-by-step descriptions of some real-life use cases.
-
-=== Recovering a (mostly) unreadable sector of a Notebook HDD ===
-
-This was done in March 2016 under Windows 7 using ''Cygwin''^[#footnote12 12.]^ ports of ''GNU ddrescue''^[#footnote13 13.]^ and ''The Sleuth Kit (TSK)''^[#footnote14 14.]^. All commands shown should work similar on other platforms and with other filesystems.
-
-==== Determine Logical Block Address of unreadable sector ====
-
-Examine smartctl output:
-{{{
-root:~# smartctl -x /dev/sdb
-smartctl 6.5 2016-02-29 r4227 [x86_64-w64-mingw32-win7-sp1] (daily-20160229)
-...
-Model Family:     SAMSUNG SpinPoint MP5
-Device Model:     SAMSUNG HM640JJ
-...
-Firmware Version: 2AK10001
-User Capacity:    640.135.028.736 bytes [640 GB]
-Sector Size:      512 bytes logical/physical
-Rotation Rate:    7200 rpm
-Form Factor:      2.5 inches
-...
-ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
-...
-  5 Reallocated_Sector_Ct   PO--CK   252   252   010    -    0
-...
-  9 Power_On_Hours          -O--CK   100   100   000    -    251  <=== See Self-test Log below
-...
-197 Current_Pending_Sector  -O--CK   100   100   000    -    1    <=== At least 1 bad sector
-...
-SMART Extended Comprehensive Error Log Version: 1 (2 sectors)
-Device Error Count: 351 (device log contains only the most recent 8 errors)
-...
-Error 351 [6] occurred at disk power-on lifetime: 251 hours (10 days + 11 hours)
-  When the command that caused the error occurred, the device was active or idle.
-
-  After command completion occurred, registers were:
-  ER -- ST COUNT  LBA_48  LH LM LL DV DC
-  -- -- -- == -- == == == -- -- -- -- --
-  40 -- 51 00 01 00 00 33 3f d8 a6 40 00  Error: UNC 1 sectors at LBA = 0x333fd8a6 = 859822246  <=== Its LBA
-
-  Commands leading to the command that caused the error were:
-  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
-  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
-  25 00 00 00 01 00 00 33 3f d8 a6 40 00     00:00:06.924  READ DMA EXT
-...
-
-SMART Extended Self-test Log Version: 1 (2 sectors)
-Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
-# 1  Short offline       Completed: read failure       90%       176         859822246  <=== Detected 75 power on hours ago
-}}}
-
-A read scan helps to verify the LBA and checks for other possible bad sectors
-(alternatively replace `/dev/null` by a file path to create a disk image):
-
-{{{
-root:~# ddrescue --ask --verbose --binary-prefixes --idirect --force /dev/sdb /dev/null disk.map
-GNU ddrescue 1.21-rc2
-About to copy 610480 MiBytes from /dev/sdb [SAMSUNG HM640JJ::...] to /dev/null [0].
-Proceed (y/N)? y
-...
-non-tried:        0 B,     errsize:      512 B,      run time:          2h
-  rescued: 610480 MiB,      errors:        1,  remaining time:         n/a
-percent rescued:  99.99%      time since last successful read:         20s
-Finished
-}}}
-
-The `ddrescue` map file now shows byte ranges of good and bad disk areas:
-
-{{{
-root:~# cat disk.map
-...
-#      pos        size      status
-  0x00000000  0x667FB14C00  +
-0x667FB14C00    0x00000200  -  <=== 512 bytes unreadable
-0x667FB14E00  0x2E8B541200  +
-}}}
-
-Translate the byte position to the LBA:
-
-{{{
-root:~# echo $((0x667FB14C00/512))
-859822246
-}}}
-
-Or convert the map file to a `badblocks` like list with `ddrescuelog` (part of recent versions of ''ddrescue'' package):
-
-{{{
-root:~# ddrescuelog --list-blocks=- disk.map
-859822246
-}}}
-
-Both match the LBA reported by `smartctl`.
-
-==== Find affected file ====
-
-Get start offset of affected partition:
-
-{{{
-root:~# fdisk --list /dev/sdb
-...
-Device     Boot Start        End    Sectors   Size Id Type
-/dev/sdb1          63 1250258624 1250258562 596.2G  7 HPFS/NTFS/exFAT
-}}}
-
-Get filesystem block (cluster) size if unknown (4096 in many cases):
-
-{{{
-root:~# fsstat /dev/sdb1
-...
-File System Type: NTFS
-...
-Sector Size: 512
-Cluster Size: 4096
-...
-}}}
-
-Calculate number of bad cluster as `(BAD_LBA - START_LBA) / SECTORS_PER_CLUSTER`:
-
-{{{
-root:~# echo $(((859822246-63)/8))
-107477772
-}}}
-
-Find inode (here: MFT entry) used by this cluster:
-
-{{{
-root:~# ifind -d 107477772 /dev/sdb1
-663-128-2
-}}}
-
-Print some info about this inode:
-
-{{{
-root:~# istat /dev/sdb1 663-128-2
-...
-Name: Backup_2015-12-17.zip
-Parent MFT Entry: 30    Sequence: 1
-Allocated Size: 4660039680      Actual Size: 4660039516
-Created:        2015-12-17 13:43:30.460000000 (CET)
-File Modified:  2015-12-17 13:46:19.647000000 (CET)
-...
-Type: $DATA (128-2)   Name: N/A   Non-Resident   size: 4660039516  init_size: 4660039516
-106950180 106950181 ...
-...
-107477772  <=== The bad cluster
-...
-108087884
-}}}
-
-Find full path of affected file:
-
-{{{
-root:~# ffind /dev/sdb1 663-128-2
-/Backups/2015/Backup_2015-12-17.zip
-}}}
-
-If the file is no longer needed, it could be overwritten in place and removed then. This is easy with `shred` from ''GNU coreutils'': `shred --iterations=1 --remove /PATH/TO/FILE`. This should reallocate the bad sector in most cases.
-
-==== Try to recover the bad sector ====
-
-Start with 100 read retries of the bad sector, write to `recovered.bin` if successful:
-
-{{{
-root:~# ddrescue --ask --verbose --binary-prefixes --idirect --retry=100 \
-                 --input-position=859822246s --output-position=0 --size=1s \
-                 /dev/sdb recovered.bin recovered.map
-...
-Current status
-     ipos: 419835 MiB, non-trimmed:        0 B,  current rate:      32 B/s
-     opos:        0 B, non-scraped:        0 B,  average rate:       4 B/s
-non-tried:        0 B,     errsize:        0 B,      run time:      1m 49s
-  rescued:      512 B,      errors:        0,  remaining time:         n/a
-percent rescued: 100.00%      time since last successful read:          0s
-Finished
-}}}
-
-We were very lucky:
-
-{{{
-root:~# cat recovered.map
-...
-#      pos        size      status
-  0x00000000  0x667FB14C00  ?
-0x667FB14C00  0x00000200    +  <=== Now OK!
-0x667FB14E00  0x2E8B541200  ?
-}}}
-
-Check whether the disk firmware took the chance to reallocate the sector using the recovered data:
-
-{{{
-root:~# dd skip=859822246 count=1 iflag=direct if=/dev/sdb of=test.bin
-dd: error reading ‘/dev/sdb’: Input/output error
-0+0 records in
-0+0 records out
-0 bytes (0 B) copied, 23.5006 s, 0.0 kB/s
-}}}
-
-No luck in this case. So overwrite the sector manually:
-
-{{{
-root:~# dd seek=859822246 count=1 oflag=direct if=recovered.bin of=/dev/sdb
-1+0 records in
-1+0 records out
-512 bytes (512 B) copied, 1.05331 s, 0.5 kB/s
-}}}
-
-Read data back and check:
-
-{{{
-root:~# dd skip=859822246 count=1 iflag=direct if=/dev/sdb of=test.bin
-1+0 records in
-1+0 records out
-512 bytes (512 B) copied, 0.0211745 s, 24.2 kB/s
-
-root:~# diff -s recovered.bin test.bin
-Files recovered.bin and test.bin are identical
-}}}
-
-Finally, run a SMART self-test and check its result:
-
-{{{
-root:~# smartctl -t short /dev/sdb
-...
-Sending command: "Execute SMART Short self-test routine immediately in off-line mode".
-...
-Please wait 2 minutes for test to complete.
-
-root:~# sleep 120 # :-)
-
-root:~# smartctl -x /dev/sdb
-...
-ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
-...
-  5 Reallocated_Sector_Ct   PO--CK   252   252   010    -    0   <=== Interesting...
-...
-  9 Power_On_Hours          -O--CK   100   100   000    -    252
-...
-197 Current_Pending_Sector  -O--CK   100   100   000    -    0   <=== As expected
-...
-SMART Extended Self-test Log Version: 1 (2 sectors)
-Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
-# 1  Short offline       Completed without error       00%       252         -          <=== Works again!
-# 2  Short offline       Completed: read failure       90%       176         859822246
-}}}
-
-Interestingly the `Reallocated_Sector_Ct` did not increase. Either the firmware did not record the reallocation or decided to reuse the original sector.
-
-Done!
+Et voilà !
 
 == Footnotes ==