Opened 5 days ago

Last modified 8 hours ago

#1213 new enhancement

Access smart data from devices behind Megaraid software stack (ESRT2)

Reported by: AndyLinux Owned by:
Priority: minor Milestone: undecided
Component: all Version: 6.6
Keywords: megaraid linux Cc:

Description

For a long time Intel platforms have supported ESRT2 which basically runs the LSI megaraid stack as a kernel module talking thru the chipset which has to be placed into ESRT2 mode.

This has the advantages that the platform operates as though it had a real RAID card and anything talking to that (e.g. the LSI/Avago/Broadcom tools MegaCLI, storcli, CmdTool264 etc.) just work unchanged.

Unfortunately smartctl does not appear to work.

I'm using an Intel S2600WF system with the latest ESRT2 drivers on Linux, using a 4.9 kernel in a custom distribution. I've tried with the RHEL6 smartmontools package (6.6.2) and don't notice any obvious changes between this version and smartmontools 7 Megaraid support in the change history.

The first problem is that the the megasr kernel module creates its device node at /dev/megaraid_swr_ioctl_node not the normal /dev/megaraid_sas_ioctl_node - this is worked around using a symlink or a secondary device node with the same major/minor numbers.

Once this is done, the smartctl fails in a fairly blunt fashion:

# smartctl -a -d sat+megaraid,0 -T permissive /dev/bus/0
smartctl 6.6 2017-11-05 r4594 [x86_64-linux-4.9.125-2.11.0.12.x86_64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

Read Device Identity failed: megasas_cmd result: 0.0 = 25/255

=== START OF INFORMATION SECTION ===
Device Model:     [No Information Found]
Serial Number:    [No Information Found]
Firmware Version: [No Information Found]
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   [No Information Found]
Local Time is:    Thu Jul 11 15:21:01 2019 UTC
SMART support is: Ambiguous - ATA IDENTIFY DEVICE words 82-83 don't show if SMART supported.
SMART support is: Ambiguous - ATA IDENTIFY DEVICE words 85-87 don't show if SMART is enabled.
A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options.

Adding the -r ioctl,3 options simple shows a dump of a frame full of 0's.

The smartctl scan options show this info:

# smartctl --scan -r ioctl,3
glob(3) found no matches for pattern /dev/hd[a-t]
sda -> /sys/class/scsi_host/host0/proc_name: "megasr"
glob(3) found no matches for pattern /dev/sd[a-c][a-z]
glob(3) found no matches for pattern /dev/nvme[0-9]
glob(3) found no matches for pattern /dev/nvme[1-9][0-9]
/dev/sda -d scsi # /dev/sda, SCSI device
# smartctl --scan-open -r ioctl,3
glob(3) found no matches for pattern /dev/hd[a-t]
sda -> /sys/class/scsi_host/host0/proc_name: "megasr"
glob(3) found no matches for pattern /dev/sd[a-c][a-z]
glob(3) found no matches for pattern /dev/nvme[0-9]
glob(3) found no matches for pattern /dev/nvme[1-9][0-9]
>>>> do_scsi_cmnd_io: sg_io_ver=3
 [inquiry: 12 00 00 00 24 00 ]
  scsi_status=0x0, sg_transport_status=0x0, sg_driver_status=0x0
  sg_info=0x0  sg_duration=0 milliseconds  resid=0
  Incoming data, len=36:
 00     00 00 05 02 fa 00 00 32  49 6e 74 65 6c 20 20 20
 10     4d 65 67 61 53 52 20 20  00 00 00 00 00 00 00 00
 20     31 2e 30 20
/dev/sda -d scsi # /dev/sda, SCSI device

I suspect that the problem relates to the fact that unlike normal Megaraid controllers, the drives do not show as being in an enclosure, with the software raid the enclosure is N/A:

# CmdTool264 pdlist a0 | grep -i -e slot -e id
Enclosure Device ID: N/A
Slot Number: 0
Device Id: 0
Media Type: Solid State Device
Enclosure Device ID: N/A
Slot Number: 1
Device Id: 1
Media Type: Solid State Device

If I boot the system using a Live CD and don't load the megasr module, the AHCI module will find the devices and report the SMART data quite happily.

I won't be able to provide debug access but it's not a production system so can run whatever debug commands are required to provide more info as needed.

Attachments (2)

os_linux-lsraid_ioc-test.patch (1.2 KB) - added by Christian Franke 3 days ago.
Patch for os_linux.cpp to do a minimal test of LSRAID_IOC_FIRMWARE
strace.cmdtool264.txt (49.1 KB) - added by AndyLinux 31 hours ago.
gdb capture of ioctl calls for "CmdTool264 pdinfo physdrv [:0] a0"

Download all attachments as: .zip

Change History (12)

comment:1 Changed 5 days ago by Christian Franke

Component: smartctlall
Keywords: megaraid linux added
Milestone: undecided
Priority: majorminor

comment:2 Changed 5 days ago by Christian Franke

Is the source code of the megasr kernel module available?
Is there any evidence that the module implements MEGASAS_IOC_FIRMWARE?

comment:3 in reply to:  2 Changed 5 days ago by AndyLinux

Replying to Christian Franke:

Is the source code of the megasr kernel module available?

Only partially, it is a mostly closed source module except for the inevitable shim layer. It can be found at:
https://downloadcenter.intel.com/download/28304/Intel-Embedded-Server-RAID-Technology-2-ESRT2-RAID-driver-for-Linux-?v=t

Is there any evidence that the module implements MEGASAS_IOC_FIRMWARE?

I have no idea; looking in the above code I see a token LSRAID_IOC_FIRMWARE used, defined as:

#define LSRAID_IOC_FIRMWARE	_IOWR('R', 1, struct megasas_iocpacket)

Which I guess is similar but different to that in the kernel code:

#define MEGASAS_IOC_FIRMWARE	_IOWR('M', 1, struct megasas_iocpacket)

If this will show up in an strace of the LSI tools being used, then I can provide that, if that helps? Let me know what CmdTool264 etc. command to use and I can grab a trace.

comment:4 Changed 4 days ago by AndyLinux

I had a quick look at an strace comparison between a 'CmdTool? pdinfo' command on a real RAID system and a megasr software RAID system.

There are 2 basic differences:

  1. It notices the difference in /proc/devices between "250 megaraid_sas_ioctl" and "250 megaraid_swr_ioctl" as an indication that it's talking to a software RAID controller, and uses that to determine the correct /dev/ node to open, as noted above
  2. It uses subtly different ioctl's - hardware RAID shows a sequence of ioctl(0x3, 0xc1944d01, 0x7f6d9f452000) whereas software RAID shows ioctl(0x3, 0xc1945201, 0x7f3485c52000) - where of course the 3rd octet is 4d or 52 - 'M' or 'R' from the above #define's

If you can give me some ideas on how to see the contents of the ioctl memory, I can forward that if it helps.

Looking at the structures referenced in the #define values above, the structures appear identical.

comment:5 Changed 3 days ago by Alex Samorukov

You can get ioctl content with a number of ways.

  1. If it is non-staic binary - just use LD_PRELOAD to override ioctl
  2. If it is static - use any kernel-level debugger (stap?) to do the same

However, without documentation that would be a lot of work.

Changed 3 days ago by Christian Franke

Patch for os_linux.cpp to do a minimal test of LSRAID_IOC_FIRMWARE

comment:6 in reply to:  4 Changed 3 days ago by Christian Franke

Summary: Cannot access smart data from devices behind Megaraid software stackAccess smart data from devices behind Megaraid software stack (ESRT2)

comment:7 Changed 3 days ago by Christian Franke

Replying to AndyLinux:

... hardware RAID shows a sequence of ioctl(0x3, 0xc1944d01, 0x7f6d9f452000) whereas software RAID shows ioctl(0x3, 0xc1945201, 0x7f3485c52000) ...

Result of _IOWR() expansion:

#define MEGASAS_IOC_FIRMWARE 0xc1944d01UL
#define LSRAID_IOC_FIRMWARE  0xc1945201UL

These values actually match the strace result. The struct size and buffer addresse are identical in both calls. This suggests that both I/O-controls may only differ in ioctl code and device node.

If possible, please apply os_linux-lsraid_ioc-test.patch to smartmontools source, build smartctl and try it on a test(!) machine. Patch should work for 6.6 and 7.0 tarballs and current SVN. Test could be run from build directory, there is no need to run make install. The patch does neither include creation of missing /dev/megaraid_swr_ioctl_node nor device scan support.

comment:8 Changed 35 hours ago by AndyLinux

Thanks for the prompt patch.

I tried this with 6.6, fixed a minor typo, tweaked the patch to replace all instances of MEGASAS_IOC_FIRMWARE with LSIRAID_IOC_FIRMWARE, also all instances of megaraid_sas_ioctl_node with megaraid_swr_ioctl_node; it's close but not quite functioning yet:

# smartctl -a -d megaraid,0 -T permissive -r ioctl,4 /dev/bus/0
smartctl 6.6 2019-06-19 r4923 [x86_64-linux-4.9.125-2.11.0.12.x86_64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

 [inquiry: 12 00 00 00 24 00 ]
Got MegaRAID inquiry.. ATA     Micron_5200_MTFDU004
 [ata pass-through(16): 85 08 0e 00 00 00 01 00 00 00 00 00 00 00 ec 00 ]
sat_device::ata_pass_through: scsi_pass_through() failed, errno=5 [megasas_cmd result: 0.0 = 0/35]
 [inquiry: 12 01 00 00 fc 00 ]
 [inquiry: 12 00 00 00 24 00 ]
 [read capacity(10): 25 00 00 00 00 00 00 00 00 00 ]
User Capacity:        960,197,124,096 bytes [960 GB]
Logical block size:   512 bytes
 [mode sense(6): 1a 00 84 00 40 00 ]
 [mode sense(6): 1a 00 1c 00 40 00 ]
Vital Product Data (VPD) INQUIRY failed [3]
Vital Product Data (VPD) INQUIRY failed [3]
Device type:          CD/DVD
 [mode sense(6): 1a 00 19 00 40 00 ]
Local Time is:        Mon Jul 15 11:04:57 2019 UTC
 [test unit ready: 00 00 00 00 00 00 ]
SMART support is:     Unavailable - device lacks SMART capability.
 [Input/output error]

=== START OF READ SMART DATA SECTION ===
 [log sense: 4d 00 40 00 00 00 00 00 04 00 ]
 [log sense: 4d 00 40 00 00 00 00 00 06 00 ]
 [log sense: 4d 00 6f 00 00 00 00 00 04 00 ]
 [log sense: 4d 00 6f 00 00 00 00 00 0c 00 ]
 [request sense: 03 00 00 00 12 00 ]
 [log sense: 4d 00 4d 00 00 00 00 00 04 00 ]
 [log sense: 4d 00 4d 00 00 00 00 00 10 00 ]

 [log sense: 4d 00 4d 00 00 00 00 00 04 00 ]
 [log sense: 4d 00 4d 00 00 00 00 00 10 00 ]
Current Drive Temperature:     29 C
Drive Trip Temperature:        0 C

Error Counter logging not supported

 [mode sense(6): 1a 00 0a 00 40 00 ]
Device does not support Self Test logging

If I use sat+megaraid,0 then I see this output - I've removed the identify packet dumps as they are all 0's.

# smartctl -a -d sat+megaraid,0 -T permissive -r ioctl,4 /dev/bus/0
smartctl 6.6 2019-06-19 r4923 [x86_64-linux-4.9.125-2.11.0.12.x86_64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org


REPORT-IOCTL: Device=/dev/bus/0 Command=IDENTIFY DEVICE
 Input:   FR=...., SC=0x01, LL=...., LM=...., LH=...., DEV=...., CMD=0xec IN
 [ata pass-through(16): 85 08 0e 00 00 00 01 00 00 00 00 00 00 00 ec 00 ]
sat_device::ata_pass_through: scsi_pass_through() failed, errno=5 [megasas_cmd result: 0.0 = 0/35]
REPORT-IOCTL: Device=/dev/bus/0 Command=IDENTIFY DEVICE returned -1 errno=5 [megasas_cmd result: 0.0 = 0/35]

===== [IDENTIFY DEVICE] DATA START (BASE-16) =====
...
===== [IDENTIFY DEVICE] DATA END (512 Bytes) =====


REPORT-IOCTL: Device=/dev/bus/0 Command=IDENTIFY PACKET DEVICE
 Input:   FR=...., SC=0x01, LL=...., LM=...., LH=...., DEV=...., CMD=0xa1 IN
 [ata pass-through(16): 85 08 0e 00 00 00 01 00 00 00 00 00 00 00 a1 00 ]
sat_device::ata_pass_through: scsi_pass_through() failed, errno=5 [megasas_cmd result: 0.0 = 0/35]
REPORT-IOCTL: Device=/dev/bus/0 Command=IDENTIFY PACKET DEVICE returned -1 errno=5 [megasas_cmd result: 0.0 = 0/35]

===== [IDENTIFY PACKET DEVICE] DATA START (BASE-16) =====
...
===== [IDENTIFY PACKET DEVICE] DATA END (512 Bytes) =====

Read Device Identity failed: megasas_cmd result: 0.0 = 0/35
(followed by empty information section)

It looks like the CmdTool2 binary is dynamically linked against glibc so I will see if I can use gdb to capture some ioctl content, unless there's anything else I can try?

comment:9 Changed 31 hours ago by AndyLinux

I managed to use gdb to capture the ioctl parameters when running CmdTool264 pdinfo physdrv [:0] a0 (which should return the data about the same drive as identified above)

For each time ioctl is called, it dumps the ioctl megasas_ioc_packet and 128 bytes pointed to by the first sgl entry in that ioc_packet, and then dumps them again after the syscall returns.

Hope this is of use, let me know if there's more data that can be gathered.

Changed 31 hours ago by AndyLinux

Attachment: strace.cmdtool264.txt added

gdb capture of ioctl calls for "CmdTool264 pdinfo physdrv [:0] a0"

comment:10 Changed 8 hours ago by Christian Franke

Most cmd types (at offset 0x14) are MFI_CMD_DCMD (0x05), only two commands are actual pass-through types (MFI_CMD_PD_SCSI_IO, 0x04).

The first pass-through command is a SAT ATA PASS-THROUGH-12 (0xa1) with a CDB for ATA IDENTIFY DEVICE (0xec). A quick look shows two fields which differ from smartctl:

  1. megasas_pthru_frame.sense_buf_phys_addr_lo/hi is set to some sense buffer address.
  2. megasas_pthru_frame.context is set to some magic value.
825	0x7ffff5e52010:	0x00	0x00	0x00	0x00	[0x04]	0x00	0xff	0x00  <== cmd
826	0x7ffff5e52018:	0x00	0x00	0x0c	0x01	[0x00	0xa2	0xe6	0xf5] <== context
827	0x7ffff5e52020:	0xff	0x7f	0x00	0x00	0x14	0x00	0x1e	0x00
828	0x7ffff5e52028:	0x00	0x02	0x00	0x00	[0x9c	0x70	0xe6	0xf5] <== sense_buf_phys_addr_lo
829	0x7ffff5e52030:	[0xff	0x7f	0x00	0x00]	[0xa1	0x08	0x0b	0x00] <== "_hi, cdb[0..3]
830	0x7ffff5e52038:	[0x00	0x00	0x00	0x00	0x00	0xec	0x00	0x00] <== cdb[4..11]

The second pass-through command is SCSI LOG SENSE (0x4d). The context is set to a slightly different value:

948	0x7ffff5e52018:	0x00	0x00	0x0a	0x01	[0x00	0xa4	0xe6	0xf5] <== context

So a nonzero context may be required.

BTW: Output data is missing for the pass-through commands because gdb interpreted megasas_pthru_frame.cdb[8..15] as the buffer address:

830	0x7ffff5e52038:	0x00	0x00	0x00	0x00	[0x00	0xec	0x00	0x00
831	0x7ffff5e52040:	0x00	0xb1	0x0d	0x54]	0xbc	0x70	0xe6	0xf5
...
843	0x540db1000000ec00:	runmegacli.gdb:24: Error in sourced command file:
844	Cannot access memory at address 0x540db1000000ec00
Note: See TracTickets for help on using tickets.