bugGNU FreeIPMI - Bugs: bug #64792, Bad IPMI DCMI response from Huawei...

 
 

bug #64792: Bad IPMI DCMI response from Huawei and Xfusion BMCs

Submitter:  Ole Holm Nielsen <oleholmnielsen>
Submitted:  Thu 19 Oct 2023 08:15:00 AM UTC
   
 
Category:  None Severity:  3 - Normal
Priority:  5 - Normal Item Group:  None
Status:  None Privacy:  Public
Assigned to:  None Open/Closed:  Open
Operating System:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Mon 20 Nov 2023 03:06:19 PM UTC, comment #4: 


> But the very new Xfusion FusionOne HPC 1288H V6 servers's datasheet claim to support DCMI, so there may be hope?  My colleague and I would be happy to test, if there's some way to get more detailed information from ipmi-dcmi when it's probing the Xfusion BMC.


It's been a long time since I wrote that tool, I can't think of anything else that can be done, but I could be forgetting something.

Does any other dcmi software work on this board?  It may be worth pinging the vendor to verify if it supports DCMI.

Albert Chu <chu11>
Group administrator
Mon 20 Nov 2023 11:01:08 AM UTC, comment #3: 

Hi Albert,

comment #2:

> What kind of workaround are you thinking of?
>
> - A workaround to make DCMI work?  I don't think this is possible, as it simply appears your board does not support DCMI.


You may well be right regarding the old Huawei servers.  But the very new Xfusion FusionOne HPC 1288H V6 servers's datasheet claim to support DCMI, so there may be hope?  My colleague and I would be happy to test, if there's some way to get more detailed information from ipmi-dcmi when it's probing the Xfusion BMC.

> - Are you looking for `ipmi-dcmi` to not return an error if an an error occurs, so that the slurm logs do not get spammed?  I think such a workaround would be better served through a wrapper script around `ipmi-dcmi`.


The fine folks at SchedMD have made a fix so that spamming will no longer occur as of the next releases 23.02.7 and 23.11 of Slurm, see https://bugs.schedmd.com/show_bug.cgi?id=17938
We're really satisfied with this fix.

Ole Holm Nielsen <oleholmnielsen>
Sun 19 Nov 2023 06:33:06 PM UTC, comment #2: 

sorry for late reply as I was on leave.

What kind of workaround are you thinking of?

- A workaround to make DCMI work?  I don't think this is possible, as it simply appears your board does not support DCMI.

- Are you looking for `ipmi-dcmi` to not return an error if an an error occurs, so that the slurm logs do not get spammed?  I think such a workaround would be better served through a wrapper script around `ipmi-dcmi`.


Albert Chu <chu11>
Group administrator
Wed 15 Nov 2023 08:20:42 AM UTC, comment #1: 

Hi,

Would a WORKAROUND be feasible to implement for Huawei and Xfusion servers?  If so, how can we help by providing debugging information?

Thanks a lot,
Ole

Ole Holm Nielsen <oleholmnielsen>
Thu 19 Oct 2023 08:15:00 AM UTC, original submission:  

We have successfully integrated the development FreeIPMI version 1.7.0 in our Linux cluster with the Slurm resource manager.  My test is described in https://bugs.schedmd.com/show_bug.cgi?id=17639#c55 and I have documented the FreeIPMI setup in my Slurm Wiki page https://wiki.fysik.dtu.dk/Niflheim_system/Slurm_configuration/#freeipmi-issues

Now we would like to deploy Slurm including the FreeIPMI power monitoring, but we have discovered a snag:

We have 196 older Huawei XH620 V3 nodes (Intel Broadwell) whose BMC doesn't seem to support the IPMI DCMI extensions.  A colleague at another university has the same problem with brand new Xfusion FusionOne HPC 1288H V6 servers (Intel IceLake, essentially rebranded Huawei servers) even though the server's BMC is documented to support DCMI 1.5!

On the Huawei and Xfusion nodes we get this error message:

$ ipmi-dcmi --get-system-power-statistics
ipmi_cmd_dcmi_get_power_reading: command invalid or unsupported

Due to this error, Slurm logs (spams) every minute in slurmd.log "error: _get_dcmi_power_reading: get DCMI power reading failed"

I've tried to find out how to query the Huawei BMC with IPMI DCMI but I only get error messages:

$ ipmi-dcmi --get-dcmi-capability-info
ipmi_cmd_dcmi_get_dcmi_capability_info_supported_dcmi_capabilities: bad completion code

I also tried each of the WORKAROUNDS listed in the ipmi-dcmi manual page, but in every case they return the same error.

The debug option gives some details:

$ ipmi-dcmi --get-dcmi-capability-info --debug
=====================================================
Group Extension - Get DCMI Capability Info Request
=====================================================
[               1h] = cmd[ 8b]
[              DCh] = group_extension_identification[ 8b]
[               1h] = parameter_selector[ 8b]
=====================================================
Group Extension - Get DCMI Capability Info Response
=====================================================
[               1h] = cmd[ 8b]
[              D6h] = comp_code[ 8b]
ipmi_cmd_dcmi_get_dcmi_capability_info_supported_dcmi_capabilities: bad completion code

The non-DCMI commands seem to be working correctly.  For example, I can read the system power:

$ ipmi-sensors -t Power_Unit
ID  | Name         | Type       | Reading    | Units | Event
22  | Power        | Power Unit | 296.00     | W     | 'OK'
(lines deleted)

Question: Would a WORKAROUND be feasible to implement for Huawei and Xfusion servers?  If so, how can we help by providing debugging information?

Or is there some other way for getting the DCMI extensions to work?

Thanks a lot,
Ole

Ole Holm Nielsen <oleholmnielsen>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #55257:  bmc-info.log added by oleholmnielsen (2KiB - text/x-log - Output from bmc-info)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by chu11 (Posted a comment)
  • -email is unavailable- added by oleholmnielsen (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follows 1 latest change.

    Date Changed by Updated Field Previous Value => Replaced by
    2023-10-19 oleholmnielsen Attached File- Added bmc-info.log, #55257

    Back to the top

    Powered by Savane 3.13-4b4b.
    Corresponding source code