mainGNU FreeIPMI - Support: sr #110963, ipmi-sensors does not return fan...

 
 

sr #110963: ipmi-sensors does not return fan rpm on Lenovo SR650 v3 but ipmitool does

Submitter:  None
Submitted:  Wed 29 Nov 2023 11:53:05 AM UTC
   
 
Category:  ipmi-sensors Priority:  5 - Normal
Severity:  3 - Normal Status:  None
Privacy:  Public Assigned to:  None
Originator Email:  -email is unavailable- Open/Closed:  Open
Operating System:  GNU/Linux
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Mon 18 Dec 2023 12:15:01 AM UTC, comment #43: 

I got a potential candidate for merger in this branch

freeipmi-1-6-0-stable-lenovo_workaround

the workaround I've created is now called "altbridging"

Albert Chu <chu11>
Group administrator
Sat 02 Dec 2023 11:52:57 PM UTC, comment #42: 

Not a problem, take your time. I can work with what you've given me so far. I really appreciate it.

Derek Rubright <derekthetech>
Sat 02 Dec 2023 11:51:06 PM UTC, comment #41: 

Cool.  If you remove the "printfs" in libfreeipmi/sensor-read/sensor-read.c you should have something that can work for you temporarily.

I just have to debate how to do it.  I'm a little reluctant to change default behavior in case it breaks things for current users.  So perhaps another workaround flag.


Albert Chu <chu11>
Group administrator
Sat 02 Dec 2023 11:23:22 PM UTC, comment #40: 

Ha i didn't even check the output, just the debug. Yes it does look like it works! If you can fold it in somehow, is it possible to 'assume bmc owner' text string ? But really, whatever allows me to use it is perfect!

Derek Rubright <derekthetech>
Sat 02 Dec 2023 11:12:55 PM UTC, comment #39: 

i think it worked

> 292 | Fan 1 Front Tach | Fan  | 14924.00   | RPM   | 'OK'


discounting my debug prints, does it work for all sensors?

if so, i need to think of how to fold this in for real

Albert Chu <chu11>
Group administrator
Sat 02 Dec 2023 08:07:48 PM UTC, comment #38: 

Trying again

(file #55392)

Derek Rubright <derekthetech>
Sat 02 Dec 2023 06:57:59 PM UTC, comment #37: 

I notice in your debug output you always run out of a `Hyperscale/SR650v3` directory.  Are you installing the debug ipmi-sensors there?

Try running out of the cloned build directory instead.  I'm wondering if your ipmi-sensors could be picking up the system libfreeipmi by accident (vs using my hacked libfreeipmi in the build directory).

Albert Chu <chu11>
Group administrator
Sat 02 Dec 2023 06:27:29 PM UTC, comment #36: 

I get the same thing when I run that command in the cloned free-ipmi directory. I can send all of the output from when I did the complile and the make if that would help ?

Derek Rubright <derekthetech>
Sat 02 Dec 2023 06:23:28 PM UTC, comment #35: 

something isn't quite right, even my printf debug statements aren't appearing in your output.  Are my hacks in the branch you are building?  Here's what I get when I run git log.

```

>git log | head -n5

commit 1db6ac40eaecaf84d7615a5a2fd6f5b63919b2bc
Author: Albert Chu <chu11@llnl.gov>
Date:   Sat Dec 2 08:38:48 2023 -0800

    hacks
```



Albert Chu <chu11>
Group administrator
Sat 02 Dec 2023 05:01:31 PM UTC, comment #34: 

Re-cloned the repo, and did a re-make.

Attached new log

(file #55391)

Derek Rubright <derekthetech>
Sat 02 Dec 2023 04:40:26 PM UTC, comment #33: 

remember to "git fetch" to update the branch (or you can re-clone too).

Albert Chu <chu11>
Group administrator
Sat 02 Dec 2023 04:39:31 PM UTC, comment #32: 

crud, i'm sorry, I think I pushed the wrong code to that branch ... can you try again.  thanks.

Albert Chu <chu11>
Group administrator
Sat 02 Dec 2023 03:58:41 PM UTC, comment #31: 

Reuploaded

(file #55390)

Derek Rubright <derekthetech>
Sat 02 Dec 2023 03:39:53 PM UTC, comment #30: 

I think you forgot to run with --workaround-flags=assumebmcowner

Albert Chu <chu11>
Group administrator
Sat 02 Dec 2023 01:36:58 PM UTC, comment #29: 

Apologies for the delay

(file #55386)

Derek Rubright <derekthetech>
Fri 01 Dec 2023 09:32:05 PM UTC, comment #28: 

Could you show the --debug output for the fan sensor?  Maybe I messed something up.

Albert Chu <chu11>
Group administrator
Fri 01 Dec 2023 09:24:10 PM UTC, comment #27: 

I think I followed all the commands correctly, but when I ran the ipmi-sensors command I got the same information. I'm not sure if you incremented the version number to ensure I made it properly, but it is listed as 1.6.12

Derek Rubright <derekthetech>
Fri 01 Dec 2023 07:30:03 PM UTC, comment #26: 

Here's a hack on a branch ... just trying to see if this logic would work (hopefully no typos in these build instructions):

https://github.com/chu11/freeipmi-mirror/tree/lenovo_workaround

git clone https://github.com/chu11/freeipmi-mirror.git

cd freeipmi-mirror
git checkout lenovo_workaround
./autogen.sh
./configure --enable-debug
make
cd ipmi-sensors
./ipmi-sensors --workaround-flags=assumebmcowner


Albert Chu <chu11>
Group administrator
Fri 01 Dec 2023 06:38:59 PM UTC, comment #25: 

Ok, seems that’s the answer.  Lemme see if I can figure out a workaround.

Albert Chu <chu11>
Group administrator
Fri 01 Dec 2023 06:35:40 PM UTC, comment #24: 

root@sr630v3-2310:/home/jacob# ipmi-raw 0 0x04 0x2d 0x81
rcvd: 2D 00 00 C0 00 80

root@sr630v3-2310:/home/jacob# ipmi-raw 3 0x04 0x2d 0x81
rcvd: 2D 00 50 C0 C0

Derek Rubright <derekthetech>
Fri 01 Dec 2023 05:59:39 PM UTC, comment #23: 

so that sorta worked, although 0 rpm doesn't sound right.

looking through more code, i have a theory what might be going on.

on your board i see

```
172.16.40.114: [              10h] = sensor_owner_id[ 7b]
172.16.40.114: [               3h] = sensor_owner_lun[ 2b]
172.16.40.114: [               0h] = sensor_owner_lun.reserved[ 2b]
172.16.40.114: [               0h] = channel_number[ 4b]
```

the sensor owner is the typical one and the lun is != 0.

but in my code I have


```
if (slave_address == IPMI_SLAVE_ADDRESS_BMC && sensor_owner_lun == IPMI_BMC_IPMB_LUN_BMC)
   don't do bridging
else
   do bridging

```

which short answer, I do "bridging" when the address & lun aren't the BMC, but it appears ipmitool may not.  It may only do bridging when the address is not the BMC.

My assumption is that the bridging I do just happens to work on most motherboards, but it doesn't on yours (thus the 0xC3 error we saw earlier).  But it does work the way ipmitool does it.

I'm not sure how to work around this for the moment, but lets see if we can verify.

ipmi-raw 0 0x04 0x2d 0x81

should emulate what i'm doing now

ipmi-raw 3 0x04 0x2d 0x81

should emulate what ipmitool is doing

my hope is that we'll see

(first case)
rcvd: 2D 00 00 XX XX

(second case)
rcvd: 2D 00 NN XX XX


where NN will not be 0 in the second case.



Albert Chu <chu11>
Group administrator
Fri 01 Dec 2023 05:12:05 PM UTC, comment #22: 

Attached

(file #55385)

Derek Rubright <derekthetech>
Fri 01 Dec 2023 04:21:23 PM UTC, comment #21: 

Could you try the "assumebmcowner" workaround (without --bridge-sensors).  AFAICT, ipmitool is not "bridging", even though they should.

Albert Chu <chu11>
Group administrator
Fri 01 Dec 2023 02:30:14 PM UTC, comment #20: 

Just to add an additional detail, no sensor information gets return from this system, except for the coin battery voltage. Nothing else does. I attached a spreadsheet that shows the difference between ipmitool and ipmi-sensors

(file #55384)

Derek Rubright <derekthetech>
Fri 01 Dec 2023 01:18:58 PM UTC, comment #19: 

Hey Albert, thanks for all the feedback. I'm working to get those commands rerun.

So ipmitool is working just fine, and is returning the appropriate RPM values that I'm seeing from the web-based BMC, so I'm not so concerned about ipmitool.

Attached are oob debug commands, one with bridge sensors and one without



(file #55382, file #55383)

Derek Rubright <derekthetech>
Fri 01 Dec 2023 05:58:51 AM UTC, comment #18: 

Actually I just noticed something in your out of band debug output.

```

>> Sending IPMI command payload
>>    netfn   : 0x04
>>    command : 0x2d

BUILDING A v2 COMMAND
Local RqAddr 0x20 transit 0:0 target 0x20:0 bridgePossible 1
<< IPMI Response Session Header
<<   Authtype                : RMCP+
<<   Payload type            : IPMI (0)
<<   Session ID              : 0xa0a2a3a4
<<   Sequence                : 0x00000309
<<   IPMI Msg/Payload Length : 32
<< IPMI Response Message Header
<<   Rq Addr    : 81
<<   NetFn      : 05
<<   Rq LUN     : 0
<<   Rs Addr    : 20
<<   Rq Seq     : 0f
<<   Rs Lun     : 0
<<   Command    : 2d
<<   Compl Code : 0x00

```

This is the "get sensor reading" command, and it apparently doesn't return an error (completion code == 0).


Could you do the out of band --debug --bridge-sensors output again?  Your debug output of that before didn't connect to the server.

i.e. ipmi-sensors -r NUMBER --debug --bridge-sensors -h host -u user -p password

Albert Chu <chu11>
Group administrator
Fri 01 Dec 2023 12:23:37 AM UTC, comment #17: 

That said, why your motherboard is returning errors for that sensor is something you'll probably want to bring up with your vendor.  Could be a firmware issue or otherwise.

Albert Chu <chu11>
Group administrator
Fri 01 Dec 2023 12:22:59 AM UTC, comment #16: 

At this point I'm unconvinced that ipmitool is doing the right thing.  It appears when the 0xC3 error occurs when reading a sensor, ipmitool will output data under the assumption the reading was valid.

I've submitted a bug to ipmitool on the mailing list for the time being.  We can see if anyone responds.


Albert Chu <chu11>
Group administrator
Thu 30 Nov 2023 11:10:34 PM UTC, comment #15: 

I looked through the ipmitool code and one chunk looks very concerning in `ipmi_sdr_read_sensor_value()`

```
        if (rsp->ccode) {                                                                                                                  
                if ( !((sr.full    && rsp->ccode == 0xcb) ||                                                                               
                       (sr.compact && rsp->ccode == 0xcd)) ) {                                                                             
                        lprintf(LOG_DEBUG,                                                                                                 
                                "Error reading sensor %s (#%02x): %s", sr.s_id,                                                            
                                sensor->keys.sensor_num,                                                                                   
                                val2str(rsp->ccode, completion_code_vals));                                                                
                }                                                                                                                          
                return &sr;                                                                                                                
        }   
```

It seems to suggest that if an error occurs during the sensor reading, output a debug message (if necessary), and then return the sensor reading as though it were successful.

So this may be a bug in ipmitool in which it gets an error, but ignores the error. 

What confidence do you have in the RPM values actually being correct?

The RPM may simply be junk data being output.


Albert Chu <chu11>
Group administrator
Thu 30 Nov 2023 08:53:24 PM UTC, comment #14: 

Unfortunately the ipmitool debug did not have much (they seem to exclude packet dumps around the area I care about).

Lemme peruse their code and see if there's something obvious they are doing.  Perhaps it's a workaround of sorts for Lenovo.

Can you tell me the OS and version of ipmitool you're using.

Albert Chu <chu11>
Group administrator
Thu 30 Nov 2023 08:03:46 PM UTC, comment #13: 

The -D made know difference. Attached are the in-band and out of band commands

(file #55379, file #55380)

Derek Rubright <derekthetech>
Thu 30 Nov 2023 06:09:10 PM UTC, comment #12: 

Hmmm I see

> Sensor reading/event_bitmask retrieval error: sensor reading
> cannot be obtained


The error code is 0xC3 indicating a timeout.

question, you are using the linux ipmi driver, right?  Normally reading from /dev/ipmi0?  If by chance it's not using it, you can force it with "ipmi-sensors -D openipmi"

could you get the debug output from ipmitool to show how it's getting the sensor reading?  Perhaps there will be a clue.

I think `ipmitool sdr -vvv get "Fan 1 Front Tach"` should get it.

(If possible both inband and outofband ... unfortunately, it doesn't appear ipmitool outputs a lot of debug info with inband).


Albert Chu <chu11>
Group administrator
Thu 30 Nov 2023 05:22:50 PM UTC, comment #11: 

A colleague of mine ran the debug commands in band on his SR650 v3 system. He used the id that matched with the fan 1 front tach. Hopefully it helps

(file #55376, file #55377)

Derek Rubright <derekthetech>
Wed 29 Nov 2023 09:54:21 PM UTC, comment #10: 

Your new debugs failed to connect to the server.  Some random mistake I assume.

Albert Chu <chu11>
Group administrator
Wed 29 Nov 2023 08:42:15 PM UTC, comment #9: 

Ah, sorry. I had to replace the board in hopes of getting the video to work. I see what happened. I changed it to 292 now, which matched the Fan 1 Front Tach (which was 260 originally).

(file #55373, file #55374)

Derek Rubright <derekthetech>
Wed 29 Nov 2023 08:27:11 PM UTC, comment #8: 


> 260 | PVNN PCH AUX PG | Voltage | N/A        | N/A   | Unknown


Are you running against a different motherboard?  I used the id of 260 based on your original screenshot.

Not sure if we need to start over with the original debug output.

Albert Chu <chu11>
Group administrator
Wed 29 Nov 2023 08:20:33 PM UTC, comment #7: 

I can't do this in-band. For some reason my video output isn't working either locally or remote.

Attached new debug log

(file #55372)

Derek Rubright <derekthetech>
Wed 29 Nov 2023 07:15:43 PM UTC, comment #6: 

Could you provide the --debug for --bridge-sensors.  like

ipmi-sensors --record-ids=260 --debug --bridge-sensors

(if possible could you do inband, so the extra out-of-band packet stuff isn't there)

Albert Chu <chu11>
Group administrator
Wed 29 Nov 2023 06:18:46 PM UTC, comment #5: 

No, that doesn't work either

Derek Rubright <derekthetech>
Wed 29 Nov 2023 06:17:08 PM UTC, comment #4: 

Does --bridge-sensors work?

Albert Chu <chu11>
Group administrator
Wed 29 Nov 2023 06:14:18 PM UTC, comment #3: 

I have attempted using the workaround, as I found it in an earlier bug. Unfortunately that didn't work, as these sensors are coded for RPM and not assert/deassert.

I ran the debug command as requested, and have attached it here.

(file #55371)

Derek Rubright <derekthetech>
Wed 29 Nov 2023 05:36:57 PM UTC, comment #2: 

Hi, does the "discretereading" workaround work, i.e. --workaround-flags=discretereading.  I have noticed this issue before only on HP motherboards.

If not, could you send me some debug info.  For just one sensor would be enough, like

ipmi-sensors --debug --record-ids=260

and we can see what's going on.  Perhaps it's a new problem that hasn't been seen before.

Albert Chu <chu11>
Group administrator
Wed 29 Nov 2023 11:54:43 AM UTC, comment #1: 

I submitted this before I created an account. Hopefully that doesn't lead to problems reviewing this.

Derek Rubright <derekthetech>
Wed 29 Nov 2023 11:53:05 AM UTC, original submission:  

While working on a Lenovo SR650 v3 system, I attempted to pull fan RPM info via ipmi-sensors command (both in-band and out-of-band), and neither returned the information. I was able to use ipmitool and was able to get the info. I have attached screenshots showing the differences.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #55384:  ipmi_differences.xlsx added by derekthetech (25KiB - application/vnd.openxmlformats-officedocument.spreadsheetml.sheet)
file #55379:  ipmitool-ib.txt added by derekthetech (45KiB - text/plain)
file #55380:  ipmitool-oob.txt added by derekthetech (501KiB - text/plain)
file #55371:  ipmi_sensors.txt added by derekthetech (57KiB - text/plain)
file #55369:  Screenshot@2023-11-28@120831.png added by None (122KiB - image/png)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by chu11 (Posted a comment)
  • -email is unavailable- added by derekthetech (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 17 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2023-12-02 derekthetech Attached File- Added dr_ipmi-sensors-debug-id292_v4.txt, #55392
    2023-12-02 derekthetech Attached File- Added dr_ipmi-sensors-debug-id292_v3.txt, #55391
    2023-12-02 derekthetech Attached File- Added dr_ipmi-sensors-debug-id292_v2.txt, #55390
    2023-12-02 derekthetech Attached File- Added dr_ipmi-sensors-debug-id292.txt, #55386
    2023-12-01 derekthetech Attached File- Added ipmi-sensor-assumebmcowner-debug.txt, #55385
    2023-12-01 derekthetech Attached File- Added ipmi_differences.xlsx, #55384
    2023-12-01 derekthetech Attached File- Added ipmi-sensors-oob-bridge.txt, #55382
        Attached File- Added ipmi-sensors-oob.txt, #55383
    2023-11-30 derekthetech Attached File- Added ipmitool-ib.txt, #55379
        Attached File- Added ipmitool-oob.txt, #55380
    2023-11-30 derekthetech Attached File- Added jl_ipmi_sensors-debug-id289.txt, #55376
        Attached File- Added jl_ipmi_sensors-debug-id289-bridge.txt, #55377
    2023-11-29 derekthetech Attached File- Added ipmi_sensors-debug-id292.txt, #55373
        Attached File- Added ipmi_sensors-debug-id292-bridge.txt, #55374
    2023-11-29 derekthetech Attached File- Added ipmi_sensors-bridge-sensors-debug.txt, #55372
    2023-11-29 derekthetech Attached File- Added ipmi_sensors.txt, #55371
    2023-11-29 None Attached File- Added Screenshot@2023-11-28@120831.png, #55369

    Back to the top

    Powered by Savane 3.13-02a9.
    Corresponding source code