I found that in openssl-1.1.1, before calling the fgets function to read stdin, setbuf(stdin, NULL) will be called

I wrote a simple demo to simulate this situation:

    $ cat main_pipe.cpp
    #include <string.h>
    #include <fcntl.h>
    #include <unistd.h>
    #include <iostream>
    #include <fstream>
    #include <sstream>
    #include <stdio.h>

    using namespace std;

    int main() {
        int herepipe[2];

        if (pipe(herepipe) < 0) {
            cout << "pipe file failed" << endl;
        }

        string read_line = "first line\nsecond line\nthird line\n";

        write(herepipe[1], read_line.c_str(), read_line.length());

        close(herepipe[1]);

        dup2(herepipe[0], 0);

        char read_str[256] = { 0 };

        while (fgets(read_str, 255, stdin))
        {
            setbuf(stdin, NULL); // clear buffer
            printf("%s", read_str);
        }

        close(herepipe[0]);
    }

The execution result of the program is:

    $ first line

I also wrote a simple demo to simulate the use of temp files in bash-5.0:

    $ cat testfile
    first line
    second line
    third line

    $ cat main_tmp_file.cpp
    #include <string.h>
    #include <fcntl.h>
    #include <unistd.h>
    #include <iostream>
    #include <fstream>
    #include <sstream>
    #include <stdio.h>

    using namespace std;

    int main() {
        int fd = -1;

        if ((fd = open("testfile", O_RDWR)) == -1) {
            cout << "open file failed" << endl;
        }

        dup2(fd, 0);

        char read_str[256] = { 0 };

        while (fgets(read_str, 255, stdin))
        {
            setbuf(stdin, NULL);
            printf("%s", read_str);
        }

        close(fd);
    }

The execution result of the program is:

    $ first line
    $ second line
    $ third line

I don't understand why openssl calls setbuf(stdin, NULL). and I don't know whether this usage is reasonable, I'm also looking to the openssl community for answers

Hello,

When I execute the openssl command to generate the rsa secret key in bash-5.1, it’s failed. But the same command can be successfully executed in bash-5.0.

LD_LIBRARY_PATH= openssl rsa -aes256 -passout stdin <<< "4567263139692f79613d254c51363046723a307e5c6124263f5935356e436a56
-----BEGIN RSA PRIVATE KEY-----
MIIG4wIBAAKCAYEAwXo4TQIBaKPRDy17IHGVE5G9pBJIgVY2D98wexktPUGE0z7K
/++dSqMJJH81r+XvAirQgzW6ZAbjImSBYuOYlXEI4MOmdTpYhTfnofOL9Pdi0Sy+
nJxv8b4NrRqqXDfEuzpVmnWRMI/ZrZXaILOSjXBBo9TujM9pQ8kYcIloQ6DX6nUC
dxNqOk+/QFEvmTwp6Q/cHZ8oMDlVeRYyP5S93lQ7Exxf1CJeqqBf1E/kFjpaK8Wn
YAFzumNDEopUwzboYJF1kEk7WXUTkaOT/BTyCBpuQfhauAMzi+iFSRV2q+ReviPq
UdvjP5RwTCA43PBLJoRV3IXrpWxOuwDB/YQX3MQ+hq034VJBCCeZT2HDYbuWvSIY
GYykOjWYthM6tleTmKuLB8T0kEfwFSTM71gzcbDCKHZYI7cuSERcwwsOzpggggBk
Y6daOi8acqJKl47KO15jEErCCgUCCzto5+x79AASicZXuGN0whk7XAMqT/y4ERvt
ZCaAwVG8WT3bvouFAgMBAAECggGAX1yMT3iDlT3ZFzAFlhIltkWJQUs5rXX+o4gQ
bzwTjjOy8iPrDobd8iG1qeWBYrdkPJ85stJSkmbKoPMEyub/F/lktNP6QaGGa/MI
GkzeWNluPUdek/aPPom8V+viuB1MvDOeack0zMi6M9K+dovL2ibQU53qDkFU0vJV
N3Kij2LByW/uzR4mp6MS0vKxEUgUKCmHBpqhxlMcC9ii36/CfcjqbRr5KKfgHR16
3jyaYVJVi2mae7l7fwzZzFOeP7HZsAQ1jEGYoNFlG4hJJ59BVuncUw490LU/yucS
8kybHjoshXMUc3Wiy0c/BBp9tdF6xcEBXy3cBb+1tVEbyZwAubk1bVt06rRvqXrO
GFbzhSs/UsxNjrHQ4h21mK1h29bSHBzQ6EGniIeAHJ1LNoHEJQezDMsrH1EnvEaF
v6Z9RsXDB5HX9FOw1bPsrxyFTLW+RnlDC5nQaeFv6geiGfD268Zj6Sx3vQoIVDLN
uMTv2NF8YgZYXQVgc1HY34ocfLxBAoHBAOI4Q+VRXWzjVwaakvlh31QFZKAC1eBS
IkKAQrdVkjIGJUg7o6q33PXNCz940v+2vak8P7hvxUM8VCb6w/wxcM9y6xv+4ZEw
IUNP5B9HjaNj0sDiXpAWJXaPOOpPqu07HDhbSJ2oCxbWocVAaapHPNpqRK8Uh1bW
EVRZfzfohpxcFmZFNDb94Pf/8mssWCnpsKh+sZiAhtlRlGZ/VVWQkL2Q/0vlT+jR
jSVLHa/mWPiiXxzMZ2rkA7oCoJadBEUOHQKBwQDa8oS8dUCjHkheOxoF8hIhqZPv
oy9aj2GlqmeuHAkZiO1t28VTOlsUpeJ2id68qJyh8gKERd2yWFaY+mWU+JqMV+kW
ceW57JAJvZsCgiBYMlrNOUVUZpGgdRDbm0gRkuq2V5se6gSb3ItyOpVsDO6cok+Z
YeEznpivhVfkNlqBXg5YfqjDVr28s624GqEnSuMsY4JSbHnVXzuHWgJIRAqPEyk+
29cXLxUZ83XJw3K/drXPV3onf1ehxNgBLpuJlokCgcBNlDQckWGztOckWaArq9fV
zZ+rjC1ip2geXDQFAXCIfTkN434GTtWU6gOAeeQKU9YJEqNbkeEZZLgLQhYyB5aD
Xgqo6CjdDJ9roRs/at90+AfPpBH/4chpN0ayfC3G+217Gjr17ioXY/XIWoWiIpoI
4Ua6sBSVwAXjYmnRd4mqV55K5lK4xSoAgqLVBDkb7otpzUEEZpdYMbTtzZ8rH8+z
YuVaVsKnui1oa7E7oIhEhBKQIr0YmCd7A+mknSZYtDUCgcEA2Dl9YOii5vS3H2ha
sNh5k4j3MkgEMfk7w/Hf5pCFZBRX9pB99U3COUgBuHM75zDVTxHD7oFZChI3o2hV
/aLubf87kBD3gH/hV91KQlXkP3WrOtV+ZHXPB5nVBiv6JnDjHySxavxNhOdvR2YX
vteepk2dMqr5TpMn0t9rB1OuWehbXoXngNIfG0aPHywdTsKkS2gSgffi8hfUoD0B
2ABqNRYFO7C8YIb83d4TyygcIH3J23RWjL8z+ex+LdyreluRAoHAHOkdXOy5sbXQ
xA0zJ5rOO3wy7hgMrodiCdzaLbmNer2qY0ymknlW64mEISVKFVerdkmyJFOvc/A+
bGMsuXPfO+r8NS8HHAplj/YmNlNksgZIdrHfKO/HxkBnEZZ8ny3Nt+i+J9z8uzMr
hgvr0zvd003hONS41q4m84ufRWfcxVfJls3fZ64q77p6UlzECY31m2n+wm1/Nmpp
pk1TJdu73LqpZ6NZ8pgIzsTVgTBRDgTJ6JgEFH86tEpEyFsXtgbf
-----END RSA PRIVATE KEY-----"

and error msg is:
unable to load Private Key
140737348022272:error:0909006C:PEM routines:get_name:no start line:crypto/pem/pem_lib.c:745:Expecting: ANY PRIVATE KEY

I find that in bash-5.1, function here_document_to_fd try to use a pipe internal instead of a temp file, this is the cause of failure.

#if defined(HEREDOC_PIPESIZE)
    /* Try to use a pipe internal to this process if the document is shorter
       than the system's pipe capacity (computed at build time). We want to
       write the entire document without write blocking. */
if (document_len <= HEREDOC_PIPESIZE)

I try to add goto use_tempfile before #if defined(HEREDOC_PIPESIZE), to ignore pip internal, it will be successful.

btw, when I execute the openssl command to generate the ecc secret key, It is both successful in bash-5.1 and bash-5.0.

"402c4258735e7c397a6041553b3e612274355939312b5a40554f305f3e784470
-----BEGIN EC PARAMETERS-----
BggqhkjOPQMBBw==
-----END EC PARAMETERS-----
-----BEGIN EC PRIVATE KEY-----
MHcCAQEEICoaFXALKhgKE1+N+TBbp8MlDxE1x8Wq+DIxw8ayxNcRoAoGCCqGSM49
AwEHoUQDQgAEZathOs55nj/ogH2G7/Wmy5IpLYwB7ngEUCu5dIukC/cWGSN5Rf3Z
GETDHzZEqUptOZEF+sQZqv7KW88guOiawQ==
-----END EC PRIVATE KEY-----"

I guess whether it is because the length of the string is limited in the pipe, and the command ulimit -a execution results are as follows:

[root@localhost ~]# ulimit -a
real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) 0
data seg size               (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) 5403
max locked memory           (kbytes, -l) 65536
max memory size             (kbytes, -m) unlimited
open files                          (-n) 1024
pipe size                (512 bytes, -p) 8
POSIX message queues         (bytes, -q) 819200
real-time priority                  (-r) 0
stack size                  (kbytes, -s) 8192
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) 5403
virtual memory              (kbytes, -v) unlimited
file locks                          (-x) unlimited