helpGnuTLS - Support: sr #107822, Testing 3.0.2 on AIX


sr #107822: Testing 3.0.2 on AIX

Submitter:  Bjørn Christensen <cybear>
Submitted:  Wed 28 Sep 2011 12:54:30 PM UTC
Category:  Core library Priority:  5 - Normal
Severity:  3 - Normal Status:  Done
Privacy:  Public Assigned to:  nmav
Open/Closed:  Closed Operating System:  None
* Mandatory Fields

Add a New Comment (Rich Markup)

Comment Type & Canned Response:


( Jump to the original submission )

Mon 03 Oct 2011 01:00:13 PM UTC, comment #24: 

Indeed. Check for the details. This is a good argument for me to disable TLS 1.2 silently if sign_callback is set.

Mon 03 Oct 2011 12:34:20 PM UTC, comment #23: 

Hangon a bit!

does the means that 2.10.5, 2.12.x and 3.0.x all wraps the hash in ASN1, when they are using TLS-1.2?

see #107785


Bjørn Christensen <cybear>
Mon 03 Oct 2011 12:10:43 PM UTC, comment #22: 

If you are using client certificates you might be seeing the bug in 2.10.1 I mentioned about. 2.10.1 was not correctly packing data for client certificate RSA signatures. If you want them to interoperate upgrade to 2.10.5 or 2.12.x.

3.0.x is expecting the data to be properly packed.

Mon 03 Oct 2011 12:06:34 PM UTC, comment #21: 

I am moving from 2.10.1 to 3.0.2, I tested the server build on 3.0.2 with the client build on 2.10.1 to limit the range of errors. I believed the 2.10.1 to work.

The problem on the 3.0.2 server is still that decode_ber_digest_info expect the hash to be wrapped in ASN1.

Bjørn Christensen <cybear>
Mon 03 Oct 2011 11:52:12 AM UTC, comment #20: 

gnutls 2.10.x is unsupported. 2.10.1 had a bug in client certificates and TLS 1.2 which was fixed in 2.10.5 (the latest release from this branch).;a=blob;f=NEWS;h=e84a83d4b072f91010c7b1cb7dff262e36e06cc7;hb=gnutls_2_10_x

I'd suggest using gnutls 2.12.x which is binary compatible with 2.10.x.

Mon 03 Oct 2011 10:49:48 AM UTC, comment #19: 

The server is 3.0.2  and should support TLS-1.2 the client I used was 2.10.1

I have not tested with gnutls-cli. I can try again and capture the stack.


Bjørn Christensen <cybear>
Mon 03 Oct 2011 10:44:37 AM UTC, comment #18: 

What is the type of the remote server? Could it be a server with incomplete TLS 1.2 support (gnutls 2.8.x had one, but it was disabled by default). About the error with the priority string what error is it? Is it reproducible with gnutls-cli?

Mon 03 Oct 2011 09:23:31 AM UTC, comment #17: 

If I disable TLS1.2 it works!

but if I give the priority string


where I try to disable  eliptic curves I get an Error when TLS1.2 is disabled.


Bjørn Christensen <cybear>
Mon 03 Oct 2011 09:08:50 AM UTC, comment #16: 

gnutls_global_init have been called.

I can decode and use certs on the platfrom through the gnutls API. (what part of certtool needs to work?)

In the function decode_ber_digest_info, it looks to me like gnutls are trying to decode the info as and ASN1 string but at that point info holds 20 bytes which is the SHA1 hash.

What is info expected to hold?

in 2.10.1 all the data send forth and back were logged, is that possible with 3.0.2?

call stack:
decode_ber_digest_info : gnutls_sig.o line 780
_pkcs1_rsa_verify_sig : gnutls_pubkey.o
pubkey_verify_hashed_data : gnutls_pubkey.o
gnutls_pubkey_verify_hash : gnutls_pubkey.o
verify_tls_hash : gnutls_sig.o
_gnutls_handshake_verify_cert_vrfy12 : gnutls_sig.o
_gnutls_handshake_verify_cert_vrfy : gnutls_sig.o
_gnutls_proc_cert_client_cert_vrfy : cert.o
_gnutls_recv_client_certificate_verify_message : gnutls_kx.o
_gnutls_handshake_server : gnutls_handshake.o
gnutls_handshake : gnutls_handshake.o

Bjørn Christensen <cybear>
Fri 30 Sep 2011 02:00:11 PM UTC, comment #15: 

This looks like a parsing error. Does certtool work on this platform? If yes, have you called gnutls_global_init()?

Nikos Mavrogiannopoulos <nmav>
Project Administrator
Fri 30 Sep 2011 12:49:21 PM UTC, comment #14: 

I did a simpel change #define RESUME_FALSE 1

and that got me further but not all the way.

I have attached a log file and now it is not trying to resume an empty session but now it fails during validation of the client certificate. Any suggestions?

(file #24054)

Bjørn Christensen <cybear>
Fri 30 Sep 2011 10:15:56 AM UTC, comment #13: 

If I could choose I would try to program more defensive!

First I would define RESUME_FALSE 0 because that is what I would expect from 0==1.

Also that would if calloc'ed buffer is used default would be not resuming.

Then I would make all comparisons session->internals.resumed == RESUME_FALSE or session->internals.resumed != RESUME_FALSE

agains the value defined as 0. In your case RESUME_TRUE, if suggested above RESUME_FALSE.

In K&R they state that the behaviour of bit fields are implementation specefic. (Code Defensive)

Does that make any sense?

Bjørn Christensen <cybear>
Fri 30 Sep 2011 08:20:23 AM UTC, comment #12: 

In that case also change:
int resumed:1  -> unsigned int resumed:1.

Which compiler are you using?

Fri 30 Sep 2011 07:56:19 AM UTC, comment #11: 

I think I have identified the problem!

session->internals.resumed end up being 1 and RESUME_FALSE is -1 and RESUME_TRUE is 0

The declaration of session->internals.resumed in gnutls_int.h line 669 is:

int resumed:1; 

that will hold 0 or 1 not negative 1.

what is the suggested fix changing RESUME_FALSE to 1?


Bjørn Christensen <cybear>
Thu 29 Sep 2011 04:22:06 PM UTC, comment #10: 

Nettle complains because the size of the AES keys provided is invalid. That's not expected. What are the values put to _gnutls_set_keys(), key_size, hash_size and IV_size?

The assertion that you mention at:
gnutls_hash_int.c:55 is indeed worrying because _gnutls_mac_is_ok() is called in _gnutls_epoch_set_keys() at gnutls_constate.c which would have ensured that it is not called with random data.

I'd suggest that would single-step _gnutls_epoch_set_keys() and find out where the values of the cipher algorithms and lengths go out of range.

Nikos Mavrogiannopoulos <nmav>
Project Administrator
Thu 29 Sep 2011 01:51:04 PM UTC, comment #9: 

Here is the stack on the server

pthread_kill : libpthread.a-.text-1
_p_raise : libpthread.a-.text-1
raise : libc.a-.text-1
abort : libc.a-.text-1
__assert_c99 : libc.a-.text-1
nettle_aes_set_encrypt_key : aes-set-encrypt-key.o
aes_bidi_setkey : cipher.o
wrap_nettle_cipher_setkey : cipher.o
_gnutls_cipher_init : gnutls_cipher_int.o
_gnutls_auth_cipher_init : gnutls_cipher_int.o
_gnutls_init_record_state : gnutls_constate.o
_gnutls_epoch_set_keys : gnutls_constate.o
_gnutls_write_connection_state_init : gnutls_constate.o
_gnutls_send_handshake_final : gnutls_handshake.o
_gnutls_handshake_common : gnutls_handshake.o
gnutls_handshake : gnutls_handshake.o

I can inspect variable at the different stack levels!
but I do not know what to look for. It seems to me like it is trying to resume a connection, but this a new connection.


Bjørn Christensen <cybear>
Thu 29 Sep 2011 10:47:05 AM UTC, comment #8: 

I have added a log file from linux where it works.
On AIX after the SERVER HELLO have been queue there is a CHANGE CIPHER SPEC. Not a CERTIFICATE as on linux

(file #24047)

Bjørn Christensen <cybear>
Thu 29 Sep 2011 07:50:06 AM UTC, comment #7: 

I have attached a log file.

Client and Server both put entries into it.

Thread 258 is the client and thread 515 is the server.

I have some ASSERT from gnutls_db but I think that is just because it does not find a saved session.

This logfile have been capture with TLS-VER1.2 disable

There is an assertion from gnutls_hash_int.c:55 which looks worrying.

Can you see more from the log file.


(file #24044)

Bjørn Christensen <cybear>
Wed 28 Sep 2011 08:29:00 PM UTC, comment #6: 

Ouch, this can be hardly related with the modification. Could it be a memory corruption? The failure is within the handshake. Could you provide a stack trace, via valgrind or a debugger?
(preferably valgrind)

Nikos Mavrogiannopoulos <nmav>
Project Administrator
Wed 28 Sep 2011 04:57:45 PM UTC, comment #5: 

I changed system.c

system_errno (gnutls_transport_ptr_t ptr)
#if defined(_AIX) || defined(AIX)
  if (errno == 0 || errno == ENOENT) errno = EAGAIN;

  return errno;

And I think it brought me further, now I got an assertion.
Assertion failed: keysize >= AES_MIN_KEY_SIZE, file  aes-set-encrypt-key.c, line 57

and a core dump.

Bjørn Christensen <cybear>
Wed 28 Sep 2011 04:24:07 PM UTC, comment #4: 

I will try that but since I was not sure I understood the mechanics I postponed it. I thought that the ENOENT were cleared somewhere else since the internale errno was 0.

Anyway I will try and see if the suggested change will solve my problem.


Bjørn Christensen <cybear>
Wed 28 Sep 2011 02:04:51 PM UTC, comment #3: 

You are correct it is the how system_errno() works. I also see that in:

It seems some common buggy code is shared among these systems. If you put an if (errno==ENOENT) errno=EAGAIN; in system_errno() does it fix the error for you?

Wed 28 Sep 2011 01:57:23 PM UTC, comment #2: 

errono=2 also means ENOENT on AIX and it makes no sense, at least IBM manuals does not list ENOENT as an error code from recv.

When gnutls_init is called it calls gnutls_transport_set_errno_function() with the default system_errno() function which can be found in system.c line 82.
The system_errno() functions just returns opsys errno.
Have I completly misunderstood the way it works?

I am also trying to find out what ENOENT means from revc!
I have googled it and there are some suggestions that one should treats it as EAGAIN.

Bjørn Christensen <cybear>
Wed 28 Sep 2011 01:23:11 PM UTC, comment #1: 

What does the errno=2 mean in AIX? In Linux it is ENOENT, which could mean that the connection is closed, or you're using an invalid file descriptor. About your question, I don't really understand. The gerrno (internals.errnum) is only set if you use the gnutls_transport_set_errno() function. Otherwise only the system errno is used. Thus having gerrno=0 and errno!=0 is quite normal.

Wed 28 Sep 2011 12:54:30 PM UTC, original submission:  

I have the following problem and I cannot understand what is going wrong:

I have the following entries in my log file:
 RPCSECURE: thread 258 : READ: -1 returned from d, errno=2 gerrno=0
 RPCSECURE: thread 258 : ASSERT: gnutls_buffers.c:275

If I look at gnutls_buffers.c at line 255 where the first statment get written, I have the pull_func return -1 from socket d, the opsys error is 2 and the internale error no is 0.

in gnutls_buffers.c line 247 the internale errno get cleared (session->internals.errnum)

when the pull_func returns -1 the get_errno(session) get called on line 253, the get_errno returns either the internal errno if not 0 else it calls the errno_func which for AIX is system_errno  and system_errno from system.c line 82
returns errno (which we from the log statement knows is 2).

How can I end up with errno=2  and gerrno (internals.errnum) =0 ?


PS: I can see that the problem I reportede where AIX recv returns -1 but forgets to set errno have been fixed in the system_errno function (thank you)

Bjørn Christensen <cybear>


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

Attached Files
file #24054:  congaRESUMEfix.log added by cybear (33KiB - application/octet-stream)
file #24047:  congalinux.log added by cybear (82KiB - application/octet-stream)
file #24044:  conga.log added by cybear (36KiB - application/octet-stream)


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by nmav (Posted a comment)
  • -email is unavailable- added by cybear (Submitted the item)

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.





    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2011-10-23 nmav StatusNone Done
        Assigned toNone nmav
        Open/ClosedOpen Closed
    2011-09-30 cybear Attached File- Added congaRESUMEfix.log, #24054
    2011-09-29 cybear Attached File- Added congalinux.log, #24047
    2011-09-29 cybear Attached File- Added conga.log, #24044

    Back to the top

    Powered by Savane 3.9