bugGNUstep - Bugs: bug #22351, GDNC not starting properly on some...

 
 

bug #22351: GDNC not starting properly on some systems.

Submitted by:  Gregory John Casamento <gcasa>
Submitted on:  Mon 18 Feb 2008 12:21:05 AM UTC  
 
Category:  Base/Foundation Severity:  3 - Normal
Item Group:  Bug Status:  Fixed
Privacy:  Public Assigned to:  Gregory John Casamento <gcasa>
Open/Closed:  Closed

Add a New Comment (Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

( Jump to the original submission)

Tue 19 Feb 2008 02:05:43 PM UTC, comment #19: 

I tested yesterday's changes to NSMessagePortNameServer.m (as applied on the stable branch) with Debian's Base 1.14.1 and everything works as expected.

Many thanks for fixing this annoying problem.

Yavor Doganov <yavor>
Tue 19 Feb 2008 10:38:04 AM UTC, comment #18: 

As expected ... the solaris64bit issue seems to have nothing to do with this bug report.

Richard Frith-Macdonald <CaS>
Project Member
Tue 19 Feb 2008 10:09:46 AM UTC, comment #17: 

I checked out svn yesterday evening, and recompiled, gdnc is staring up successfully as before, but still crashing on openbsd sparc64, as described on the other bug report.

Sebastian Reitenbach <buzzdee>
Project Member
Mon 18 Feb 2008 06:31:18 PM UTC, comment #16: 

As I commented on IRC, the current solution seems to have improved my NetBSD system, although I did only a quick test, if it is intermittent, only further testing will verify it. Surely it improved things, since before on NetBSD it always locked up.

I verified the problem on netbsd/x86, netbsd/ppc32, netbsd/sparc32. So it is surely not a 64bit-only problem

I will check ppc32 and Sparc32 asap.

Gregory had it on SuSE x86 instead, so it was not a NetBSD specific thing.

On FreeBSD/x86 it did work before and it continues to work.

Riccardo Mottola <rmottola>
Project Member
Mon 18 Feb 2008 06:24:49 PM UTC, comment #15: 

Richard, 

I tried your fix and it works properly on my machine.   I think this is definitely a better fix than the previous one.  

Thanks, GJC

Gregory John Casamento <gcasa>
Project AdministratorIn charge of this item.
Mon 18 Feb 2008 05:44:31 PM UTC, comment #14: 

> No it's not safe to make any such assumption...


Ah, thanks.  So my approach is entirely unacceptable, as you said.

> Hopefully the changes to NSPortMessageNameServer.m to implement
> locking...


I'll test extensively tomorrow at work, they should apply cleanly and work with 0.14 as well.

Yavor Doganov <yavor>
Mon 18 Feb 2008 05:30:22 PM UTC, comment #13: 

I saw a comment from Riccardo saying that the NSMessagePortNameServer change seems to have fixed the problem ... but since it's an intermittent problem, it would be good to get the fix confirmed for other systems and/or by other people.

NB. I don't expect that this is the same problem as the segmentation fault in 64bit sparc systems, so I would not be surprised if this fails to cure bug #21415

Richard Frith-Macdonald <CaS>
Project Member
Mon 18 Feb 2008 04:41:50 PM UTC, comment #12: 

> "If there is a problem running out of file descriptors,
> then a fix would be to use fewer of them"


Sure ... but it turns out that the problem was nothing to do with running out of file descriptors.

> The root of the problem is that the polling in that method for a connection
>is causing the process to block because the file descriptors allowed for that
>process are quickly used up. This happens because, at least on linux, the
> send and receive ports are created by making special files in /tmp.


But ensuring that those descriptors are closed immediately (ie so that the
process cannot possibly run out of descriptors) did not cure the problem, and of course the use of a large number of descriptors in one client cannot explain why other clients could then not contact the server ... so the descriptors were a red herring.

> Giving the server time to come up and register itself means that
> we'll do less polling


Yes ... good to poll less frequently ... I put a 0.05 of a second delay between each poll.  Enough to cut down significantly on cpu usage without causing a discernable delay in application startup.

> and, thus, use fewer file descriptors.


Each time we poll we we re-use the same file descriptors, so frequency of polling doesn't affect that.

Hopefully the changes to NSPortMessageNameServer.m to implement locking will have cureed this issue by preventing two processes from interfering with each other (your long sleep on startup would have reduced, but not eliminated, the chance of the two processes both trying to access the port name at the same time).

Richard Frith-Macdonald <CaS>
Project Member
Mon 18 Feb 2008 04:09:16 PM UTC, comment #11: 

"If there is a problem running out of file descriptors, then a fix would be to use fewer of them"

I agree that what I did is not a perfect solution.

The root of the problem is that the polling in that method for a connection is causing the process to block because the file descriptors allowed for that process are quickly used up.   This happens because, at least on linux, the send and receive ports are created by making special files in /tmp.

Riccardo also verified that it's the file descriptors on his NetBSD box by using the code prior to my change and setting the number of allowed file descriptors to the maximum.

Giving the server time to come up and register itself means that we'll do less polling and, thus, use fewer file descriptors.

Nevertheless, it does address the issue on a number of platforms where this problem was occurring.  I suggest we leave the present "hack" in place until a proper fix is found.   GC.

Gregory John Casamento <gcasa>
Project AdministratorIn charge of this item.
Mon 18 Feb 2008 03:50:48 PM UTC, comment #10: 

> But this happens at a time when gdnc is either not started or is non-functional,
> so it is safe to assume that no such connections exist. Isn't it so?


No it's not safe to make any such assumption since programs could register their own names (or try to contact other processes) before they access gdnc. Also, if gdnc has died, there may be a whole lot of other processes whose ports and names would be wiped out at the point when you tried to start gdnc.

Richard Frith-Macdonald <CaS>
Project Member
Mon 18 Feb 2008 03:25:00 PM UTC, comment #9: 

But this happens at a time when gdnc is either not started or is non-functional, so it is safe to assume that no such connections exist.  Isn't it so?  I may be totally wrong, I don't understand the code very well.

Here's how I've been able to trigger this bug, although not reliably (as I said I cannot reproduce with my patch, whether right or wrong -- and I certainly don't claim it's right):

1. Kill all daemons as suggested in the original submission.
2. Start an app (Ink, Preferences) that would lead to gdnc launching.
3. Kill the app, so that the nameserver database remains (quiting the app in a normal way should usually clean the database).
4. Kill gdnc.
5. Start the app again.

(Repeat steps 2-4 if the bug doesn't show yet.)

Yavor Doganov <yavor>
Mon 18 Feb 2008 03:09:57 PM UTC, comment #8: 

Looks like a problem with the NSMessagePortNameServer code failing to protect some changes with locks.  Deleting  the nameserver database and all the ports (as in the patch) is not really a solution since it effectively breaks everything else using distributed objects for private host-local connections (the default for inter-app communications).

Richard Frith-Macdonald <CaS>
Project Member
Mon 18 Feb 2008 10:54:18 AM UTC, comment #7: 

This problem happens quite often in Debian as of Base 0.14 (which has the --auto option by default), with many applications; users have reported such problems on i386, powerpc and amd64.  The solution in comment #1 does not help, unfortunately.

I've been using a modified GNUstep Base on all my machines for some time and I can't reproduce the problem with the attached patch.  This is not a clean solution either, since it generally slows down app startup time.

Either way, debugging this obscure problem is really hard.

(file #15064)

Yavor Doganov <yavor>
Mon 18 Feb 2008 10:16:46 AM UTC, comment #6: 

Is this a duplicate of #21415 ?

I think we are talking about two separate isssues, but it's not clear to me.

The issue in #21415 is that gdnc sometimes crashes (I think this is only on startup, but  perhaps when a client connects to it?)

The issue here seems to be that clients fail to connect to gdnc ... now is there really a problem in the client at all, or is this just that the client fails to connect because gdnc has crashed?

Richard Frith-Macdonald <CaS>
Project Member
Mon 18 Feb 2008 10:11:56 AM UTC, comment #5: 

Certainly not a generic 64bit problem ... because it works fine on my development machine (debian intel 64bit).  However, a sparc specific 64bit issue would not surprise me (since sparc is picky about  word alignments), and there is always the possibility of a problem with the ffcall library on a specific platform (or with gnustep's use of it).

Richard Frith-Macdonald <CaS>
Project Member
Mon 18 Feb 2008 07:50:34 AM UTC, comment #4: 

gdnc is working well on OpenBSD i386, cannot remember having seen a problem there. I've a OpenBSD sparc64 system, there the problem manifests "every time". Start gdnc, then try starting an application, e.g. GNUmail or AddressManager, and gdnc segfaults.
Maybe it has sth. to do with 64Bit systems. I think this bug report is a duplicate of this, still open one: https://savannah.gnu.org/bugs/?21297

however, I could offer a ssh account to the OpenBSD sparc64 box, in case someone wants to investigate. sebastia[at]l00-bugdead-prods.de

Sebastian Reitenbach <buzzdee>
Project Member
Mon 18 Feb 2008 07:02:09 AM UTC, comment #3: 

I would describe that as a hack to hide a problem (and therefore make it harder to fix any real bug).
If there is a problem running out of file descriptors, then a fix would be to use fewer of them, so we can try that instead.
Unfortunately this issue seems to be quite a rare, system-specific thing... Riccardo let me try to reproduce it on his BSD system for a while, but I could't get it to occur and it doesn't happen on any of the GNU/Linux systems I've access to.

Richard Frith-Macdonald <CaS>
Project Member
Mon 18 Feb 2008 12:50:18 AM UTC, comment #2: 

Richard, please take a look at the code which was changed and give any feedback you feel necessary.  GC.

Gregory John Casamento <gcasa>
Project AdministratorIn charge of this item.
Mon 18 Feb 2008 12:26:52 AM UTC, comment #1: 

Research found that this might be linked to file descriptors being exhausted by the polling in the connect method after starting the gdnc server.

I added a wait of three seconds which seems to address the issue.  GJC

Gregory John Casamento <gcasa>
Project AdministratorIn charge of this item.
Mon 18 Feb 2008 12:21:05 AM UTC, original submission:  

On many systems OpenBSD, NetBSD, and some Linux distros (notably SuSE 9.3) GDNC is failing to start properly.

Steps to reproduce:
1) make sure all daemons are stopped...
2) start an application using gopen or openapp
Result: the application will time out and give the "unable to contact GDNC server" error.
Expected: the application should start properly and GDNC should be launched.

This problem has been on these systems for the past couple of years.  GJC

Gregory John Casamento <gcasa>
Project AdministratorIn charge of this item.

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #15064:  clean-tmpdir.diff added by yavor (1KiB - text/x-patch)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rmottola (Posted a comment)
  • -email is unavailable- added by yavor (Updated the item)
  • -email is unavailable- added by buzzdee (Posted a comment)
  • -email is unavailable- added by CaS (Posted a comment)
  • -email is unavailable- added by gcasa (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

     

     

    Follow 4 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2008-03-20 rmottola Open/ClosedIn Test => Closed
    2008-02-18 yavor Attached File- => Added clean-tmpdir.diff, #15064
    2008-02-18 gcasa StatusIn Progress => Fixed
        Open/ClosedOpen => In Test

    Back to the top


    Powered by Savane 3.6