I backported the patch to exit with info=2 when a singularity is reached (http://hg.savannah.gnu.org/hgweb/octave/rev/68e2daa29222).
I added a BIST test on stable to verify that the patch above works correctly (http://hg.savannah.gnu.org/hgweb/octave/rev/c29a2107c559).
Finally, I checked in a third patch to return a success code (info=1) if the initial guess for the solution is already accurate enough (http://hg.savannah.gnu.org/hgweb/octave/rev/b3c35a130f94). I added a BIST test for that too.
Marking bug as fixed and closing report.

I checked in a changeset to address the problem of bug #39000, mentioned here in comment #4. See that report for details.

What about something like this

I checked in a change to avoid the crash at a stationary point (http://hg.savannah.gnu.org/hgweb/octave/rev/dffd9f6ee85c). I used this test
Since norm (x,1) is an absolute column sum I needed to scale macheps by the number of rows in the matrix.
I will probably backport this to stable, but not before the other issue is solved. You are correct that if the user happens to specify a solution as the initial starting point then fsolve returns info=3. A simple test is just a line,
The simplest thing would just be to copy the test for success to be at the top of the file just outside the start of the outer loop.
However, we don't have xn available at that time. Maybe someone can work out what xn should be for the first case of iteration.

@Rik: my first thought was that info = 2 should be set when fjac is small AND fvec is still large. So I tried
(pi is the right solution), but I get flag = 3. I would conclude that:
1) fsolve should check whether the initial solution is already fine before the ## Outer loop
2) when it computes a new fjac, it means that the previous fvec was not fine. Therefore it is correct just to check whether the new fjac is not too small, as you did
3) I would use norm (fjac, 1). The 1norm is easier to compute.
4) For unconverged cases, Matlab gives a message (not a warning).

@Marco: Can you review this patch for fsolve?
If the Jacobian is very small this stops fsolve before calling _dogleg_ with a zero value for the first argument which would eventually lead to an illconditioned matrix.
One question, should the test be for any component of fjac or should we be taking the norm first? In other words, this
OR
I expect it is something like the latter. The first thing that the __dogleg__routine does is calculate
If fjac is a vector or matrix having a zero would be okay. Only an all zero array would be a problem.

I think there are two separate bug reports here. The first is that calling balance can lead to a segfault on Windows platforms.
Using the test code from comment #1 on a Linux machine I get
The segfault is probably back to the inability to overide the XERBLA handler on Windows (bug #39000). The question is how far we want to keep patching things. It seems like possibly every routine which ever interacts with the BLAS or LAPACK library would potentially need to look for NaN values before continuing. At the very least, it would seem that this should be restricted to Windows platforms which would require putting in #ifdef tests.
The second bug report is about fsolve not returning a correct error code when it encounters a stationary point. This should be fixed, and it shouldn't be that difficult.

I think the problem is in the algorithm itself. Matlab returns with the exit flag 2 which means
This happens when the Jacobian fjac has a small norm (0, in this report) but the solution fvec is far from 0. This can be detected before calling _dogleg_ at line 307.

The problem is in the call to DGEBAL in liboctave/numeric/aepbalance.cc

The line that cause the crash is
The bug seems to be related to bug #39000

Trying to solve equation det (B(lam)) = 0 for lam, if starting point lam0 = 0, Octave crashes without error or warning.
If lam0 = 0.1, solution found correctly.
Here the script, with a plot of B vs. lam:
