bugGNU Octave - Bugs: bug #49214, Readline library does not support...

 
 

bug #49214: Readline library does not support writing to Windows files with "hidden" attribute

Submitted by:  Hartmut <hardy>
Submitted on:  Wed 28 Sep 2016 06:17:33 PM UTC  
 
Category:  Octave Function Severity:  1 - Wish
Priority:  3 - Low Item Group:  Incorrect Result
Status:  Wont Fix Assigned to:  None
Originator Name:  Open/Closed:  Closed
Release:  4.2.0-rc2 Operating System:  Microsoft Windows

Add a New Comment (Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

( Jump to the original submission)

Wed 05 Oct 2016 06:42:59 PM UTC, comment #28: 

I have added the information that I think is still important to the new bug report, thanks.

Hartmut <hardy>
Wed 05 Oct 2016 06:29:21 PM UTC, comment #27: 

I just filed bug #49275 as a distinct feature request based on the bug dicussion here, so that part of this bug history will remain active.

Other than implementing an option to append to the history file instead of overwriting it when Octave exits, is there anything else in this bug that you think is unique an desirable to act on?

Mike Miller <mtmiller>
Project Administrator
Wed 05 Oct 2016 05:54:01 PM UTC, comment #26: 

This bug report has covered two different topics:

  • (1) The original issue of Octave not beeing able to use an .octave_hist file under Windows OS, if this file has the "hidden" file attribute set.
  • (2) The idea to solve this issue with a much more general solution: Wanting that Windows Octave would in general be able to write to files that have the "hidden" attribute set.

I understand that the much wider topic (2) has been decided to NOT beeing solved. The reason is that Windows OS in general does not support writing to such hidden files (it only allows appending to hidden files).

But it IS still possible to fix the original topic (1):

  • Windows OS ALLOWS this (comment #21).
  • There is also a reasonable PROPOSAL how to fix Octave's behavior (Mike's comment #23). It would mean that Octave only appends to the history file during its shutdown process (and not writes it entirely new).
  • It would be a USEFUL change to Octave (see my original submission).

So my wish persists to re-open this bug report, and to change its title back to the original topic (1), which CAN (maybe quite easily) be fixed.

Hartmut <hardy>
Tue 04 Oct 2016 10:28:11 PM UTC, comment #25: 

The policy is not to leave a bug open if it will not be fixed.  All bug reports, open or closed, are searchable.  We find that most people search for a bug matching their symptoms, rather than simply browse through the now greater than 800 bug reports.

Rik <rik5>
Project Administrator
Tue 04 Oct 2016 08:58:04 PM UTC, comment #24: 
  • I have tried to do some tests with mingw the bash that is included in Octave's Windows installer. But this bash doesn't even seem to create any history file. And it also does not remember the commands from the previos bash session at all. So I think I personally are not capable of doing the proposed bash tests.
  • Could you nevertheless please
    • change the bug title back to something like "an .octave_hist file with hidden file attribut should work (Windows)"
    • re-open this bug report (as a "wish"), so it can more easily be found by the next person who stumbles over this issue, and maybe is capable of fixing it.
Hartmut <hardy>
Thu 29 Sep 2016 09:30:24 PM UTC, comment #23: 

The behavior for readline history is up to the program to decide how to write. Octave has decided to emulate bash, whose default behavior is to overwrite the history file when the shell exits. The shell and user are thus in control of how large the history buffer is.

When the bash shell starts, it reads the ~/.bash_history file into its in-memory history buffer. As commands are run, they are appended to the end of the in-history list (up to HISTSIZE in-memory entries). When the shell exits, it overwrites the entire ~/.bash_history file with the N last entries, where N is controlled by the HISTFILESIZE variable.

Bash does have a "histappend" option, which instructs the shell to always append history to the file rather than overwriting it on exit, meaning it will grow indefinitely instead of being truncated to HISTFILESIZE (but still limited by the in-memory limit HISTSIZE).

Octave could add a history_append() boolean option to mimic bash "histappend".

It would be nice if you or someone could thoroughly test that this would actually work, for example with the bash shell under mingw where ~/.bash_history has the "hidden" attribute set.

Mike Miller <mtmiller>
Project Administrator
Thu 29 Sep 2016 09:06:30 PM UTC, comment #22: 

Wouldn't it be a possible fix for the original problem (a nicely hidden .octave_hist file) to use the "append_history" command (Octave's "history -a") from the readline library in the Octave shutdown code, instead of the currently used "write_history" command (="history -w")?

Because the only thing that normally happens during an Octave session is to APPEND new commands to the already existing history. And appending to already existing "hidden" files seems to be supported by Windows OS as well as by the "fopen(...,'a')" command in Octave for example. And for clearing the full history list, the history file can just be deleted, this is also allowed for hidden Windows files.

Or are there any usecases, where someone can actually change the already pre-existing history list during an Octave session?

(If you think this is a doable fix, then please reopen this bug report. Otherwise just leave it closed.)

Hartmut <hardy>
Thu 29 Sep 2016 06:21:17 PM UTC, comment #21: 

I just wrote a test script in Perl on windows and it has the same problem.  It can't open a hidden file for writing, although it can open such a file for appending.

This seems to be pretty universal.

Documentation for Windows Powershell and redirection states

If the specified file already exists, the redirection operators that do not
append data (> and n>) overwrite the current contents of the file without
warning. However, if the file is a read-only, hidden, or system file, the
redirection fails. The append redirection operators (>> and n>>) do not
write to a read-only file, but they append content to a system or hidden
file.

Rik <rik5>
Project Administrator
Thu 29 Sep 2016 05:42:50 PM UTC, comment #20: 

Ok, then that's exactly what readline is doing.

Readline has a write_history() function and a append_history() function, each of which map to the "history -w" and "history -a" commands. When Octave exits it does write_history, which overwrites the file. This is what is failing.

I expect that this would affect any program, whether it's C++, Octave, or Python, attempting to overwrite an existing hidden file.

I'd mark this as completely won't fix, talk to Microsoft about fixing it if you want this changed :)

Mike Miller <mtmiller>
Project Administrator
Thu 29 Sep 2016 05:22:30 PM UTC, comment #19: 

fopen worked fine for me.  This was with a Windows XP virtual machine.  Perhaps newer versions of Windows are stricter?

Also, I used

fopen ("filename", "a")

because I thought that was probably how the internals were working.

I just checked with "w" and that does not work.

[fid,msg] = fopen ('filename', 'w')
fid = -1
msg = Permission denied

Rik <rik5>
Project Administrator
Thu 29 Sep 2016 04:56:39 PM UTC, comment #18: 

Hm? I thought comment #10 explicitly says that calling fopen (with options set to overwrite the file) on an existing file with the hidden attribute set does not work.

Can we get some more testing? It seems to me that it is a general OS / system library problem than a bug specific to readline, if I'm reading this right.

Mike Miller <mtmiller>
Project Administrator
Thu 29 Sep 2016 04:27:04 PM UTC, comment #17: 

This is only a problem with Readline.  Octave can use fopen, fprintf, fclose on a hidden file without problems which I have verified.

The best thing is to report this upstream.  Readline is old, but still supported, so there is a bug tracker for it somewhere.

I've changed the title to reflect the true cause and marked it as Postponed since we need an upstream fix for this.

Rik <rik5>
Project Administrator
Thu 29 Sep 2016 11:38:05 AM UTC, comment #16: 

So the root cause seems to be: The readline library does not support writing to Windows files with "hidden" attribute. (We might change the title accordingly.)

I would find it a good feature, if Octave could do this (writing into "hidden" files). Not only for the mentioned use case of a hidden .octave_hist file, but in general (e.g. with fopen). (I assume Matlab or Pyhton do not have a problem with this.)

Does someone know where such a feature request for the readline library could be submitted?

Hartmut <hardy>
Thu 29 Sep 2016 04:39:41 AM UTC, comment #15: 

Actually, a little more testing shows that writing the history file always fails when the hidden attribute is set.

The issue is not with Octave, but with the Readline library that Octave is using.  I don't see an easy way to fix this.

The code is eventually in liboctave/util/oct-rl-hist.c.

Rik <rik5>
Project Administrator
Thu 29 Sep 2016 04:30:56 AM UTC, comment #14: 

If I use File Explorer to change .octave_hist to hidden, and then start octave-cli.exe from a cmd.exe window I get the following error on leaving Octave.

error: writing file 'C:\Documents and Settings\Rik\My Documents\Linux\home\rik\.octave_hist': Permission denied
error: ignoring const octave::execution_exception& while preparing to exit

I used the function history_file() to change the name of the file to something without a leading '.' character, but I get the same error message if the new file has the hidden attribute.

On the other hand, directly writing the history file with

history -w

works just fine in an Octave session.  So it may be something special about the shutdown procedure for Octave.

Rik <rik5>
Project Administrator
Wed 28 Sep 2016 10:32:16 PM UTC, comment #13: 

Whatever we do, I don't think it makes sense to add special cases all over Octave to work around this behavior.

John W. Eaton <jwe>
Project Administrator
Wed 28 Sep 2016 10:31:44 PM UTC, comment #12: 

I suppose it would be possible to get the file attributes, fix them up so that the file can be written, then restore them after.  But why should this particular file get special treatment in that regard?  What about other files?  If only this file, then I think the place to fix the bug is in the readline library, not Octave itself, since Octave just calls readline to write the file.

Or, is there some way to set an option for writing hidden files so that it will work for all files?

John W. Eaton <jwe>
Project Administrator
Wed 28 Sep 2016 09:21:30 PM UTC, comment #11: 

Thank you for the tests and clarification. So overwriting an existing file with the hidden attribute set is not allowed at the standard library level on Windows. There's not much we can do about that, this is the way readline operates when it is saving the session history.

Mike Miller <mtmiller>
Project Administrator
Wed 28 Sep 2016 09:17:40 PM UTC, comment #10: 

The result of a rename command (comment #6) on an existing but hidden windows file, is exactly the one displayed in comment #7. So renaming of hidden files DOES work. But writing (at least fopen) to hidden files does NOT work from within Octave.

Anonymous
Wed 28 Sep 2016 09:14:51 PM UTC, comment #9: 

The error message of the fopen command in comment #8 on hidden Win files is "Permission denied".

Anonymous
Wed 28 Sep 2016 09:11:31 PM UTC, comment #8: 

(from Hartmut)

  • to question in comment #1 : see Philips comment #5
  • to question in comment #2 and #4: the Octave "rename" command works fine on both hidden and un-hidden files, also on filenames beginning with a ".".

Another observation: The command "fid = fopen('file.txt', 'w')" does also NOT work if "file.txt" is an existing but "hidden" file.

Anonymous
Wed 28 Sep 2016 09:01:11 PM UTC, comment #7: 

Sorry, that should have been

>> [status, errmsg] = rename ("file1.txt", "file2.txt")
status = 0
errmsg =

Mike Miller <mtmiller>
Project Administrator
Wed 28 Sep 2016 08:59:03 PM UTC, comment #6: 

You didn't say whether it's possible to rename a normal file with the hidden attribute set.

Can one of you please try creating a normal plain text file, file1.txt, set the NTFS hidden attribute, and then try

>> rename file1.txt file2.txt;

Does this work or do you get the same "Permission denied" error?

The old bug report you are remembering was about packages not passing --no-history to an "octave --eval" command line when they have a src/configure script as part of their installation. May still be an open issue.

Mike Miller <mtmiller>
Project Administrator
Wed 28 Sep 2016 08:13:31 PM UTC, comment #5: 

Renaming hidden files with a name starting with a period works perfectly fine in Octave's file browser.

It does not work in Windows Explorer ("you must type a file name" -> it thinks one entered just a file extension), nor in a cmd window "The system cannot find the file specified." -> it is really hidden (!).

On Windows systems, files starting with a "." are not automatically hidden files. That is simply how Windows file system work under Windows. That *nix etc. treat Windows file systems like NTFS and FAT differently makes no difference here.

I can confirm Hartmut's bug; just before closing Octave-GUI with .octave_hist set to "hidden" I see two error messages flashing by.
A fix might be a question of merely a different system call (in the MinGW libs, probably needing #ifdef WIN32 clauses), but the "bug" may also be associated with more fundamental issues on Windows and thus harder to solve (if at all possible)

"history_save" returns 1 but it didn't really update the file.

>> history -w
error: history: writing file 'C:\Users\philip\.octave_hist': Permission denied

As there is a workaround by simply leaving .octave_hist to be not hidden I fully agree with lowering priority and severity.

BTW, IIRC jwe has adapted .octave_hist processing some time ago (a year+ ?) related to an old bug report that may even have been entered by me. Again IIRC it was related to writing in the root dir of a drive (different from some OF pkg configure issues that Mike or Rik investigated). Maybe there are some clues there.

Philip Nienhuis <philipnienhuis>
Project Member
Wed 28 Sep 2016 08:04:09 PM UTC, comment #4: 

And the other questions in comment #1?

As to comment #2, the important part is whether the system call works or throws an error. How about creating a normal file, setting the hidden attribute, and then trying to rename it using the rename() function in Octave?

Mike Miller <mtmiller>
Project Administrator
Wed 28 Sep 2016 07:02:40 PM UTC, comment #3: 

The question in comment #2 is hard to anwer. When you try to rename a file (hidden nor not) where the filename starts with a "." , like ".octaverc", then the Windows file manager will NOT let you do this, but throws an error message "please do choose a proper filename". So the Windows file explorer doesn't consider a filename starting with a dot as being alright. (Even though it is an allowed filename for the used fat and ntfs file systems).

Hartmut <hardy>
Wed 28 Sep 2016 06:46:38 PM UTC, comment #2: 

Just a wild guess: I have found with some searching that attempting to rename a file with the "hidden" attribute will result in an error, is that still correct on current Windows systems?

Mike Miller <mtmiller>
Project Administrator
Wed 28 Sep 2016 06:30:16 PM UTC, comment #1: 

Does history_save return true?

Does "history -w" update the file?

Can you determine what is the error if the OS is refusing to allow the file to be written when Octave is attempting to do so?

Mike Miller <mtmiller>
Project Administrator
Wed 28 Sep 2016 06:17:33 PM UTC, original submission:  

This happens under Windows OS (Win7 in my case). I have seen it with Octave 4.2.0-rc2, but it's already there in Octave 4.0.3 and probably also in earlier versions:

The file ".octave_hist" is not updated during an Octave session, if this file has the Windows file attribute "hidden"! In this case the octave_hist file is properly read during startup of Octave, during the Octave session the newly issued command are properly displayed in the GUI history window. But after closing and restarting Octave, the last commands have disappeard, again.

Just for comparison: The file ".octaverc" is properly read under Window OS, regardless if it has "normal" file attributes, or the attribute "hidden".

I usually like to "hide" files like octave_hist, because I am not ment to manually edit them anyways. And under Linux OS the intention of the "." at the beginning of the file name is also to hide this file away. So I think it would be reasonable if also the Octave Windows version could deal with "hidden" octave_history files, it might even be a good idea to "hide" this file under Windows as well, when Octave needs to newly generate it (or runs for the first time).

Hartmut <hardy>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by mtmiller (Posted a comment)
  • -email is unavailable- added by hardy (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

     

     

    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2016-09-29 mtmiller StatusPostponed => Wont Fix
        Open/ClosedOpen => Closed
    2016-09-29 rik5 StatusNeed Info => Postponed
        Summary".octave_hist" not updated if it is a "hidden" Windows file => Readline library does not support writing to Windows files with "hidden" attribute
    2016-09-28 mtmiller Severity3 - Normal => 1 - Wish
        Priority5 - Normal => 3 - Low
    2016-09-28 mtmiller StatusNone => Need Info

    Back to the top


    Powered by Savane 3.5