bugGNU Octave - Bugs: bug #64363, [octave forge] (io) xlswrite to...

 
 

bug #64363: [octave forge] (io) xlswrite to existing file fails if temp dir is on different drive

Submitter:  Georg Wiora <gwiora>
Submitted:  Wed 28 Jun 2023 02:11:50 PM UTC
   
 
Category:  Octave Package Severity:  3 - Normal
Priority:  3 - Low Item Group:  Unexpected Error or Warning
Status:  Works For Me Assigned to:  philipnienhuis
Originator Name:  georgw Open/Closed:  * Open
Release:  * 8.2.0 Operating System:  * Microsoft Windows
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Fri 19 Jan 2024 12:55:13 PM UTC, comment #7: 

@gwiora:
What maybe might be at play here is that behind your back, Windows may use another temp directory inside the temp directory.
That is, I often find that the environment variables %TMP% and %TEMP% point to e.g., C:\Windows\Temp but in reality C:\Windows\Temp\1\  or C:\Windows\Temp\2\ are being used.

However, xlswrite.m and helper functions invoke Octave's function 'tempname' which in turn invokes regular Windows functions, so AFAICS xlswrite.m etc. should just work "as advertised".

This bug report is some 6 months old now with status "Need info". I'll set status to "Works for me" and I'll leave it open for at most another month, or until you've reacted here (whichever comes first).

Thanks

Philip Nienhuis <philipnienhuis>
Group Member
Sun 23 Jul 2023 09:24:24 PM UTC, comment #6: 

(Slightly OT: As to comment #5: I've entered bug #64462 as it is fileparts.m that mangles the file name.)

Anyway, I have a hard time reproducing the issue. E.g., with PWD to be f:\tmp (on same computer as where I ran Octave on) and Q: a drive on my NAS:

>> getenv ("TEMP")
ans = C:\Users\philip\AppData\Local\Temp
>> getenv ("TMP")
ans = C:\Users\philip\AppData\Local\Temp
>> xlswrite ("Q:/tmp/bug64363A.xlsx", 15.3, "FirstSheet", "C2", "oct")
ans = 1
>> xlswrite ("Q:/tmp/bug64363A.xlsx", 16.3, "FirstSheet", "C3", "oct")
ans = 1

so the TEMP dirs are on another drive but xlswrite happily writes to yet another drive.

@OP (gwiora):

  1. Is the location where you are writing to actually writable when accessed as network location?
  2. How did you assess that the location of your TEMP directory is related in the first place?
  3. As you seem the only one affected I'd urge you to debug this yourself along the lines Iset out in comment #3



Philip Nienhuis <philipnienhuis>
Group Member
Sun 23 Jul 2023 10:15:27 AM UTC, comment #5: 

The error seen in the 3rd verbatim block of comment #1 seems to be due to an issue with the 'make_absolute_filename' function. That is conspicuous in the first lines there but I overlooked it:

>> cd c:
>> xlswrite ("z:subst2.xlsx", {"Col1", "Col2"; 5, 6; 7, 8}, "", "", "oct")
xlsclose: could not zip files in C:\Users\philip\AppData\Local\Temp\oct-IOmPy1 to c:\\z:subst2.xlsx
:


Setting a breakpoint in xlsclose.m at L.139
( xls = _OCT_spsh_close_ (xls, "xlsclose", force); )
and doing this:

debug> xls
xls =
  scalar structure containing the fields:
    xtype = OCT
    app = xlsx
    filename = Z:bug64363B.xlsx
:

debug> make_absolute_filename (xls.filename)
ans = c:\Z:bug64363B.xlsx

shows that make_absolute_filename cannot process / is oblivious to the working directory on another drive.
ISTR this issue has been reported, or at least noted, several years ago in the course of other issues pertaining to 'make_absolute_filename' and/or 'canonicalize_file_name' functions and how these work on Windows. But maybe here it's due to the subst command.

However this is not the bug the OP posted.
I'll check if I can reproduce that some other way.

Philip Nienhuis <philipnienhuis>
Group Member
Fri 30 Jun 2023 07:33:44 AM UTC, comment #4: 

Thank you Philip for your debugging tips.

I see a different behavior between SUBST and NET SHARE. Windows shows a SUBST drive as a local drive and the xlswrite is working as expected. A drive mapped with NET SHARE is shows up as a network drive. I think this could explain the different behavior.

I can not promise if I find time to test further, but I will try to.

Georg Wiora <gwiora>
Thu 29 Jun 2023 06:25:15 PM UTC, comment #3: 

@gwiora:
Maybe it isn't 'movefile' but rather the zip function that's at fault, but I doubt that.

What you can do is enter a 'breakpoint' statement in the file private/__OCT_spsh_close__.m (in the io package subdirectory) just above the lines where the zip is created (that way search path issues are avoided for private functions) and try the xlswrite command. It'll stop at the 'breakpoint' statement. From then on 'dbstep' through the file (just use the buttons in the GUI editor) and inspect if files are created properly. Or better:
Hint: I usually select (highlight) a statement and press <F9> to execute it manually, one by one. That way you can avoid being thrown out of debug mode when an error is hit, and you can also find out exactly which statement provokes the error (which is otherwise masked because of the try-catch block around it all).
If it indeed is the 'movefile' function, you can try to mimic the command in a cmd32 window outside Octave. Maybe the underlying 'move' command in Windows itself is the culprit.

Anyway this is how I would start tackling this issue. Hopefully it helps you to debug this further.

BTW I think the SUBST command might merely be some wrapper for the 'net share' or 'net use' command. Whatever, the resulting drive mapping is the same.

Philip Nienhuis <philipnienhuis>
Group Member
Thu 29 Jun 2023 07:18:50 AM UTC, comment #2: 

I shared a folder "c:\myfolder_with_a_very_longpath" as "\\mypc\myfolder" in the gui and then mapped a network drive letter to "\\mypc\myfolder". I did not use the "SUBST" command.

What I did is probably more like

NET SHARE ...
NET USE ...

Pointing to octaves movefile command seems to be the right direction for me.

Thanks for your constant work!

Georg Wiora <gwiora>
Wed 28 Jun 2023 08:52:51 PM UTC, comment #1: 

Hmmm, I often save spreadsheet files to some other drive, network or not, and had not met your issue yet.

"drive Z is linked to a network drive mapping a folder on my drive C"
what do you mean exactly? Did you do something along the lines of the "SUBST" command?
What are the commands you used to map your folder on C: into a network drive letter Z: ?

I tried the following:
In a cmd32 window:

C:\>subst z: c:\users\philip\mydocs\octave

C:\>dir z:
 Volume in drive Z has no label.
 Volume Serial Number is 1E0D-1C75

 Directory of Z:\

09-Mar-23  09:59    <DIR>          .
09-Mar-23  09:59    <DIR>          ..
07-Dec-19  21:46            15 446 3D-clip.pdf
07-Dec-19  21:45            21 562 3D-clip.pptx
27-Aug-22  23:10    <DIR>          a
20-Mar-12  23:11         1 261 949 a.csv
15-Feb-15  18:31            14 336 a.xls
02-Apr-13  23:03             1 075 aa.ods
:


and drive Z: neatly shows up in 'My Computer' as a mapped network drive.

Next, in Octave 9.0.0 (dev):

>> cd c:\
>> getenv ("TEMP")
ans = C:\Users\philip\AppData\Local\Temp
>> getenv ("TMP")
ans = C:\Users\philip\AppData\Local\Temp
>> cd z:
>> xlswrite ("subst.xlsx", {"Col1", "Col2"; 1, 2; 3, 4}, "", "", "oct")
ans = 1
>> ls -l subst.xlsx
-rw-r--r-- 1 philip None 7265 Jun 28 22:02 subst.xlsx
>> pwd
ans = z:\
>>


so I cannot reproduce your issue using a "SUBST"ed network drive that way, with Z: as current drive.

However:

>> cd c:
>> xlswrite ("z:subst2.xlsx", {"Col1", "Col2"; 5, 6; 7, 8}, "", "", "oct")
xlsclose: could not zip files in C:\Users\philip\AppData\Local\Temp\oct-IOmPy1 to c:\\z:sub
st2.xlsx
warning: xlsclose: Moving file "C:\Users\philip\AppData\Local\Temp\oct-IOmPy1\z:subst2.zip"
 to "c:\\z:subst2.xlsx" failed with message: "no files to move"
warning: called from
    __OCT_spsh_close__ at line 76 column 9
    xlsclose at line 139 column 13
    xlswrite at line 242 column 11

warning: xlsclose: file z:subst2.xlsx could not be saved.
Read-only? unsupported file format? in use elsewhere?
(File pointer preserved.
 Try saving again later, or with different file name, or as different file type.)
error: warning: STATE structure must have fields 'identifier' and 'state'
error: called from
    __OCT_spsh_close__ at line 77 column 9
    xlsclose at line 139 column 13
    xlswrite at line 242 column 11


so writing a spreadsheet file on a network drive that isn't the current drive doesn't work well.

AFAICS this isn't a bug in xlswrite.m (or rather, the private function __OCT_spsh_close__.m that does the write-to-disk) but rather due to the way Octave's movefile function handles TEMP files with network drives.
There are already a few kludges in __OCT_spsh_close__.m that are meant to help successfully move the .xlsx file created in %TEMP% to the network drive in question, and IIRC all these kludges were needed especially for Windows systems.
While writing this comment I got reminiscences of various 'movefile' issues when I wrote and later on repeatedly patched the OCT interface spreadsheet I/O; there must be old bug reports about it.

I'm inclined to adapt this bug report's title to reflect that it is Octave's 'movefile' function that may be the problem but that has to be confirmed first.

As to "regression", I'm unsure. It may well be that you're just the unlucky first person to hit this. First we have to see if this bug also happens with older io package and Octave releases.
Adapting "Item Group" to mere "Unexpected error".

ATM I don't have time to dive further into this; mid-July I hope to be able to investigate. Hopefully in the mean time other developers / contributors can help finding out what goes wrong with the movefile function and/or how it is used in __OCT_spsh_close__.m .

Philip Nienhuis <philipnienhuis>
Group Member
Wed 28 Jun 2023 02:11:50 PM UTC, original submission:  

When I use XSLWRITE to insert data in an existing excel file the operation fails if the temp folder is on a different windows drive as the target file. In my case %TEMP% points to a directory on drive C while drive Z is linked to a network drive mapping a folder on my drive C. That means it looks like a network connection to windows, but it isn't.


Saving data to file "z:/testdir/myexcelfile.xlsx" ...
mv: inter-device move failed: './myexcelfile.zip' to 'z:/testdir/myexcelfile.xlsx'; unable to remove target: Invalid argument
mv: inter-device move failed: 'myexcelfile.zip' to 'z:/testdir/myexcelfile.xlsx'; unable to remove target: Invalid argument
xlsclose: could not zip files in z:/Users/georg/AppData/Local/Temp/oct-azWA6N to z:/testdir/myexcelfile.xlsx
warning: xlsclose: Moving file "z:/Users/georg/AppData/Local/Temp/oct-azWA6N/myexcelfile.zip" to "z:/testdir/myexcelfile.xlsx" failed with
 message: ""
warning: called from
    __OCT_spsh_close__ at line 76 column 9
    xlsclose at line 139 column 13
    xlswrite at line 242 column 11


Georg Wiora <gwiora>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by philipnienhuis
  • -email is unavailable- added by mmuetzel (Updated the item)
  • -email is unavailable- added by gwiora (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 9 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2024-01-19 philipnienhuis Priority5 - Normal 3 - Low
        StatusNeed Info Works For Me
    2023-07-23 philipnienhuis StatusIn Progress Need Info
    2023-07-23 philipnienhuis StatusConfirmed In Progress
        Assigned toNone philipnienhuis
    2023-06-30 philipnienhuis Carbon-Copy- Added philipnienhuis
    2023-06-28 philipnienhuis Item GroupRegression Unexpected Error or Warning
        StatusNone Confirmed
    2023-06-28 mmuetzel Summary[IO] xlswrite to existing file failes if temp dir is on different drive [octave forge] (io) xlswrite to existing file fails if temp dir is on different drive

    Back to the top

    Powered by Savane 3.13-02a9.
    Corresponding source code