Mon 25 Sep 2017 04:31:08 AM UTC, comment #82:
I don't know what to say. Thank you both for solve this task. But me very pity that I didn't have a opportunity to realize all my brain.
Have a good day
Andrey
|
Sun 24 Sep 2017 06:47:13 PM UTC, comment #81:
Markus,
I pushed your fix in comment #79 here:
http://hg.code.sf.net/p/octave/io/rev/66b5b97039f0
Thank you very much for your perseverance & patience.
Closing report
|
Sun 17 Sep 2017 07:27:16 PM UTC, comment #80:
I'll have a look this week. Thank YOU for your patience and persistence.
|
Sun 17 Sep 2017 07:08:22 PM UTC, comment #79:
Hi Philip,
The attached patch should be cleaner and also fix some typos in the previous one. Thanks for your patience.
(file #41820)
|
Tue 12 Sep 2017 08:13:20 PM UTC, comment #78:
Thx Markus.
I'll await a new patch (no hurry, when you have time :-) )
|
Mon 11 Sep 2017 05:09:05 PM UTC, comment #77:
Hi Philip,
No worries. Whenever you have time.
Below some comments to your questions:
I forgot to delete this line, which was only useful during debugging.
If I recall correctly, InfoZip messed with relative paths if the zip-file that it creates was NOT located in the same directory as the source files. Those lines move the zipped file created in the temp directory (line 27) to the destination directory and rename it from .zip to .xlsx.
Debug variable I forgot to delete for the final patch. The same with t_unpack in line 60. Sorry.
Same as above. Please ignore.
Looking further below in the patch (lines 92-97), it looks like I might have messed up the indentation (not sure if I read the patch correctly). You might want to ignore that hunk as well...
Sorry for the messy patch. I shouldn't try and hurry...
When I have some minutes, I'll clean the patch.
|
Sun 03 Sep 2017 12:57:59 PM UTC, comment #76:
@Markus
sorry for long delay, after returning from vacation there were (and still are) other (private) distractions keeping me busy.
I went through your patch of comment #74 and have some Q. (line numbers are those of the patch):
_OCT_spsh_close_.m
- Where is variable adddToFile (L.23) used?
- What is the use of hunk L.30-35?
_OCT_spsh_open_.m
-------------------
- What is the use of variable i_open in L.76? It is nowhere used
- Why do you want:
- fid = fopen (sprintf ('%s/xl/workbook.xml', tmpdir));
to be changed by
+ workbook_xml = sprintf ('%s/xl/workbook.xml', tmpdir);
+ fid = fopen (workbook_xml);
It merely introduces a temporary variable that is nowhere used anymore.
|
Thu 20 Jul 2017 09:16:37 AM UTC, comment #75:
@Markus
Dear Markus, I want to offer some my thinks:
Do you think, may be the next title is more informative?
"io's xlsopen/close with "OCT" interface are not stable in non empty workfolder (ssd only)"?
|
Wed 19 Jul 2017 08:46:36 AM UTC, comment #74:
Philip, thanks for pushing that patch, too.
I attached a patch with the remaining changes and changed the title accordingly. (Maybe you can come up with something a little less wordy?)
We can come back to this (rare) issue after you have returned if you like.
Enjoy your free days!
(file #41242)
|
Tue 18 Jul 2017 08:10:43 PM UTC, comment #73:
@Markus,
I pushed the patch in comment #69 as well, thanks.
What shall we do with this bug report?
By the end of next week I'll be away for 3 weeks and until then I have little time left to devote to Octave.
- If you think the unicode issues are solved we can at least adapt the title.
That may have been caused (in part or completely) by the (lack of) unicode conversion too, I hit that several times myself.
As to Infozip and "cd": I know the issue is real but rare, and chasing it down has shown to be quite hard.
In the original bug report Andrey states that it is the COM interface that gave him errors. probably that issue is not so much that the file is corrupt but rather that is has the wrong extension, or contents => Excel stubbornly creates .xlsx files even if the filename extension is .xls (an Excel bug if you ask me).
That reminds me of the fact that the default file extension currently still is .xls. But now that most people have MS-Office >= 2007 .xlsx is a better default. So I just changed that:
http://hg.code.sf.net/p/octave/io/rev/5b5b31fa4020
|
Mon 17 Jul 2017 08:10:07 AM UTC, comment #72:
@Markus
No-No. It's my eng words.
Don't worry. Take a log.
No even 1 bug. And this is a clean result
This is my reply of your Saturday's letter. I'm write here to quickly answer. I said my mail works fine home only.
(file #41219)
|
Mon 17 Jul 2017 08:03:22 AM UTC, comment #71:
@Andrey: I am not sure whether I understood correctly, sorry. Did the tests fail horribly? Or did it succeed without any error?
Did you try with the version without the loop around the "cd" command?
|
Mon 17 Jul 2017 04:40:30 AM UTC, comment #70:
@Dear Markus:
2000 iterations w 0.4-0.5 sec per iteration - no even 1 - clean result. At my work with ssd.
|
Sun 16 Jul 2017 08:23:30 PM UTC, comment #69:
Thank you, Philip, for pushing that patch.
I attached yet another one for a few changes in _OCT_spsh_open_.m and _OCT_spsh_close_.m that should be less controversial.
(file #41213)
|
Sun 16 Jul 2017 04:53:00 PM UTC, comment #68:
@Markus:
I pushed your cset (comment #25) for using native2unicde rather than utf82unicode (and reverse functions) here:
http://hg.code.sf.net/p/octave/io/rev/dbd2adbb3ef1
with some minor style fixes (parens around if conditions).
Thanks for this patch.
|
Sat 15 Jul 2017 11:43:48 AM UTC, comment #67:
.ods and .xlsx are zipped archives containing 2 or more trees of subdirs, all (including the "root" dir) containing XML files.
xlswrite (or rather some private functions that are called in a chain of other functions) unzips such an archive to a temporary subdirectory, modifies it, and in the end zips it back into the original place, overwriting the original .xlsx/.ods file.
After processing the temporary subdir with the unzipped archive is wiped.
xlsread and xlsfinfo (or rather private functions .....) merely do the same but don't modify it and in the end just wipe the temporary subdir.
The hairy issue here is that this bug is very hard to reproduce; sometimes it pops up, most of the time it doesn't. It makes one think of OS issues behind the scenes (like temporary swapping to disk of, or briefly blocking access to, required system libraries).
At one time I figured that maybe the original archive was somehow locked by zip/unzip, and releasing the lock might be delayed, but AFAIU that isn't the case.
But the thing that I suspect the most on at least Windows is virus/malware scanners. These are explicitly blocking all access to new files until those files can be declared "safe". (Which doesn't explain why I saw this bug mostly on fast HDDs/SSDs.)
In the io package's spreadsheet test functions there are pause(0.25) statements after each call to xlswrite (or odswrite) to avoid any occurrence of this bug.
On my two dev boxes I just tried with all delays set to 0 (i.e., pause(0), or no delay) and I couldn't reproduce this bug at all anymore. Next week I can try on my PC at work where I hit this bug more often.
|
Fri 14 Jul 2017 08:58:49 PM UTC, comment #66:
I tried using the script in comment #65 and didn't see a problem.
I'm using io 2.4.7 on a Windows system with a build of dev Octave (Octave hg id 5296657cde44) but I'm not sure what mxe-octave version. It was built around the beginning of May.
My approach to debugging this problem would be to simplify the xlswrite function to the point where it just does the minimum to demonstrate the problem.
Exactly what is xlswrite doing in this case?
|
Fri 14 Jul 2017 08:29:52 PM UTC, comment #65:
Attached is a changed patch that doesn't use the loop around "cd".
Andrey used the following script to trigger the bug:
Could you please first try with the original files in io 2.4.7 whether the script triggers the bug for you? Does the attached patch still fix the issue?
(file #41206)
|
Tue 11 Jul 2017 02:51:20 PM UTC, comment #64:
Philip, can you increase "max_no_change" to 200 (or something higher) and try again. Maybe also increase t_max if this were necessary.
In the discussion in bug #51388 I understood that the very similar code there was triggering the bug for you. Maybe it needs more iterations to be triggered?
I will try and get the dependencies for some (or all) of the other interfaces within the next few days. Will report back when I have some results.
|
Mon 10 Jul 2017 04:56:20 PM UTC, comment #63:
This is what I get, writing your script as 'test_cd_delay.m':
To run my suggestions in comment #61 you'd need at least Java (just a JRE), plus the required Java .jar libraries for each interface. LibreOffice is an extra. If you're on Windows and have Excel installed, you'd also need the Windows package loaded.
I have the following:
and LibreOffice (64-bit) is in 'C:\Program Files\LibreOffice 5' (needing the following .jars & other Java stuff), taken from chk_spreadsheet_support.m's listing:
If you don't want to install all of that (which I'd understand), maybe we can wait for someone else who runs Windows and does have Java etc installed.
|
Mon 10 Jul 2017 03:38:28 PM UTC, comment #62:
Philip,
Currently, I don't have LibreOffice or even Java installed. I always was using the OCT interface. If I understood correctly, both are needed to trial all available interfaces. Am I missing anything else?
Meanwhile, can you check the following code snippet for two directories that failed the test in bug #51388 (comment #2 there)?
It runs quite fast on a machine that doesn't experience the error (50-60ms for me).
You might need to tweak the abort criteria (first 4 lines) to get reliable results...
|
Sun 09 Jul 2017 08:05:10 PM UTC, comment #61:
Markus,
Only the OCT interface invokes "cd" explicitly.
If you look in the test functions
io_xls_testscript.m
io_ods_testscript.m
you'll note many pause statements for most if not all interfaces. All of those were needed to circumvent errors due to the OS apparently not being ready for the next command in those scripts.
Setting the value of "dly" (the delay time) to zero will help to illustrate where the errors exactly occur for which interface.
Results on your computer(s) would be interesting.
I was thinking that the "while strcmpi (pwd, ..)" loop could be morphed into a piece of code that assesses a machine-specific required delay time for each machine it runs on at the start of each first invocation of xlsopen or xlsclose, using persistent or global variables. That while loop may need to exit after the total accumulated delay would reach a max. of 1 second or so to avoid a hang.
|
Wed 05 Jul 2017 09:46:11 PM UTC, comment #60:
@Markus
As to other affected interfaces, I'll sort that out (again).
A notable example is UNO (invoking LibreOffice behind the scenes using a Java-UNO bridge). LibreOffice invokes a built-in zip procedure to unzip/zip .ods and .xlsx files, and it was with UNO that I first hit this bug some years ago.
As to move to core:
I doubt if all of io (spreadsheet & XML) can be moved to core; I think only XML and the OCT spreadsheet interface will do.
The COM and Java-based interfaces could/should probably be left behind in a weeded-out io package. COM relies on Excel itself + the windows package so that is obviously out-of-the-question (although COM is a "system library" in GPL terms). Furthermore most octave devs don't care much for Java and if we include Java-based interfaces we'd introduce new dependencies.
AFAICS most of the required preparatory work has already been done, I need to polish and update it.
|
Wed 05 Jul 2017 05:09:55 PM UTC, comment #59:
Philip,
can you give me a pointer to which other interfaces are affected? I could only find OCT using zip/unzip or unpack or manually changing directories.
I read through the threads you linked to. I would definitely like to see the io package merged into core.
If you think I can help getting the files ready for the move, please send a PM or an email with what you think should be done (or something more public if that is preferred). I will try and see if I can be of any support.
|
Wed 05 Jul 2017 12:40:39 PM UTC, comment #58:
Markus,
No i'm not at all opposed to fixing io for the time being. But it'll have to be fixed for all affected interfaces, not just OCT.
Later this week I'll look into that.
FYI, fixing OF packages ahead of core releases while keeping backward compatibility is merely business-as-usual :-)
Some other background that may be relevant:
At OctConf 2017 I spoke with JWE about moving spreadsheet + XML I/O to core Octave (an old wish of him, see
http://octave.1599824.n4.nabble.com/Spreadsheet-support-in-core-Octave-instead-of-io-package-tt4666947.html#a4666955
and
https://lists.gnu.org/archive/html/octave-maintainers/2015-05/msg00074.html).
At OctConf 2017 I hoped it could happen around (last) June. Well it now looks like that'll be a bit later... :-)
One thing I've been preparing for a.o. that move to core, is eliminating duplicate code. We now have various xls....m functions doing similar things as ods....m counterparts; some time ago I've combined them and made the one just a wrapper for the other. In the mean time io package and core development have continued so this needs to be merged. I think any patches for tackling zip functions would need to be adapted to the merged functions.
|
Wed 05 Jul 2017 09:27:13 AM UTC, comment #57:
I opened bug #51388 to track the issue with the "sluggishness" of "cd" on some systems.
@Philip: No hurry. I just wanted to make sure the other patch does not get lost as the scope of this bug report changed direction...
Since Octave packages get updates more regularly than core, are you still interested in working around that in io rather than waiting for an update of core Octave?
|
Wed 05 Jul 2017 06:24:10 AM UTC, comment #56:
Yeah it would be a hard sell :-)
That leaves us with adapting unpack.m / unzip.m / gunzip.m.
Again I'd rather do this in a central place (i.e., core Octave) rather than for individual functions in an OF package.
BTW I mentioned before there is a similar bug report where this issue with zip was also mentioned and where (IIRC) Mike figured it would be a (fast SSD/HD) driver issue. But I failed to dig it up.
I'll cc him, hopefully he still remembers where it was.
(as it may help to tackle/close that bug report as well.)
...oh but he IS cc'd...
{slightly OT]
@Markus, now that I got some issues out of the way with mxe cross-builds I'll review and push your patch from comment #25, thanks for your patience :-)
|
Tue 04 Jul 2017 10:11:18 PM UTC, comment #55:
InfoZIP is the default zip and unzip programs on all GNU systems, so these should be the exact same versions (3.0 and 6.0 respectively) that are on every GNU/Linux system out there, and are provided to mxe-octave via msys standard packages.
I think trying to get Octave core to use something other than the default zip and unzip programs to deal with zip format archives is kind of a hard sell.
The "while-strcmp-cd" loops in this patch look very suspect to me, there should be no situation where the pwd does not equal the directory that was just cd'd to. Are these loops actually necessary? Do we need to fix cd / chdir?
The rest of the patch I guess has to do with zipping and unzipping to a temporary file and then moving the result over the real destination file name.
Are there bug reports about why the pack and unpack functions are not good enough to do what is needed here?
|
Tue 04 Jul 2017 06:25:52 PM UTC, comment #54:
@Andrey and Markus:
Good detective work, thanks. It sounds very convincing to me - I've always wondered why it is on systems with a fast HD / SSD that we see this bug; your explanation fits this observation perfectly: such systems finished the actual write operations before infozip itself is ready.
I am not prepared (yet) to just apply your patch for the OCT interface:
- All io pkg spreadsheet interfaces that rely on zip need a similar patch.
- There's also UNO (AFAIK relying on gzip) that has similar issues.
As I have a hard time believing that issues with zip / infozip are limited to the io package I'd suggest to bring this (= replacing infozip and/or fixing unpack.m) up in the maintainers ML.
|
Tue 04 Jul 2017 08:48:04 AM UTC, comment #53:
Thanks to a lot of trial and error with Andrey's help, we could pin the second issue down to some strange behavior of InfoZip:
InfoZip relies on the current working directory (cwd) to function properly. On some systems (like on Andrey's and others), it looks like it takes some time to update the cwd properly. That does not seem to be handled by Octave's "unpack" function.
When calling InfoZip directly and taking special care that the cwd has changed properly, the behavior that Andrey observed disappears on his system.
The attached patch includes these changes.
Btw, we quickly checked with using 7-Zip (because we both had it at hand). That program accepts absolute paths for source and destination (unlike InfoZip that relies on the cwd for relative paths in the archive). Using 7-Zip, that issue disappeared as well (without any additional hassle).
I am still not sure why on some systems the issue with the slow cwd change exists in the first place. But we might consider using a more modern program than InfoZip (which dates back to 2008 for the zip part and 2009 for the unzip part) that better handles those systems.
PS: The patch for the other issue (transcode to UTF-8 from any supported code page) is in comment #25 of this already quite long discussion.
(file #41114)
|
Wed 21 Jun 2017 02:53:30 PM UTC, comment #52:
when I deleted ";" in your _OCT:
|
Wed 21 Jun 2017 02:24:51 PM UTC, comment #51:
Output file is absent always (xlsx)
|
Wed 21 Jun 2017 02:06:00 PM UTC, comment #50:
your correction in c:\octave-4.2.1\share\octave\packages\io-2.4.7\private\__OCT_spsh_open__.m
script:
|
Wed 21 Jun 2017 01:58:52 PM UTC, comment #49:
Of course. It is need to reopen Octave to clean results, otherwise - it works from cache:
|
Wed 21 Jun 2017 01:24:58 PM UTC, comment #48:
Andrey, could you please find the folder where "__OCT_spsh_open__.m" is and replace it with the one I attached?
This folder is for me: C:\Octave\Octave-4.2.1\share\octave\packages\io-2.4.7\private
Can you try whether you can still reproduce the issue and post any error messages or warnings you see at the command prompt?
(file #41012)
|
Wed 21 Jun 2017 05:44:39 AM UTC, comment #47:
May be at the empty folder "delete" command not expend time to search a file, work quickly and release the hdd.
Otherwise at the full folder "delete" command expend time to searching file among other but time is continue and next command "xlswrite" attempt to work and collision is appear.
I know my thinks, words are not professionally but they are. (I have no more time to study in depth your subject now). I viewed you understood me. Thank you!
--
About second bug - xlsx. Thank you too for understanding and for a patch. I will try compiling development packages later.
I don't want to use your time in vain. Therefore I think it is the part of end of topic. If I recieve new results I will write here.
Have a good day. Thank you very much both (Philip Nienhuis and Markus Mützel). I discovered new thinks for me.
Best regards
|
Wed 21 Jun 2017 05:12:29 AM UTC, comment #46:
You said in detail. Thank you
|
Tue 20 Jun 2017 06:36:04 PM UTC, comment #45:
Andrey,
The bug you are referring to now (apparently unreliable writing to file) has been reported before (can't find it now, was about 3-4 months back IIRC). It has to do with very fast hard disks. I hit this already 3-4 years ago.
Mike Miller and John Eaton figured it would be a bug in hardware drivers, but now I think it is really a bug in zip (that zips the various XML files together, .xlsx is a mere zip archive):
- zip hands over the actual writing of the zip file to disk to lower level routines and returns.
- The lower level routines may take their time to do the actual writing.
- Next however, as zip has returned, unzip is invoked and it tries to read the .xlsx file that hasn't been finished yet.
I think zip should return only after having verified that its output file has been finished completely.
In the io_xls_testcript.m and io_ods_testscript.m function files in the io package you can see that I've entered various pause commands to separate xlswrite and xlsread (odswrite and odsread) calls. Those delays are there for the very same reason you complain about.
|
Tue 20 Jun 2017 03:07:39 PM UTC, comment #44:
Re: It would be easier to follow if you post the errors and warning directly as text.
Ok. I will write warnings and errors here. Thank you
As Philip said:
Repeating of xlswrite until result or this pause can do stable output by me.
Me very pleasantly your both attention to me. But I want say I try to recognize this bugs to help you both to write Octave's library and Octave 4.4.
It's not a problem that stopped my work. But I wanted to help you.
Philip said has no time to me today but will have later. And I didn't want to divert him today. I wrote to the future
I hope you receive my bug's results in later time. I don't think the reason in my computer and not need to correct whole Octave with pause only for me. Therefore I try to understand this bug many times
Thanks very much. I will absent until tomorrow
|
Tue 20 Jun 2017 02:51:30 PM UTC, comment #43:
Those files aren't really strange. That is what the more recent MS Office files are: zip-files with a certain structure.
It would be easier to follow if you post the errors and warning directly as text.
@Philip: There indeed seem to be two instances of "return" in _OCT_spsh_open_ (and at least in _COM_spsh_open_, too) where the last output argument "lastintf" is not yet set. I will leave that error to you because I am not sure what that argument is used for and what should be returned in those cases.
There seems to already be a manual pause after unpack in line 63:
It might be that this pause is too short on Andrey's system and the file I/O needs more time to flush. Maybe waiting some time whilst checking for the last file in the return value of unpack to be available might be necessary? Any better ideas?
|
Tue 20 Jun 2017 02:43:45 PM UTC, comment #42:
It's an example. This bug appeared when I wrote a middle size script. I change my script, run it and view results in xlsx. But sometimes I changed my script, ran it, but result was absent
|
Tue 20 Jun 2017 02:38:19 PM UTC, comment #41:
result is same on another computer. Stable non-assembled xlsx after 1-4 attempts to run with 1-9 folders and successfully in empty.
(file #40957)
|
Tue 20 Jun 2017 02:06:47 PM UTC, comment #40:
You can take any xlsx file. Change it file type from .xlsx to .zip and unpack it to new directory. You will see same strange files as I sent you
|
Tue 20 Jun 2017 02:01:56 PM UTC, comment #39:
I attached pictures.
It's work very quickly with 'oct'.
HDD is flash sata2. Very quick.
Strange situation is:
All is correct in empty work directory and xlsx dissamble by parts when work dir is full.
This strange files is a part of one xlsx file, but I don't can assemble it manual. It need to do by Octave automatically.
Files attached. Their contents same as in dissasembled xlsx but they are not a strong clone.
I'm sorry, I forgot to attach warning last time
(file #40953, file #40954, file #40955, file #40956)
|
Tue 20 Jun 2017 01:43:53 PM UTC, comment #38:
Still cannot reproduce here even with bigger files.
Good to here that the short pause solves the issue for you.
If that is helping, the issue might be due to a busy or slow hard disk when the file is created.
What does system("set temp") return for you? Is that a local drive?
If you do not see any errors or warnings in the command window, there is probably little we can do to help.
|
Tue 20 Jun 2017 12:12:40 PM UTC, comment #37:
Dear Markus Mützel,
Is I wrote that you request me?
It's not an utf bug, but this bug I described in first message
|
Tue 20 Jun 2017 12:05:52 PM UTC, comment #36:
"pause(0.05);"
all correct.
I don't know how this delete's command affect to no stable output, but pause is an alternative soluition.
addiction:
bug is absent when use with 'com' interface .
output script is:
|
Tue 20 Jun 2017 12:04:05 PM UTC, comment #35:
I'm sorry for long time. I don't know a reason, but I found an alternative solution. Below about it.
Well.
Octave 4.2.1 from zip: https://ftp.gnu.org/gnu/octave/windows/octave-4.2.1-w64.zip without any patches (had no time to try their)
io *| 2.4.7
windows *| 1.2.4:
Initial conditions:
empty work folder (05octave to me)
Close and open your Octave.
We create a new script: o076xls.m
with this content:
and run it by pressing F5.
Running it many times it do right output xlsx.
Then we execute manually this command:
to create 9 folders (this is an example a non-empty folder with main script).
Then run our script by pressing F5 many times. 3-4
And sometimes we have no an output file xlsx, but have some parts of it.
I found just now:
If we write a pause:
|
Tue 20 Jun 2017 11:12:15 AM UTC, comment #34:
Thanks. I doing it now. 10-15 minutes to write steps
|
Tue 20 Jun 2017 11:05:19 AM UTC, comment #33:
I cannot reproduce this here. But I cannot read Russian, so I might have gotten some steps wrong. Can you please provide easy steps in English if possible with the code you are running and what warning/errors you see in a new bug report.
Please, keep this report dedicated to the Unicode issue with xlswrite.
|
Tue 20 Jun 2017 09:20:49 AM UTC, comment #32:
I posted in pictures:
I don't know, how to assemble this strange files in one, but I understand they are parts of file xlsx that I waited.
Bug in this:
in empty workspace xlxs is created, in full directory - not always
(file #40944, file #40945, file #40946)
|
Tue 20 Jun 2017 08:47:31 AM UTC, comment #31:
Good day to all! It's a good day really.
I found conditions my main bug and this behavior's description. I am posting now (not with encoding - another bug - first bug of this topic)
|
Thu 15 Jun 2017 08:26:53 AM UTC, comment #30:
Thank you both
|
Wed 14 Jun 2017 07:00:48 PM UTC, comment #29:
Markus, thanks for the patch.
I'll review and probably push but I'm busy with other things now.
|
Wed 14 Jun 2017 10:11:04 AM UTC, comment #28:
Thank you. I will try it if I can. I will read their
|
Wed 14 Jun 2017 09:05:42 AM UTC, comment #27:
Andrey, that patch applies to the Mercurial repository of the io package. Philip who is its maintainer might review and push it there.
Anyway - like I already wrote - the patch would help only with the current development version of Octave which will eventually become version 4.4. You could test that by cross compiling Octave for Windows.
If you are interested in helping to test the next version of Octave, you could start by reading the Wiki pages:
http://wiki.octave.org/Building
http://wiki.octave.org/Windows_Installer
|
Wed 14 Jun 2017 05:54:06 AM UTC, comment #26:
Thank you. Is I correct try to use a patch? Please, see a attached screen.
I found in Internet next instructions:
#! /bin/sh
#
# To apply this patch, cd to the top level Octave source directory and
# run this file through /bin/sh. It will first remove any files that
# have been deleted from the source distribution since the last
# release and then update the sources with patch(1)...
Is same actions need in Windows?
I use: bin\sh.exe bug51203_xls_unicode.patch
I added "#" in line 7.
Is need to run via Windows Octave-editor? I'm sorry for this simple question.
(file #40912)
|
Tue 13 Jun 2017 08:12:44 PM UTC, comment #25:
Attached is a patch for Octave IO that should fix the issue on current development builds. I only tested with system code page CP1250 but it should work with other system code pages, too.
(file #40910)
|
Tue 13 Jun 2017 06:27:11 PM UTC, comment #24:
Very thanks, Markus Mützel. Me pleasantly you are thinking about it.
It is not a problem by me today. I write English only ascii 32-127 scripts today.
I have a some perfectionist's behavior and I like to do a work perfect. Therefore I interest it. Because native characters are very friendly to default user.
I will wait Octave 4.4 patiently. I think I will understand some important features during and until an it's creation. May be it will be useful to me or other.
I view this forum and will view. I understand you. Thank you!
|
Tue 13 Jun 2017 05:04:22 PM UTC, comment #23:
Andrey, the behavior that you are describing is due to the fact that Excel needs the text encoded in UTF-8. If you save your file in UTF-8 and do not mangle with the strings, Excel is happy.
However as you wrote, that does not match Octave's nor Matlab's default encoding (and you might hit other bugs like bug #51210).
As of Octave version 4.2, there is no Octave function to manually convert between arbitrary encodings. Octave 4.4 will contain "native2unicode" and "unicode2native" which could be used for this. Unfortunately, there is no way I can think of to solve your issues at the moment (other than manually writing a function that converts strings between your local encoding "Windows-1251" and UTF-8).
|
Tue 13 Jun 2017 03:15:54 PM UTC, comment #22:
Good day.
I found next:
Linux & Oct work ideal - plots, xls-es.
If in Windows in Octave to set editor encoding UTF-8 and to write script, It will be in UTF-8. And it do output in xlsx (...'oct') fine as said Markus Mützel. Thanks. But this script incompatible with default users.
But. If to write script in UTF-8, for example in Notepad++, to convert, to save in UTF-8 and to open in Windows Octave, then file will be not readable by me, but it correctly do output in Excel w/o flag "convert_utf". Text is readable.
But if you to run UTF-8-script in Windows-Matlab, then you can see in Excel same characters as in Matlab/Octave editor.
I don't can to understand it now. I see, that Matlab to do output this character, that view.
But Octave to do output character that it load. But it don't display it. That is: There are a script in UTF-8. Octave with 1251's editor don't may to display it correctly. But It do output exactly as scriptt saved.
Although, it must to do output that it display. In my opinion.
I'm sorry for my long comments and my write style. I will think further.
(file #40909)
|
Sun 11 Jun 2017 02:06:11 PM UTC, comment #21:
Title adapted
|
Fri 09 Jun 2017 11:59:30 AM UTC, comment #20:
I can confirm Andrey's observation regarding text in graphics. I opened a separate report for this issue in bug #51210.
|
Fri 09 Jun 2017 09:33:46 AM UTC, comment #19:
All right. I am a new user here. I'm studying behavior on this site.
Yes, I use alternatively way to output by repeating by cycle today.
Your ideas and reflections are useful by me.
I found, that UTF8 encoding by Octave editor allow to do output via 'oct' wide-ASCII characters correct. utf8.m files are opened on Linux without troubles on Linux in any math soft (because Linux has utf8, I think). In Windows Matlab utf8.m file don't readable.
I use Octave to write .m files, but default users may use both soft in both or in three Operating Systems. Therefore this solution may be using by me today, but this is not universal.
And, when I changed Octave-Editor locale to utf-8 by default - Plotting fonts don't display normally.
I will to study encoding problem now, but there is available to write English scripts only by me to current tasks today.
--
Yea. I accept your action - to set status "postponed".
I'm sorry that my answers are too long.
Thank you all very much. Me pleasantly for your time and thoughts. You can write me later in any time. I will answer and I will to try any your cases and examples.
With best wishes
(file #40892)
|
Fri 09 Jun 2017 08:55:58 AM UTC, comment #18:
Dear Andrey,
Currently I lack time to dive into your issues more thoroughly, but I will do that later.
But I understand you have a workaround so for the time being I'm lowering priority of this bug report and set status to "postponed" as a real fix (unicode processing) will be implemented later on.
|
Fri 09 Jun 2017 08:08:00 AM UTC, comment #17:
But, Thank you very much, that you solve my problem with encoding.
I will to study it soon today or in the coming days too.
Best regards.
|
Fri 09 Jun 2017 06:29:03 AM UTC, comment #16:
Good morning.
Very thanks, that you are discussing about them. I don't carefully understand all information, that you wrote to each other with 49222, I've read it, but I understand a common idea and I will understand it.
Do I understand correctly, that future (and being developed now) Octave 4.4 will contain native2unicode and unicode2native functions?
In Matlab they work non ideal with all characters but work.
If Octave will contain their, I think it will be good. Although how binary data in unicode write in cell when transfer to xls/xlsx? But it will be good in any case.
I found 'UTF-8' encoding in Octave. It's one of solutions. Although this code is not readable by Matlab users. Well.
Thanks very much you both.
It about encoding.
How about this no stable output with 'oct', when other files or/and folders are exist? It's interested by me first. And I think, that this is not small inaccuracy, but this is a more important bug. Both: xlswrite and xlsopen-oct2xls-xlsclose don't do a guaranteed output, when another folders/files are.
(file #40890)
|
Thu 08 Jun 2017 05:26:33 PM UTC, comment #15:
@Markus:
yes I'd appreciate cooperation.
I knew the limitations of what is now implemented in the io package.
Note that only some of the available spreadsheet I/O interfaces need conversion, the others do it OOTB.
I've seen native2unicode and its sibling but they consistently give errors. As soon as that is solved we can proceed.
|
Thu 08 Jun 2017 05:05:15 PM UTC, comment #14:
Andrey,
Working with encodings is a mess. Especially on Windows.
To make your live easier, I would always use UTF-8 with Octave. To change your default encoding in the editor, please open your preferences (menu "Edit" -> "Preferences"). On the very bottom on the "Editor" tab, select "UTF-8" from the drop-down list.
Now close all open tabs in the editor. When you open your m-files again, you might notice that your Cyrillic characters look messed up. You can either convert your m-files to UTF-8 (e.g. with Notepad++) or correct the messed up characters manually.
With these settings, the following example works for me:
Do not use the "convert_utf" setting in this case. Your input is already UTF-8 encoded.
Philip,
The dirty hack that we used to solve bug #49222 only works for encodings which match Unicode in the 128-255 area. That means it only works for "Latin-1" (or to some extend "ANSI") encoding. I did not realize this at the time (sorry).
The current dev builds include "native2unicode" and "unicode2native". If you like, we could work on integrating these functions to make the io package work with all supported encodings when Octave 4.4 comes out.
|
Thu 08 Jun 2017 02:37:42 PM UTC, comment #13:
I'm sorry, I read ASCII 128-255 localized by each language as I understand
|
Thu 08 Jun 2017 02:10:22 PM UTC, comment #12:
ascii http://book.itep.ru/10/ascii.htm
|
Thu 08 Jun 2017 01:44:51 PM UTC, comment #11:
Russian letters are char codes: 168 and 192-255, that wider than [32:127] and I think, their need to convert as says in https://octave.sourceforge.io/io/function/oct2xls.html but today converting is not right. It need to think about it at second time
|
Thu 08 Jun 2017 01:25:13 PM UTC, comment #10:
I don't think, that my solution is correct. And I thought, that I must to notify a world as says in https://octave.sourceforge.io/bugs.php
Thanks.
|
Thu 08 Jun 2017 12:59:21 PM UTC, comment #9:
Yea, Andrey
|
Thu 08 Jun 2017 12:56:55 PM UTC, comment #8:
Is my English so bad? I'm sorry. I tell only in forums and shops. I try to speak better. I'm very sorry again.
--
I understood about Linux and utf8. Thanks. I will remember it feature. I will try to solve an encoding problem later.
('com'-interface in windows correct do output in xls file, but don't do in xlsx. In Linux com interface don't work as I think, and oct don't output in xls.)
--
I don't touch this xlsx file during running script. I understand it.
I solve this task by using 'oct'-interface and repeating output again and again while file is not exist or counter t<=3.
--
I try to use xlsopen->oct2xls->xlsclose, but result is same as xlswrite:
When work folder with script(which contain oct2xls or xlswrite) is empty. The script work ideal.
When I create 9 folders (for example:)
a result is not guaranteed. After 1 times, after 2-5 times, my script don't output xlsx-file, displays warnings or errors and create strange xml-folders and files.
If I continue and I press F5 to run script. It will be work and do output to xlsx.
--
Thank you very much for answer. You can say me any words.
|
Thu 08 Jun 2017 12:29:20 PM UTC, comment #7:
Sorry Aleshin (or is it Andrey?) I may not understand you very well; I'm sorry I don't know better Russian :-)
Is your problem solved now?
The warning:
is telling and may be caused by having o076.xlsx opened in another program so that it is locked and cannot be overwritten.
Did you try with another filename?
It does not surprise me that you do not need the "convert-utf" option on Linux. The Octave terminal on Linux is a better one than the Octave terminal that is used on Windows; therefore on Linux double-byte chars are probably treated much better by Octave so that they do not need treatment with the "convert-urf" option.
|
Thu 08 Jun 2017 12:22:58 PM UTC, comment #6:
while (exist(filename)~=2)&&t<=3
|
Thu 08 Jun 2017 11:58:36 AM UTC, comment #5:
xlswrite(filename,cell1,'sh1','A1');
t=1;
while (exist(filename)~=2)&&t<=1
xlswrite(filename,cell1,[],[],'oct');
t=t+1;
end
if (exist(filename)~=2)
fprintf('Please repeat script\n');
end
------
This is not liked by me, but today it is works only that. Me smile.
|
Thu 08 Jun 2017 11:57:16 AM UTC, comment #4:
|
Thu 08 Jun 2017 11:55:54 AM UTC, comment #3:
|
Thu 08 Jun 2017 11:54:54 AM UTC, comment #2:
Me pleasantly, that you answered me. Thank you very much. Thank you very much. I studied, understood and tried new variant a long time, therefore I write only now.
I replace xlswrite by oct2xls as you recommended.
Russian symbols with 'oct' output correct only in Linux without structure "convert_utf". Me smile. I attached screenshots and xlsx-es.
But this is not priority task. Today, it is may to read and write in English only.
-------------
Main task - is to do sure output in xlsx - was not obtain by me with oct2xls. In Linux - no problems. Sure and guaranteed. In Windows, I have same result. Therefore, problem may be is not be related with xlswrite and oct2xls. I don't know, what.
In two words: when script to be with files and folders - often there are xml-files instead file.xlsx.
Yea, io-2.4.7:
oct2xls:
xlswrite:
Today I use a not right solution - to repeat output:
|
Thu 08 Jun 2017 09:24:53 AM UTC, comment #1:
The issue is that you're being hit by double-byte character issues; I'm sorry for that. Octave isn't very good at handling double-byte characters esp. on Windows.
Luckily there's a way out for you.
First, be sure to upgrade the io package to 2.4.7 (latest version).
Then in your script change the line:
xlswrite(filename,cell1,[],[],'oct');
into:
xls = xlsopen ("o076xls.xlsx", 1, "oct");
xls = oct2xls (cell1, xls, 1, "", struct ("convert_utf", 1));
xls = xlsclose (xls);
and try again.
If I do that I get the attached file that is read perfectly fine by Excel and LibreOffice.
FYI: the option "convert_utf" is what you need. Try:
help oct2xls
or
help oct2ods
and
help xls2oct
or
help ods2oct
for more info.
Please report back!
Release => other (as it is an OF package issue and not core Octave)
(file #40885)
|
Thu 08 Jun 2017 07:12:03 AM UTC, original submission:
Welcome.
function xlswrite with interface 'com' output in both formats: xls and xlsx. But an Excel don't open xlsx, send error "file corrupted".
with interface 'oct' in Windows: If variable contain Russian symbols, file is outputed, but Excel don't may to read this. It may attempt to open, but it view only some digits.
with interface 'oct' in Linux(Ubuntu) and LibreOffice: errors are absent. All right and correct.
But in Windows xlswrite sometimes not output data in file. Instead Octave send warnings listed below, and there are appear a strange folders and files:
When Octave's work directory is empty - all right. But, if no, for example, I create 9 folders. You can open a file with xlswrite(...'oct') and don't get output.
To view this bug, you can create a some amount folders, open m-file and run it. From 1 to 10 attempts - bug will appear guaranted
|