bugGNU Octave - Bugs: bug #46771, symbolic/sympy oct-package fails

 
 

bug #46771: symbolic/sympy oct-package fails

Submitter:  None
Submitted:  Mon 28 Dec 2015 07:19:55 PM UTC
   
 
Category:  Octave Package Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Other
Status:  Fixed Assigned to:  None
Originator Name:  Jari-Pekka Ikonen Originator Email:  -email is unavailable-
Open/Closed:  * Closed Release:  * 4.0.0
Operating System:  * Microsoft Windows Fixed Release:  None
Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sat 30 Apr 2016 05:12:55 AM UTC, comment #30: 

This should be fixed in the next Symbolic release (2.4.0).

Here's the branch: https://github.com/cbm755/octsympy/pull/302

Thanks to everyone who reported this and helped debug/test.

Colin Macdonald <cbm>
Thu 14 Jan 2016 12:55:28 AM UTC, comment #29: 

Are you still willing/able to test things?  I'm experimenting with passing only ascii over the pipe (escaping all high characters above 127).

See here:

https://github.com/cbm755/octsympy/pull/302

You can probably just patch your own installation rather than git cloning if you prefer.  You need the do_highbyte_escapes.m file (its part of that diff).

Colin Macdonald <cbm>
Mon 11 Jan 2016 11:08:04 PM UTC, comment #28: 

Maybe a fix would be to escape all the non-ascii utf-8 chars and pass only ascii back to Octave... Then it won't matter what the intermediate terminal encoding is.  If I try that, I'll post back here and see if you are still able to test!

thanks for your testing so far!

Colin Macdonald <cbm>
Fri 08 Jan 2016 02:52:12 PM UTC, comment #27: 

Having added similar database debug-printing about the missing UTF-encoding I attached part of the output especially when

y=2*x

gave the encoding error.

OctSymbolicDebug1.txt  : error debug output

OctSymbolicDebug2.txt  : lines of debug code

I think this just tells that UTF-encoding characters were not found using this syntax. With the 'backwards compatible' syntax for versions < 3.0 I have no problems encoding the prettyprint.

(file #35946, file #35947)

Jari-Pekka Ikonen <jpikonen>
Fri 08 Jan 2016 01:53:01 PM UTC, comment #26: 

I tested python_header.py change:

/
        DOM = minidom.parseString(ET.tostring(xroot, encoding="utf-8"))
        if sys.version_info >= (3, 0):
            print(DOM.toprettyxml(indent="", newl="\n", encoding="utf-8").decode("utf-8"))
/
and got the missing UTF-characters error back:

  File "C:\Users\Jari-Pekka\AppData\Local\Programs\Python\Python35-32\lib\encodings\cp1252.py",
line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u22c5' in position 170: character m
aps to <undefined>
status =  127
out = <?xml version="1.0" encoding="utf-8"?>

etc.

Jari-Pekka Ikonen <jpikonen>
Wed 06 Jan 2016 10:29:02 PM UTC, comment #25: 

Can you test something?  I wonder if this works for you:


ET.tostring(xroot, encoding="utf-8")
DOM.toprettyxml(indent="", newl="\n", encoding='utf-8').decode("utf-8")


See here where I also use "dbout()" to debug:

https://github.com/cbm755/octsympy/pull/284

"f = ceil(x)" is a good test...

Colin Macdonald <cbm>
Wed 06 Jan 2016 08:42:52 PM UTC, comment #24: 

Great!  Let me look into this a bit more, as Python 3 doesn't work for me on GNU/Linux when using that "encoding=" bit...

Colin Macdonald <cbm>
Wed 06 Jan 2016 10:04:29 AM UTC, comment #23: 

About the integral, at least Reduce and Xcas give some result. Maybe those can be used in octave even if SymPy doesn't integrate it.

I uninstalled the 'bundle' and reinstalled Python, Sympy and symbolic-2.2.4 (without the 'bundle').

The UTF- 'can't encode' -errors returned.

Then, when I change private/python_header.py to

/
if sys.version_info >= (3, 0):
            print(DOM.toprettyxml(indent="", newl="\n", encoding="utf-8"))
        else:
            print(DOM.toprettyxml(indent="", newl="\n", encoding="utf-8"))
/

everything is normal again without 'UTF-missing' -errors.

So it seems like versions 3.0 and above can not implicitly return UTF-encodings. The question is, where should this UTF-info locate? (Windows->updates, services,Python->path,link or a variable, Sympy->path, link or a variable, Octave->update or default settings, XML-version? )

I don't know what is missing from the XML-handling. Maybe it just doesn't default to UTF. Should I change windows settings for this?

Anonymous
Tue 05 Jan 2016 07:04:16 PM UTC, comment #22: 

1.  There is a still a bug here because it should work with your system Python too (without the bundle).  From the error message, perhaps this logic should be changed to set encoding="utf-8" on Python 3 as well:

private/python_header.py:

        if sys.version_info >= (3, 0):
            print(DOM.toprettyxml(indent="", newl="\n"))
        else:
            print(DOM.toprettyxml(indent="", newl="\n", encoding="utf-8"))


(the bundle ships Python 2.x so...)

"git blame" says this commit did this:
https://github.com/cbm755/octsympy/commit/eead64a6333aff5aec941df08ebc68532d49e8c2

Would be awesome if you want to try to hack on this to track it down further!

2.  Re: the integral that doesn't evaluate: I don't know of any version of SymPy that can do this...  File a bug with upstream SymPy if you want.

Colin Macdonald <cbm>
Mon 04 Jan 2016 07:08:08 PM UTC, comment #21: 

So next I uninstalled the symbolic-package, octave, sympy, python and removed them from Windows path. Then I reinstalled octave and the python/sympy windows bundle -package.

Now all symcalc tests so far go through without problems. Some integration examples still don't return the integrated version, but only give back the integral, like:

>>int(sqrt(x+sqrt(x)),x)


ans = (sym)

    /
   |
   |    _________
   |   /   _
   | \/  \/ x  + x  dx
   |
  /

For this it would be nice to have a later version of symbolic/SymPy available for Octave also.

Anyway the earlier 'one by one' -missing character fixing started to be rather frustrating as the error message didn't tell which UTF-character was actually missing and causing the error. And there are quite many UTF-characters any prettyprint output could use.

So, thanks!

Anonymous
Sun 03 Jan 2016 11:37:02 PM UTC, comment #20: 

Ok. I tested several different integrals, found what UTF-characters were missing, changed the decoding_table and now I seem to get proper integral answers from my symbolic 2.2.4 with python 3.5 (35-32) and SymPy 0.7.6.1.

I could still test the 'bundle' if I have time for that. Thank you for your answers.

I will make some more extensive tests to see whether the integration is stable for all inputs.

Anonymous
Sun 03 Jan 2016 09:32:37 PM UTC, comment #19: 

Usually I just do:

pkg install symbolic-win-py-bundle-2.2.4.zip
pkg load symbolic

Philip Nienhuis <philipnienhuis>
Group Member
Sun 03 Jan 2016 08:11:38 PM UTC, comment #18: 

Following is ok:

>> g

g = (sym)

                          / 2\
                          \x /
  /       / 2\      \ / 2\
  \2*x*log\x / + 2*x/*\x /

>> int(g,1,y)

ans = (sym)

      2    /   2\
   4*x *log\4*x /
  e               - 1

>>

Anonymous
Sun 03 Jan 2016 08:01:34 PM UTC, comment #17: 

I can also do the following:

>> g=x^3


g = (sym)

   3
  x

>> g=diff(f^f,x)

g = (sym)

                          / 2\
                          \x /
  /       / 2\      \ / 2\
  \2*x*log\x / + 2*x/*\x /

But as the int(f,x)-error did not mention which characters caused the encode-error (only position) I don't know what UTF-characters are missing.

Anonymous
Sun 03 Jan 2016 07:23:26 PM UTC, comment #16: 

Ok, I downloaded the 'bundle' package. How should I use it?

1) Should I uninstall symbolic (Sympy) 2.2.4 package that I already have?
2) Should I uninstall Octave 4.0.0 before reinstalling with the 'bundle'
3) Do I need to have some Windows path set?
4) Do I need to uninstall my Python and Sympy?
5) I found following in README.pyexe.txt:

/
To build, make sure you have the following files:
  py.py          # the main source file
  py_version.py  # simple version information
  setup.py       # the file needed for py2exe
type:
  python setup.py py2exe
/

Do I need to set a path? Do I type the above to CMD/DOS, Windows, Python, Octave or elsewhere?

6) How do I actually then install this 'bundle'? I didn't find an installer?
7) Is it OK that Windows 10 display-language is Finnish and English(UK) is another installed language? I would prefer to keep Finnish as the display-language due to other compatibility issues.

Sorry for so many questions.

Anonymous
Sun 03 Jan 2016 06:32:09 PM UTC, comment #15: 

I'm fairly sure modifying the code page is not the right solution.

I want to know why this happens on your system and not to other people :(  Maybe we need to force utf-8 somewhere.

Can you try the "bundle" pkg, instead of your own Python install?

https://github.com/cbm755/octsympy/releases

Colin Macdonald <cbm>
Sun 03 Jan 2016 04:50:10 PM UTC, comment #14: 

Sorry, the earlier comment was slightly innaccurate:

One can not move the additional UTF-definitions to the end of decoding_table, because that just brings the "undefined"-error back. Also not "all encodings" were still a problem, for example when additional UTF lines are located just after the original UTF-lines in decode_table, then for example following functions:

>> diff(f,x)

ans = (sym) 2*x

>>



Anonymous
Sun 03 Jan 2016 04:07:08 PM UTC, comment #13: 

The decoding was fixed by adding following lines to the end of decoding_table in file cp1252.py:

    '\u23a1'   #  0x5B -> LEFT SQUARE BRACKET
    '\u23a2'   #  0x7C -> VERTICAL LINE
    '\u23a3'   #  0x5B -> LEFT SQUARE BRACKET
    '\u23a4'   #  0x5D -> RIGHT SQUARE BRACKET
    '\u23a5'   #  0x7C -> VERTICAL LINE
    '\u23a6'   #  0x5D -> RIGHT SQUARE BRACKET
    '\u22c5'   #  0x2A -> ASTERISK

However, all encoding errors, for example for

int(f,x)

.
'charmap' codec can't encode characters in position 196-197: character maps to &lt;undefined&gt;
.

continued. And the encoding possibly should be unique (?) not including several alternatives. I don't know if these fixes are correct (of course more precise character to character matching is clearly necessary). There are ASCII-characters \xC4 and \xC5 both defined in decoding_table (and thus in encoding_table also, i suppose) but still error is that they are "undefined". I tested that these \xC4 and \xC5 characters function expectedly in octave and in python without any errors.

Anonymous
Sun 03 Jan 2016 11:14:53 AM UTC, comment #12: 

I edited the file cp1252.py decoding_table to include lines for the missing characters:

    '\u23a1'   #  0x5B -> LEFT SQUARE BRACKET
    '\u22c5'   #  0x2A -> ASTERISK

etc.

Now, when

y=2*x

returned an error message before the change, I get now

>> y=2*x

y = (sym) 2*x

>>


The file cp1252.py was created from file cp1252.txt, which I did not change. But to fix this for all further similar errors the change should be made to these *.txt files also.

I guess this is to the developers, as they also probably found out in several similar cases. Windows changes in language settings that you recommended did not help in this case.

Thanks for helping!

Anonymous
Fri 01 Jan 2016 09:13:42 PM UTC, comment #11: 

This is something to do with character encoding...  "2*x" should give the error as well.

Is your windows "locale" in another language?  Can you try again with it set to English?

Also, if you try the "octave windows bundle" and see if that works around it?

Colin Macdonald <cbm>
Fri 01 Jan 2016 07:28:42 PM UTC, comment #10: 

These last 2 reports seem to indicate tat there is a problem with the fonts that you are using. Could you try some common font, and report back?

Doug Stewart <dastew>
Fri 01 Jan 2016 05:57:45 PM UTC, comment #9: 

...additionally, matrix definition gives:

>> [x,x;x,x]


Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<string>", line 2, in <module>
  File "<string>", line 158, in octoutput_drv
  File "C:\Users\Jari-Pekka\AppData\Local\Programs\Python\Python35-32\lib\encodings\cp1252.py",
line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u23a1' in position 285: character m
aps to <undefined>
status =  127
out = <?xml version="1.0" ?>
<output_block>
<item>
<f>1003</f>
<f>PYTHON: successful variable import</f>
</item>
</output_block>

<?xml version="1.0" ?>
<output_block>
<item>
<f>1003</f>
<f>PYTHON: Error in var export</f>
</item>
</output_block>

<output_block>
<item>
<f>9999</f>
<f>
&lt;class 'UnicodeEncodeError'&gt;
</f><f>
'charmap' codec can't encode character '\u23a1' in position 285: character maps to &lt;undefined
&gt;
</f>
</item>
</output_block>


error: system() call failed!
error: called from
    python_ipc_sysoneline at line 92 column 5
    python_ipc_driver at line 34 column 15
    python_cmd at line 138 column 9
    vertcat at line 51 column 11

>>

Anonymous
Wed 30 Dec 2015 06:56:50 PM UTC, comment #8: 

done with :

>> sympref IPC system

Resetting the octsympy communication mechanism
Forcing the system() octsympy communication mechanism

>> syms x


OctSymPy v2.2.4: this is free software without warranty, see source.
Using system()-based communication with Python [sysoneline].
Warning: this will be SLOW.  Every round-trip involves executing a
new Python process and many operations involve several round-trips.

>> f=x^2


f = (sym)

   2
  x

>> fi=int(f,x)


Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<string>", line 2, in <module>
  File "<string>", line 158, in octoutput_drv
  File "C:\Users\Jari-Pekka\AppData\Local\Programs\Python\Python35-32\lib\encodings\cp1252.py",
line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position 196-197: character maps
to <undefined>
status =  127
out = <?xml version="1.0" ?>
<output_block>
<item>
<f>1003</f>
<f>PYTHON: successful variable import</f>
</item>
</output_block>

<?xml version="1.0" ?>
<output_block>
<item>
<f>1003</f>
<f>PYTHON: Error in var export</f>
</item>
</output_block>

<output_block>
<item>
<f>9999</f>
<f>
&lt;class 'UnicodeEncodeError'&gt;
</f><f>
'charmap' codec can't encode characters in position 196-197: character maps to &lt;undefined&gt;

</f>
</item>
</output_block>


error: system() call failed!
error: called from
    python_ipc_sysoneline at line 92 column 5
    python_ipc_driver at line 34 column 15
    python_cmd at line 138 column 9
    int at line 145 column 7

>>


>> fd=diff(f,x)

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<string>", line 2, in <module>
  File "<string>", line 158, in octoutput_drv
  File "C:\Users\Jari-Pekka\AppData\Local\Programs\Python\Python35-32\lib\encodings\cp1252.py",
line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u22c5' in position 154: character m
aps to <undefined>
status =  127
out = <?xml version="1.0" ?>
<output_block>
<item>
<f>1003</f>
<f>PYTHON: successful variable import</f>
</item>
</output_block>

<?xml version="1.0" ?>
<output_block>
<item>
<f>1003</f>
<f>PYTHON: Error in var export</f>
</item>
</output_block>
                                                                                               
<output_block>
<item>
<f>9999</f>
<f>
&lt;class 'UnicodeEncodeError'&gt;
</f><f>
'charmap' codec can't encode character '\u22c5' in position 154: character maps to &lt;undefined
&gt;
</f>
</item>
</output_block>


error: system() call failed!
error: called from
    python_ipc_sysoneline at line 92 column 5
    python_ipc_driver at line 34 column 15
    python_cmd at line 138 column 9
    diff at line 106 column 5

>>


I think the octave version is 4.0.0. Should I try another one?

Anonymous
Wed 30 Dec 2015 12:43:38 PM UTC, comment #7: 

Sure, after

pkg load symbolic
sympref IPC system

on both 4.0.1-rc1 and 4.1.0+ all commands work w/o any problems.

Philip Nienhuis <philipnienhuis>
Group Member
Wed 30 Dec 2015 08:58:32 AM UTC, comment #6: 

Thanks for helping, can you also try with "sympref IPC system" then repeat the commands?

Colin Macdonald <cbm>
Tue 29 Dec 2015 01:02:08 PM UTC, comment #5: 

FWIW, on my Win7 64-bit system:

Octave-4.1.0+ for 64bitWindows (dev version): symbolic-2.2.4 just works with all command mentioned;

Octave-4.0.1-rc1 as-is from alpha.gnu.org (I suppose 32-bit):
Immediately after installing:

pkg install -verbose symbolic-win-py-bundle-2.2.4.zip

these commands:

x = sum ("x")
f = x^2
fi = int (f, x)

all work but Octave crashes after:

diff (f, x)

After restarting Octave-4.0.1-rc1, all commands work fine, incl. the initially offending diff (f, x)

@OP:
Do symbolic 2.2.3 and/or 2.2.2 work more reliably?

Philip Nienhuis <philipnienhuis>
Group Member
Tue 29 Dec 2015 08:49:00 AM UTC, comment #4: 

Some other tests:

>> x=sym('x')
>> f=x^2


>> fd=diff(f,x)

OctSymPy v2.2.4: this is free software without warranty, see source.
Initializing communication with SymPy using a popen2() pipe.
Detected Windows: using "winwrapy.bat" to workaround Octave bug #43036
Some output from the Python subprocess (pid 6148) might appear next.

OctSymPy: Communication established.  SymPy v0.7.6.1.
Python 3.5.1 (v3.5.1:37a07cee5969, Dec  6 2015, 01:38:48) [MSC v.1900 32 bit (Intel)]
PYTHON: Error in var export
Resetting the octsympy communication mechanism
Closing the Python pipe...
error: python_cmd: unexpected return
error: called from
    python_cmd at line 144 column 6
    diff at line 106 column 5

>> fd=sin(x)

OctSymPy v2.2.4: this is free software without warranty, see source.
Initializing communication with SymPy using a popen2() pipe.
Detected Windows: using "winwrapy.bat" to workaround Octave bug #43036
Some output from the Python subprocess (pid 11596) might appear next.

OctSymPy: Communication established.  SymPy v0.7.6.1.
Python 3.5.1 (v3.5.1:37a07cee5969, Dec  6 2015, 01:38:48) [MSC v.1900 32 bit (Intel)]
fd = (sym) sin(x)

>> sin(f(x))

OctSymPy v2.2.4: this is free software without warranty, see source.
Initializing communication with SymPy using a popen2() pipe.
Detected Windows: using "winwrapy.bat" to workaround Octave bug #43036
Some output from the Python subprocess (pid 11568) might appear next.

OctSymPy: Communication established.  SymPy v0.7.6.1.
Python 3.5.1 (v3.5.1:37a07cee5969, Dec  6 2015, 01:38:48) [MSC v.1900 32 bit (Intel)]
error: subscript indices must be integers or boolean
error: called from
    subsindex at line 49 column 5
    subsref at line 37 column 22
error: evaluating argument list element number 1

>> f(x)

OctSymPy v2.2.4: this is free software without warranty, see source.
Initializing communication with SymPy using a popen2() pipe.
Detected Windows: using "winwrapy.bat" to workaround Octave bug #43036
Some output from the Python subprocess (pid 10080) might appear next.

OctSymPy: Communication established.  SymPy v0.7.6.1.
Python 3.5.1 (v3.5.1:37a07cee5969, Dec  6 2015, 01:38:48) [MSC v.1900 32 bit (Intel)]
error: subscript indices must be integers or boolean
error: called from
    subsindex at line 49 column 5
    subsref at line 37 column 22

>> [x,x;x,x]

OctSymPy v2.2.4: this is free software without warranty, see source.
Initializing communication with SymPy using a popen2() pipe.
Detected Windows: using "winwrapy.bat" to workaround Octave bug #43036
Some output from the Python subprocess (pid 220) might appear next.

OctSymPy: Communication established.  SymPy v0.7.6.1.
Python 3.5.1 (v3.5.1:37a07cee5969, Dec  6 2015, 01:38:48) [MSC v.1900 32 bit (Intel)]
PYTHON: Error in var export
Resetting the octsympy communication mechanism
Closing the Python pipe...
error: python_cmd: unexpected return
error: called from
    python_cmd at line 144 column 6
    vertcat at line 51 column 11

Anonymous
Tue 29 Dec 2015 07:44:43 AM UTC, comment #3: 

I cannot reproduce either (also on GNU/Linux).

Can you try to narrow it down a bit?  e.g., does "diff(f,x)" or some other commands work?

(Symbolic is annoying to debug on Windows because we can't see Python stderr, its eaten by winwrapy.bat)

Colin Macdonald <cbm>
Tue 29 Dec 2015 02:37:11 AM UTC, comment #2: 

adding maintainer to cc list.

Carnë Draug <carandraug>
Group Member
Mon 28 Dec 2015 07:28:57 PM UTC, comment #1: 

On ubuntu 15.10 and symbolic 2.2.4

x = (sym) x

>>  f=x^2

f = (sym)

   2
  x

>> int(f,x)

ans = (sym)

   3
  x
  ──
  3

Doug Stewart <dastew>
Mon 28 Dec 2015 07:19:55 PM UTC, original submission:  


Octave start...

>> pkg load symbolic

>> x=sym('x')
OctSymPy v2.2.4: this is free software without warranty, see source.
Initializing communication with SymPy using a popen2() pipe.
Detected Windows: using "winwrapy.bat" to workaround Octave bug #43036
Some output from the Python subprocess (pid 1732) might appear next.

OctSymPy: Communication established.  SymPy v0.7.6.1.
Python 3.5.1 (v3.5.1:37a07cee5969, Dec  6 2015, 01:38:48) [MSC v.1900 32 bit (Intel)]
x = (sym) x
>> f=x^2
f = (sym)

   2
  x

>> fi=int(f,x)
PYTHON: Error in var export
Resetting the octsympy communication mechanism
Closing the Python pipe...
error: python_cmd: unexpected return
error: called from
    python_cmd at line 144 column 6
    int at line 145 column 7
>>

(Windows 10, 64 bit,4 cores laptop)
Sympy tested in Python without errors.


Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #35946:  OctSymbolicDebug1.txt added by jpikonen (4KiB - text/plain - UTF-encoding fails)
file #35947:  OctSymbolicDebug2.txt added by jpikonen (417B - text/plain - UTF-encoding fails)

 

Carbon-Copy List
  • -email is unavailable- added by mtmiller (Updated the item)
  • -email is unavailable- added by jpikonen (Posted a comment)
  • -email is unavailable- added by cbm (Posted a comment)
  • -email is unavailable- added by carandraug (Colin Macdonald - maintainer of symbolic package)
  • -email is unavailable- added by dastew (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2016-11-01 mtmiller Dependencies- bugs #47925 is dependent
    2016-11-01 mtmiller StatusNone Fixed
        Open/ClosedOpen Closed
    2016-04-20 mtmiller Dependencies- bugs #46847 is dependent
    2016-01-08 jpikonen Attached File- Added OctSymbolicDebug1.txt, #35946
        Attached File- Added OctSymbolicDebug2.txt, #35947
    2015-12-29 carandraug Carbon-Copy- Added -email is unavailable-

    Back to the top

    Powered by Savane 3.13-02a9.
    Corresponding source code