bugGNU Octave - Bugs: bug #55765, Missing function: normalize (same...

 
 

bug #55765: Missing function: normalize (same function name used in ltfat and matgeom pkgs)

Submitter:  Nicholas Jankowski <nrjank>
Submitted:  Fri 22 Feb 2019 05:07:51 PM UTC
   
 
Category:  Octave Function Severity:  1 - Wish
Priority:  3 - Low Item Group:  Feature Request
Status:  Fixed Assigned to:  None
Originator Name:  Nicholas Jankowski Open/Closed:  * Closed
Release:  * dev Operating System:  * Any
Fixed Release:  8.1.0 Planned Release:  8.1.0
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Wed 12 Apr 2023 11:45:28 PM UTC, comment #24: 

agreed. closing as fixed.

Nicholas Jankowski <nrjank>
Group Member
Wed 12 Apr 2023 11:25:00 PM UTC, comment #23: 

@nrjank: This bug report was about adding a missing function.  Since it has  now been added I think it is okay to close this report.

Any issues with the the implementation of normalize (such as failure to preserve sparseness) can be the subject of a new bug report.

Rik <rik5>
Group administrator
Wed 12 Apr 2023 10:55:43 PM UTC, comment #22: 

seems the current function pushed to 8.1 is ok. no complaints at least.

two potential outstanding issues.  (1) this version still doesn't preserve sparseness.  (2) there are some matlab incompatibilities due to us not having had compatible mean, median, std, etc., functions when i wrote this one.

could retitle to keep a placeholder for eventually fixing that. or close and leave as a later report.

Nicholas Jankowski <nrjank>
Group Member
Wed 12 Apr 2023 09:05:51 PM UTC, comment #21: 

@nrjank: patch was pushed back in October, 2022.  Okay to close report?

Rik <rik5>
Group administrator
Mon 31 Oct 2022 08:27:06 PM UTC, comment #20: 

following on comment #15, i pushed a change to replace the two locations with automatic broadcasting with bsxfun similar to that done with center.m in bug #51249. won't worry about any workarounds to preserve sparseness for now, but this prevents the nonconformance errors. added bists to capture sparse inputs.
 
https://hg.savannah.gnu.org/hgweb/octave/rev/84fa33608593

Nicholas Jankowski <nrjank>
Group Member
Mon 31 Oct 2022 05:19:59 PM UTC, comment #19: 

@Markus:
Yep, I know, but I'd rather leave negotations with "upstream" to the OF matgeom package maintainer (Juanpi).

Matlab is a moving target for not just Octave - from time to time several other non-official and Matlab Central/FileExchange packages fall victim to TMW's decisions, including the one underlying the OF matgeom package at https://github.com/mattools/matGeom.

Philip Nienhuis <philipnienhuis>
Group Member
Mon 31 Oct 2022 10:42:10 AM UTC, comment #18: 

@Philip: IIUC, the matgeom package was originally targeting Matlab. That probably means that shadowing that function there could also lead to unexpected results.
It might be worth asking upstream if they would like to avoid shadowing that function for both programs (Octave and Matlab).

Markus Mützel <mmuetzel>
Group administrator
Sun 30 Oct 2022 08:32:36 PM UTC, comment #17: 

There's also a function 'normalize' in the OF matgeom package:

>> which normalize
'normalize' is a function from the file C:\Programs\Octave\Octave-8.0.0_20220824H\mingw64\
share\octave\packages\matgeom-1.2.3\geom2d\normalize.m


Fixing this may be a bit more difficult than the the 'normalize.m' in the ltfat package; AFAIK the matgeom package is actually depending 1:1 on an external package primarily meant for Matlab.
I'm cc-ing the matgeom maintainer anyway.

Philip Nienhuis <philipnienhuis>
Group Member
Thu 27 Oct 2022 06:02:00 PM UTC, comment #16: 

AFAIK, bsxfun and broadcasting hasn't been fully implemented for sparse matrices yet so converting to full was a way to at least get a result.

For a couple of months now I've been meaning to post some ideas about refactoring broadcasting to make better use of templates.  I'll try to do that soon as I think that should be done before or simultaneously with adding support for sparse matrices.

John W. Eaton <jwe>
Group administrator
Thu 27 Oct 2022 05:42:59 PM UTC, comment #15: 

hadn't tested sparse matrices before, but notice it runs into the same problem as center.m, where in bug #51249 they added a temporary workaround with bsxfun to prevent errors. I'll plan to do the same here with a similar note. I did notice, though that although it doesn't error, bsxfun always outputs a full matrix. i didn't see any bug reports on that. std does the same.

Nicholas Jankowski <nrjank>
Group Member
Thu 27 Oct 2022 02:23:39 PM UTC, comment #14: 

thanks. docstring and manual linking pushed as
https://hg.savannah.gnu.org/hgweb/octave/rev/4c6c8f14766c

Nicholas Jankowski <nrjank>
Group Member
Thu 27 Oct 2022 06:54:02 AM UTC, comment #13: 

I also can't find where the documentation of the new function was added to the manual.

Markus Mützel <mmuetzel>
Group administrator
Thu 27 Oct 2022 06:41:19 AM UTC, comment #12: 

I haven't tested the function (yet). Just a remark:
The docstring is lacking a one-line description as its first line after the prototypes.

Markus Mützel <mmuetzel>
Group administrator
Wed 26 Oct 2022 10:06:11 PM UTC, comment #11: 

ignore the last comment. of course they fail if you split them like a char array and forget to enclose them in [].

one more push. everything passes. sorry for the noise.
https://hg.savannah.gnu.org/hgweb/octave/rev/fa3ae8fd8449

Nicholas Jankowski <nrjank>
Group Member
Wed 26 Oct 2022 09:38:30 PM UTC, comment #10: 

linked nan related xtests to bug #50571 and remembered to clean up the line lengths.

https://hg.savannah.gnu.org/hgweb/octave/rev/014648ce3267

also noticed that splitting an error message across two lines causes the error BISTs to fail. a quick search didn't find anything but will create a separate bug report if that hasn't been seen before.

Nicholas Jankowski <nrjank>
Group Member
Wed 26 Oct 2022 09:08:27 PM UTC, comment #9: 

(*for select definitions of near future)

have completed a cleaned up normalized.m against the current default branch.  includes some simplification now that sum includes an optional mean output (bug #62395), and from std/var now properly handling an Inf input (bug #63203). General cleanup of code, comments, and docstring as well.

pushing to default in statistics folder along with module.mk and a NEWS edits as https://hg.savannah.gnu.org/hgweb/octave/rev/ecfb41743fd2

marking as Ready for Test.

I see that there's a ltfat 2.5.0 in their repository (with the rename of their normalize function) that hasn't been pushed to either package repository yet (currently at 2.3.1). Even if that doesn't get released by the Octave v8 release, there's a manual download option that should help anyone running into shadowing issues with normalize.

Nicholas Jankowski <nrjank>
Group Member
Tue 26 Jul 2022 03:51:33 PM UTC, comment #8: 

note that ltfat has deprecated their normalize function, renaming it to setnorm.
https://github.com/ltfat/ltfat/commit/577aee742d49195301e33574e67327681793a09c

looking back at this function it could use some cleanup. will revisit and update in the near future.

Nicholas Jankowski <nrjank>
Group Member
Thu 17 Jun 2021 07:51:26 PM UTC, comment #7: 

updated v3 patch and m file with corrected unix sytle line endings.

(file #51576, file #51577)

Nicholas Jankowski <nrjank>
Group Member
Thu 17 Jun 2021 04:19:32 AM UTC, comment #6: 

noticed that recent version of matlab added the ability to get scale and center value outputs from function, as well as ability to call with both center and scale as inputs. updated function to add both of those features, along with a few BISTs for them, and updated patch to recent codebase.  add_normalize_v2.diff and updated normalize.m attached.  note this is still dependent on patched iqr.m from bug #59636.



(file #51570, file #51571)

Nicholas Jankowski <nrjank>
Group Member
Thu 22 Apr 2021 03:22:40 PM UTC, comment #5: 

noting a lot of interest in getting the Tabilicious package up to the point of providing compatibility with Matlab's Table class.  Since a completely compatible version of this function would need to work on Table objects, how should that be handled if Table functionality is dependent on an external package? Should the Table package shadow this (and any other similar) function? Is that unnecessary since you'd need the package loaded to have a Table object to work on in the first place?  Are there cases that wouldn't trigger a "that is the function of the XYZ package which isn't loadad" error, or would an if(package_loaded) need to be hard coded into this function?

Nicholas Jankowski <nrjank>
Group Member
Mon 14 Dec 2020 06:28:28 AM UTC, comment #4: 

attached is an implementation of a matlab compatible normalize.m and a patch to add it to /scripts/statistics in octave core.  It implements all of the the different options/methods except for DataVariable, as that depends on Tables objects that have not yet been implemented. the input handling is in place however once that can be implemented.

It handles most all of the oddities of the Matlab version including inf and nan handling, except for a few specific NaN containing inputs. Matlab has ways of skipping over NaN inputs and processing the rest of the vector as if it wasn't there (like NaNflag options in some other functions) that don't appear trivial to emulate.

BISTs are added for all of the options.  I left in two xtests for particular NaN incompatibilities mentioned above.

note that a few options depend on the modified matlab compatible iqr implemented in bug #59636. breaking that dependency would require adding in a lot more input/option processing the few places it's used.


Last, i realize this will create a situation where ltfat's normalize would shadow this one.  The maintainer noted on github that they noted the potential namechange in their release.

(file #50456, file #50457)

Nicholas Jankowski <nrjank>
Group Member
Fri 22 Feb 2019 07:27:27 PM UTC, comment #3: 

did so over at GitHub, linked back here
https://github.com/ltfat/ltfat/issues/101

Nicholas Jankowski <nrjank>
Group Member
Fri 22 Feb 2019 05:41:32 PM UTC, comment #2: 

nrjank - You may also want to file a bug with ltfat (either here or https://github.com/ltfat/ltfat/issues) to rename their 'normalize' function.

Mike Miller <mtmiller>
Group Member
Fri 22 Feb 2019 05:34:45 PM UTC, comment #1: 

I'm marking this as a feature request because even though there is a function of the same name somewhere, that isn't the same as having a function in Octave that attempts to do the same thing but is incompatible in some way(s).

John W. Eaton <jwe>
Group administrator
Fri 22 Feb 2019 05:07:51 PM UTC, original submission:  

Matlab 2018a introduced a function 'normalize' that "returns the vectorwise z-score of the data in A with center 0 and standard deviation 1."

see
https://www.mathworks.com/help/matlab/ref/normalize.html

Note that there is a function in the ltfat package that has the same name but very different functionality:

https://octave.sourceforge.io/ltfat/function/normalize.html

Nicholas Jankowski <nrjank>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #51576:  add_normalize_v3.diff added by nrjank (27KiB - application/octet-stream - line endings fix)
file #51577:  normalize.m added by nrjank (25KiB - text/plain - line endings fix)
file #51570:  add_normalize_v2.diff added by nrjank (28KiB - application/octet-stream - v2 - adds [z, c, s] output option and simultaneous center and scale input options )
file #51571:  normalize.m added by nrjank (25KiB - text/plain - v2 - adds [z, c, s] output option and simultaneous center and scale input options )
file #50456:  normalize.m added by nrjank (20KiB - text/plain - patch to add normalize.m to core octave)
file #50457:  add_normalize.diff added by nrjank (23KiB - application/octet-stream - patch to add normalize.m to core octave)

 

Carbon-Copy List
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by philipnienhuis
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by nrjank (Submitted the item)
  • -email is unavailable- added by nrjank (ltfat pkg manager)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 22 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2023-04-12 nrjank StatusReady For Test Fixed
        Open/ClosedOpen Closed
        Fixed ReleaseNone 8.1.0
    2023-04-12 rik5 Planned ReleaseNone 8.1.0
    2022-10-31 nrjank SummaryMissing function: normalize (same function name used in ltfat pkg) Missing function: normalize (same function name used in ltfat and matgeom pkgs)
    2022-10-30 philipnienhuis Carbon-Copy- Added juanpi
    2022-10-26 nrjank Dependencies- Depends on bugs #63203
        Dependencies- Depends on bugs #62395
    2022-10-26 nrjank StatusPatch Submitted Ready For Test
    2021-10-06 nrjank StatusNone Patch Submitted
        Dependencies- Depends on bugs #59636
    2021-06-17 nrjank Attached File- Added add_normalize_v3.diff, #51576
        Attached File- Added normalize.m, #51577
    2021-06-17 nrjank Attached File- Added add_normalize_v2.diff, #51570
        Attached File- Added normalize.m, #51571
    2020-12-14 nrjank Attached File- Added normalize.m, #50456
        Attached File- Added add_normalize.diff, #50457
    2019-02-22 jwe Release5.0.91 dev
    2019-02-22 jwe Severity3 - Normal 1 - Wish
        Priority5 - Normal 3 - Low
        Item GroupMatlab Compatibility Feature Request
    2019-02-22 nrjank Carbon-Copy- Added -email is unavailable-

    Back to the top

    Powered by Savane 3.13-3230.
    Corresponding source code