bugGNU Octave - Bugs: bug #64563, datevec: "mmm dd yyyy"...

 
 

bug #64563: datevec: "mmm dd yyyy" matched instead of "mmmyyyy" standard format

Submitter:  Nicholas Jankowski <nrjank>
Submitted:  Thu 17 Aug 2023 04:53:15 PM UTC
   
 
Category:  Octave Function Severity:  2 - Minor
Priority:  5 - Normal Item Group:  Documentation
Status:  Ready For Test Assigned to:  None
Originator Name:  Open/Closed:  * Open
Release:  * dev Operating System:  * Any
Fixed Release:  None Planned Release:  8.4.0
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sat 13 Jan 2024 01:51:04 AM UTC, comment #14: 

Possibly. So far we have a doc patch warning pushed and marked as ready and then some inconclusive beck and forth over whether there should be an actual code change or not.  if it's just going to stop at a doc fix then yes it can be closed.

Nicholas Jankowski <nrjank>
Group Member
Fri 12 Jan 2024 09:55:19 PM UTC, comment #13: 

Bug is marked as Ready for Test since September 2023.  Can it be closed as Fixed?

Rik <rik5>
Group administrator
Fri 15 Sep 2023 08:35:05 PM UTC, comment #12: 


> I'm not sure if it is possible in principle that datevec can correctly match all formats that have a numeric identifier in datestr.

Yeah I'm also unsure (just started investigating, then real life stepped in) but we can at least try to match as much as possible (or reasonable).

>It took me a while to notice that "AUG2013" can be matched by the pattern 'mmm dd yyyy'.

That's what this bug report is about. I suppose datevec.m should catch & if needed preprocess input before handing over to strptime(). Easiest is to provide a new format matching one in datestr.m.

strptime() is a beast and on top of that its behavior is partly implementation-defined (I hit that in the past that when trying to fix daylight saving time issues). So yes, surprises are due ...

I'm not so afraid of surprises if they pertain to undocumented behavior.
I'm more concerned (but not that much) with consistency of functions that provide, or implicitly or explicitly suggest to provide, each other's inverse, like datestr.m and datevec.m.

Philip Nienhuis <philipnienhuis>
Group Member
Wed 13 Sep 2023 02:13:13 PM UTC, comment #11: 

I'm not sure if it is possible in principle that datevec can correctly match all formats that have a numeric identifier in datestr. (I haven't checked that though.)

It took me a while to notice that "AUG2013" can be matched by the pattern 'mmm dd yyyy'. You could say that's a surprise. But it's documented behavior of strptime which is the backbone of datevec.

It's very difficult to predict which new surprises are possible if new patterns are added.

I'm not against adding new patterns in principle. Just be aware that you need to be very, very careful when doing that.

I agree that the issue about the day being set to 0 deserves its own report.

Markus Mützel <mmuetzel>
Group administrator
Wed 13 Sep 2023 01:47:44 PM UTC, comment #10: 

Both Matlab and Octave accept numeric 28 for datestr.m's 2nd argument FormatOut, i.e. 'mmmyyyy', and neither Matlab's nor Octave's datevec.m have a matching builtin format.
One can interpret this as "glitch-for-glitch" Matlab-compatibility of datevec.m.
If datevec.m cannot properly parse a builtin output format produced by datestr.m, IMHO that wouldn't fit with the principle of "least surprise". An unwary user would expect to be each other's complement to some extent, I suppose.

> At the moment, it's unclear to me if that could break existing code.

Nor to me. But as AFAIK the output of datevec ('mmmyyyy') hasn't been documented, I doubt if relying on specific behavior of datevec.m for such input would be good practice in the first place. So I think we can safely patch datevec.m to accept 'mmmyyyy' input.
My patch would still need a bit more polishing (a.o., BIST).


I'm looking into other non-matches between datestr.m and datevec.m as well.
E.g., Matlab's datevec.m won't accept '2023' (output of "datestr ([2023, 1, 1], 10)") while Octave's returns [2023, 1, 0, 0, 0, 0]. I'd say Octave does "better" here, apart from a day value of zero; Matlab's "datevec ('2023', 'yyyy')" returns [2023, 1, 1, 0, 0, 0]

I think the day value equal to 0 for some formats should be fixed as well but would need another bug report.

Philip Nienhuis <philipnienhuis>
Group Member
Sat 26 Aug 2023 11:38:10 AM UTC, comment #9: 

I went ahead and pushed an update to the docstring here:
https://hg.savannah.gnu.org/hgweb/octave/rev/2e8b8d38431c

I'm still hesitant to add new format strings to somewhere in the middle of the existing list. At the moment, it's unclear to me if that could break existing code.

I'm marking this documentation change as ready for test because imho this is all we need for the original issue.
Feel free to set different flags if you disagree.

Markus Mützel <mmuetzel>
Group administrator
Wed 23 Aug 2023 04:40:01 PM UTC, comment #8: 


comment #7:

> > Not sure which inconsistency you mean.
> See 2nd verbatim block in my comment #5


Could you please elaborate? I still don't understand what is inconsistent there...

Markus Mützel <mmuetzel>
Group administrator
Wed 23 Aug 2023 12:04:50 PM UTC, comment #7: 


> Not sure which inconsistency you mean.

See 2nd verbatim block in my comment #5

Philip Nienhuis <philipnienhuis>
Group Member
Wed 23 Aug 2023 06:20:35 AM UTC, comment #6: 

Yeah, I quoted that part on purpose. I was hoping a native speaker could come up with something more technical. Maybe, something along the lines of:

> If no format string is provided, heuristics are used to guess the input format.


And if that is not clear enough, we could follow with something like:

> These heuristics might not give the expected result.


Not sure which inconsistency you mean.
In Matlab R2023a:

>> dstr = datestr (datenum ([2013, 8, 1, 0, 0, 0]), 28)
dstr =
    'Aug2013'
>> datevec(dstr, 'mmmyyyy')
ans =
        2013           8           1           0           0           0


For the roundtrip to work, you'd need to use the same (or equivalent) format specifiers in both directions.
Or if you don't want to use format specifiers, you must not use them in both directions:

>> dstr = datestr (datenum ([2013, 8, 1, 0, 0, 0]))
dstr =
    '01-Aug-2013'
>> datevec(dstr)
ans =
        2013           8           1           0           0           0


Format "29" was a typo. I meant to write "28". Which Matlab incompatibility are you referring to?


Markus Mützel <mmuetzel>
Group administrator
Tue 22 Aug 2023 09:52:50 PM UTC, comment #5: 

The underlying message in:

> "all bets are off" for ambiguous date strings if no format string is explicitly supplied

is right but the wording could be a little more rounded.
Yet I regularly run into data files where dates/times in a column have mixed formats. I never tried if that'll work with a given format string but, well, I guess in such cases indeed "all bets are off" :-)

Good catch, that "2013th day of August". But in Matlab r2020b:

>> datestr (datenum (2013, 8, 1, 0, 0, 0), 28)
ans =
    'Aug2013'
>>

so even Matlab has inconsistencies between its datestr.m and datevec.m.

In Octave's datestr.m we have these entries:

% Texinfo header
## @item 28 @tab mmmyyyy                @tab Sep2000
:
:
% further down in code
    dateform{29} = "mmmyyyy";

so there we have another inconsistency and Matlab incompatibility.
Given Matlab's output my patch could be amended with format no. 28 I guess.

Maybe a check for matching format numbers in datestr.m and datevec.m in both Octave and Matlab, resulting in a separate patch, would be a Good Idea :-)

Philip Nienhuis <philipnienhuis>
Group Member
Tue 22 Aug 2023 10:28:47 AM UTC, comment #4: 

I guess Matlab interprets that input as the "2013th day of August" in the current year:

>> datevec(datenum([2023 8 2013 0 0 0]))

ans =

        2029           2           2           0           0           0


Octave's behavior seems more reasonable to me. So, I don't see a reason to follow Matlab's behavior in that case.

Stepping through the code in Octave, the input is matched to the format string `mmm dd yyyy`. IIUC, that is because `[tm, nc] = strptime('AUG2013', '%b %d %Y')` matches the entire input string (i.e., nc == 8).

That seems to be the behavior defined by POSIX for that function:
https://man7.org/linux/man-pages/man3/strptime.3.html
In particular:

> All other characters in the format string must have a matching character in the input string, except for whitespace, which matches zero or more whitespace characters in the input string.


It is not clear what the "correct behavior" of `datevec` would be in this particular case. Are we sure that adding new strings to the format list doesn't have unexpected side effect? (And if we do want to add it, it should come with a comment that it's format type 29.)
The docstring already encourages to supply a date format for performance reasons. Maybe, we should just add a note to the docstring that "all bets are off" for ambiguous date strings if no format string is explicitly supplied.

Markus Mützel <mmuetzel>
Group administrator
Tue 22 Aug 2023 08:50:40 AM UTC, comment #3: 


> there are a number of dates where Octave's datevec uses 0 as the starting day

OK, but what does Matlab output in those cases? Is that valid datevec output? I can imagine that functions devouring datevec.m's output need some consistent output and where needed could (should) neglect starting day values. If user code would rely on a zero starting day I'd think that code should fix it for itself as well, not rely on datevec idiosyncrasies. But if Matlab leads the way here, well ... :-)
In this case, where Matlab returns [2013, 8, 1, 0, 0, 0] for "datevec ('AUG2013', 'mmmyyyy')" (where no day is included in the format) I think my patch would do for a start - apart from the BIST.

Philip Nienhuis <philipnienhuis>
Group Member
Mon 21 Aug 2023 07:03:34 PM UTC, comment #2: 

there are a number of dates where Octave's datevec uses 0 as the starting day instead of 1.  don't have them linked but there were other bug conversations about that point so i didn't call it out here.

Nicholas Jankowski <nrjank>
Group Member
Mon 21 Aug 2023 06:28:44 PM UTC, comment #1: 


> since mmmyyyy is one of the specified standard datestr formats

AFAICS that standard format isn't defined in Octave's datevec.m. However 'mmmyy' is.
Anyway attached patch would fix that. It adds 'mmmyyyy' under the header "These are other formats that Octave tries' because, as you showed, Matlab doesn't cope with it.

I haven't looked at the related BISTs in detail (sorry no time), I think I saw that the BIST that tries this format erroneously sets day no. to (0) zero rather than 1.


(file #55066)

Philip Nienhuis <philipnienhuis>
Group Member
Thu 17 Aug 2023 04:53:15 PM UTC, original submission:  

Octave output appears to partially match an format undefined by datestr or mentioned within datevec, but it isn't expanding two digit year to yyyy in the vector.

Octave 9.0.0 (hg id: caef3e5f755e)

>> datevec("AUG2013", 'mmmyyyy')
ans =

   2013      8      1      0      0      0

 datevec("AUG2013", 'mmmddyy')
ans =

   2013      8     20      0      0      0

>> datevec("AUG2013")
ans =

   13    8   20    0    0    0


matlab 2023a behavior isn't much help here.  probably should file a mathworks bug:


>> datevec("AUG2013", "mmmyyyy")
ans =
        2013           8           1           0           0           0
>> datevec("AUG2013", "mmmddyy")
ans =
        2013           8          20           0           0           0
>> datevec("AUG2013")
ans =
        2029           2           2           0           0           0


since mmmyyyy is one of the specified standard datestr formats, I would think the matching routine should find that one first.

Nicholas Jankowski <nrjank>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #55066:  bug64563_datevec.cset added by philipnienhuis (780B - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by nrjank (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2023-08-26 mmuetzel Item GroupIncorrect Result Documentation
        StatusPatch Submitted Ready For Test
        Planned ReleaseNone 8.4.0
        Summarydatevec: undefined mmmddyy partially matched instead of mmmyyyy standard format datevec: "mmm dd yyyy" matched instead of "mmmyyyy" standard format
    2023-08-21 philipnienhuis Attached File- Added bug64563_datevec.cset, #55066
        StatusNone Patch Submitted

    Back to the top

    Powered by Savane 3.13-54b4.
    Corresponding source code