bugGNU Octave - Bugs: bug #59979, Unexpected format for output from...

 
 

bug #59979: Unexpected format for output from datevec

Submitted by:  Alan <dekalog>
Submitted on:  Sat 30 Jan 2021 06:36:50 PM UTC  
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Inaccurate Result
Status:  Confirmed Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  6.1.0 Operating System:  Any

Add a New Comment (Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

( Jump to the original submission)

Wed 24 Feb 2021 05:53:44 PM UTC, comment #23: 

And, as predicted, unless these routines reproduce Matlab results we are going to have a stream of bug reports.  See bug #60107 where datenum() is off by 1 day with respect to Matlab.

Rik <rik5>
Project Administrator
Wed 24 Feb 2021 05:51:41 PM UTC, comment #22: 

These functions are a core part of Matlab, and will be for years, so I don't think we can abandon them in favor of duration objects.

Rik <rik5>
Project Administrator
Mon 01 Feb 2021 06:24:25 PM UTC, comment #21: 

Matlab moved away from these function and uses datetime and duration objects now.
I believe Andrew Janke has worked on that.

Maybe we should just abandon these functions and try to get the more modern types that better describe a date or a duration.

But if someone wants to try to get this function to better mimick what Matlab returns, I'll be happy to run tests or review.

Octave seems to implement the carry-over rules correctly from datevectors to datestrings (although it needs an additional step through `datenum`):

>> datestr(datenum([2009 22 03 00 00 00]))
ans = 03-Oct-2010
>> datestr(datenum([2010 07 -05 00 00 00]))
ans = 25-Jun-2010

IIUC, this report is not about carry-over behavior but about rounding behavior.

Fwiw, in Matlab R2020b:

>> datevec(datenum([2021 01 02 00 00 59.9999]))

ans =

   1.0e+03 *

    2.0210    0.0010    0.0020         0         0    0.0600

Markus Mützel <mmuetzel>
Project Member
Mon 01 Feb 2021 06:10:43 PM UTC, comment #20: 

How matlab deals with 'carryover' is described here:
https://uk.mathworks.com/help/matlab/matlab_prog/carryover-in-date-vectors-and-strings.html

How matlab deals with time "zero" as a special case is described here (see first point in Limitations) https://uk.mathworks.com/help/matlab/ref/datevec.html#btfyhto-4

Note that there is nothing special about the 'zero' date. It is simply the 31st Dec 2BC. Effectively it seems that matlab has chosen to represent the 31st Dec of year -1 (i.e. 2BC, since the year '0' corresponds to 1BC.) as 0,0,0 instead of -1,12,31 for purely cosmetic reasons (i.e. to provide a 'zero' date). It does this by treating it as a special exception to the datevec output.

I honestly don't care that octave chooses to represent this 'properly' as opposed to 'cosmetically'. In fact I think it makes more sense to represent the 'real' date rather than a 'zero'.

However, when it comes to the carryover issue, that is a separate issue, one that I think matlab does correctly.

Tasos Papastylianou <tpapastylianou>
Mon 01 Feb 2021 06:08:24 PM UTC, comment #19: 

If I pass a duration to datevec, I would probably expect it to return the number of years, months, hours, minutes, and seconds of the elapsed time represented by the duration.  In that case, I also wouldn't expect elapsed time to be negative.  But if datevec is supposed to return an actual date, then it doesn't make sense to have the month or day set to zero.  OTOH, it isn't always returning correct calendar dates anyway because calendar dates have done some strange things.

Maybe this is just a poorly defined function?

I'm not sure what the best fix is, but the current behavior seems bad because sometimes it just looks wrong and it is also not consistent with the results Matlab produces, so without some changes we are sure to see more bug reports.

John W. Eaton <jwe>
Project Administrator
Mon 01 Feb 2021 05:12:26 PM UTC, comment #18: 
Markus Mützel <mmuetzel>
Project Member
Mon 01 Feb 2021 04:49:42 PM UTC, comment #17: 

Regarding the date with day 0 and month 0:
I see the following in Matlab R2020b:

>> datevec(-2:0.5:2)

ans =

    -1    12    29     0     0     0
    -1    12    29    12     0     0
    -1    12    30     0     0     0
    -1    12    30    12     0     0
     0     0     0     0     0     0
     0     0     0    12     0     0
     0     1     1     0     0     0
     0     1     1    12     0     0
     0     1     2     0     0     0

>> datevec(365:0.5:368)

ans =

     0    12    30     0     0     0
     0    12    30    12     0     0
     0    12    31     0     0     0
     0    12    31    12     0     0
     1     1     1     0     0     0
     1     1     1    12     0     0
     1     1     2     0     0     0

There seems to be no 31st of December in the year 1 BCE. But there is a 31st of December in the year 0 CE.
I find that rather surprising because I assumed `datevec` returns the date and time.
But it looks like others expect it to return something different.

From Matlab's documentation:

> DateVector = datevec(t) converts the datetime or duration value t to a date vector—that is, a numeric vector whose six elements represent the year, month, day, hour, minute, and second components of t.

Markus Mützel <mmuetzel>
Project Member
Mon 01 Feb 2021 04:37:44 PM UTC, comment #16: 

Neglecting for a moment any strange display issues that might be due to rounding (like showing 19 minutes and 60 seconds instead of 20 minutes and 0 seconds) I find the following results quite surprising:

octave> datevec (0)
ans =

   -1   12   31    0    0    0

Shouldn't that be 0 years, 0 months, 0 days, 0 hours, 0 minutes, and 0 seconds?  For comparision, it looks like datevec (1) is correct:

octave> datevec (1)
ans =

   0   1   1   0   0   0

Zero years, the first month and day, zero hours, minutes, and seconds.  Similarly, I see what appears to be the correct result for datevec (1.1):

octave> datevec (1.1)
ans =

    0    1    1    2   24    0

0.1 days is 144 minutes, or 2 hours and 24 minutes.

So maybe we just need to do the right thing when the datenum is between 0 and 1?

Looking at datevec.m, I see:

## Algorithm: Peter Baum (http://vsg.cape.com/~pbaum/date/date0.htm)

That link is no longer valid.  Maybe we could update it?  And also

  else   # datenum input

    if (! iscolumn (date))
      date_sz = size (date);
      do_resize = true;
    endif
    date = date(:);

    ## Move day 0 from midnight -0001-12-31 to midnight 0000-3-1
    z = double (floor (date) - 60);
    ## Calculate number of centuries; K1 = 0.25 is to avoid rounding problems.
    a = floor ((z - 0.25) / 36524.25);
    ## Days within century; K2 = 0.25 is to avoid rounding problems.
    b = z - 0.25 + a - floor (a / 4);
    ## Calculate the year (year starts on March 1).
    y = floor (b / 365.25);
    ## Calculate day in year.
    c = fix (b - floor   else   # datenum input

    if (! iscolumn (date))
      date_sz = size (date);
      do_resize = true;
    endif
    date = date(:);

    ## Move day 0 from midnight -0001-12-31 to midnight 0000-3-1
    z = double (floor (date) - 60);
    ## Calculate number of centuries; K1 = 0.25 is to avoid rounding problems.
    a = floor ((z - 0.25) / 36524.25);
    ## Days within century; K2 = 0.25 is to avoid rounding problems.
    b = z - 0.25 + a - floor (a / 4);
    ## Calculate the year (year starts on March 1).
    y = floor (b / 365.25);
    ## Calculate day in year.
    c = fix (b - floor (365.25 * y)) + 1;
    ## Calculate month in year.
    m = fix ((5 * c + 456) / 153);
    d = c - fix ((153 * m - 457) / 5);
    ## Move to Jan 1 as start of year.
    ++y(m > 12);
    m(m > 12) -= 12;

    ## Convert hour-minute-seconds.  Attempt to account for precision of
    ## datenum format.

    fracd = date - floor (date);
    tmps = abs (eps*86400*date);
    tmps(tmps == 0) = 1;
    srnd = 2 .^ floor (- log2 (tmps));
    s = round (86400 * fracd .* srnd) ./ srnd;
    h = floor (s / 3600);
    s -= 3600 * h;
    mi = floor (s / 60);
    s -= 60 * mi;

  endif
(365.25 * y)) + 1;
    ## Calculate month in year.
    m = fix ((5 * c + 456) / 153);
    d = c - fix ((153 * m - 457) / 5);
    ## Move to Jan 1 as start of year.
    ++y(m > 12);
    m(m > 12) -= 12;

    ## Convert hour-minute-seconds.  Attempt to account for precision of
    ## datenum format.

    fracd = date - floor (date);
    tmps = abs (eps*86400*date);
    tmps(tmps == 0) = 1;
    srnd = 2 .^ floor (- log2 (tmps));
    s = round (86400 * fracd .* srnd) ./ srnd;
    h = floor (s / 3600);
    s -= 3600 * h;
    mi = floor (s / 60);
    s -= 60 * mi;

  endif

There seems to be some ad hoc fudging here.  What is the real justification for "K1 = 0.25 is to avoid rounding problems" and "K2 = 0.25 is to avoid rounding problems"?  Are these arbitrary values really helpful in all cases?  Maybe there is a better way?  This code appears to be mostly unchanged since it was added to Octave.  (Note also that it was added long ago, before we used Mercurial, when I was the only one committing changes, so looking at the Mercurial logs won't give you proper author info.)

In any case, I don't think the algorithm is correct for small datenum values and that the problems are not just display issues.

John W. Eaton <jwe>
Project Administrator
Mon 01 Feb 2021 01:07:25 PM UTC, comment #15: 

@Markus comment #12:
Maybe Matlab deliberately rounds up as soon as the smallest fraction in datevec's output for seconds (milliseconds, 'FFF') is prone to be rounded (i.e., seconds >= 59.9995 milliseconds)?
(I suppose that's the actual question asked in comment #6.)

If so, Matlab's behavior wouldn't be unreasonable IMO. That is, I think datenums are simply the most accurate and suitable for date/time calculations, while IMO datevec and datestr are just interpretations/translations of some time value for display purposes and shouldn't be used for (much) further processing.
Even if the display is wrong the totals are still correct, so it's primarily a cosmetics bug.

Philip Nienhuis <philipnienhuis>
Project Member
Sun 31 Jan 2021 06:58:55 PM UTC, comment #14: 

Copy-paste error in previous comment. Instead of

octave:2> for DELTA = [0 59 60 61], tmp = datevec(((0:1200:1e6) + DELTA) / 86400 + floor(now)); unique(tmp(:,end)), end

please use

octave:2> for DELTA = [0 1 59 60 61], tmp = datevec(((0:1200:1e6) + DELTA) / 86400 + floor(now)); unique(tmp(:,end)), end

Anonymous
Sun 31 Jan 2021 06:57:02 PM UTC, comment #13: 

This code gives correct results in the seconds position. Rounding works correctly.

for DELTA = [0 1 59 60 61]
    datevec(((0:1200:1e5) + DELTA) ./ 86400 + floor(now))
end

This test also works accurately where [0 1 59 60 61] are mapped to [0 1 59 0 1] respectively:

octave:2> for DELTA = [0 59 60 61], tmp = datevec(((0:1200:1e6) + DELTA) / 86400 + floor(now)); unique(tmp(:,end)), end
ans = 0
ans = 1
ans = 59
ans = 0
ans = 1

Is there any need for extra rounding within datevec?

Anonymous
Sun 31 Jan 2021 03:23:12 PM UTC, comment #12: 

I'd still argue that what Matlab seems to do is somewhat strange (hence the item group I chose):
AFAICT, the type of datenum is double in Matlab. But still it seems to do the conversion to a datevec vector with less than double precision.

Does it convert to single precision before conversion? Or does it do something entirely different?

A digital clock doesn't round to the closest minute. It switches to the next minute when that minute has started. (That's why you don't see something like "19:60".) If you want to get output similar to what a clock would show, use `floor`:

>> a = datenum ([2021, 1, 27, 17, 0, 0]);
>> b = datenum ([2021, 1, 27, 16, 40, 0]);
>> floor (datevec (a-b))
ans =

   -1   12   31    0   19   59

Also, the output format is completely independent of floating point accuracy (which is the reason for the round-off errors here).
The result of `datevec` is "just a normal vector". In Octave, neighboring elements in a vector don't interact when displayed.
If a user prefers to round the result of `datevec` differently, they should do that in their own code, imho.

Markus Mützel <mmuetzel>
Project Member
Sun 31 Jan 2021 02:57:47 PM UTC, comment #11: 

PMFJI
I quite agree with comment #2 and comment #5 as regards "formatted display" and representation.
Yet I wouldn't rely too much on Matlab as a reference as that occasionally gets it wrong as well; e.g., I recently hit this with patch #9953 (see midway comment #10) where Matlab screws up with coordinates display:

> ... for -181.7: while <Matlab> gets the total right as it returns <degrees> 41' 60", it should of course round to 42' 00" ...

...and I've regularly hit similar issues as the OP in both Matlab and Octave in my tidal analysis project. (generally Matlab seems to do better, though.)

My conclusion was (and is) that representation and "formatted display" is unavoidably prone to rounding error here and there, and working around that requires special care on a case-by-case basis.
I suppose this supports Tasos' stance in comment #9.

@comment #6:
I'm only vaguely familiar with IEEE number representations, but AFAIK floating point machine precision depends on number magnitude. 'dec2bin (datenum (2021, 1, 27))' gives '10110100001110000111', parts of which probably get morphed into the exponent part, but either way you loose precision with larger numbers.
So your question depends on what date range you have in mind.

Getting better accuracy when working with dates starts with subtracting the start date from your period of interest (comment #1: datenum (2021,1,27) = 738183), work with those numbers, and at the end add it back, as then you'll have a much better resolution to start with.
Just to show this, compare Markus' example output in comment #7 with this:

>> format long
>> a = datenum ([0, 0, 0, 17, 0, 0]);
>> b = datenum ([0, 0, 0, 16, 40, 0]);
>> datevec (a - b)
ans =
Columns 1 through 3:
  -1.000000000000000e+00   1.200000000000000e+01   3.100000000000000e+01
Columns 4 through 6:
                       0   1.900000000000000e+01   5.999999999999591e+01

>> format short
>> datevec (a - b)
ans =
   -1.0000   12.0000   31.0000         0   19.0000   60.0000

to see that

  • accuracy gets quite a bit better for dates close to Octave's and Matlab's epoch, but
  • Octave's rounding in datevec could be better. Matlab gives:

    >> a = datenum ([0, 0, 0, 17, 0, 0]);
    >> b = datenum ([0, 0, 0, 16, 40, 0]);
    >> format long
    >> datevec (a-b)
    ans =
         0     0     0     0    20     0
    >> format short
    >> datevec (a-b)
    ans =
         0     0     0     0    20     0

IMO "Item group" should be "Inaccurate Result" rather than "WTF, Matlab?!?", and "Status" should be "confirmed".
Setting field accordingly.

Philip Nienhuis <philipnienhuis>
Project Member
Sun 31 Jan 2021 01:19:20 PM UTC, comment #10: 

I should point out the background context which triggered this bug report, which the original author forgot to mention: https://stackoverflow.com/q/65938193/4183191

Tasos Papastylianou <tpapastylianou>
Sun 31 Jan 2021 01:17:14 PM UTC, comment #9: 

Hi Markus!

Just to clarify, the point of that example is not the year-month-day difference (I agree that the octave representation makes more sense). Whether that part should be treated as a "wtf matlab" bug or not is an entirely unrelated issue to this bug report (and personally, I would not fix that part)

The main issue here is the rounding of [19, 59.999] to [19, 60.0] as opposed to [20, 0] in the resulting datevec vector, when operating at a particular output precision!

Matlab does this "correctly" (imo), under the assumption that the datevec is to be interpreted as a structure conveying time semantics, despite the fact that functionally it is a normal array. Therefore [19, 60] is meaningless: in the same way there is no "zero" date on a calendar, there is no "19:60" time on a clock.

I appreciate how the 60 may have come about as a rounding of 59.99. The argument is, since a datevec generated vector represents time semantics, such a rounding should be treated in time semantics as well, and therefore round [19, 59.99] to [20, 0] instead (as matlab does it).

However I also appreciate the argument that this is simply a representational issue w.r.t. a particular output_precision, and the underlying precision does not change. And also that passing this to datestr will produce the expected result.

Therefore the solution to this bug (at least from a 'differs from matlab' point of view) is either:
A) to fix it, i.e. detect the output precision before rounding, and round appropriately using time semantics before returning the resulting datevec, in a way compatible with matlab,
OR
B) leave it as it is, (i.e. incompatible with matlab) but perhaps add a simple 'caveat emptor' note in the documentation that the datevec output is not rounded using time semantics, and should therefore be treated with caution, i.e. it should not be relied on for inspecting individual elements of a date for equality in this fashion (and that the right way to do this is via string representations or floating-point calculations involving datenum, and the usual caveats involving comparisons in floating-point arithmetic).

My personal opinion is that B is probably better than A, since one presumably does not want to 'lose' actual precision in the seconds for the sake of output precision, and I don't necessarily think that [2000, 1, 1, 0, 20, -0] is particularly better than [2000, 1, 1, 0, 19, 60] in denoting 'a small number of milliseconds short of the full minute'.

Tasos Papastylianou <tpapastylianou>
Sun 31 Jan 2021 11:23:51 AM UTC, comment #8: 

I should have written: "There is no 0th day of the 0th month on a Gregorian calendar."

Markus Mützel <mmuetzel>
Project Member
Sun 31 Jan 2021 11:20:00 AM UTC, comment #7: 

The code in comment #5 leads to a time just short of 20 minutes after midnight in Octave:

>> format long
>> a = datenum ([2021, 1, 27, 17, 0, 0]);
>> b = datenum ([2021, 1, 27, 16, 40, 0]);
>> datevec (a-b)
ans =

  -1.000000000000000e+00   1.200000000000000e+01   3.100000000000000e+01                       0   1.900000000000000e+01   5.999999888241291e+01

The component with the seconds rounds to 60 when displayed with less digits. I'm not sure if it makes sense to "arbitrarily" round is this case...

The other question regarding the date part in Matlab's result is: There is no 0th of January on a Gregorian calendar. How does Matlab's datenum represent the 31st of December 1 BCE?

Markus Mützel <mmuetzel>
Project Member
Sat 30 Jan 2021 11:14:22 PM UTC, comment #6: 

Maybe MATLAB rounds to milliseconds ore something like that, to avoid this issue with floating-point rounding errors. What is the e smallest difference you can represent in MATLAB using datevec?

Anonymous
Sat 30 Jan 2021 08:38:02 PM UTC, comment #5: 

I have just checked with matlab.

This code:

a = datenum ([2021, 1, 27, 17, 0, 0]);
b = datenum ([2021, 1, 27, 16, 40, 0]);
datevec (a-b)

Results in

ans =     0     0     0     0    20     0

in matlab, and

ans =   -1.0000   12.0000   31.0000         0   19.0000   60.0000

in octave.

Tasos Papastylianou <tpapastylianou>
Sat 30 Jan 2021 08:33:35 PM UTC, comment #4: 

I have tried the 1200/86400 dateum difference suggestion and, whilst it lessens the problem, it still renders 20 minutes past the hour as 19 minutes and 60 seconds.

Alan <dekalog>
Sat 30 Jan 2021 08:22:59 PM UTC, comment #3: 

E.g. the following, very reasonable produces:

datevec( datenum( [2000, 1, 1, 0, 0, 60] ) )
ans =
   2000      1      1      0      1      0

as opposed to

ans =
   2000      1      1      0      0      60

Tasos Papastylianou <tpapastylianou>
Sat 30 Jan 2021 08:20:43 PM UTC, comment #2: 

I disagree. This is not a question of accuracy. It's a matter of reasonable representation.

Regardless of the underlying mechanics, a datevec representation should never be rounded to 60s; it should be rounded to the next minute.

Tasos Papastylianou <tpapastylianou>
Sat 30 Jan 2021 08:06:45 PM UTC, comment #1: 

This sounds like no more than a floating point rounding issue. The value of your 20 minute difference is 0.01388888887595385.

octave:2> datenum_20_min_diff=datenum([2021,1,27,17,0,0])-datenum([2021,1,27,16,40,0])
datenum_20_min_diff = 0.01388888887595385

Which upon multiplying by 86400 seconds in a day falls short of 20 minutes by 1.117 microseconds:

octave:3> datenum_20_min_diff * 86400
ans = 1199.999998882413
octave:4> datenum_20_min_diff * 86400 - 1200
ans = 1.117587089538574e-06

You may want to try setting your 20 minute difference to 1200/86400 since you know it's exactly 20 minutes, or even better, work in units that don't need small fractions until you're ready to print.

In either case, this doesn't look like a bug.

Anonymous
Sat 30 Jan 2021 06:36:50 PM UTC, original submission:  

Create a datevec series of dates and times, spaced by 20 minutes, with the minutes column values being exactly 0, 20 and 40 with a given starting and ending time in datevec format.

First create a typical 20 minute datenum difference value

datenum_20_min_diff=datenum([2021,1,27,17,0,0])-datenum([2021,1,27,16,40,0]);

Next, create a sequence of datenums, spaced by datenum_20_min_diff, from one given datevec to another datevec and display output as a datevec, e.g.

datevec((datenum([2021,1,27,22,0,0]):datenum_20_min_diff:datenum([2021,1,28,11,0,0]))')

gives unexpectedly formatted output in stout. After a few lines the minutes and seconds columns of the datevec output display e.g. 59 and 60 respectively, instead of the expected 0 and 0.

Alan <dekalog>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by tpapastylianou (Posted a comment)
  • -email is unavailable- added by dekalog (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

     

     

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2021-01-31 philipnienhuis Item GroupWTF, Matlab?!? => Inaccurate Result
        StatusNone => Confirmed
    2021-01-31 mmuetzel CategoryNone => Octave Function
        Item GroupNone => WTF, Matlab?!?
        Operating SystemGNU/Linux => Any

    Back to the top


    Powered by Savane 3.5