Too bad that Octave's behavior suffered from a regression compared to Mike's fix in comment #34.
Closing this bug because (interpreted in my words) "it is documented that Octave behaves buggy here" feels a bit awkward.
I wonder what caused this regression. Currently I have no time to sort it out; but within a few months I hope to have time to return to my tidal hobby project where I hit this (and hard - it was a big PITA for me and a good reason to resort to the proprietary competition that does it correctly).

So I've just set status to "postponed". Hopefully not for too long.

That said, IMO datevec.m is an opaque and entangled piece of code. I've hit some strange issues with my patch #9964 and I have trouble deducing what datevec's code actually does vs. what it is supposed to be doing.

I can't confirm it's the same bug but it sure looks like it.
With a recent Octave-7.0.0 Windows build:

>> DD_0=datenum('2019/03/31 00:00','yyyy/mm/dd HH:MM');
>> DD_0
DD_0 = 737515
>> for ii=1:4;
  my_datestr=sprintf('2019/03/31 0%d:00',ii);
  DD_r=datenum(my_datestr,'yyyy/mm/dd HH:MM');(DD_r-DD_0)*24
end
ans = 1.0000
ans = 3
ans = 3
ans = 4.0000
>>

I lack time to verify further before I'll be away for some weeks, sorry. It's also quite a while ago that I was involved in this bug so I don't remember all details.

I think I've found the same bug again. I run Octave 5.2.0 on Windows 10 Enterprise, 64-bit. Below my description.

The datenum function introduces unexpected daylight saving time (DST) issues. I’ve found this for spring 2019 (not for autumn) and checked for 2018 as well. According to the Octave help, datenums “do not take into account time zones […], nor seasonal changes”. This would be perfect for me, as I use UTC timestamps. However, it seems that sometimes DST comes in.

Sample code:
DD_0=datenum('2019/03/31 00:00','yyyy/mm/dd HH:MM');
for ii=1:4;
  my_datestr=sprintf('2019/03/31 0%d:00',ii);
  DD_r=datenum(my_datestr,'yyyy/mm/dd HH:MM');(DD_r-DD_0)*24
end

Output in Octave, issue at 02:00:
ans =  1.00000
ans =  3
ans =  3
ans =  4.0000

Output in Matlab (I have access to R2017b release) is correct:
ans =    1.0000
ans =    2.0000
ans =    3
ans =    4.0000

Could anybody confirm this is the same bug? Should the bug Status be changed from Fixed to Open? Or should I submit a new bug report? Any workaround to suggest?

Good, let's call this fixed for now, and you know where to come if you still hit problems in the future :)

Sorry this took so long to fix.

I pushed this fix on the default branch

https://hg.savannah.gnu.org/hgweb/octave/rev/df726a12c757

Please build and test when you can on Windows. As we showed, the unit test won't work reliably on Windows, but you can test date strings that you know would be problematic in your local time zone.

It's not that bad, it just means the test case will only work on Unixy systems.

The good news is the actual fix I wrote will work, because your output did not show the bug occurring when TZ is set to "UTC0". So the system does support the TZ variable, it probably just expects its own formats.

I'll add a "if (isunix ())" to the test and this can be considered fixed. If someone can extend the test later to use some kind of TZ variable that Windows supports, then we can look at extending it to work there also.

Here we go:

bug #36954 occurs on 2017-03-26 02:15:00 with TZ=EST+5EDT,M3.2.0/2,M11.1.0/2
bug #36954 occurs on 2017-03-26 02:15:00 with TZ=CET-1CEST,M3.5.0/2,M10.5.0/2
bug #36954 occurs on 2017-03-26 02:15:00 with TZ=CLT+4CLST,M8.2.0/0,M5.2.0/0
bug #36954 occurs on 2017-03-26 02:15:00 with TZ=LHST-10:30LHDT-11,M10.1.0/2,M4.1.0/2
bug #36954 occurs on 2017-03-26 02:15:00 with TZ=:America/Caracas

Oops, there's a small typo in the output, but the test is correct, the third string is actually "2017-08-13 00:15:00". That is what is actually being passed to datevec, only the output is wrong when I copy-pasted it.

I've updated the patch a bit, just added a comment and moved things around a little.

I'll try to work up some test cases for a few different time zones.

I think the best approach for the tests would be to iterate over a few different time zone specifications, and make sure that a set of test cases all work, each test sample being something that would fail in a particular time zone.

Please try the attached patch with a test case that works for you in your time zone. Also please make sure that time functions work normally and return the correct current local time after testing this in the same instance of Octave.

This change sets TZ=UTC0 for the block where strptime is used to convert a string into a struct tm. This ensures that no local time zone information is factored in to the conversion at all. It then tries to restore TZ by either unsetting it or setting it back if it had a value.

The only missing part here is that I am not sure if there is a way to tell whether an environment variable is missing or is set to an empty string in the m-file language, which is an important distinction here.

(file #41626)

Here is one case that breaks the approach in this patch:

>> getenv TZ
ans = America/Caracas
>> datevec ('2016-05-01 02:25:00')
ans =

   2016      5      1      2     25      0

>> datevec ('2016-05-01 02:30:00')
ans =

   2016      5      1      3      0      0

>> datevec ('2016-05-01 02:40:00')
ans =

   2016      5      1      3     10      0

Here is another case

>> getenv TZ
ans = Australia/LHI
>> datevec ('2017-10-01 01:50:00')
ans =

   2017     10      1      1     50      0

>> datevec ('2017-10-01 02:00:00')
ans =

   2017     10      1      2     30      0

>> datevec ('2017-10-01 02:10:00')
ans =

   2017     10      1      2     40      0

>> datevec ('2017-10-01 02:20:00')
ans =

   2017     10      1      2     50      0

>> datevec ('2017-10-01 02:30:00')
ans =

   2017     10      1      2     30      0

Source: https://www.timeanddate.com/time/time-zones-interesting.html

Yes, the patch also worked for me with minimal testing, and with no noticable performance impact. It makes me uncomfortable to fix this problem by converting the time structure back into a string and running regular expressions and string matching on it, there has got to be a better way... I just don't know what that is.

Now that I've done a bit of research, I also think this approach is fragile because it assumes all daylight savings shifts are exactly one hour, which I have just learned is not always the case.

The test case in this patch is definitely not useful, because it's entirely dependent on the user's local time zone setting.

I think this still needs some work and testing, and maybe a different approach. I'll see if I can come up with some cases that can't be solved this way.

The status is that it "works for me", in my timezone (CEST).

Someone more knowledgeable in DST etc. will have to morph into something that has a chance to be included in core Octave. I don't have that knowledge and I lack time/urgency.

As to Mike -IMO legitimate- performance concerns, just above the place where my patch would apply there are already three consecutive unconditionally executed strrep statements - 'nuf said.

Here's some timings (on Win7 64-bit, octave-4.3.0+ as of late May)

I used the following script that I ran twice w. & w/o patch:

t_tot = t_max = 0;
t_min = Inf;
dats = [datenum(2009, 1, 1, 0, 1, 0) : (1/24/6) : datenum(2009, 12, 31, 23, 51, 0)];
for ii=1:100
  tic; datv = datevec (dats); tt = toc;
  t_tot += tt;
  t_min = min (t_min, tt);
  t_max = max (t_max, tt);
endfor
t_avg = t_tot / 100;
t_tot
t_min
t_max
t_avg

Without my patch:
+verbatim-
Run 1:

>> t_tot

t_tot =  1.1781

>> t_min

t_min =  0.011000

>> t_max

t_max =  0.029002

>> t_avg

t_avg =  0.011781

Run 2:

>> t_tot

t_tot =  1.1891

>> t_min

t_min =  0.011000

>> t_max

t_max =  0.032002

>> t_avg

t_avg =  0.011891
-verbatim-

... and with the patch:

Run 1:

>> t_tot
t_tot =  1.1771
>> t_min
t_min =  0.011000
>> t_max
t_max =  0.028002
>> t_avg
t_avg =  0.011771

Run 2:

>> t_avg = t_tot / 100;
>> t_tot
t_tot =  1.1981
>> t_min
t_min =  0.011000
>> t_max
t_max =  0.021002
>> t_avg
t_avg =  0.011981

I'd say there's no discernable slowdown.

Just to note, looking in the cod for datevec I see in subfunc _date_vfmt2sfmt_ already 5 consecutive calls to regexprep and 3 to strrep. So those two I add can't make that much difference, can they.

Also your test is very locale / time-zone specific, will not actually test anything for US users for example. Test must set a specific time zone environment variable to be useful.

cset attached including test.

Admittedly it is a bit kludgy but the most reliable way I could come up with is to
- reconstruct the original time string with strftime from strptime's output, and
- compare that with the original string
while both have been stripped of spaces and zero digits, to get rid of a.o., leading zeros and extra spaces introduced by strftime (see below).

I've run "make check" with this cset applied to my local repo to check if any other date/time related output is affected; no FAILs were introduced by it so I consider this patch fairly safe :-)

IMO strptime isn't the most suited tool to parse date/time strings due to this DST issue, but as it is probably the only efficient tool available we'll have to make do with it.


BTW, I noted that the internal format string fed to strptime can sometimes be off in the sense that it adds leading zeros that weren't in he original date/time string, e.g., by using %H rather than %j; while in turn  %j introduces an extra space instead of a zero digit.
But as those internal formats aren't exposed I didn't investigate further.


(file #38336)

Delving a bit into it I conclude that it is actually strptime() (called by datevec.m in turn called by datenum.m) that is the culprit.

It looks like strptime() thinks that in the night of switching to DST, DST starts at 02:00 AM sharp.
The fix could simply be making strptime() interpret DST start time to be 3:00 AM rather than 2:00 AM.  After all, actual times between 2:00 AM and 3:00 AM (2:00 AM incl.) do not exist in the night of switching to DST in spring, so if a user supplies a date/time string with a time between 2:00 AM and 3:00 AM it is probably explicitly meant to be a non-DST time string.

The complementary bug might exist at the time of switching back in fall but there it is probably automatically neutralized. At least, I surmise that must be the reason that no errors show up at those times.

I'm not so proficient in C++ so I hope someone more knowledgeable in C++ and time.cc can fix this.

I had a look but I couldn't find where in time.cc the octave::sys::time t variable is actually created.

For my work, we've generally elected to use UTC timestamps on all our data to avoid issues with DST changes in local time.  I'd find it highly unfortunate if octave decided to try to enforce local time DST convention on me in such a case.  Note that while my data timestamps are in UTC, my computer clock is generally not.

Can we have someone with access to Matlab verify whether datenum produces the same result when fed either a string or a datevector?

In Octave, the code forks.  A datevector is used to directly calculate a datenum.  If a string is presented to datenum it must first be converted to a datevector by calling datevec() and this starts to involve localtime() and other system specific features.

My computer's locale is US, so Daylight Savings Time does explain the behavior I'm seeing. However, it is confusing that datenum returns different values for March 11, 2007 at 2:00am when it is expressed as a string and a vector.

code:

s = datenum('11-Mar-2007 02:00:00')
v = datenum([2007 3 11 2 0 0])

output:

s =  733112.125000000
v =  733112.083333333

What is your computer's locale? Are you in Canada or the US? I think given DST, there was no such time, two in the morning of March 11, 2007, since at that instant time was advanced by one hour. If this matches your locale settings, I think Octave is behaving correctly.

I have noticed that the functions datenum and datevec incorrectly convert date strings. Specifically, a date string that represents 2:00am is converted to a serial date number and date vector as 3:00am. I have only seen this behavior for the hour of 2:00am in the month of March in the years 2007-2010. The day on which the error occurs depends on the year. In 2007, 2008, 2009 and 2010, the error occurs on March 11, 9, 8 and 14, respectively. The following example illustrates the error for March 11, 2007.

code:

for h=1:4
  s = sprintf('03/11/2007 %02d:00',h)
  n = datenum(s)
  v = datevec(s)
end

output:

s = 03/11/2007 01:00
n =  733112.041666667
v =

   2007      3     11      1      0      0

s = 03/11/2007 02:00
n =  733112.125000000
v =

   2007      3     11      3      0      0

s = 03/11/2007 03:00
n =  733112.125000000
v =

   2007      3     11      3      0      0

s = 03/11/2007 04:00
n =  733112.166666667
v =

   2007      3     11      4      0      0