bugGNU Octave - Bugs: bug #51340, datevec jump hour on 29/03/2009...

 
 

bug #51340: datevec jump hour on 29/03/2009 02:*

Submitter:  Juan Pablo Carbajal <juanpi>
Submitted:  Thu 29 Jun 2017 07:57:29 AM UTC
   
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Incorrect Result
Status:  Duplicate Assigned to:  None
Originator Name:  juanpi Open/Closed:  * Closed
Release:  * dev Operating System:  * GNU/Linux
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Wed 23 Aug 2017 07:56:45 AM UTC, comment #20: 

Closing as a dupe of bug #36954.

Mike Miller <mtmiller>
Group Member
Sun 02 Jul 2017 03:47:22 AM UTC, comment #19: 

output from Mike's request in comment #13:


>> format compact
>> format long
fmt = 'yyyy-mm-dd HH:MM:SS';
x = datenum ('2017-03-12 01:59:59', fmt)
x(2) = datenum ('2017-03-12 02:00:01', fmt)
x(3) = datenum ('2017-03-12 02:59:59', fmt)
x(4) = datenum ('2017-03-12 03:00:01', fmt)
round (86400 * diff (x))
x =
     7.367660833217592e+05
x =
   1.0e+05 *
   7.367660833217593   7.367660833449074
x =
   1.0e+05 *
   7.367660833217593   7.367660833449074   7.367661249884259
x =
   1.0e+05 *
  Columns 1 through 3
   7.367660833217593   7.367660833449074   7.367661249884259
  Column 4
   7.367661250115742
ans =
           2        3598           2


Nicholas Jankowski <nrjank>
Group Member
Sat 01 Jul 2017 10:30:13 AM UTC, comment #18: 

@JuanPi:  AFAIK we usually close later bug reports as duplicates in favor of the earliest one.
TTBOMK the actual cause / bug / deficiency lies with strptime(); but that is no Octave function so we must relate it to a function callable in Octave. I'm indifferent to whichever that is.

Maybe we let Mike decide?

Philip Nienhuis <philipnienhuis>
Group Member
Sat 01 Jul 2017 02:55:07 AM UTC, comment #17: 

Not so important, but the bug dependence seems inverted. datenum calls datevec calls strptime.

Juan Pablo Carbajal <juanpi>
Group Member
Fri 30 Jun 2017 09:19:57 PM UTC, comment #16: 

Philip - yes, I believe the same underlying bug affects both datevec and datenum. We can call this a duplicate of bug #36954. When datenum is given a string, it calls datevec to convert the string into a datevec array. And both are due to the behavior of strptime and mktime in correcting a non-existent time value into a valid time value.

Mike Miller <mtmiller>
Group Member
Fri 30 Jun 2017 08:11:59 PM UTC, comment #15: 

I'm not convinced that the underlying issue is different than bug #36954.
The issue there is in the end identical to what is shown in comment #9 here.

That this bug is about datevec.m is a red herring - what matters is that any function that in the end relies on strptime() is affected.

What I found -after some web-searching- is that what strptime() does for non-existent times is implementation-dependent - see bug #36954 comment #11.
Octave apparently uses a "faulty" strptime() implementation.

Philip Nienhuis <philipnienhuis>
Group Member
Thu 29 Jun 2017 11:08:14 PM UTC, comment #14: 

Mike,

Ok got it now. Due to time changes some hours get lost and the system is aware of this, even if my TZ is empty (I guess it is getting the CET from somewhere else).

Regarding the "correct" behavior, I would say that date string should be read 'asis' no timezone awareness at all (I guess this really means that the timezone is always UTC). Let the user do the conversion.

I guess since there is no metion of timezones in the mathworks documentation that is what they do

https://ch.mathworks.com/help/matlab/ref/datevec.html

and I think they offer datetime (not implemented in Octave)

https://ch.mathworks.com/help/matlab/ref/datetime.html

to actually apply a timezone to a datestr or datevec

I would say this is the behavior implemented by the pair, strptime (system's. It is timezone unaware),  and date (GNU program. It is timezone aware).


Juan Pablo Carbajal <juanpi>
Group Member
Thu 29 Jun 2017 10:38:23 PM UTC, comment #13: 

What I mean is that this wallclock time does not exist in your local time zone. If you had been recording some kind of time series data at say one minute intervals on your system, the data log would have gone from "01:58:00" to "01:59:00" to "03:00:00" because the hour between 02:00:00 and 02:59:59 does not exist on that particular day.

When I use GNU date to try to parse such a date string, I get back a definitive error:


$ date -d '2017-03-12 01:59:59'
Sun Mar 12 01:59:59 PST 2017
$ date -d '2017-03-12 02:00:00'
date: invalid date ‘2017-03-12 02:00:00’
$ date -d '2017-03-12 03:00:00'
Sun Mar 12 03:00:00 PDT 2017


If you adjust your C program to look at the time_t value returned by mktime(), you'll see that that value actually represents 03:00:00:


   strptime("29/03/2009 02:00:00", "%d/%m/%Y %H:%M:%S", &tm);
   time_t t = mktime(&tm);
   localtime_r(&t, &tm);
   strftime(buf, sizeof(buf), "%d %b %Y %H:%M", &tm);


That's what I mean when I say that this example time is specifying an invalid / nonexistent time in your time zone.

There may be a Matlab compatibility issue here, from comment #11 it looks like Matlab parses the string into a datevec regardless of whether it can represent a valid time or not.

nrjank - since you have Matlab 2017 available, and you're on the same US timezone rules, can you try the following:


format long
fmt = 'yyyy-mm-dd HH:MM:SS';
x = datenum ('2017-03-12 01:59:59', fmt)
x(2) = datenum ('2017-03-12 02:00:01', fmt)
x(3) = datenum ('2017-03-12 02:59:59', fmt)
x(4) = datenum ('2017-03-12 03:00:01', fmt)
round (86400 * diff (x))


In Octave I get


x =

   736766.083321759   736766.125011574   736766.166655093   736766.125011574

ans =

   3602   3598  -3598


Mike Miller <mtmiller>
Group Member
Thu 29 Jun 2017 08:32:46 PM UTC, comment #12: 

Hi Mike,

Thanks for the effort. I still dot not understand how the current behavior is acceptable. It just misbehaves for the period of 1 hour and for that day.

1. In the first assert indeed the "_" should be erased.

2. The element-wise comparisson was intentional to show that it fails in the span of an hour

3. What do you mean by invalid time string/specification? '29/03/2009 02:00:00' is a valid time string.

4. I do not get any error with my system's strptime, it always works as expected (I would expect Octave to do the same, but maybe it is a compatibility issue). I gave the example program in comment #1

5. I get empty when getting TZ

>> getenv TZ
ans =


6. The example in comment #9 works fine here.

>> getenv TZ
ans =
>> fmt = "yyyy-mm-dd HH:MM:SS";
>> datevec ("2017-03-12 02:00:01", fmt)
ans =

   2017      3     12      2      0      1

>> datevec ("2017-03-12 03:00:01", fmt)
ans =

   2017      3     12      3      0      1

I understand that in the octave wrapper of strptime we are adding some TZ awareness. I also assume this is an inherited bug from Matlab compatibility. Right?

7. Setting TZ to UTC solves the original issue. I just wonder whether it will be triggered somewhen else.

Juan Pablo Carbajal <juanpi>
Group Member
Thu 29 Jun 2017 08:28:32 PM UTC, comment #11: 

correcting my earlier comment #4, my timezone behaves similarly to Mikes, although I don't have a TZ env variable.  Don't know what the equivalent is in Windows. (local TS is EDT/EST, UTC-4/-5)


>> getenv TZ
ans =
>>  datevec ("2017-03-12 02:00:01", fmt)
error: 'fmt' undefined near line 1 column 34
>> fmt = "yyyy-mm-dd HH:MM:SS";
>>  datevec ("2017-03-12 02:00:01", fmt)
ans =

   2017      3     12      3      0      1


If we want to consider compatibility, the Matlab 2017a behavior seems to ignore this issue:

>> fmt = 'yyyy-mm-dd HH:MM:SS'
fmt =
    'yyyy-mm-dd HH:MM:SS'
>> datevec ('2017-03-12 02:00:01', fmt)
ans =
  Columns 1 through 5
        2017           3          12           2           0
  Column 6
           1


Nicholas Jankowski <nrjank>
Group Member
Thu 29 Jun 2017 07:49:50 PM UTC, comment #10: 

What should the behavior of datevec be when given an invalid time string?

In my time zone there is no such thing as "2017-03-12 02:00:00". The GNU date program correctly reports an error. The next time increment after "2017-03-12 01:59:59" is "2017-03-12 03:00:00".

In your example and in your time zone, "29/03/2009 02:10:05" is an invalid time specification.

Would you prefer Octave to fold the time over by one hour the way it does now, or throw an error when given an invalid time string like this?

Or were those times recorded in UTC and you should be setting your local time zone to UTC before processing them?

Mike Miller <mtmiller>
Group Member
Thu 29 Jun 2017 07:42:13 PM UTC, comment #9: 

I can reproduce what I think you are reporting with the following here:


>> getenv TZ
ans = PST8PDT
>> fmt = "yyyy-mm-dd HH:MM:SS";
>> datevec ("2017-03-12 02:00:01", fmt)
ans =

   2017      3     12      3      0      1

>> datevec ("2017-03-12 03:00:01", fmt)
ans =

   2017      3     12      3      0      1


Mike Miller <mtmiller>
Group Member
Thu 29 Jun 2017 07:38:10 PM UTC, comment #8: 

juanpi - can you please show your TZ environment variable, or "timedatectl status | grep zone", or some other way so others can try with your time zone setting?

Like Dmitri I am unable to reproduce any kind of error so far.

Please note that you have a couple of mistakes.

In the original post, I think you meant "assert (vec_, vec)", which works for me.

In comment #1, I think you meant "assert (datevec ('29/03/2009 02:10:05',fmt), datevec ('29/03/2009 03:10:05',fmt))", because the != is elementwise, you want a scalar for assert. For me they are equal elementwise except for the hour position.

Mike Miller <mtmiller>
Group Member
Thu 29 Jun 2017 06:18:53 PM UTC, comment #7: 

It might be, but the issue is really datevec not datenum.
also it the underlying system function works fine...so it is somewhere between datevec and strptime

Juan Pablo Carbajal <juanpi>
Group Member
Thu 29 Jun 2017 04:49:39 PM UTC, comment #6: 

Could be related to bug #36954 ?
That bug is about wrong evaluation of DST in spring (I hit it because I very often work with time series of hourly measurements, and I added a patch there that I've been using w/o issues for a fairly long time now).

Philip Nienhuis <philipnienhuis>
Group Member
Thu 29 Jun 2017 03:27:35 PM UTC, comment #5: 


octave:19> fmt = 'dd/mm/yyyy HH:MM:SS';
octave:20> datevec ('29/03/2009 02:10:05',fmt)
ans =

   2009      3     29      2     10      5

hg id
251cb33c6570 tip @

(same with 4.2.1)

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Thu 29 Jun 2017 01:37:10 PM UTC, comment #4: 

just to verify this appears to be linux only, on windows 7 64bit, octave 4.2.1 i get:


>> fmt = 'dd/mm/yyyy HH:MM:SS';
>> datevec ('29/03/2009 02:10:05',fmt)
ans =

   2009      3     29      2     10      5



Nicholas Jankowski <nrjank>
Group Member
Thu 29 Jun 2017 11:27:03 AM UTC, comment #3: 

Dmitri,

Thanks for checking, hat do you get if you do


fmt = 'dd/mm/yyyy HH:MM:SS';
datevec ('29/03/2009 02:10:05',fmt)


Juan Pablo Carbajal <juanpi>
Group Member
Thu 29 Jun 2017 11:11:11 AM UTC, comment #2: 

I cannot reproduce this on Fedora 26
But may be I misunderstood your bug report.

Dmitri.

Dmitri A. Sergatskov <dasergatskov>
Thu 29 Jun 2017 08:20:32 AM UTC, comment #1: 

Another way of triggering the bug is

fmt = 'dd/mm/yyyy HH:MM:SS';
assert (datevec ('29/03/2009 02:10:05',fmt) != datevec ('29/03/2009 03:10:05',fmt))


Juan Pablo Carbajal <juanpi>
Group Member
Thu 29 Jun 2017 07:57:29 AM UTC, original submission:  

Hi,

datevec adds one hour to dates on 29/03/2009 02:*

I chased this bug down to datevec (actually I think the bug is in octave::sys::strptime)


vec        = repmat ([2009 3 29 2 0 0],6,1);
vec(:,5:6) = randi(59,6,2);  # random minutes and seconds
fmt        = 'dd/mm/yyyy HH:MM:SS';
vstr       = datestr (vec, fmt);
vec_       = datevec (vstr, fmt);
assert (vec_, vec_);


I checked that the bug is not in my system's strptime. The following program outputs the right values


#define _XOPEN_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <time.h>

int
main(void)
{
   struct tm tm;
   char buf[255];

   memset(&tm, 0, sizeof(struct tm));
   //strptime("2009-03-29 14:00:00", "%Y-%m-%d %H:%M:%S", &tm);
   //strptime("2009-03-29 02:00:00", "%Y-%m-%d %H:%M:%S", &tm);
   strptime("29/03/2009 02:00:00", "%d/%m/%Y %H:%M:%S", &tm);
   strftime(buf, sizeof(buf), "%d %b %Y %H:%M", &tm);
   puts(buf);
   exit(EXIT_SUCCESS);
}


Juan Pablo Carbajal <juanpi>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Digest:
   bug dependencies.

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by mtmiller (Posted a comment)
  • -email is unavailable- added by nrjank (Posted a comment)
  • -email is unavailable- added by dasergatskov (Posted a comment)
  • -email is unavailable- added by juanpi (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2017-08-23 mtmiller StatusNone Duplicate
        Open/ClosedOpen Closed
    2017-06-30 mtmiller Dependencies- Depends on bugs #36954

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code