Zone/Rule file distribution & Olson stuff

Zone/Rule file distribution & Olson stuff

by Martín Langhoff -
Number of replies: 5
Jon, I see you're working on the DST stuff, I hope all is going well. Let me know if I can help.

With regards to distribution, MD mentioned file size as a concern for the main distribution of files. Checked which of the Olson files are really needed, and stripped away the comments. CSV files are probably going to be even smaller, but I thinkthis info is also relevant for future distribution from Moodle.org

These are the required files, stripping comments and "Link" records which we don't need:

cat africa antarctica asia australasia europe northamerica southamerica  | grep -v '^\(#\|Link\)' > allzones

It's about 130K, which gzip reduces to 26K. I am skipping a few legacy files, and the files for Riyadh which are mind-boggling because the place has strange day lengths so they have 365 rules per year.

Ah, this brilliant idea of making the world spherical.
Average of ratings: -
In reply to Martín Langhoff

Re: Zone/Rule file distribution & Olson stuff

by John Papaioannou -
Sorry for not replying earlier, but I seem to have my hands fuller than I 'd like to these days. After this depressing preamble, though, let's start sharing the good news: smile

All the internal code that deals with DST offsetting is now in HEAD! cool
(and the people rejoiced)

Now, what we need to do first and foremost is:

  1. Make a good and correct import procedure.
  2. Provide seamless transition from the legacy $USER->timezone system.

The import procedure is currently being done by admin/dst_update.php (this will probably get a rename before long). You can reach that through the administration UI, but right now it's barely working correctly.

What it does do, is:
  • Kill all your mdl_timezone table entries
  • Parse $CFG->$CFG->dataroot.'/temp/olson.txt'
  • Insert all the records it parses into mdl_timezone
  • Insert a bunch of generic "Unspecified location" records as well
  • die()
The first order of the day here is to make sure that all data is being parsed correctly. I 'm saying this because I 'm having some problems with parsing, namely olson_parse_at() seems to have problems with the America/St_Johns, America/Goose_Bay (these two use the StJohns rule) and Asia/Karachi zones. It leaves the times NULL. Some help from Martin L would be wonderful here. Regarding this process, olson_parse_at() is the only thing that doesn't give me a warm fuzzy feeling inside.

Once this crude import procedure works 100% correctly, we can refine it to be more gentle and have a pretty interface.

Another important thing here is the code snippet that inserts the "unspecified location" timezones in the table:
$timezone = new stdClass;
$timezone->year = 1970;
for($i = -13; $i <= 13; $i += .5) {
$timezone->gmtoff = $i * HOURSECS;
$timezone->name = get_string('timezoneunspecifiedlocation').' / GMT';
$timezone->sortkey = 'UNSPECIFIED_'.sprintf('%05d', 13 * HOURSECS + $timezone->gmtoff);
if($i < 0) {
$timezone->name .= $i;
}
else if($i > 0) {
$timezone->name .= '+'.$i;
}
insert_record('timezone', $timezone);
}

I 'm meaning for this to go into /lib/db/mysql.php exactly as it stands, but it's not there yet because of one thing. You 'll notice that it's setting a "sortkey" field in there, which doesn't exist in the database. The reason for this is that in the end "name" alone is not enough. Sorting these "unspecified location" timezones by name is ultra-yucky, it suffices to say that + comes before - in an alpha sort (not to mention that +10 comes after +1 and before +2).

It therefore seems that since we 're going to have to somehow sort nicely in order to show that "select timezone" drop down in the user profile, there's need for an additional field. In my dev install this is "sortkey varchar(100)", and it's only used for sorting that one drop down menu (talk about waste for the sake of prettiness). Does anyone have a better idea?

The above is one of the two things that are holding the "migration procedure" up right now. The other one is this:

Moodle currently has a "Server's TZ" setting for timezones. We probably need to have one that's equivalent under the new system. It's not technically required, because the admin can always upgrade, go to the vars config, pick the correct GMT+-X timezone and that's it. However, since people might have set that to "Server's TZ" they would be in for a big time-shifting surprise if they fail to do that, and I wouldn't want to be Martin's Moodle Services in that situation.

The problem is that our current GMT offset field is called "gmtoff", and it holds the offsets in seconds. There is going to be one record among the thousands in that table that we have to somehow mark as "this one isn't using a real offset, it's a placeholder for the server's offset". One possible idea would be to use the special value "-1" for that case. Another idea would be to use "999999999". An even more sly idea would be to not have such a record, and hardwire Moodle's internals to recognize a special "Default/Server" timezonename and act accordingly. I 'm not sure which one would be better, or even if there's an even slicker solution I haven't thought of. Thoughts anyone?

Once this is also taken care of, we can write an upgrade snippet that enters these records into mdl_timezone, updates the "timezonename" field for each user, updates $CFG->timezonename, drops the "timezone" field from mdl_user and follow up with code that totally uses the new timezone system for calculations.

(Right now, the "new" timezone is only used for DST offsetting; the "old" timezone is still used to determine offset in relation to GMT).

That's probably enough rhetoric for one day... any input on the highlighted issues?


Jon


PS: There's enough code in HEAD right now to test the DST offsetting yourselves if you like.

In reply to John Papaioannou

Re: Zone/Rule file distribution & Olson stuff

by Martin Dougiamas -
Picture of Core developers Picture of Documentation writers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers
Thanks, Jon! approve

I've started playing with the latest changes on test.moodle.com and have a few notes along the way:

Import

I did the import OK, but for some reason I've got multiple copies of many of the zones in my list. Later I still really want to see the CSV version by default in lib/timezones.csv smile


The "unspecified" zones

I don't think there needs to be a sort field. In fact why do these standard timezones need to be in the table at all? I think these could be created at the time the menu is displayed (just as in 1.4, except now we will append all the other named timezones to the list).

If we use the same values as we use in 1.4 for the $user->timezone field (-12, -11.5 ...0 ... 12.5, 13) then we can upgrade nicely with no effort. It should be pretty easy to use is_float() during calculation time, yeah?


User table

Following on from this ... I would keep the user->timezone field and just change the type from int to varchar (and not have a timezonename field at all)... with my scheme above we'll just keep the same values for all users and everything continues smoothly.

What does the new code do when the timezone is 99?

Translations

I think we need to provide a new lang/en/regions.php file that allows translations of the timezone region names when displaying the list to users ... even if the English regional names are used internally in the timezone field and the timezone table.

There could also be entries for displaying the numeric timezones in there, like GMT+8 etc
In reply to John Papaioannou

Re: Zone/Rule file distribution & Olson stuff

by Martín Langhoff -
Cool! I've been on Easter holiday, sorry I haven't answered any earlier. I'm checking those zones that are giving you trouble now.
In reply to John Papaioannou

Re: Zone/Rule file distribution & Olson stuff

by Martín Langhoff -
Just committed two small fixes to the Olson parser, one of which resolves the problems with the StJohn's rules. My regex was being too accommodating ;)

(at last! sorry about the delay)
In reply to Martín Langhoff

Re: Zone/Rule file distribution & Olson stuff

by John Papaioannou -
No worries, I 've just come back from 4 days in Istanbul as well. The bad timing award is mine still. wink