incorrect sesskey - occasional failures

incorrect sesskey - occasional failures

by Garret Gengler -
Number of replies: 27

Greetings folks...

We recently upgraded our Moodle 1.7 installation to Moodle 1.9.2.  For the most part, things are running smoothly.

Several of our users, however, have reported that forum posts have been lost when they got an "Incorrect sesskey" message.   The error message is in the formslib.php... it appears to be thrown when Moodle believes the form they are submitting was not generated for them.   (I.e., sesskey in the hidden form field does not match sesskey stored in the user's session data.)

My first thought was that users were leaving the forum post interface open for a long time, and their session was expiring.  After reviewing the logs and interviewing affected users, I'm sure that's not the case.

Has anyone experienced this?   We're almost at a loss for ways to troubleshoot this... As you can imagine, it's extremely frustrating to our users.   We are averaging two or three sesskey errors per day.  Each means the loss of a forum post or other form data, and contributes to a mistrust of Moodle.

I see that I could disable sesskey checking in various ways, but I'm afraid to do so because of the security implications.

Best regards,
-Garret

Average of ratings: -
In reply to Garret Gengler

Re: incorrect sesskey - occasional failures

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
What sort of authentication system are you using?

Sesskey errors can only happen if the users session gets messed up, and an authentication plugin is the most likely thing to be messing things up. I suppose that a firewall stripping off cookies might also cause problems. It is possible to turn on logging of cookies into the apache logs. That might help you find out what is going on.
Average of ratings: -
In reply to Tim Hunt

Re: incorrect sesskey - occasional failures

by Garret Gengler -

Thanks for the ideas, Tim...  We're using CAS.

I'll have to look into cookie logging.  That sounds like a disk space killer, but may be what we need to do to get to the bottom of this.

Is it your understanding that these are the two ways that a sesskey could fail to match?

1) the form you submit is (somehow) a stale one, because of client side browser or proxy-based caching.  I thought this was next-to-impossible these days with pragma: no-cache directives.

2) your browser fails to send your session ID cookie with a given request, which will lead moodle to generate you a new session.

 

Average of ratings: -
In reply to Garret Gengler

Re: incorrect sesskey - occasional failures

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Cookie logging is something you would probably only want to turn on briefly while diagnosing a particular problem.

I don't think either 1) or 2) quite describes it.

When you submit a Moodle form, your web browser sends back two things:
  • Your Moodle session cookie.
  • The form data.
The session cookie is what lets the server know that you are logged in to Moodle as you, and so, indirectly, what you have permission to do. The session can also store some persistent data, to get over the fact that HTTP is stateless, but that is not so important here.

The form data, and the URL it is sent to, lets the server know what you are trying to do.

Now the problem that sesskey is supposed to solve is this: suppose you are logged in as admin, so you can do anything, and you come across a malicious page which someone has set up to do a redirect to http://yoursite/moodle/course/index.php?delete=2&fulldelete=1&submitbutton=Delete. If there was no other protection in place, this would immediately delete a course category on your site.

So the sesskey is a secret stored in the session data on the server, and added to the form as a hidden field when the form is generated. So when Moodle receives the request it will only continue if the sesskey in the request matches the one stored in your session. That makes it harder (although not impossible) for malicious people to fake a request. (So there is an additional level of protection for some operations like deleting courses and categories.)

So, how can the two things fail to match on what should be a legitimate request? The form data will be OK, we do add enough no-cache headers. It is more likely to be a problem with the session, particularly if you are using a single sign-on system. With SSO, if you Moodle session times out, but you are still logged in to SSO, then it will probably automatically and silently log you back into Moodle, which will create a session with a new sesskey. The re-login may additionally do some redirects, as part fo setting up new cookies, which might also cause the form submission to be lost.
Average of ratings: Useful (3)
In reply to Tim Hunt

Re: incorrect sesskey - occasional failures

by Janet Smith -

Hi,

We currently have a very similar issue related to the sesskey and it causing login failure and it’s gotten me a little confused on sesskeys in Moodle.

To start with we use Moodle’s external database authentication and store usernames and pws in a table within this database. Additionally, we have an external portal that uses the same username and password to authenticate into it. From this Portal they click a link which automatically authenticates them to Moodle by posting their username and a session password (which expires every 20 mins) rather than the plaintext password for security. We have modified the auth.php file so that Moodle can accept either the session password or the plain text password in case they want to login to Moodle directly.

The problem we are having is that when users login from the Portal automatically to Moodle and then logout of the system they are taken back to the Moodle login screen and from there if more than 20 mins (the length of the session pw) has passed they can’t login with their plaintext password. When we look in the mdl_sessions2 table the sesskey is still there and still matches the sesskey in our local cookie. But when trying to login directly to Moodle it tells me the login is invalid. If we go back to the Portal and click the link to come back into Moodle it authenticates us fine and creates a new sesskey.

So why doesn’t it just create a new sesskey when the user is trying to login with the username and plain text pw? Or why doesn’t it just pickup that existing sesskey for that user- is the other password somehow stored in the existing sesskey?

Thanks so much for any assistance.

Janet

Average of ratings: -
In reply to Tim Hunt

Re: incorrect sesskey - occasional failures

by Garret Gengler -

Thanks for your detailed explanation, Tim.

I suspected the session was timing out too, which would result in the user redirecting to the SSO system to prove their auth credentials, then back to moodle, where they would get a new session and sesskey. 

And some of our errors appear to be that case.   I.e., you leave moodle on a form, and let your session time out, then try to post the form.   What happens next depends on whether the form requires particular user permissions.   If for instance you were posting to a course forum, and posting is restricted to class participants, then you will get a permissions error before the codebase notices the sesskey mismatch.   If you are submitting a form that does not require special credentials, you can get the sesskey error.

But all that requires the session to timeout, and that shouldn't happen before three hours of idle time, according to how we have Moodle and our SSO system configured.  Looking at the log files for our SSO system, and for Moodle, I'm seeing users who are using moodle actively, with Moodle activity log entries every minute.. the suddenly a sesskey error.

And before/during/after the sesskey error, there is no activity for that user's ip address in our SSO system's log.

Average of ratings: -
In reply to Tim Hunt

Re: incorrect sesskey - occasional failures

by Garret Gengler -

EUREKA!  After three weeks of head scratching, I can reproduce the problem.   And it's not just for sites with SSO... I can reproduce the problem with manual auth too.

Here's how to reproduce the problem in Moodle 1.9.2:

  • Log into Moodle with Firefox.
  • Navigate to a discussion forum, and open the editor to start a post.   Leave the editor open.
  • Open a second tab, or browser window.
  • In the second window, request http://host.domain.com/login/index.php
  • In the second window, enter your username and password if requested.  (You will not be asked if you use a SSO system.)
  • Return to the first browser window, and try to submit your post.   POOF!  Sesskey mismatch, and you lose what you typed.

Here's what I think is going on:

  • In Firefox, the same session is shared across all browser windows and tabs.   (MSIE is different... you get a new session for each browser window, but share sessions in tabs.)
  • login/index.php is not normally called except "internally."   I don't know why it's getting called by my users.  But they are somehow requesting that page, perhaps from bookmarks, or perhaps from a CAS/SSO redirect.
  • login/index.php, and the downstream authentication logic, does not check if the session already exists.   Eventually, the authentication routines call get_complete_user_data(), which generates a new sesskey and stores the value in the session.
  • So!  After your request to login/index.php, your sesskey has been changed for your existing session.
  • The editor you left open has a stale sesskey in the hidden field, and it fails to match the new sesskey when you try to post.

Now to fix the problem.   Perhaps it is safe to load the existing sesskey if we find an active session, instead of generating a new one.

The security ramifications of that have my head spinning.. but I'm thinking it should be safe.

Average of ratings: -
In reply to Garret Gengler

Re: incorrect sesskey - occasional failures

by Garret Gengler -

Okay this is untested, but it seems to work.. I haven't thought it through at all yet, including the security implications... so don't use this on any production systems:

In lib/moodlelib.php, round about line 8166

change this line...
   $user->sesskey  = random_string(10);

to this....

/// make a new sesskey unless we already have one
    if (!isset($_SESSION["USER"]->sesskey)) {
       $user->sesskey  = random_string(10);
    }
    else {
       $user->sesskey = $_SESSION["USER"]->sesskey;
    }

Average of ratings: -
In reply to Garret Gengler

Re: incorrect sesskey - occasional failures

by Petr Skoda -
Picture of Core developers Picture of Documentation writers Picture of Peer reviewers Picture of Plugin developers
Hello,

I do not agree. I think that if somebody opens new window and relogs in it, it is expected that the sesskey tests fail. In your example if you keep the sesskey in new session after login and submit the form, the form may get submitted as different user and I am sure that is what you do not want.

I think that solution is:
1/ educate users that they have to use different browsers if they want to login as two different users at the same time
2/ improve login page so thta it shows "You are already logged in, please logout first and close all other browser windows".

Petr
Average of ratings: -
In reply to Petr Skoda

Re: incorrect sesskey - occasional failures

by Garret Gengler -

Hi Peter.  Thanks for your input.   I think we actually agree, to a point.  But I do strongly believe there is a bug here.   I have faculty who are livid, and threatenting to abandon Moodle, in favor of the Campus WebCT installation, because they are losing forum posts!   This may be a small and subtle bug, but it has catastrophic consequences for the affected users.  It's like if you used an ATM that once in a blue moon, eats your card.   Once it happens to you one time, you never go there again.

I've been talking to the users who have experienced this bug... and none of them are intentionally entering the /login/index.php url.  It's just happening that way through the SSO login process.

So the users are just opening a new tab and typing the url for our moodle website.   And due to the existing SSO credentials and the shared session in Firefox, they are seemlessly logged in to Moodle.   And when that happens, their existing session gets assigned a new sesskey.    Unless I'm missing something, that should never happen.   A sesskey should only be stored in the session when the session is first created.

For manual authentication, your suggestion #2 can work.  I.e, we don't show the username and password form when you are already logged.   But with CAS (and possibly other SSO hooks), the user never enters the username and password.  The re-login is a seemless thing.

I'll take a look at the login/index.php code and see if I can find a clean way to trap for attempts to "re-login", in a way that works for all authentication types.

Average of ratings: -
In reply to Garret Gengler

Re: incorrect sesskey - occasional failures

by Garret Gengler -

Okay, my fix above was clearly the wrong approach.... way too deep to fix something that is easily prevented at a higher level.  And it was ugly to read from _SESSION at that level. 

Here's a new fix which is much cleaner and works just as well.

In login/index.php, add this, around line 45 (after the auth_instructions stuff).

// check if the user is already logged in
// if so, skip the login process... otherwise we can mess up
// the existing sesskey and other session data
if (isloggedin()) {
 redirect($CFG->wwwroot.'/index.php');
}

This code basically says, if you somehow happen to request the login page with a valid, logged-in session, we're not going to let you log in again. 

The login process is not safe to run against an existing session, because it generates a new sesskey, among other things.   And it's unnecessary, because you are already logged in.   So we'll just redirect you to the site homepage, and you can use your existing, logged-in session.

I'll do some more testing, then put this up on our production site asap.   We're losing about 5 posts a day right now, and the phone is ringing often with irate users.  So I have to do something.

Average of ratings: Useful (1)
In reply to Garret Gengler

Re: incorrect sesskey - occasional failures

by Petr Skoda -
Picture of Core developers Picture of Documentation writers Picture of Peer reviewers Picture of Plugin developers
If you add that redirect to index.php, what happens if you login as different CAS user? Would it break it? Does it do the moodle logout first before trying to login as different user?
Average of ratings: -
In reply to Petr Skoda

Re: incorrect sesskey - occasional failures

by Garret Gengler -

In Firefox, you basically can't do it.  You can only log into CAS and Moodle as one user at a time, because you only have one common session across all brower windows and tabs.  If you want to change users, you have to log out first.

In MSIE, it's different.  You get an independent session for each browser window, but you share a common session for all tabs.  This is very nice as a developer, if you need to test things. smile

If the session in your new browser or tab is the same, and you are currently logged-in, you cannot log in again through CAS.  Your CAS credentials are stored in session data as well.. so CAS will refuse to show you the login prompt.

It makes sense to me that the login/index.php page should not break an existing, already-logged-in session.   Either the login processing should fall through gracefully, or the login attempt should be ignored.    Currently, it almost falls through gracefully.. you do get into Moodle, and you have your existing session and your credentials.   But some things are done to your existing session that should not be... your sesskey being reset is one example.

I'm almost 100% positive it's not timeout-related.  Our timeout is set to 4 hours, and the logs show the error happening to people who have only been in Moodle for a short time.

Thank you for your prompt attention and help.  I'll put a ticket in the tracker with my proposed patch.

Best regards,
-Garret

Average of ratings: -
In reply to Garret Gengler

Re: incorrect sesskey - occasional failures

by Clark Moodler -
Garret,

I have a user getting this same issue -she's editing a forum post, opens a new window or tab in Firefox to look at something else in the course, closes the other window, submits the forum post and gets the sesskey error, loses the post.

I'm trying to figure out what to tell her - leave the other window open until you submit the post? I'm at a loss. I routinely have many tabs/windows open in Firefox and usually only get issues such as this if I have been using "login as" for other users. Also, I'm usually not posting important messages, so it doesn't bother me, but I need to find a fix for my users.

Any idears?
thanks,
clark

Average of ratings: -
In reply to Clark Moodler

Re: incorrect sesskey - occasional failures

by Garret Gengler -

It could be a similar problem.   I believe the hack I proposed is safe to add to any site... so it won't hurt to try it.  Just be prepared to roll it back out in a panic if it breaks everything.  smile

In /login/index.php, look around line 27 for this comment... 
  // auth plugins may override these - SSO anyone?

And paste this stuff just above it:

  // check if the user is already logged in
  // if so, we skip the login process... otherwise we can mess up
  // the existing sesskey and other session data
  if (isloggedin()) {
   redirect($CFG->wwwroot.'/index.php');
  }

-Garret

Average of ratings: -
In reply to Garret Gengler

Re: incorrect sesskey - occasional failures

by Clark Moodler -
Thanks Garret,

Just to follow up, I used Yugma to do screensharing with this user, and it turns out when she was opening up another tab/window with Moodle, she was using bookmarks to go to moodle and going to the login page and hitting enter, (thus she was logging in a second time, and getting another session key...)

So, it was not a system error as much as a user error, and I informed her that she should right-click a link or breadcrumb in Moodle to open another tab/window or start up a new one and NOT go to the login page...

Best,
Clark
Average of ratings: -
In reply to Clark Moodler

Re: incorrect sesskey - occasional failures

by José Cedeño -
I think it's more of a design problem. When a user who is already authenticated into a system goes to the login page, the webpage should identify that the user is already authenticated and redirect them somewhere. A separate session key shouldn't be created, the user shouldn't be logged out either. I don't know a lot about Moodle, but that's how I've coded web based applications in the past and how I expect most applications to work.
Average of ratings: -
In reply to José Cedeño

Re: incorrect sesskey - occasional failures

by Garret Gengler -

Nicely stated, Jose.   (I think) that's exactly what my hack/patch does.

If a user who is currently logged-in to Moodle requests the login url, it shouldn't break their current sesson.

I really don't care why the login url is getting requested... via a bookmark, or as a side-effect of CAS authentication.  Moodle should never reset the sesskey for a user who is already authenticated..

That's my story, and I'm sticking with it.

I put this in the tracker a while back.. vote for my issue if you want to make sure this gets fixed.
http://tracker.moodle.org/browse/MDL-16640

Average of ratings: Useful (1)
In reply to Garret Gengler

Re: incorrect sesskey - occasional failures

by Petr Skoda -
Picture of Core developers Picture of Documentation writers Picture of Peer reviewers Picture of Plugin developers
Hmm, this looks like a timeout related problem - either Moodle or CAS times out. If that code change above works around the problem, then I suppose it would be the CAS timing out because if Moodle times out the session is deleted and you would not know the old session key.

I have looked at the code, get_complete_user_data() sets the ip and sesskey - this seems really wrong in most of the cases (except loginas which is really special), it should be set in complete_user_login() instead. Unfortunately any change here might cause major regressions sad

I think your patch above is a suitable temporary solutions for 1.9.x when using CAS SSO. You should imho add username checks there to verify that the previous and new user is the same.

Going to change the sesskey and sessionIP logic a bit in HEAD(2.0) which should solve it in the long term.

thanks for all the information and patch!
Average of ratings: Useful (1)
In reply to Petr Skoda

Re: incorrect sesskey - occasional failures

by Rex Tan -
I still actually receive this error only for IE but not firefox. It seems like after CAS re-authentication, it regenerates the sess-key

I posted a screenshot:

IE-Screenshot-2.png

I tried the patch above but it doesn't seem to work for this particular error since it's not related to logins but when you click the (Turn editing on) button.
Average of ratings: -
In reply to Rex Tan

Re: incorrect sesskey - occasional failures

by Rex Tan -

I think i've resolved the problem.

This error will occur with all authentication methods (when sesskey is involved) on the Internet Explorer browser after the Google chrome frame is added.

The google chrome frame is an IE extension that will mess up the sesskey of Moodle (tested) - Someone to confirm?

Average of ratings: -
In reply to Garret Gengler

Re: incorrect sesskey - occasional failures

by Kameron Fulp -

I am a student user.  I clicked on a link to start a quiz and received this instead. I was not logged in on Moodle. I was not logged into more than one session on the website. I did not have additional tabs open. I was just trying to get my work done. I have checked my cookies and my cache. Nothing seems to work. Whenever I try to take the quiz again, I receive the same message.

So what am I supposed to do about that? I have downloaded the most recent moodle, from three days ago, and I still can't get in.

Average of ratings: -
In reply to Garret Gengler

Re: incorrect sesskey - occasional failures

by april brandon -

I have experienced the loss of a post 4 or 5 times. After reading these posts I realize that every time I did have other tabs open. I just lost 1000 words that I do not care to re-write but I must. My only option, as the professor does not want an attatchement, is to write a word document and copy and paste. It will allow me to do this, even though is says that it won't. I can copy and paste once per post, though often it will tell me that I cannot several times. I have to hit the cancel button and try again but so far it has worked every time. I hope this helps. I am truly fed up with moodle.

Average of ratings: -
In reply to Garret Gengler

Re: incorrect sesskey - occasional failures

by Thomas Bristol -

I'm somewhat new to the moodle scene but I felt like posting this could be helpful. 

I'd been troubleshooting this one for days and none of the solutions offered were solving my problem. We had several students trying to self register for a course and they would get this error when trying to create their account. It turns out that the web address given to the students was the direct web address for registering with www. at the beginning. The form would load and create a sesskey for www.domain.com/login/... When the form was submitted to the domain in the config file (http://domain.com/...)  a new sesskey was created because the domain changed. So when the scripts compared the form submitted sesskey with the new one created the would not match.

My solution was to use a rewrite rule on the server that would force all incoming trafic from www.domain.com to domain.com

Average of ratings: -
In reply to Garret Gengler

Re: incorrect sesskey - occasional failures

by Safwan Ahmad -

This error also appears with moodle install on nginx setup. I used this tutorial: http://www.lowendguide.com/1/getting-started/installing-wordpress-using-minstall/

 

But i got the error while setting the admin password.

 

I used the hack here: https://moodle.org/mod/forum/discuss.php?d=234656

 

And the error went away and i am having peace for now.

Average of ratings: -
In reply to Garret Gengler

Re: incorrect sesskey - occasional failures

by Fiona McGarry -

I just submitted my post to a discussion forum and it said error and now I cannot find my post.  Please tell me it is not lost!!!   How do I get it back?  

Fiona.  

Average of ratings: -
In reply to Garret Gengler

Re: incorrect sesskey - occasional failures

by Николай Nikolas -

I find it.

In table mdl_config

Key in string dbsessions was 0, change it on 1 - all work!!! smile

 



Average of ratings: -