LDAPS has stopped working

LDAPS has stopped working

por heli g -
Number of replies: 8

LDAPS stopped working this morning. This occurred in several Moodle installations on different servers. I have no idea where to look to try resolve this.

My servers are Ubuntu 14.04 (PHP 5.5, mysql 5.5, Moodle 2.9) and we are authenticating against MS Active Directory. LDAPS has been working fine for just under a year.

I have verified the certificates using

openssl verify -verbose -CApath /etc/ssl/certs /etc/ssl/certs/server1.domain.pem

I can connect to the LDAPS server over SSL using

openssl s_client -connect  server1.domain:636

The one issue I notice is below (but we have always had this and until this morning authentication and password change worked):

No client certificate CA names sent
...
Verify return code: 21 (unable to verify the first certificate)
------------------

The Debug info when I try login is below:

LDAP-module cannot connect to any servers: Server: 'ldaps://ldapserver1.domain, Connection: 'Resource id #21', Bind result: '' Server: 'ldaps://server2.domain', Connection: 'Resource id #22', Bind result: ''

More information about this error

Debug info:
Error code: auth_ldap_noconnect_all
Stack trace:
  • line 481 of /lib/setuplib.php: moodle_exception thrown
  • line 2154 of /auth/ldap/auth.php: call to print_error()
  • line 208 of /auth/ldap/auth.php: call to auth_plugin_ldap->ldap_connect()
  • line 4187 of /lib/moodlelib.php: call to auth_plugin_ldap->user_login()
  • line 144 of /login/index.php: call to authenticate_user_login()
Students can login because I have changed the configuration to use LDAP, but I would really like to get this issue fixed. I have been assured that there have been no changes that could affect Moodle logins in the Domain Controllers or AD.

Ideas anyone? Can anyone help?

Heli
Average of ratings: -
In reply to heli g

Re: LDAPS has stopped working

por Emma Richardson -
Imaxe de Documentation writers Imaxe de Particularly helpful Moodlers Imaxe de Plugin developers

Did your certificate expire?

Your code:21 is definitely showing a problem with the certificate.

In reply to Emma Richardson

Re: LDAPS has stopped working

por heli g -
Hi Emma,

I got LDAPS working again by exporting the directory servers' DER certificates, converting them to PEM, creating the certificate hashes and restarting apache (basically redoing what I did when I first enabled LDAPS https://docs.moodle.org/30/en/LDAP_authentication#Using_LDAPS_.28LDAP_over_SSL.29 ).

This indicates that there was something wrong with the certificates, however if this were the case I do not understand how it only affected my 3 Moodle servers and not the entire University... (Out institution is mostly Microsoft, Moodle is the only core system running on Linux, and I am not very familiar with Windows and AD DCs - so I may be missing something obvious).

Do you know how to check a certificates expiry? openssl verify's response was "OK" even when there was obviously something wrong and you could not login.

It is working now, you can login to Moodle and change passwords, however I still see errors when running:

openssl s_client -connect  server1.domain:636
The error codes are below, but since it is working, I am not sure how significant they are

verify error:num=20:unable to get local issuer certificate
verify error:num=27:certificate not trusted
verify error:num=21:unable to verify the first certificate

My main concern is now is to try understand how this happened and how to avoid it happening again.

Thanks for getting back to me

Heli
In reply to heli g

Re: LDAPS has stopped working

por Ken Task -
Imaxe de Particularly helpful Moodlers

Can't really speak from experience with ldaps but have had to move Moodle servers from http to https with valid crts from a Certificate of Authority.

When certificate request (csr) are generated to submit to a true CA, there is an option for setting the number of days a certificate would be valid ... 1 yr, 2 yrs, etc.   Could be the .pem's, which, from what I've read, are antiquated and many recommend they no longer be used, had a 'ttl' (time to live) setting which your testing didn't show/mention.  So certs expired.  The errors related to certificate NOT trusted means generated for/by 'localhost' or FQDN for the server by the server itself and never submitted to a true CA thus no true crts.   Which, I guess, is ok to do between your Moodle server and your own LDAPS server ... IF they are gen'd by server admins.

Crts must match server key.   Below from a CentOS server ... not Ubuntu.

to check:

openssl x509 -noout -modulus -in certs/FQDN.crt | openssl md5 > crtmodulus;
openssl rsa -noout -modulus -in private/FQDN.key | openssl md5 > keymodulus;

should show the same modulus (example obscured):

(stdin)= fc529832ac8e5jhkl5ca84609a0dc133c

for both the crt and the key - they are a valid pair.

One can inspect the contents of a crt (and probably a .pem) via:

openssl x509 -in FQDN.crt -text

Will show info contained in the .crt file.

Dunno that the above helps or not ...

'spirit of sharing', Ken



In reply to heli g

Re: LDAPS has stopped working

por Emma Richardson -
Imaxe de Documentation writers Imaxe de Particularly helpful Moodlers Imaxe de Plugin developers

If the cert expired it would have been renewed everywhere else effectively creating a new cert.  If the new cert was not imported to Moodle it could have had that effect.  You can check the expiration of certificates on your ad server in the certificates management.

In reply to Emma Richardson

Re: LDAPS has stopped working

por heli g -

Thanks Ken and Emma, 

It certainly appears that something changed with the certificates, but if this is the case, whatever changed must have been automatically propagated to Domain joined Windows systems without human intervention (is this possible?)

- or - 

somehow in the conversion the certificates were given a 6 month time to live. Just musing here...

I've checked my documentation and the certificates which were no longer working were exported in DER format and converted to PEM 6 months ago.

We use different certificates for https - a Commercial wildcard - here I exported a .pfx from Windows and converted it to a .pem and .key file. eg.

openssl pkcs12 -in Wildcard.pfx -clcerts -nokeys -out *.domain.pem
openssl pkcs12 -in Wildcard.pfx -nocerts -nodes  -out *.domain.key  

I also placed the intermediary certificate in my certs directory.

For LDAPS, however, I merely exported the root CA and DC certificates as DER (without private key) and converted them to PEM. eg.

openssl x509 -in local-SERVERNAME-CA.cer -inform DER -out
servername-ca.local.domain.pem -outform PEM
placed them in the /etc/ssl/certs directory, added the paths to /etc/ldap/ldap.conf, restarted Apache ... And ldaps just worked...

It seemed almost too easy.

@Ken do you recommend a different method and output for conversion - any examples on how to do this?

Regarding LDAPs do you think I should include the private key and intermediary certificates?

Or perhaps I should just let this go, since the system is working now...

Below is a graph showing our active users and why I would very much like to know how to avoid this ever happening again.

Moodle active users


Thank you both for your time.

Best wishes

Heli

In reply to heli g

Re: LDAPS has stopped working

por heli g -

PS: Most of what I have learnt about Moodle comes from trawling these forum pages - it is because of this that I may go overboard typing in commands. I do it just in case someone like me may have a similar problem, stumble upon the post - and it may help.

I am so very grateful to this wonderful and supportive community.


Heli

Average of ratings:Useful (1)
In reply to heli g

Re: LDAPS has stopped working

por Iñaki Arenaza -
Imaxe de Core developers Imaxe de Documentation writers Imaxe de Particularly helpful Moodlers Imaxe de Peer reviewers Imaxe de Plugin developers

It certainly appears that something changed with the certificates, but if this is the case, whatever changed must have been automatically propagated to Domain joined Windows systems without human intervention (is this possible?)

Yes, this coud be possible. Windows systems don't use LDAP+SSL to securely connect to Active Directory. They use GSS-SPNEGO, which usually means they end up using GSSAPI with Kerberos tickets, unless they are very old Windows systems.

Kerberos tickets are not related to X.509v3 certificates, so that's why even if your root CA and DC certificates are rotated, the Windows systems can talk to AD without problem (but note that AD can push lots of stuff to Windows clients silently, especially credentials to access AD-related services and resources, and things related to Enterprise CAs managed through Windows Certificate Services or whatever it's called nowadays).

@Ken do you recommend a different method and output for conversion - any examples on how to do this?

I'm not Ken, but I hope you don't mind me answering the question sorrisa PEM is the right format if you are using OpenLDAP's libraries (which is what PHP uses unless you compile it yourself specifying a different supported library). It's the format OpenLDAP's libraries expect.

Regarding LDAPs do you think I should include the private key and intermediary certificates?

You don't need the private key to verify the DC certificate at all (in fact you SHOULDN'T copy the private key anywhere, except for disaster recovery purposes). Even more surprisingly, you don't even need a copy of the DC certificate! sorrisa You just need the public key (which is part of the certificate itself) for the CA. Or if you are using intermediate CAs, the certificates of all the CAs in the trust chain starting from the root CA and up to the CA signing the DC certificate (this is what some people call the "CA certificate bundle").

When connecting to the DC via SSL, the DC will send the certificate back to the connecting system as part of the SSL handshake (see http://www.moserware.com/2009/06/first-few-milliseconds-of-https.html for all the gory details). In order to verify the authenticity of the certificate (in addition to temporal validity, the name of the remote system, etc.) you'll need to check the digital signature of the certificate. And for that, you'll use the certificate of the CA issuing the DC's certificate (and the upper CA issuing that CA's certificate, and so on, up to the root CA). And that CA's certificate must be valid. Which means, among other things, it shouldn't have expired.

So you need to make sure that the COPY of the CA's certificate that you copied to your Moodle server has not expired. Which means you should check when your CA's certificate is going to be renewed (which happens on your Windows Certificates Server, and might happen way before it will expire) and then export and copy the new CA's certificate to your Moodle server. I would keep the old CA's certificate around until it expires, because some certificates could be using the new CA's cert and some others the old CA's cert.

Yes, it's a bit complicated, with lots of moving parts that can break down when you don't expect them tristayu

Saludos. Iñaki.

Average of ratings:Useful (1)
In reply to heli g

Re: LDAPS has stopped working

por Ken Task -
Imaxe de Particularly helpful Moodlers

Sorry ... didn't see the @Ken there and have been busy doing other stuff ... like living! :|

Anyhoooo ... the command you used was one for a 'wildcard' cert ... can be used with any server within a FQDN.   They have the same example shown in Entrust/Digicert (the only two commercial CA's with which I had to work).

Creates a pfx file that then can be restored to any server within that domain ... has to do with getting matching pairs ... crt and key.  There's a pfx file converter that will gen the cert with a matching key for the server then one uses the .key file gen'd by that to setup on web server (normally, one generates the key and crs on the server using FQDN to submit to a CA.  CA takes the crs and gens the crts that matches your key.  key is always kept private.).

So no, would not use anything different for cert for a Moodle to run https - one has to do what the CA says. ;)

Like I said ... don't do LDAPS so will defer to Iñaki there for sure.

'spirit of sharing', Ken