Moodle in English: failed to start vpl jail server (SSL

failed to start vpl jail server (SSL_CTX issue)

by Garth Brady - Sunday, February 7, 2021, 5:59 AM

After running through the install on a fresh (updated) copy of Ubuntu20, I get the following error:

SSL_CTX_use_certificate_chain_file() fail: error:02001002:system library:fopen:No such file or directory

I allowed the VPL script to create a self-signed certificate and all seemed to go well. I can see the cert.pem and key.pem files in "/etc/vpl/" directory. However, the vpl-jail-system fails to start. Any idea why this would be?

Re: failed to start vpl jail server (SSL_CTX issue)

by Garth Brady - Monday, February 8, 2021, 4:33 AM

On an AWS EC2, Self-Signed certificates are a royal pain to deal with (an intermediate cert is required but this process is not straight forward in an AWS environment... Instead, I ended up opting for Let's Encrypt. Be sure to use an alias as Let's Encrypt's certbot writes the key pairs to a non vpl directory so you must:
"Replace /etc/vpl/cert.pem [in the vpl configuration file] with a link to /etc/letsencrypt/live/yourservername/fullchain.pem and /etc/vpl/key.pem with a link to /etc/letsencrypt/live/yourservername/privkey.pem. Another option is to copy files."

Average of ratings: -

Re: failed to start vpl jail server (SSL_CTX issue)

by Juan Carlos Rodríguez-del-Pino - Monday, February 8, 2021, 5:10 PM

Dear Garth,
version 2.6.0 of VPL-Jail-System adds two new configuration parameters that allow setting the server's certificate and private key files.
See
https://vpl.dis.ulpgc.es/documentation/vpl-jail-system-2.6.0/configuration.html#ssl-cert-file

Best regards,
Juan Carlos.

Average of ratings: -

Re: failed to start vpl jail server (SSL_CTX issue)

by Garth Brady - Tuesday, February 9, 2021, 8:37 PM

Sorry, continuing to have issues... I read the documentation but have not yet been successful.

Using the vpl installer I successfully received certificate files from let's encrypt and the .pem files exist in /etc/letsencrypt/live/myservername/. I added the paths the vpl .conf file:

SSL_CERT_FILE=/etc/letsencrypt/live/myservername/fullchain.pem
SSL_KEY_FILE=/etc/letsencrypt/live/myservername/privkey.pem

However, when I try to start vpl I get this error still:

vpl-jail-system[13570]: SSL_CTX_use_certificate_chain_file() fail: error:02001002:system library:fopen:No such file or directory

Average of ratings: -

Re: failed to start vpl jail server (SSL_CTX issue)

by Juan Carlos Rodríguez-del-Pino - Tuesday, February 9, 2021, 9:37 PM

Dear Garth,

please, set the LOGLEVEL to 8, try restarting the service, and show the log here.
I also will try to replay the problem.

Best regards,
Juan Carlos.

Average of ratings: -

Re: failed to start vpl jail server (SSL_CTX issue)

by Garth Brady - Wednesday, February 10, 2021, 12:13 AM

Here's the system control status log. Hopefully this helps!

Loaded: loaded (/etc/systemd/system/vpl-jail-system.service; enabled; vendor preset: enabled)
Active: failed (Result: protocol) since Tue 2021-02-09 15:56:19 UTC; 12min ago
Docs: https://github.com/jcrodriguez-dis/vpl-xmlrpc-jail
https://vpl.dis.ulpgc.es
Process: 1838 ExecStart=/etc/vpl/vpl-jail-system start (code=exited, status=0/SUCCESS)
Process: 2010 ExecStopPost=/etc/vpl/vpl-jail-system stop (code=exited, status=0/SUCCESS)

Feb 09 15:56:17 ip-removed systemd[1]: Starting VPL Jail System Service...
Feb 09 15:56:17 ip-removed vpl-jail-system[1838]: Starting vpl-jail-system service:
Feb 09 15:56:17 ip-removed vpl-jail-system[2008]: SSL_CTX_use_certificate_chain_file() fail: error:02001002:system library:fopen:No such file o> Feb 09 15:56:18 ip-removed vpl-jail-system[1838]: Daemon not started
Feb 09 15:56:18 ip-removed systemd[1]: vpl-jail-system.service: Can't open PID file /run/vpl-jail-server.pid (yet?) after start: Operation not > Feb 09 15:56:18 ip-removed vpl-jail-system[2010]: Stopping vpl-jail-system service:
Feb 09 15:56:19 ip-removed systemd[1]: vpl-jail-system.service: Failed with result 'protocol'.
Feb 09 15:56:19 ip-removed systemd[1]: Failed to start VPL Jail System Service.

Average of ratings: -

Re: failed to start vpl jail server (SSL_CTX issue)

by Juan Carlos Rodríguez-del-Pino - Wednesday, February 10, 2021, 2:15 AM

Sorry, I did not explain how to get the logs.
Please, next time follow the instructions at
https://vpl.dis.ulpgc.es/documentation/vpl-jail-system-2.6.0/troubleshooting.html#reviewing-logs

Anyway, the error message says that the file does not exist.
As expected, I have reproduced the problem on Ubuntu 20.04 when I remove/rename the file.

Please, check the files (file names that appear in the logs) exist and do not have stranger characters in their names.

Best regards,
Juan Carlos.

Average of ratings: -

Re: failed to start vpl jail server (SSL_CTX issue)

by Garth Brady - Wednesday, February 10, 2021, 4:46 AM

Thanks again for the help... I'm certainly stumped... Is it possibly a permissions issue since the files are clearly there? To double-check I'm not crazy, I copied the directory for each of the .pem files and then edited them using the command "sudo nano /etc/letsencrypt/live/my-domain/privkey.pem" (and also fullchain.pem)). Both commands executed correctly. I also browsed to them like so:

ubuntu@ip-myip:/etc/letsencrypt$ sudo ls live/my-domain
READMEcert.pem chain.pem fullchain.pem privkey.pem

Clearly, the files are there and match the directory structure I have in the vpl conf. To test if it was a permissions issue, I ran "sudo chmod -R 0755 /etc/letsencrypt/live", but the same file not found error came up on vpl start. Any ideas!?

Average of ratings: -

Re: failed to start vpl jail server (SSL_CTX issue)

by Garth Brady - Wednesday, February 10, 2021, 6:19 AM

In addition to my comments above, not only do the files appear to be in the correct location and the /conf file accurate, but the directory permissions (/etc/letsencrypt/live/my-domain) seem correct to me as well:
lrwxrwxrwx 1 root root 46 Feb 9 11:37 cert.pem -> ../../archive/my-domain/cert2.pem
lrwxrwxrwx 1 root root 47 Feb 9 11:37 chain.pem -> ../../archive/my-domain/chain2.pem
lrwxrwxrwx 1 root root 51 Feb 9 11:37 fullchain.pem -> ../../archive/my-domain/fullchain2.pem
lrwxrwxrwx 1 root root 49 Feb 9 11:37 privkey.pem -> ../../archive/my-domain/privkey2.pem

Here's the perms on the /archive:

drwxr-xr-x 2 root root 4096 Feb 9 11:37 ./

drwx------ 3 root root 4096 Feb 9 00:56 ../

-rw-r--r-- 1 root root 1850 Feb 9 00:56 cert1.pem

-rw-r--r-- 1 root root 1850 Feb 9 11:37 cert2.pem

-rw-r--r-- 1 root root 1586 Feb 9 00:56 chain1.pem

-rw-r--r-- 1 root root 1586 Feb 9 11:37 chain2.pem

-rw-r--r-- 1 root root 3436 Feb 9 00:56 fullchain1.pem

-rw-r--r-- 1 root root 3436 Feb 9 11:37 fullchain2.pem

-rw------- 1 root root 1708 Feb 9 00:56 privkey1.pem

-rw------- 1 root root 1704 Feb 9 11:37 privkey2.pem

Average of ratings: -

Re: failed to start vpl jail server (SSL_CTX issue)

by Juan Carlos Rodríguez-del-Pino - Wednesday, February 10, 2021, 7:12 PM

It seems difficult to spot the problem.

I tested the use of symbolic links with no success in reproducing the problem.

If you, as root, can "cat" the content of the file, Vpl also must can.

sudo cat /etc/letsencrypt/live/my-domain/cert.pem

I guess you are using the latest version of the vpl-jail-system, V2.6.0.

Best regards,

Juan Carlos.

Average of ratings: -

Re: failed to start vpl jail server (SSL_CTX issue)

by Garth Brady - Wednesday, February 10, 2021, 8:33 PM

That's what I presumed. I ran 'sudo cat /etc/letsencrypt/live/my-domain/cert.pem' and it outputted the contents of the cert. As you can see in my example above, I've tried creating the cert twice with no success

I'm having trouble determining next steps... I even tried copying the fullchain and privkey to a public directory and adjusting the conf to point there and they still won't load.

Any ideas?

Average of ratings: -

Re: failed to start vpl jail server (SSL_CTX issue)

by Garth Brady - Thursday, February 11, 2021, 5:52 AM

I installed a fresh copy of Ubuntu18.04 and then VPL and have the same experience with the same error message (02001002) as I did on 20.04. I've been looking on let's encrypt's forums and see a few people with the same error all related to nginx). I have yet to resolve the matter.

In /etc/letsencrypt/live/my-domain name, the domain was actually created like so: my-domain.tk-0001. Is it possible this -0001 is causing the issue? I named the path identically in the vpl-system.conf.

Average of ratings: -

Re: failed to start vpl jail server (SSL_CTX issue)

by Garth Brady - Thursday, February 11, 2021, 9:21 PM

I am fully confused at this point... But all of a sudden it's working, and I'm still not sure why. Here's what I did:

I copied both fullchain1.pem and privkey1.pem from /etc/letsencrypt/archive to /etc/vpl (so that I could play with permissions) and modded vpl-jail-system.conf to match.
I then chmod'd 755 both cert files in /etc/vpl/ and tried starting the service and got the same error. Since that still produced the same error (error:02001002:system library:fopen:No such file or directory) I chmod'd them back to 400 (still owned by root, root).
Out of frustration with certbot, I then re-ran the vpl installer, chose self-signed cert (NOT LETSENCRYPT) and vpl starts up without changing the .conf???!!!!

Question: The vpl-jail-system.conf file is still pointing to certbot's SSL fullchain1.pem and privkey1.pem files. Why did running a self-signed cert process through the vpl installer suddenly make things work correctly?

Average of ratings: -

Re: failed to start vpl jail server (SSL_CTX issue)

by Juan Carlos Rodríguez-del-Pino - Friday, February 12, 2021, 1:09 AM

Dear Garth,
congratulation on winning this battle, but I am disconcerted with this behavior.
I think that something is wrong with the access to the cert/key files. I need time to reproduce/find the problem and resolve it.

Best regards,
Juan Carlos.

Average of ratings: -

Re: failed to start vpl jail server (SSL_CTX issue)

by Garth Brady - Friday, February 12, 2021, 4:18 AM

Hi Juan,
To confuse matters even more, I reverted the .conf to point to the original symbolic links in the letsencrypt/live folder and vpl still starts. I am going to try to re-create the resolution to this issue on a separate server.

Average of ratings: -

Re: failed to start vpl jail server (SSL_CTX issue)

by Garth Brady - Friday, February 12, 2021, 5:38 AM

In addition to what I just posted, it seems that this issue can be entirely reproduced (I just did) on an EC2 instance (free tier) at Amazon on a Ubuntu 18/20 install. For some reason, after generating the Let's Encrypt certificate, the error described above occurs. However, re-running the vpl installer and selecting 'self-signed' certificate resolves the conflict (still not sure what it is). And, when browsing to https://yourJailserver.com/ok, the site reports that it is secured with a Let's Encrypt certificate.

Average of ratings: -

Re: failed to start vpl jail server (SSL_CTX issue)

by Juan Carlos Rodríguez-del-Pino - Friday, February 12, 2021, 9:07 PM

Thanks Garth.

Best regards,

Juan Carlos

Average of ratings: -

Virtual Programming Lab (plugin)

failed to start vpl jail server (SSL_CTX issue)