Hi there,
This kind of relates to the general post, but I am replying to this specific post since it relates most to my problem.
I thought I would write a bit about my findings for the next person coming to this thread since I had a rough time figuring out what the actual problem was causing "Cannot obtain task lock". So here it goes.
Moodle runs crons in the background which are usually long running processes that can potentially overlap each other. Meaning before it's finished running something is can start running that same thing again ... e.g. backups. This obviously is not ideal and moodle has come to the table with a great solution - locks. It took a while to figure out and understand these locks, but moodle has a couple of different ways of handling locks. The default uses the files system. It has a set of files in moodledirectory/locks that are used to keep track of what is locked or what isn't locked. If you are interested in seeing how things fit together check out lib/classes/lock directory. All the files are in there. Another important file to check out is admin/tool/task/cli/schedule_task.php
Anyway the main gist and problem of the file locking mechanism seems like using flock (php function to lock files found in lib/classes/lock/file_lock_factory.php function get_lock) is that it's brittle. I'm not sure why they would use something that's not great. If you look at the documentation they say "On windows, a second attempt to get a lock will block indefinitely instead of timing out.". I am not using windows, but I think the same thing is happening for us. The great thing is you can swap out locking mechanisms. You can use the database or in memory storage like memcache (I think this requires a plugin).
All you need to do to swap over to the database is update your config with
$CFG->lock_factory = "\\core\\lock\\db_record_lock_factory";
This obviously comes with the cost of hitting your database, but for us we would much rather know that it's working.
If you want to check what's locked you can check the table mdl_lock_db
After changing over to this everything started working.
Some references
https://docs.moodle.org/28/en/Administration_via_command_line#Running_cron_via_command_line
https://tracker.moodle.org/browse/MDL-11309
https://tracker.moodle.org/browse/MDL-25500
Some tips
If you ever find yourself in this situation, dive into the code and drop some echo statements or var_dumps to find out what is actually happening. Disclaimer - don't do this on production and make sure you delete any traces once your are done.