Hey Guys,
Just to update everyone, I figured the issue. It was pretty silly of me.
The moodle data directory which I was using was being accessed through LAN and was mounted on my VM.
Apparently that was slowing things down as even cache was maintained in it. I tried using memcache but no great results.
I am now thinking to switching to NFS using an instance of VM.
Will keep you guys posted.