Just to keep everyone posted, here are the updates on the hosting issue so far:
To recap, TPFC was running on a PS on DH with Apache proxying to my private copy of lighttpd. As you can see, this is not the happiest of situation because Apache takes up a lot of memory and does nothing much. It would be much better if the PS runs lighttpd on port 80 directly.
It so happens that DH was beta-testing lighttpd on PS, so I decided to order a new PS to try it out (I didn't want to break the existing PS by switching it over to lighttpd, 'cos I did some research and apparently some folks had major breakages switching an existing Apache PS to lighttpd).
The first problem came with migrating the domain to the new PS/lighttpd. It didn't work and I got a script error which I had to contact support. Thankfully that was only a test subdomain, so it didn't break TPFC. Once support sorted it out, I was able to get the test domain running in no time.
After a day or so of testing, everything seems to work, so I decided to mirror the main domain to PS/lighttpd (I didn't use the migration script because I know the script wasn't fixed yet, so I created another subdomain - which triggered the script error; got support to fix it - and mirror TPFC to it). Once the DNS propagated, everything seemed to work, and I was happy. The memory usage was fantastically low and stayed at around the 100MB mark. Response was fast too. So I went to bed.
The next morning, I checked and lo-and-behold, I got the "500 Internal server error". I restarted the server, and it worked for 8-10 hours before getting stuck again. I did some research and it seems older versions of lighttpd had this issue when the load gets too high with the PHP backends, and it is
unable to recover gracefully. I worked with support for a couple of days to tweak the lighttpd conf parameters but was unable to resolve the issue.
So I had no choice but to revert back to Apache/lighttpd. I am not sure why my own config works. It could be the version of lighttpd or something in the config file, but I became too discouraged to find out, since this back-and-forth cycle between contacting support and getting things fixed was getting too long and tiring. The workflow is fine when things more-or-less works, but when stuff are broken, it is really painful when you don't have root access. The DH web panel is a nice frontend that hides all the ugly *nix commands, but is useless when things don't work and you need bare-metal access.
When this realization dawned on me, I finally decided to move on to a real VPS with root access. A couple of days back, I ordered a VPS from
vpslink.com (note: link contains my referral code 6FP20Z, which gives you 10% lifetime discount and me a one-time service credit) and spending most of my spare time now configuring it. Hopefully I can get it to production level soon, and then I will cut TPFC over to it.
Cheers!