Thanks SYSTEM! So webfork was right that it happened about 2 years back, but I couldn't find the topic because it was buried in the "New designs for TPFC by tproli" thread.
Since the problem has recurred, I think it's good we have a dedicated topic for it. After re-reading the messages in 2015
, I am surprised the symptoms and analysis are the same. I suspect this is caused by some kind of unidentified, rare, race condition in the code somewhere.
I need to think about how to deal with this, at the very least, put in additional logs so that if it occurs again, we have more info to work on next time.
If you guys have any idea or suggestions, do lemme know. Thanks!