Gotta lean hard in the other direction to flip the gaslighting onto him. If he says he paid for your college then start insisting you never went to college. If he says you owe him for keeping your car running during X-Y years then tell him you didn’t even own a car.
More seriously, though, my mom was the same way. Hers was due to being extremely manipulative instead of memory issues. Either way you aren’t going to get through. Better to start distancing yourself now until you’ve reached a level of contact that isn’t detrimental to you. If that eventually means no contact then so be it; they bring it on themselves and you owe them nothing.
Yeah, my work has an anti-union section of the employee handbook. It says they are proud to provide such good benefits that a union isn’t needed (definitely not true). And how we need to talk to HR first if we ever feel like our working conditions would warrant one. I suspect you wouldn’t be allowed back to your desk if you did that.