Patriarchy isn't referring to men, it's referring to a collective mentality that men are supposed to act a certain way, enforce women to act a certain way, and holds the belief that men are above women, while degrading both women and men who don't meet the mentality's standards.