More or less. A couple of caveats. If the serfs build a merry men type system where they oppress the feudal lords that would still essentially be feudalism. Because it describes the system not the individuals. So it still wouldn’t really be reversed.
Where it get cumbersome with racism (or sexism) is that the “power” refers more to the power structure than it does to the individual (which is why I said conflating then gets confusing).
The easiest way to see it is probably with systemic sexism. We live is a system which specifically maintains its power structure in a way that generally lowers the station of women. But the same power structure also affect men. Usually this is in men’s benefit. But sometimes it’s not. For instance in a divorce we tend to side on awarding custody to the mother over the father. This is an aspect of a systemic enforcement of women as caregivers for children. But it still adversely affects men. Similarly the placement of women as the gatekeepers of sexuality forces men into a role of being the ones to pursue relationships. Stuff like that.