I’m using segment-polygon tests and they are definitely colliding on both sides of the polygon.
I found a much simpler solution that works for me by just counting it as visible if there’s only one collision. There’s no way a collision can pass through two one-way collisions so it works out fine in practice.