Aluminum Intakes dont seal on X heads

I have tried three different new manifolds and the would not seal at the bottom of the port. They pulled oil in an big vaccum leak. I have also had the heads off and I am not sure if I swapped them side to side or not. The cast manifold is the only manifold that will run without problems.