Are you using the factory roller lifter or aftermarket? The factory ones are as stout as any brand name. Theoretically those should not cause any geometry issues from being too tall.
Get a checking pushrod and order custom ones to ensure proper lift and the best possible geometry.