I'm thinking the same. Thing is, I'm not being so painful about this for nothing. I think that most people overestimate the size of exhaust that they need. As with most tuning not getting greedy can help. Sure a slight small diameter might rob me of 5BHP when it's running a higher lift cam, but I assume that it will help keep the wodge of torque lower down.
I've read up on this back pressure and torque thing I can see how it all works. Essentially, going too large makes the gas speed too low in the system and it is less good at pulling the charge out of the manifold.