My wife and I just bought a 05 Highlander std 2WD V6 less than a month ago. Coming from a 99 VW Passat wagon I'll say the Highlander is noisier. Based on recommendations from the Lexus RX330 forum, today I removed the roof rack cross bars in attempt to lower the wind noise. It doesn't sound like this is what you're hearing but you may want to give it a try. Takes about 15mins and is easily reversible if you need them.
Can't give any feedback on weather it worked or not (still sitting in driveway) but personally I think it looks better without them.
Good luck.
Mike