This topic is quite old so likely wasting my time answering this one.
I had the same issue earlier this year and resolved by putting in a new steering rack, new front-end alignment and balance.
Be sure to go to a Toyota mechanic if possible b/c some shops don't have the right equipment to do alignments and balances and try to fake it. I experienced a dealership here in Colorado who tried to tell me my whole front end needed replacing. BS. They were trying to balance my wheels on a balance machine built for a car. Be careful.
Runner.