Just saw a TSB for an issue like this. T-SB-0220-17. Your Rav is not on it, but they may not know all the effected vehicles yet.
If the latch switch gets hit by the nozzle or the splash shield and trips it, an automatic filler vapor block is activated. At that point you can't fill it any more.
The fix is a shield for the switch, but you can experiment to see if that is what is happening.
TSB fuel door MC-10161532-9999.pdf