Why Ben Affleck hasn't harassed Drew Barrymore or Reese Witherspoon.

He knows that he can't get away with it.  Besides, what fun is there in harassing another bully?