In aggregate, your phone is not listening to you, but if you are of great interest to a powerful adversary, it very well might be. But at that point, I would wager that's one of the smaller things on your plate.