Is picking fruit from other towns bushes or fruit trees against the rules?

(Assume there is no sign in the town that says something to the effect of "Feel free to pick the fruit" or "Please don't pick the fruit".)