[If] he deferred to a minor until he becomes of age, or to the eldest until he arrives from overseas, or to a deaf-mute or to an idiot, we do not listen to him; but we say to him: “The mitzvah is [incumbent] upon you —- perform either chalitzah or yibum!”