This seems an obvious reading of Matt 1:19, but a text to consider is the Infancy Gospel of James. Here Joseph is an old man asked by the priests to care for a 12-year-old girl Mary, who’d been raised in the Temple.

And Joseph refused, saying: I have sons, and I am an old man, but she is a girl: lest I became a laughing-stock to the children of Israel.

I’ve wondered if this might dovetail with the gospel narratives. Joseph does just disappear from them, never being quoted, suggesting he might be an older Jewish man who wasn’t too spiritually illuminated. (The ‘brothers of Jesus’ might then be Joseph’s children from a previous wife.) An alternate motivation for Joseph may then be: he feared being embarrassed.