The smaller watches throughout history like the datejust/tank were worn by what I suppose you could call "manly" guys so I wouldnt worry too much about that! But honestly if it feels better and you're happy who cares what it might be/look like :p Plus from what I've said the shape of the case at 35mm would suit more wrists

I think if you change your perspective from feminine to classic or sophisticated it can help. Think 35/36/37 is a sweet spot for guys trying to get the classic look