As usual, consuming things in good measures won't do much harm.
It's rather a problem when you take it in daily and in higher doses.
Soy beans comtain genistein and daidzein. These are so similar to estrogen that they are categorized as phytoestrogens. Which is a plant produced estrogen.
So of course if you take them in big amounts you might risk collecting some unwanted fat layers and maybe even risk growing some small man-boobs.
And it also obviously depends from what region of the earth you come. Asian people consume way more soy products than we do, and are mostly fairly sleek. And not every asian man is carrying manboobs. Having had soybeans in their diet over the course of many centuries their bodies usually digest it better and they are also more adapted.
Soymilk is a very concentrated product so rather could inflict said phenomena when taken in considerable amounts over a longer period. Whilst adding some soybeans to a dish won't grow you instant man boobs.