You brought up a very good point, something i was trying to explain to our anglers as well. For example you destroy a good deal of natural environment and have big influence on for example ocean ecosystems to catch source of protein for raising trout, you pollute rivers with outflow from fish farms to raise some 1-5kg "trophy", which is then stocked in wild water, where it has negative impacts on wild fish, most probably in C&R section, where it either dies from fish damage, starvation or in flood, whatever comes first and all that damage to the environment went nowhere. And next year, we do it again. Just so that there is enough of easy to catch fish in water.
Regarding taking of wild fish, natural mortality in trout waters can go as high as 65%, normaly it is qouted between 30-65%. So very often the anglers harvest is neglegible compared to this.
In my own club we had a rule about taking either 1 brown trout or 1 graylling per fishing day for some years now (ad 3 salmonidae in general, so oters must be rainbows). Otherwise we are allowed to take 3 salmonidae per day and in the past there was no limitation for grayling or brownie. I have suggested to do that again. From very simple reason. There is no change in population of brownies and graylings, at least for brownies we could say it went down. But not becouse of anglers harvest which is more than obvious neglegible compared to other couses, for example domination of wild rainbows, and huge amounts of stockies. There are many more factors that influence wild fish populations, but anglers normaly just see their harvest. Unfortunately it is very often used to spread some fog over other facts and as excuse to hide other stuff like stocking policy and its influence or introduction of competitive species etc.
I have pasted this link before, i shall do it again. There are many more articles about this if you search for those , but this is an interview and is therefore a more interesting read:
Why Montana Went Wild