I do tend to agree with you about the Wye. I was at Symonds Yat in the summer and it was crawling with canoeists, with a hire outfit set up on the bankside too. Quite a different prospect to e.g. the Ribble, where you only seems to get the odd one floating by.
I guess the answer is only fish scary rivers with plenty of difficult sections, in bad weather! - Benign ones attract too many fine weather/easy riders.