This is also true when it comes to speaking. Taking advantage of many opportunities to speak (whether in a formally organized group like Toastmasters International, or in an informal workplace-based group) does indeed offer the opportunity to hone, to refine, to perfect. But you need to know what it is you are aiming for, what habits of yours need correcting, what new skills you need to acquire. The problem with peer-to-peer groups is that they may help you become more aware of what needs fixing, but they don't always offer an effective way to fix it. Often, someone will energetically advise you to "try this; it worked for me." But using tactics that worked for them may not help you at all. And the anxiety produced by this sense of failure can tie you up in even more knots.
As a speaker trainer/presentation skills consultant I spend a lot of time undoing those knots. I work with people who have been beating themselves up for years because they need to use notes, or have to let speeches "marinate" before they are fully formed. All because some "expert" (who is good at doing this himself but has no idea how to teach) proclaimed "only losers need notes" or "what's the big deal? You know your stuff, just get up there and talk about it." You can guess what I say to that! I have blogged about these issues, here and here and probably several other places as well.
If glossophobia.com is to be believed, as much as 75% of the population suffers from a fear of public speaking. So whoever is offering you advice likely had his own issues to deal with. And whatever they were, they were overcome. So hurray for him! But here's the thing--your issues are not the same. Following the same methods for fixing them is like taking someone else's pills. They might work--or they might disastrously backfire. You need your own prescription.
In speaking, as in baseball, you need to train with a coach, not just other players on the team. Sure, you need to show up on the field and practice pitching, batting, and catching, but your technique won't improve with practice alone. You need to work on the skills with someone who can recognize what you are doing wrong and offer you exercises, techniques and strategies for improvement. Then you can go out and practice. And practice, practice, practice. Ask your coach for continued help as you progress. She'll make adjustments and you'll keep working.
Stay strong! Your peers, most of whom mean well, will weigh in. Take their feedback as just that, not instruction or advice. They aren't up at bat with you, they're just watching from the bench. So smile and thank them. And then go do what you were trained to do!