I never considered S. fryeri to be agressive to be honest. When I kept it, it basically minded his own business most of the time (which was sitting in a relatively small corner of the tank), only to show up during feeding time. The male was not stressed or supressed by other fish - they didn't dare take him on
Ocassionally, he swam a lane or two, immediately to return again to his corner.
However, in the mating season, they can be very dominant, claiming a large territory (an entire 170 gallon tank, in my case) and will chase their tank mates all over the place.
Are they very young or (nearly) adult fish? That makes a difference of course. Also, when you replace one male with one or two females, you will probably see more action