I think of replacing my aging switch witha RB1100 for a cluster of up to 6 servers (well, 6, but one does not need to be on the switch).
The Rb1100 would allow me to run 5 ports for the front side, and the other 5 for the server backend.
Now, there is sometimes a LOT of traffic going there. as in: When I transfer files etc., I need full 1gbit between 2 ports. While another 2 ports may do another transfer.
Possible?
Since you have in essence two 5-port gigabit switches connected together with 1 Gbit, on a RB1100 you can not do that.
Switchgroup A <—> Switchgroup B will always only share 1 Gbit.
So, All 5 servers together will share only 1 Gbit towards the “front”.
The five ports within each Switchgroup however can push 5 Gbit among themselves.
So all 5 servers can each use 1 full gbit towards other servers on the same Switchgroup.
The second scenario would pretty much be it. If I keep the 5 servers within one group, I can actually use full performance between them, correct? 
Given the small amount of servers this would allow me to use 8 ports as “router” and 5 groups as “fast inter server backbone”.
So, the performance of a switch group is fast enough to actually real handle full throughput?
Correct, you can use full performance between the 5 servers on the same switchgroup - just remember to configure switching for the interfaces! Out of the box, the Ports work as standard routed interfaces.
The other 8 ports can be used individually (keeping in mind that the second switchgroup also shares 1 gbit to the other ports)