Actually, no. It's there for them to tell their story and in hindsight they might emerge from there about maybe a level or two higher than the party average by Chapter 6, but since it's that much at most, I find it negligible unless it becomes very noticeable. xD
But then their levels in Chapter 6 will be different to the ones in Chapter 5,and isn't 5x suppose to be based on events prior to Chapter 5?
Can't you make the enemies in 5x not give EXP?