Do you know which bit of the code starts to fail at higher opt levels? The toolchain supplied is a completely standard gcc Arm compiler, so although there may be corner cases where -O3 gives problems I would not expect any issues for the vast majority of code.
There are notably differences in debug vs release builds also, for example, allocated memory is not cleared on release builds but might be on debug builds. Do you have uninitialised data somewhere? That a common culprit.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Please direct all questions to the forum, I do not do support via PM.