The problem is usually other software that you have no control over. Chromium, for example, will fail to build without setarch and finding every spot they incorrectly try to determine the architecture can be too tedious. Dpkg is rarely used because that will only work on debian derivatives.
In most cases, the best thing to do is to rely on the compiler defaults and provide configure flags where appropriate. Then things either just work and cause the least headaches.