First:
BCM2708 is the SoC family and BCM2835 is the individual. Up until Pi2, it was the only one in the family, but now we have BCM2836 as well.
When building the Linux kernel, 2 main choices have to be made: cpu architecture (ARCH=arm) and board/platform (ARCH_BCM2835).
ARCH_BCM2708 is the kernel platform code that Broadcom made. This is the core code that makes it possible to boot Linux on the BCM2835 SoC. This code could not go into mainline Linux, because at that time (Pi arrival) new ARM boards was required to use Device Tree which ARCH_BCM2708 didn't.
I'm not familiar with the early work, but Pi support ended up in mainline as
ARCH_BCM2835
The problem now is that we have to carry a set of patches between kernel versions to support ARCH_BCM2708, while there is Pi1 support in mainline. So if we could use ARCH_BCM2835, we could drop some patches, and if all Pi specific drivers where in mainline as well, even less work had to be spent on adapting and testing patches for each new kernel version.
And since mainline uses Device Tree, none of the Pi specific drivers in mainline could be used.
In July last year I added Device Tree support to ARCH_BCM2708 and later that year Phil Elwell added support so we could build a kernel that could boot the old ATAGS way and the new Device Tree way. This made it possible to move to Device Tree without to much disruption, and we could start to use the drivers that where already in mainline. But as long as ATAGS booting is supported, some old drivers still have to be carried along.
So at this point we can use most of the Pi drivers in mainline with ARCH_BCM2708, but we still have to carry many patches.
The next step was to prepare a move from ARCH_BCM2708 to ARCH_BCM2835 which involved changing many of the downstream only Pi drivers.
In 4.1 all downstream drivers are updated except for lirc_rpi.
The next step is to drop ATAGS, but there's no schedule yet. ARCH_BCM2835 doesn't support ATAGS booting.
So, benefits with ARCH_BCM2835:
Don't have to carry alot of patches.
The code have been reviewed by the some of the best kernel programmers.
If there are changes to internal kernel interfaces, Pi code in mainline gets updated.
When mainline gets Pi2 support (work in progress), both Pi1 and Pi2 can be booted with one kernel image (instead of kernel.img and kernel7.img).
Many kernel build projects (Yocto, buildroot) prefer to use mainline.
So should you use ARCH_BCM2835?
In production systems, no. But I really would like to see some wide scale testing, so we can fix up the missing pieces.
Ref:
https://github.com/raspberrypi/linux/wiki/Upstreaming