It depends on what you want to do with all those I2S channels.
I'd synchronise the Pi's with just one signal, possibly from a master, and transfer the data over Ethernet or USB, using a host to re-assemble/partition the channels - I would not even try to transfer the data between the Pi's over GPIO (basically same speed/complexity issue as I2S bit banging).
The idea of DSP's is attractive, but then begs the question - how will you get all that data to/from the DSP's fast enough?
1) If this is a personal project, dive in. It will be a ton of work, frustrating at times, but you will learn a lot. You may not get it running.
2) If this is a commercial project, especially if it has less than a year or two to get done, put together a comprehensive specification of what it needs to do, and hire an experienced consultant to decide if it is feasible, and come up with some approaches. Do not try to dictate the tools to someone more experienced. Expect to pay a lot.
Frankly I run across similar issues often.
The price and popularity of the Pi make people want to use it for applications it was not designed for, even if there are better alternatives - and as they are not in the industry, they are not aware of alternatives.
I am NOT trying to discourage you - if this is a personal project, I actually encourage you to try, as I guarantee that you will learn a lot by trying.
The problem I see with using 4 pi's is that I'd have to cross-link them with the remaining GPIO pins to each other pi, which has limited scalability and might be even less possible to run reliably than a single pi with all the IO used, at that point I might as well just stick with dedicated DSP chips.
Though it may be possible to hit it half-way with a single pi and multiple DSP chips...