Since I have a Pi in-hand, it might make sense for me to look at the installer stuff. I have no experience with the traditional Debian installer, so it will take some research on my end.
One thought - we have to have the firmware on the SD card for the Pi to even boot into the installer. So the installer setup will be a little different then say, installing from a LiveCD.
I am thinking along the lines of the following:
-The installer be in a 2GB SD card image, just like the pre-installed Debian images
-The image have 2 partitions, the /boot for the firmware, and a 256MB partition with the installer (this is about the size of the Debian netinst images, plus some room to add specific packages for the Pi)
-Once the installer is running it partitions the remainder of the SD card as desired by the user
-The installer installs a basic system from its image and downloads any additional packages at install time (netinst)
-At the end of installation, or at first run, the 256MB installer partition is deleted and replaced by /swap
An alternate implementation would be to put just the firmware/installer image on the SD card and have a package library on a USB key, in which case a 4GB image could be used with more packages available without going out to the network....