In search of a small simple GNU/Linux server setup, I started with a Debian 7 installation through a network based install - netinst. Using that image is simple, either by writing to a CD, or simply to a USB drive or memory card:
(Replace X with your flash drive, but be careful; everything will be overwritten, without any recovery option).

sudo dd if=debian-7.8.0-i386-netinst.iso.torrent of=/dev/sdX
The installation was straight forward, but it has to be hand-held since there are multiple prompts from various parts of the installation throughout. Unfortunately, the final step of writing out the GRUB configuration failed, since the install medium, the USB flash reader, was included in the GRUB device map. Removing it from /boot/grub/device.map fixed that, and a little rescue operation resolve the rest.

Once booted, there was a problem with the start-stop-daemon; for some reason, it was set to a fake mock implementation. That caused all services to not start. Swapping in with the real implementation took care of that:

mv /sbin/start-stop-daemon /sbin/start-stop-daemon.FAKE
ln -s /sbin/start-stop-daemon.REAL /sbin/start-stop-daemon

Finally, some essentials are always missing:

apt-get install emacs atop htop iftop iotop tree git tig sudo autossh iptables-persistent wpasupplicant cryptsetup smartmontools