Hmmm ... Yah, it sounds like you are really aiming high here. while your application is in the development and alpha and beta states, you might be able to get away with a single box, but Rye is definitely right, you should consider more than one mahcine and delegation of roles / duties between them.
I feel I should warn you that you need to think about security from the very beginning. Especially if you are going to be either directly processing payments, or at least brokering the transactions. If any personaly identifiable data or financial data is going to be stored in your application, or handled / processed by your application, you will at the bare minimum have some regulatory concerns to keep in mind.
As for the "unlimitted number of subdomains", this is trivial if you control your own DNS (BIND is still the gold standard) and the web server. You can fairly trvially automate the process of creating new records in DNS along with creating new virtual hosts in your web server.
As far as messaging goes, if your only concern is "internal" messaging, with no *real* email interface, that should be simple enough to implement via your database and integrate directly into the web app. DB based mail storage is often preferable, too, if messages tend to be distributed to multiple users, since you won't have to store multiple coppies of the same message, but rather just provide links in the DB (e.g. join tables). If you do want to integrate real email, then it's simple enough to "weld" that onto the existing internal / web app framework. Implmenting virus checking and spam filtering on an internal mail system might be trickier, if the messages never hit an "external" MTA, but it's not impossible I would still look at the APIs for ClamAV and SpamAssassin, if I were you.
Really, though, if there's anything I can tell you, it's to focuss on building your app securely from the ground up. ALWAYS sanitize your user input, do not use user input to directly build SQL querries, always expect unexepected input and detroy it / throw vague errors (to the user, log usefull error on the server), never try to implement your own encryption, that bears repeating: NEVER try to implment your own encryption - use an accepted / tested / proven standard.
and, good luck. :)