Ram is like real food and swap is like army rations. You only eat the army rations when you run out of real food, and when you have to eat army rations, life sucks. Whatever solution you come up with, using swap is probably not what you want to do.

What virtualization system are you using? Vmware? Xen? KVM? What other things is the virtualization host doing? How many other VMs are you running? How are they behaving?

What kind of HDD setup do you have? If the host/dom0/metal just has a regular disk, and your'e sharing it among multiple VMs including this IO intensive one, that's probably your bottleneck. Is the host OS using lots of swap? I could imagine something like you're describing if the host OS is thrashing but the guest claims to have free ram.

What are you doing that's using so much network? You may have to figure out a better way to make everyone play nicely. Do you know for a fact that things only go bad when multiple hosts do these big transfers, or is it just a slow creeping death?

Really, as I'm typing this, it smells like you have a memory leak that you either need to plug or restart the responsible process automatically. That should be easy to set up. Do you know what processes are eating the ram? You should find out.

Hi all. I'm a linux sysadmin and I'm trying to find good utilities to help me automate some of the tasks I do a lot. There are already good utils for things like log rotation, but there are countless tasks that could be similarly automated. I recently noticed one called "certwatch" that keeps an eye on the expiration date of your SSL certs and gives you a warning before they expire.

Anyway, here are some tasks that I'd like my servers to do on their own:

[ul]
[li]restart important processes if they die (httpd,mysqld etc...)[/li]
[li]re-connect to the network if it gets lost for some reason (like stupid me accidentally disables it while I'm offsite...)[/li]
[li]delete old backup files if disk gets too full[/li]
[li]restart processes that hold large deleted files open (run lsof +L1 to see what I'm talking about)[/li]
[li]alert me if I left something that's filling up disk really fast (tcpdump -w or apache rewrite logs)[/li]
[/ul]

Does anyone know of any utilities that might help me make my servers do some of that for me? Or, heaven forbid, something that tries to do the whole shooting match?

Thanks!
-Dylan

What happens if you don't redirect the output to a file? Does it look right then? What about if you log onto <host> and run /mnt/monitor.sh. Does it look right when you're not piping through ssh? Try and strip away as much as you can and you'll probably find where the problem lies.