There is probably has some memory-eating / inactive processes having many inactive pages. Linux kernel prefers to swap out inactive portions of programs to swap since most likely the memory is better used as cache.
If u wish to change this behaviour, try vm.swappiness in sysctl tunable but be careful , outsmarting the kernel developers / distro maintainers is not that easy way. Also try swapon / swapoff