You only need the kernel sources if you are compiling modules for it ( Nvidia drivers or Vritualbox are good examples for things you need the kernel sources)

/lib/modules does not change because of loading or unloading modules, at all. That's only the place to look upon for modules to load, that's all. Loading a modules reads it from disk, that is /lib/modules, and loads it into memory. When unloading... you get the thing, right? ;)

As for the symlinks: you can have symlinks pointing nowhere.

You can check where a symlink is pointing to with

ls -l

Regards