Linux.com

Home Linux Community Community Blogs Business (or Enterprise)

Community Blogs



HOWTO - Using rsync to move a mountain of data

In this installment of my blog, I want to document the proper use of rsync for folks who are tasked with moving a large amount of data.  I'll even show you a few things you can do from the command line interface to extend the built-in capability of rsync using a little bash-scripting trickery.

I use rsync to migrate Oracle databases between servers at least a few times per year.  In a snap, its one of the easiest ways to clone a database from a Production server to a Pre-Production/Development server or even a Virtual Machine.  You don't have to have a fancy Fibre-Channel or iSCSI storage array attached to both servers, in order to do a data LUN clone, thanks to rsync.

I hope you enjoy this in-depth article.  Please feel free to comment: if you need clarification, find it useful, or something I wrote is just plain wrong.

Read more... Comment (3)
 

Clone a Virtual Machine from the shell (The Script)

After few comments on my previous blog related on how to manually clone a Virtual Machine from the shell I've decided to write a simple script to do everything automatically. Maybe this could be useful for newbies but basically it reproduces all the information reported on my latest blog.

There's no rocket science here and I've tried to keep the script simple and hackable for anyone, it required me some time (less than 1h) due to my poor sed knowledge, I've taken it as an exercise to improve my sed capabilities.

As in open source feel free to improve or modify it as you wish, send me an updated copy so I can publish your best version as well, error checking it's quite simple now. You may input absolute or relative paths but there're few limitations around.

 

Basic Usage:

VMCopy <old name> <new name>
oldname is the name of the directory with original VMWare files
newname
is the name of the directory with newly created VMWare files

simple, isn't it ?

 

Here's the script:

#!/bin/bash
#
# @name VMCopy - Copy/Clone a VMWARE Virtual machine with a new name
#
# @author Andrea Benini (Ben)
# @since 2011-02
# @website http://www.linux.com
# @email andrea benini (at domain name) gmail [DoT] com
# @package Use it to get a physical copy of an existing machine, no snapshots or
# VMWare tools involved in this operation, it's a plain text bash script
# @require This tool should be portable to many UNIX platforms, it just requires:
# sed, dirname, basename, md5sum, $RANDOM (shell variable) and few more
# shell builtins commands
#
# @license GPL v2 AND The Beer-ware License
# See GPL details from http://www.gnu.org/licenses/gpl-2.0.html
# "THE BEER-WARE LICENSE" (Revision 43)
# Andrea Benini wrote this file. As long as you retain this notice you
# can do whatever you want with this stuff. If you make modification on
# the file please leave author notes on it, if you improve/alter/modify
# it please send me an updated copy by email. If we meet some day, and
# you think this stuff is worth it, you can buy me a beer in return.
# Andrea Benini
#
SOURCEPATH=$(dirname "$1")
TARGETPATH=$(dirname "$2")
SOURCEMACHINE=$(basename "$1")
TARGETMACHINE=$(basename "$2")

if [[ $# -ne 2 ]]; then
echo -e "$0 "
echo -e " Copies a VMWare virtual machine"
echo " and are names"
echo " of the machine you'd like to copy and the new destination name"
echo ""
exit
fi

exec 2> /dev/null
echo "VMCopy - VMWare Virtual Machines cloner"
echo " - Copying source machine '$SOURCEMACHINE' with the new name '$TARGETMACHINE'..."
rm -rf "$TARGETPATH/$TARGETMACHINE"
cp -R "$SOURCEPATH/$SOURCEMACHINE" "$TARGETPATH/$TARGETMACHINE"

echo " - Removing unnecessary files for '$TARGETMACHINE'"
rm -f "$TARGETPATH/$TARGETMACHINE"/*.log

echo " - Renaming files for '$TARGETMACHINE'"
for OLDNAME in "$TARGETPATH/$TARGETMACHINE"/*; do
NEWNAME=${OLDNAME/$SOURCEMACHINE/$TARGETMACHINE}
mv -f "$OLDNAME" "$NEWNAME"
done

echo " - Remapping Hard Disks for the new machine"
ls "$TARGETPATH/$TARGETMACHINE"/*.vmdk | grep -v -e "-s....vmdk" | while read DISKNAME; do
sed -i "s/$SOURCEMACHINE/$TARGETMACHINE/g" "${DISKNAME}"
done

echo " - Changing resource files (if any)"
if [[ -f "$TARGETPATH/$TARGETMACHINE/$TARGETMACHINE.vmxf" ]]; then
sed -i "s/$SOURCEMACHINE/$TARGETMACHINE/g" "$TARGETPATH/$TARGETMACHINE/$TARGETMACHINE.vmxf"
fi

echo " - Changing $TARGETMACHINE.vmx file"
# Massive character substitutions
sed -i "s/$SOURCEMACHINE/$TARGETMACHINE/g" "$TARGETPATH/$TARGETMACHINE/$TARGETMACHINE.vmx"
# Change ethernet mac addresses
MACADDRESSES=`cat "$TARGETPATH/$TARGETMACHINE/$TARGETMACHINE.vmx"|grep "generatedAddress ="| sed -e "s/.*=."//" -e "s/"//"`
REGEXP="[0-9 A-Z a-z][0-9 A-Z a-z]"
for OLDMAC in $MACADDRESSES; do
NEWMAC=$(echo $RANDOM$RANDOM |md5sum| sed -r 's/(..)/1:/g; s/^(.{17}).*$/1/;')
sed -i "s/$OLDMAC/$NEWMAC/" "$TARGETPATH/$TARGETMACHINE/$TARGETMACHINE.vmx"
done

echo -e " - Operation Complete, '$TARGETMACHINE' cloned successfully"

 

 

Share your ideas

If you find errors or you'd like to change some parts let me know, share your ideas to improve the script, I'll always post here the improved version

 

 

Manually clone a VMWare Virtual machine from the shell

Introduction

Sometimes you've VMWare appliances and you need to get a physical copy instantly and you don't have VMWare Tools with you or you're doing everything from command line (on a remote console), sometimes you don't even have VMWare (ESX/GSX/VSphere/player) installed or you've just the Player (no cloning from there) but you still need to get a clone of a working machine. I usually create my own appliances with my own utilities, packages and tools installed, I store them as .TAR.GZ and I use them as a base for new machines. Here's what I do to have an exact copy of a machine; it's not a geek trick, it's just a plain basic task, this always works, no matter about the OS inside your VM (Win/Linux/BSD/Plan9/BeOS/...).

 

First: of all you need to do is stop your source machine (in my example “Debian 6”) and locate its directory, then copy the whole source Dir to a new path (in my example “new.machine”)

$ cp -R Debian 6   new.machine
$ ls -la new.machine/
total 534520
drwxr-xr-x 2 ben ben 4096 2011-02-09 09:53 .
drwxr-xr-x 12 ben ben 4096 2011-02-09 09:53 ..
-rw------- 1 ben ben 8684 2011-02-09 09:53 Debian 6.nvram
-rw------- 1 ben ben 211550208 2011-02-09 09:53 Debian 6-s001.vmdk
-rw------- 1 ben ben 234356736 2011-02-09 09:53 Debian 6-s002.vmdk
-rw------- 1 ben ben 107347968 2011-02-09 09:53 Debian 6-s003.vmdk
-rw------- 1 ben ben 2621440 2011-02-09 09:53 Debian 6-s004.vmdk
-rw------- 1 ben ben 65536 2011-02-09 09:53 Debian 6-s005.vmdk
-rw------- 1 ben ben 639 2011-02-09 09:53 Debian 6.vmdk
-rw-r--r-- 1 ben ben 0 2011-02-09 09:53 Debian 6.vmsd
-rwxr-xr-x 1 ben ben 1652 2011-02-09 09:53 Debian 6.vmx
-rw-r--r-- 1 ben ben 263 2011-02-09 09:53 Debian 6.vmxf
-rw-r--r-- 1 ben ben 88558 2011-02-09 09:53 vmware-0.log
-rw-r--r-- 1 ben ben 49667 2011-02-09 09:53 vmware-1.log
-rw-r--r-- 1 ben ben 64331 2011-02-09 09:53 vmware-2.log
-rw-r--r-- 1 ben ben 63492 2011-02-09 09:53 vmware.log

Now delete unnecessary files like the logs

$ rm *.log

Do a massive rename, source/previous virtual machine was named “Debian 6”, you need to replace it with “new.machine” (our new name)

$ mv "Debian 6.nvram" new.machine.nvram
$ mv "Debian 6-s001.vmdk" new.machine-s001.vmdk
$ mv "Debian 6-s002.vmdk" new.machine-s002.vmdk
$ mv "Debian 6-s003.vmdk" new.machine-s003.vmdk
$ mv "Debian 6-s004.vmdk" new.machine-s004.vmdk
$ mv "Debian 6-s005.vmdk" new.machine-s005.vmdk
$ mv "Debian 6.vmdk" new.machine.vmdk
$ mv "Debian 6.vmsd" new.machine.vmsd
$ mv "Debian 6.vmx" new.machine.vmx
$ mv "Debian 6.vmxf" new.machine.vmxf

NOTE: .vmxf file is present on newer releases of VMWare appliances, if you don't have it just ignore it

Now it's time to change information inside your virtual machines, you just need to use your favorite text editor to change few things, keep this files as they're

new.machine-s*
new.machine.nvram
new.machine.vmsd

NVRam is your bios/nvram, it's a binary file and you don't need to change it, *.vmdk are your disks, you just need to change the information header of the disk (new.machine.vmdk), leave the other VMDK files as they are (new.machine-s*.vmdk); VMSD file is usually empty, don't need to change it.

 

Modify your hard disks

If you've more than one hard disk you've more than one .VMDK master file, you need to apply few mods on it, here's the content of the original file (was “Debian 6.vmdk”, now “new.machine.vmdk”)

# Disk DescriptorFile
version=1
encoding="UTF-8"
CID=71ad0a67
parentCID=ffffffff
isNativeSnapshot="no"
createType="twoGbMaxExtentSparse"

# Extent description
RW 4192256 SPARSE "Debian 6-s001.vmdk"
RW 4192256 SPARSE "Debian 6-s002.vmdk"
RW 4192256 SPARSE "Debian 6-s003.vmdk"
RW 4192256 SPARSE "Debian 6-s004.vmdk"
RW 8192 SPARSE "Debian 6-s005.vmdk"

# The Disk Data Base
#DDB
ddb.virtualHWVersion = "7"
ddb.longContentID ="86aa7ebbb50ab88b973ea60271ad0a67"
ddb.uuid = "60 00 C2 9f 9a e3 43 6a-ea 70 c7 fa 35 72 7c 04"
ddb.geometry.cylinders = "1044"
ddb.geometry.heads = "255"
ddb.geometry.sectors = "63"
ddb.adapterType = "lsilogic"

Row order and content may vary, VMWare configuration files don't have a fixed order, you may change row order, add comments and some other stuff inside it. Here's what you need to change:

RW 4192256 SPARSE "new.machine-s001.vmdk"
RW 4192256 SPARSE "new.machine-s002.vmdk"
RW 4192256 SPARSE "new.machine-s003.vmdk"
RW 4192256 SPARSE "new.machine-s004.vmdk"
RW 8192 SPARSE "new.machine-s005.vmdk"

So all you need to do is change references to physical hard disk files, nothing more, just change the lines above in your new.machine.vmdk file and nothing else

 

Other Descriptors

It's time to change the VMXF file (extra configs from VMWare), if you don't have it, just skip this step. Your new.machine.vmxf file could be something like that:

52 62 73 9d 7f 10 1b 58-8e 3c 8e 15 8e ef f4 a3 Debian 6.vmx

It's an XML file as you may see, content and VMIDs may change a little bit but it doesn't matter. All you need to do here is to replace this string:

Debian 6.vmx

with this one

new.machine.vmx

and nothing more, here's the result:

52 62 73 9d 7f 10 1b 58-8e 3c 8e 15 8e ef f4 a3 new.machine.vmx

VMX Main configuration file

new.machine.vmx is the machine main configuration file, inside it you find hardware description and file references, it may vary a lot according to virtual hardware, player version and hardware machine version, here's a copy of my new.machine.vmx (original copy from Debian 6.vmx)

.encoding = "UTF-8"
config.version = "8"
virtualHW.version = "7"
scsi0.present = "TRUE"
scsi0.virtualDev = "lsilogic"
memsize = "256"
scsi0:0.present = "TRUE"
scsi0:0.fileName = "Debian 6.vmdk"
ethernet0.present = "TRUE"
ethernet0.connectionType = "bridged"
ethernet0.wakeOnPcktRcv = "FALSE"
ethernet0.addressType = "generated"
pciBridge0.present = "TRUE"
pciBridge4.present = "TRUE"
pciBridge4.virtualDev = "pcieRootPort"
pciBridge4.functions = "8"
pciBridge5.present = "TRUE"
pciBridge5.virtualDev = "pcieRootPort"
pciBridge5.functions = "8"
pciBridge6.present = "TRUE"
pciBridge6.virtualDev = "pcieRootPort"
pciBridge6.functions = "8"
pciBridge7.present = "TRUE"
pciBridge7.virtualDev = "pcieRootPort"
pciBridge7.functions = "8"
vmci0.present = "TRUE"
roamingVM.exitBehavior = "go"
displayName = "Debian 6"
guestOS = "other26xlinux"
nvram = "Debian 6.nvram"
virtualHW.productCompatibility = "hosted"
gui.exitOnCLIHLT = "FALSE"
extendedConfigFile = "Debian 6.vmxf"
ethernet0.generatedAddress = "00:0c:29:b1:8b:e6"
uuid.location = "56 4d 05 92 24 e8 b0 b3-f7 37 1f d9 51 b1 8b e6"
uuid.bios = "56 4d 05 92 24 e8 b0 b3-f7 37 1f d9 51 b1 8b e6"
cleanShutdown = "TRUE"
replay.supported = "FALSE"
replay.filename = ""
scsi0:0.redo = ""
pciBridge0.pciSlotNumber = "17"
pciBridge4.pciSlotNumber = "21"
pciBridge5.pciSlotNumber = "22"
pciBridge6.pciSlotNumber = "23"
pciBridge7.pciSlotNumber = "24"
scsi0.pciSlotNumber = "16"
ethernet0.pciSlotNumber = "32"
vmci0.pciSlotNumber = "34"
vmotion.checkpointFBSize = "16777216"
ethernet0.generatedAddressOffset = "0"
vmci0.id = "1370590182"
vmi.present = "FALSE"
ide1:0.present = "FALSE"
floppy0.present = "FALSE"

Now let's focus on the changes, it's basically straightforward to understand it but I need to mention:

change previous VMDK references to newly created hard disk, basically you need to replace “Debian 6.vmdk” with “new.machine.vmdk” everywhere in your file (just one occurrence)

scsi0:0.fileName = "new.machine.vmdk"

Now it's time to change the label for your new machine in the Server (ESX,GSX,VSphere) or Player with your favorite name (“My new Machine Name” in my case), here's:

displayName = "My new Machine Name"

NVRam file with the new name:

nvram = "new.machine.nvram"

Extended configuration file (only if this is present) with the new one:

extendedConfigFile = "new.machine.vmxf"

Now change the Ethernet mac address with a new one or your machines cannot be on the same network with the same address (as in real cases), just respect mac address notations and change something random in it

ethernet0.generatedAddress = "00:0c:29:b1:ab:ab"

You may change UIDs inside the file but you don't need to bother about them. Save everything and import your newly created/cloned machine inside your favorite player/server.

 

Ready, Set, Go!

Locate your new .VMX file and open it with your Server/Player, you'll see your new machine inside the remote/local repository and you're ready to start it.

We didn't change the machine UID because it's not necessary, VMWare will do it for us, when you run your machine the first time you'll see a window like this

Just select “I copied it” button and VMWare will generate the serial UID for the new machine. Now the machine runs an exact copy of your previous one with the same operating system and configurations inside it, please read these hints to solve possible problems:

  • If you're using a static IP address you need to change it in your new machine to avoid conflicts with the previous one (obviously)

  • If you're using a MS Windows OS you need to change the machine name or you'll have a “name duplicated” error when you start the machine, just change the name and make a new reboot

  • If you're serving clients with a basic service (DHCP, DNS, MS Domain Controller) you need to stop it or you'll have few network troubles (as with real servers) due to two services running in the same network (two DHCP servers in the same net are a bad thing...)

  • udev troubles and linux networking, please read below if you're running a perfect Linux machine but without networking capabilities

 

No networking ? Please read

Everything is fine with your new virtual machine but.... you don't have a network card properly configured ? Keep reading.

If you're using UDEVD (http://linuxmanpages.com/man8/udevd.8.php) you may have a problem, it's just a minor trouble as you'll see.

UDEVD defines plugged network cards in a proper configuration file, network cards are located generally in /etc/udev/rules.d, there's a file called: “z25_persistent-net.rules” | “70-persistent-net.rules” (Debian | Gentoo) or something like that, it's not hard to find it (or let me know and I'll add your information here), generically it's called *persistent-net.rules, let's see it to understand how it works:

~$ cat /etc/udev/rules.d/70-persistent-net.rules
# This file was automatically generated by the /lib64/udev/write_net_rules
# program run by the persistent-net-generator.rules rules file.
#
# You can modify it, as long as you keep each rule on a single line.


# PCI device 0x10ec:0x8168 (r8169)
SUBSYSTEM=="net", DRIVERS=="?*", ATTR{address}=="00:21:85:c1:79:37", KERNEL=="eth*", NAME="eth0"

 

Here's the “old” network card (the one you cloned), it's called “eth0” and this card is not available any more (you've just changed the mac address), you may:

  • Delete the line reporting the “eth0” device, just delete this line

  • Change the line with the proper mac address (the one you've changed in the VMX file)

I usually prefer to delete the line so a new one will be created for you on the next reboot

 

NOTE: If you've a line with “eth1” device and you don't have two network cards it means UDEVD has created the line for your with your new (according to him) network card and it left the previous one already there and configured, you may remove eth0 line and rename eth1 to eth0 OR delete both lines. UDEVD will recreate what it needs on the next reboot, don't change your network configuration (etc/networking and so on), just leave UDEVD with the proper card and you'll see it running fine from the next boot

NOTE SAMPLE: If you're following my own example with a Debian 6 installation you don't need to worry about udev, previous versions (etc, lenny, …) are affected by this


IMPROVED SCRIPT
: After few comments reported on this blog I've decided to write a new blog with an automated script, the script does everything reported it by itself, check it out here


I hope this small guide will assist you in some way if you decide to clone your VMWare machines on your own, file formats are basically the same for a long time and that's what I do for basic sys Admin when I don't have the hypervisor or proper tools with me.

Share your comments

Hope it helps

 

Andrea (Ben) Benini

 

 

WIKIBEN: Virtualization Articles Collection

Here's a list of my articles related to virtualization, I've collected all of them in this page to summarize my results, I'll always keep the page updated so you can periodically check my activities related to it.

I've sorted subjects on technology, product and topic, feel free to ask me new topics or arguments if you need more information about something, hope this helps


Generic
VMWare Server 2.x thoughts



VMWare Troubleshooting and Tricks
Manually clone a VMWare Virtual machine from the shell
Clone a virtual machine from the shell (THE SCRIPT)
Mouse/Keyboard not responding on VMWare Player with Linux
Accessing VMWare Server 2 with vSphere Client (the unsupported way)
Access VMWare Server 2 remote virtual machine without web interfaceHOWTO: VMWare Server 2, Disable Web Server Interface



Linux Distro Related Topics

Install latest VMWare Player on Gentoo (without portage)
Install VMWare Player on Gentoo (amd64), the easy way
HOWTO: Install VMWare Server 2 on Debian Lenny, AMD64 (64bit)

 

 

 

 

Ben

 

 

VPN-O-Rama : IPCop to IPCop with IPSec

After a short introduction (http://www.linux.com/community/blogs/vpn-o-rama-vpns-intro-practical-howtos-screenshots.html) it's time to face the facts and see something practical.

As previously mentioned I'd like to focus on ready made Linux distros so you can create a VPN connection on the fly and easily in just few easy steps. In my first episode I'll approach IPCop (www.ipcop.org) and I'll create a VPN connection between two IPCop machines, screenshots are something nice to see but our first step is to plan the example.

This example is built in a private network with virtual appliances, you can obviously modify it to fit your needs, I'll use fake names and networks, translate them to your current network if needed


 

Network topology:

Office

Network

Subnet

Headquarter (Coruscant)

10.0.2.0

255.255.255.0

Subsidiary 1 (Alderaan)

10.0.3.0

255.255.255.0

 

 

Firewalls:

Location/Name

Linux Distro

Private IP (LAN)

Public IP (WAN)

Coruscant

IPCop v1.4.21

10.0.2.94

10.0.0.94

Alderaan

IPCop v1.4.21

10.0.3.95

10.0.0.95

 

For simplicity I've two private and separate networks (representing two offices) connected to a private net (10.0.0.0/24) representing the Internet. It's an easy example quite portable to everything. I also have static IP Addresses (LAN and WAN) and no NAT traversal troubles around (at least in this example, I'll come back with NAT traversal and dynamic IP addresses later...).

IPCop installation is pretty straightforward, I'll assume you're familiar with it or you can install it without serious issues (or let me know and I'll write something for you if needed); from a basic installation without additional modules or plugins you've everything you need to setup an IPSec connection between your machines.

I'll use IPCop IPSec built in capabilities to set everything up, first connect to your machine in you headquarter (Coruscant), just go to: https://10.0.2.95:445, then select VPNs Menu and choose VPNs option or go directly to https://10.0.2.94:445/cgi-bin/vpnmain.cgi if you prefer. Here's what you see with a clean installation

 


If you've done some tests or you've some previous configurations you may press “Remove all CA and certs” to wipe everything. If you want to use IPSec on this host you need to check the “Enabled” flag (on top left) and issue a fully qualified domain name (FQDN) or public IP address for this machine (in our case is 10.0.0.94) than it “Save” button to start IPSec on IPCop.

Now as your first step you'll create root/host certificates, press “Generate Root/Host Certificates” to create an X509 cert from Coruscant firewall (10.0.2.94), in the next screen you need to fill some data related to your host and office, here are mine :

You've just created an X509 certificate inside your firewall (Coruscant) with a root and host certificate for your machine, here's what you'll see after this:

Now save your root and host certificates by hitting the two little disk icons on the bottom right (download root certificate / download host certificate) and name them as:

  • cacert.coruscant.pem
  • hostcert.coruscant.pem

Now we need to do the same in the other IPCop machine (Alderaan), here are screenshots taken from https://10.0.3.95:445/cgi-bin/vpnmain.cgi :

Generate Root/Host Certificates button

hit Generate Root/Host Certificates button and here's the result

Now save your root and host certificates by hitting the two little disk icons on the bottom right (download root certificate / download host certificate) and name them as:

  • cacert.alderaan.pem
  • hostcert.alderaan.pem

 

Importing Certificates on both sides

Now on Coruscant firewall (10.0.2.94) you need to import Alderaan root certificate, in VPN page type “Alderaan” as CA Name and select cacert.alderaan.pem by hitting the “browse” button, see image for details:

Hit “Upload CA Certificate” to continue, here's the result

Now you know Alderaan certification authority on this machine and you're ready to create a VPN tunnel, let's do the same on Alderaan firewall (10.0.3.95), see screenshots:


 

Ready, Set, Go !

Where's my VPN tunnel ? Relax, we're creating it now; we've done the tough part related to certificates and authorities, now let's establish the tunnel.

On Coruscant firewall hit the “Add” button in the middle of “Connection status and control” tab so you can decide the type of VPN connection; we're trying to connect two networks so we choose Net-to-Net Virtual Private Network in the following screen, then we press add to continue (screenshot)


Now fill remote data (Alderaan) with proper values, as you may see they match remote Alderaan network (10.0.3.0/255.255.255.0) and Alderaan public/static IP address.

In the authentication section you need to select “Upload a certificate” and use hostcert.alderaan.pem certificate downloaded before, in the bottom of the page hit the “Save” button to continue,

you'll now see a new closed VPN connection on Coruscant firewall


Now do the same on Alderaan firewall to establish the connection, go to https://10.0.3.95:445/cgi-bin/vpnmain.cgi (Alderaan) and press the “Add” button to create a new VPN connection, select NET-TO-NET as done before

 

Now fill remote data (Coruscant) with proper values, you're on the other side so you need to reverse everything: Coruscant network (10.0.2.0/255.255.255.0) and Coruscant public/static IP address (10.0.0.94). In the authentication section select “Upload a certificate” and hit “browse” to select hostcert.coruscant.pem , see the screenshot

Then press save on the bottom of the page to continue


 

Yeah ! we're up and running

Now these two networks are fully connected and working, I hope you'll benefit from this article and find it useful for your work; let me know if you want further details or additional information.

Next episodes will cover, different Linux and BSD distro, more configurations, NAT and dynamic IP addresses as well.

 

 

Previous:
VPN-O-Rama: VPNs intro, practical HOWTOs

Next:
IPCop to PFSense with IPSec

 

Andrea Benini

 

VPN-O-Rama: VPNs intro, practical HOWTOs, screenshots

I've spent few days on corporate VPNs with few Linux and BSD distros and I've decided to write down some notes and publish few screenshots for practical usage, even for newbies.

I've read some docs but I've never found a quick guide with practical examples for newbies to create a VPN from scratch, in these episodes I'll create VPNs with real examples. As you may know you can create VPN between two machines/networks by using a lot of different security mechanisms like:

  • IPSec (my favorite) IPv4 and IPv6 capable

  • OpenVPN (SSL/TLS based), nice for roadwarrior connections but you may have troubles with NAT and firewall policies

  • MPVPN, never used it, I've seen it during certification exams but I really don't know who uses it

  • PPTP, Microsoft Point to Point encryption system, avoid it like a plague if possible, buggy and it had several security issues

  • SSTP, Secure Socket Tunneling protocol introduced by Microsoft with their Windows Server 2008 and Vista/7, seems to be nice but not so portable or available on third party systems

  • DTLS, mainly from Cisco Systems

 

There are even more VPN solutions but mostly proprietary based, this saga has several different achievements in mind:

  • It has to be portable. I'd like to use my favorite security mechanism with available hardware or software, we don't want to rely on specific OS or platform. I even want to use it on very cheap hardware or embedded devices (read: high class smart phones)

  • It has to be secure, so we don't want security issues or known troubles around us

  • It has to be free and publicly available so everyone may take a look at it

 

When you need to connect two different hosts/networks you may have different scenarios:

  • you need to connect a single host to a remote network

  • you need to connect a network to a remote network

  • you've public and static IPs on on both side

  • you've dynamic IP at least on one side

  • you've one or more firewalls in the middle with one or more blocking rules (and sometimes you cannot modify them)

 

As you may know from the top I'd like to use IPSec because that's what I'm using now for these reasons:

  • It's available everywhere, from cheap DLink DSL routers to heavy BSD servers, it's not tied to a particular operating system

  • It's stable and solid

  • no security issues (yet) [http://bsd.slashdot.org/story/10/12/15/004235/FBI-Alleged-To-Have-Backdoored-OpenBSDs-IPSEC-Stack is it real or a fake ??]

  • IPv4 and IPv6 ready

  • few troubles with NAT'd networks compared to others

  • works great with static IPs (and that's my case), but even with dynamic if you cheat something; by the way I'll show you even something more from OpenVPN, that is my favorite roadwarrior solution

 

I'll start with IPSec in different scenarios but I'll go further with other solutions like OpenVPN or PPTP if you want, I'll try to publish a single and detailed article for every case or you can suggest me your needs.

 

Resources

If you've a lot of time and you'd like to know everything on IPSec you may take a look at (http://www.ipsec-howto.org/), it's a good guide for a Linux sys admin. Also read Openswan documentation (http://www.openswan.org/), Openswan is an implementation of IPSec for Linux. It's quite hard to start from scratch with Openswan on the command line but this is the definitive guide (now) for it. If you've enough time to set everything up and fine tune every aspect of your connection I suggest you to use only these components: the Linux kernel, IPTables, Openswan. If you've limited time and you want to deal with ready made distros oriented to firewall/VPN solutions you may follow next articles.

 

Next Step:
IPCop to IPCop with IPSec

IPCop to PFSense with IPSec

 

Glad to read your comments

Andrea Benini

 

 

Mouse/Keyboard not responding on VMWare Player with Linux

Many of you use everyday virtualization products to emulate other machines and run specific tasks on them.

As many of you already know I only use Linux machines at work, it ain't that easy if you need to survive in a corporate Windows forest (AD controller and windows environment) but I'm still fighting for it. Sometimes you need to use certain Windows apps or developer tools and you don't want to install WINE or something like that, as many of you I use customized and virtualized Windows machines. VMWare player outside "mainstream" distros have some lacks or troubles, expecially when you deal with GTK.

it happens to have mouse garbled, or if you move it inside VMPlayer window it disappears or acts in a weird mode, the same happens to the keyboard (not proper working), after a while I've figured how to solve it, it's not that strange or particular, it's a quite known problem and the fix is quite easily available if you use google for a while.

If you work with the latest version of Gnome your VMWare Player won't work well because it was supplied with previous version of GTK, even if you have "grab when cursor enters window" option set. It won't grab the pointer and it looks strange when moving it inside the VM Window, I hope this workaround will help you until VMWare solves compatibilities with the GTK library (and even release a VIC/vsphere like client for linux !!!).
You need to force VMPlayer to use shipped version of GTK. here's what you need to do:

  • locate vmplayer program path (`which vmplayer`), /opt/vmware/player/bin/vmplayer in my linux gentoo distro
  • It's a text file so you can edit it with your favorite editor (nano or whatever)
  • add a line with  `  export VMWARE_USE_SHIPPED_GTK="force"   ` after "set -e" line, so it will look like:
# is installed.
#

set -e
export VMWARE_USE_SHIPPED_GTK="force"

ETCDIR=/etc/vmware

line "export VMWARE_USE_SHIPPED_GTK="force"" is what you need to add, it works even with "export VMWARE_USE_SHIPPED_GTK=yes", choose whatever you like

 

Now when you run it you'll see an application with a bad look, the older GTK version is used there and it ain't that nice but at least it works fine

Now run your favorite virtual machine and you'll see no mouse garbling now. This solved my troubles on Gentoo but even with other distros. Hope it helps

 

Glad to read your comments

Andrea (Ben) Benini

 

 

Easy backup with RSync, windows client configuration

Now for the latest article of this first rsync saga I'd like to share with you my current Windows clients configuration, as I've already explained, rsync is quite easy to understand and configure; in your windows client you only need these files:

  • rsync.exe
  • cygpopt-0.dll
  • cygwin1.dll
  • Copying.txt

Copying.txt is the GPL public license, it's needed only for license purpouses, the other files: 2 DLLs (cygpopt-0.dll, cygwin1.dll) and 1 EXE file (rsync.exe) are part of your solution. As you imagine these files are taken from a working cygwin installation: you can download them from the net, from a working cygwin environment or directly from [this link where I've provided them for you]. These files allows rsync to work properly and syncing your local disk with a remote RSync server.

Rsync has an excellent manual and configuration page, a ton of options ready for you, here's just what I'm using for my backups:

rsync --verbose --recursive --compress --delete --perms --owner --group --specials --stats --devices 
--links --times --exclude=”system*” “/cygdrive/c/backuphomedir/”
“array1_backup@myfavoritenas::array1_backup/ComputerID/” 1> “errorlog.txt” 2>&1

This is the raw command used for tests, of course you need to script it in a batch file or wherever you want. I don't want to enter into rsync syntax flags deeply when you've an excellent man page (like this one [http://www.samba.org/ftp/rsync/rsync.html]), but briefly:

--verbose be verbose on operations, useful for generated log file
--recursive copy directory specified and go recursively into subdirs
--compress compress file data during transfer
--delete delete file from remote host if local files doesn't exists anymore
--group --specials preserve group and special files
--stats give some transfer statistics (again, useful for logging)
--devices preserve device files
--links preserve links
--times preserve modification times

This command works with previous samples from past episodes, now pay attention to parameters related to your own machine settings like:

/cygdrive/c/backuphomedir/ it's the home dir where you want to start from copying your data, for Windows users the translated path is “C:ackuphomedir“, I've reported “/cygdrive/c/backuphomedir/” because cygwin environment starts with “/” (as UNIX), then it appends “cygdrive” (all local drives) then the name of the Windows drive “c”, then your backup sourcedir; (backuphomedir).

--exclude flag: directories to exclude from your backup path (inside c:ackuphomedir), so directories starting with “system” (like: c:ackuphomedirsystem, c:backuphomedirsystem32, …) will be excluded from your copy. Why ? Because I store rsync, DLLs and my batch file inside this dir

array1_backup@myfavoritenas::array1_backup/ComputerID is the rsync path where you want to sync your data, “myfavoritenas” is the machine name, “array1_backup” are username and “rsync share point”. ComputerID is just a directory inside your rsync share point, use different directories for different PCs so you've a dir for each pc of your net (use MachineID, username or whatever you want).

errorlog.txt is my log file where rsync operations are stored, very useful for further readings

If your rsync share point has a password as mine you need to type it on command line if you're executing this command as it is; if you're scripting it into your own program you need to export a Windows variable called RSYNC_PASSWORD (set RSYNC_PASSWORD=”your pass”) to have it running, read rsync manual page for details.

Be careful about locked files, rsync and every windows program will fail the backup if the file you want to copy is locked by someone else, an example ? Take a look at Outlook .PST files, you cannot copy them when Outlook is opened

Here are few considerations for applying this solution in a real environment, here's my experience:

  • I've set a “resume time” in BIOS for every PC I've in my net (example: wakeup at 00:01 am)

  • I've inserted a scheduled job (windows scheduler is fine for me even if it's nothing fancy) which runs a C++ application created by me (example: start app at 00:30 am)
  • My applications acts as a wrapper to rsync, makes its own copies and so on
  • When copy is finished to powers off the PC again, if the PC was already on (user leaved the computer on from yesterday) I leave it on by checking computer uptime

That's it !

 

Why using this kind of solution against others ?

  • GPL, no license fees, easy configuration, easy customization

  • Easy server installation, no custom packages or services, rsync is well known for its simplicity and configuration
  • RSync support and documentation is HUGE across internet, there're a lot of things done with it
  • Multiplatform: linux/OS-X/Windows/... you don't care about the operating system, you just need to have rsync compiled (cygwin for windows is ok) but you can also have your XYZ operating system or whatever you want. Not so many backup solutions have a port for nearly every existing platform like rsync has
  • No installation on Windows platform, I know, this is a rude solution but it works fine and flawlessly; I've just paired windows scheduler with rsync. No setup, no install, no virtual machines/.NET/... just copy the files

This episode concludes my first RSync saga, nothing strange or complex, just a common use of rsync in a real environment for handling daily operations in a network.

 

Previous Steps:
Easy backup with RSync, introduction
RSync server side config on linux platform
Linux RSync client side configuration
Windows RSync client side configuration

 

Hope it helps you to save time

Glad to read your comments here if you find it useful

 

Andrea (Ben) Benini

 

Easy backup with RSync, linux client configuration

After the basics and the server configuration here's a quick 'n' dirty example of my linux client configuration.

Each Linux client (but generically speaking a UNIX or OS/X client) only needs the rsync program installed, no additional dependencies, each linux distribution has it, just type:

~$ rsync --version
rsync version 3.0.6 protocol version 30
Copyright (C) 1996-2009 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
append, ACLs, no xattrs, iconv, symtimes

rsync comes with ABSOLUTELY NO WARRANTY. This is free software, and you
are welcome to redistribute it under certain conditions. See the GNU
General Public Licence for details.

To see if it's properly installed.

This example coming from real world is connected to an RSync server as detailed in my previous article,the script itself is not complex, it just feet my needs and I use it with cron each day, in the first section there is a configuration part for remote host setup, the second part is where the business logic resides, a little bit of logging is inserted as well

~/bin$ cat backup.rsync
#!/bin/bash
# Sync sensible data of this pc to a remote rsync host
#


# Configuration Section, change your parameters below

# Remote host name
REMOTE_HOSTNAME=myfavoritenas

# RSync connection on remote host (username as well)
REMOTE_SYNC_POINT=array1_backup

# Remote directory name where rsync copies will be created
REMOTE_DIR=linux_client_host

# Local directory name to sync (not the whole disk...)
LOCALE_DIR=$HOME

# RSync password for the connection (see rsyncd.secret)
RSYNC_PASSWORD=idonttellmypasswdtoyou

# Exclude FILES
EXCLUDE_FILES="
--exclude "$HOME/.Trash" --exclude "$HOME/.bittorrent" --exclude "$HOME/.dbus" --exclude "$HOME/.evolution" --exclude "$HOME/.fontconfig" --exclude "$HOME/.gnochm" --exclude "$HOME/.icons" --exclude "$HOME/.macromedia" --exclude "$HOME/.metacity" --exclude "$HOME/.mozilla" --exclude "$HOME/.mysqlgui" --exclude "$HOME/.nautilus" --exclude "$HOME/.nx" --exclude "$HOME/.python" --exclude "$HOME/.qt" "


# Business logic, don't change anything below this line
MY_RSYNC_OPTIONS="
--verbose --recursive --compress --perms --owner --group --specials --stats --devices --links --times --delete $EXCLUDE_FILES "

# Exec command
RSYNC_COMMAND="`which rsync`"
export RSYNC_PASSWORD
DATE_BEGIN="Begin : `date`"
if [ "$1" != "" ]; then
$RSYNC_COMMAND $MY_RSYNC_OPTIONS $LOCALE_DIR/ $REMOTE_SYNC_POINT@$REMOTE_HOSTNAME::$REMOTE_SYNC_POINT/$REMOTE_DIR
echo $DATE_BEGIN
echo "End : `date`"

else
$RSYNC_COMMAND $MY_RSYNC_OPTIONS $LOCALE_DIR/ $REMOTE_SYNC_POINT@$REMOTE_HOSTNAME::$REMOTE_SYNC_POINT/$REMOTE_DIR > /$LOCALE_DIR/backup.rsync.log 2>&1
echo $DATE_BEGIN >> $LOCALE_DIR/backup.rsync.log
echo "End : `date`" >> $LOCALE_DIR/backup.rsync.log
fi

As you can see from example you can invoke this script just by typing:

~/bin$ ./backup.rsync

If you provide an additional parameter on command line you can store rsync operations in a log file (backup.rsync.log)

~/bin$ ./backup.rsync log

 

Glad to improve my example if someone of you wants to contribute or add something
Next article covers windows client configuration, stay tuned

Next:
Windows RSync client side configuration

Steps:
Easy backup with RSync, introduction
RSync server side config on linux platform
Linux RSync client side configuration
Windows RSync client side configuration

 

Glad to see your comments
Andrea (Ben) Benini

 

Easy backup with RSync, server configuration

Now after this quick solution preview I'd like to share with you my current rsync server configuration, as I've already explained you rsync is quite easy to understand and configure, just install it in your favorite distro and configure these things:

  • /etc/rsyncd.secret file, this file contains rsync shared connection and password
  • /etc/rsyncd.conf file, this file contains rsync server configuration itself

Now let's have a look of rsycnd.secret file, it's something like:

root@myfavoritenas:/etc# cat rsyncd.secret
array1_backup:idonttellmypasswdtoyou
.... (and so on)....

this config file was taken from my current NAS, each line have two columns: share name and password, in the example above share name is "array1_backup" and password is "idonttellmypasswdtoyou", more lines may follow depending on your config. Due to security reasons you cannot read the file except if you're root, so file have chmod 0400

root@myfavoritenas:/etc# ls -la |grep rsyncd.secret
-rw------- 1 root root 368 Jul 10 2009 rsyncd.secret

Now the big part is inside rsyncd.conf, let's have a look:

root@myfavoritenas:/etc# cat rsyncd.conf
uid = root
gid = root
use chroot = yes

[array1_backup]
path = /mnt/array1/backup/.
read only = no
auth users = array1_backup
secrets file = /etc/rsyncd.secret

This config has been taken from a Buffalo TeraStation, I think you may have something similar in your favorite distro, I don't like so much buffalo configuration but this is really simple and easy to understand, I can even attach a config from your favorite distro if you like.

As you can see there's uid/gid for rsync process (don't like to see root run it), rsync chrooting, and you can see a section for each share you define (array1_backup in my example)

Next: Linux Client Configuration

Steps:
Easy backup in a Windows/Linux network with RSync
Easy backup with RSync, server configuration
Easy backup with RSync, linux client configuration
Easy backup with RSync, windows client configuration

 

Glad to see your comments

Andrea (Ben) Benini

 

Easy backup in a Windows/Linux network with RSync

This time I'd like to show you a really easy backup method using open source solutions as usual.

I mainly work as a network administrator in a company, keeping services running and having an efficient network is my primary task. One of my big troubles in the network is not related to servers or network physical backbone, troubles comes frequently from users and client desktop PCs, mainly Windows machines.

Problems are divided in two different categories: hardware and software failures; crappy PCs nowadays are frequent and cheap hardware is always a problem, software failures are mainly related to the operating system (mainly XP and Vista, no 7 in my net yet).

You cannot mess with hardware, you can try to buy better PCs or from well known hw manufacturers but if you're trying to recycle old PCs sometimes you run into troubles. Hard Disks are one of my biggest point of failuires. Operating system and mostly user misconfigurations also drives you to every kind of mess, in a regular network like mine corporate users don't have custom or strange apps, they're using Office Automation tools and few more; real problems comes from user custom data, not always backed up as it should be ("what are backups ?") and always locally stored on workstation hard disk even if they're not allowed to do it.
This solution covers data backup only, I don't care about applications setup, operating system setup or something more, just only data backup. In my case, in my network, operating system setup takes me just an hour and applications needed are only Office Automation tools as I told you before. One tool comes in mind to me to achieve this kind of backup: "rsync".

Here are main benefits:

  • You don't need to install additional software or tools in each client, rsync app compiled with cygwin comes with few dll dependencies, no install required
  • It comes for free, just deploy GPLv2 file with the license along with rsync files
  • no strange windows config on it, just need to schedule a task to run rsync periodically, windows scheduler is enough for running this kind of task
  • you don't need to rely on windows auth, permissions, samba, Active Directory or whatever
  • you only need a network and rsync server. RSYNC servers are really easy to setup, you can run them on linux (my choice) or windows or whatever you want
  • you've plenty of options for easy setup from client side as well as from server side

Next:
RSync server side config on linux platform

Steps:
Easy backup with RSync, introduction
RSync server side config on linux platform
Linux RSync client side configuration
Windows RSync client side configuration

Hope it helps

Andrea (Ben) Benini

 
Page 3 of 7

Upcoming Linux Foundation Courses

  1. LFD331 Developing Linux Device Drivers
    25 Aug » 29 Aug - Virtual
    Details
  2. LFS422 High Availability Linux Architecture
    08 Sep » 11 Sep - Raleigh, NC
    Details
  3. LFS426 Linux Performance Tuning
    08 Sep » 11 Sep - New York
    Details

View All Upcoming Courses


Who we are ?

The Linux Foundation is a non-profit consortium dedicated to the growth of Linux.

More About the foundation...

Frequent Questions

Join / Linux Training / Board