Linux.com

Community Blogs



Find a string in files (using grep easily)

Here's a quick command for a quite common problem:

Is there a way to find a text string inside a bunch of files in a dir ?

Sure it is, here's one :

grep 'MyCoolString' /my/preferred/directory/*

Searches "MyCoolString" inside the specified path and willcard, or if you prefer to ignore case take a look at:

grep -i 'mycoolstring' /my/preferred/directory/*

Easy isn't it ?

Obviously this is not the only method but it's my preferred one, easy syntax, easy to remember, glad to see your comments or alternative methods for this task

Hope it helps

 

Ben 

 

Xen VM migration on SLES 10

This how-to describes the process for migrating a SLES Xen virtual machine from one physical host to another. In this scenario, the new VM will be hosted on shared storage mounted via NFS. This scenario may also work for Physical to Virtual (P2V) migration.

 

1. Shut down source vm on Dom0, mount the hard drives.

xm shutdown vm1

# list the drives to find which one is the root

fdisk -l /dev/sda

# Gain access to the old virtual hard drive: create the mapping under /dev/mapper

kpartx -a /dev/sda3
mount -o loop /dev/mapper/sda3p2 /mnt/temp

2. Create a new hard drive image file (60GB) on the new location using dd:

dd if=/dev/zero of=/mnt/vm2/disk0-new.img bs=1M count=1 seek=60000

# Make the file system
/sbin/mkreiserfs -q -f disk0-new.img

# Make the swapfile
dd if=/dev/zero of=/var/lib/xen/images/vm2/vm2-8g.swap bs=1k count=8392930

mkswap vm2-8g.swap # it is large because of oracle reqs

# Mount the new filesystem

mount -o loop /mnt/vm2/disk0-new.img /mnt/img

3. Create new virtual machine minimal install with SLES 10 SP2 on new Xen host, pointing to new hard drive files. Use Yast's "Create a Virtual Machine."

4. rsync the file system from /dev/mapper/sda3p2 to /mnt/img

### exclude_file.txt
/dev/
/mnt/
/lost+found/
/sys/
/boot/
/media/
/proc/
/selinux/
/tmp/
sysconfig/

$ rsync -rlpogt --progress --exclude-from="exclude_file.txt" /dev/mapper/sda3p2/ /mnt/img/

5. Correct entries in files in the newly replaced /etc for the new vm configuration: /etc/fstab (now /dev/xvda1 instead of hda2 for /, /dev/xvdb1 for swap instead of hda1), /etc/passwd, /etc/group, /etc/sysconfig/network/ifcfg_eth..  Kept the new vm's mac address and updated the IP for vm2 in the config file /etc/xen/vm/vm2.

6. Start up the new virtual machine. Don't forget to do an 'xm delete vm1' on the old system once you are sure it is no longer needed.
 

Hylafax, send a fax from the command line

In these days I'm stressing hylafax a lot, I think it's a nice and powerful program, stable, complete and reliable. When my job will be completed I'll publish some thoughts about it.

Server part is so stable and secure, client part, expecially for Windows clients have some lacks, there're a lot of win client all around but every software I've tried has some lacks so as a lot of you I've decided to write my own.

Few of them have everything I need, except the license and price, I mean I think it's right to charge for your work but while talking about an high ranking opensource software I was thinking it has good opensource clients as well.

No matter I'm deploying a base installation with a lot of different clients so I think I'll buy some commercial clients suitable for my needs (I was really impressed about ifax.com HylaFSP client) but by the way I think I'll deploy even few clients maded by me, I'm in an Active Directory network and few considerations and limitations about existing open source clients made me take the decision of writing some clients on my own.

Now let's start with the basics, you've started reading this article for getting information about sending faxes from command line, don't you ?

While deploying my new client I've read about how to send something from command line, this solution will be integrated in my new client. First you need to install command line hylafax-client, check out your favorite distro, most of them have a package called "hylafax-client" (Gentoo, Debian based and others), if don't have it take a log at a command called sendfax, your distro should have it somewhere.

here's the command:

sendfax -f " This e-mail address is being protected from spambots. You need JavaScript enabled to view it " -R -r "Fax Subject" -c "Coverpage comments" -x "Your recipient" -d "Recipient@1234567890" sendfax.example.document.ps

First you need to create your example document in PostScript format, if you've installed hylafax you'll have ps2xx utilities (ps2pdf, ps2ascii, ps2txt, pdf2ps, ...), use them to convert from your current format to Postscript if you don't have a PS ready document (dummy example: pdf2ps input.pdf output.ps)

then modify parameters according to your email address if you want to receive notification about the job status (failed, success), fax subject, coverpage comments if you've it and so on, obviously change 1234567890 with your destination fax number.

Pretty easy, isn't it ?

That's why I'm writing my own windows and linux wrapper, backend sendfax program is so easy to use so I just need few changes for adapting my AD integration

 

Let me know your thoughts

Ben

 

 

ssh, scp without password to remote host (Look Ma, no Password !)

Here I'm, back again on SSH stuff, as you can see from my previous posts (search blogger name = "ben") OpenSSL and SSH stuff is very interesting and useful for me, so I wrote down a lot of notes on them, this time I'll show you how to connect to an SSH host without password input.

Yeah, I know, there're a lot of folks all around explaining you how to do that but I promise to make it easy 'n' dirty, without hassling you too much, just the basic steps for connecting to your remote host and make it working.

What would you do with this tutorial ? for example:

  • you can ssh to your remote host without requiring a password, this is safe and secure (it uses SSH public/private keys) until you keep your private keys for yourself. A quite recurring task if you've a lot of machines to manage
  • Copy files from an host to another, not only as utility but even for basic administration task, if you manage a network you know what I mean
  • Grant someone access to certain hosts for his job (be careful ok ?)
  • Use all the other SSL suite across hosts, this is not only for ssh or scp, all SSL suite is involved, look at my articles on SSH port forwarding for example, there are a lot of them (blogger: ben)
  • Impress your boss or whatever you'd like


Ok, let's get started

Let's assume you've two hosts:
mylocal - the host from where you want to connect
myremote - the host where you want to connect to

 

1) From mylocal create an ssh rsa key pair for host validation, here's how: 

mylocal:~# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx user@mylocal
The key's randomart image is:
+--[ RSA 2048]----+
....
+-----------------+



2) Now you need to copy the public key to your remote host, do NOT copy the private key, obviously use scp to do this

mylocal:~# scp ~/.ssh/id_rsa.pub myremote:~
Password:
id_rsa.pub 100% 391 0.4KB/s 00:00

so now you've your public key copied fine, let's connect to remote host now



 

3) Connect to your remote machine (myremote)

mylocal:~# ssh root@myremote (root or your remote username)
Password:
Last login: Wed May xx xx:xx:xx xxxx 2009 from mylocal on ssh
myremote ~ #



4) Check out .ssh stuff, if .ssh dir doesn't exist you need to create it

myremote ~ # ls -la ~/.ssh
ls: cannot access /root/.ssh: No such file or directory
If you get something like this you need to create the dir, so:
myremote ~ # mkdir .ssh
myremote ~ # chmod 700 .ssh



5) Now copy your ssh public key into authorized keys file and delete it when finished, so:

myremote ~ # cat ~/id_rsa.pub >> .ssh/authorized_keys
myremote ~ # chmod 600 .ssh/authorized_keys
myremote ~ # rm id_rsa.pub
NOTE: If you've a Debian remote host you MUST use this instead:
myremote ~ # cat ~/id_rsa.pub >> .ssh/authorized_keys2
myremote ~ # chmod 600 .ssh/authorized_keys2
myremote ~ # rm id_rsa.pub
First row is used for all major distros (Gentoo in my real example), Debian users must use the second one, check your ssh man page for details on your setup (first is the most common case)



6) FINAL TEST
Ok let's go back to our local host and try to make something to see what happens:

mylocal:~# scp example.file root@myremote:/tmp/
example.file 100% 169 0.2KB/s 00:00
mylocal:~# ssh root@myremote
Last login: Wed May xx xx:xx:xx xxxx 2009 from mylocal on ssh
myremote ~ # ls -la /tmp/example.file
total 1
-rwxr-xr-x 1 root root 169 May xx xx:xx example.file
Did you see it ? I'll hope so.
As you can see you can copy or connect to host without supplying passwds




Note (read)

Sometimes additional configurations are requested on remote ssh daemon, this may vary from your distro setup and basic security configuration, if final test failed you'll probably have PublicAuthentication or RSA disabled.
In this case you need to change them, don't worry it doesn't affect or lower your current security, tipically this change is done by editing /etc/ssh/sshd_config file, you need root access for it.
sshd_config path may vary between different distro even it's the most common name

To get the correct configuration, see that the following attributes are set (not commented or set to "no") in your sshd_config file

RSAAuthentication    yes
PubkeyAuthentication yes

If you change sshd_config file with these values you need to restart ssh daemon (something like: /etc/init.d/sshd restart)

 

Hope it helps someone
Let me know if you need help or further suggestions
Andrea Benini Ben

 

SLES10

I still say: I love the Suse linux enterprise server software. It is easy to use and has a great integration with Wind##s server product. So till the moment where we can enjoy a 100% Linux enviroment an excellent choice.
 

Using SSHFS to remote mount a Windows System.

Try hard as we might, it's impossible to get away from the OS from Redmond.  All too often we need to be able to get a file off or onto a Windows system in a secure manor.  Samba won't work over the Internet, NFS isn't an option either over the internet.  Linux to the rescue, along with some FOSS for Windows. 

FUSE or File sytem in USEr space, allows Linux to create file systems of multiple kinds that can be mounted and manipulated by an unpriviledged user, meaning no root privileges are needed at any time.  SSHFS or SSH File System, is the type of file system we will use. Other Windows connectable file systems exist but since we are going over the open Internet and we want data integrity, and security I've chosen SSHFS for this.  I've tested it on win2k XP and 2003 server so far without a problem.

Step one: go to  http://sshwindows.sourceforge.net/download/ and grab the binary installation file for Windows. (Currently i386 version only) Install this on the Windows system you wish to mount from a remote location.  The process on how to install an application on Windows and create Users is an exercise I'll leave to the reader.  However to someone who is familiar at all with SSH running the install and answering it's questions should be straight forward.  NOTE: If you have cygwin installed you should install the SSH server with cygwins package manager as it will be less likely to cause a conflict. At this point, from your Linux box ssh into the windows box and make sure it all works.  Once it does.  You are done with the windows work.

 Step two:  If you are running RHEL/CentOS 5.x  the modules you need will already be installed in the running kernel, and the packages will be available in the normal repository.  RHEL/CentOS 4.x will require that you use either DAG's repository or do the normal configure/make/make install routine.  I highly recommend DAG's work as it's always rock solid, and his packages are correctly integrated into RH/CentOS system so as to not cause future conflicts.  Ubuntu and Deb (sarge and Lenny) both have FUSE available in the apt repository and the dependency will be brought in when you install SSHFS.  

Step three:  Install SSHFS With RHEL/CentOS you will need DAG's repository setup.  Then just yum install fuse-sshfs, or Ubuntu/Debian do apt-get sshfs.  All dependency's will be pulled.  The following should be taken care of before you begin.  Edit /etc/group (I use vigr) and add all users who you want to be able to use SSHFS into the group fuse.  Once this is done we are ready to test things out.

Now we get to mount our remote windows system.  Create a dir in your home directory to mount the remote system and then prepare to do a simple test run. 

 usage: sshfs [user@]host:[dir] mountpoint [options] 

 Is the proper format for running the command, and remember that if you skip the [user@] portion, just as in Linux it will assume that your current logged in user is the username to use on your remote server as well.  

 #> sshfs myname@mywinbox: /myhome/mountpoint  

This will grab the user home or C: (depending on System) mount it locally and give you access to read/write these files as if they were locally held.  Also at the same time any user on the Windows box will also be able to read/write the same file (no file locking is provided. ) so it's best to co-ordinate with the user on the other end if needed.  

At this point the movement of files will be at the same speed as a normal ssh, man sshfs can be read and used to fine tune the sshfs mount the most important from my viewpoint is the ability to set the UID and GID of the files so that the correct user has access to them.  Again since this isn't a comprehensive tutorial, I'll leave it to the reader to learn from the man page. 

 

Xen DomU clock issues

I just setup a Xen DomU (Debian Lenny) and found that when I tried to live migrate it between hosts, any pings would just hang.  I found my logs filled with clock warnings about going backwards in time.  Here is how I resolved this.

Read more... Comment (0)
 

Xen: Create a Jaunty DomU using xen-create-image

These are instructions for creating an Ubuntu Jaunty DomU on Debian Lenny Dom0 using Xen 3.2.1 that comes with Lenny.  I'm installing into an LVM rather than using a file image.  See this HowtoForge article for background of my particular setup.  This tutorial assumes you're working from a similar setup with an iscsi target and LVM-based VMs.

 The reason I'm posting this is because I had such a hard time finding some combination of tools that worked correctly to do this.  I tried a few options but had little success:

  • debootstrap - Didn't seem to work with Jaunty at all.  It would just crash
  • vm-install - Doesn't support LVM-based VMs by default, but there is a procedure for converting the qcow image to an LVM
What I found is that I could find no way of creating the LVM-based VM in Lenny.  I actually had to create a sid instance first and then install my open-iscsi and xen-utils into the sid VM, then use SID to populate the VM. 

Perform the following actions in your SID vm or workstation

Note:  You'll need your /etc/xen-tools/xen-tools.conf setup like the instructions in the HowtoForge article (things like gateway, netmask, broadcast, passwd, fs type, install-method, etc).

Whenever I would try to build the vm with the default partitions, it would always fail.  For some reason the resulting vm would have an fstab that shows something like sda1 and sda2 instead of xvda1 and xvda2.  I had to create my own partition scheme before anything would work right.  Here's mine:

$ sudo cat /etc/xen-tools/partitions.d/with-data
[root]
size=4G
type=ext3
mountpoint=/
options=sync,errors=remount-ro

[swap]
size=512M
type=swap

[data]
size=4G
type=ext3
mountpoint=/data
options=nodev

 You will need to create a symlink for jaunty so that the option is recognized at the command-line. 

$ cd /usr/lib/xen-tools

$ sudo ln -s edgy.d jaunty.d 

Now, finally, you can create the image.  This step may take some time

$ xen-create-image --hostname=myserver --dhcp --lvm=vg_xen --dist=jaunty --mirror=http://archive.ubuntu.com/ubuntu --size=4Gb --memory=512Mb --swap=512MB --arch=i386 --partitions=with-data

 Since you're building the LVM on a different machine than it will eventually run on, you'll need to  copy the resulting xen config to the correct server:

$ scp /etc/xen/myserver.cfg root@realhost:/etc/xen/

That should be all you need to get this working.

 

Remote port tunnelling with SSH

Hi there, here's a quick blog about SSH port forwarding, let's describe the scenario with an example, of course port forwarding may be applied to everythin, not only to mysql as reported in the sample

 

Assume you've a remote host with MySQL server installed and running, of course for security reasons you've forbidden TCP connections from every machine except localhost, or at least this is how I usually configure my services. Your Python, PHP, Java apps and even CLI apps are happy with it, they can access mysql backend by connecting to localhost on 3306 port.

For security reasons when you're inside the mysql server you can connect to my by using:
myserver:~$ mysql --host=127.0.0.1 --user= --password=
pretty safe and good, I usually configure MySQL in this way:
myserver:~$ cat /etc/mysql/my.cnf|grep "bind-address"
bind-address = 127.0.0.1

so far, everything is perfect now but if you need to manage your remote db with MySQL Administrator or with your preferred tool how can you connect to this machine ? Easy, let's forward remote 3306 port to local 3306 or other port if needed, then you can connect to localhost and use the SSH tunnel in between. from your local machine:
localmachine:~$ ssh -l -L 3306:localhost:3306
So you open an ssh console to your machine from your localhost, with the connection you ask remote to port forward its 3306 port to your local 3306.
Now try to open your remote db from localhost, so if you use mysql command line utility you need to type:
localmachine:~$ mysql --host=127.0.0.1 --user= --password=
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 254
Server version: 5.0.51a-24 (Debian)

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql>

And that's it !

Obviously you can even use your favorite admin tool, not only mysql cli

 

Pretty easy and quick

Hope it helps someone

Ben

 

Zeroshell Linux: Captive Portal, Internet Gateway and Router (part3)

Zeroshell Linux: Captive Portal, Internet Gateway and Router (part3) is the last part of Eric Geier's excellent series on using Zeroshell Linux to provide secure LAN services, such as wireless access point, name services, firewall, and lots more. Enjoy!
 

Preventing unauthorized SSH access using Denyhosts

Once when I was doing a regular tail -f /var/log/messages, I came across a number of messages like these.

sshd[29924]: PAM_NAM: User donk unknown to the authentication module
sshd[29924]: Failed password for invalid user donk from 'IP address here' port 63410 ssh2

My SSH was under continuous attack! . Hmm.., until I found DenyHosts..

DenyHosts is a cool little python script by Phil Schwartz, which will parse the logs and identify repeated authentication failures and add the IP address of the offenders to /etc/hosts.deny, thus preventing them to connect to the server in the first place.

Installation

As the program was not available in the official repositories for SLES 10 SP1, I had to do some manual configuration. The installation steps were detailed in the ‘Readme.txt' file within the package.

First, the python-devel package has to be installed. It is not installed by default

zypper install python-devel

Download the latest version of DenyHosts from http://denyhosts.sourceforge.net/

The version available at the time of my setup was 2.6. After uncompressing the sources

tar zxvf DenyHosts-2.6.tar.gz

cd DenyHosts-2.6

python setup.py install

The above step install the scripts and config files in /usr/share/denyhosts and in the site-packages of the python directory.

Configuration

Before proceeding the file denyhosts.cfg must be edited to suit the installation environment.The example config file is fully commented so it should be easy to follow. I had the following config

#/usr/share/denyhosts/denyhosts.cfg

SECURE_LOG = /var/log/messages
HOSTS_DENY = /etc/hosts.deny
LOCK_FILE = /var/run/denyhosts.pid

After this, I did the following step (as mentioned in the readme) to run denyhosts as a daemon during system start.

cd /usr/share/denyhosts

chmod 700 daemon-control

ln -s /usr/share/denyhosts/daemon-control /etc/init.d/denyhosts

/etc/init.d/denyhosts start

tail -f /var/log/denyhosts # will contain messages related to the start

If it is working as intended, enable it to start automatically by doing

chkconfig denyhosts on

It had happend occassionally that some valid IP's are listed in /etc/hosts.deny. To prevent this, the genuine IPs from which users connect can be added to a file called ‘allowed-hosts' in /usr/share/denyhosts/data. There is no specific format. Just add the IPs to the file one below the other. Also, edit denyhosts.cfg to change the following variable and restart denyhosts.

ALLOWED_HOSTS_HOSTNAME_LOOKUP=YES

That's it..

 
Page 6 of 8

Upcoming Linux Foundation Courses

  1. LFS426 Linux Performance Tuning
    21 Apr » 24 Apr - Virtual
    Details
  2. LFS520 OpenStack Cloud Architecture and Deployment
    05 May » 09 May - Virtual
    Details
  3. LFD320 Linux Kernel Internals and Debugging
    12 May » 16 May - Virtual
    Details

View All Upcoming Courses


Who we are ?

The Linux Foundation is a non-profit consortium dedicated to the growth of Linux.

More About the foundation...

Frequent Questions

Join / Linux Training / Board