Linux.com

Community Blogs



One shot RSync between two hosts without installing services

I was syncing two Linux hosts, just need to copy data between them. I don't need and neither don't want to install Samba as a Service for a quick sync, RSync was my preferred method.

RSync is very easy to use and quick but in this case I don't either want to install RSync service daemon as well on the destination machine (the machine with data to transfer), so I've opted for rsync through ssh tunnel without a service installed.

Here's a quick sample:

- You've to transfer data from remote host machine (name: "oldhost")
- You've to copy data into another machine (name: "newhost")
- You don't want to install/configure rsync daemon on these two machines
- You've at least ssh server access to "oldhost" from "newhost"
- You've at least rsync program installed on "newhost"

Ok, don't configure rsync daemon on these two machines, just login to "newhost", go to target directory (the directory where you'd like to have data copied) and issue a command like this:

newhost:/target# rsync --verbose --recursive --copy-links -perms --owner --group --compress --specials --stats --devices --times --delete -e "ssh -l root" oldhost:/source/ .

Please substitute /target with your target directory on newhost, /source with your source directory from oldhost, root with your favorite username on remote host

This command uses rsync through ssh shell on remote host and copies data from there to local host on specified directory (current directory as latest "."  on command reported)

You can substitute "." with local target directory as needed

 

In this way rsync is just used without the service (rsyncd), as a new blog I can create few easy steps for rsyncd configuration if you need it.
I always use this method for one shot synchronization, when your rsync operation are scheduled of programmed periodically it's better to use rsyncd service, this is my favorite backup system

Glad to read your comments

 

Andrea Ben Benini

 

Samba public users directory (quick howto)

This quick post shows you how to create a samba share for a network, every user is forced to a specific username and each file belongs to this username. This is useful when dealing with public folders for some sort of exchange between users in a network

Read/Write access  to everyone for directories and files, this is a tipical configuration for a swap area

Check it out:

[public]
comment = Public folder for my network
available = yes
browseable = yes
path = /home/public
guest ok = yes
public = yes
writable = yes
write list = *

force group = commongroup
force user = commonuser
create mask = 0644
directory mask = 0755

printable = no

 

Comments welcomed

 

Ben 

 

Transparent dynamic reverse proxy with nginx

A while back I wrote about using Apache as a dynamic reverse proxy. Anyone who has done even minimal research into web servers knows that Apache is the swiss army knife. It trys to be everything for everyone, and like a swiss army knife may not be as good as a more refined too at least as far as efficiency is concerned. (Read More)
 

Installing Oracle on Linux

Installing Oracle  on Linux


Requirements

  1. Create Users and groups

  2. Creating directories for installing oracle

  3. Change Kernel Parameters

  4. Install Oracle


  1. Create User and groups


#groupadd oinstall

#groupadd dba

#useradd –m –g oinstall –G dba oracle

#passwd oracle


  1. Creating directories for installing oracle

For Applicaiton

#Mkdir –p /u01/app/oracle

#Chown –R oracle:oinstall /u01/app/oracle

For data

#mkdir -p /u02/oradata

#chown –R oracle:oinstall /u02/


  1. Change Kernel Parameters

Sysctl file

# vi /etc/sysctl.conf

kernel.shmall = 2097152

kernel.shmmax = 536870912

kernel.shmmni = 4096

kernel.sem = 250 32000 100 128

fs.file-max = 65536

net.ipv4.ip_local_port_range = 1024 65000

net.core.rmem_default=262144

net.core.wmem_default=262144

net.core.rmem_max=262144

net.core.wmem_max=262144


Limits file

#vi /etc/security/limits.conf

oracle soft nproc 2047

oracle hard nproc 16384

oracle soft nofile 1024

oracle hard nofile 65536



Login file

#vi /etc/pam.d/login


session required /lib/security/pam_limits.so


profile file

#vi /etc/profile

if [ $USER = "oracle" ]; then

if [ $SHELL = "/bin/ksh" ]; then

ulimit -p 16384

ulimit -n 65536

else

ulimit -u 16384 -n 65536

fi

umask 022

fi



  1. Install Oracle

# cd /opt/database

#./runInstaller

  1. Advance installation

  2. Enterprise edition

  3. Application path  /u01/app/oracle/product/10.2.0/db_1

  4. Create database

  5. Transaction processing

  6. Global database (gtldev.localdomain sid gtldev)

  7. Unicode standard utf-8 AL32F8

  8. File system storage ( /u02/oradata/

  9. Do not enable backup

  10. Password

  11. Install

After installation run two scripts

  1. # cd /home/orainventory

  2. #./orainstroot.sh

  3. #cd /u01/app/oracle/product/10.2.0_db1/

  4. #./root.sh


Web Console


http://192.168.*.*/em





 

WebServices Wars: "The Phantom Menace": Creating the WebService WSDL file

WebServices Wars: "The Phantom Menace": Creating the WebService WSDL file


Now if you've followed the introduction "WebServices Wars: Creating a basic webservice using Eclipse, php and apache" you're now ready for Episode I

Intro

This webservice saga begins with the base of every good project: the planning and design phase.
Here's what I'm going to do:
  • Create a WSDL definition file for the new service, it will be called sample.wsdl, it will be accessible from the following URL: http://my.website.com/sample.wsdl
  • Create a webservice server called sample.php it will provide sample.wsdl implementation, it will act as a service for etherogeneous applications requiring for it from this url: http://my.website.com/sample.php
  • Create a webservice client file, this is just for seeing some output and understand how php can access services, it doesn't matter if you plan to use php on the backend server or not. The url could be http://my.website.com/client.php
  • Make some considerations on debugging techniques and caching in development environments and production areas

This example will use SOAP techniques, SOAP is quick and easy to use, PHP already provide native SOAP classes, they're fast and well written in low level, not an external php library for SOAP. Personally I don't like NuSoap or third party libs for PHP, native support is reliable, fast, documented and stable.

I'll assume you already know what a webservice is, what you can do with it and how to use it in a real environment, please read this intro if you need it, it's a nice starting point.

As W3C suggests you can create a webservice with or without WSDL definition file, but if you're planning a big project or you just want to make things easy and well documented I suggest you to use WSDL files from the beginning, it's more easy to understand internals and speeds up your programming

Interface

Our sample webservice will provide these methods:

Method 1

twostringinput

input:

(string) param1

input:

(string) param2

output

(boolean) reply


Method 2

noinput_stringoutput

input:

(void: nothing)

output

(string) reply


These two methods are just samples to understand how you can use them, some sort of Hello World service.

Let's start

Now we need to write the webservice definition file (sample.wsdl), this file defines these two methods and their interfaces, the webservice url and other useful things, more documentation on WSDL can be found here from the official site.
You can obviously write this sample by hand, it's not difficult but when you do it the first time you can run into roubles if you don't know W3C documentation well. If you use Eclipse you already have a powerful tool called "The WSDL Editor", this amazing toy can create your WSDL files from a GUI interface in few minutes without orrying too much about WSDL internals, it's fast, intuitive and easy.
I'll suggest you to install the Eclipse Web Standard Tools (WST plugin) as well as WSDL tools and editors (WSDL), hese packages provides you HTML validators, html code completition, debugging tools, editors (web and wsdl) and lot of other useful functions.

Next episode will require PDT (PHP Development Tools) (Eclipse PDT plugin) so if you collect eclipse and these extensions you'll have a complete PHP/WebServices development tool, check out requirements from introduction article

When you're a beginner and you want to create a WSDL file with Eclipse you're probably looking for something good, already working and editable with WSDL Editor with no hassles, when I was googling around for wsdl files the first time I've found a lot of non-W3C compliants, malformed, wrong files and I've lost a lot of time to understand where errors were located, here's my sample for you.

According to my example and the two methods planned above here's the good W3C Compliant WSDL file:



























































transport="http://schemas.xmlsoap.org/soap/http" />


soapAction="http://my.website.com/twostringinput" />









soapAction="http://my.website.com/noinput_stringoutput" />













Create a new wsdl file (Eclipse: File, New, File) and name it sample.wsdl
Save the file and close it so you've this working sample.

When WSDL Editor is installed you can directly open it in GUI mode (WSDL Editor Itself),
it's easy to read, maintain, rewrite and modify, take a look at the following picture:


Amazing, isn't it ?
With this tool you can see in a nice gui mode your webservice, easy for a newbie and for an expert as well. When you select an item you can select and modify its properties in the properties window, take a look at my sample data and see where they are located


If you click on the arrows in the right side you can open another window with input parameters for the two methods, take a look at the picture below

Here are the methods



Play with the file, it's a good starting point for future projects, it was created with Eclipse 3.4 Ganymede and latest tools synced from Eclipse central repository, you can easily open it with outdated eclipse versions or without WSDL editor as well; Eclipse doesn't have backwards compatibility issues (like other IDEs)

When finished just put this file in your webserver and make it accessible from outside, in my example this file it's located in /var/www/htdocs/sample.wsdl, according to my current Apache2 config this file is available from the following url: http://my.website.com/sample.wsdl, sample.php (webservice server) and client.php (webservice client) will refer to this url for getting WSDL properties

For a closer look at WSDL Editor take a read at the WSDL Editor Documentation, it's THE starting point for everything, refer to W3C for reliable SOAP and WSDL documentation


Stay tuned for the next episode...

Introduction: "WebServices Wars: Creating a basic webservice using Eclipse, php and apache"

Episode I "The Phantom Menace": Creating the WebService WSDL file"

Episode 2 "Attack of the Clones": Creating the WebService php Server (coming soon)

 

 

 

WebServices Wars: Creating a basic webservice using Eclipse, php and apache

Here I am,
That's a lot of time since my last blog, well, I was quite busy with something else (still have my private life ok ?) but now I'm back again with WebServices, PHP, Eclipse, Apache.

A lot of folks are asking me about some sort of short howto/sample/easy doc for dealing with webservices and php, there's a lot of information spread all around the web so I've decided to collect information from my projects and write down some notes for a quickstart howto with webservices and php.

Now if you like to write down everything from scratch you can surely do but if you're working on a big project you're surely already using an IDE of some sort.

I've tried a lot of different commercial and open source IDEs and after a lot of evaluation I've choosen Eclipse because it's mature, stable (quite), reliable, studied for big enterprise class projects ...and I've a fast and huge pc with a ton of ram on it.

Eclipse is reliable, well known and supported, one of its major drawbacks are about system resources, it eats a lot of ram but if you've a recently updated PC with a good linux distro and a good amount of ram you can surely use it and you'll never go away from it. PHP folks will surely install PDT (http://www.eclipse.org/pdt/): PHP Development Toos Project for Eclipse, it's now a mature plugin for eclipse, reliable and suited for php professional development.

I don't wanna bother you about details on configuring eclipse/pdt/apache/php and so on (maybe another blog argument ?) I'll assume you've this configuration :

  • Eclipse and PDT up and running in your workstation, don't care about operating system but of course if you're using linux it's better :-) (gentoo linux like me even better). But I really don't care, just need Eclipse up and running
  • Web Tools Platform plugin for Eclipse, not really required but useful, it provides: web page editor inside Eclipse, WSDL editor, HTML Validators. WSDL editor is very important if you wish to create webservices with a GUI interface, if you're mastering xml/soap/rpc files and you want to write them on your own you're free to do this.
  • A webserver with php extension installed, Apache2 and PHP5 are used in this sample but every webserver suitable for php is good (here again, apache2 and php5 are better than everything else)
  • Common PHP knowledge and some Object Oriented programming skills, here used for these samples
  • Basic SOAP and WebServices knowledge, at least you need to know what they can do and what is this technology for, later you'll better understand soap after reading samples

This blog is just an intro and a "bill list of materials" before getting started

 

This is an intro, next article will be the first "hands on" with eclipse and WSDL creation
As usual feel free to ask and write down some comments when needed, hope it helps


Next, featuring: Episode 1: WebServices Wars: "The Phantom Menace": Creating the WebService WSDL file


Andrea Benini (Ben)

 

 

8 Ways to Fix Slow Email (Part 1)

Email Server Distance MapIn two earlier articles, we discussed ways to make your website load faster. Another issue many users face is the sometimes painfully slow speed of email. "What do you mean, you haven't gotten it yet? I sent that email over an hour ago?" Does this sound familiar? There are several steps that must take place between your hitting the "send" button and the recipient viewing your message. A delay in your email's journey can occur at one or more of these steps.

Part 1 of this article will address ways to optimize the speed of your network connection, in addition to the website-specific tips we focused upon in "9 Tips to Make Your Website Load Much Faster." These suggestions may increase the speed at which you can send and receive email, as well as improving the performance of your web pages and other file transfers.

1. Problem: Delays caused by the distance between you and your web server.

Solution: As with your website, you can remove one potential cause of slow email by selecting a web hosting provider that has a data center near you. This is the single most important thing you can do to increase your email speed.

While you can't control how long it will take your email to reach your recipient once it leaves your email server, or how long a message from another person will take to reach your email server, you can minimize the time it takes incoming emails to get from your server to your mailbox, and vice versa for outgoing emails.

For example, let's say Tony Traceroute, an employee in your Newark office, wants to send a large file - a CAD diagram of his new widget design - by intraoffice email to Patty Ping, who works just down the hall from Tony. If your company's server is located in Los Angeles, then when Tony hits the send button, his email will travel all the way to sunny California, through whatever routers and packet switches it encounters along the way, then all the way back to Delaware - an odyssey of some 5,000 miles - before it arrives in Patty's inbox! It might have been faster if Tony had just printed out the diagram and walked down the hall to drop it off!

The bottom line: if you rely heavily on email in your business, choosing a web hosting company with a nearby data center may save you a considerable amount of time and money.

2. Problem: Overuse of server capacity on shared servers. Every user on a shared server is allocated a certain amount of bandwidth and other server resources. When a user goes over this limit, it can slow down the other accounts on the server.

Solution: Make sure your web hosting company has policies in place to manage excessive usage by individual clients on its shared servers. Some hosting providers automatically suspend accounts that use excessive resources. While this works, the best way for a web hosting company to handle these accounts is to migrate them to a separate server where they can't negatively impact the performance of other shared accounts. If you're on a shared server, ask your web hosting company how they handle such accounts.

In addition, talk to your web hosting company to make sure that they aggressively combat spammers and other system abusers who are likely to consume system resources and slow down your website and email.

Rerouting Data3. Problem: Your email may encounter a bottleneck if there is an issue with the bandwidth provider that your web hosting company uses. Most large web hosting companies have partnerships with multiple major bandwidth providers to ensure this does not happen. Smaller hosts may not be able to afford these partnerships, leaving you susceptible to a slowdown.

Solution: Find out what types of ISP partnerships your web hosting company has. You can run a simple trace route test to see what partnerships your hosting company has with bandwidth providers. Look for major communications companies (Tier 1 or highly reputable Tier 2 providers) such as MCI-Verizon, AT&T, Mzima, Level(3), Global Crossing, etc.

Ideally, your web hosting company should have a variety of bandwidth partners. That way, your web host will be able to reroute email data quickly if one of the bandwidth providers goes down. If your web host does not have major partnerships, it may be in your best interest to look for a web hosting company that does.

Speedtest Results4. Problem: Connection problem on your end - If you're using "home-grade" equipment, and you think your connection may be slow, try running a speed test at a site such as speedtest.net. Sometimes modems and routers will go out of sync after being powered on for weeks or months at a time. This is especially true if you are using a modem and wireless router combination.

Solution: Try refreshing your connection. Simply power down both your modem and your router. Then turn the modem back on, wait a few minutes, and turn the router on (the order of these steps is very important).

If this doesn't increase your connection speed, notify your cable or DSL provider. They will test your connection speed, and they may be able to optimize your connection from their operations center.

Conclusion


• Be sure that you are located as close as possible to your web hosting company's data center.
• Ask your provider about their excess usage policies if you are hosted on a shared server.
• Make sure your web host has partnerships with the top bandwidth providers.
• Conduct basic tests with your internet provider and on your own home router/wireless setup.

 

Part 2 of this article will focus on optimization techniques that are specific to email. We'll look at issues that can slow email such as spam, misconfigured spam filters, bloated emails, and reverse DNS lookup - and what you can do to address these problems.

 

Qwiki : Order IP Addresses in MySQL

Recently I came across unexpected results when trying to order a table by stored IP Addresses.

mysql> select ip from table order by ip desc;

The results looked like this:

...

| 10.5.0.91 | | 10.5.0.90 | | 10.5.0.9 | | 10.5.0.89 | | 10.5.0.88 | ...

| 10.5.0.81 | | 10.5.0.80 | | 10.5.0.8 | | 10.5.0.79 | | 10.5.0.78 | ...

Here's the quick fix ...

Read more... Comment (0)
 

Disable/Enable ping response on a Linux machine

Few days ago I was dealing with SYSCTL (man sysctl) utility and I was looking for a certain kernel parameter, I wish to set it on the fly and I've found other useful information too.

sysctl is used to modify kernel parameters at runtime, one of these parameter could be ping daemon response, if you want to disable ping reply on your network you just simply need to issue something like:

~# sysctl -w net.ipv4.icmp_echo_ignore_all=1
net.ipv4.icmp_echo_ignore_all = 1
(root access required)

Now try to ping your machine, no replies at all, look nice isn't it ?
To  re-enable ping reply just issue:

~# sysctl -w net.ipv4.icmp_echo_ignore_all=0
net.ipv4.icmp_echo_ignore_all = 0
(root access required)

-w flag is used if you want to change some settings, take a look at kernel flags you can set at runtime (linux sources)

Ben 

 

File transfer with Netcat

There's a basic usage for netcat helping you to transfer data/files across hosts without ssh or ftp services

Here's what I usually do on server (example IP: 1.2.3.4)

nc -l -p 6666 | uncompress -c | tar xvfp -

Basically I open TCP port 6666 (or whatever you want) on server and listen for raw compressed data

 

On my client I need to issue a command like this

tar cpf - /my/cool/dir | compress -c | nc -w 3 1.2.3.4 6666

Where /my/cool/dir is what I want to transfer, 1.2.3.4 is the destination host (server) and 6666 is the previously opened TCP port; you can even set a timeout (in seconds) for connection, suggested if you're in a busy network

 

Data on this stream are compressed so faster downloads are possible, this kind of trick is useful when you need to transfer data on the fly or when you don't have ssh shell host, scp or ftp server; I use it heavily expecially on embedded devices or dumb machines

 

Hope it helps

 

Andrea Ben Benini

 

Shared Samba Printer for PDF/PS Storage

As many of you already know I'm experiencing some sort of hylafax/samba integration for a mixed Windows environment, this blog is for creating a PDF/PostScript printer easily, no matter of what you do with the printer; when you print on it you'll produce a Postscript/PDF document

requirements:
A samba server installation up and running

achievements:
Create a PostScript/PDF document from a spool so every Windows user in your net can generate documents without client side software installation (Exception for the windows printer driver)

notes:
If you want to create your PDF files and customize them each time (custom passwd protection, comments, tags) you'd probably use PDFCreator, it has a windows installation and a lot of funny options to deal with. If you're searching for a dumb massive pdf generation for average windows users (lazy users without knowledge and time) you'd better stay here to see my config.

I'm using it for a Windows network where users just need to use a document management and storage system, they simply want to archive files on storage array in PDF format, no matter for file input (DOC, XLS, EMails, ...). This is particularly useful in big environments and when input file may vary, here's a nice Windows spooler usage

After installing samba and integrating it in your network (Workgroup, Domain, Active Directory, ... need some help or need a new blog for it ?) just add a section to your /etc/samba/smb.conf (or wherever your samba config file is) like this one:

# SAMBA PDF SHARED PRINTER 
# for windows clients
[pdfprinter]
comment = my cool samba printer
public = yes
print command = ( ps2pdf "%s" "/my/storage/dir/`date +"%Y-%m-%d.%H-%M.pdf"`" > /dev/null; rm "%s" ) &
printable = yes
writable = no
path = /var/spool/samba
use client driver = yes

This config was taken from a running server, exception for the print command line, in my case I've a custom script making other things, in this sample when a file is received it will be converted from PS (input) to PDF (output) and temp PS file is then deleted.

PDF in the example will have this name /my/storage/dir/YYYY-MM-DD.HH-MM.pdf
change it to whatever you want

 

Now the big question is: what should I have to install on the windows side ?
Here's the easiest part of the trick.
Install a "Apple LaserWriter 12-640 PS" printer, common alternatives are:
"Apple LaserWriter 12-600 PS"
"HP Laserjet 4/4M PS"

I strongly suggest you to install Apple Laserwriter printer family, these are already supported and driver already available in Windows XP, You don't even need to supply Windows install CD

Vista users may choose HP Laserjet (no cd required, already installed), Laserwriter drivers are not there

These printers are PostScript printers so each file you send to them will be translated to postscript language (thanks to Windows Printer Spooler).

 

I use this system to store files in a common and unique format (PDF) in a mixed environments when different types of files exists and problems are raised when you only need to read a simple document

Hope it helps
Glad to read your comments

 

Andrea Ben Benini

 
Page 5 of 8

Upcoming Linux Foundation Courses

  1. LFS426 Linux Performance Tuning
    21 Apr » 24 Apr - Virtual
    Details
  2. LFS520 OpenStack Cloud Architecture and Deployment
    05 May » 09 May - Virtual
    Details
  3. LFD320 Linux Kernel Internals and Debugging
    12 May » 16 May - Virtual
    Details

View All Upcoming Courses


Who we are ?

The Linux Foundation is a non-profit consortium dedicated to the growth of Linux.

More About the foundation...

Frequent Questions

Join / Linux Training / Board