Linux.com

Feature

Backing up MySQL data

By Tom Adelstein and Bill Lubanovic on May 29, 2007 (8:00:00 AM)

Share    Print    Comments   

Backing up files and directories is relatively easy; databases, however, have some special quirks that you need to address. Our examples use MySQL, but the same principles apply to PostgreSQL and other relational databases.

This article is excerpted from O'Reilly's recently published book Linux System Administration .

If your MySQL server does not need to be available 24x7, a fast and easy offline raw backup method is:

  1. Stop the MySQL server:

    # /etc/init.d/mysqld stop
  2. Copy MySQL's data files and directories. For example, if your MySQL data directory is /var/lib/mysql and you want to save it to /tmp/mysql-backup:

    # cp -r /var/lib/mysql /tmp/mysql-backup

    Instead of cp, you can use rsync, tar, gzip, or other commands.

  3. Start the server again:

    # /etc/init.d/mysqld start

Online backups are trickier. If you have mutually independent MyISAM tables (no foreign keys or transactions), you could lock each one in turn, copy its files, and unlock it. But you may have InnoDB tables, or someone could write a transaction involving multiple tables. Fortunately there are several reasonable noncommercial solutions, including mysqlhotcopy, mysqlsnapshot, replication, and mysqldump.

mysqlhotcopy is a Perl script that does online raw backups of ISAM or MyISAM tables. The manpage includes many options, but here's how to back up a single database named drupal:

# mysqlhotcopy -u user -p password drupal /tmp
Locked 57 tables in 0 seconds.
Flushed tables ('drupal'.'access', 'drupal'.'accesslog', 'drupal'.'aggregator_
category', 'drupal'.'aggregator_category_feed', 'drupal'.'aggregator_category_item',
'drupal'.'aggregator_feed', 'drupal'.'aggregator_item', 'drupal'.'authmap', 'drupal'.
'blocks', 'drupal'.'book', 'drupal'.'boxes', 'drupal'.'cache', 'drupal'.'client',
'drupal'.'client_system', 'drupal'.'comments', 'drupal'.'contact', 'drupal'.'file_
revisions', 'drupal'.'files', 'drupal'.'filter_formats', 'drupal'.'filters',
'drupal'.'flood', 'drupal'.'forum', 'drupal'.'history', 'drupal'.'locales_meta',
'drupal'.'locales_source', 'drupal'.'locales_target', 'drupal'.'menu', 'drupal'.
'node', 'drupal'.'node_access', 'drupal'.'node_comment_statistics', 'drupal'.'node_
counter', 'drupal'.'node_revisions', 'drupal'.'permission', 'drupal'.'poll',
'drupal'.'poll_choices', 'drupal'.'poll_votes', 'drupal'.'profile_fields', 'drupal'.
'profile_values', 'drupal'.'role', 'drupal'.'search_dataset', 'drupal'.'search_
index', 'drupal'.'search_total', 'drupal'.'sequences', 'drupal'.'sessions', 'drupal'.
'system', 'drupal'.'term_data', 'drupal'.'term_hierarchy', 'drupal'.'term_node',
'drupal'.'term_relation', 'drupal'.'term_synonym', 'drupal'.'url_alias', 'drupal'.
'users', 'drupal'.'users_roles', 'drupal'.'variable', 'drupal'.'vocabulary',
'drupal'.'vocabulary_node_types', 'drupal'.'watchdog') in 0 seconds.
Copying 171 files...
Copying indices for 0 files...
Unlocked tables.
mysqlhotcopy copied 57 tables (171 files) in 1 second (1 seconds overall).

mysqlsnapshot is even easier. It backs up all the ISAM or MyISAM tables on your server to one tar file per database:

# ./mysqlsnapshot -u user -p password -s /tmp --split -n
checking for binary logging... ok
backing up db drupal... done
backing up db mysql... done
backing up db test... done
snapshot completed in /tmp

If you've set up MySQL replication for 24x7 availability, you can back up from a slave server using one of the methods just decribed. You'll also need to save replication info (logs, configuration files, and so on).

For extra protection from hardware corruption (but not human error), set up replication and provide your slave (and/or master) with RAID 1 (mirrored) disks.

Many MySQL sites migrate data from MyISAM to InnoDB tables to get true database transactions and better write performance. The authors of the InnoDB module have a commercial product for online InnoDB backups named InnoDB Hot Backup.

The last method is usually the first mentioned in most documentation: mysqldump. Rather than a raw (verbatim) copy, mysqldump produces an ASCII dump of the specified databases and tables. It works with all MySQL table types, including InnoDB. It's relatively slow, and the text files it produces are large, although they compress fairly well. It's useful to create these dumps from time to time, because they contain a straightforward script for recreating your databases and tables from scratch. You can use editors, grep, and other text tools to search through or modify the dump files.

To lock all of your tables and dump them to a single file, enter:

# mysqldump -u user -ppassword -x --all-databases > /tmp/mysql.dump

You can pipe the output through gzip to save some time and space:

# mysqldump -u user -ppassword -x --all-databases | gzip > / tmp/mysql.dump.gz

A new open source tool (free download, pay for support) called Zmanda Recovery Manager for MySQL provides a useful front end to many of these alternatives. Notable features:

  • Has a command-line interface.

  • Backs up local databases, or remote databases over SSL.

  • Emails the status of the backup procedure.

  • Handles all table types, including InnoDB.

  • Does not provide any new backup methods. Instead, it chooses among mysqldump, mysqlhotcopy, MySQL replication, or LVM snapshots.

  • Supports recovery to a particular transaction or point in time.

Zmanda provides .tar.gz and .rpm files for many Linux distributions. For an installation how-to for Debian, see HowtoForge.

Share    Print    Comments   

Comments

on Backing up MySQL data

Note: Comments are owned by the poster. We are not responsible for their content.

Additionally... if I may... this is crucial

Posted by: Anonymous Coward on May 29, 2007 04:41 PM
Wow: database maintenance is now mainstream!

Anyone running any form of CMS/CRM or Wiki should take note of this article (and maybe even read my comments)

Everyone should be aware that doing a plain database file copy *may work* on certain databases even while the database server is still running, if there is no activity on the database, all writes have been flushed to disk, and the file system allows read access to the database file. This is very dangerous because all the assumptions may not be valid during the lifetime of the copy process (for a large database this can easily be several minutes).

Most RDBMS's come with command-line tools to do live backups, essentially a script to start a new transactional snapshot, get a copy of full metadata, then get a copy of full data typically ignoring in-limbo or deleted transactions / content. If this live-backup fails, it may indicate on-disk errors (due to various reasons) and this should be a trigger for the database maintainer to prepare for pre-disaster recovery.

So a regular live-backup scheduled and run automatically but upon failure notifying some administrator is invaluable.

Restore-testing is also prudent, since a "successful backup" is only really worth anything if it can be restored, so modify your backup script to not only do a backup, but also a restore to a temporary file (immediately) to see if the backup file is uncorrupted. If this process is successful, delete the "temp restored database", and make sure you make an off-site copy of the backup, otherwise you're not insured against disk failure.

I hope someone may find this addition useful, and I apologise if I offended anyone in posting this.

Donald Klopper
donald.klopper@gmail.com

#

Re:Additionally... if I may... this is crucial

Posted by: Anonymous Coward on May 29, 2007 05:40 PM
What I found dubious is that the backup copy is directly placed into the<nobr> <wbr></nobr>/tmp folder. Some systems have that folder under public read/write permissions wich makes the whole copy of your database accessible by any user if you're out of luck.

I guess the author ment to use the example<nobr> <wbr></nobr>/tmp as any temporary directory usable for a backup, but please don't take this example as the best practice.

- Unomi -

#

Re:Additionally... if I may... this is crucial

Posted by: Anonymous Coward on May 29, 2007 05:47 PM
What I forgot to mention is that with these command-line tools (mysqlhotcopy, mysqldump etc.) you pass a username and a password directly to the command. When executed, this command ends up in the processlist (try ps aux for example) with the credentials fully readable.

When your backup takes a while, this display of credentials will be up there as long as it takes. So, be cautious doing so. You better make a<nobr> <wbr></nobr>.my.cnf in the user's home directory under wich you make these backups. Place your needed credentials and parameters in there so the are kept hidden while the backup is executed and visible in the processlist.

- Unomi -

#

Re:There's a lot more to it than that

Posted by: Anonymous Coward on May 30, 2007 12:35 AM
also a good tool: <a href="http://sourceforge.net/projects/automysqlbackup" title="sourceforge.net">http://sourceforge.net/projects/automysqlbackup</a sourceforge.net>

#

Re:There's a lot more to it than that

Posted by: Anonymous Coward on May 30, 2007 08:36 AM

You may be saying "yeah, but the backup took 1 second for all 57 tables." True, but we don't know how much data was in those 57 tables. How long does it take you to copy 100 megs of data? 100 gigs?



This is a good point, and it's why I store my mysql data on an lvm logical volume. I can flush the tables, create a snapshot of the logical volume, and then unlock the tables all in under a second regardless of the data size. I then have as long as I want to backup the data.



FWIW I found it quite easy to roll my own script, rather than using one of the ones mentioned in the article.

#

how to restore the backup

Posted by: Anonymous Coward on May 30, 2007 07:00 PM
Hi

I think it would be good to point out how
to restore the backup. I finally found out that
date I backup with myslqdum say for the DB wikidb
I have to restore with

mysql -D wikidb dump.sql

#

There's a lot more to it than that

Posted by: Administrator on May 30, 2007 12:22 AM
The astitute reader will recognize that the mysqlhotcopy script does, in fact, lock all tables involved in the backup, which may or may not have a serious impact on the users/applications using the database (although the output doesn't show it, the mysqlsnapshot script does a "FLUSH TABLES WITH READ LOCK", too):

Locked 57 tables in 0 seconds.
[...]
Unlocked tables.

You may be saying "yeah, but the backup took 1 second for all 57 tables." True, but we don't know how much data was in those 57 tables. How long does it take you to copy 100 megs of data? 100 gigs?

The highly astitute reader will also notice that mysqlhotcopy and mysqlsnapshot only work for ISAM and MyISAM tables, not Innodb tables. If you want transaction and foreign key support, you must use InnoDB tables - which means neither of these backup methods will work for you.

mysqldump will work with any storage engine since it makes a logical backup. It writes SQL statements, which can be pumped right back into MySQL.

Replication is, by far, the most flexible method of making online, consistent, reliable backups without affecting your database's performance. This is akin to Oracle Data Guard, where the data is replicated to another physical server. To make the backup, simply stop replication, flush the logs, then you can:

- shutdown MySQL, perform a cold backup of everything (data files, config files, logs, etc.)
- use mysqldump to get a logical backup (you don't have to worry about users/apps since they don't touch that database anyway)

When the backup is complete, simply start replication back up.

One thing the article doesn't make clear is InnoDB Hot Backup can make online, hot-backups *without locking* tables using the InnoDB storage engine, only. If you have some tables using any of the other engines, MyISAM, BDB, CSV, etc., then you will incur locking issues and may or may not get a consistent snapshot of the data.

It's also licensed on a per-server basis.

#

This story has been archived. Comments can no longer be posted.



 
Tableless layout Validate XHTML 1.0 Strict Validate CSS Powered by Xaraya