Chinaunix

标题: ［每周讨论专题］－－第六期－－WEB服务器的数据远程同步 [打印本页]

作者: HonestQiao 时间: 2004-06-13 11:00
标题: ［每周讨论专题］－－第六期－－WEB服务器的数据远程同步
［每周讨论专题］－－第六期－－WEB服务器的数据远程同步
每周讨论专题【第六期】.......................................................点这里查看其他讨论专题

　　WEB服务器的数据远程同步

本期讨论主旨，WEB服务器的数据远程同步！！！

感谢 wingger 点题
本期主持：wingger

作者: wingger 时间: 2004-06-21 20:20
标题: ［每周讨论专题］－－第六期－－WEB服务器的数据远程同步
两个WEB远程同步可以吗

或是如何负载匀横

作者: 不想发言 时间: 2004-06-24 21:35
标题: ［每周讨论专题］－－第六期－－WEB服务器的数据远程同步
I think only web can't do so.maybe you can use scp or sftp/ftp do so.

作者: xxxs 时间: 2004-06-25 14:38
标题: ［每周讨论专题］－－第六期－－WEB服务器的数据远程同步
如果是数据同步，而且使用了数据库的话，可以考虑采用数据库的同步复制功能；
当然也可以对web系统自己预留接口，定期自动查询同步。最笨了，而且性能不好控制

作者: gig2600 时间: 2004-06-28 14:21
标题: ［每周讨论专题］－－第六期－－WEB服务器的数据远程同步
rsync这个软件可以实现mirror.

作者: wingger 时间: 2004-07-07 10:08
标题: ［每周讨论专题］－－第六期－－WEB服务器的数据远程同步
使用rsync

下面是rsync网站上的一些例子

每隔七天将数据往中心服务器做增量备份

#!/bin/sh

# This script does personal backups to a rsync backup server. You will end up
# with a 7 day rotating incremental backup. The incrementals will go
# into subdirectories named after the day of the week, and the current
# full backup goes into a directory called "current"
# tridge@linuxcare.com

# directory to backup
BDIR=/home/$USER

# excludes file - this contains a wildcard pattern per line of files to exclude
EXCLUDES=$HOME/cron/excludes

# the name of the backup machine
BSERVER=owl

# your password on the backup server
export RSYNC_PASSWORD=XXXXXX

########################################################################

BACKUPDIR=`date +%A`
OPTS="--force --ignore-errors --delete-excluded --exclude-from=$EXCLUDES
--delete --backup --backup-dir=/$BACKUPDIR -a"

export PATH=$PATH:/bin:/usr/bin:/usr/local/bin

# the following line clears the last weeks incremental directory
[ -d $HOME/emptydir ] || mkdir $HOME/emptydir
rsync --delete -a $HOME/emptydir/ $BSERVER:

USER/$BACKUPDIR/
rmdir $HOME/emptydir

# now the actual transfer
rsync $OPTS $BDIR $BSERVER:

USER/current

备份至一个空闲的硬盘

I do local backups on several of my machines using rsync. I have an
extra disk installed that can hold all the contents of the main
disk. I then have a nightly cron job that backs up the main disk to
the backup. This is the script I use on one of those machines.

#!/bin/sh

export PATH=/usr/local/bin:/usr/bin:/bin

LIST="rootfs usr data data2"

for d in $LIST; do
mount /backup/$d
rsync -ax --exclude fstab --delete /$d/ /backup/$d/
umount /backup/$d
done

DAY=`date "+%A"`

rsync -a --delete /usr/local/apache /data2/backups/$DAY
rsync -a --delete /data/solid /data2/backups/$DAY

The first part does the backup on the spare disk. The second part
backs up the critical parts to daily directories.  I also backup the
critical parts using a rsync over ssh to a remote machine.

对vger的cvs树进行镜像

The vger.rutgers.edu cvs tree is mirrored onto cvs.samba.org via
anonymous rsync using the following script.

#!/bin/bash

cd /var/www/cvs/vger/
PATH=/usr/local/bin:/usr/freeware/bin:/usr/bin:/bin

RUN=`lps x | grep rsync | grep -v grep | wc -l`
if [ "$RUN" -gt 0 ]; then
      echo already running
      exit 1
fi

rsync -az vger.rutgers.edu::cvs/CVSROOT/ChangeLog $HOME/ChangeLog

sum1=`sum $HOME/ChangeLog`
sum2=`sum /var/www/cvs/vger/CVSROOT/ChangeLog`

if [ "$sum1" = "$sum2" ]; then
      echo nothing to do
      exit 0
fi

rsync -az --delete --force vger.rutgers.edu::cvs/ /var/www/cvs/vger/
exit 0

Note in particular the initial rsync of the ChangeLog to determine if
anything has changed. This could be omitted but it would mean that the
rsyncd on vger would have to build a complete listing of the cvs area
at each run. As most of the time nothing will have changed I wanted to
save the time on vger by only doing a full rsync if the ChangeLog has
changed. This helped quite a lot because vger is low on memory and
generally quite heavily loaded, so doing a listing on such a large
tree every hour would have been excessive.

在家自动备份

我用rsync to backup my wifes home directory across a modem link each
night. The cron job looks like this

#!/bin/sh
cd ~susan
{
echo
date
dest=~/backup/`date +%A`
mkdir $dest.new
find . -xdev -type f $ -mtime 0 -or -mtime 1 $ -exec cp -aPv "{}"
$dest.new \;
cnt=`find $dest.new -type f | wc -l`
if [ $cnt -gt 0 ]; then
   rm -rf $dest
   mv $dest.new $dest
fi
rm -rf $dest.new
rsync -Cavze ssh . samba:backup
} >;>; ~/backup/backup.log 2>;&1

note that most of this script isn't anything to do with rsync, it just
creates a daily backup of Susans work in a ~susan/backup/ directory so
she can retrieve any version from the last week. The last line does
the rsync of her directory across the modem link to the host
samba. Note that I am using the -C option which allows me to add
entries to .cvsignore for stuff that doesn't need to be backed up.

Fancy footwork with remote file lists

One little known feature of rsync is the fact that when run over a
remote shell (such as rsh or ssh) you can give any shell command as
the remote file list. The shell command is expanded by your remote
shell before rsync is called. For example, see if you can work out
what this does:

rsync -avR remote:'`find /home -name "*.[ch]"`' /tmp/

note that that is backquotes enclosed by quotes (some browsers don't
show that correctly).

作者: wingger 时间: 2004-07-07 10:43
标题: ［每周讨论专题］－－第六期－－WEB服务器的数据远程同步
Rsync for Windows（我没测试成功，有兴趣的可以试试，我有空会把这篇翻译成中文）

导言
我写这份文档可以帮助那些想通过使用Rsync备份 windowsrs 到LINUX servers的人或 Windows 工作站/服务器. 这是经过测试过的。

--------------------------------------------------------------------------------

1: Cygwin环境安装

你需要下载和安装Cygwin系统 .
安装时记住要选择 Rsync from the +Net package list, 和a suitable Editor (因为这个系统不会默认安装). 要用到一些编辑器  Pico (在 +Mail之下的part of Pine living ) 和 Nano (living under +Editors).

下面这段都是讲如何在win下设置环境变量的
在环境变量中增加C:\Cygwin\bin.

否则会报错：apps called from outside Cygwin will fail.

On Windows 2000/XP, open the Control Panel and double click on the System applet. Click on the Advanced tab, then click the Environment Variables button. Double click on the PATH statement in the 'System Variable' screen (lower of the two), add the path on the end, and click OK. Click OK to close the Environment Variables screen, then click OK to close the System Properties dialogue box. The path will be dynamically reloaded (no need to reboot).

注意：
If the end of the path looks something like this: C:\Somepath don't forget to add ; before you add the C:\Cygwin\bin; e.g. C:\Somepath;C:\Cygwin\bin;

--------------------------------------------------------------------------------

2: 服务端分LINUX和WIN服务端两种

Linux Rsync 服务安装
1. 确认 Rsync 已经安装在你的LINUX上 [rpm -q rsync (Red Hat)] .

2. Choose the path for your backup area: This can be on a per-user basis (backup a Users data to their /home/user area) or on a system level basis (a single machine backing up to one directory.)

Real world example:
All rsync data on the Gaztronics Server sits on /dev/hda2 as /data/rsync_dump

3. Create the /etc/rsyncd.conf and /etc/rsyncd.secrets files. [Check out the Manual pages for rsyncd.conf, or see the documentation on the Rsync website for more info.]

Here is an example of an rsyncd.conf file where the backup area drops into the user 'Fred's' home drive:

[computername]

path = /home/fred/backup
comment = Fred's Offsite storage area (requires authentication)
uid = fred
gid = users
read only = false
auth users = fred
secrets file = /etc/rsyncd.secrets

The permissions for this file should be: -rw-r--r-- (644) and root root.

The corresponding rsyncd.secrets file contains the following entry:

fred:BackUpPassword

The permissions for this file should be: -rw------- (600) and root root.

4. Start Rsync in daemon mode.

Linux Tip: Linux Distros usually run rsync from xinetd. You might need to run /usr/sbin/setup (Red Hat) and select 'rsync' in the 'System Services'; or you can edit the 'rsync' file in /etc/xinetd.d and set disable = no (don't forget to restart xinetd!).

Windows Rsync 服务安装

Setting up Rsync as a Server under Windows is a little more tricky, due to the differences in security and paths, and the inability to use the 'authenticate user' mode of Rsync. For this reason, I would not recommend setting up an Rsync Server on Windows for use over the Internet. Keep the installation within a secure Local Area Network.

Method update:  Windows 2003 Server has thrown a spanner in the works. Microsoft have set the paranoid level to maximum which has resulted in Cygwin based services failing to start. This method has been updated to take this into account.

This method comes without warranty, but it should work for: Windows NT 4.0 Server; Windows NT 4.0 Workstation; Windows 2000 Server; Windows 2000 Workstation; Windows XP Workstation.

Method 1. Without installing Cygwin in the full: You might be interested in this Rsync Server project. The Windows package installs the bare essentials to create an Rsync Server.
This method has yet to be tested with 2003 Server!

Method 2. Install Cygwin in full and use its directory structure for storage. (This is recommended if you are new to Linux as the directory permissions are stored *nix style and you will need Cygwin to change them - you cannot change them from Windows!)

Both methods require a valid rsyncd.conf, such as the one below.

--------------------------------------------------------------------------------

Method 2. Step 1: Install Cygwin as in Section 1.

Method 2. Step 2: Choose an area to backup the files to. (In this example I have used /var/rsync_dump in the Cygwin tree.)

Method 2. Step 3: Create the /etc/rsyncd.conf file, as in the example below:

use chroot = false
strict modes = false

[modulename]

path = /cygdrive/c/cygwin/var/rsync_dump
comment = Rsync storage area
read only = false

Note: The path = /cygdrive/c/cygwin/var/rsync_dump looks a little odd. This is Cygwin convention for defining Windows paths from within a *nix emulator.

Method 2. Step 4: If you are setting up on Windows 2003 Server (otherwise skip to the next step):

(i) Open the Windows File Explorer and go to the C: drive.

(ii) Right click on the 'Cygwin' directory and select 'Properties'.

(iii) Click on the 'Security' tab. The user 'Administrator' should be the first in the list and it will not have any permissions set for this folder.
   (If the user 'Administrator' is not listed, you will need to add it.)

(iv) Tick the 'Allow - Full Control' box in the "

ermissions for Administrator" window.

(v) Click the Advanced button and tick the box for "Replace permission entries on all child objects with entries shown here that apply to child objects".

(vi) Click the Apply button to set the permissions.

(vii) Click the OK button to close the Advanced settings dialogue box.

(viii) Click the OK button to close the Cygwin properties dialogue box.

Method 2. Step 5 Install Rsync as a Service from a 'Command Prompt' window with the following command line:

cygrunsrv.exe -I "Rsync" -p /cygdrive/c/cygwin/bin/rsync.exe -a "--config=/cygdrive/c/cygwin/etc/rsyncd.conf --daemon --no-detach"
-f "Rsync" -u Administrator -w password

Note: This is all one line!

The section -u Administrator -w password installs the service to run as the user 'Administrator' (password is the Administrator's account password) and is required by Windows 2003 Server, else the service will fail to start correctly. The extra settings have not been needed for Windows NT 4.0 or Windows 2000, but may be a good idea.

Windows Tip: Once you have the service running as Administrator, you can, for good security practice, create an account with Admin priveleges which Rsync can use. Remember to change the permissions on the C:\Cygwin directory and for the Service.

Method 2. Final Step From the same command prompt, start the Service with net start rsync.

If all has gone well, you should be ready to accept incoming client connections.

--------------------------------------------------------------------------------

3: 客户端

In order to backup your Windows machine effectively, I would recommend the use of a batch file, as in the following:

@ cls
@ echo off

rem Rsync job control file

C:\Cygwin\bin\rsync -vrtz --password-file=c:\cygwin\secret --delete /cygdrive/d/Data fred@company.com::computername

An explanation:

C:\Cygwin\bin\rsync - is the full path to 'rysync.exe'.

-vrtz - See the rsync documentation for details.

--password-file=c:\cygwin\secret - Path to 'secret' file. (Note: Remember this is for backup to a Linux based Rsync Server; a Windows based Rsync Server cannot authenticate!)

--delete - delete remote files that are deleted locally.

/cygdrive/d/Data - in this example means D:\Data.

fred@company.com::computername - is the user ID, hostname (can be IP address if over Local Network), and the module connection name (in this example 'computername').

Windows Tip: We assume you wish to run the backup automatically! In that case, you will have to create the 'secret' file that Rsync uses for its authenticating password. Here is how to do that:

1. Login to Cygwin.

2. Create a file called 'secret' in the root of the Cygwin application (i.e. cd /) with the Rsync server password (in this example: 'BackUpPassword') and give it 600 permissions (chmod 600 secret). This is the file '--password-file=c:\cygwin\secret' as referred to above.

3. Exit Cygwin.

Windows Tip: You may call this batch file from the Startup Group, the Scheduler (and AT scheduler), or from the Logon / Logoff features of the Group Policy (Win2k/XP) by running gpedit.msc.

--------------------------------------------------------------------------------

Further reading: check out the documentation on the Rsync website.

Happy backing up!!

作者: wingger 时间: 2004-07-10 20:54
标题: ［每周讨论专题］－－第六期－－WEB服务器的数据远程同步
版主去哪了，

作者: ljh1405 时间: 2004-07-12 18:32
标题: ［每周讨论专题］－－第六期－－WEB服务器的数据远程同步
其实远程数据同步的方案是比较多的。
如果数据量不大用rsync就完全可以实现了，可以参看
http://www.fanqiang.com/a1/b1/20011019/0800011478.html
另外如果数据量大的话可以采用rsync＋sgi－fam
sgi－fam在redhat AS3 里面就有，不用编译内核
具体做法可以参看
http://oldsite.linuxaid.com.cn/training/showtri.jsp?i=221
如果是专业的网站就可以考虑用商业的数据同步软件来做
比如snapmirror等等，甚至可以用netapp等专业的存储设备加以实现。

作者: 飘雪心辰 时间: 2004-08-05 09:36
标题: ［每周讨论专题］－－第六期－－WEB服务器的数据远程同步
如果用root根用户，用rsync之前最好小心点，此命令像rm命令一样“杀伤力”具大。

作者: fxyabc 时间: 2004-09-06 16:25
标题: ［每周讨论专题］－－第六期－－WEB服务器的数据远程同步
不懂，先收藏！！！

作者: Ice_D. 时间: 2004-09-07 20:35
标题: ［每周讨论专题］－－第六期－－WEB服务器的数据远程同步
为什么在这里我只有灌水的份？
天了啊！

－－－－－－－－－－－
但是我会学习，我现在是菜鸟，但我不是永远菜鸟～

作者: platinum 时间: 2004-10-11 15:15
标题: ［每周讨论专题］－－第六期－－WEB服务器的数据远程同步
测试过了，可行，下面谈一些我用后的感想

原理很简单
1、借助cygwin把source编译成windows下可以用的exe程序
2、在cmd里直接使用，路径用c:\xxx这样就可以
3、做服务时有些麻烦，需要用到cygrunsrv.exe这个cygwin的程序

一些感触
1、windows做client，linux做server的使用方法
这种方法不太好
linux区分大小写，而windows不
linux有“软链”的概念，windows没有
linux有很复杂的chown和chmod，和windows不太一样
这样的话，即使备份下来了，恢复的时候也未必可用

2、linux做client，windows做server
个人认为这种方法还可以，但是也同样存在文件权限的不兼容性问题

总结
怎么用，关键看应用领域了，如果只是FTP和SAMBA或WEB，就无所谓了
（有些WEB除外，因为需要设置复杂的属主和属性）

如果是和系统有关的一些文件，最好还是win-win或linux-linux

作者: zjzf_1 时间: 2004-10-28 09:17
标题: ［每周讨论专题］－－第六期－－WEB服务器的数据远程同步
rsync 可一定是同步如果想实时同步就用共享储存的办法来解决

但是服务器到共享储存的速度一定要快

作者: fz-L 时间: 2005-12-04 21:59
有实际实践通过的吗，rsync远程实现多大的数据量比较好呢

作者: zhengwei_zw 时间: 2006-12-13 15:52
有一个没有说到。那就算client and server 都用rsync同步
上边全部实现过。
求在win下自动rsync的方法。应为我的rsync都走的ssh

作者: zdm 时间: 2007-05-25 16:35
win下也有ssh：
http://sourceforge.net/project/s ... p;package_id=111688

作者: benkernel 时间: 2007-07-18 09:51
我有十几台web服务器，用rsync可以实现吗？

作者: HonestQiao 时间: 2007-07-18 10:28

原帖由 benkernel 于 2007-7-18 09:51 发表
我有十几台web服务器，用rsync可以实现吗？

可以使用，但是你要顶一个好的同步规则。

欢迎光临 Chinaunix (http://bbs.chinaunix.net/)