脚本军火库：使用Amazon S3服务备份指定文件

最新推荐文章于 2021-02-18 18:52:13 发布

原创最新推荐文章于 2021-02-18 18:52:13 发布 · 1.6k 阅读

0 ·

本内容遵循CC 4.0 BY-SA版权协议

标签

#脚本 #date #cmd #groovy #variables

Linux 应用专栏收录该内容

32 篇文章

订阅专栏

本文介绍了一种基于AWS S3服务的文件备份与恢复方案，包括s3cmd工具的使用，以及通过自定义脚本实现自动化的备份与恢复流程。

以下将提供一个基于S3服务的备份方案。AWS虽然属于收费服务，但对于小规模的文件备份而言还是很便宜的（我的两美元到现在没花完……汗）。开通后可以安装一个叫's3cmd'的工具简化操作（参考Linux下通过s3cmd使用Amazon S3服务）。

首先准备配置文件和目录，运行以下脚本s3init

#!/bin/ksh

mkdir -p $XMETA_HOME/restore $XMETA_HOME/backup

cat > $XMETA_HOME/xmeta.conf << EOF

s3_bucket_backup=Xmeta

EOF

cat > $XMETA_HOME/backup.list << EOF

/etc/profile.d/custom.sh

~/.bash_profile

~/.vimrc

~/.vim

~/tools/xmeta/scripts

~/tools/xmeta/xmeta.conf

EOF

每次备份都将生成一个tarball保存到backup文件夹并上传到s3，还原的时候，则从s3下载最新的备份文件并解压到restore文件夹。注意：还原操作并不会覆盖原有文件，毕竟大多数人不想一不小心就把自己的系统还原到两年以前吧。

xmeta.conf文件将成为Xmeta的主要配置文件，将Xmeta用到的环境变量集中在这个配置文件中可以大大降低对.bash_profile的“污染”。目前这个文件仅仅包含了一个变量：$s3_bucket_backup，这个变量定义了用于存放备份文件的s3 bucket。无需多言，在使用前要在s3新建对应的bucket，命令如下（用你自己的bucket代替命令中的$bucket_name，可惜s3不支持命名空间，好名字已经所剩无几……多试几次吧）

s3cmd mb s3://$bucket_name

backup.list则用于指定备份范围，里面每一行代表一个文件或目录，可以根据自己的需要添减。

准备工作完成后，就可以开始做正事了。首先建立一个用于打印格式化日期的脚本（意义不大，纯粹为了方便）。

datestr

#!/bin/env ksh

#

# Copyright (C) Young, Fey 

#

# This program is free software: you can redistribute it and/or modify

# it under the terms of the GNU General Public License as published by

# the Free Software Foundation, either version 3 of the License, or

# (at your option) any later version.

#

# This program is distributed in the hope that it will be useful,

# but WITHOUT ANY WARRANTY; without even the implied warranty of

# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the

# GNU General Public License for more details.

#

# You should have received a copy of the GNU General Public License

# along with this program.  If not, see .

#

# SCRIPT: datestr

# AUTHOR: Young, Fey

# DATE:   2010-8-30

# REV:    1.0.P

#

# PURPOSE: print the current datetime in specified format

#

# set -n # Uncomment to check syntax without execution

#

# set -x # Uncomment to debug

#

################################# FUNCTIONS ####################################

function usage {

    echo "Usage: $0 [option]"

    echo "  -s, --short       YYYY-mm-dd"

    echo "  -l, --long        YYYY-mm-dd HH:MM:SS"

    echo "  -c, --compact     YYYYmmdd"

    echo "  -d, --default     YYYYmmddHHMMSS"

    exit 1

}

#################################### MAIN ######################################

if (( $# > 1 ))

then

    usage

fi

flag='-d'

if (( $# == 1 ))

then

    flag=$1

fi

case $flag in

    '-d')

        date '+%Y%m%d%H%M%S'

        ;;

    '-s')

        date '+%Y-%m-%d'

        ;;

    '-c')

        date '+%Y%m%d'

        ;;

    '-l')

        date '+%Y-%m-%d %H:%M:%S'

        ;;

    *)

        usage

esac

# End of script

其次是备份脚本s3backup，备份逻辑很简单，就是遍历列表中的每一行，把对应的文件（夹）塞到tarball里，最后上传。

#!/bin/env ksh

#

# Copyright (C) Young, Fey 

#

# This program is free software: you can redistribute it and/or modify

# it under the terms of the GNU General Public License as published by

# the Free Software Foundation, either version 3 of the License, or

# (at your option) any later version.

#

# This program is distributed in the hope that it will be useful,

# but WITHOUT ANY WARRANTY; without even the implied warranty of

# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the

# GNU General Public License for more details.

#

# You should have received a copy of the GNU General Public License

# along with this program.  If not, see .

#

# SCRIPT: s3backup

# AUTHOR: Young, Fey

# DATE:   2010-8-30

# REV:    0.1.A

#

# PURPOSE: backup files to amazon s3 service

#

# set -n # Uncomment to check syntax without execution

#

# set -x # Uncomment to debug

#

######################### DEFINE FILES AND VARIABLES ###########################

. $XMETA_HOME/xmeta.conf

backup_list=$XMETA_HOME/backup.list

tarball=$XMETA_HOME/backup/xmeta_`datestr`.tar

############################## DEFINE FUNCTIONS ################################

function add_file {

    while read LINE

    do

        file=${ eval "echo $LINE" }

        tar rf $tarball $file

    done < $backup_list

}

#################################### MAIN ######################################

tar cf $tarball $backup_list

add_file

gzip $tarball

s3cmd put $tarball.gz s3://$s3_bucket_backup

# End of script

美中不足的是tar在处理绝对路径时会提示 tar: Removing leading `/' from member names，鉴于安全性考虑，我并没有使用-P参数来保留'/'，如果哪位大侠有消除这条提示的方法，一定要告诉我。（郁闷的是，比特桶在这里无效，在我加上 2>&1 > /dev/null后，tar依然风雨无阻的输出了警告）

最后是还原脚本 s3restore

#!/bin/env ksh

#

# Copyright (C) Young, Fey 

#

# This program is free software: you can redistribute it and/or modify

# it under the terms of the GNU General Public License as published by

# the Free Software Foundation, either version 3 of the License, or

# (at your option) any later version.

#

# This program is distributed in the hope that it will be useful,

# but WITHOUT ANY WARRANTY; without even the implied warranty of

# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the

# GNU General Public License for more details.

#

# You should have received a copy of the GNU General Public License

# along with this program.  If not, see .

#

# SCRIPT: s3restore

# AUTHOR: Young, Fey

# DATE:   2010-8-30

# REV:    0.1.A

#

# PURPOSE: restore the latest backup to $XMETA_HOME/restore

#

# set -n # Uncomment to check syntax without execution

#

# set -x # Uncomment to debug

#

#################################### MAIN ######################################

. $XMETA_HOME/xmeta.conf

echo 'check s3 bucket'

latest_backup=`s3cmd ls s3://$s3_bucket_backup |sort | tail -n 1 | sed 's/.*s3:////.*////'`

if [ -z $latest_backup ]

then

    echo 'no backup found'

    exit 1

fi

cd $XMETA_HOME/restore

s3cmd get s3://$s3_bucket_backup/$latest_backup

tar xf $latest_backup

rm $latest_backup

# End of script

执行s3restore的时候，脚本会遍历对应的s3 bucket并将其中的文件名进行排序，因此，请勿在该bucket中存放其它文件，否则脚本可能无法找到所需的备份文件。（如果非要放的话，用数字开头的还有a-w开头的文件，这样不影响排序）。解压的文件保存在restore文件夹，需要手动将其转移到相应的位置。（如果你真的很勇敢，可以把s3backup中的tar命令加上-P参数然后以足够高的权限来运行，比如你要是备份了/etc/fstab，那么你需要root的权限来完成覆盖）

PS:
Live Writer下的"Source Code Formatter by Amer Gerzic"插件实在不算好用，不支持代码下载，连语法高亮支持都很不够，没有bash，没有ksh，没有groovy……（残念），最后用Ruby的高亮坚持了下。有谁推荐更好的吗？