MySQL——日志管理(binarylog、querylog)

本文深入解析MySQL中的查询日志和二进制日志(binlog),包括日志的作用、配置参数、事件类型及其组成,以及如何使用binlog进行数据恢复,特别关注GTID模式下的日志管理和恢复策略。

一、查询日志

说明:

查询日志记录了数据库客户端所有的语句,二进制日志不包含查询数据的SQL

参数说明:

3306[youngboy]>select @@general_log;	
+---------------+
| @@general_log |
+---------------+
|             0 |
+---------------+
1 row in set (0.00 sec)
general_log=1/0		是否开启querylog 默认0关闭

3306[youngboy]>select @@general_log_file;
+-------------------------------------+
| @@general_log_file                  |
+-------------------------------------+
| /opt/mysql-data/mysql/localhost.log |
+-------------------------------------+
1 row in set (0.00 sec)
general_log_file=/opt/mysql-data/mysql/localhost.log 	路径

3306[youngboy]>select @@log_output;
+--------------+
| @@log_output |
+--------------+
| FILE         |
+--------------+
1 row in set (0.00 sec)
value值可以是TABLEFILE、NONE
TABLE:表示日志输出到表
FILE:表示日志输出到文件
NONE:表示日志不写到表也不写到文件

如果设置输出到表则:
querylog:记录在mysql库的general_log表中
slowlog:记录在mysql库的slow_log表中
默认输出到文件,性能更高,因为表会比文件占到更多的资源。

注:querylog在数据库中用的不多,会记录所有数据库的操作,需要经常访问的数据库,这个日志对数据库的性能影响会比较大,所以一般是关闭的。

二、binlog(二进制日志) *****

binlog作用及参数

binlog记录了所有的DDL/DML语句,不包括DQL语句,是用event形式存放的,在备份恢复和主从环境中很重要。
vim /etc/my.cnf
server_id=6                                    ----->5.6中,单机可以不需要此参数              
log_bin=/opt/mysql-data/binlog/mysql-bin
binlog_format=row
sync_binlog=1
max_binlog_size=1024000000				日志文件到达1个G自动滚动日志到下一个
重启数据库生效
systemctl restart mysqld

注:MySQL数据库默认不开启二进制日志
相关命令:

开关及路径:
3306[youngboy]>select @@log_bin;					查看二进制日志是否开启
+-----------+
| @@log_bin |
+-----------+
|         1 |
+-----------+
1 row in set (0.00 sec)

3306[(none)]>select @@log_bin_basename;				查看二进制日志文件路径
+----------------------------------+
| @@log_bin_basename               |
+----------------------------------+
| /opt/mysql-data/binlog/mysql-bin |
+----------------------------------+
1 row in set (0.01 sec)

服务ID:
3306[(none)]>select @@server_id;					查看ID号,主要用在主从
+-------------+
| @@server_id |
+-------------+
|           6 |
+-------------+
1 row in set (0.00 sec)

二进制日志格式:
3306[(none)]>select @@binlog_format;						
+-----------------+	
| @@binlog_format |
+-----------------+
| ROW             |
+-----------------+
1 row in set (0.00 sec)

双一标准的另一个参数:
3306[(none)]>select @@sync_binlog;					同步二进制日志
+---------------+
| @@sync_binlog |
+---------------+
|             1 |
+---------------+
1 row in set (0.00 sec)
3306[youngboy]>show binary logs;					查看有几个二进制文件
+------------------+-----------+
| Log_name         | File_size |
+------------------+-----------+
| mysql-bin.000007 |  89629964 |
| mysql-bin.000008 |       509 |
+------------------+-----------+
2 rows in set (0.00 sec)

3306[youngboy]>flush logs;							刷新一个新的binlog日志
Query OK, 0 rows affected (0.02 sec)
日志滚动重启mysql也会自动刷新一个,达到设置日志的大小也会自动滚动一个。

3306[(none)]>show master status ;					查看正在使用的binlog日志
+------------------+----------+--------------+------------------+---------------------------------------------------------------------------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set                                                                     |
+------------------+----------+--------------+------------------+---------------------------------------------------------------------------------------+
| mysql-bin.000008 |      234 |              |                  | 0a99e3f9-3a95-11ea-87df-000c29b6e0ec:1-12,
979de5b8-3c01-11ea-8252-000c29b6e0ec:1-326 |
+------------------+----------+--------------+------------------+---------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

二进制日志格式参数说明(binlog_format):

statement(SBR):原封不动的记录用户操作的DML语句
ROW(RBR):记录用户进行SQL操作后每一行的记录,也就是数据行的变化(日志量比statement大很多,因为需要记录行的变化,大表数据量会非常大)
select查询语句不记录
mixed(MBR):以上两种模式的混合,用两种模式的优点,避开他们的缺点。

事件(event)

事件的组成

1、事件在二进制中的开始标识
2、事件的内容
3、事件在二进制中的结束标识
4、事件的示例以及参数说明

3306[youngboy]>show binlog events in 'mysql-bin.000008';
+------------------+-----+----------------+-----------+-------------+---------------------------------------------------------------------------------------+
| Log_name         | Pos | Event_type     | Server_id | End_log_pos | Info                                                                                  |
+------------------+-----+----------------+-----------+-------------+---------------------------------------------------------------------------------------+
| mysql-bin.000008 |   4 | Format_desc    |         6 |         123 | Server ver: 5.7.20-log, Binlog ver: 4                                                 |
| mysql-bin.000008 | 123 | Previous_gtids |         6 |         234 | 0a99e3f9-3a95-11ea-87df-000c29b6e0ec:1-12,
979de5b8-3c01-11ea-8252-000c29b6e0ec:1-326 |
| mysql-bin.000008 | 234 | Gtid           |         6 |         299 | SET @@SESSION.GTID_NEXT= '979de5b8-3c01-11ea-8252-000c29b6e0ec:327'                   |
| mysql-bin.000008 | 299 | Query          |         6 |         375 | BEGIN                                                                                 |
| mysql-bin.000008 | 375 | Table_map      |         6 |         427 | table_id: 267 (youngboy.t1)                                                           |
| mysql-bin.000008 | 427 | Write_rows     |         6 |         478 | table_id: 267 flags: STMT_END_F                                                       |
| mysql-bin.000008 | 478 | Xid            |         6 |         509 | COMMIT /* xid=1506670 */                                                              |
+------------------+-----+----------------+-----------+-------------+---------------------------------------------------------------------------------------+
7 rows in set (0.00 sec)
如果一个binlog中日志太多还可以做限制,显示几条信息:
3306[youngboy]>show binlog events in 'mysql-bin.000008' limit 6;
+------------------+-----+----------------+-----------+-------------+---------------------------------------------------------------------------------------+
| Log_name         | Pos | Event_type     | Server_id | End_log_pos | Info                                                                                  |
+------------------+-----+----------------+-----------+-------------+---------------------------------------------------------------------------------------+
| mysql-bin.000008 |   4 | Format_desc    |         6 |         123 | Server ver: 5.7.20-log, Binlog ver: 4                                                 |
| mysql-bin.000008 | 123 | Previous_gtids |         6 |         234 | 0a99e3f9-3a95-11ea-87df-000c29b6e0ec:1-12,
979de5b8-3c01-11ea-8252-000c29b6e0ec:1-326 |
| mysql-bin.000008 | 234 | Gtid           |         6 |         299 | SET @@SESSION.GTID_NEXT= '979de5b8-3c01-11ea-8252-000c29b6e0ec:327'                   |
| mysql-bin.000008 | 299 | Query          |         6 |         375 | BEGIN                                                                                 |
| mysql-bin.000008 | 375 | Table_map      |         6 |         427 | table_id: 267 (youngboy.t1)                                                           |
| mysql-bin.000008 | 427 | Write_rows     |         6 |         478 | table_id: 267 flags: STMT_END_F                                                       |
+------------------+-----+----------------+-----------+-------------+---------------------------------------------------------------------------------------+
6 rows in set (0.00 sec)

Log_name:binlog文件名
Pos:事件开始的position    
Event_type:事件类型
Format_desc:MySQL识别binlog必要信息,没有这些信息识别不了
Server_id:mysql服务号,在主从中主要对应哪台机器
End_log_pos:事件结束的位置号 
Info:事件内容,二进制一般看不懂,只知道做了什么操作

注:上一个结束的pos号就是下一个开始的pos号,不需要加1。
如果不知道格式,还可以使用help show binlog events;来查看events的使用
通过过滤的方式查看到二进制日志里面insert的操作
[root@localhost binlog]# mysql -e "show binlog events in 'mysql-bin.000007'" | grep insert
mysql-bin.000007	737771	Query	6	738457	use `mysql`; CREATE TABLE `ndb_binlog_index` (\n  `Position` bigint(20) unsigned NOT NULL,\n  `File` varchar(255) NOT NULL,\n  `epoch` bigint(20) unsigned NOT NULL,\n  `inserts` int(10) unsigned NOT NULL,\n  `updates` int(10) unsigned NOT NULL,\n  `deletes` int(10) unsigned NOT NULL,\n  `schemaops` int(10) unsigned NOT NULL,\n  `orig_server_id` int(10) unsigned NOT NULL,\n  `orig_epoch` bigint(20) unsigned NOT NULL,\n  `gci` int(10) unsigned NOT NULL,\n  `next_position` bigint(20) unsigned NOT NULL,\n  `next_file` varchar(255) NOT NULL,\n  PRIMARY KEY (`epoch`,`orig_server_id`,`orig_epoch`)\n) ENGINE=MyISAM DEFAULT CHARSET=latin1

事件的说明

1、事件是二进制日志中最小的记录单元
2、相对于DDL、DCL来说,一条语句就是一个event
3、相对于DML来说已提交的事务中begin和commit在内的每一条语句都是一个事件
例:

begin;								第一个事件			
insert into xxx values(xxx,xxx);	第二个事件	
insert into xxx values(xxx,xxx);	...(可以包括N个事件,看事务语句有多少)
commit;								最后一个事件

binlog使用

binlog文件内容查看
日志说明:
1、从BEGIN到COMMIT就是一个日志的完整binlog日志

2、日志中binlog中绿色的两行就是ROW模式下的日志,加密数据,需要解密才能查看

3、详细日志中的“@1=8”可以看出来是插入了一行数据到第8行

直接查看binlog日志信息:
[root@localhost binlog]# mysqlbinlog mysql-bin.000008		
/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=1*/;
/*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/;
DELIMITER /*!*/;
# at 4
#200224 10:39:10 server id 6  end_log_pos 123 CRC32 0xeb936dbc 	Start: binlog v 4, server v 5.7.20-log created 200224 10:39:10
BINLOG '
TjdTXg8GAAAAdwAAAHsAAAAAAAQANS43LjIwLWxvZwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAEzgNAAgAEgAEBAQEEgAAXwAEGggAAAAICAgCAAAACgoKKioAEjQA
Abxtk+s=
'/*!*/;
# at 123
#200224 10:39:10 server id 6  end_log_pos 234 CRC32 0x51fdd69a 	Previous-GTIDs
# 0a99e3f9-3a95-11ea-87df-000c29b6e0ec:1-12,
# 979de5b8-3c01-11ea-8252-000c29b6e0ec:1-326
# at 234
#200224 10:45:37 server id 6  end_log_pos 299 CRC32 0xb51fc4a2 	GTID	last_committed=0	sequence_number=1	rbr_only=yes
/*!50718 SET TRANSACTION ISOLATION LEVEL READ COMMITTED*//*!*/;
SET @@SESSION.GTID_NEXT= '979de5b8-3c01-11ea-8252-000c29b6e0ec:327'/*!*/;
# at 299
#200224 10:45:28 server id 6  end_log_pos 375 CRC32 0xfc0d9d46 	Query	thread_id=53	exec_time=0	error_code=0
SET TIMESTAMP=1582512328/*!*/;
SET @@session.pseudo_thread_id=53/*!*/;
SET @@session.foreign_key_checks=1, @@session.sql_auto_is_null=0, @@session.unique_checks=1, @@session.autocommit=1/*!*/;
SET @@session.sql_mode=1436549152/*!*/;
SET @@session.auto_increment_increment=1, @@session.auto_increment_offset=1/*!*/;
/*!\C utf8 *//*!*/;
SET @@session.character_set_client=33,@@session.collation_connection=33,@@session.collation_server=45/*!*/;
SET @@session.lc_time_names=0/*!*/;
SET @@session.collation_database=DEFAULT/*!*/;
BEGIN
/*!*/;
# at 375
#200224 10:45:28 server id 6  end_log_pos 427 CRC32 0xe113aadb 	Table_map: `youngboy`.`t1` mapped to number 267
# at 427
#200224 10:45:28 server id 6  end_log_pos 478 CRC32 0x49664d8c 	Write_rows: table id 267 flags: STMT_END_F

BINLOG '
yDhTXhMGAAAANAAAAKsBAAAAAAsBAAAAAAEACHlvdW5nYm95AAJ0MQACAw8CUAAD26oT4Q==
yDhTXh4GAAAAMwAAAN4BAAAAAAsBAAAAAAEAAgAC//wIAAAACmhkZmVuZzIzNDWMTWZJ
'/*!*/;
# at 478
#200224 10:45:37 server id 6  end_log_pos 509 CRC32 0x3681e7dd 	Xid = 1506670
COMMIT/*!*/;

---------------------------------------------------------------------------------------------------------------
查看更详细的日志信息:
[root@localhost binlog]# mysqlbinlog --base64-output=decode-rows -vv /data/binlog/mysql-bin.000009
/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=1*/;
/*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/;
DELIMITER /*!*/;
mysqlbinlog: File '/data/binlog/mysql-bin.000009' not found (Errcode: 2 - No such file or directory)
SET @@SESSION.GTID_NEXT= 'AUTOMATIC' /* added by mysqlbinlog */ /*!*/;
DELIMITER ;
# End of log file
/*!50003 SET COMPLETION_TYPE=@OLD_COMPLETION_TYPE*/;
/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=0*/;
[root@localhost binlog]# mysqlbinlog --base64-output=decode-rows -vvv mysql-bin.000008
/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=1*/;
/*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/;
DELIMITER /*!*/;
# at 4
#200224 10:39:10 server id 6  end_log_pos 123 CRC32 0xeb936dbc 	Start: binlog v 4, server v 5.7.20-log created 200224 10:39:10
# at 123
#200224 10:39:10 server id 6  end_log_pos 234 CRC32 0x51fdd69a 	Previous-GTIDs
# 0a99e3f9-3a95-11ea-87df-000c29b6e0ec:1-12,
# 979de5b8-3c01-11ea-8252-000c29b6e0ec:1-326
# at 234
#200224 10:45:37 server id 6  end_log_pos 299 CRC32 0xb51fc4a2 	GTID	last_committed=0	sequence_number=1	rbr_only=yes
/*!50718 SET TRANSACTION ISOLATION LEVEL READ COMMITTED*//*!*/;
SET @@SESSION.GTID_NEXT= '979de5b8-3c01-11ea-8252-000c29b6e0ec:327'/*!*/;
# at 299
#200224 10:45:28 server id 6  end_log_pos 375 CRC32 0xfc0d9d46 	Query	thread_id=53	exec_time=0	error_code=0
SET TIMESTAMP=1582512328/*!*/;
SET @@session.pseudo_thread_id=53/*!*/;
SET @@session.foreign_key_checks=1, @@session.sql_auto_is_null=0, @@session.unique_checks=1, @@session.autocommit=1/*!*/;
SET @@session.sql_mode=1436549152/*!*/;
SET @@session.auto_increment_increment=1, @@session.auto_increment_offset=1/*!*/;
/*!\C utf8 *//*!*/;
SET @@session.character_set_client=33,@@session.collation_connection=33,@@session.collation_server=45/*!*/;
SET @@session.lc_time_names=0/*!*/;
SET @@session.collation_database=DEFAULT/*!*/;
BEGIN
/*!*/;
# at 375
#200224 10:45:28 server id 6  end_log_pos 427 CRC32 0xe113aadb 	Table_map: `youngboy`.`t1` mapped to number 267
# at 427
#200224 10:45:28 server id 6  end_log_pos 478 CRC32 0x49664d8c 	Write_rows: table id 267 flags: STMT_END_F
### INSERT INTO `youngboy`.`t1`
### SET
###   @1=8 /* INT meta=0 nullable=1 is_null=0 */
###   @2='hdfeng2345' /* VARSTRING(80) meta=80 nullable=1 is_null=0 */
# at 478
#200224 10:45:37 server id 6  end_log_pos 509 CRC32 0x3681e7dd 	Xid = 1506670
COMMIT/*!*/;
---------------------------------------------------------------------------------------------------------------
mysqlbinlog -d world mysql-bin.000008								查看在某个库下的二进制日志
mysqlbinlog --start-datetime='2020-02-24 8:00:00' --stop-datetime='2020-02-24 9:00:00' mysql-bin.000008 按时间进行查看

日志的截取
–start-position=xxx --stop-position=xxx
下列是截取的内容

[root@localhost binlog]# mysqlbinlog --start-position=299 --stop-position=478 mysql-bin.000008
# at 299
#200224 10:45:28 server id 6  end_log_pos 375 CRC32 0xfc0d9d46 	Query	thread_id=53	exec_time=0	error_code=0
SET TIMESTAMP=1582512328/*!*/;
SET @@session.pseudo_thread_id=53/*!*/;
SET @@session.foreign_key_checks=1, @@session.sql_auto_is_null=0, @@session.unique_checks=1, @@session.autocommit=1/*!*/;
SET @@session.sql_mode=1436549152/*!*/;
SET @@session.auto_increment_increment=1, @@session.auto_increment_offset=1/*!*/;
/*!\C utf8 *//*!*/;
SET @@session.character_set_client=33,@@session.collation_connection=33,@@session.collation_server=45/*!*/;
SET @@session.lc_time_names=0/*!*/;
SET @@session.collation_database=DEFAULT/*!*/;
BEGIN
/*!*/;
# at 375
#200224 10:45:28 server id 6  end_log_pos 427 CRC32 0xe113aadb 	Table_map: `youngboy`.`t1` mapped to number 267
# at 427
#200224 10:45:28 server id 6  end_log_pos 478 CRC32 0x49664d8c 	Write_rows: table id 267 flags: STMT_END_F

BINLOG '
yDhTXhMGAAAANAAAAKsBAAAAAAsBAAAAAAEACHlvdW5nYm95AAJ0MQACAw8CUAAD26oT4Q==
yDhTXh4GAAAAMwAAAN4BAAAAAAsBAAAAAAEAAgAC//wIAAAACmhkZmVuZzIzNDWMTWZJ
'/*!*/;
ROLLBACK /* added by mysqlbinlog */ /*!*/;
SET @@SESSION.GTID_NEXT= 'AUTOMATIC' /* added by mysqlbinlog */ /*!*/;
DELIMITER ;
# End of log file
/*!50003 SET COMPLETION_TYPE=@OLD_COMPLETION_TYPE*/;
/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=0*/;
---------------------------------------------------------------------------------------------------------------
[root@localhost binlog]# mysqlbinlog --start-position=299 --stop-position=478 mysql-bin.000008 > /tmp/2020-02-24binlog.sql
将截取的binlog输出到sql文件,进行恢复使用

日志的删除

单机模式下:
方法1:

reset master;	直接删除所有日志从000001开始
查看当前数据库下binlog日志文件:
3306[(none)]>system ls -ltrh /opt/mysql-data/binlog		
总用量 16K
-rw-r-----. 1 mysql mysql 1.7K 2月  24 17:15 mysql-bin.000001
-rw-r-----. 1 mysql mysql  257 2月  24 17:15 mysql-bin.000002
-rw-r-----. 1 mysql mysql  120 2月  24 17:16 mysql-bin.index
-rw-r-----. 1 mysql mysql  447 2月  24 17:38 mysql-bin.000003
注:单机模式下可以这样进行删除,如果是主从模式下会导致主从数据不一致等一系列问题。

方法2:

purge master logs to 'mysql-bin.000002';
删除000002之前所有的binlog日志
3306[(none)]>system ls -ltrh /opt/mysql-data/binlog
总用量 8.0K
-rw-r-----. 1 mysql mysql 447 2月  24 17:38 mysql-bin.000003
-rw-r-----. 1 mysql mysql  40 2月  24 17:49 mysql-bin.index
删除了000003之前所有的日志

方法3:

用日期的方式删除,删除1751分之前的所有日志
3306[(none)]>system ls -ltrh /opt/mysql-data/binlog
总用量 44K
-rw-r-----. 1 mysql mysql 494 2月  24 17:50 mysql-bin.000003
-rw-r-----. 1 mysql mysql 281 2月  24 17:50 mysql-bin.000004
-rw-r-----. 1 mysql mysql 281 2月  24 17:50 mysql-bin.000005
-rw-r-----. 1 mysql mysql 281 2月  24 17:50 mysql-bin.000006
-rw-r-----. 1 mysql mysql 281 2月  24 17:50 mysql-bin.000007
-rw-r-----. 1 mysql mysql 281 2月  24 17:50 mysql-bin.000008
-rw-r-----. 1 mysql mysql 281 2月  24 17:50 mysql-bin.000009
-rw-r-----. 1 mysql mysql 281 2月  24 17:50 mysql-bin.000010
-rw-r-----. 1 mysql mysql 281 2月  24 17:51 mysql-bin.000011
-rw-r-----. 1 mysql mysql 234 2月  24 17:51 mysql-bin.000012
-rw-r-----. 1 mysql mysql 400 2月  24 17:51 mysql-bin.index

3306[(none)]>purge master logs before '2020-02-24 17:51:00';
Query OK, 0 rows affected (0.02 sec)
3306[(none)]>system ls -ltrh /opt/mysql-data/binlog
总用量 12K
-rw-r-----. 1 mysql mysql 281 2月  24 17:51 mysql-bin.000011
-rw-r-----. 1 mysql mysql 234 2月  24 17:51 mysql-bin.000012
-rw-r-----. 1 mysql mysql  80 2月  24 18:42 mysql-bin.index

方法4:
使用设置参数的方法去定期的让MySQL自己去删除日志

3306[(none)]>select @@expire_logs_days;
+--------------------+
| @@expire_logs_days |
+--------------------+
|                 15 |
+--------------------+
1 row in set (0.00 sec)
可以设置成:
set global expire_logs_days=8;设置全局模式的清理时间为8天
也可以在数据库配置文件中加入:
vim /etc/my.cnf
expire_logs_days=8;

binlog模拟日志丢失进行恢复(pos方式)

步骤:
1、创建测试库,以及数据录入,删除库的操作

3306[(none)]>create database binlogtest charset utf8mb4;
Query OK, 1 row affected (0.00 sec)
3306[(none)]>show binlog events in 'mysql-bin.000010';
+------------------+-----+----------------+-----------+-------------+---------------------------------------------------------------------------------------+
| Log_name         | Pos | Event_type     | Server_id | End_log_pos | Info                                                                                  |
+------------------+-----+----------------+-----------+-------------+---------------------------------------------------------------------------------------+
| mysql-bin.000010 |   4 | Format_desc    |         6 |         123 | Server ver: 5.7.20-log, Binlog ver: 4                                                 |
| mysql-bin.000010 | 123 | Previous_gtids |         6 |         234 | 0a99e3f9-3a95-11ea-87df-000c29b6e0ec:1-12,
979de5b8-3c01-11ea-8252-000c29b6e0ec:1-327 |
| mysql-bin.000010 | 234 | Gtid           |         6 |         299 | SET @@SESSION.GTID_NEXT= '979de5b8-3c01-11ea-8252-000c29b6e0ec:328'                   |
| mysql-bin.000010 | 299 | Query          |         6 |         427 | create database binlogtest charset utf8mb4                                            |
+------------------+-----+----------------+-----------+-------------+---------------------------------------------------------------------------------------+
4 rows in set (0.00 sec)
3306[(none)]>use binlogtest;
3306[binlogtest]>create table binlog1(id int,sname varchar(20));
Query OK, 0 rows affected (0.04 sec)
3306[binlogtest]>insert into binlog1 values(1,'sfsdfwr'),(2,'hdfeng1'),(3,'ljwfesf'),(4,'cxwsdf'),(5,'hdfeng2'),(6,'wet3465sdf');
Query OK, 6 rows affected (0.00 sec)
Records: 6  Duplicates: 0  Warnings: 0
3306[binlogtest]>drop database binlogtest;
Query OK, 1 row affected (0.05 sec)

2、库被删了,需要进行恢复,得先去找数据的开始点和删除表之前的结束点

3306[(none)]>show master status ;
+------------------+----------+--------------+------------------+---------------------------------------------------------------------------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set                                                                     |
+------------------+----------+--------------+------------------+---------------------------------------------------------------------------------------+
| mysql-bin.000010 |     1147 |              |                  | 0a99e3f9-3a95-11ea-87df-000c29b6e0ec:1-12,
979de5b8-3c01-11ea-8252-000c29b6e0ec:1-331 |
+------------------+----------+--------------+------------------+---------------------------------------------------------------------------------------+
1 row in set (0.00 sec)
3306[(none)]>show binlog  events in 'mysql-bin.000010';
+------------------+------+----------------+-----------+-------------+---------------------------------------------------------------------------------------+
| Log_name         | Pos  | Event_type     | Server_id | End_log_pos | Info                                                                                  |
+------------------+------+----------------+-----------+-------------+---------------------------------------------------------------------------------------+
| mysql-bin.000010 |    4 | Format_desc    |         6 |         123 | Server ver: 5.7.20-log, Binlog ver: 4                                                 |
| mysql-bin.000010 |  123 | Previous_gtids |         6 |         234 | 0a99e3f9-3a95-11ea-87df-000c29b6e0ec:1-12,
979de5b8-3c01-11ea-8252-000c29b6e0ec:1-327 |
| mysql-bin.000010 |  234 | Gtid           |         6 |         299 | SET @@SESSION.GTID_NEXT= '979de5b8-3c01-11ea-8252-000c29b6e0ec:328'                   |
| mysql-bin.000010 |  299 | Query          |         6 |         427 | create database binlogtest charset utf8mb4                                            |
| mysql-bin.000010 |  427 | Gtid           |         6 |         492 | SET @@SESSION.GTID_NEXT= '979de5b8-3c01-11ea-8252-000c29b6e0ec:329'                   |
| mysql-bin.000010 |  492 | Query          |         6 |         624 | use `binlogtest`; create table binlog1(id int,sname varchar(20))                      |
| mysql-bin.000010 |  624 | Gtid           |         6 |         689 | SET @@SESSION.GTID_NEXT= '979de5b8-3c01-11ea-8252-000c29b6e0ec:330'                   |
| mysql-bin.000010 |  689 | Query          |         6 |         767 | BEGIN                                                                                 |
| mysql-bin.000010 |  767 | Table_map      |         6 |         826 | table_id: 298 (binlogtest.binlog1)                                                    |
| mysql-bin.000010 |  826 | Write_rows     |         6 |         941 | table_id: 298 flags: STMT_END_F                                                       |
| mysql-bin.000010 |  941 | Xid            |         6 |         972 | COMMIT /* xid=1506711 */                                                              |
| mysql-bin.000010 |  972 | Gtid           |         6 |        1037 | SET @@SESSION.GTID_NEXT= '979de5b8-3c01-11ea-8252-000c29b6e0ec:331'                   |
| mysql-bin.000010 | 1037 | Query          |         6 |        1147 | drop database binlogtest                                                              |
+------------------+------+----------------+-----------+-------------+---------------------------------------------------------------------------------------+
13 rows in set (0.00 sec)
发现是从299-1037为止。

3、找到二进制日志以后,进行二进制日志的截取,并且导出

[root@localhost binlog]# mysqlbinlog --start-position=299 --stop-position=1037 mysql-bin.000010 > /tmp/binlogtest.sql
[root@localhost tmp]# ll
-rw-r--r--. 1 root  root  3145 2月  24 14:27 binlogtest.sql

4、截取成功以后,因为二进制日志会记录在数据库里面的操作,所以恢复之前需要先把二进制日志给关掉。

3306[(none)]>set sql_log_bin=0;
Query OK, 0 rows affected (0.00 sec)

5、进行数据的恢复,再打开二进制日志,进行正常数据的日志记录。

3306[(none)]>source /tmp/binlogtest.sql;
Query OK, 0 rows affected (0.01 sec)

Query OK, 0 rows affected (0.00 sec)

Charset changed
Query OK, 0 rows affected (0.00 sec)

Query OK, 0 rows affected, 1 warning (0.00 sec)

Database changed
Query OK, 0 rows affected (0.00 sec)

Query OK, 0 rows affected (0.03 sec)

执行成功以后看看库是否恢复
3306[binlogtest]>show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| binlogtest         |
| day1               |
| gtid               |
| mysql              |
| performance_schema |
| sys                |
| test               |
| world              |
| youngboy           |
+--------------------+
10 rows in set (0.00 sec)
3306[binlogtest]>select * from binlog1;
+------+------------+
| id   | sname      |
+------+------------+
|    1 | sfsdfwr    |
|    2 | hdfeng1    |
|    3 | ljwfesf    |
|    4 | cxwsdf     |
|    5 | hdfeng2    |
|    6 | wet3465sdf |
+------+------------+
6 rows in set (0.00 sec)
数据恢复成功!
开启binlog:
3306[binlogtest]>set  sql_log_bin=1;
Query OK, 0 rows affected (0.00 sec)
3306[binlogtest]>select @@sql_log_bin;
+---------------+
| @@sql_log_bin |
+---------------+
|             1 |
+---------------+
1 row in set (0.00 sec)

binlog(GTID方式)

说明:GTID(Global Transaction ID)全局事务编号,记录在日志中的每一个事务都会有一个这样的编号,并且这个编号是在数据库全局唯一的一个编号。
在这里插入图片描述
版本说明:

5.6开始这个新的特性,不开启没有这个功能
5.7对于GTID的功能进行了加强,不开也自动生成,关闭比较麻烦,需要3个步骤

相对于pos方式的优点:pos如果遇到截取的内容是分段的,就要分次进行截取,而GTID则不用,直接截取相应的gtid号即可截出对应日志,而且还方便恢复。

GTID参数说明

参数介绍:

vim /etc/my.cnf
gtid-mode=on					GTID模式开启
enforce-gtid-consistency=true	强制GTID一致性

GTID的使用

1、查看GTID号,下列Executed_Gtid_Set中就是GTID号码,冒号前的是source ID,后面的1-12,1-335就是事务号。

3306[gtid]>show master status ;
+------------------+----------+--------------+------------------+---------------------------------------------------------------------------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set                                                                     |
+------------------+----------+--------------+------------------+---------------------------------------------------------------------------------------+
| mysql-bin.000016 |      843 |              |                  | 0a99e3f9-3a95-11ea-87df-000c29b6e0ec:1-12,
979de5b8-3c01-11ea-8252-000c29b6e0ec:1-335 |
+------------------+----------+--------------+------------------+---------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

2、DDL/DML/DCL语句都会被记录到GTID内,每一个操作都会有一个全局事务号,如果是DML事务,从begin到commit会生产一个事务号。

3306[gtid]>show binlog events in 'mysql-bin.000016';
+------------------+-----+----------------+-----------+-------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Log_name         | Pos | Event_type     | Server_id | End_log_pos | Info                                                                                                                                           |
+------------------+-----+----------------+-----------+-------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| mysql-bin.000016 |   4 | Format_desc    |         7 |         123 | Server ver: 5.7.20-log, Binlog ver: 4                                                                                                          |
| mysql-bin.000016 | 123 | Previous_gtids |         7 |         234 | 0a99e3f9-3a95-11ea-87df-000c29b6e0ec:1-12,
979de5b8-3c01-11ea-8252-000c29b6e0ec:1-332                                                          |
| mysql-bin.000016 | 234 | Gtid           |         7 |         299 | SET @@SESSION.GTID_NEXT= '979de5b8-3c01-11ea-8252-000c29b6e0ec:333'                                                                            |
| mysql-bin.000016 | 299 | Query          |         7 |         409 | create database gtid charset utf8mb4                                                                                                           |
| mysql-bin.000016 | 409 | Gtid           |         7 |         474 | SET @@SESSION.GTID_NEXT= '979de5b8-3c01-11ea-8252-000c29b6e0ec:334'                                                                            |
| mysql-bin.000016 | 474 | Query          |         7 |         571 | use `gtid`; create table t(id int )                                                                                                            |
| mysql-bin.000016 | 571 | Gtid           |         7 |         636 | SET @@SESSION.GTID_NEXT= '979de5b8-3c01-11ea-8252-000c29b6e0ec:335'                                                                            |
| mysql-bin.000016 | 636 | Query          |         7 |         708 | BEGIN                                                                                                                                          |
| mysql-bin.000016 | 708 | Table_map      |         7 |         752 | table_id: 230 (gtid.t)                                                                                                                         |
| mysql-bin.000016 | 752 | Write_rows     |         7 |         812 | table_id: 230 flags: STMT_END_F                                                                                                                |
| mysql-bin.000016 | 812 | Xid            |         7 |         843 | COMMIT /* xid=82 */                                                                                                                            |
| mysql-bin.000016 | 843 | Gtid           |         7 |         908 | SET @@SESSION.GTID_NEXT= '979de5b8-3c01-11ea-8252-000c29b6e0ec:336'                                                                            |
| mysql-bin.000016 | 908 | Query          |         7 |        1129 | use `gtid`; GRANT SELECT ON *.* TO 'hdfeng'@'localhost' IDENTIFIED WITH 'mysql_native_password' AS '*23AE809DDACAF96AF0FD78ED04B6A265E05AA257' |
+------------------+-----+----------------+-----------+-------------+------------------------------------------------------------------------------------------------------------------------------------------------+
13 rows in set (0.00 sec)

3306[gtid]>drop database gtid;
Query OK, 1 row affected (0.02 sec)
模拟数据库被删除的过程
使用GTID查看binlog以及GTID恢复日志

查看及截取命令:

--include-gtids  值为需要恢复的gtid事务号范围
--exclude-gtids	 值为需要路过的gtid事务号范围
当需要跳过多个号码的时候可以用逗号格开

mysqlbinlog --include-gtids='979de5b8-3c01-11ea-8252-000c29b6e0ec:333-336' --exclude-gtids='979de5b8-3c01-11ea-8252-000c29b6e0ec:xxx' /opt/mysql-data/binlog/mysql-bin.000016 
因为中间没有其它事务,所以不需要跳过事务ID
[root@localhost ~]# mysqlbinlog --include-gtids='979de5b8-3c01-11ea-8252-000c29b6e0ec:333-336'  /opt/mysql-data/binlog/mysql-bin.000016
/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=1*/;
/*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/;
DELIMITER /*!*/;
# at 4
#200224 18:44:10 server id 7  end_log_pos 123 CRC32 0x8dc43445 	Start: binlog v 4, server v 5.7.20-log created 200224 18:44:10
# Warning: this binlog is either in use or was not closed properly.
BINLOG '
+qhTXg8HAAAAdwAAAHsAAAABAAQANS43LjIwLWxvZwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAEzgNAAgAEgAEBAQEEgAAXwAEGggAAAAICAgCAAAACgoKKioAEjQA
AUU0xI0=
'/*!*/;
# at 123
#200224 18:44:10 server id 7  end_log_pos 234 CRC32 0x2bd7914a 	Previous-GTIDs
# 0a99e3f9-3a95-11ea-87df-000c29b6e0ec:1-12,
# 979de5b8-3c01-11ea-8252-000c29b6e0ec:1-332
# at 234
#200225 10:29:32 server id 7  end_log_pos 299 CRC32 0xa6a3bfb1 	GTID	last_committed=0	sequence_number=1	rbr_only=no
SET @@SESSION.GTID_NEXT= '979de5b8-3c01-11ea-8252-000c29b6e0ec:333'/*!*/;
# at 299
#200225 10:29:32 server id 7  end_log_pos 409 CRC32 0x418618ab 	Query	thread_id=7	exec_time=0	error_code=0
SET TIMESTAMP=1582597772/*!*/;
SET @@session.pseudo_thread_id=7/*!*/;
SET @@session.foreign_key_checks=1, @@session.sql_auto_is_null=0, @@session.unique_checks=1, @@session.autocommit=1/*!*/;
SET @@session.sql_mode=1436549152/*!*/;
SET @@session.auto_increment_increment=1, @@session.auto_increment_offset=1/*!*/;
/*!\C utf8 *//*!*/;
SET @@session.character_set_client=33,@@session.collation_connection=33,@@session.collation_server=45/*!*/;
SET @@session.lc_time_names=0/*!*/;
SET @@session.collation_database=DEFAULT/*!*/;
create database gtid charset utf8mb4
/*!*/;
# at 409
#200225 10:34:52 server id 7  end_log_pos 474 CRC32 0xee989acd 	GTID	last_committed=1	sequence_number=2	rbr_only=no
SET @@SESSION.GTID_NEXT= '979de5b8-3c01-11ea-8252-000c29b6e0ec:334'/*!*/;
# at 474
#200225 10:34:52 server id 7  end_log_pos 571 CRC32 0x70133894 	Query	thread_id=7	exec_time=1	error_code=0
use `gtid`/*!*/;
SET TIMESTAMP=1582598092/*!*/;
create table t(id int )
/*!*/;
# at 571
#200225 10:36:57 server id 7  end_log_pos 636 CRC32 0x1b423a86 	GTID	last_committed=2	sequence_number=3	rbr_only=yes
/*!50718 SET TRANSACTION ISOLATION LEVEL READ COMMITTED*//*!*/;
SET @@SESSION.GTID_NEXT= '979de5b8-3c01-11ea-8252-000c29b6e0ec:335'/*!*/;
# at 636
#200225 10:36:54 server id 7  end_log_pos 708 CRC32 0xee84dc06 	Query	thread_id=7	exec_time=0	error_code=0
SET TIMESTAMP=1582598214/*!*/;
BEGIN
/*!*/;
# at 708
#200225 10:36:54 server id 7  end_log_pos 752 CRC32 0x97603303 	Table_map: `gtid`.`t` mapped to number 230
# at 752
#200225 10:36:54 server id 7  end_log_pos 812 CRC32 0x5d3de002 	Write_rows: table id 230 flags: STMT_END_F

BINLOG '
RohUXhMHAAAALAAAAPACAAAAAOYAAAAAAAEABGd0aWQAAXQAAQMAAQMzYJc=
RohUXh4HAAAAPAAAACwDAAAAAOYAAAAAAAEAAgAB//4BAAAA/gIAAAD+AwAAAP4EAAAA/gUAAAAC
4D1d
'/*!*/;
# at 812
#200225 10:36:57 server id 7  end_log_pos 843 CRC32 0x2804d8c9 	Xid = 82
COMMIT/*!*/;
# at 843
#200225 10:44:24 server id 7  end_log_pos 908 CRC32 0x54ab549a 	GTID	last_committed=3	sequence_number=4	rbr_only=no
SET @@SESSION.GTID_NEXT= '979de5b8-3c01-11ea-8252-000c29b6e0ec:336'/*!*/;
# at 908
#200225 10:44:24 server id 7  end_log_pos 1129 CRC32 0xda23c80a 	Query	thread_id=7	exec_time=0	error_code=0
SET TIMESTAMP=1582598664/*!*/;
GRANT SELECT ON *.* TO 'hdfeng'@'localhost' IDENTIFIED WITH 'mysql_native_password' AS '*23AE809DDACAF96AF0FD78ED04B6A265E05AA257'
/*!*/;
SET @@SESSION.GTID_NEXT= 'AUTOMATIC' /* added by mysqlbinlog */ /*!*/;
DELIMITER ;
# End of log file
/*!50003 SET COMPLETION_TYPE=@OLD_COMPLETION_TYPE*/;
/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=0*/;

---------------------------------------------------------------------------------------------------------------
进行日志导出恢复:
[root@localhost ~]# mysqlbinlog --include-gtids='979de5b8-3c01-11ea-8252-000c29b6e0ec:333-336'  /opt/mysql-data/binlog/mysql-bin.000016 >/tmp/gtid.sql
---------------------------------------------------------------------------------------------------------------
3306[(none)]>set sql_log_bin=0;
Query OK, 0 rows affected (0.00 sec)
3306[(none)]> source /tmp/gtid1.sql;

ERROR 1049 (42000): Unknown database 'gtid'
Query OK, 0 rows affected (0.00 sec)

ERROR 1046 (3D000): No database selected
Query OK, 0 rows affected (0.00 sec)
发现报错,这是为啥?这个是因为GTID有幂等性的原因。
GTID的幂等性

什么又叫GTID的幂等性呢?这个就是开启了GTID之后,恢复binlog的时候,MySQL的重复GTID不读取的操作。数据库一旦发现有相同的GTID号,就会直接跳过不执行,这也是为了保证主从复制一致性。

需要加参数:

--skip-gtids 		跳过重复的GTID号
[root@localhost ~]# mysqlbinlog --skip-gtids --include-gtids='979de5b8-3c01-11ea-8252-000c29b6e0ec:333-336'  /opt/mysql-data/binlog/mysql-bin.000016 >/tmp/gtid1.sql
加入以后再进行恢复就不会报错了
3306[(none)]>source /tmp/gtid1.sql;
可以查看与原来截取出来的日志进行对比:
[root@hdfeng ~]# vimdiff /tmp/gtid.sql /tmp/gtid1.sql
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值