salt多master使用failover方式倒换后minion连接master异常

woshiqhj

测试环境：
    2台salt-master
      - 192.168.57.187
      - 192.168.57.190
    1台salt-minion
      - 192.168.57.188

salt版本：

1. [root@at-saltmaster-1 ~]# salt --versions-report
   
2. Salt Version:
   
3.            Salt: 2015.8.3
   
4. 
   
5. Dependency Versions:
   
6.          Jinja2: 2.7.2
   
7.        M2Crypto: 0.21.1
   
8.            Mako: Not Installed
   
9.          PyYAML: 3.11
   
10.           PyZMQ: 14.7.0
    
11.          Python: 2.7.5 (default, Jun 17 2014, 18:11:42)
    
12.            RAET: Not Installed
    
13.         Tornado: 4.2.1
    
14.             ZMQ: 4.0.5
    
15.            cffi: Not Installed
    
16.        cherrypy: 3.2.2
    
17.        dateutil: Not Installed
    
18.           gitdb: Not Installed
    
19.       gitpython: Not Installed
    
20.           ioflo: Not Installed
    
21.         libnacl: Not Installed
    
22.    msgpack-pure: Not Installed
    
23. msgpack-python: 0.4.6
    
24.    mysql-python: Not Installed
    
25.       pycparser: Not Installed
    
26.        pycrypto: 2.6.1
    
27.          pygit2: Not Installed
    
28.    python-gnupg: Not Installed
    
29.           smmap: Not Installed
    
30.         timelib: Not Installed
    
31. 
    
32. System Versions:
    
33.            dist: centos 7.1.1503 Core
    
34.         machine: x86_64
    
35.         release: 3.10.0-229.el7.x86_64
    
36.          system: CentOS Linux 7.1.1503 Core
    

复制代码

salt配置：
master-1：

1. interface: 192.168.57.187
   
2. master_sign_pubkey: True
   
3. master_use_pubkey_signature: True
   
4. master_pubkey_signature: at-saltmaster-1

复制代码

master-2：

1. interface: 192.168.57.190
   
2. master_sign_pubkey: True
   
3. master_use_pubkey_signature: True
   
4. master_pubkey_signature: at-saltmaster-1

复制代码

minion-1：

1. default_include: minion.d/*.conf
   
2. master:
   
3. - 192.168.57.187
   
4. - 192.168.57.190
   
5. random_master: False
   
6. master_type: failover
   
7. master_alive_interval: 15
   
8. verify_master_pubkey_sign: True
   
9. always_verify_signature: True
   
10. ipv6: False
    
11. retry_dns: 0
    
12. master_port: 4506
    
13. user: root
    
14. color: True
    
15. rejected_retry: True
    
16. random_reauth_delay: 15
    
17. auth_timeout: 15
    
18. auth_tries: 1
    
19. auth_safemode: False
    
20. recon_default: 1000
    
21. recon_max: 5000
    
22. recon_randomize: Tru

复制代码

启动2台salt-master和1台salt-minion后，minion能够连接到master-1，在master-1上能够正常操作minion-1，当把master-1关闭后，minion-1能够自动切换到master-2，但是在master-2上操作minion-1，总是提示

1. [root@saltmaster-2 ]# salt "*" cmd.run "df -h"
   
2. minion-1:
   
3.     Minion did not return. [No response]

复制代码

查看两台master的debug日志都显示已鉴权通过，但是查看minion的debug日志看到有一点异常：
这是连接master-1的鉴权日志：

1. [DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/minion', 'minion-1', 'tcp://192.168.57.187:4506', 'clear')
   
2. [DEBUG   ] salt.crypt.verify_signature: Loading public key
   
3. [DEBUG   ] salt.crypt.verify_signature: Verifying signature
   
4. [DEBUG   ] Successfully verified signature of master public key with verification public key master_sign.pub
   
5. [INFO    ] Received signed and verified master pubkey from master 192.168.57.187
   
6. [DEBUG   ] Decrypting the current master AES key

复制代码

这是切换后连接master-2的鉴权日志：

1. [DEBUG   ] salt.crypt.verify_signature: Loading public key
   
2. [DEBUG   ] salt.crypt.verify_signature: Verifying signature
   
3. [DEBUG   ] Successfully verified signature of master public key with verification public key master_sign.pub
   
4. [INFO    ] Received signed and verified master pubkey from master <tornado.concurrent.Future object at 0x202c750>
   
5. [DEBUG   ] Decrypting the current master AES key

复制代码

查看代码这句日志是由crypt.py中verify_signing_master方法中打印的：

1. def verify_signing_master(self, payload):
   
2.     720         try:
   
3.     721             if self.verify_pubkey_sig(payload['pub_key'],
   
4.     722                                       payload['pub_sig']):
   
5.     723                 log.info('Received signed and verified master pubkey '
   
6.     724                          'from master {0}'.format(self.opts['master']))

复制代码

想请各位帮忙看看为何会打印出<tornado.concurrent.Future object at 0x202c750>，是不是因为这个导致切换后master-2与minion-1的zmq连接异常？

woshiqhj

master和minion的日志：

woshiqhj

已经确认是个bug