A机器无法稳定调用B机器4567端口服务，如何troubleshooting?

我是个野鸭子

现有情况：
A机器是一个linux主机，上面有一个java application
B机器是一个linux主机，有一个application在端口4567提供服务
当前A机器的java application不停的调用B机器的4567服务，
但是不稳定，一天会出现1-2次的调用悬停。。

进展情况：
A机器的java application，和B机器的application，都是正常的
这里的话题，请排除关于application级别的讨论

现在的问题是：
1. 如何从网络的角度，实时的探查当时的网络状态？A B两个主机的网络是否连通？ 如果连通，A访问B的4567是否当时连通？
2. 如何从主机稳定的角度，实时的探查当时的主机运行的稳定性？出现异常时，A,B两个主机是否正常？（我很零星的发现，发现异常时，SSH登录A主机会悬停; 再次登录，会变得正常。另外，我们的主机,storage坏过好几次）
3. 如何从A, B主机内,linux相关网络模块或服务的稳定性来实时探查异常？ 如何监控A, B主机在OS层级的网络模块？及其基于这些网络模块的关键services?比如是否出现异常时，firewall突然被什么其他应用所打开？
4. 如何从A, B主机相关硬件，比如网卡稳定性的角度来实时探查异常？ 我10几年前用的商业unix主机，提供实时探查硬件设备的方法。不知道linux是否提供相关的硬件状态的探查方法？


解决问题的角度：
1. 日志
我应该去重点查看哪些日志？我需要怎样打开这些日志？这些日志都在哪里？这些日志是否实时输出信息？
2. 指令
有什么好指令来做相关的实时探查？
比如在网络连通性方面， telnet B 4567 ;  nmap B | grep 4567   还有什么其他更好的指令码？
3. 脚本
我想解决这个问题后，写个脚本。所以请大家在提供相关的指令时，最好能做实时capture的指令，而不是telnet B 4567这样的交互型指令


其他重点：
请根据各位的经验，除了以上方面，我还遗漏了哪些方面？我该如何探查？

q1208c

先查 swap. 从你说的现象上看, 很象是 内存用光了, 机器在用 swap了.

我是个野鸭子

thanks a lot for your information
do you have any more information?

it's really what I need.....

我是个野鸭子

其他人还有没有好点的建议？

我是个野鸭子

sorry...

can any one help me?

You know, stackoverflow could not be post any questions since gmail is forbidden...
So I have to ask for your nice help at this forum...

after all, chinaunix is the top site for Chinese Unix/Linux guys..
we need to improve our ability to answer some high level questions
we need to overwhelm stackoverflow...  then china will overwhelm american...   )

q1208c

你贴了这么大一堆的问题, 那么, 你有没有做过什么? 又得出什么结论呢?

没有人能透过网络, 精确的定位的你的问题, 而且, 你没有贴上任何的日志, 也没有任何的可供参考的数据, 仅仅只有一个问题, 你觉得谁有这么大的本事能猜出原因呢?

我是个野鸭子

q1208c 发表于 2014-06-24 08:04
你贴了这么大一堆的问题, 那么, 你有没有做过什么? 又得出什么结论呢?

没有人能透过网络, 精确的定位的 ...

1. 如何从网络的角度，实时的探查当时的网络状态？A B两个主机的网络是否连通？ 如果连通，A访问B的4567是否当时连通？
[What I had done]
At that issue time, I checked the network connection immediately by below ways:
A.  I used "telnet B 4567".  I got a success result
B.  I used "nmap B". Then I found it provided 4567 service.
But I don't know why the app could not connect B's 4567 service.
So I think I need to have a real-time capture shell to capture whether or not the connectivity is stable.
What's this command to do so?  I don't think "telnet B 4567" is good, because it is an interactive command. Right?
So "nmap B | grep 4567" is good, right? I can put it in a loop and capture the status in every 2 seconds.
Do you have any other ways? I suspect I missed some MORE BETTER ways. One useful utilities is "sar". I remembered solaries contribute this wonderful utility to linux fundation. right? I don't try it. I'm not sure do you have some clue for me?

我是个野鸭子

q1208c 发表于 2014-06-24 08:04
你贴了这么大一堆的问题, 那么, 你有没有做过什么? 又得出什么结论呢?

没有人能透过网络, 精确的定位的 ...

2. 如何从主机稳定的角度，实时的探查当时的主机运行的稳定性？出现异常时，A,B两个主机是否正常？（我很零星的发现，发现异常时，SSH登录A主机会悬停; 再次登录，会变得正常。另外，我们的主机,storage坏过好几次）
[What I had done]
Sorry. I really could not find any ways to check it.
If it's a Unix(AIX, HP-UX or Solaris), there DO are may ways.
But for linux, I don't find some kind of ways, except some raw commands like iostat...
In generally, if there are some key issues, we could find it out at /var/log. Right?
But I also could not check any logs at this folder.
After all, my issue is not stable, it is not reproduced. /var/log is more like for reproduced issues. right?

One more thing I would like to highlight here is that our system's storage is not stable. We set up a RAID5 at bottom and then install RHEL6.4 at it.
But looks the storage is not stable. Is this issue impact our system's stable?
Do you have any experiences?
Looking forward to sharing your story.

我是个野鸭子

q1208c 发表于 2014-06-24 08:04
你贴了这么大一堆的问题, 那么, 你有没有做过什么? 又得出什么结论呢?

没有人能透过网络, 精确的定位的 ...

3. 如何从A, B主机内,linux相关网络模块或服务的稳定性来实时探查异常？ 如何监控A, B主机在OS层级的网络模块？及其基于这些网络模块的关键services?比如是否出现异常时，firewall突然被什么其他应用所打开？
[What I had done]
Based on the basic linux network theory, any network services between different hosts is based at socket level. Right?
So if there DID are many network issues at OS network related modules, I could find the key information at syslog. Right?
But I could not find it.
So I have to ask for your nice help how to do.
What's other log could help me?
How to turn on these logs file?
Yes, you may also ask me "hi guy, why not have sniff??"  yes.. it's good utility, but you known, it's a production machine... hmmmmmmm.. so.. you known...

我是个野鸭子

q1208c 发表于 2014-06-24 08:04
你贴了这么大一堆的问题, 那么, 你有没有做过什么? 又得出什么结论呢?

没有人能透过网络, 精确的定位的 ...

4. 如何从A, B主机相关硬件，比如网卡稳定性的角度来实时探查异常？ 我10几年前用的商业unix主机，提供实时探查硬件设备的方法。不知道linux是否提供相关的硬件状态的探查方法？
[What I had done]
Sorry. I really don't know how to do it at linux...
At popular Unix IV platforms, like AIX HP-UX or Solaris, every hardware maps a command to check its status
But how to do it at linux?
Our linux is RHEL6.4
Our hardware is from DELL. I suspect I should ask it from DELL?
Give me some advice..