[基于Epoll内置Leader-Follower服务端实现, 已可达50万echo qps（全新支持Lua啦）]

windoze

回复 249# yulihua49

Fiberized.IO也有一模一样的问题，每一个fiber scheduler共用一个asio::io_service，里面就是一个epfd，一个scheduler下的fiber太多性能就会下降。
所以最后我在程序里加了个功能，让一个进程支持多个fiber schduler，每个scheduler使用独立的io_service和独立的线程池。

yulihua49

answer3y 发表于 2015-09-07 12:00
回复 249# yulihua49

实际上每秒1000000的qps意义不大。如果每个request有2K数据（这是一般事务的量）你需要2K*8*1M=16G的带宽，现在的网络还普及不了这样的资源。

现在我的指标是65000Tps，2KByte/T,2K*8*65000=1G,正好是普及局域网的带宽，内部也正好是13Wqps。各服务器的CPU也正好是85%。（某种业务的特例，每笔业务大约是30k的json格式，压缩后大约2K）。

个别指标超过这个，是个浪费，因为别的资源跟不上，为个别指标付出代价是否值得。

yulihua49

windoze 发表于 2015-09-07 12:38
回复 249# yulihua49

Fiberized.IO也有一模一样的问题，每一个fiber scheduler共用一个asio::io_servic ...

哪个客户端分给哪个io_service，就需要负载均衡。
如果任务是异质的，就需要动态智能负载均衡，我做过这个东西，效果还是很不错的。
可惜的是，它也是依赖epoll和线程池，也受到15Wqps的限制。
我那个东西用于一个带数据库的业务，总能力为3W Tps。所以应付的还不错。

所以我说，你不必搞什么多sched，这个指标，到这是个平台，超过这个数，就需要全新的架构，成本上台阶了。

windoze

回复 253# yulihua49

我正在搞GPU thread pool，它要和CPU thread pool分开调度，所以我才做了multi-scheduler。

yulihua49

windoze 发表于 2015-09-07 15:18
回复 253# yulihua49

我正在搞GPU thread pool，它要和CPU thread pool分开调度，所以我才做了multi-sc ...

show一个我的协程池的工作日志。应该说明的是，由于打日志，改变了实际流程。否则应该交替的更频繁

1. 
   
2. 5 CLIBG:127.0.0.1:70 09/08 14:52'14 RecvNet:tid=7FE9921FC700,socket=14,yield to schedle bcount=16348/402650
   
3. 5 CLIBG:127.0.0.1:70 09/08 14:52'14 thread_work:tid=7FE9921FC700,fiber yield from TCB_no=70
   
4. 5 CLIBG:127.0.0.1:70 09/08 14:52'14 thread_work:tid=7FE9921FC700,resume to TCB_no=70
   
5. 5 CLIBG:127.0.0.1:70 09/08 14:52'14 RecvNet:tid=7FE9921FC700,socket=14,yield to schedle bcount=32732/402650
   
6. 5 CLIBG:127.0.0.1:70 09/08 14:52'14 thread_work:tid=7FE9921FC700,fiber yield from TCB_no=70
   
7. 5 CLIBG:127.0.0.1:70 09/08 14:52'14 thread_work:tid=7FE990DFA700,resume to TCB_no=70
   
8. 5 CLIBG:127.0.0.1:70 09/08 14:52'14 RecvNet:tid=7FE990DFA700,socket=14,yield to schedle bcount=98268/402650
   
9. 5 CLIBG:127.0.0.1:70 09/08 14:52'14 thread_work:tid=7FE990DFA700,fiber yield from TCB_no=70
   
10. 5 CLIBG:127.0.0.1:70 09/08 14:52'14 thread_work:tid=7FE990DFA700,resume to TCB_no=70
    
11. 5 CLIBG:127.0.0.1:70 09/08 14:52'14 RecvNet:tid=7FE990DFA700,socket=14,yield to schedle bcount=376796/402650
    
12. 5 CLIBG:127.0.0.1:70 09/08 14:52'14 thread_work:tid=7FE990DFA700,fiber yield from TCB_no=70
    
13. 5 CLIBG:127.0.0.1:70 09/08 14:52'14 thread_work:tid=7FE990DFA700,resume to TCB_no=70
    
14. 5 CLIBG:127.0.0.1:71 09/08 14:52'14 thread_work:7FE9921FC700 create fiber for TCB_no=71
    
15. 5 CLIBG:127.0.0.1:71 09/08 14:52'14 RecvNet:tid=7FE9921FC700,socket=15,yield to schedle bcount=16348/422169
    
16. 5 CLIBG:127.0.0.1:71 09/08 14:52'14 thread_work:tid=7FE9921FC700,fiber yield from TCB_no=71
    
17. 5 CLIBG:127.0.0.1:71 09/08 14:52'14 thread_work:tid=7FE9921FC700,resume to TCB_no=71
    
18. 5 CLIBG:127.0.0.1:71 09/08 14:52'14 RecvNet:tid=7FE9921FC700,socket=15,yield to schedle bcount=32732/422169
    
19. 5 CLIBG:127.0.0.1:71 09/08 14:52'14 thread_work:tid=7FE9921FC700,fiber yield from TCB_no=71
    
20. 5 CLIBG:127.0.0.1:71 09/08 14:52'14 thread_work:tid=7FE9921FC700,resume to TCB_no=71
    
21. 5 CLIBG:127.0.0.1:71 09/08 14:52'14 RecvNet:tid=7FE9921FC700,socket=15,yield to schedle bcount=163804/422169
    
22. 5 CLIBG:127.0.0.1:71 09/08 14:52'14 thread_work:tid=7FE9921FC700,fiber yield from TCB_no=71
    
23. 5 CLIBG:127.0.0.1:71 09/08 14:52'14 thread_work:tid=7FE9921FC700,resume to TCB_no=71
    
24. 5 CLIBG:127.0.0.1:71 09/08 14:52'14 RecvNet:tid=7FE9921FC700,socket=15,yield to schedle bcount=180188/422169
    
25. 5 CLIBG:127.0.0.1:71 09/08 14:52'14 thread_work:tid=7FE9921FC700,fiber yield from TCB_no=71
    
26. 5 CLIBG:127.0.0.1:71 09/08 14:52'14 thread_work:tid=7FE9921FC700,resume to TCB_no=71
    
27. 5 CLIBG:127.0.0.1:71 09/08 14:52'14 RecvNet:tid=7FE9921FC700,socket=15,yield to schedle bcount=196572/422169
    
28. 5 CLIBG:127.0.0.1:71 09/08 14:52'14 thread_work:tid=7FE9921FC700,fiber yield from TCB_no=71
    
29. 5 CLIBG:127.0.0.1:71 09/08 14:52'14 thread_work:tid=7FE9921FC700,resume to TCB_no=71
    
30. 5 CLIBG:127.0.0.1:72 09/08 14:52'14 RecvNet:tid=7FE992BFD700,socket=16,yield to schedle bcount=32732/397007
    
31. 5 CLIBG:127.0.0.1:72 09/08 14:52'14 thread_work:tid=7FE992BFD700,fiber yield from TCB_no=72
    
32. 5 CLIBG:127.0.0.1:71 09/08 14:52'14 thread_work:tid=7FE9921FC700,resume to TCB_no=71
    
33. 5 CLIBG:127.0.0.1:71 09/08 14:52'14 RecvNet:tid=7FE9921FC700,socket=15,yield to schedle bcount=180188/422169
    
34. 5 CLIBG:127.0.0.1:71 09/08 14:52'14 thread_work:tid=7FE9921FC700,fiber yield from TCB_no=71
    
35. 5 CLIBG:127.0.0.1:71 09/08 14:52'14 thread_work:tid=7FE9921FC700,resume to TCB_no=71
    
36. 5 CLIBG:127.0.0.1:71 09/08 14:52'14 RecvNet:tid=7FE9921FC700,socket=15,yield to schedle bcount=196572/422169
    
37. 5 CLIBG:127.0.0.1:71 09/08 14:52'14 thread_work:tid=7FE9921FC700,fiber yield from TCB_no=71
    
38. 5 CLIBG:127.0.0.1:71 09/08 14:52'14 thread_work:tid=7FE9921FC700,resume to TCB_no=71
    
39. 5 CLIBG:127.0.0.1:72 09/08 14:52'14 RecvNet:tid=7FE992BFD700,socket=16,yield to schedle bcount=32732/397007
    
40. 5 CLIBG:127.0.0.1:72 09/08 14:52'14 thread_work:tid=7FE992BFD700,fiber yield from TCB_no=72
    
41. 5 CLIBG:127.0.0.1:72 09/08 14:52'14 thread_work:tid=7FE992BFD700,resume to TCB_no=72
    
42. 5 CLIBG:127.0.0.1:72 09/08 14:52'14 RecvNet:tid=7FE992BFD700,socket=16,yield to schedle bcount=65500/397007
    
43. 5 CLIBG:127.0.0.1:72 09/08 14:52'14 thread_work:tid=7FE992BFD700,fiber yield from TCB_no=72
    
44. 5 CLIBG:127.0.0.1:73 09/08 14:52'14 thread_work:7FE992BFD700 create fiber for TCB_no=73
    
45. 5 CLIBG:127.0.0.1:72 09/08 14:52'14 thread_work:tid=7FE9917FB700,resume to TCB_no=72
    
46. 5 CLIBG:127.0.0.1:72 09/08 14:52'14 RecvNet:tid=7FE9917FB700,socket=16,yield to schedle bcount=163804/397007
    
47. 5 CLIBG:127.0.0.1:72 09/08 14:52'14 thread_work:tid=7FE9917FB700,fiber yield from TCB_no=72
    
48. 5 CLIBG:127.0.0.1:73 09/08 14:52'14 RecvNet:tid=7FE992BFD700,socket=17,yield to schedle bcount=163804/435097
    
49. 5 CLIBG:127.0.0.1:73 09/08 14:52'14 thread_work:tid=7FE992BFD700,fiber yield from TCB_no=73
    
50. 5 CLIBG:127.0.0.1:72 09/08 14:52'14 thread_work:tid=7FE992BFD700,resume to TCB_no=72
    
51. 5 CLIBG:127.0.0.1:72 09/08 14:52'14 RecvNet:tid=7FE992BFD700,socket=16,yield to schedle bcount=278492/397007
    
52. 5 CLIBG:127.0.0.1:72 09/08 14:52'14 thread_work:tid=7FE992BFD700,fiber yield from TCB_no=72
    
53. 5 CLIBG:127.0.0.1:72 09/08 14:52'14 thread_work:tid=7FE9917FB700,resume to TCB_no=72
    
54. 5 CLIBG:127.0.0.1:73 09/08 14:52'14 thread_work:tid=7FE992BFD700,resume to TCB_no=73
    
55. 5 CLIBG:127.0.0.1:73 09/08 14:52'14 RecvNet:tid=7FE992BFD700,socket=17,yield to schedle bcount=212956/435097
    
56. 5 CLIBG:127.0.0.1:73 09/08 14:52'14 thread_work:tid=7FE992BFD700,fiber yield from TCB_no=73
    
57. 5 CLIBG:127.0.0.1:73 09/08 14:52'14 thread_work:tid=7FE992BFD700,resume to TCB_no=73
    
58. 4 CLIBG:127.0.0.1:70 09/08 14:52'14 do_work: TCB_no=70,tid=7FE9921FC700,PROTO_NUM=3 pkg_len=5519827,t_len=402650,O_NODE=0,USEC=3650770334140237
    
59. 4 CLIBG:127.0.0.1:71 09/08 14:52'14 do_work: TCB_no=71,tid=7FE9921FC700,PROTO_NUM=3 pkg_len=5549250,t_len=422169,O_NODE=0,USEC=3650770334148224
    
60. 4 CLIBG:127.0.0.1:72 09/08 14:52'14 do_work: TCB_no=72,tid=7FE992BFD700,PROTO_NUM=3 pkg_len=5658893,t_len=397007,O_NODE=0,USEC=3650770334187568
    
61. 4 CLIBG:127.0.0.1:73 09/08 14:52'14 do_work: TCB_no=73,tid=7FE992BFD700,PROTO_NUM=3 pkg_len=5568378,t_len=435097,O_NODE=0,USEC=3650770334219294
    
62. 
    

复制代码

注意39-56行，两个线程对两个任务，交替进行。
可以说，在NIO下，qps会更少，因为收一个大包要激活epoll好多次。对于一个任务而言，显然要比SIO耗时长。

windoze

回复 255# yulihua49

这不是显然的么？AsyncIO的优势本来就不在响应处理速度，而在于支持大规模并发。

yulihua49

windoze 发表于 2015-09-08 16:55
回复 255# yulihua49

这不是显然的么？AsyncIO的优势本来就不在响应处理速度，而在于支持大规模并发。

是。网络上有人企图使用AIO/NIO提升处理速度。在非并发环境这是不可能的。

p_loco

也做了一个，但是是作为plugin的形式存在的

wlmqgzm

顶一下, 我的ECHO SERVER目前双核上已经达到32万QPS ECHO,  是基于BOOST ASIO的,  我是第一次做网络服务器代码.
你算前辈了, 向你学习.

ylwxb

#define CONN_MAXFD 65536
struct connection_st g_conn_table[CONN_MAXFD] = {0};

看了下楼主的源码，定义CONN_MAXFD为65536安全吗，accept返回的socket是不是都在这个范围之内。