免费注册 查看新帖 |

Chinaunix

  平台 论坛 博客 文库
最近访问板块 发新帖
楼主: sun_wanx

[FastDFS] 【求助】java客户端并发访问报错。 [复制链接]

论坛徽章:
2
IT运维版块每日发帖之星
日期:2016-05-09 06:20:00IT运维版块每日发帖之星
日期:2016-08-11 06:20:00
发表于 2016-05-06 15:17 |显示全部楼层
环境:
服务器FastDFS版本5.0.7
java客户端版本1.24
window7 64位、ubuntu、centos

部署结构:
2台trackage,2台storage

问题:
并发访问单个或者多个文件,少数可以下载成功,大部分报如下多种错误:
1.java.io.IOException: Socket Closed
        at java.net.AbstractPlainSocketImpl.getOutputStream(AbstractPlainSocketImpl.java:429)
        at java.net.PlainSocketImpl.getOutputStream(PlainSocketImpl.java:232)
        at java.net.Socket$3.run(Socket.java:927)
        at java.net.Socket$3.run(Socket.java:925)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.Socket.getOutputStream(Socket.java:924)
        at org.csource.fastdfs.ProtoCommon.closeSocket(ProtoCommon.java:350)
        at org.csource.fastdfs.TrackerServer.close(TrackerServer.java:76)
        at org.csource.fastdfs.StorageClient.download_file(StorageClient.java:131
        at org.csource.fastdfs.StorageClient.download_file(StorageClient.java:1280)
=========================================================================
2.java.io.IOException: recv cmd: 64 is not correct, expect cmd: 100
        at org.csource.fastdfs.ProtoCommon.recvHeader(ProtoCommon.java:219) ~[fastdfs-client-5.0.4.jar:na]
        at org.csource.fastdfs.ProtoCommon.recvPackage(ProtoCommon.java:250) ~[fastdfs-client-5.0.4.jar:na]
        at org.csource.fastdfs.StorageClient.download_file(StorageClient.java:1301) ~[fastdfs-client-5.0.4.jar:na]
        at org.csource.fastdfs.StorageClient.download_file(StorageClient.java:1280) ~[fastdfs-client-5.0.4.jar:na]
        at com.github.mtdp.dfs.server.service.impl.FastDFSFileSystemServiceImpl.download(FastDFSFileSystemServiceImpl.java:106) ~[classes/:na]
        at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source) ~[na:na]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_79]
        at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_79]
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302) [spring-aop-4.2.1.RELEASE.jar:4.2.1.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) [spring-aop-4.2.1.RELEASE.jar:4.2.1.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) [spring-aop-4.2.1.RELEASE.jar:4.2.1.RELEASE]
        at com.alibaba.druid.support.spring.stat.DruidStatInterceptor.invoke(DruidStatInterceptor.java:72) [druid-1.0.18.jar:1.0.18]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.2.1.RELEASE.jar:4.2.1.RELEASE]
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207) [spring-aop-4.2.1.RELEASE.jar:4.2.1.RELEASE]
        at com.sun.proxy.$Proxy31.download(Unknown Source) [na:na]
==================================================================================================================

论坛徽章:
2
IT运维版块每日发帖之星
日期:2016-05-09 06:20:00IT运维版块每日发帖之星
日期:2016-08-11 06:20:00
发表于 2016-05-06 15:21 |显示全部楼层
补充服务端没有报错

论坛徽章:
2
IT运维版块每日发帖之星
日期:2016-06-03 06:20:00IT运维版块每日发帖之星
日期:2016-06-04 06:20:00
发表于 2016-05-11 08:17 |显示全部楼层
同问
centos7.1,使用5.05,1台tracker,2台storage,javaclient低并发上传,有几率报timeout错误

查询日志,在tracker的server中报
[2016-05-11 08:00:38] DEBUG - file: tracker_nio.c, line: 226, client ip: 172.27.12.86, recv error event: 25, close connection
[2016-05-11 08:00:38] DEBUG - file: tracker_nio.c, line: 226, client ip: 172.27.12.86, recv error event: 25, close connection
[2016-05-11 08:00:38] DEBUG - file: tracker_nio.c, line: 226, client ip: 172.27.12.86, recv error event: 25, close connection

看到以前的帖子说升级版本,不是很确定升级是否能解决问题

论坛徽章:
2
IT运维版块每日发帖之星
日期:2016-06-03 06:20:00IT运维版块每日发帖之星
日期:2016-06-04 06:20:00
发表于 2016-05-11 13:53 |显示全部楼层
yxs1112003 发表于 2016-05-11 08:17
同问
centos7.1,使用5.05,1台tracker,2台storage,javaclient低并发上传,有几率报timeout错误

问题解决了,升级到5.08,不在出现timeout异常了。

论坛徽章:
2
IT运维版块每日发帖之星
日期:2016-05-09 06:20:00IT运维版块每日发帖之星
日期:2016-08-11 06:20:00
发表于 2016-05-12 15:31 |显示全部楼层
回复 4# yxs1112003

感谢分享,
如果在上传/下载文件时添加syc锁,就不会报错,我的问题应该不是服务端问题。
我的代码是:
QQ图片20160512153113.png QQ图片20160512153036.png


   

论坛徽章:
2
IT运维版块每日发帖之星
日期:2016-06-03 06:20:00IT运维版块每日发帖之星
日期:2016-06-04 06:20:00
发表于 2016-05-31 19:08 |显示全部楼层
本帖最后由 yxs1112003 于 2016-05-31 19:11 编辑

这么传好像特别慢啊,我是上传的时候每次都获取一个新的storageClient,我是这样多线程上传的,不过这样并发数大一些就报错了。。。在网上看到说需要写线程池。。。

  1.     /**
  2.      * 每次执行上传下载动作时需要获取一个新的StorageClient,因为旧的StorageClient会被关闭
  3.      *
  4.      * @return 返回一个新的StorageClient1实例
  5.      */
  6.     StorageClient getStorageClient() {
  7.         return new StorageClient1();
  8.     }
  9.    / **
  10.      * 上传小文件
  11.      *
  12.      * @param nvp 文件数据元
  13.      * @return Meta类型的数据元,保存了保存组和上传路径
  14.      */
  15.     public Meta upload(NameValuePair[] nvp) {

  16.         String[] fileIds;
  17.         Meta meta = null;
  18.         StorageClient client;
  19.         try {
  20.             // nameValuePair是一个记录上传信息的类,可以存储name,value字符串
  21.             String local_filename = configManager.getLocal_filename();
  22.             client = getStorageClient();
  23.             fileIds = client.upload_file(local_filename, null, nvp);
  24.             meta = new Meta(fileIds[0], fileIds[1]);

  25.             // 将上传的文件路径记录到文件
  26.             recorder.recordUploadedFileList(meta);
  27.             logger.info("组名:" + fileIds[0] + " 路径: " + fileIds[1] + "\t");

  28.         } catch (Exception e) {
  29.             logger.error(null, e);
  30.         }
  31.         return meta;
  32.     }
复制代码

论坛徽章:
2
IT运维版块每日发帖之星
日期:2016-06-03 06:20:00IT运维版块每日发帖之星
日期:2016-06-04 06:20:00
发表于 2016-06-01 10:43 |显示全部楼层
哪位大大帮着看看可好?

论坛徽章:
2
IT运维版块每日发帖之星
日期:2016-06-03 06:20:00IT运维版块每日发帖之星
日期:2016-06-04 06:20:00
发表于 2016-06-01 11:42 |显示全部楼层
本帖最后由 yxs1112003 于 2016-06-01 11:47 编辑
yxs1112003 发表于 2016-05-31 19:08
这么传好像特别慢啊,我是上传的时候每次都获取一个新的storageClient,我是这样多线程上传的,不过这样并发 ...

报的错误,异常解释如下http:// frenchmay.iteye.com/ blog/ 253123

  1. 2016-06-0111:39:41,554[ERROR] pool-1-thread-49:com.common.impl.FDFSBaseDaoImpl.upload(87)       
  2. java.net.SocketException: Software caused connection abort: socket write error
  3.         at java.net.SocketOutputStream.socketWrite0(Native Method)
  4.         at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
  5.         at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
  6.         at org.csource.fastdfs.UploadStream.send(UploadStream.java:55)
  7.         at org.csource.fastdfs.StorageClient.do_upload_file(StorageClient.java:972)
  8.         at org.csource.fastdfs.StorageClient.upload_file(StorageClient.java:192)
  9.         at org.csource.fastdfs.StorageClient.upload_file(StorageClient.java:153)
  10.         at org.csource.fastdfs.StorageClient.upload_file(StorageClient.java:126)
  11.         at com.common.impl.FDFSBaseDaoImpl.upload(FDFSBaseDaoImpl.java:80)
  12.         at com.common.impl.FDFSBaseDaoImpl.upload(FDFSBaseDaoImpl.java:63)
  13.         at com.service.impl.FDFSServiceImpl.upload(FDFSServiceImpl.java:20)
  14.         at com.multi.UploadThread.run(UploadThread.java:21)
  15.         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  16.         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  17.         at java.lang.Thread.run(Thread.java:745)
  18. 2016-06-0111:39:41,554[ERROR] pool-1-thread-285:com.common.impl.FDFSBaseDaoImpl.upload(87)       
  19. java.io.IOException: recv package size -1 != 10
  20.         at org.csource.fastdfs.ProtoCommon.recvHeader(ProtoCommon.java:214)
  21.         at org.csource.fastdfs.ProtoCommon.recvPackage(ProtoCommon.java:250)
  22.         at org.csource.fastdfs.StorageClient.do_upload_file(StorageClient.java:976)
  23.         at org.csource.fastdfs.StorageClient.upload_file(StorageClient.java:192)
  24.         at org.csource.fastdfs.StorageClient.upload_file(StorageClient.java:153)
  25.         at org.csource.fastdfs.StorageClient.upload_file(StorageClient.java:126)
  26.         at com.common.impl.FDFSBaseDaoImpl.upload(FDFSBaseDaoImpl.java:80)
  27.         at com.common.impl.FDFSBaseDaoImpl.upload(FDFSBaseDaoImpl.java:63)
  28.         at com.service.impl.FDFSServiceImpl.upload(FDFSServiceImpl.java:20)
  29.         at com.multi.UploadThread.run(UploadThread.java:21)
  30.         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  31.         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  32.         at java.lang.Thread.run(Thread.java:745)
  33. 2016-06-0111:39:41,950[INFO] pool-1-thread-585:com.common.impl.FDFSBaseDaoImpl.upload(85)        Meta [groupName=group1, upPath=M00/00/86/rBsMQFdOVsqAG7KiAABFapaYBmE906.png]
  34. 2016-06-0111:39:41,554[ERROR] pool-1-thread-335:com.common.impl.FDFSBaseDaoImpl.upload(87)       
  35. java.net.SocketException: Software caused connection abort: socket write error
  36.         at java.net.SocketOutputStream.socketWrite0(Native Method)
  37.         at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
  38.         at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
  39.         at org.csource.fastdfs.UploadStream.send(UploadStream.java:55)
  40.         at org.csource.fastdfs.StorageClient.do_upload_file(StorageClient.java:972)
  41.         at org.csource.fastdfs.StorageClient.upload_file(StorageClient.java:192)
  42.         at org.csource.fastdfs.StorageClient.upload_file(StorageClient.java:153)
  43.         at org.csource.fastdfs.StorageClient.upload_file(StorageClient.java:126)
  44.         at com.common.impl.FDFSBaseDaoImpl.upload(FDFSBaseDaoImpl.java:80)
  45.         at com.common.impl.FDFSBaseDaoImpl.upload(FDFSBaseDaoImpl.java:63)
  46.         at com.service.impl.FDFSServiceImpl.upload(FDFSServiceImpl.java:20)
  47.         at com.multi.UploadThread.run(UploadThread.java:21)
  48.         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  49.         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  50.         at java.lang.Thread.run(Thread.java:745)
  51. 2016-06-0111:39:41,553[INFO] pool-1-thread-540:com.common.impl.FDFSBaseDaoImpl.upload(85)        Meta [groupName=group1, upPath=M00/00/86/rBsMQFdOVsmAdRdiAABFapaYBmE981.png]
  52. 2016-06-0111:39:41,553[ERROR] pool-1-thread-224:com.common.impl.FDFSBaseDaoImpl.upload(87)       
  53. java.net.SocketException: Software caused connection abort: socket write error
  54.         at java.net.SocketOutputStream.socketWrite0(Native Method)
  55.         at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
  56.         at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
  57.         at org.csource.fastdfs.UploadStream.send(UploadStream.java:55)
  58.         at org.csource.fastdfs.StorageClient.do_upload_file(StorageClient.java:972)
  59.         at org.csource.fastdfs.StorageClient.upload_file(StorageClient.java:192)
  60.         at org.csource.fastdfs.StorageClient.upload_file(StorageClient.java:153)
  61.         at org.csource.fastdfs.StorageClient.upload_file(StorageClient.java:126)
  62.         at com.common.impl.FDFSBaseDaoImpl.upload(FDFSBaseDaoImpl.java:80)
  63.         at com.common.impl.FDFSBaseDaoImpl.upload(FDFSBaseDaoImpl.java:63)
  64.         at com.service.impl.FDFSServiceImpl.upload(FDFSServiceImpl.java:20)
  65.         at com.multi.UploadThread.run(UploadThread.java:21)
  66.         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  67.         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  68.         at java.lang.Thread.run(Thread.java:745)
  69. 2016-06-0111:39:41,553[INFO] pool-1-thread-550:com.common.impl.FDFSBaseDaoImpl.upload(85)        Meta [groupName=group1, upPath=M00/00/86/rBsMQFdOVsmAbY_SAABFapaYBmE342.png]
  70. 2016-06-0111:39:41,552[INFO] pool-1-thread-542:com.common.impl.FDFSBaseDaoImpl.upload(85)        Meta [groupName=group1, upPath=M00/00/86/rBsMQFdOVsmAaFklAABFapaYBmE183.png]
  71. 2016-06-0111:39:41,552[ERROR] pool-1-thread-346:com.common.impl.FDFSBaseDaoImpl.upload(87)       
  72. java.io.IOException: recv package size -1 != 10
复制代码

论坛徽章:
2
IT运维版块每日发帖之星
日期:2016-05-09 06:20:00IT运维版块每日发帖之星
日期:2016-08-11 06:20:00
发表于 2016-06-12 15:01 |显示全部楼层
上传还行,不是很慢。回复 6# yxs1112003


   

论坛徽章:
0
发表于 2016-06-24 14:52 |显示全部楼层
客户端上传时需要添加一个连接池,本人测试过了,可以解决多线程上传导致的问题。有需要的可以发邮件给我841439685@qq.com
您需要登录后才可以回帖 登录 | 注册

本版积分规则 发表回复

  

北京盛拓优讯信息技术有限公司. 版权所有 京ICP备16024965号-6 北京市公安局海淀分局网监中心备案编号:11010802020122 niuxiaotong@pcpop.com 17352615567
未成年举报专区
中国互联网协会会员  联系我们:huangweiwei@itpub.net
感谢所有关心和支持过ChinaUnix的朋友们 转载本站内容请注明原作者名及出处

清除 Cookies - ChinaUnix - Archiver - WAP - TOP