linux搭建分布式文件系统FastDFS并整合spring boot

青旅半醒 2022-06-01 09:06 207阅读 0赞

# linux搭建分布式文件系统FastDFS并整合spring boot #

## 目录 ##

*  FastDFS是什么
 *  FastDFS安装
 *  spring boot集成FastDFS上传下载文件
 *  FastDFS配置nginx
 *  问题&注意事项

## FastDFS是什么： ##

是由C语言实现的开源轻量级分布式文件系统，实现了对文件管理，同步，访问，下载等功能，解决了大容量存储和负载均衡。其包含两个角色：Tracker（跟踪器）和Storage（存储节点）

*  Tracker：资源调度，负载均衡，管理所有的group和storage，每一个storage启动就会链接Tracker
 *  Storage：存储节点，提供存储和备份服务，以group为单位，每组可以有多台Storage，数据互相备份。
    
        官网架构图：
    
    ![架构图][SouthEast]

## FastDFS安装 ##

多台服务器都需要安装FastDFS，为了方便测试单机安装也可以，配置相差不多。本次环境centos，如果安装过程中出现问题翻到最后查看注意事项可能会找到解决方法。

*  下载安装libfastcommon：
    
        cd /opt/fastdfs
        
        wget https://github.com/happyfish100/libfastcommon/archive/V1.0.7.tar.gz
        
        tar -zxvf V1.0.7.tar.gz
        
        cd libfastcommon-1.0.7
        
        ./make.sh
        
        //安装：
        ./make.sh install
        
        //设置软连接:
        ln -s /usr/lib64/libfastcommon.so /usr/local/lib/libfastcommon.so
        ln -s /usr/lib64/libfastcommon.so /usr/lib/libfastcommon.so
        ln -s /usr/lib64/libfdfsclient.so /usr/local/lib/libfdfsclient.so
        ln -s /usr/lib64/libfdfsclient.so /usr/lib/libfdfsclient.so
 *  下载安装FastDFS，每台文件服务器都需要安装：
    
        wget https://github.com/happyfish100/fastdfs/archive/V5.05.tar.gz
        
        tar -zxvf V5.05.tar.gz
        
        cd fastdfs-5.05
        
        ./make.sh
        
        ./make.sh install
    
    安装成功后会在/etc/fdfs 目录下生成三个示例配置文件
 *  配置Tracker服务器
    
        cp tracker.conf.sample tracker.conf
        
        //创建文件夹留后用
        mkdir -p /data/fastdfs/tracker
        
        vim tracker.conf
    
    编辑tracker.conf配置文件，修改：
    
        # the base path to store data and log files
        base_path=/data/fastdfs/tracker     
        
        http.server_port=8055
    
    保存退出，启动tracker服务：
    
        fdfs_trackerd /etc/fdfs/tracker.conf start
    
    启动成功，查看监听服务：
    
        netstat -unltp|grep fdfs
    
    可以看到tracker默认端口为22122，并且在/data/fastdfs/tracker文件夹下新增数据和日志存储文件。
 *  配置Storage服务器：
    
        cd /etc/fdfs
        
        cp storage.conf.sample storage.conf
        
        mkdir -p /data/fastdfs/storage
        
        vim storage.conf
    
    修改storage.conf配置文件：
    
        # the base path to store data and log files
        base_path=/data/fastdfs/storage
        
        # store_path#, based 0, if store_path0 not exists, it's value is base_path
        # the paths must be exist
        store_path0=/data/fastdfs/storage
        
        # tracker_server can ocur more than once, and tracker_server format is
        #  "host:port", host can be hostname or ip address
        #配置tracker跟踪器ip端口
        tracker_server=192.168.5.26:22122
    
    保存退出，并启动Storage：
    
        fdfs_storaged /etc/fdfs/storage.conf start
    
    如果启动过慢或卡死，查看日志文件：
    
        /data/fastdfs/storage/logs
    
    查看端口监听，发现Storage默认端口号为23000
    
        netstat -unltp|grep fdfs
    
    所有节点都安装成功后可以在任意一个节点查看集群的状态：
    
        /usr/bin/fdfs_monitor /etc/fdfs/storage.conf

## spring boot集成FastDFS上传下载文件： ##

*  pom.xml引入，参考自纯洁的微笑：
    
    ` <dependency> <groupId>net.oschina.zcx7878</groupId> <artifactId>fastdfs-client-java</artifactId> <version>1.27.0.0</version> </dependency>`
 *  添加FastDFS配置文件到classpath：fdfs\_client.conf
    
        connect_timeout = 60
        network_timeout = 60
        charset = UTF-8
        http.tracker_http_port = 8055
        http.anti_steal_token = no
        http.secret_key = 123456
        
        tracker_server = 192.168.5.26:22122
    
    其中http.tracker\_http\_port = 8055 与tracker.conf中的http.server\_port一致，并且不能与其他端口程序冲突。tracker\_server修改相应ip，如果有多台，则可以写多个trackrt\_server配置项。
 *  封装文件基础信息类：
    
        @Data
        public class FastDFSFile {
        
            private String name;
        
            private byte[] content;
        
            private String ext;
        
            private String md5;
        
            private String author;
        }
 *  封装client操作类，用于上传下载删除等操作：
    
        import java.io.ByteArrayInputStream;
        import java.io.IOException;
        import java.io.InputStream;
        
        import org.csource.common.NameValuePair;
        import org.csource.fastdfs.ClientGlobal;
        import org.csource.fastdfs.FileInfo;
        import org.csource.fastdfs.ServerInfo;
        import org.csource.fastdfs.StorageClient;
        import org.csource.fastdfs.StorageServer;
        import org.csource.fastdfs.TrackerClient;
        import org.csource.fastdfs.TrackerServer;
        import org.slf4j.LoggerFactory;
        import org.springframework.core.io.ClassPathResource;
        
        public class FastDFSClient {
            private static org.slf4j.Logger logger = LoggerFactory.getLogger(FastDFSClient.class);
            private static TrackerClient trackerClient;
            private static TrackerServer trackerServer;
            private static StorageClient storageClient;
            private static StorageServer storageServer;
        
            static {
                try {
                    String filePath = new ClassPathResource("fdfs_client.conf").getFile().getAbsolutePath();;
                    ClientGlobal.init(filePath);
                    trackerClient = new TrackerClient();
                    trackerServer = trackerClient.getConnection();
                    storageServer = trackerClient.getStoreStorage(trackerServer);
                } catch (Exception e) {
                    logger.error("FastDFS Client Init Fail!",e);
                }
            }
        
            public static String[] upload(FastDFSFile file) {
                logger.info("File Name: " + file.getName() + "File Length:" + file.getContent().length);
        
                NameValuePair[] meta_list = new NameValuePair[1];
                meta_list[0] = new NameValuePair("author", file.getAuthor());
        
                long startTime = System.currentTimeMillis();
                String[] uploadResults = null;
                try {
                    storageClient = new StorageClient(trackerServer, storageServer);
                    uploadResults = storageClient.upload_file(file.getContent(), file.getExt(), meta_list);
                } catch (IOException e) {
                    logger.error("IO Exception when uploadind the file:" + file.getName(), e);
                } catch (Exception e) {
                    logger.error("Non IO Exception when uploadind the file:" + file.getName(), e);
                }
                logger.info("upload_file time used:" + (System.currentTimeMillis() - startTime) + " ms");
        
                if (uploadResults == null) {
                    logger.error("upload file fail, error code:" + storageClient.getErrorCode());
                }
                String groupName = uploadResults[0];
                String remoteFileName = uploadResults[1];
        
                logger.info("upload file successfully!!!" + "group_name:" + groupName + ", remoteFileName:" + " " + remoteFileName);
                return uploadResults;
            }
        
            public static FileInfo getFile(String groupName, String remoteFileName) {
                try {
                    storageClient = new StorageClient(trackerServer, storageServer);
                    return storageClient.get_file_info(groupName, remoteFileName);
                } catch (IOException e) {
                    logger.error("IO Exception: Get File from Fast DFS failed", e);
                } catch (Exception e) {
                    logger.error("Non IO Exception: Get File from Fast DFS failed", e);
                }
                return null;
            }
        
            public static InputStream downFile(String groupName, String remoteFileName) {
                try {
                    storageClient = new StorageClient(trackerServer, storageServer);
                    byte[] fileByte = storageClient.download_file(groupName, remoteFileName);
                    InputStream ins = new ByteArrayInputStream(fileByte);
                    return ins;
                } catch (IOException e) {
                    logger.error("IO Exception: Get File from Fast DFS failed", e);
                } catch (Exception e) {
                    logger.error("Non IO Exception: Get File from Fast DFS failed", e);
                }
                return null;
            }
        
            public static void deleteFile(String groupName, String remoteFileName)
                    throws Exception {
                storageClient = new StorageClient(trackerServer, storageServer);
                int i = storageClient.delete_file(groupName, remoteFileName);
                logger.info("delete file successfully!!!" + i);
            }
        
            public static StorageServer[] getStoreStorages(String groupName)
                    throws IOException {
                return trackerClient.getStoreStorages(trackerServer, groupName);
            }
        
            public static ServerInfo[] getFetchStorages(String groupName,
                                                        String remoteFileName) throws IOException {
                return trackerClient.getFetchStorages(trackerServer, groupName, remoteFileName);
            }
        
            public static String getTrackerUrl() {
                return "http://"+trackerServer.getInetSocketAddress().getHostString()+":"+ClientGlobal.getG_tracker_http_port()+"/";
            }
        }
 *  编写控制器提供REST 服务器：
    
        @PostMapping("/upload") 
        public String singleFileUpload(@RequestParam("file") MultipartFile file,
                                       RedirectAttributes redirectAttributes) {
            if (file.isEmpty()) {
                redirectAttributes.addFlashAttribute("message", "Please select a file to upload");
                return "redirect:uploadStatus";
            }
            try {
                // Get the file and save it somewhere
                String path=saveFile(file);
                redirectAttributes.addFlashAttribute("message",
                        "You successfully uploaded '" + file.getOriginalFilename() + "'");
                redirectAttributes.addFlashAttribute("path",
                        "file path url '" + path + "'");
            } catch (Exception e) {
                logger.error("upload file failed",e);
            }
            return "redirect:/uploadStatus";
        }
        
          public String saveFile(MultipartFile multipartFile) throws IOException {
            String[] fileAbsolutePath={};
            String fileName=multipartFile.getOriginalFilename();
            String ext = fileName.substring(fileName.lastIndexOf(".") + 1);
            byte[] file_buff = null;
            InputStream inputStream=multipartFile.getInputStream();
            if(inputStream!=null){
                int len1 = inputStream.available();
                file_buff = new byte[len1];
                inputStream.read(file_buff);
            }
            inputStream.close();
            FastDFSFile file = new FastDFSFile(fileName, file_buff, ext);
            try {
                fileAbsolutePath = FastDFSClient.upload(file);  //upload to fastdfs
            } catch (Exception e) {
                logger.error("upload file Exception!",e);
            }
            if (fileAbsolutePath==null) {
                logger.error("upload file failed,please upload again!");
            }
            String path=FastDFSClient.getTrackerUrl()+fileAbsolutePath[0]+ "/"+fileAbsolutePath[1];
            return path;
        }
 *  上传测试，如果操作成功，上传接口将返回文件上传到FastDFS集群中的位置URL信息，类似：[http://192.168.5.26:8888/group1/M00/00/00/rBF80FppaNaASaqvAAvWFkcZHjA372.jpg][http_192.168.5.26_8888_group1_M00_00_00_rBF80FppaNaASaqvAAvWFkcZHjA372.jpg]，但该链接还并不能直接访问，需要继续配置FastDFS与nginx集成进行提供外部访问，继续。
 *  示例项目位置：[github][]

## FastDFS配置nginx ##

*  nginx 相关操作：[链接][Link 1]
 *  安装fastdfs-nginx-module,所有Storage节点都需要安装
    
        cd /opt/fastdfs
        
        wget http://jaist.dl.sourceforge.NET/project/fastdfs/FastDFS%20Nginx%20Module%20Source%20Code/fastdfs-nginx-module_v1.16.tar.gz
        
        tar -zxvf fastdfs-nginx-module_v1.16.tar.gz
        
        cd /opt/fastdfs/fastdfs-nginx-module/src
        
        vim config
    
    修改配置，否则nginx编译报错
    
        CORE_INCS="$CORE_INCS /usr/local/include/fastdfs /usr/local/include/fastcommon/"
        
        改为如下：
        
        CORE_INCS="$CORE_INCS /usr/include/fastdfs /usr/include/fastcommon/"
    
    复制fastdfs-nginx-module 的配置文件到/etc/fdfs并修改
    
        cd /opt/fastdfs-nginx-module/src
        
        cp mod_fastdfs.conf /etc/fdfs
        
        cd /etc/fdfs
        
        vim mod_fastdfs.conf
    
    修改如下：
    
        tracker_server=192.168.7.73:22122 # tracker服务IP和端口
        url_have_group_name=true # 访问链接前缀加上组名
        store_path0=/data/fastdfs/storage # 文件存储路径
 *  重新编译安装nginx，相关操作链接在上面。
    
        cd /usr/local/nginx
        
        ./configure \
        --prefix=/usr/local/nginx \
        --pid-path=/var/local/nginx/nginx.pid \
        --lock-path=/var/lock/nginx/nginx.lock \
        --error-log-path=/var/log/nginx/error.log \
        --http-log-path=/var/log/nginx/access.log \
        --with-http_gzip_static_module \
        --http-client-body-temp-path=/var/temp/nginx/client \
        --http-proxy-temp-path=/var/temp/nginx/proxy \
        --http-fastcgi-temp-path=/var/temp/nginx/fastcgi \
        --http-uwsgi-temp-path=/var/temp/nginx/uwsgi \
        --http-scgi-temp-path=/var/temp/nginx/scgi \
        --add-module=/opt/fastdfs/fastdfs-nginx-module/src
        
        make && make install
        
        //重启,如何添加到service查看上面链接
        service nginx restart
 *  配置文件：
    
    复制FastDFS部分配置文件到/etc/fdfs
    
        cd /opt/fastdfs/fastdfs-5.05/conf
        
        cp http.conf mime.types /etc/fdfs
    
    修改nginx配置文件：
    
        vim /usr/local/nginx/conf/nginx.conf
    
    修改端口与/etc/fdfs/storage.conf 中的http.server\_port=8888 相对应
    
        server {
            listen 8888;  
            server_name localhost;
        
           location / {
              root html;
              index index.html index.htm;
           }
           location ~/group([0-9])/M00 {
              alias /data/fastdfs/storage/;
              ngx_fastdfs_module;  
            }
            location /group1/M00 {
                root /data/fastdfs/storage/;
                ngx_fastdfs_module;
            }
          }
    
    重启nginx
    
        service nginx restart
    
    查看nginx状态：
    
        netstat -unltp|grep nginx
    
    可以看到nginx监听了8888端口。此时可以访问[http://192.168.5.26:8888/group1/M00/00/00/rBF80FppaNaASaqvAAvWFkcZHjA372.jpg][http_192.168.5.26_8888_group1_M00_00_00_rBF80FppaNaASaqvAAvWFkcZHjA372.jpg]查看到文件
 *  Tracker配置nginx
    
        cd /usr/local/nginx
        
        ./configure \
        --prefix=/usr/local/nginx 
        
        make && make install
    
    设置group负载均衡，设置nginx端口为8000
    
        #设置 group1 的服务器
        upstream fdfs_group1 {
             server 192.168.5.26:8888 weight=1 max_fails=2 fail_timeout=30s;
             server 192.168.5.27:8888 weight=1 max_fails=2 fail_timeout=30s;
        }
        
        server {
            listen       8000;
            server_name  localhost;
        
            #charset koi8-r;
        
            #access_log  logs/host.access.log  main;
        
            location / {
                root   html;
                index  index.html index.htm;
            }
        
            #设置 group 的负载均衡参数
            location /group1/M00 {
                proxy_next_upstream http_502 http_504 error timeout invalid_header;
                proxy_pass http://fdfs_group1;
                expires 30d;
            }
        }
    
    此时通过Tracker和直接从Storage直接访问都可以访问到文件。
    
        http://tracker_server:8000/group1/M00/00/00/rBF80FppaNaASaqvAAvWFkcZHjA372.jpg
        http://storage_server:8888/group1/M00/00/00/rBF80FppaNaASaqvAAvWFkcZHjA372.jpg

## 问题&注意事项 ##

*  Tracker启动成功，Storage启动失败：
    
    检查防火墙配置：
    
        vim /etc/sysconfig/iptables
    
    有关iptables详细配置查看：[iptables快速配置][iptables]
    
    使用阿里云或其他云服务器注意平台安全组，以阿里云或腾讯云为例，进入ECS服务器–>查看安全组–>添加安全组规则，暴露出相应的端口：22122，23000，8050，8888等。
 *  使用spring boot上传文件失败，发现tracker默认http端口为8080，可能与tomcat冲突，修改tracker端口，并重新配置防火墙和安全组。
 *  nginx安装失败，可能是缺少依赖，运行：
    
        yum -y install gcc
        
        yum -y install pcre pcre-devel
        
        yum -y install zlib zlib-devel
        
        yum -y install openssl openssl-devel
    
    安装相应依赖。
 *  常用命令：
    
     *  netstat -anp|grep 80
     *  fdfs\_trackerd /etc/fdfs/tracker.conf start
     *  fdfs\_storaged /etc/fdfs/storage.conf start
     *  service nginx start
     *  netstat -unltp|grep fdfs
     *  tar zxvf nginx-1.12.1.tar.gz
     *  vim /etc/sysconfig/iptables

[SouthEast]: /images/20220601/b48d7c5eb402434696315e18944f3c69.png
[http_192.168.5.26_8888_group1_M00_00_00_rBF80FppaNaASaqvAAvWFkcZHjA372.jpg]: http://192.168.5.26:8888/group1/M00/00/00/rBF80FppaNaASaqvAAvWFkcZHjA372.jpg
[github]: https://github.com/Xchunguang/spring-cloud-framework/tree/master/cloud-mysoft-fastdfs
[Link 1]: http://blog.csdn.net/xcg132566/article/details/79161756
[iptables]: http://blog.csdn.net/xcg132566/article/details/78797638