Linux 搭建分布式文件存储FastDFS

缺乏、安全感 2023-10-05 13:38 33阅读 0赞

#### 分布式文件存储FastDFS ####

*  软件下载
 *  安装FastDFS
 *   *  1. 上传文件到
     *  2. 安装依赖
     *  安装FastDFS
     *  启动tracker
     *  启动storage
     *  安装FastDFS的Nginx模块
     *  安装Nginx
     *  关闭防火墙
 *  spring Cloud 整合文件存储微服务
 *   *  添加依赖
     *  FastDFS配置
     *  application.yml配置
     *  文件上传

## 软件下载 ##

链接：[https://pan.baidu.com/s/1Cle7VaKBUce6K3xxBFcxFg][https_pan.baidu.com_s_1Cle7VaKBUce6K3xxBFcxFg]  
提取码：737r

## 安装FastDFS ##

### 1. 上传文件到 ###

将资料中的文件上传到linux下的/usr/local/fdfs目录  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NTYyNjI4OA_size_16_color_FFFFFF_t_70_pic_center]

### 2. 安装依赖 ###

安装GCC依赖  
yum -y install gcc  
安装unzip工具  
yum install -y unzip zip  
安装libevent  
yum -y install libevent  
安装Nginx所需依赖  
yum -y install pcre pcre-devel zlib zlib-devel openssl openssl-devel

安装libfastcommon-master

1.  解压 libfastcommon-master.zip
2.  cd /usr/local/fdfs
3.  unzip libfastcommon-master.zip
4.  编译安装
5.  cd libfastcommon-master
6.  ./make.sh
7.  ./make.sh install

### 安装FastDFS ###

解压

1.  cd /usr/local/fdfs/
2.  tar -xvf FastDFS\_v5.08.tar.gz

进入目录  
cd FastDFS

编译并安装

1.  ./make.sh
2.  ./make.sh install

校验安装结果  
查看FastDFS启动脚本

1.  cd /etc/init.d/
2.  ll /etc/init.d/ | grep fdfs  
    ![在这里插入图片描述][20201114091955487.png_pic_center]  
    **fdfs\_trackerd** 是tracker启动脚本  
    **fdfs\_storaged** 是storage启动脚本

查看配置文件模板  
cd /etc/fdfs  
![在这里插入图片描述][20201114092131325.png_pic_center]  
**tarcker.conf.sample** 是tracker的配置文件模板  
**storage.conf.sample** 是storage的配置文件模板  
**client.conf.sample** 是客户端的配置文件模板

### 启动tracker ###

编辑tracker配置

1.  cd /etc/fdfs/
2.  cp tracker.conf.sample tracker.conf
3.  vim tracker.conf

修改base\_path配置  
4. tracker的数据和日志存放目录  
5. base\_path=/usr/local/fdfs/tracker

创建目录

1.  mkdir -p /usr/local/fdfs/tracker

启动tracker

1.  sh /etc/init.d/fdfs\_trackerd start
2.  或
3.  service fdfs\_trackerd start

停用tracker

1.  service fdfs\_trackerd stop  
    设置开机启动
2.  chkconfig fdfs\_trackerd on

### 启动storage ###

编辑storage配置

1.  cd /etc/fdfs/
2.  cp storage.conf.sample storage.conf
3.  vim storage.conf

修改base\_path配置

1.  storage的数据和日志存放目录
2.  base\_path=/usr/local/fdfs/storage
3.  storage的上传文件存放路径
4.  store\_path0=/usr/local/fdfs/storage
5.  tracker的地址（更换成自己机器的IP）
6.  tracker\_server=192.168.xx.xx:22122

创建目录

1.  mkdir -p /usr/local/fdfs/storage

启动storage

1.  sh /etc/init.d/fdfs\_storaged start
2.  或
3.  service fdfs\_storaged start

停用storage

1.  service fdfs\_storaged stop

设置开机启动

1.  chkconfig fdfs\_storaged on

查看进程

1.  ps -ef | grep fdfs  
    ![在这里插入图片描述][20201114095338242.png_pic_center]

### 安装FastDFS的Nginx模块 ###

解压

1.  cd /usr/local/fdfs/
2.  tar -xvf fastdfs-nginx-module\_v1.16.tar.gz

配置config文件  
进入配置目录

1.  cd fastdfs-nginx-module/src/

修改配置

1.  vim config

执行下面命令

1.  :%s+/usr/local/+/usr/+g

配置mod\_fastdfs.conf

1.  cd /usr/local/fdfs/fastdfs-nginx-module/src/
2.  cp mod\_fastdfs.conf /etc/fdfs/
3.  vim /etc/fdfs/mod\_fastdfs.conf

修改为以下配置  
connect\_timeout=10 \# 客户端访问文件连接超时时长（单位：秒）  
tracker\_server=192.168.1.120:22122 \# tracker服务IP和端口（更换成自己机器的IP）  
url\_have\_group\_name=true \# 访问链接前缀加上组名  
store\_path0=/usr/local/fdfs/storage \# 文件存储路径

复制 FastDFS的部分配置文件到/etc/fdfs目录

1.  cd /usr/local/fdfs/FastDFS/conf/
2.  cp http.conf mime.types /etc/fdfs/

### 安装Nginx ###

解压

1.  cd /usr/local/fdfs/
2.  tar -xvf nginx-1.16.1.tar.gz

配置  
添加FastDFS模块

1.  cd nginx-1.16.1/
2.  ./configure --add-module=/usr/local/fdfs/fastdfs-nginx-module/src

重新编译

1.  make && sudo make install

配置nginx整合fastdfs-module模块  
修改nginx配置文件

1.  vim /usr/local/nginx/conf/nginx.conf

将文件中，原来的server 80\{ …\} 部分代码替换为如下代码：

server {
    	listen 80;
    	server_name localhost;
    	# 监听域名中带有group的，交给FastDFS模块处理
    	location ~/group([0-9])/ {
    		ngx_fastdfs_module;
    	}
    	location / {
    		root html;
    		index index.html index.htm;
    	}
    	error_page 500 502 503 504 /50x.html;
    	location = /50x.html {
    		root html;
    	}
    }

启动nginx

1.  cd /usr/local/nginx/sbin
2.  ./nginx \# 启动
3.  ./nginx -s stop \# 停止
4.  ./nginx -s reload \# 重新加载配置

### 关闭防火墙 ###

查看防火墙的状态的命令为：sudo systemctl status firewalld。  
打开防火墙的方式有两种，一种是打开后重启会恢复回原来的状态，命令为：sudo systemctl start firewalld；  
另一种是打开后重启不会恢复到原来的状态，命令为：sudo systemctl enable firewalld，这种方式输入命令后要重启系统才会生效。  
关闭防火墙的方式也有两种，和打开相对应，命令分别为  
**sudo systemctl stop firewalld**  
**sudo systemctl disable firewalld**

## spring Cloud 整合文件存储微服务 ##

### 添加依赖 ###

<dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>net.oschina.zcx7878</groupId>
            <artifactId>fastdfs-client-java</artifactId>
            <version>1.27.0.0</version>
        </dependency>
    </dependencies>

### FastDFS配置 ###

在resources文件夹下创建fasfDFS的配置文件fdfs\_client.conf

connect_timeout=60
    network_timeout=60
    charset=UTF-8
    http.tracker_http_port=80
    # 配置成自己的IP地址
    tracker_server=192.168.xx.xx:22122

connect\_timeout：连接超时时间，单位为秒。  
network\_timeout：通信超时时间，单位为秒。发送或接收数据时。假设在超时时间后还不能发送或接收数据，则本次网络通信失败  
charset： 字符集  
http.tracker\_http\_port ：.tracker的http端口  
tracker\_server： tracker服务器IP和端口设置

### application.yml配置 ###

在resources文件夹下创建application.yml

spring:
      servlet:
        multipart:
          max-file-size: 10MB # 单个文件大小
          max-request-size: 10MB # 总上传数据大小
      application:
        name: file-server
    server:
      port: 7008

max-file-size是单个文件大小，max-request-size是设置总上传的数据大小

### 文件上传 ###

**件信息封装**  
文件上传一般都有文件的名字、文件的内容、文件的扩展名、文件的md5值、文件的作者等相关属性，我们可以创建一个对象封装这些属性，代码如下：  
创建com.ddz.entity.FastDFSFile代码如下：

package com.ddz.entity;
    
    import java.io.Serializable;
    import java.util.Arrays;
    
    public class FastDFSFile implements Serializable {
        
        //文件名字
        private String name;
        //文件内容
        private byte[] content;
        //文件扩展名
        private String ext;
        //文件MD5摘要值
        private String md5;
        //文件创建作者
        private String author;
    
        public String getName() {
        
            return name;
        }
    
        public void setName(String name) {
        
            this.name = name;
        }
    
        public byte[] getContent() {
        
            return content;
        }
    
        public void setContent(byte[] content) {
        
            this.content = content;
        }
    
        public String getExt() {
        
            return ext;
        }
    
        public void setExt(String ext) {
        
            this.ext = ext;
        }
    
        public String getMd5() {
        
            return md5;
        }
    
        public void setMd5(String md5) {
        
            this.md5 = md5;
        }
    
        public String getAuthor() {
        
            return author;
        }
    
        public void setAuthor(String author) {
        
            this.author = author;
        }
    
        @Override
        public String toString() {
        
            return "FastDFSFile{" +
                    "name='" + name + '\'' +
                    ", content=" + Arrays.toString(content) +
                    ", ext='" + ext + '\'' +
                    ", md5='" + md5 + '\'' +
                    ", author='" + author + '\'' +
                    '}';
        }
    }

(可选)测试文件相关操作

import com.zb.pojo.FastDFSFile;
    import org.csource.common.MyException;
    import org.csource.common.NameValuePair;
    import org.csource.fastdfs.*;
    import org.springframework.core.io.ClassPathResource;
    import org.springframework.stereotype.Component;
    
    import java.io.File;
    import java.io.FileOutputStream;
    import java.io.IOException;
    
    @Component
    public class FastDFSUtil {
        
    
        static {
        
            String path = new ClassPathResource("fdfs_client.conf").getPath();
            try {
        
                ClientGlobal.init(path);
            } catch (IOException e) {
        
                e.printStackTrace();
            } catch (MyException e) {
        
                e.printStackTrace();
            }
        }
    
        public static String [] upload(FastDFSFile fastDFSFile)throws Exception{
        
            NameValuePair [] meta_list = new NameValuePair[1];
            meta_list[0]=new NameValuePair(fastDFSFile.getAuthor());
            /***
             * 文件上传后的返回值
             * uploadResults[0]:文件上传所存储的组名，例如:group1
             * uploadResults[1]:文件存储路径,例如：M00/00/00/wKjThF0DBzaAP23MAAXz2mMp9oM26.jpeg
             */
            String [] uploadResults = null;
            TrackerClient trackerClient = new TrackerClient();
            TrackerServer connection = trackerClient.getConnection();
            StorageClient storageClient = new StorageClient(connection, null);
            uploadResults= storageClient.upload_file(fastDFSFile.getContent(),fastDFSFile.getExt(),meta_list);
            return uploadResults;
        }
    
    
    
        public void upload() throws Exception {
        
            TrackerClient trackerClient = new TrackerClient();
            TrackerServer connection = trackerClient.getConnection();
            StorageClient storageClient = new StorageClient(connection, null);
            String[] mp4s = storageClient.upload_file("E:\\14bd8e05-84e0-40e8-88ce-9712831698c8.mp4", "mp4", null);
            for (String mp4 : mp4s) {
        
                System.out.println(mp4);
            }
        }
    
        public void delete()throws Exception{
        
            //创建TrackerClient客户端对象
            TrackerClient trackerClient = new TrackerClient();
            //通过TrackerClient对象获取TrackerServer信息
            TrackerServer trackerServer = trackerClient.getConnection();
            //获取StorageClient对象
            StorageClient storageClient = new StorageClient(trackerServer, null);
            //执行文件上传
            int group1 = storageClient.delete_file("group1", "M00/00/00/wKgBbl-ag1aAQe-LADQRbSP7hcU218.mp4");
            System.out.println(group1);
        }
    
        /***
         * 获取Tracker服务地址
         */
        public static String getTrackerUrl(){
        
            try {
        
                //创建TrackerClient对象
                TrackerClient trackerClient = new TrackerClient();
                //通过TrackerClient获取TrackerServer对象
                TrackerServer trackerServer = trackerClient.getConnection();
                //获取Tracker地址
                return "http://"+trackerServer.getInetSocketAddress().getHostString()+":"+ClientGlobal.getG_tracker_http_port();
            } catch (IOException e) {
        
                e.printStackTrace();
            }
            return null;
        }
    
        public void download(String filename)throws Exception{
        
            TrackerClient trackerClient = new TrackerClient();
            TrackerServer trackerServer = trackerClient.getConnection();
            StorageClient storageClient = new StorageClient(trackerServer, null);
            byte[] bytes = storageClient.download_file("group1", filename);
            File file = new File("d://a.mp4");
            FileOutputStream fos = new FileOutputStream(file);
            fos.write(bytes);
            fos.close();
        }
    
        public void getFileInfo(String filename)throws Exception{
        
            TrackerClient trackerClient =new TrackerClient();
            TrackerServer trackerServer = trackerClient.getConnection();
            StorageClient storageClient = new StorageClient(trackerServer, null);
            FileInfo group1 = storageClient.get_file_info("group1", filename);
            System.out.println(group1);
        }
    }

创建一个FileController，在该控制器中实现文件上传操作，代码如下：

@RestController
    @CrossOrigin
    public class FileController {
        
        /***
         * 文件上传
         * @return
         */
        @PostMapping(value = "/upload")
        public String upload(@RequestParam("file") MultipartFile file) throws Exception {
        
            //封装一个FastDFSFile
            FastDFSFile fastDFSFile = new FastDFSFile(
                    file.getOriginalFilename(), //文件名字
                    file.getBytes(),            //文件字节数组
                    StringUtils.getFilenameExtension(file.getOriginalFilename()));//文件扩展名
    
            //文件上传
            String[] uploads = FastDFSUtil.upload(fastDFSFile);
            //组装文件上传地址
            return FastDFSUtil.getTrackerUrl()+"/"+uploads[0]+"/"+uploads[1];
        }
    }

Postman测试文件上传  
选择post请求方式，输入请求地址  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NTYyNjI4OA_size_16_color_FFFFFF_t_70_pic_center 1]

[https_pan.baidu.com_s_1Cle7VaKBUce6K3xxBFcxFg]: https://pan.baidu.com/s/1Cle7VaKBUce6K3xxBFcxFg
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NTYyNjI4OA_size_16_color_FFFFFF_t_70_pic_center]: https://img-blog.csdnimg.cn/20201114091312120.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NTYyNjI4OA==,size_16,color_FFFFFF,t_70#pic_center
[20201114091955487.png_pic_center]: https://img-blog.csdnimg.cn/20201114091955487.png#pic_center
[20201114092131325.png_pic_center]: https://img-blog.csdnimg.cn/20201114092131325.png#pic_center
[20201114095338242.png_pic_center]: https://img-blog.csdnimg.cn/20201114095338242.png#pic_center
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NTYyNjI4OA_size_16_color_FFFFFF_t_70_pic_center 1]: https://img-blog.csdnimg.cn/2020111410130257.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NTYyNjI4OA==,size_16,color_FFFFFF,t_70#pic_center