【图像分类】实战——AlexNet实现图像分类（pytroch）

桃扇骨 2022-10-05 10:52 266阅读 0赞

# 摘要 #

AlexNet是由Alex Krizhevsky 提出的首个应用于图像分类的深层卷积神经网络，该网络在2012年ILSVRC（ImageNet Large Scale Visual Recognition Competition）图像分类竞赛中以15.3%的top-5测试错误率赢得第一名。也是在那年之后，更多的更深的神经网络被提出，比如优秀的vgg,GoogLeNet。 这对于传统的机器学习分类算法而言，已经相当的出色。网络结构如下：

![2071a471613d86fd23a878d085c4f81a.png][]

对于AlexNet的详细介绍可以参考我的博文：[【图像分类】 一文读懂AlexNet\_AI浩-CSDN博客][_AlexNet_AI_-CSDN]

下面讲讲如何使用AlexNet实现猫狗分类，由于本文使用的Loss函数是CrossEntropyLoss，所以只需更改类别的个数就可以实现多分类。

# 新建项目 #

新建一个图像分类的项目，data里面放数据集，dataset文件夹中自定义数据的读取方法，这次我不采用默认的读取方式，太简单没啥意思。然后再新建train.py和test.py

在项目的根目录新建train.py，然后在里面写训练代码。

# 导入所需要的库 #

# 设置全局参数 #

设置BatchSize、学习率和epochs，判断是否有cuda环境，如果没有设置为cpu。

# 设置全局参数
    modellr = 1e-4
    BATCH_SIZE = 64
    EPOCHS = 20
    DEVICE = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

# 图像预处理 #

在做图像与处理时，train数据集的transform和验证集的transform分开做，train的图像处理出了resize和归一化之外，还可以设置图像的增强，比如旋转、随机擦除等一系列的操作，验证集则不需要做图像增强，另外不要盲目的做增强，不合理的增强手段很可能会带来负作用，甚至出现Loss不收敛的情况。

# 数据预处理
     
    transform = transforms.Compose([
        transforms.Resize((224, 224)),
        transforms.ToTensor(),
        transforms.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5])
     
    ])
    transform_test = transforms.Compose([
        transforms.Resize((224, 224)),
        transforms.ToTensor(),
        transforms.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5])
    ])

# 读取数据 #

数据集地址：链接：https://pan.baidu.com/s/1ZM8vDWEzgscJMnBrZfvQGw 提取码：48c3   
将其下载后解压放到data文件夹中。数据的目录如下图：

![20210608174508334.png][]

然后我们在dataset文件夹下面新建 \_\_init\_\_.py和dataset.py，在dataset.py文件夹写入下面的代码：

# coding:utf8
    import os
    from PIL import Image
    from torch.utils import data
    from torchvision import transforms as T
    from sklearn.model_selection import train_test_split
     
    class DogCat(data.Dataset):
     
        def __init__(self, root, transforms=None, train=True, test=False):
            """
            主要目标： 获取所有图片的地址，并根据训练，验证，测试划分数据
            """
            self.test = test
            self.transforms = transforms
            imgs = [os.path.join(root, img) for img in os.listdir(root)]
            if self.test:
                imgs = sorted(imgs, key=lambda x: int(x.split('.')[-2].split('/')[-1]))
            else:
                imgs = sorted(imgs, key=lambda x: int(x.split('.')[-2]))
     
            if self.test:
                self.imgs = imgs
            else:
                trainval_files, val_files = train_test_split(imgs, test_size=0.3, random_state=42)
                if train:
                    self.imgs = trainval_files
                else:
                    self.imgs = val_files
     
        def __getitem__(self, index):
            """
            一次返回一张图片的数据
            """
            img_path = self.imgs[index]
            if self.test:
                label =-1
            else:
                label = 1 if 'dog' in img_path.split('/')[-1] else 0
            data = Image.open(img_path)
            data = self.transforms(data)
            return data, label
     
        def __len__(self):
            return len(self.imgs)

然后我们在train.py调用DogCat读取数据

dataset_train = DogCat('data/train', transforms=transform, train=True)
    dataset_test = DogCat("data/train", transforms=transform_test, train=False)
    # 读取数据
    print(dataset_train.imgs)
     
    # 导入数据
    train_loader = torch.utils.data.DataLoader(dataset_train, batch_size=BATCH_SIZE, shuffle=True)
    test_loader = torch.utils.data.DataLoader(dataset_test, batch_size=BATCH_SIZE, shuffle=False)

# 设置模型 #

使用CrossEntropyLoss作为loss，模型采用alexnet，选用预训练模型。更改全连接层，将最后一层类别设置为2，然后将模型放到DEVICE。优化器选用Adam。

# 实例化模型并且移动到GPU
    criterion = nn.CrossEntropyLoss()
    model_ft = alexnet(pretrained=True)
    model_ft.classifier = nn.Sequential(
                nn.Dropout(),
                nn.Linear(256 * 6 * 6, 4096),
                nn.ReLU(inplace=True),
                nn.Dropout(),
                nn.Linear(4096, 4096),
                nn.ReLU(inplace=True),
                nn.Linear(4096, 2),
            )
    model_ft.to(DEVICE)
    # 选择简单暴力的Adam优化器，学习率调低
    optimizer = optim.Adam(model_ft.parameters(), lr=modellr)
    
    
    def adjust_learning_rate(optimizer, epoch):
        """Sets the learning rate to the initial LR decayed by 10 every 30 epochs"""
        modellrnew = modellr * (0.1 ** (epoch // 50))
        print("lr:", modellrnew)
        for param_group in optimizer.param_groups:
            param_group['lr'] = modellrnew

# 设置训练和验证 #

# 定义训练过程
    
    def train(model, device, train_loader, optimizer, epoch):
        model.train()
        sum_loss = 0
        total_num = len(train_loader.dataset)
        print(total_num, len(train_loader))
        for batch_idx, (data, target) in enumerate(train_loader):
            data, target = Variable(data).to(device), Variable(target).to(device)
            output = model(data)
            loss = criterion(output, target)
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
            print_loss = loss.data.item()
            sum_loss += print_loss
            if (batch_idx + 1) % 50 == 0:
                print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                    epoch, (batch_idx + 1) * len(data), len(train_loader.dataset),
                           100. * (batch_idx + 1) / len(train_loader), loss.item()))
        ave_loss = sum_loss / len(train_loader)
        print('epoch:{},loss:{}'.format(epoch, ave_loss))
    
    
    # 验证过程
    def val(model, device, test_loader):
        model.eval()
        test_loss = 0
        correct = 0
        total_num = len(test_loader.dataset)
        print(total_num, len(test_loader))
        with torch.no_grad():
            for data, target in test_loader:
                data, target = Variable(data).to(device), Variable(target).to(device)
                output = model(data)
                loss = criterion(output, target)
                _, pred = torch.max(output.data, 1)
                correct += torch.sum(pred == target)
                print_loss = loss.data.item()
                test_loss += print_loss
            correct = correct.data.item()
            acc = correct / total_num
            avgloss = test_loss / len(test_loader)
            print('\nVal set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\n'.format(
                avgloss, correct, len(test_loader.dataset), 100 * acc))
    
    
    # 训练
    
    for epoch in range(1, EPOCHS + 1):
        adjust_learning_rate(optimizer, epoch)
        train(model_ft, DEVICE, train_loader, optimizer, epoch)
        val(model_ft, DEVICE, test_loader)
    torch.save(model_ft, 'model.pth')

完成上面的代码后就可以开始训练，点击run开始训练，如下图：

由于我们使用了预训练模型，所以收敛速度很快。

# 测试 #

我介绍两种常用的测试方式，第一种是通用的，通过自己手动加载数据集然后做预测，具体操作如下：

测试集存放的目录如下图：

第一步 定义类别，这个类别的顺序和训练时的类别顺序对应，一定不要改变顺序！！！！我们在训练时，cat类别是0，dog类别是1，所以我定义classes为(cat,dog)。

第二步 定义transforms，transforms和验证集的transforms一样即可，别做数据增强。

第三步 加载model，并将模型放在DEVICE里，

第四步 读取图片并预测图片的类别，在这里注意，读取图片用PIL库的Image。不要用cv2，transforms不支持。

import torch.utils.data.distributed
    import torchvision.transforms as transforms
    import torchvision.datasets as datasets
    from PIL import Image
    from torch.autograd import Variable
    import os
    classes = ('cat', 'dog')
    transform_test = transforms.Compose([
             transforms.Resize((224, 224)),
            transforms.ToTensor(),
            transforms.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5])
    ])
    
    DEVICE = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
    model = torch.load("model.pth")
    model.eval()
    model.to(DEVICE)
    
    path='data/test/'
    testList=os.listdir(path)
    for file in testList:
            img=Image.open(path+file)
            img=transform_test(img)
            img.unsqueeze_(0)
            img = Variable(img).to(DEVICE)
            out=model(img)
            # Predict
            _, pred = torch.max(out.data, 1)
            print('Image Name:{},predict:{}'.format(file,classes[pred.data.item()]))

运行结果：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70][]

第二种使用我们刚才定义的dataset.py加载测试集。代码如下：

import torch.utils.data.distributed
    import torchvision.transforms as transforms
    from dataset.dataset import DogCat
    from torch.autograd import Variable
    
    classes = ('cat', 'dog')
    transform_test = transforms.Compose([
        transforms.Resize((224, 224)),
        transforms.ToTensor(),
        transforms.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5])
    ])
    
    DEVICE = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
    model = torch.load("model.pth")
    model.eval()
    model.to(DEVICE)
    
    dataset_test =DogCat('data/test/', transform_test,test=True)
    print(len(dataset_test))
    # 对应文件夹的label
    
    for index in range(len(dataset_test)):
        item = dataset_test[index]
        img, label = item
        img.unsqueeze_(0)
        data = Variable(img).to(DEVICE)
        output = model(data)
        _, pred = torch.max(output.data, 1)
        print('Image Name:{},predict:{}'.format(dataset_test.imgs[index], classes[pred.data.item()]))
        index += 1

运行结果：

# ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 1][]       完整代码 #

train.py

import torch.optim as optim
    import torch
    import torch.nn as nn
    import torch.nn.parallel
    import torch.optim
    import torch.utils.data
    import torch.utils.data.distributed
    import torchvision.transforms as transforms
    from dataset.dataset import DogCat
    from torch.autograd import Variable
    from torchvision.models import alexnet
    
    # 设置全局参数
    modellr = 1e-4
    BATCH_SIZE = 32
    EPOCHS = 10
    DEVICE = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
    
    # 数据预处理
    
    transform = transforms.Compose([
        transforms.Resize((224, 224)),
        transforms.ToTensor(),
        transforms.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5])
    
    ])
    transform_test = transforms.Compose([
        transforms.Resize((224, 224)),
        transforms.ToTensor(),
        transforms.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5])
    ])
    dataset_train = DogCat('data/train', transforms=transform, train=True)
    dataset_test = DogCat("data/train", transforms=transform_test, train=False)
    # 读取数据
    print(dataset_train.imgs)
    
    # 导入数据
    train_loader = torch.utils.data.DataLoader(dataset_train, batch_size=BATCH_SIZE, shuffle=True)
    test_loader = torch.utils.data.DataLoader(dataset_test, batch_size=BATCH_SIZE, shuffle=False)
    
    # 实例化模型并且移动到GPU
    criterion = nn.CrossEntropyLoss()
    model_ft = alexnet(pretrained=True)
    model_ft.classifier = nn.Sequential(
                nn.Dropout(),
                nn.Linear(256 * 6 * 6, 4096),
                nn.ReLU(inplace=True),
                nn.Dropout(),
                nn.Linear(4096, 4096),
                nn.ReLU(inplace=True),
                nn.Linear(4096, 2),
            )
    model_ft.to(DEVICE)
    # 选择简单暴力的Adam优化器，学习率调低
    optimizer = optim.Adam(model_ft.parameters(), lr=modellr)
    
    
    def adjust_learning_rate(optimizer, epoch):
        """Sets the learning rate to the initial LR decayed by 10 every 30 epochs"""
        modellrnew = modellr * (0.1 ** (epoch // 50))
        print("lr:", modellrnew)
        for param_group in optimizer.param_groups:
            param_group['lr'] = modellrnew
    
    
    # 定义训练过程
    
    def train(model, device, train_loader, optimizer, epoch):
        model.train()
        sum_loss = 0
        total_num = len(train_loader.dataset)
        print(total_num, len(train_loader))
        for batch_idx, (data, target) in enumerate(train_loader):
            data, target = Variable(data).to(device), Variable(target).to(device)
            output = model(data)
            loss = criterion(output, target)
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
            print_loss = loss.data.item()
            sum_loss += print_loss
            if (batch_idx + 1) % 50 == 0:
                print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                    epoch, (batch_idx + 1) * len(data), len(train_loader.dataset),
                           100. * (batch_idx + 1) / len(train_loader), loss.item()))
        ave_loss = sum_loss / len(train_loader)
        print('epoch:{},loss:{}'.format(epoch, ave_loss))
    
    
    # 验证过程
    def val(model, device, test_loader):
        model.eval()
        test_loss = 0
        correct = 0
        total_num = len(test_loader.dataset)
        print(total_num, len(test_loader))
        with torch.no_grad():
            for data, target in test_loader:
                data, target = Variable(data).to(device), Variable(target).to(device)
                output = model(data)
                loss = criterion(output, target)
                _, pred = torch.max(output.data, 1)
                correct += torch.sum(pred == target)
                print_loss = loss.data.item()
                test_loss += print_loss
            correct = correct.data.item()
            acc = correct / total_num
            avgloss = test_loss / len(test_loader)
            print('\nVal set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\n'.format(
                avgloss, correct, len(test_loader.dataset), 100 * acc))
    
    
    # 训练
    
    for epoch in range(1, EPOCHS + 1):
        adjust_learning_rate(optimizer, epoch)
        train(model_ft, DEVICE, train_loader, optimizer, epoch)
        val(model_ft, DEVICE, test_loader)
    torch.save(model_ft, 'model.pth')

test1.py

test2.py

完整的代码连接：[AlexNet图像分类实战.zip-深度学习文档类资源-CSDN下载][AlexNet_.zip-_-CSDN]

[2071a471613d86fd23a878d085c4f81a.png]: /images/20221005/876c4e55f75c480ba795dae39ac46197.png
[_AlexNet_AI_-CSDN]: https://blog.csdn.net/hhhhhhhhhhwwwwwwwwww/article/details/117675363?spm=1001.2014.3001.5501
[20210608174508334.png]: /images/20221005/be0b350545cd4e769515acc49274a4c6.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70]: /images/20221005/296c220703294b5e9ccc8cf7cbda80a3.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 1]: /images/20221005/8303f389ae6242bd8137df4136f5c791.png
[AlexNet_.zip-_-CSDN]: https://download.csdn.net/download/hhhhhhhhhhwwwwwwwwww/19478456