CN113887503B - Improved attention convolution neural network-based five-classification method for white blood cells - Google Patents

The invention belongs to the field of medical microscopic image classification, and provides an improved attention convolution neural network leukocyte five-classification-based method for recognizing blood cell images by using deep learning. The method takes ResNeXt-50 as a backbone network, a residual error module uses packet convolution to reduce the number of model parameters, an independent attention module structure is added at the end of each stage of the network in parallel, aiming at a white blood cell characteristic diagram output by a convolutional neural network at different stages, a white blood cell key region is extracted by using the attention part of the attention module, a prediction type and a confidence score are output by the output part of the attention module, the output of a final network model is obtained by weighted averaging of the prediction type and the confidence score, and five white blood cell classification is realized based on improvement under the original ResNeXt-50 network framework. The invention utilizes the parallel attention modules to output class prediction and confidence score at different stages of the network, thereby improving the accuracy of leukocyte classification.

一种基于改进的注意力卷积神经网络白细胞五分类方法A five-class classification method of white blood cells based on an improved attention convolutional neural network

技术领域technical field

本发明属于医学显微图像分类领域，涉及一种并行嵌入注意力模块卷积神经网络的白细胞分类方法。The invention belongs to the field of medical microscopic image classification, and relates to a leukocyte classification method with parallel embedded attention module convolutional neural network.

背景技术Background technique

本部分的陈述仅仅是提供了与本发明相关的背景技术信息，不必然构成在先技术。The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

白细胞是免疫系统的一部分，它们负责摧毁和清除旧的或异常的细胞和细胞碎片，以及攻击病原体和异物。通常在血液中发现的白细胞是成熟的中性粒细胞、淋巴细胞和单核细胞，其中嗜酸性粒细胞和嗜碱性粒细胞的数量较少。白细胞数量的增加或减少可能代表某些疾病的发病迹象，各种白细胞的形态和比例可以反映一个人的健康状况。因此白细胞的精确分类对临床诊断和治疗起着至关重要的作用。人工镜检是临床上白细胞检测的“金标准”，可以准确的对白细胞进行分类并且可以观察到白细胞的病理学变化。但这种方法需要将血液样本经过繁琐的处理制作成血液涂片在显微镜下由专业的检验人员进行操作镜检，由于其操作繁琐，耗时时间长，镜检工作重复性强，可能会导致检验人员产生疲惫对白细胞的类别进行误判或漏判，进而影响对疾病的诊断和治疗。White blood cells are part of the immune system and they are responsible for destroying and removing old or abnormal cells and cellular debris, as well as attacking pathogens and foreign bodies. The white blood cells usually found in the blood are mature neutrophils, lymphocytes and monocytes, with eosinophils and basophils in small numbers. An increase or decrease in the number of white blood cells may represent a sign of the onset of certain diseases, and the shape and ratio of various white blood cells can reflect a person's health. Therefore, accurate classification of leukocytes plays a crucial role in clinical diagnosis and treatment. Manual microscopy is the "gold standard" for clinical leukocyte detection, which can accurately classify leukocytes and observe the pathological changes of leukocytes. However, this method requires the tedious processing of blood samples to make blood smears under the microscope by professional inspectors for microscopic examination. Examiners are tired and misjudge or miss the type of white blood cells, which in turn affects the diagnosis and treatment of diseases.

近年来，由于卷积神经网络在处理多阵列呈现的数据表现出良好的性能，被广泛应用在医疗图像分类领域。然而，目前很多深度学习方法在执行卷积运算时无法充分利用关键特征和重要信息，这在很大程度上限制了神经网络的性能。目前，注意力机制已经广泛应用在自然语言处理和计算机视觉等人工智能相关领域。使用注意力机制可以使模型学会注意力-能够忽略无关信息而关注关键信息，从而提高计算机视觉任务的性能。一些研究者已经把加入注意力机制的卷积神经网络用于白细胞的分类，主要包括：In recent years, convolutional neural networks have been widely used in the field of medical image classification due to their good performance in processing data presented in multiple arrays. However, many current deep learning methods cannot fully utilize key features and important information when performing convolution operations, which largely limits the performance of neural networks. At present, attention mechanism has been widely used in artificial intelligence related fields such as natural language processing and computer vision. Using an attention mechanism enables models to learn to pay attention - the ability to ignore irrelevant information and focus on key information, improving performance on computer vision tasks. Some researchers have used convolutional neural networks with attention mechanisms for the classification of white blood cells, including:

专利《基于深层卷积神经网络的多类白细胞自动识别方法》(CN 110059568 A)提出了一种加入了注意力的深度卷积神经网络对白细胞进行分类。该深度卷积神经网络利用9个inception模块级联组成，在第4个和第7个inception模块分别添加了辅助分类器，并且在每个inception模块级联的时候加入SE-Net通道注意力。在ILSVRC数据集对模型进行预训练，在训练集上对网络进行微调，实现对白细胞的分类。The patent "Multiple Types of White Blood Cell Automatic Identification Method Based on Deep Convolutional Neural Network" (CN 110059568 A) proposes a deep convolutional neural network with added attention to classify white blood cells. The deep convolutional neural network is composed of 9 inception modules cascaded, auxiliary classifiers are added to the 4th and 7th inception modules respectively, and SE-Net channel attention is added when each inception module is cascaded. The model is pre-trained on the ILSVRC dataset, and the network is fine-tuned on the training set to classify white blood cells.

缺点：该方法添加的辅助分类器利用了两个中间特征层来预测类别只是为了防止梯度消失等问题，并没有为网络最终决策提供更多帮助；此外inception网络超参数过多，泛化能力较差，不一定有一个较好的结果。Disadvantages: The auxiliary classifier added by this method uses two intermediate feature layers to predict the category just to prevent problems such as gradient disappearance, and does not provide more help for the final decision of the network; in addition, the inception network has too many hyperparameters, and the generalization ability is relatively low. Poor, not necessarily a better result.

专利《基于混合残差注意力残差网络实现白细胞自动分类方法及系统》(CN113343799A)提出一种混合注意力机制的残差网络对白细胞进行自动分类，该混合注意力机制残差网络通过堆叠使用残差注意力模块构成一个混合注意力残差网络，将卷积后得到的特征图通过混合注意力模块提取关键特征实现白细胞的分类。The patent "Method and System for Automatic White Blood Cell Classification Based on Mixed Residual Attention Residual Network" (CN113343799A) proposes a mixed attention mechanism residual network to automatically classify white blood cells. The mixed attention mechanism residual network is used by stacking The residual attention module constitutes a mixed attention residual network, and the feature map obtained after convolution is extracted through the mixed attention module to extract key features to achieve the classification of white blood cells.

缺点：此网络由多个残差注意力模块堆叠而成，忽略了低层次特征在遍历多次残差注意连接后受到的噪声影响，并且低层次特征(如纹理)在细粒度识别中很重要，可以帮助区分两个相似的类。Disadvantages: This network is composed of multiple residual attention modules stacked, ignoring the noise impact of low-level features after traversing multiple residual attention connections, and low-level features (such as texture) are important in fine-grained recognition , which can help distinguish two similar classes.

发明内容SUMMARY OF THE INVENTION

针对目前方法中存在的堆叠使用残差注意力模块带来的噪声影响以及未能完全利用不同层次特征的问题，本发明提供了一种基于改进的注意力卷积神经网络白细胞五分类方法，以ResNext-50为骨干网路，网络中的分组卷积可以大大减少模型参数量，在骨干网络并行嵌入一种独立的注意力机制模块对低层次特征图和高层次特征图提取关键信息，并通过注意力模块分别得到不同层次特征图的预测类别和置信度分数，最后对所有预测类别通过置信度加权和平均得到模型输出类别，达到了提高白细胞分类准确率的目的。Aiming at the problems of noise effects caused by stacking and using residual attention modules and failure to fully utilize different levels of features in the current method, the present invention provides an improved attention convolutional neural network-based five-classification method for white blood cells. ResNext-50 is the backbone network. The grouped convolution in the network can greatly reduce the amount of model parameters. An independent attention mechanism module is embedded in the backbone network to extract key information from low-level feature maps and high-level feature maps. The attention module obtains the predicted categories and confidence scores of the feature maps at different levels, and finally obtains the model output category by weighting and averaging all the predicted categories, which achieves the purpose of improving the accuracy of white blood cell classification.

为了实现上述目的，本发明采用的技术方案是：In order to achieve the above object, the technical scheme adopted in the present invention is:

一种基于改进的注意力卷积神经网络的白细胞分类方法，包括以下步骤：A method for leukocyte classification based on an improved attentional convolutional neural network, including the following steps:

步骤(1)：采集白细胞图像，并把完整的血液显微图像裁剪为单独的图像，并由血液检验专家对白细胞进行类别标注；Step (1): collect leukocyte images, and crop the complete blood microscopic image into separate images, and the leukocytes are classified by blood test experts;

步骤(2)：对于步骤(1)采集到的白细胞显微图片进行图像增强操作，并进行预处理；Step (2): performing image enhancement operations on the leukocyte microscopic pictures collected in step (1), and performing preprocessing;

步骤(3)：将步骤(2)处理后的白细胞图像数据集随机地按照一定的比例划分为训练集和测试集；Step (3): The leukocyte image data set processed in step (2) is randomly divided into a training set and a test set according to a certain proportion;

步骤(4)：构建改进的注意力卷积神经网络模型，使用步骤(3)划分的训练集对模型进行巡练，此过程为一次前向传播过程；Step (4): construct an improved attention convolutional neural network model, and use the training set divided in step (3) to conduct patrol training of the model, and this process is a forward propagation process;

步骤(5)：经过一次向前传播后，使用交叉熵损失函数计算预测值和真实值之间的误差，通过使用Adam算法来不断地减小损失误差，并更新网络模型每层的参数，此过程为一次后向传播过程；Step (5): After one forward propagation, use the cross entropy loss function to calculate the error between the predicted value and the true value, and use the Adam algorithm to continuously reduce the loss error and update the parameters of each layer of the network model. The process is a backward propagation process;

步骤(6)：反复经过步骤(4)的前向传播和步骤(5)的后向传播，不断更新网络层参数，当训练轮数达到设置的最大训练轮数时，网络模型收敛，训练结束，保存训练集准确率最高的网络模型为最优网络模型；Step (6): Repeat the forward propagation of step (4) and the backward propagation of step (5), and continuously update the network layer parameters. When the number of training rounds reaches the set maximum number of training rounds, the network model converges and the training ends. , save the network model with the highest accuracy in the training set as the optimal network model;

步骤(7)：使用步骤(6)保存的最优网络模型进行白细胞分类，输入一张白细胞图像到训练好的模型中，输出白细胞的类别。Step (7): Use the optimal network model saved in step (6) to classify white blood cells, input a white blood cell image into the trained model, and output the type of white blood cells.

步骤(1)的具体实施过程包括：首先使用显微镜对医院提供的血液涂片进行白细胞显微图像采集，在整个显微图像上以一个完整白细胞为中心裁剪大小为256*256的白细胞图像，并由血液专业对裁剪后的白细胞图像类别进行标注。The specific implementation process of step (1) includes: first, use a microscope to collect a leukocyte microscopic image on the blood smear provided by the hospital, and crop a leukocyte image with a size of 256*256 on the entire microscopic image with a complete leukocyte as the center, and The cropped leukocyte image category is annotated by the hematology professional.

步骤(2)的具体实施过程包括：针对白细胞进行数据增强操作，包括上下左右裁剪、随机旋转、增强图像对比度，镜像翻转。The specific implementation process of step (2) includes: performing data enhancement operations on white blood cells, including cropping up, down, left, and right, randomly rotating, enhancing image contrast, and mirroring flipping.

步骤(3)中将步骤(2)处理过后的图像数据集随机地按照7:3的比例划分为训练集和测试集。In step (3), the image data set processed in step (2) is randomly divided into a training set and a test set according to a ratio of 7:3.

所述的网络模型以残差网络ResNeXt-50为骨架使用分组卷积减少模型参数量，使用一种轻量型的注意力机制并行嵌入残差网络ResNeXt-50的各个阶段，对网络模型的不同层次的特征图生成空间注意热图，并输出基于局部信息的类别预测和置信度分数。最终的输出预测由所有类别预测和归一化置信度分数加权而成。The described network model takes the residual network ResNeXt-50 as the skeleton, uses grouped convolution to reduce the amount of model parameters, and uses a lightweight attention mechanism to embed all stages of the residual network ResNeXt-50 in parallel. Hierarchical feature maps generate spatial attention heatmaps and output class predictions and confidence scores based on local information. The final output prediction is weighted by all class predictions and normalized confidence scores.

我们所采用的注意力机制可以添加在每个卷积层之后，并且不改变网络整体的结构。注意力模块主要包括两个子模块：注意头H，它提取了特征图对类别决策最相关的区域；输出头O，通过全局池化和全连接生成类别预测，并且为每个注意头输出置信门得分。每个注意力机制将得到类别预测和置信度得分，最后将所有的类预测由置信度分数加权平均得到最终的预测类别。The attention mechanism we adopt can be added after each convolutional layer without changing the overall structure of the network. The attention module mainly consists of two sub-modules: the attention head H, which extracts the most relevant regions of the feature map for class decisions; the output head O, which generates class predictions through global pooling and full connection, and outputs a confidence gate for each attention head Score. Each attention mechanism will get a class prediction and a confidence score, and finally all class predictions are weighted by the confidence score to get the final predicted class.

与之前的技术相比，本发明具有以下优势：Compared with the previous technology, the present invention has the following advantages:

1、本发明改进的注意力卷积神经网络同时采用注意力机制对低层次特征和高层次特征提取最感兴趣的部分，并利用这些特征输出类别预测和置信度得分来帮助最后模型的分类决策。1. The improved attention convolutional neural network of the present invention simultaneously adopts the attention mechanism to extract the most interesting parts of low-level features and high-level features, and uses these features to output category prediction and confidence score to help the classification decision of the final model .

2、本发明采用了残差网络模型，在残差模块中使用了分组卷积，让网络学习不同的特征，降低网络参数的同时，也提高了模型的精度。2. The present invention adopts a residual network model, and uses grouped convolution in the residual module to allow the network to learn different features, reduce network parameters, and improve the accuracy of the model.

3、本发明采用了一种轻量化的注意力机制，并不会给模型带来复杂的参数，且注意力机制模块十分灵活，可以在不同的深度和宽度进行扩展，并且表现出很好的性能。3. The present invention adopts a lightweight attention mechanism, which does not bring complex parameters to the model, and the attention mechanism module is very flexible, can be expanded in different depths and widths, and shows good performance. performance.

附图说明Description of drawings

图1为本发明方法流程图。Fig. 1 is the flow chart of the method of the present invention.

图2为本发明中使用的ResNeXt-50(34×4d)的残差模块。FIG. 2 is the residual module of ResNeXt-50 (34×4d) used in the present invention.

图3为本发明中使用的注意力模块示意图。FIG. 3 is a schematic diagram of an attention module used in the present invention.

图4为本发明中的注意力卷积神经网络示意图。FIG. 4 is a schematic diagram of an attention convolutional neural network in the present invention.

图5为五类白细胞显微图像。Figure 5 is a microscopic image of five types of leukocytes.

具体实施方式Detailed ways

以下结合附图和技术方案，进一步说明本发明的具体实施方式。The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and technical solutions.

如图1所示，本发明的总体步骤如下：As shown in Figure 1, the overall steps of the present invention are as follows:

步骤1：数据集的采集和准备，将血液检验专家制作好的血液涂片在相同条件下用装有工业相机的生物显微镜(放大倍数为1000倍)进行白细胞显微图像的采集。以完整的单个白细胞为中心在整张图像裁剪出大小为256*256的白细胞图像，并且由血液检验专家将这些大小为256*256的白细胞图像进行类别标注，准确地分出嗜中性粒细胞、嗜酸性粒细胞、嗜碱性粒细胞、单核细胞和淋巴细胞。Step 1: Collection and preparation of the data set, the blood smear made by the blood test expert is used to collect the microscopic image of white blood cells with a biological microscope equipped with an industrial camera (magnification of 1000 times) under the same conditions. A leukocyte image with a size of 256*256 is cropped from the entire image centered on a complete single leukocyte, and these leukocyte images with a size of 256*256 are labeled by the blood test experts to accurately separate neutrophils , eosinophils, basophils, monocytes and lymphocytes.

步骤2：对步骤1采集和标注后的白细胞图像进行数据增强操作，具体的，对白细胞图像进行裁剪，分别以图像四个角和中心各截取大小为224*224的白细胞图像，并对进行镜像翻转，以及30度、60度旋转，增强图像对比度，以此来获取不同位置和不同环境下的白细胞图像，防止模型过拟合，增加模型的泛化能力。Step 2: Perform data enhancement operations on the leukocyte image collected and labeled in step 1. Specifically, the leukocyte image is cropped, and the leukocyte image with a size of 224*224 is intercepted from the four corners and the center of the image, and mirrored. Flip, as well as 30-degree, 60-degree rotation, enhance the image contrast, so as to obtain white blood cell images in different positions and different environments, prevent the model from overfitting, and increase the generalization ability of the model.

步骤3：将步骤2增强过后的白细胞数据集随机按照7:3的比例划分为训练集和测试集，训练集用于对卷积网络模型的参数训练过程，测试集用于检验整个白细胞五分类识别算法的效率和参数权重的更新；Step 3: The leukocyte data set enhanced in step 2 is randomly divided into a training set and a test set according to the ratio of 7:3. The training set is used for the parameter training process of the convolutional network model, and the test set is used to test the entire white blood cell five classifications. Identify the efficiency of the algorithm and update the parameter weights;

步骤4：构建改进的注意力卷积神经网络模型，本发明设计的网络模型以ResNeXt-50为骨干网络，在每个阶段后面并行嵌入了一种注意力机制来利用不同层次特征最有用的部分生成类别预测和置信度得分辅助最终模型的决策。Step 4: Build an improved attention convolutional neural network model. The network model designed by the present invention uses ResNeXt-50 as the backbone network, and an attention mechanism is embedded in parallel after each stage to utilize the most useful parts of different levels of features Generate class predictions and confidence scores to aid in the decision of the final model.

1)ResNeXt-50网络将inception、ResNet、VGG中的优秀思想归纳并演绎，得到一个结构简洁的强大的网络结构。ResNeXt-50由一个普通卷积结构，一些残差块，和一个全连接层组成。如图2所示，每个残差模块的左半部分由两个1*1的卷积核和3*3的卷积核组成卷积操作，右半部分则是一个快速连接操作，两部分的结果经过加操作经激活函数得到输出。具体地，对于左半部分的卷积操作首先通过1*1的卷积核实现整体升降维，然后采用分组卷积的思想对通道切分为32个分支，每个32分支的4通道特征图经过3*3卷积核分别做运算，将得到的变换结果(特征图)进行聚合。与Resnet类似，整个ResNeXt-50有4个layer层，layer1含有3个残差块，layer2含有4个残差块，layer3含有6个残差块，layer4含有3个残差块。1) The ResNeXt-50 network summarizes and deduces the excellent ideas in inception, ResNet, and VGG, and obtains a powerful network structure with a simple structure. ResNeXt-50 consists of a common convolutional structure, some residual blocks, and a fully connected layer. As shown in Figure 2, the left half of each residual module consists of two 1*1 convolution kernels and 3*3 convolution kernels to form a convolution operation, and the right half is a fast connection operation. The result of the addition operation is output through the activation function. Specifically, for the convolution operation of the left half, the overall dimension reduction is realized through a 1*1 convolution kernel, and then the channel is divided into 32 branches by the idea of grouped convolution, and each 32 branches has a 4-channel feature map. After 3*3 convolution kernels are operated separately, the obtained transformation results (feature maps) are aggregated. Similar to Resnet, the entire ResNeXt-50 has 4 layers, layer1 contains 3 residual blocks, layer2 contains 4 residual blocks, layer3 contains 6 residual blocks, and layer4 contains 3 residual blocks.

2)在ResNeXt-50不同阶段末嵌入注意力模块，如图3所示，每个注意力模块包括了注意头和输出头两大部分。注意头部分对卷积过后得到的特征图Z使用1*1的卷积核进行卷积操作和空间softmax输出注意力热图M，注意力热图M与输入特征图Z通道的乘积通过广播机制得到注意头的输出H，其中M是一个2维平面，空间softmax用于模型学习图像中最相关的区域。每个注意模块注意头的的输出H由一个空间降维层(即全局池化层)组成，后面通过一个全连接层产生类别预测o，每个注意力模块根据其局部信息做出类别预测o。然而，在某些情况下，局部特征并不足以输出一个好的预测。为了缓解这个问题，我们让每个注意力模块以及骨架网络输出，通过与权矩阵的内积来预测置信度分数c，然后通过softmax函数对置信度分数进行归一化，得到权值g，网络的最终输出是各输出的类别预测和置信度分数的加权和，计算公式为2) Embed attention modules at the end of different stages of ResNeXt-50, as shown in Figure 3, each attention module includes two parts: attention head and output head. Note that the head part uses a 1*1 convolution kernel to perform the convolution operation on the feature map Z obtained after convolution and spatial softmax to output the attention heat map M. The product of the attention heat map M and the input feature map Z channel passes through the broadcasting mechanism The output H of the attention head is obtained, where M is a 2-dimensional plane, and the spatial softmax is used for the model to learn the most relevant regions in the image. The output H of the attention head of each attention module consists of a spatial dimensionality reduction layer (ie, a global pooling layer), followed by a fully connected layer to generate category prediction o, and each attention module makes category prediction o based on its local information . However, in some cases, local features are not enough to output a good prediction. In order to alleviate this problem, we let each attention module and the skeleton network output, predict the confidence score c through the inner product with the weight matrix, and then normalize the confidence score through the softmax function to obtain the weight g, the network The final output of is the weighted sum of the category predictions and confidence scores for each output, calculated as

output＝g_net·output_net+∑∑g^l _k·o^l _k output=g _net ·output _net +∑∑g ^l _k ·o ^l _k

其中output为整个网络模型的最终输出，g_net为骨干网络的输出的归一化后的置信度分数，output_net为骨架网络的的类别预测，g^l _k为每个注意力模块的输出的归一化后的置信度分数，o^l _k为每个注意力模块的类别预测。where output is the final output of the entire network model, g _net is the normalized confidence score of the output of the backbone network, output _net is the category prediction of the backbone network, and g ^l _k is the normalized output of each attention module ^The normalized confidence score, ol _k is the class prediction for each attention module.

使用步骤3得到的白细胞训练集数据对步骤4设计的网络模型进行训练，此过程为一次前向传播。Use the white blood cell training set data obtained in step 3 to train the network model designed in step 4, and this process is a forward propagation.

经过一次向前传播后，使用交叉熵损失函数计算预测值和真实值之间的误差，通过使用随机梯度下降算法来不断地减小损失误差，并更新网络模型每层的参数，采用学习率固定步长递减策略，步长为7个epoch，gamma系数为0.1，此过程为一次后向传播过程。After one forward propagation, the cross-entropy loss function is used to calculate the error between the predicted value and the true value, the loss error is continuously reduced by using the stochastic gradient descent algorithm, and the parameters of each layer of the network model are updated, using a fixed learning rate The step size decreases strategy, the step size is 7 epochs, and the gamma coefficient is 0.1. This process is a back propagation process.

反复经过步骤5的前向传播和步骤6的后向传播，不断更新网络层参数，当训练轮数达到设置的最大训练轮数时，网络模型收敛，训练结束，保存训练集准确率最高的网络模型为最优网络模型。Repeat the forward propagation of step 5 and the backward propagation of step 6, and continuously update the network layer parameters. When the number of training rounds reaches the maximum number of training rounds set, the network model converges, the training ends, and the network with the highest accuracy of the training set is saved. The model is the optimal network model.

利用步骤7保存的最优模型对白细胞测试集进行五分类预测，测试测准确率如下表所示。Use the optimal model saved in step 7 to perform five classification predictions on the white blood cell test set, and the test accuracy is shown in the following table.

应当理解的是，本说明书未详细阐述的部分均属于现有技术。对于本领域的技术人员来说，本发明可以有各种更改和变化。凡在本发明的精神和原则之内，所作的任何修改、等同替换、改进等均应包含在本发明的保护范围内。It should be understood that the parts not described in detail in this specification belong to the prior art. Various modifications and variations of the present invention are possible for those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.