CN114757906A - Detection method of leukocytes in microscopic images based on DETR model - Google Patents

The invention discloses a method for detecting leucocytes in a microscopic image based on a DETR model, which comprises the following steps: 1) collecting a microscopic image containing white blood cells, manually marking the white blood cells in the microscopic image to construct a data set, and then dividing the data set into a training set and a verification set; 2) enhancing the brightness and the contrast of the data set, adjusting the brightness and the contrast as hyper-parameters, and determining the optimal hyper-parameters according to the detection precision of the verification set; 3) training the DETR model through a training set by using a transfer learning algorithm; 4) and inputting the microscopic image to be detected into the trained DETR model to obtain a leucocyte detection result in the microscopic image. The invention firstly applies the DETR model to the field of leukocyte target detection of microscopic images, realizes the faster convergence of the DETR model in the leukocyte target detection training and obtains higher target detection precision and robustness.

基于DETR模型的显微图像中白细胞检测方法Detection method of leukocytes in microscopic images based on DETR model

技术领域technical field

本发明涉及医学显微图像的智能检测领域，特别涉及一种基于DETR模型的显微图像中白细胞检测方法。The invention relates to the field of intelligent detection of medical microscopic images, in particular to a method for detecting white blood cells in microscopic images based on a DETR model.

背景技术Background technique

外周血细胞分析已成为临床实验室的常规检测项目。虽然全自动五分类血液细胞分类仪在各级临床实验室中被普及，但仍有大量样本需要的人工形态学复检以进行疾病的诊断。血细胞形态学检验是血液细胞形态学和分类计数的“金标准”，人工显微镜镜检需要检验师具有相应资历及工作经验，并且容易出现观察者之间的差异，长时间的显微镜下工作对工作人员的健康造成极大的损害。Peripheral blood cell analysis has become a routine test item in clinical laboratories. Although the automatic five-part blood cell sorter is popularized in clinical laboratories at all levels, there are still a large number of samples that require manual morphological re-examination for disease diagnosis. Blood cell morphology test is the "gold standard" for blood cell morphology and classification and counting. Manual microscope inspection requires inspectors to have corresponding qualifications and work experience, and differences between observers are prone to occur. Working under a microscope for a long time is not conducive to work. Great damage to the health of people.

随着深度学习概念在2006年由Hinton等[1]提出，基于卷积神经网络(CNN)的自动化检测算法相继提出并应用于医学图像检测领域，表现出比传统图像处理方法更高的性能。如检测性能较高的Faster R-CNN，它的检测过程分为两步：先预测出包含物体的很多候选框，再进行最大值抑制的后处理过程。但是它的性能受到后处理、锚点集的设计以及将目标框分配给锚点等步骤的影响[2]。With the concept of deep learning proposed by Hinton et al. [1] in 2006, automatic detection algorithms based on convolutional neural networks (CNN) have been successively proposed and applied in the field of medical image detection, showing higher performance than traditional image processing methods. For example, Faster R-CNN with high detection performance, its detection process is divided into two steps: firstly, many candidate frames containing objects are predicted, and then the post-processing process of maximum suppression is performed. But its performance is affected by steps such as post-processing, anchor set design, and assignment of target boxes to anchors [2].

最近基于Transformer的模型在计算机视觉领域表现出超越CNN模型的性能。其中DETR模型将目标检测看作是一个直接的集合预测问题，简化了以往基于人工锚点识别、合并重复结果的两步流程，是一种端到端的目标检测方法。DETR模型有望在更多的图像处理问题中进行应用，但现在未见公开将DETR模型用于白细胞检测的可靠方案。Recently Transformer-based models have shown performance that surpasses CNN models in the computer vision domain. Among them, the DETR model regards target detection as a direct set prediction problem, which simplifies the previous two-step process based on manual anchor point identification and merging repeated results, and is an end-to-end target detection method. The DETR model is expected to be applied in more image processing problems, but there is no published reliable scheme for using the DETR model for leukocyte detection.

发明内容SUMMARY OF THE INVENTION

本发明所要解决的技术问题在于针对上述现有技术中的不足，提供一种基于DETR模型的显微图像中白细胞检测方法、存储介质及计算机设备。The technical problem to be solved by the present invention is to provide a method for detecting leukocytes in a microscopic image based on the DETR model, a storage medium and a computer device, aiming at the deficiencies in the above-mentioned prior art.

为解决上述技术问题，本发明采用的技术方案是：一种基于DETR模型的显微图像中白细胞检测方法，包括：In order to solve the above-mentioned technical problems, the technical solution adopted in the present invention is: a method for detecting leukocytes in a microscopic image based on a DETR model, comprising:

1)采集包含白细胞的显微图像，人工标注出显微图像中的白细胞，构建成数据集，然后将数据集划分为训练集和验证集；1) Collect a microscopic image containing leukocytes, manually mark the leukocytes in the microscopic image, construct a data set, and then divide the data set into a training set and a validation set;

2)使用colorjitter算法对数据集亮度和对比度的增强，将亮度和对比度的作为超参数进行调节，由验证集的检测精度确定最佳超参数；2) Use the colorjitter algorithm to enhance the brightness and contrast of the dataset, adjust the brightness and contrast as hyperparameters, and determine the best hyperparameters by the detection accuracy of the validation set;

3)使用迁移学习算法通过训练集对DETR模型进行训练，达到设定的精度后，得到训练好的DETR模型；3) Use the migration learning algorithm to train the DETR model through the training set, and obtain the trained DETR model after reaching the set accuracy;

4)将待检测的显微图像输入训练好的DETR模型中，得到显微图像中的白细胞检测结果。4) Input the microscopic image to be detected into the trained DETR model to obtain the white blood cell detection result in the microscopic image.

优选的是，所述步骤1)中，将数据集的图片和标注信息做成MSCOCO数据集的格式后再进行划分。Preferably, in the step 1), the pictures and annotation information of the data set are made into the format of the MSCOCO data set and then divided.

优选的是，所述步骤1)中，将按照训练集：验证集＝8:2的比例对数据集进行划分。Preferably, in the step 1), the data set will be divided according to the ratio of training set:validation set=8:2.

优选的是，所述步骤2)具体包括：Preferably, the step 2) specifically includes:

2-1)调用pytorch框架中的ColorJitter类，加入到DETR数据增强方法中，用于对图片亮度和对比度进行增强；2-1) Call the ColorJitter class in the pytorch framework and add it to the DETR data enhancement method to enhance the brightness and contrast of the picture;

2-2)首先给定特定亮度值a和对比度值b，通过将该值下增强的图片与其他图片对比，定性确定是否在数据集图片的预设亮度和对比度范围内，以此为基准进行训练；2-2) First, given a specific brightness value a and contrast value b, by comparing the enhanced picture at this value with other pictures, qualitatively determine whether it is within the preset brightness and contrast range of the data set picture, and use this as a benchmark to carry out train;

2-3)将亮度值a和对比度值b作为模型训练的超参数，根据训练得到的模型检测精度进行调参，调节为分别包含a和b的区间范围，以达到最优检测精度，作为最终的超参数。2-3) Take the brightness value a and the contrast value b as the hyperparameters of model training, adjust the parameters according to the model detection accuracy obtained by training, and adjust the parameters to include the interval range of a and b respectively, so as to achieve the optimal detection accuracy, as the final result. hyperparameters.

优选的是，所述步骤3)具体包括：获取MSCOCO数据集上的DETR预训练权重，导入到DETR模型中，通过步骤2)得到的训练集对DETR模型进行训练，达到设定的精度后，得到训练好的DETR模型。Preferably, the step 3) specifically includes: obtaining the DETR pre-training weight on the MSCOCO data set, importing it into the DETR model, and training the DETR model through the training set obtained in step 2), and after reaching the set accuracy, Get the trained DETR model.

优选的是，所述步骤3)中，将DETR预训练权重文件中的类别层中的class_embed相关类别参数更新为6，以适应迁移学习的任务。Preferably, in the step 3), the relevant category parameter of class_embed in the category layer in the DETR pre-training weight file is updated to 6, so as to adapt to the task of transfer learning.

优选的是，所述步骤3)中，将DETR模型主体中定义的类别参数num_classes改成6，以适应迁移学习的任务。Preferably, in the step 3), the class parameter num_classes defined in the body of the DETR model is changed to 6, so as to adapt to the task of transfer learning.

优选的是，所述DETR模型包括输入层、colorjitter数据增强模块、CNN骨干网络、图像特征集合模块、基于Transformer的编码器与解码器以及输出层。Preferably, the DETR model includes an input layer, a colorjitter data enhancement module, a CNN backbone network, an image feature collection module, a Transformer-based encoder and decoder, and an output layer.

本发明还提供一种存储介质，其上存储有计算机程序，该程序被执行时用于实现如上所述的方法。The present invention also provides a storage medium on which a computer program is stored, and when the program is executed, the method is used to implement the above-mentioned method.

本发明还提供一种计算机设备，包括存储器、处理器以及存储在所述存储器上并可在处理器上运行的计算机程序，所述处理器执行所述计算机程序时实现如权上所述的方法。The present invention also provides a computer device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor implements the method described above when the processor executes the computer program .

本发明的有益效果是：The beneficial effects of the present invention are:

本发明专利首次将DETR模型应用在显微图像的白细胞目标检测领域，本发明提供的方法基于transformer和二部图匹配损失进行直接集合预测，实现了端到端的白细胞目标检测；The patent of the present invention applies the DETR model to the field of white blood cell target detection in microscopic images for the first time. The method provided by the present invention performs direct set prediction based on transformer and bipartite graph matching loss, and realizes end-to-end white blood cell target detection;

由于医学数据集需要医生进行专业标注，具有收集困难且数量少的特点；而在小型医学数据集上对模型进行从头开始的训练，往往达不到良好的性能；迁移学习是在大型数据集上对模型进行预训练，并在小型数据集上进行微调的训练方式，有效解决了小型数据集上模型训练的问题；本发明在公开发表的显微图像白细胞检测数据集上，采用迁移学习的方式对DETR进行训练，有效提高模型的检测精度及收敛速度；Because medical datasets require professional labeling by doctors, it is difficult to collect and has a small number; however, training models from scratch on small medical datasets often cannot achieve good performance; transfer learning is performed on large datasets. The training method of pre-training the model and fine-tuning on the small data set effectively solves the problem of model training on the small data set; the invention adopts the method of migration learning on the published microscopic image leukocyte detection data set Train DETR to effectively improve the detection accuracy and convergence speed of the model;

显微图像容易受光强、玻片染色条件等因素的影响而产生较大的差异，降低了检测模型的鲁棒性；本发明在训练过程中增加了对图片亮度、对比度等的数据增强方式，能有效提高检测模型的鲁棒性及检测精度；Microscopic images are easily affected by factors such as light intensity, glass slide staining conditions and other factors, resulting in large differences, which reduces the robustness of the detection model; the invention adds data enhancement methods for image brightness, contrast, etc. in the training process. It can effectively improve the robustness and detection accuracy of the detection model;

本发明实现了DETR模型在白细胞目标检测训练中的较快收敛，且取得了较高的目标检测精度及鲁棒性，有助于推动深度学习模型在显微图像目标检测领域的应用。The invention realizes the faster convergence of the DETR model in the training of leukocyte target detection, and achieves high target detection accuracy and robustness, which is helpful for promoting the application of the deep learning model in the field of microscopic image target detection.

附图说明Description of drawings

图1为本发明的基于DETR模型的显微图像中白细胞检测方法的流程图；1 is a flowchart of a method for detecting leukocytes in a microscopic image based on the DETR model of the present invention;

图2为本发明的DETR模型的网络结构示意图；Fig. 2 is the network structure schematic diagram of DETR model of the present invention;

图3为本发明中训练好的DETR模型的检测结果；Fig. 3 is the detection result of DETR model trained in the present invention;

图4为本发明的迁移学习与从头开始训练的精度对比结果。FIG. 4 is the accuracy comparison result between the transfer learning of the present invention and the training from scratch.

具体实施方式Detailed ways

下面结合实施例对本发明做进一步的详细说明，以令本领域技术人员参照说明书文字能够据以实施。The present invention will be further described in detail below with reference to the embodiments, so that those skilled in the art can implement according to the description.

应当理解，本文所使用的诸如“具有”、“包含”以及“包括”术语并不排除一个或多个其它元件或其组合的存在或添加。It should be understood that terms such as "having", "comprising" and "including" as used herein do not exclude the presence or addition of one or more other elements or combinations thereof.

实施例1Example 1

参照图1，本实施例提供一种基于DETR模型的显微图像中白细胞检测方法，包括：Referring to FIG. 1 , the present embodiment provides a method for detecting leukocytes in a microscopic image based on the DETR model, including:

1)采集包含白细胞的显微图像，在医生指导下人工标注出显微图像中的白细胞，构建成数据集，将数据集的图片和标注信息做成MSCOCO数据集的格式，然后将按照训练集：验证集＝8:2的比例对数据集进行划分；1) Collect a microscopic image containing leukocytes, manually mark the leukocytes in the microscopic image under the guidance of a doctor, and construct a data set. The pictures and annotation information of the data set are made into the format of the MSCOCO data set, and then the training set will be used according to the training set. : Validation set = 8:2 ratio to divide the dataset;

2)使用colorjitter算法对数据集亮度和对比度的增强，将亮度和对比度的作为超参数进行调节，由验证集的检测精度确定最佳超参数；具体包括：2) Use the colorjitter algorithm to enhance the brightness and contrast of the dataset, adjust the brightness and contrast as hyperparameters, and determine the best hyperparameters based on the detection accuracy of the validation set; specifically:

2-1)调用pytorch框架中的torchvision.transforms模块中的ColorJitter类，加入到DETR数据增强方法中，用于对图片亮度和对比度进行增强；由于不同光强及染色条件差异主要影响图片亮度和对比度，因此针对图片亮度和对比度进行增强；2-1) Call the ColorJitter class in the torchvision.transforms module in the pytorch framework and add it to the DETR data enhancement method to enhance the brightness and contrast of the picture; due to the difference in light intensities and staining conditions, the brightness and contrast of the picture are mainly affected , so the brightness and contrast of the picture are enhanced;

2-2)首先给定特定亮度值a和对比度值b，通过将该值下增强的图片与其他图片对比，定性确定是否在数据集图片的预设亮度和对比度范围内，以此为基准进行训练；2-2) First, given a specific brightness value a and contrast value b, by comparing the enhanced picture at this value with other pictures, qualitatively determine whether it is within the preset brightness and contrast range of the data set picture, and use this as a benchmark to carry out train;

2-3)将亮度值a和对比度值b作为模型训练的超参数，根据训练得到的模型检测精度进行调参，调节为分别包含a和b的区间范围，以达到最优检测精度，作为最终的超参数。2-3) Take the brightness value a and the contrast value b as the hyperparameters of model training, adjust the parameters according to the model detection accuracy obtained by training, and adjust the parameters to include the interval range of a and b respectively, so as to achieve the optimal detection accuracy, as the final result. hyperparameters.

3)获取MSCOCO数据集上的DETR预训练权重，导入到DETR模型中，通过步骤2)得到的训练集对DETR模型进行训练，达到设定的精度后，得到训练好的DETR模型。3) Obtain the DETR pre-training weights on the MSCOCO data set, import them into the DETR model, train the DETR model through the training set obtained in step 2), and obtain the trained DETR model after reaching the set accuracy.

由于MSCOCO数据集有90类，而白细胞数据集只有5类，对预训练的权重文件进行修改，以适应权重导入过程中的类别匹配。即将预训练权重文件中的类别层中的class_embed相关类别参数更新为6(包含空集类)。2.对DETR模型主体中定义的类别参数进行修改，即将num_classes改成6(包含空集类)，以适应迁移学习的任务。Since the MSCOCO dataset has 90 classes and the leukocyte dataset has only 5 classes, the pretrained weights file is modified to accommodate class matching during the weight import process. That is, update the class_embed related category parameter in the category layer in the pre-training weight file to 6 (including the empty set class). 2. Modify the category parameters defined in the body of the DETR model, that is, change num_classes to 6 (including empty set classes) to adapt to the task of transfer learning.

4)将待检测的显微图像输入训练好的DETR模型中，得到显微图像中的白细胞检测结果。4) Input the microscopic image to be detected into the trained DETR model to obtain the white blood cell detection result in the microscopic image.

DETR模型包括输入层、colorjitter数据增强模块、CNN骨干网络、图像特征集合模块、基于Transformer的编码器与解码器以及输出层。参照图2，为DETR模型的网络结构示意图。The DETR model includes an input layer, a colorjitter data enhancement module, a CNN backbone network, an image feature collection module, a Transformer-based encoder and decoder, and an output layer. Referring to FIG. 2 , it is a schematic diagram of the network structure of the DETR model.

参照图3，为训练好的DETR模型的检测结果，可以看出，该方法能够实现端到端的检测，即输入一张图片直接输出检测结果；对于显微图像中多个小目标的白细胞检测十分准确，置信度也较高，说明了检测方法的准确性。Referring to Figure 3, which is the detection result of the trained DETR model, it can be seen that this method can realize end-to-end detection, that is, input a picture and output the detection result directly; it is very useful for the detection of white blood cells of multiple small targets in the microscopic image. Accurate, the confidence is also high, indicating the accuracy of the detection method.

参照图4为迁移学习与从头开始训练的精度对比结果，可以看出，本发明采用的基于迁移学习的DETR模型训练，收敛速度更快且检测精度更高；经过ColorJitter图像增强的训练能够取得更高的检测精度，验证了本专利所提出改进的有效性。Referring to FIG. 4 for the comparison results of the accuracy of migration learning and training from scratch, it can be seen that the DETR model training based on migration learning adopted in the present invention has faster convergence speed and higher detection accuracy; after the training of ColorJitter image enhancement, it is possible to obtain better results. The high detection accuracy verifies the effectiveness of the improvement proposed in this patent.

实施例2Example 2

本实施例提供一种存储介质，其上存储有计算机程序，该程序被执行时用于实现实施例1的方法。This embodiment provides a storage medium on which a computer program is stored, which is used to implement the method of Embodiment 1 when the program is executed.

实施例3Example 3

本实施例提供一种计算机设备，包括存储器、处理器以及存储在存储器上并可在处理器上运行的计算机程序，处理器执行计算机程序时实现实施例1的方法。This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and running on the processor, and the method of Embodiment 1 is implemented when the processor executes the computer program.

尽管本发明的实施方案已公开如上，但其并不仅仅限于说明书和实施方式中所列运用，它完全可以被适用于各种适合本发明的领域，对于熟悉本领域的人员而言，可容易地实现另外的修改，因此在不背离权利要求及等同范围所限定的一般概念下，本发明并不限于特定的细节。Although the embodiment of the present invention has been disclosed as above, it is not limited to the application listed in the description and the embodiment, and it can be applied to various fields suitable for the present invention. For those skilled in the art, it can be easily Therefore, the invention is not limited to the specific details without departing from the general concept defined by the appended claims and the scope of equivalents.