CN113160192B - Visual sense-based snow pressing vehicle appearance defect detection method and device under complex background - Google Patents

The invention provides a visual-based snow pressing vehicle appearance defect detection method under a complex background, which comprises the following steps of: s1: acquiring an image of a region to be detected of a target object by using a mobile terminal in a random shooting mode, and transmitting image information to an upper computer; s2: the upper computer receives the image information and detects the defects of the image; s3: and the upper computer stores the detection result and finally generates a detection report. The invention combines object-level defect detection and semi-supervised defect classification, and provides a semi-supervised defect detection method for realizing object-level automatic labeling under a complex background.

复杂背景下基于视觉的压雪车外观缺陷检测方法及装置Appearance defect detection method and device for snow-pumping vehicles based on vision under complex background

技术领域technical field

本发明涉及压雪车外观缺陷检测方法领域，具体地涉及一种基于视觉的能够在复杂背景下仅依靠少量标注即可实现对象级自动标注的压雪车外观缺陷检测技术。The invention relates to the field of detection methods for the appearance defects of snowbore vehicles, in particular to a vision-based snowbottle appearance defect detection technology that can realize object-level automatic labeling only by a small number of labels in a complex background.

背景技术Background technique

产品外观检测是生产制造过程中产品质量控制的关键步骤。基于视觉的外观缺陷检测方法凭借其快速、鲁棒的特性，在织物、金属、木材等多个领域得到广泛应用。基于视觉的缺陷检测核心是缺陷特征提取，根据特征提取方式的不同，可将检测方法分为以下两类：基于人工特征提取的数字图像处理方法和基于神经网络特征自提取的深度学习方法。Product appearance inspection is a key step in product quality control in the manufacturing process. Vision-based appearance defect detection methods have been widely used in many fields such as fabrics, metals, and woods due to their fast and robust properties. The core of vision-based defect detection is defect feature extraction. According to different feature extraction methods, detection methods can be divided into the following two categories: digital image processing methods based on artificial feature extraction and deep learning methods based on neural network feature self-extraction.

数字图像处理方法是根据缺陷的具体情况，手动设计缺陷特征提取算子。尽管该方法具有无监督、可解释性强的优点，但泛化性不足，对于复杂背景下的多目标检测，往往容易造成漏检以及误判。近年来，基于卷积神经网络的方法不断应用到缺陷检测任务上，取得了不错的效果。得益于CNN强大的特征提取能力，在有大量样本可供训练的通用缺陷数据集上，采用深度学习的检测方法可以达到较高的精度。然而实际的生产过程中，不仅缺陷样本数量少、种类多，且缺陷样本的对象级标注需要专业人员耗费大量的时间。The digital image processing method is to manually design the defect feature extraction operator according to the specific situation of the defect. Although this method has the advantages of unsupervised and strong interpretability, its generalization is insufficient, and for multi-target detection in complex backgrounds, it is easy to cause missed detection and misjudgment. In recent years, methods based on convolutional neural networks have been continuously applied to defect detection tasks and achieved good results. Thanks to the powerful feature extraction capability of CNN, the detection method using deep learning can achieve high accuracy on the general defect dataset with a large number of samples for training. However, in the actual production process, not only the number and variety of defective samples are small, but also the object-level annotation of defective samples requires professionals to spend a lot of time.

针对复杂背景下多目标多尺度缺陷，且对象级标注的缺陷样本数量较少的问题，一些学者采用了半监督学习策略。半监督学习介于监督学习和无监督学习之间，该方法能充分利用有标签数据训练初始分类器，并利用大量无标签数据优化模型，能有效解决有监督学习依赖大量标注的问题，同时相较于无监督学习，其检测精度更高。Aiming at the problem of multi-target and multi-scale defects in complex backgrounds and a small number of defect samples annotated at the object level, some scholars have adopted semi-supervised learning strategies. Semi-supervised learning is between supervised learning and unsupervised learning. This method can make full use of labeled data to train the initial classifier, and use a large amount of unlabeled data to optimize the model, which can effectively solve the problem of supervised learning relying on a large number of labels, while the same Compared with unsupervised learning, its detection accuracy is higher.

目前尚没有将半监督学习和复杂背景下多目标对象级自动标注相结合的压雪车外观缺陷检测方法。At present, there is no method for detecting the appearance defects of snow-pumping vehicles that combines semi-supervised learning and multi-object object-level automatic annotation in complex backgrounds.

发明内容SUMMARY OF THE INVENTION

为了解决上述现有技术的不足，本发明将复杂背景下缺陷对象级自动标注与半监督学习相结合，在标签样本较少的情况下，提高多目标缺陷定位与识别的精度。为此建立适当的模型，将复杂背景下多目标对象级标注和半监督学习相结合，共同检测压雪车外观缺陷，提高缺陷检测的准确度。In order to solve the above-mentioned shortcomings of the prior art, the present invention combines defect object-level automatic labeling and semi-supervised learning in complex backgrounds, and improves the accuracy of multi-target defect location and identification in the case of fewer label samples. To this end, an appropriate model is established, which combines multi-target object-level annotation and semi-supervised learning in complex backgrounds to jointly detect the appearance defects of snowbore vehicles and improve the accuracy of defect detection.

本专利的检测系统主要包含图像采集，缺陷检测，检测结果生成三个部分。其中，图像采集使用工业相机对压雪车外观随意拍摄，不限制距离以及角度，但需确保待检测区域占据图像像素80％以上。缺陷检测部分包含多维图像分割算法、半监督分类优化模型、融合算法三个模块，多维缺陷分割算法从多个维度提取可能存在缺陷的区域，提高多目标缺陷的检出率；基于半监督方法构建了缺陷识别网络的优化模型，采用伪标签机制防止因输入错误标签而导致训练精度下降，结合基于特征距离聚类的标记模型，提高了识别网络的分类精度；基于准确的分类结果，融合算法对带标签图像块进行筛选融合，提高定位精度。检测结果生成共有两部分内容，一是通过数据线直接回传，以单张图像为单位；另一种方式是终端保存，直至该检测任务全部完成，以单个任务为单位，并生成检测报告。The detection system of this patent mainly includes three parts: image acquisition, defect detection, and detection result generation. Among them, an industrial camera is used for image acquisition to randomly shoot the appearance of the snow-pressing vehicle, and the distance and angle are not limited, but it is necessary to ensure that the area to be detected occupies more than 80% of the image pixels. The defect detection part includes three modules: multi-dimensional image segmentation algorithm, semi-supervised classification optimization model, and fusion algorithm. The multi-dimensional defect segmentation algorithm extracts areas that may have defects from multiple dimensions to improve the detection rate of multi-target defects; it is constructed based on semi-supervised methods. The optimization model of the defect identification network is adopted, and the pseudo-label mechanism is used to prevent the training accuracy from being reduced due to the input of wrong labels. Combined with the labeling model based on feature distance clustering, the classification accuracy of the identification network is improved. Based on the accurate classification results, the fusion algorithm Labeled image blocks are filtered and fused to improve positioning accuracy. There are two parts to the generation of detection results. One is to send back directly through the data line, with a single image as the unit;

具体地，本发明提出一种复杂背景下基于视觉的压雪车外观缺陷检测方法，其包括以下步骤：Specifically, the present invention proposes a vision-based method for detecting the appearance defects of snowbore vehicles under a complex background, which includes the following steps:

S1：利用移动终端通过随意拍摄的方式采集目标物体待检测区域图像，将图像信息传输到上位机；S1: Use the mobile terminal to capture the image of the area to be detected of the target object by random shooting, and transmit the image information to the host computer;

S2：上位机接收图像信息，并对图像进行缺陷检测，其包括以下步骤：S2: The upper computer receives the image information and performs defect detection on the image, which includes the following steps:

S21：多维图像分割算法从梯度、阈值和区域三个方向构建三个维度的特征图，对特征图经过形态学处理、轮廓提取和面积筛选后，生成基于各维度特征图的缺陷候选区域，保留区域图像块及其位置信息，获取缺陷候选区域；S21: The multi-dimensional image segmentation algorithm constructs three-dimensional feature maps from three directions of gradient, threshold and region. After morphological processing, contour extraction and area screening of the feature maps, defect candidate regions based on the feature maps of each dimension are generated and reserved. Region image blocks and their location information to obtain defect candidate regions;

S22：将步骤S21中获取的缺陷候选区域分为有标签样本集D^L和无标签样本集D^U，有标签样本集D^L进一步分为有标签训练集和有标签测试集，共C+1类，其中C类为缺陷类别，1类为背景类，标签采用one-hot编码，各类样本所包含的样本数量均相等；无标签样本集D^U分为m个patch，每个patch包含相同数量的样本；S22: Divide the defect candidate regions obtained in step S21 into a labeled sample set ^DL and an unlabeled sample set ^DU , and the labeled sample set DL is further divided into a labeled training set and a labeled test set, with a total of ^C +1 Among them, class C is the defect class, class 1 is the background class, the label is one-hot encoding, and the number of samples contained in each type of sample is equal; the unlabeled sample set D ^U is divided into m patches, each patch contains the same number of samples;

S23：构建CNN模型，采用Resnet50模型作为初始分类模型，利用步骤S22中的有标签训练集对所述初始分类模型进行训练，使用adam算法更新初始分类模型中的参数，使用交叉熵损失作为损失函数，当模型收敛时，终止训练，计算测试组上的所述初始分类模型的预测误差；S23: Build a CNN model, use the Resnet50 model as the initial classification model, use the labeled training set in step S22 to train the initial classification model, use the adam algorithm to update the parameters in the initial classification model, and use the cross entropy loss as the loss function , when the model converges, the training is terminated, and the prediction error of the initial classification model on the test group is calculated;

S24：将S23中训练后的初始分类模型拆除softmax分类层，将特征提取层连接特征距离相似性度量网络，对S22中的无标签样本集的patch1进行初步类别预测；S24: Remove the softmax classification layer from the initial classification model trained in S23, connect the feature extraction layer to the feature distance similarity measurement network, and perform preliminary category prediction on patch1 of the unlabeled sample set in S22;

其中，特征距离相似性度量网络将有标签数据集D^L作为支持集，通过计算各类别样本的聚类中心，找到每个类的参考特征向量，计算候选样本与各类别聚类中心特征向量的欧式距离，选取最小值所在类别作为该候选样本的预测类别；Among them, the feature distance similarity measurement network will use the labeled data set ^DL as the support set, find the reference feature vector of each class by calculating the cluster center of each class of samples, and calculate the difference between the candidate sample and the feature vector of each class cluster center. Euclidean distance, select the category of the minimum value as the predicted category of the candidate sample;

假设

表示有k个类别，每个类别有n个样本的有标签数据集，其中

代表第i类有标签样本集合，

代表第i类缺陷第j张图像的矩阵化表示，

为其对应类别的one-hot编码，初始分类模型的特征提取层用映射函数f_θ表示，将输入图像的矩阵化表示

转换为特征向量

如下式Assumption

Represents a labeled dataset with k categories and n samples in each category, where

represents the set of labeled samples of the i-th class,

represents the matrix representation of the jth image of the i-th defect,

For the one-hot encoding of its corresponding category, the feature extraction layer of the initial classification model is represented by the mapping function f _θ , and the matrix representation of the input image

Convert to feature vector

as follows

即f_θ：

That is f _θ :

其中，H、W、C分别为图像矩阵的行数、列数及维度数，即图像的高度、宽度及通道数，D是特征向量的维度，为2048维，θ是分类网络的权重参数；Among them, H, W, C are the number of rows, columns and dimensions of the image matrix, namely the height, width and number of channels of the image, D is the dimension of the feature vector, which is 2048 dimensions, and θ is the weight parameter of the classification network;

特征距离相似性度量网络将D^L作为支持集，构建

的聚类中心，得到代表

的参考向量C_i，如下式：The feature distance similarity metric network uses ^DL as a support set to construct

The cluster centers of , get the representative

The reference vector C _i of , as follows:

得到各类别支持集参考向量C_i后，计算测试样本特征向量F与各参考向量的欧式距离d_i,将距离最小的类别作为该测试样本的预测类别y，d_i如下式：After obtaining the reference vector C _i of the support set of each category, calculate the Euclidean distance d _i between the feature vector F of the test sample and each reference vector, and take the category with the smallest distance as the predicted category y of the test sample, and d _i is as follows:

其中，F_j表示测试向量F第j维的值，

代表参考向量C_i第j维的值；Among them, F _j represents the value of the jth dimension of the test vector F,

represents the value of the jth dimension of the reference vector C _i ;

S25：将S24中预测的无标签样本集D^U中patch1及其预测的类别标签归类到伪标签样本集D^P中，从有标签训练集和伪标签样本集中联合采样，输入分类网络进行训练，采用伪标签机制同时考虑两个不同样本集的损失函数，并采用动态更新的方式，优化损失函数，使用adam算法更新分类模型中的参数，当模型收敛时，终止训练，计算测试组上的所述分类模型的预测误差，得到新的分类模型；S25: Classify ^patch1 and its predicted class labels in the unlabeled sample set D ^U predicted in S24 into the pseudo-labeled sample set DP, jointly sample from the labeled training set and the pseudo-labeled sample set, and input it into the classification network for training , using the pseudo-label mechanism to consider the loss functions of two different sample sets at the same time, and using the dynamic update method to optimize the loss function, use the adam algorithm to update the parameters in the classification model, when the model converges, terminate the training, and calculate the value of the test group. the prediction error of the classification model to obtain a new classification model;

S26：将步骤S25中训练后的新的分类模型拆除softmax分类层，将特征提取层连接特征距离相似性度量网络(具体方法与前述步骤S24中方法一致)，对步骤S22中的无标签样本集的patch1和patch2进行类别预测，更新patch1的预测标签；S26: Remove the softmax classification layer from the new classification model trained in step S25, connect the feature extraction layer to the feature distance similarity measurement network (the specific method is the same as the method in the aforementioned step S24), and analyze the unlabeled sample set in step S22. The patch1 and patch2 are used for category prediction, and the predicted label of patch1 is updated;

S27：重复步骤S25和S26，直到所有无标签样本全部参与训练，得到最终的优化后的分类模型作为最优分类模型，利用最优分类模型对步骤S1中得到的缺陷候选区域进行预测，得到各缺陷候选区域的类别；S27: Repeat steps S25 and S26 until all unlabeled samples participate in the training, obtain the final optimized classification model as the optimal classification model, and use the optimal classification model to predict the defect candidate area obtained in step S1, and obtain each Category of defect candidate area;

S28：融合层将结合各缺陷候选区域的类别及位置信息，形成各维度的带标签缺陷图像，再融合三个维度的图像，进行缺陷候选区域的筛选及回归，得到标注有最终缺陷区域的图像；S28: The fusion layer will combine the category and position information of each defect candidate area to form a labeled defect image of each dimension, and then fuse the three-dimensional images to screen and regress the defect candidate area, and obtain an image marked with the final defect area. ;

S3：上位机保存检测结果，最终生成检测报告，其包括以下步骤：S3: The upper computer saves the test results and finally generates a test report, which includes the following steps:

S31、上位机将标注有缺陷区域的单个图像保存并回传至移动终端，并且在移动终端显示；S31, the host computer saves and returns the single image marked with the defective area to the mobile terminal, and displays it on the mobile terminal;

S32、在该检测任务的所有图像缺陷区域标记完成后，上位机生成检测报告，报告内容包含各图像是否存在缺陷、缺陷类别、缺陷面积以及缺陷位置。S32. After all image defect areas of the inspection task are marked, the host computer generates an inspection report, and the report content includes whether each image has defects, defect type, defect area, and defect location.

优选地，步骤S1具体包括以下子步骤：Preferably, step S1 specifically includes the following sub-steps:

S11、利用带有显示屏的移动终端进行随意拍摄的图像采集；S11. Use a mobile terminal with a display screen to capture images taken at will;

S12、随意拍摄的方式包含自然的角度、环境和距离，图像中目标物体的像素占比达到整张图像的80％及以上，图像中可以夹杂地面背景；S12. The random shooting method includes natural angles, environments and distances. The pixel ratio of the target object in the image reaches 80% or more of the entire image, and the ground background can be mixed in the image;

S13、采用有线传输或者无线传输的方式进行数据传输。S13, using wired transmission or wireless transmission for data transmission.

优选地，步骤S25中，采样规则为1:1采样，伪标签机制所采用的损失函数为Preferably, in step S25, the sampling rule is 1:1 sampling, and the loss function adopted by the pseudo-label mechanism is

其中n，m分别为一个batch-size中有标签样本和伪标签样本的数量，y_i为人工标签，y′_i为伪标签，

为模型预测结果，∝是一个权重系数，用来度量伪标签的置信度；where n and m are the number of labeled samples and pseudo-labeled samples in a batch-size, respectively, _yi is the artificial label, y′ _i is the pseudo-label,

For the model prediction result, ∝ is a weight coefficient used to measure the confidence of the pseudo-label;

随着训练深入，不断有带有伪标签的样本加入到样本集里，模型得到优化，识别精度提高，同时∝开始呈线性增长，逐步扩大伪标签样本的损失值权重，∝的变化函数如下式：With the deepening of training, samples with pseudo-labels are continuously added to the sample set, the model is optimized, and the recognition accuracy is improved. At the same time, ∝ begins to increase linearly, and gradually expands the weight of the loss value of pseudo-label samples. The change function of ∝ is as follows :

其中，t为当前迭代次数，T₁，T₂是预设的迭代次数。Among them, t is the current iteration number, and T ₁ and T ₂ are preset iteration numbers.

优选地，S28中，筛选时首先在单个图像上采用矩形框画出代表缺陷的候选区域，并在矩形框上标注该候选区域的位置及类别；Preferably, in S28, during screening, a rectangular frame is used to draw a candidate area representing the defect on a single image, and the position and category of the candidate area are marked on the rectangular frame;

其次，删除背景类矩形框，将相同目标类别的矩形框，按照位置和大小进行筛选，得到代表缺陷的最优区域：Second, delete the background rectangular box, and filter the rectangular boxes of the same target category according to the position and size to obtain the optimal area representing the defect:

如矩形框为包含关系，则删去内部矩形框；如矩形框为重叠关系，则计算IOU,当IOU大于某个阈值I₁且两个图像为同一类时，判断为多维度图像分割处理是的是同一个缺陷，对两个矩形框进行合并；当两个图像块的IOU大于阈值I₂，且图像标签为不同类别，则为不同目标类别的矩形框重叠，按照两者的置信度得分高低进行筛选，保留得分高的类别的矩形框作为最优缺陷区域。If the rectangular frame is in an inclusive relationship, delete the inner rectangular frame; if the rectangular frame is in an overlapping relationship, calculate the IOU. When the IOU is greater than a certain threshold I ₁ and the two images are of the same class, it is judged that the multi-dimensional image segmentation process is If the defect is the same defect, the two rectangular boxes are merged; when the IOU of the two image blocks is greater than the threshold I ₂ and the image labels are of different categories, the rectangular boxes of different target categories overlap, according to the confidence score of the two High and low are screened, and the rectangular box of the high-scoring category is reserved as the optimal defect area.

优选地，步骤S21中三个维度的特征提取包括以下子步骤：Preferably, the three-dimensional feature extraction in step S21 includes the following sub-steps:

S211、梯度维的特征提取：首先将RGB图像转为灰度图，再使用结合直方图均衡的双重滤波算法，双重滤波算法第一步采用小尺度的高斯卷积核对灰度图像进行低通滤波，去除图像中的高斯噪声；第二步采用自适应直方图均衡算法，将图像划分为8×8的子区域，在每个子区域采用累积分布函数(式4)使得像素值均匀分布，采用双线性插值对每个子区域进行拼接，增强图像信息；第三步采用较大尺度的高斯卷积核进行高斯滤波去除因直方图均衡放大了的噪声。S211. Gradient dimension feature extraction: first convert the RGB image into a grayscale image, and then use a double filtering algorithm combined with histogram equalization. The first step of the double filtering algorithm uses a small-scale Gaussian convolution kernel to perform low-pass filtering on the grayscale image. , remove the Gaussian noise in the image; the second step adopts the adaptive histogram equalization algorithm to divide the image into 8 × 8 sub-regions, and the cumulative distribution function (Equation 4) is used in each sub-region to make the pixel values evenly distributed. Linear interpolation splices each sub-region to enhance image information; the third step uses a larger-scale Gaussian convolution kernel to perform Gaussian filtering to remove the noise amplified by histogram equalization.

其中n是图像中像素的总和，n_j是当前灰度级的像素个数，L是图像中可能的灰度级总数；where n is the sum of the pixels in the image, n _j is the number of pixels in the current gray level, and L is the total number of possible gray levels in the image;

在边缘检测阶段，分别采用x轴方向和y轴方向的Sobel算子计算图像水平梯度G_x和垂直梯度G_y，当前点的梯度为G_x和G_y的L2范数。将图像每个点的灰度强度转化为梯度强度，对梯度图进行非极大值抑制，去除非边界上的点，最后采用双阈值分割对梯度图进行二值化。本发明首先利用最大类间方差法(OTSU)计算自适应阈值，得到一个使得前景、背景差异最大的阈值T_OTSU。为了减少图像中噪声对边缘的干扰，本发明将T_OTSU设定为低阈值T_L，根据经验将高阈值T_H设定为低阈值的三倍，实现双阈值的自适应，提高灰度-边缘维度特征提取的泛化性；In the edge detection stage, the Sobel operator in the x-axis direction and the y-axis direction is used to calculate the image horizontal gradient G _x and vertical gradient G _y , and the gradient of the current point is the L2 norm of G _x and G _y . The gray intensity of each point of the image is converted into gradient intensity, the gradient map is suppressed by non-maximum value, the points on the non-boundary are removed, and finally the gradient map is binarized by double-threshold segmentation. The present invention first calculates the adaptive threshold by using the maximum inter-class variance method (OTSU), and obtains a threshold T _OTSU which maximizes the difference between the foreground and the background. In order to reduce the interference of noise on the edge in the image, the present invention sets T _OTSU as the low threshold _TL , and sets the high threshold _TH as three times the low threshold according to experience, so as to realize the self-adaptation of double thresholds and improve the gray-scale- Generalization of edge dimension feature extraction;

S212、阈值维的特征提取：当输入图像为缺少颜色信息的灰度图时，难以采用S分量的阈值分割方法。因此本文针对灰度图像的阈值信息，采用OTSU法求得最佳阈值进行分割；S212 , feature extraction of threshold dimension: when the input image is a grayscale image lacking color information, it is difficult to adopt the threshold segmentation method of the S component. Therefore, according to the threshold information of grayscale images, the OTSU method is used to obtain the best threshold for segmentation;

当输入图像为RGB格式时，本文基于S分量采用固定阈值的方式，设定阈值TS，检测部分呈现彩色的缺陷，同时采用混合滤波的方式进行预处理。首先对S分量图像进行高斯滤波，然后在对其进行均值滤波，有效消除高斯噪声和椒盐噪声；When the input image is in RGB format, this paper adopts a fixed threshold method based on the S component, and sets the threshold value TS to detect some color defects, and at the same time adopts a mixed filtering method for preprocessing. First, Gaussian filtering is performed on the S-component image, and then the mean filtering is performed on it, which effectively eliminates Gaussian noise and salt and pepper noise;

其中h(x,y)为阈值分割后的二值化图像，s(x,y)为s分量原始图像。Among them, h(x, y) is the binarized image after threshold segmentation, and s(x, y) is the original image of s component.

S213、区域维的特征提取：将图像分为3×3的子图像，选取四个顶点所在子图像的中心点作为种子点，由该点像素为基准，进行区域填充，采用泛洪填充算法，基于八邻域像素填充法从一个点开始将附近像素点填充成新的颜色，直到封闭区域内的所有像素点都被填充新颜色为止。S213. Feature extraction of regional dimension: Divide the image into 3×3 sub-images, select the center point of the sub-image where the four vertices are located as the seed point, and use the pixels of this point as the benchmark to fill the region, using the flood filling algorithm, Based on the eight-neighbor pixel filling method, the nearby pixels are filled with a new color from a point until all the pixels in the enclosed area are filled with the new color.

本发明还提供一种用于所述的复杂背景下基于视觉的压雪车外观缺陷检测方法的检测装置，所述检测装置包括移动终端以及上位机，所述移动终端用于进行图像采集，所述上位机用于进行缺陷检测及生成检测报告；所述移动终端与所述上位机之间通讯连接；The present invention also provides a detection device for the method for detecting the appearance defects of snow-pumping vehicles based on vision under the complex background, the detection device includes a mobile terminal and a host computer, the mobile terminal is used for image acquisition, and the The upper computer is used for defect detection and generation of a detection report; the communication connection between the mobile terminal and the upper computer;

所述上位机包括缺陷检测模块、缺陷标记模块以及检测报告生成模块，所述缺陷检测模块包括多维图像分割算法单元、半监督分类优化单元以及融合算法单元；The host computer includes a defect detection module, a defect marking module and a detection report generation module, and the defect detection module includes a multi-dimensional image segmentation algorithm unit, a semi-supervised classification optimization unit and a fusion algorithm unit;

所述多维图像分割算法单元从多个维度提取可能存在缺陷的区域，提高多目标缺陷的检出率；所述半监督分类优化单元基于半监督方法构建缺陷识别网络的优化模型，得到准确的分类结果；所述融合算法对带标签图像块进行筛选融合，提高定位精度；The multi-dimensional image segmentation algorithm unit extracts areas that may have defects from multiple dimensions to improve the detection rate of multi-target defects; the semi-supervised classification optimization unit builds an optimization model of the defect identification network based on the semi-supervised method to obtain accurate classification Results: The fusion algorithm screened and fused the labeled image blocks to improve the positioning accuracy;

所述缺陷标记模块用于利用矩形框进行缺陷标记并将标记后的单个图像返回至移动终端，所述检测报告生成模块用于生成检测报告。The defect marking module is used for marking defects with a rectangular frame and returning the marked single image to the mobile terminal, and the detection report generating module is used for generating a detection report.

优选地，所述移动终端为带有显示屏的便携式工业相机。Preferably, the mobile terminal is a portable industrial camera with a display screen.

优选地，所述移动终端接收所述上位机回传的数据并以单张图像为单位进行显示，所述上位机的检测报告生成模块保存检测任务中所有图像，直至该检测任务全部完成，以单个任务为单位，生成检测报告。Preferably, the mobile terminal receives the data returned by the host computer and displays it in a single image, and the detection report generation module of the host computer saves all the images in the detection task until the detection task is completed, so as to A single task is used as a unit to generate a detection report.

与现有的技术相比，本发明具有以下有益效果：Compared with the prior art, the present invention has the following beneficial effects:

(1)提出了一种复杂背景下基于对象级自动标注的半监督缺陷检测方法。该方法首先利用图像纹理、颜色等信息，从多个维度构建多维图像分割算法，得到缺陷候选区域。然后利用半监督学习提高少量标注情况下的分类精度，并且对模型的标签预测方式、预测评价指标、标签更新机制进行了改进。最后对带有类别标签的候选区融合，实现多目标对象级缺陷自动检测。本专利中，多维图像分割算法能够在少量有标签样本的指导下，实现多目标缺陷的精准定位，仅用少量的候选框，即使在严格的IOU下依然能得到不错的召回率；改进的半监督优化模型能够在初始分类器精度有限的情况下，能够加快模型收敛，提高类别预测精度。(1) A semi-supervised defect detection method based on object-level automatic annotation in complex background is proposed. The method firstly uses information such as image texture and color to construct a multi-dimensional image segmentation algorithm from multiple dimensions to obtain defect candidate regions. Then, semi-supervised learning is used to improve the classification accuracy in the case of a small number of labels, and the label prediction method, prediction evaluation index, and label update mechanism of the model are improved. Finally, the candidate regions with category labels are fused to realize automatic detection of multi-target object-level defects. In this patent, the multi-dimensional image segmentation algorithm can achieve accurate positioning of multi-target defects under the guidance of a small number of labeled samples. With only a small number of candidate frames, a good recall rate can still be obtained even under strict IOU; The supervised optimization model can accelerate the model convergence and improve the category prediction accuracy when the initial classifier accuracy is limited.

(2)本发明将对象级缺陷检测与半监督缺陷分类相结合，提出了一种在复杂背景下实现对象级自动标注的半监督缺陷检测方法，该方法能够在少样本训练的情况下，识别多目标、多类别、小尺度的缺陷；(2) The present invention combines object-level defect detection and semi-supervised defect classification, and proposes a semi-supervised defect detection method that realizes object-level automatic labeling in complex backgrounds. Multi-objective, multi-category, small-scale defects;

(3)本发明提出一种改进的半监督训练和标记方法，优化了分类模型，提高了缺陷识别的精度。(3) The present invention proposes an improved semi-supervised training and labeling method, which optimizes the classification model and improves the accuracy of defect identification.

附图说明Description of drawings

图1为本发明的工作流程示意图；Fig. 1 is the workflow schematic diagram of the present invention;

图2为本发明的流程示意框图；Fig. 2 is a schematic flow diagram block diagram of the present invention;

图3为本发明的多目标图像分割流程示意图；Fig. 3 is the multi-target image segmentation flow schematic diagram of the present invention;

图4为本发明的半监督分类优化流程示意图；Fig. 4 is the semi-supervised classification optimization flow schematic diagram of the present invention;

图5为本发明的边框回归算法流程示意图；5 is a schematic flowchart of a frame regression algorithm of the present invention;

图6为本发明的多维图像分割对比示意图；Fig. 6 is the multi-dimensional image segmentation contrast schematic diagram of the present invention;

图7为本发明的结构示意框图。FIG. 7 is a schematic block diagram of the structure of the present invention.

具体实施方式Detailed ways

以下将参考附图详细说明本发明的示例性实施例、特征和方面。附图中相同的附图标记表示功能相同或相似的元件。Exemplary embodiments, features and aspects of the present invention will be described in detail below with reference to the accompanying drawings. The same reference numbers in the figures denote elements that have the same or similar functions.

本发明提供一种基于视觉的压雪车外观缺陷检测技术。下面结合图1以及图2所示的方法流程，以压雪车外观缺陷为例，对本发明的建模方法做进一步描述，其包括以下步骤：The present invention provides a vision-based detection technology for the appearance defects of snow plows. Below in conjunction with the method flow shown in Fig. 1 and Fig. 2, taking the appearance defect of snow plow vehicle as an example, the modeling method of the present invention is further described, which comprises the following steps:

步骤一：利用移动终端通过随意拍摄的方式采集目标物体待检测区域图像，将图像信息传输到上位机。Step 1: Use the mobile terminal to capture the image of the area to be detected of the target object by random shooting, and transmit the image information to the upper computer.

首先进行图像采集，采用移动终端(如单反相机、工业相机、手机或平板设备等)在与压雪车保持正常距离下，采集压雪车外观图像。利用无线传输，将图像传输到上位机(如台式电脑、笔记本电脑、工控机或服务器等)，进行数据集的制作。First, image acquisition is performed, and a mobile terminal (such as a single-lens reflex camera, an industrial camera, a mobile phone or a tablet device, etc.) is used to collect the appearance image of the snowmobile while maintaining a normal distance from the snowmobile. Using wireless transmission, the image is transmitted to the host computer (such as desktop computer, notebook computer, industrial computer or server, etc.), and the data set is produced.

首先将无缺陷的样本去除，仅保留有缺陷的样本制作缺陷数据集。压雪车外观缺陷数据集共有4种典型的外观缺陷，每类缺陷300张图像，图像中缺陷像素占总像素比值不超过一半，每张图像上并非只有一个缺陷，存在多个缺陷以及多类缺陷，并且受光照和对比度等因素影响，缺陷类间差异大，类内差异小。数据集带有对象级标注，包含缺陷的类别与所在位置。First, the non-defective samples are removed, and only the defective samples are retained to make a defect data set. There are 4 typical appearance defects in the snow car appearance defect data set. Each type of defect contains 300 images. The ratio of defective pixels in the image to the total pixels does not exceed half. There is not only one defect in each image, but there are multiple defects and multiple types of defects. Defects, and affected by factors such as illumination and contrast, the differences between defect classes are large, and the differences within classes are small. The dataset has object-level annotations, including the category and location of defects.

数据集的分割按照7:3进行设置，即每类210张图像用于模型训练，90张图像用于性能评估。为了模拟有标签样本较少的情况下，评估算法的检测性能，本文将训练集中各类别部分样本的标签抹去，形成10％、25％、50％的有标签样本比例。The segmentation of the dataset is set according to 7:3, that is, 210 images per class are used for model training and 90 images are used for performance evaluation. In order to simulate the detection performance of the algorithm when there are few labeled samples, this paper erases the labels of some samples of each category in the training set to form 10%, 25%, and 50% of labeled samples.

步骤二：上位机接收图像信息，通过本发明所提出的复杂背景下基于对象级自动标注的半监督缺陷检测方法进行缺陷检测。如图3所示，多目标图像分割过程总首先将图片进行缩小，减少后续处理时间；之后采用双边滤波和中值滤波的混合滤波算法对图像进行平滑操作。本专利为了在复杂背景的干扰下，实现多目标缺陷的检测，采用多维分割的方式，在HSV颜色维度、灰度阈值维、边缘纹理维度分别采用不同的图像分割算法；分割完成后，在各维度分别进行形态学膨胀，避免遗漏小目标缺陷，通过轮廓提取、面积筛选得到包含缺陷的区域；最后将ROIs裁剪出来并保存。其具体包括以下各个步骤：Step 2: The upper computer receives the image information, and performs defect detection through the semi-supervised defect detection method based on object-level automatic annotation under the complex background proposed by the present invention. As shown in Figure 3, the multi-target image segmentation process always firstly reduces the image to reduce the subsequent processing time; then uses a hybrid filtering algorithm of bilateral filtering and median filtering to smooth the image. In order to realize the detection of multi-target defects under the interference of complex background, this patent adopts multi-dimensional segmentation, and adopts different image segmentation algorithms in HSV color dimension, gray threshold dimension and edge texture dimension. The dimensions are morphologically expanded to avoid missing small target defects, and regions containing defects are obtained through contour extraction and area screening; finally, the ROIs are cropped out and saved. It specifically includes the following steps:

步骤21：多维图像分割Step 21: Multidimensional Image Segmentation

对于每张输入的图像，都从梯度、阈值、区域三个角度出发，形成三个维度的特征图。根据不同维度的特点，设计相应的检测算子，提取不同维度特征，利用特征进行分割，经过优化、筛选，得到缺陷候选区域。其中，各个维度的特征提取算子的设计，各算子的阈值是一个重要的参数。在本例中，多维图像分割算法，在各个维度上采用的方法分别如下：For each input image, a three-dimensional feature map is formed from the perspectives of gradient, threshold, and region. According to the characteristics of different dimensions, the corresponding detection operators are designed to extract the features of different dimensions, use the features to segment, and obtain defect candidate regions after optimization and screening. Among them, the design of feature extraction operators of each dimension, the threshold of each operator is an important parameter. In this example, the multi-dimensional image segmentation algorithm adopts the following methods in each dimension:

(1)梯度维(1) Gradient dimension

首先将RGB图像转为灰度图，再使用结合直方图均衡的双重滤波算法。双重滤波算法第一步采用小尺度的高斯卷积核对灰度图像进行低通滤波，去除图像中的高斯噪声；第二步采用自适应直方图均衡算法，将图像划分为8×8的子区域，在每个子区域采用累积分布函数(式4)使得像素值均匀分布，采用双线性插值对每个子区域进行拼接，增强图像信息；第三步采用较大尺度的高斯卷积核进行高斯滤波去除因直方图均衡放大了的噪声。First convert the RGB image to grayscale, and then use a double filtering algorithm combined with histogram equalization. The first step of the double filtering algorithm uses a small-scale Gaussian convolution to perform low-pass filtering on the grayscale image to remove the Gaussian noise in the image; the second step uses an adaptive histogram equalization algorithm to divide the image into 8 × 8 sub-regions , the cumulative distribution function (Equation 4) is used in each sub-region to make the pixel values evenly distributed, and bilinear interpolation is used to splicing each sub-region to enhance image information; the third step uses a larger-scale Gaussian convolution kernel for Gaussian filtering Removes noise amplified by histogram equalization.

其中n是图像中像素的总和，n_j是当前灰度级的像素个数，L是图像中可能的灰度级总数。where n is the sum of the pixels in the image, n _j is the number of pixels in the current gray level, and L is the total number of possible gray levels in the image.

在边缘检测阶段，分别采用x轴方向和y轴方向的Sobel算子计算图像水平梯度G_x和垂直梯度G_y，当前点的梯度为G_x和G_y的L2范数。将图像每个点的灰度强度转化为梯度强度，对梯度图进行非极大值抑制，去除非边界上的点，最后采用双阈值分割对梯度图进行二值化。本发明首先利用最大类间方差法(OTSU)计算自适应阈值，得到一个使得前景、背景差异最大的阈值T_OTSU。为了减少图像中噪声对边缘的干扰，本发明将T_OTSU设定为低阈值T_L，根据经验将高阈值T_H设定为低阈值的三倍，实现双阈值的自适应，提高灰度-边缘维度特征提取的泛化性。In the edge detection stage, the Sobel operator in the x-axis direction and the y-axis direction is used to calculate the image horizontal gradient G _x and vertical gradient G _y , and the gradient of the current point is the L2 norm of G _x and G _y . The gray intensity of each point of the image is converted into gradient intensity, the gradient map is suppressed by non-maximum value, the points on the non-boundary are removed, and finally the gradient map is binarized by double-threshold segmentation. The present invention first calculates the adaptive threshold by using the maximum inter-class variance method (OTSU), and obtains a threshold T _OTSU which maximizes the difference between the foreground and the background. In order to reduce the interference of noise on the edge in the image, the present invention sets T _OTSU as the low threshold _TL , and sets the high threshold _TH as three times the low threshold according to experience, so as to realize the self-adaptation of double thresholds and improve the gray-scale- Generalization of edge dimension feature extraction.

(2)阈值维(2) Threshold dimension

当输入图像为缺少颜色信息的灰度图时，难以采用S分量的阈值分割方法。因此本文针对灰度图像的阈值信息，采用OTSU法求得最佳阈值进行分割。When the input image is a grayscale image lacking color information, it is difficult to use the S-component threshold segmentation method. Therefore, this paper uses the OTSU method to obtain the best threshold for segmentation based on the threshold information of grayscale images.

当输入图像为RGB格式时，本文基于S分量采用固定阈值的方式，设定阈值TS，检测部分呈现彩色的缺陷，同时采用混合滤波的方式进行预处理。首先对S分量图像进行高斯滤波，然后在对其进行均值滤波，有效消除高斯噪声和椒盐噪声。When the input image is in RGB format, this paper adopts a fixed threshold method based on the S component, and sets the threshold value TS to detect some color defects, and at the same time adopts a mixed filtering method for preprocessing. First, Gaussian filtering is performed on the S-component image, and then the mean filtering is performed on it, which effectively eliminates Gaussian noise and salt and pepper noise.

其中h(x,y)为阈值分割后的二值化图像，s(x,y)为s分量原始图像。Among them, h(x, y) is the binarized image after threshold segmentation, and s(x, y) is the original image of s component.

(3)区域维(3) Regional dimension

将图像分为3×3的子图像，选取四个顶点所在子图像的中心点作为种子点，由该点像素为基准，进行区域填充。采用泛洪填充算法，基于八邻域像素填充法从一个点开始将附近像素点填充成新的颜色，直到封闭区域内的所有像素点都被填充新颜色为止。The image is divided into 3×3 sub-images, and the center point of the sub-image where the four vertices are located is selected as the seed point, and the region is filled with the pixel of this point as the benchmark. Using the flood filling algorithm, based on the eight-neighbor pixel filling method, the nearby pixels are filled with a new color from a point until all the pixels in the enclosed area are filled with the new color.

本例通过实验验证不同的IOU阈值下几种候选区域提取算法的召回率，IOU表示预测框与标注框的交集和并集之比。根据实验结果，本发明的多维图像分割算法取得最优的结果。In this example, the recall rate of several candidate region extraction algorithms under different IOU thresholds is verified through experiments, and IOU represents the ratio of the intersection and union of the prediction frame and the annotation frame. According to the experimental results, the multi-dimensional image segmentation algorithm of the present invention achieves the best results.

从图6可以看出，在IOU大于0.5时，基于边缘的分割法、基于阈值的分割法和滑窗法三种方法的召回率都已经低于0.3。同时，采用50％标注训练的RPN网络，其召回率也在IOU阈值大于0.5后快速下降。相对应的，本发明所提出的基于多维度特征的多维分割算法在IOU大于0.5后，下降平缓，在IOU等于0.7时尚有0.45的召回率。该数据显示了本算法在复杂背景下，查全率和缺陷区域提取精度相较于其他几个算法是最高的。事实上，由于缺陷以及背景的纹理复杂性，在单一维度上的特征并不能完全的、显著的代表所有缺陷。因此多维度图像分割算法综合考虑颜色、纹理等信息，使得在某一维度上，得到最精准的缺陷区域提取。It can be seen from Figure 6 that when the IOU is greater than 0.5, the recall rates of the edge-based segmentation method, the threshold-based segmentation method and the sliding window method are all lower than 0.3. At the same time, the recall rate of the RPN network trained with 50% annotation also drops rapidly after the IOU threshold is greater than 0.5. Correspondingly, the multi-dimensional segmentation algorithm based on multi-dimensional features proposed by the present invention decreases gently when the IOU is greater than 0.5, and has a recall rate of 0.45 when the IOU is equal to 0.7. The data shows that this algorithm has the highest recall rate and defect area extraction accuracy compared with other algorithms under complex background. In fact, due to defects and the texture complexity of the background, features in a single dimension cannot fully and significantly represent all defects. Therefore, the multi-dimensional image segmentation algorithm comprehensively considers information such as color and texture, so that the most accurate defect area extraction can be obtained in a certain dimension.

另外，考虑到虚警框对于分类网络的影响，我们计算了几种方法在每张图像上的平均候选框数，以此来评价不同候选区域提取器的效率。结果如表3所示。In addition, considering the impact of false alarm boxes on the classification network, we calculated the average number of candidate boxes per image for several methods to evaluate the efficiency of different candidate region extractors. The results are shown in Table 3.

表1单张图像各方法平均候选区域数Table 1 Average number of candidate regions for each method in a single image

Edge-basedEdge-based Threshold-basedThreshold-based Sliding-windowsSliding-windows RPNRPN OursOurs AveragenumberAveragenumber 33 77 5050 5050 1515

表1展示了不同方法的平均提取框的数量，其中滑窗法和RPN都是基于50％标注量的Top-50个提取框。结合图2以及表1的数据可以看出，在IOU小于0.5时，召回率与本发明方法相近的RPN算法单张图像需要50个提取框，平均候选框少的基于边缘的分割法和基于阈值的分割法召回率远低于本发明方法。综合来看，本发明提出的多维图像分割算法在比较严格的IOU阈值下，依旧能保持较高召回率，并且，所采用的候选框也较少，这有助于后续的图像分类过程受到较小的类别不平衡影响。Table 1 shows the average number of extraction boxes for different methods, in which both the sliding window method and the RPN are based on the Top-50 extraction boxes with 50% annotation volume. Combining with the data in Figure 2 and Table 1, it can be seen that when the IOU is less than 0.5, the RPN algorithm with a recall rate similar to the method of the present invention requires 50 extraction frames for a single image, and the edge-based segmentation method with fewer average candidate frames and the threshold-based segmentation method. The recall rate of the segmentation method is much lower than that of the method of the present invention. On the whole, the multi-dimensional image segmentation algorithm proposed in the present invention can still maintain a high recall rate under a relatively strict IOU threshold, and also uses fewer candidate frames, which is helpful for the subsequent image classification process to be more vulnerable. Small class imbalance effects.

步骤22，进行半监督分类优化，其步骤如图4所示，Step 22, carry out semi-supervised classification optimization, the steps are shown in Figure 4,

半监督优化模型将多维图像分割获取的缺陷候选区域分为有标签样本集D^L、无标签样本集D^U，有标签样本集又分为有标签训练集、有标签测试集，共5类，其中4类为缺陷类别，1为背景类，标签采用one-hot编码。无标签样本集分为m个patch，每个patch包含相同数量的样本，特别的，各类所包含的数量都相等。The semi-supervised optimization model divides the defect candidate regions obtained by multi-dimensional image segmentation into a labeled sample set D ^L and an unlabeled sample set D ^U , and the labeled sample set is further divided into a labeled training set and a labeled test set, with a total of 5 categories. Among them, 4 categories are defect categories, 1 is background category, and the labels are encoded by one-hot. The unlabeled sample set is divided into m patches, and each patch contains the same number of samples. In particular, the number of each type is equal.

然后构建CNN模型，采用成熟的Resnet50模型作为初始分类器。利用有标签训练集对所述分类模型进行训练，使用adam算法更新resnet模型中的参数，使用交叉熵损失作为损失函数，当模型收敛时，终止训练。计算测试组上的所述分类模型的预测误差。Then a CNN model is constructed, and the mature Resnet50 model is used as the initial classifier. The classification model is trained using the labeled training set, the parameters in the resnet model are updated using the adam algorithm, the cross-entropy loss is used as the loss function, and the training is terminated when the model converges. Calculate the prediction error of the classification model on the test set.

将训练后的分类模型拆除softmax分类层，将特征提取层连接特征距离相似性度量网络，对无标签样本集的patch1进行初步类别预测；其中，特征距离相似性度量网络将有标签数据集D^L作为支持集，通过计算各类别样本的聚类中心，找到每个类的参考特征向量，计算候选样本与各类别聚类中心特征向量的欧式距离，选取最小值所在类别作为该候选样本的预测类别；The softmax classification layer is removed from the trained classification model, the feature extraction layer is connected to the feature distance similarity measurement network, and the patch1 of the unlabeled sample set is preliminarily predicted. Among them, the feature distance similarity measurement network will have a label data set D ^L As a support set, by calculating the cluster center of each class of samples, find the reference feature vector of each class, calculate the Euclidean distance between the candidate sample and the feature vector of the cluster center of each class, and select the class with the minimum value as the predicted class of the candidate sample ;

假设

表示有k个类别，每个类别有n个样本的有标签数据集，其中

代表第i类有标签样本集合，

代表第i类缺陷第j张图像的矩阵化表示，

为其对应类别的one-hot编码。分类网络Resnet50的特征提取层可以用映射函数f_θ表示，它将输入图像的矩阵化表示

转换为特征向量

如下式Assumption

Represents a labeled dataset with k categories and n samples in each category, where

represents the set of labeled samples of the i-th class,

represents the matrix representation of the jth image of the i-th defect,

One-hot encoding for its corresponding category. The feature extraction layer of the classification network Resnet50 can be represented by a mapping function f _θ , which represents the matrixed representation of the input image

Convert to feature vector

as follows

即f_θ：

其中，H、W、C分别为图像矩阵的行数、列数、维度数，即图像的高度、宽度、通道数。D是特征向量的维度，为2048维。θ是分类网络的权重参数；That is f _θ :

Among them, H, W, C are the number of rows, columns, and dimensions of the image matrix, that is, the height, width, and number of channels of the image. D is the dimension of the feature vector, which is 2048 dimensions. θ is the weight parameter of the classification network;

特征距离相似性度量网络将D^L作为支持集，构建

的聚类中心，得到代表

的参考向量C_i，如下式：The feature distance similarity metric network uses ^DL as a support set to construct

The cluster centers of , get the representative

The reference vector C _i of , as follows:

得到各类别支持集参考向量C_i后，计算测试样本特征向量F与各参考向量的欧式距离d_i,将距离最小的类别作为该测试样本的预测类别y，d_i如下式：After obtaining the reference vector C _i of the support set of each category, calculate the Euclidean distance d _i between the feature vector F of the test sample and each reference vector, and take the category with the smallest distance as the predicted category y of the test sample, and d _i is as follows:

其中，F_j表示测试向量F第j维的值，

代表参考向量C_i第j维的值。Among them, F _j represents the value of the jth dimension of the test vector F,

represents the value of the jth dimension of the reference vector C _i .

预测的无标签样本集D^U中patch1及其预测的类别标签归类到伪标签样本集D^P中，从有标签训练集和伪标签样本集中联合采样，输入分类网络进行训练，采用伪标签机制同时考虑两个不同样本集的损失函数，并采用动态更新的方式，优化损失函数，使用adam算法更新分类模型中的参数，当模型收敛时，终止训练。计算测试组上的所述分类模型的预测误差。The patch1 and its predicted class labels in the predicted unlabeled sample set D ^U are classified into the pseudo-labeled sample set D ^P , jointly sampled from the labeled training set and the pseudo-labeled sample set, and input into the classification network for training, using the pseudo-label mechanism At the same time, the loss functions of two different sample sets are considered, and the loss function is optimized by a dynamic update method, and the parameters in the classification model are updated by the adam algorithm. When the model converges, the training is terminated. Calculate the prediction error of the classification model on the test set.

将训练后的新的分类模型拆除softmax分类层，将特征提取层连接特征距离相似性度量网络，对无标签样本集的patch1、2进行类别预测，更新patch1的预测标签。其中，将特征提取层连接特征距离相似性度量网络的步骤与前一步骤中的相似性度量网络相同。The softmax classification layer is removed from the new classification model after training, the feature extraction layer is connected to the feature distance similarity measurement network, the category prediction is performed on patches 1 and 2 of the unlabeled sample set, and the predicted label of patch 1 is updated. The step of connecting the feature extraction layer to the feature distance similarity measurement network is the same as the similarity measurement network in the previous step.

重复上述两步骤，直到所有无标签样本全部参与训练，得到最终的优化后的分类模型。利用最优分类模型对缺陷候选区域进行预测，得到各缺陷候选区域的类别。The above two steps are repeated until all unlabeled samples participate in the training, and the final optimized classification model is obtained. The optimal classification model is used to predict the defect candidate area, and the category of each defect candidate area is obtained.

本例将实验三个改进的有效性，分别是：基于特征距离的相似性度量、基于伪标签机制的评价机制和标签更新机制。表3展示了各个改进在50％标注量的训练集上达到指定精度的批次，以及在测试集上的准确率。This example will test the effectiveness of three improvements, namely: similarity measure based on feature distance, evaluation mechanism based on pseudo-label mechanism and label update mechanism. Table 3 shows the batches where each improvement achieves the specified accuracy on the 50% annotated training set, and the accuracy on the test set.

如表2结果所示，同时应用了三个改进的方法具有最高的准确率和最快的模型收敛速度，该结果表明三个改进对于提高模型精度，加快模型收敛速度是有效的。此外单独从标签评价机制和特征距离相似性度量两个改进对于模型的影响来看，采用了伪标签机制的模型在准确率上相比没有改进的自训练方式有8.93％的显著提高，采用基于特征距离的相似性度量的方法的收敛速度也比自训练方法快了32％。这两组结果表明在初始训练集较小，初始分类器精度较低的情况下，合适的标签预测策略和训练时伪标签的评价策略对于模型收敛和最终精度至关重要。As shown in the results in Table 2, the three improved methods applied at the same time have the highest accuracy and the fastest model convergence speed. The results show that the three improvements are effective for improving the model accuracy and speeding up the model convergence speed. In addition, from the impact of the two improvements of the label evaluation mechanism and the feature distance similarity measure on the model alone, the accuracy of the model using the pseudo-label mechanism is 8.93% higher than that of the self-training method without improvement. The method of similarity measure of feature distance also converges 32% faster than the self-training method. These two sets of results show that in the case where the initial training set is small and the initial classifier accuracy is low, a suitable label prediction strategy and an evaluation strategy for pseudo-labels during training are critical for model convergence and final accuracy.

表2半监督优化模型各改进部分的有效性Table 2 Effectiveness of each improved part of the semi-supervised optimization model

Label-updateLabel-update Pseudo-labelPseudo-label Feature-distenceFeature-distence EpochEpoch Accuracy％Accuracy% 330330 83.7483.74 √√ 224224 89.2289.22 √√ 177177 92.6792.67 √√ √√ 112112 96.8696.86 √√ √√ √√ 9898 97.0297.02

步骤23Step 23

融合层fusion layer

融合层将结合各缺陷候选区域的类别及位置信息，形成各维度的带标签缺陷图像，再融合三个维度的图像，进行缺陷候选区域的筛选及回归。其规则为：A.同类缺陷包含关系：当一个图像块的每个边都在另一个图像块内部，且两张图像为同一类缺陷，则判断为同类缺陷包含关系，将属于内部的图像块删去，保留大图。B.同类缺陷部分重叠关系：同类缺陷重叠时，需要考虑重叠部分的比例，通过计算IOU，当IOU大于某个阈值I₁且两个图像为同一类时，判断为多维度图像分割处理是的是同一个缺陷，将两张图像取并集，合为一张图像。C.异类缺陷重叠关系：当两个图像块的IOU大于阈值I₂，且图像标签为不同类别，则判断为分类网络对于同一处缺陷产生了误分类。通过比较两张图像标签的置信度，选择更大的一个标签作为该区域缺陷的最终预测标签。The fusion layer will combine the category and location information of each defect candidate area to form labeled defect images of each dimension, and then fuse the images of three dimensions to screen and return defect candidate areas. The rules are: A. Inclusion relationship of similar defects: When each edge of an image block is inside another image block, and the two images are of the same type of defects, it is judged as a similar defect inclusion relationship, and the image blocks belonging to the inner image block are judged to be included. Delete, keep the big picture. B. Partial overlap relationship of similar defects: When similar defects overlap, the proportion of the overlapping part needs to be considered. By calculating the IOU, when the IOU is greater than a certain threshold I ₁ and the two images are of the same type, it is judged that the multi-dimensional image segmentation process is Yes It is the same defect, and the two images are merged into one image. C. Overlapping relationship of heterogeneous defects: When the IOU of two image blocks is greater than the threshold I ₂ and the image labels are of different categories, it is judged that the classification network has misclassified the same defect. By comparing the confidence of the two image labels, the larger one is selected as the final predicted label for defects in the region.

表3展示了不同检测算法在不同标注量下的最终检测结果。从表3可以看出，当有标签训练集较小的情况下，本文所提的方法相较于其它的基于目标检测模型的缺陷检测方法效果更好。在10％和25％标注量下，本文方法在各个类别的AP值均高于其余方法，并且，相较于其余方法的最优结果，分别具有22％和20％的提升。随着有标签样本数量的增加，本文所提方法在压雪车外观缺陷数据集上平均AP值表现为小幅增长Table 3 shows the final detection results of different detection algorithms under different labels. As can be seen from Table 3, when the labeled training set is small, the method proposed in this paper is better than other defect detection methods based on target detection models. Under 10% and 25% annotation, the AP value of our method in each category is higher than other methods, and compared with the best results of other methods, it has 22% and 20% improvement respectively. With the increase of the number of labeled samples, the average AP value of the method proposed in this paper shows a small increase in the data set of appearance defects

表3不同标注量下各缺陷检测算法结果比较Table 3 Comparison of the results of each defect detection algorithm under different labeling quantities

步骤三：如图5所示，采用矩形框画出代表缺陷的候选区域，并在矩形框上标注该区域的类别。得到矩形框位置信息和对应类别，将背景类不予标注；将相同目标类别的框，按照位置和大小进行筛选：包含关系删去内部框，重叠关系计算IOU,当IOU大于阈值，两者合并；不同目标类别的框重叠时，按照两者的分类得分高低筛选，保留得分高的类别。在图像上标注出矩形框和对应类别。将该图像回传给移动终端的显示器，并且在移动终端保存。在该检测任务完成后，生成检测报告，报告内容包含各图像是否存在缺陷、缺陷类别、缺陷面积、缺陷位置等信息。Step 3: As shown in Fig. 5, a rectangular frame is used to draw a candidate area representing the defect, and the category of the area is marked on the rectangular frame. Obtain the position information and corresponding category of the rectangular box, and do not mark the background category; screen the boxes of the same target category according to their position and size: delete the inner box from the inclusion relationship, and calculate the IOU for the overlap relationship. When the IOU is greater than the threshold, the two are combined. ; When the boxes of different target categories overlap, they are filtered according to the classification scores of the two, and the category with the highest score is reserved. Label the rectangular box and the corresponding category on the image. The image is transmitted back to the display of the mobile terminal and stored in the mobile terminal. After the inspection task is completed, an inspection report is generated, and the report content includes information such as whether each image has defects, defect type, defect area, defect location, etc.

本发明还提供一种用于所述的复杂背景下基于视觉的压雪车外观缺陷检测方法的检测装置，如图7所示，检测装置包括移动终端1以及上位机2，移动终端1用于进行图像采集，上位机2用于进行缺陷检测及生成检测报告；移动终端1与上位机2之间通讯连接。The present invention also provides a detection device for the method for detecting the appearance defects of snow-pumping vehicles based on vision in a complex background. As shown in FIG. 7 , the detection device includes a mobile terminal 1 and a host computer 2, and the mobile terminal 1 is used for For image acquisition, the upper computer 2 is used for defect detection and generation of inspection reports; the mobile terminal 1 and the upper computer 2 are connected for communication.

上位机2包括缺陷检测模块21、缺陷标记模块22以及检测报告生成模块23，缺陷检测模块21包括多维图像分割算法单元211、半监督分类优化单元212以及融合算法单元213。The host computer 2 includes a defect detection module 21 , a defect marking module 22 and a detection report generation module 23 . The defect detection module 21 includes a multi-dimensional image segmentation algorithm unit 211 , a semi-supervised classification optimization unit 212 and a fusion algorithm unit 213 .

多维图像分割算法单元211从多个维度提取可能存在缺陷的区域，提高多目标缺陷的检出率；半监督分类优化单元212基于半监督方法构建缺陷识别网络的优化模型，得到准确的分类结果；融合算法单元213对带标签图像块进行筛选融合，提高定位精度。The multi-dimensional image segmentation algorithm unit 211 extracts areas that may have defects from multiple dimensions to improve the detection rate of multi-target defects; the semi-supervised classification optimization unit 212 constructs an optimization model of the defect identification network based on the semi-supervised method to obtain accurate classification results; The fusion algorithm unit 213 filters and fuses the labeled image blocks to improve the positioning accuracy.

缺陷标记模块22用于利用矩形框进行缺陷标记并将标记后的单个图像返回至移动终端，检测报告生成模块用于生成检测报告。The defect marking module 22 is used for marking defects with a rectangular frame and returning the marked single image to the mobile terminal, and the detection report generating module is used for generating a detection report.

优选地，移动终端1为带有显示屏的便携式工业相机。Preferably, the mobile terminal 1 is a portable industrial camera with a display screen.

优选地，移动终端1接收上位机2回传的数据并以单张图像为单位进行显示，上位机的检测报告生成模块保存检测任务中所有图像，直至该检测任务全部完成，以单个任务为单位，生成检测报告。Preferably, the mobile terminal 1 receives the data returned by the host computer 2 and displays it in a single image unit, and the detection report generation module of the host computer saves all the images in the detection task until the detection task is completely completed, and takes a single task as a unit to generate a test report.

以上所述的实施例仅是对本发明的优选实施方式进行描述，并非对本发明的范围进行限定，在不脱离本发明设计精神的前提下，本领域普通技术人员对本发明的技术方案做出的各种变形和改进，均应落入本发明权利要求书确定的保护范围内。The above-mentioned embodiments are only to describe the preferred embodiments of the present invention, and do not limit the scope of the present invention. Without departing from the design spirit of the present invention, those of ordinary skill in the art can make various modifications to the technical solutions of the present invention. Such deformations and improvements shall fall within the protection scope determined by the claims of the present invention.