patents.google.com

CN111462012A - SAR image simulation method for generating countermeasure network based on conditions - Google Patents

  • ️Tue Jul 28 2020
一种基于条件生成对抗网络的SAR图像仿真方法A SAR Image Simulation Method Based on Conditional Generative Adversarial Networks

技术领域technical field

本发明属于基于条件生成对抗网络的SAR图像仿真技术领域，尤其涉及一种基于条件生成对抗网络的SAR图像仿真方法。The invention belongs to the technical field of SAR image simulation based on conditional generation confrontation network, and in particular relates to a SAR image simulation method based on conditional generation confrontation network.

背景技术Background technique

随着遥感观测技术的发展，多源遥感图像的种类越来越丰富。在图像配准中，采用不同的传感器对同一区域成像所获取到的图像称为异源图像。由于传感器之间的成像原理、处理机制、传感器参数等存在较大差异，使得两幅异源图像之间相关性较小，因此若简单地使用针对同源图像的配准算法来对异源图像进行处理已无法实现较好的结果。要实现异源图像配准，关键技术是将异源图像转换为同源图像，以降低配准难度。With the development of remote sensing observation technology, the types of multi-source remote sensing images are becoming more and more abundant. In image registration, images obtained by imaging the same area with different sensors are called heterologous images. Due to the large differences in imaging principles, processing mechanisms, and sensor parameters between sensors, the correlation between two heterologous images is small. Processing has not been able to achieve better results. To achieve heterologous image registration, the key technology is to convert heterologous images into homologous images to reduce the difficulty of registration.

光学图像视觉体验感较好，呈现出丰富的纹理特征、灰度等信息，当处于光线充足、视野开阔的条件下时，通过光学传感器可获得质量较高的可见光图像。但在较差的气候情况下(如光线不足、云雾遮挡等)，低质量的光学图像就失去了应用价值。而SAR图像正可以弥补这一不足。合成孔径雷达(Synthetic Aperture Rader,SAR)成像可以不受光照和天气等因素的影响，并且SAR成像可达极高的分辨率。通过结合光学与SAR传感器的成像优势，将异源图像中携带的不同数据进行相互融合，可以在诸如自然灾害监测、目标检测等特定场景中产生重要的应用价值。The optical image has a good visual experience, showing rich texture features, grayscale and other information. When it is under the conditions of sufficient light and a wide field of view, high-quality visible light images can be obtained through the optical sensor. But in poor climate conditions (such as insufficient light, cloud occlusion, etc.), low-quality optical images lose their application value. And SAR images can make up for this shortcoming. Synthetic Aperture Rader (SAR) imaging can not be affected by factors such as light and weather, and SAR imaging can reach extremely high resolution. By combining the imaging advantages of optical and SAR sensors, the fusion of different data carried in heterogeneous images can produce important application values in specific scenarios such as natural disaster monitoring and target detection.

SAR图像生成的技术难点在于：①SAR图像相干斑噪声的影响；②光学图像到伪SAR的图像转换框架的搭建。The technical difficulties of SAR image generation are: ① the influence of speckle noise in SAR images; ② the construction of the image conversion framework from optical image to pseudo-SAR.

发明内容SUMMARY OF THE INVENTION

本发明的目的是提供一种基于条件生成对抗网络的SAR图像仿真方法。The purpose of the present invention is to provide a SAR image simulation method based on conditional generative adversarial network.

为实现上述目的，本发明采用的技术方案是：一种基于条件生成对抗网络的SAR图像仿真方法，包括以下步骤：In order to achieve the above object, the technical solution adopted in the present invention is: a SAR image simulation method based on conditional generative adversarial network, comprising the following steps:

步骤1、利用滚动引导滤波方法对目标SAR图像进行去噪；Step 1. Denoise the target SAR image by using the rolling guided filtering method;

步骤2、将光学图像与对应的预处理后的SAR图像对制作数据集；包括以下子步骤：Step 2, pair the optical image with the corresponding preprocessed SAR image to make a data set; including the following sub-steps:

步骤2.1、读入已配准的异源图像对；Step 2.1, read in the registered heterologous image pair;

步骤2.2、提取光学图像中的特征点，并在对应的SAR图像上以同一坐标位置的像素点为中心截取相同大小的图像块，待几组异源图像均处理完之后，进行数据扩增，将光学图像和SAR图像合并，最后划分训练集和测试集；Step 2.2: Extract the feature points in the optical image, and intercept image blocks of the same size with the pixels at the same coordinate position as the center on the corresponding SAR image. After several groups of heterologous images have been processed, data amplification is performed. Combine the optical image and SAR image, and finally divide the training set and test set;

步骤3、搭建基于条件生成对抗网络的图像转换网络结构；包括以下子步骤：Step 3. Build an image conversion network structure based on conditional generative adversarial network; including the following sub-steps:

步骤3.1、以U-Net网络结构为基础，结合Res-Net对条件生成对抗网络进行优化；Step 3.1, based on the U-Net network structure, combined with Res-Net to optimize the conditional generative adversarial network;

步骤3.2、构建五层卷积判别器网络结构；Step 3.2, construct a five-layer convolutional discriminator network structure;

步骤4、以光学图像和SAR图像对组成的训练集作为输入，多次迭代训练模型，并使用Adam算法对目标函数进行优化；Step 4. Take the training set consisting of optical image and SAR image pair as input, train the model multiple times iteratively, and use the Adam algorithm to optimize the objective function;

步骤5、在测试集上进行SAR效果图的转换和测试。Step 5: Convert and test the SAR renderings on the test set.

在上述的基于条件生成对抗网络的SAR图像仿真方法中，步骤1的实现包括：对原SAR图像进行高斯滤波，滤波后的图像作为引导图像并且在此基础上进行迭代滤波操作，对图像中大区域的物体边缘进行恢复，具体步骤如下；In the above-mentioned SAR image simulation method based on conditional generative adversarial network, the implementation of step 1 includes: performing Gaussian filtering on the original SAR image, using the filtered image as a guide image and performing an iterative filtering operation on this basis, and performing an iterative filtering operation on the image. The edge of the object in the area is restored, and the specific steps are as follows;

步骤1.1、对原SAR图像进行高斯滤波；滤除小区域的斑点与细小结构，表达式为：Step 1.1. Perform Gaussian filtering on the original SAR image; filter out spots and small structures in small areas, the expression is:

Figure BDA0002437477570000031

Figure BDA0002437477570000031

式(1)中，J1(p)表示像素点p经高斯滤波后的像素值大小，N(p)代表该点的周围用于滤波计算的邻域，q表示这一邻域中涉及到的所有像素点，kp是归一化系数，用于保持计算结果的范围；In formula (1), J 1 (p) represents the pixel value of the pixel point p after Gaussian filtering, N(p) represents the neighborhood around the point for filtering calculation, and q represents the neighborhood involved in this neighborhood. All pixels of , k p is the normalization coefficient, which is used to maintain the range of the calculation result;

步骤1.2、采用引导的方式做后续的迭代，加强图像的边缘结构，数学表达式为：Step 1.2. Use the guided method to do subsequent iterations to strengthen the edge structure of the image. The mathematical expression is:

Figure BDA0002437477570000032

Figure BDA0002437477570000032

式(2)中，Jt(p)表示像素点p经过第t次滤波后的像素值大小；Jt-1(p)、Jt-1(q)分别表示像素点p和q经过第t-1次滤波后的像素值大小；δs和δr分别表示空间尺度和距离尺度。In formula (2), J t (p) represents the pixel value of pixel p after the t-th filtering; J t-1 (p) and J t-1 (q) represent the pixel p and q after the The size of the pixel value after t-1 filtering; δ s and δ r represent the spatial scale and distance scale, respectively.

在上述的基于条件生成对抗网络的SAR图像仿真方法中，步骤2的实现包括：In the above-mentioned SAR image simulation method based on conditional generative adversarial network, the implementation of step 2 includes:

步骤2.1、读入已配准的异源图像对，采用SIFT方法对每一幅光学图像提取所有可能的SIFT特征点，并排除掉两个特征点之间的距离小于d的像素点，利用d来调整在图像中获取到的SIFT特征点的稠密度，取d＝20；Step 2.1. Read in the registered heterologous image pair, use the SIFT method to extract all possible SIFT feature points for each optical image, and exclude the pixels whose distance between the two feature points is less than d, and use d To adjust the density of the SIFT feature points obtained in the image, take d=20;

步骤2.2、进行筛选后，以每一个选中的特征点为中心，截取大小为256×256的图像块，若所选区域超出图像边界则丢弃该点；随后在对应的SAR图像上以同一坐标位置的像素点为中心截取相同大小的图像块；将光学图像和SAR图像以相同的命名分别保存到两个对应的文件夹中；Step 2.2. After screening, take each selected feature point as the center, and intercept an image block with a size of 256×256. If the selected area exceeds the image boundary, discard the point; then use the same coordinate position on the corresponding SAR image. The pixel points of the SAR image are centered on the same size image block; save the optical image and the SAR image to two corresponding folders with the same name;

步骤2.3、待几组异源图像均处理完后，对光学图像和SAR图像的两个文件夹中的所有图像进行旋转、镜像、翻转操作以完成数据扩增；Step 2.3. After several groups of heterologous images are processed, rotate, mirror, and flip all images in the two folders of optical images and SAR images to complete data amplification;

步骤2.4、将光学图像和SAR图像两个文件夹中的图像合并为一张512×256的图像，并保存在另一个文件夹中；Step 2.4. Combine the images in the optical image and SAR image folders into a 512×256 image and save it in another folder;

步骤2.5、将步骤2.4所述另一个文件夹中的样本按照80％/20％的比例随机分为训练集和测试集。Step 2.5: Divide the samples in the other folder described in step 2.4 into training set and test set randomly according to the ratio of 80%/20%.

在上述的基于条件生成对抗网络的SAR图像仿真方法中，步骤3.1所述条件生成对抗网络的生成器采用U-Net网络结构，包括编码器模块和解码器模块；In the above-mentioned SAR image simulation method based on conditional generative adversarial network, the generator of conditional generative adversarial network described in step 3.1 adopts U-Net network structure, including encoder module and decoder module;

编码器模块包括8层，每一层均为double(conv+bn+lrelu)+shortcut结构，卷积核数目从64逐层倍增直到512后不变，bn为批量归一化优化，lrelu表示激活函数使用LeakyReLU函数，shortcut指残差网络中的捷径；The encoder module includes 8 layers, each layer is a double(conv+bn+lrelu)+shortcut structure, the number of convolution kernels is doubled from 64 layer by layer until 512 and remains unchanged, bn is batch normalization optimization, and lrelu means activation The function uses the LeakyReLU function, and shortcut refers to the shortcut in the residual network;

解码器模块包括8层，与编解码器层数相同的单元具有相同的卷积核数目，而解码器中每一个单元的结构为conv+bn+relu，relu表示激活函数使用ReLU函数，并对编解码模块对应的层进行拼接；每一卷积操作的卷积核大小均为3×3，每一个单元之间均接有一层2*2的最大池化层。The decoder module includes 8 layers, and the units with the same number of layers as the encoder and decoder have the same number of convolution kernels, and the structure of each unit in the decoder is conv+bn+relu, relu means that the activation function uses the ReLU function, and The layers corresponding to the encoding and decoding modules are spliced; the size of the convolution kernel of each convolution operation is 3 × 3, and a 2 × 2 maximum pooling layer is connected between each unit.

在上述的基于条件生成对抗网络的SAR图像仿真方法中，步骤3.2所述五层卷积判别器网络采用PatchGAN结构，PatchGAN将图像分为N×N个固定大小的图像块，五层卷积判别器分别对每一块的真假性作判断，最后对一张图像中所得响应求均值后作为输出结果；且patch size设置为70*70；所述五层卷积判别器网络结构的前四层用于对样本进行特征提取，卷积核数目从64开始递增，卷积核大小为3×3，步长为2；最后一层卷积用于将特征映射到一维输出，将Sigmoid函数作为激活函数；前四层卷积后均进行批量归一化处理，激活函数为LeakyReLU，取值为0.2。In the above-mentioned SAR image simulation method based on conditional generative adversarial network, the five-layer convolution discriminator network described in step 3.2 adopts the PatchGAN structure, and PatchGAN divides the image into N×N fixed-size image blocks, and five layers of convolution discriminate The device judges the authenticity of each block separately, and finally averages the responses obtained in an image as the output result; and the patch size is set to 70*70; the first four layers of the five-layer convolutional discriminator network structure It is used for feature extraction of samples, the number of convolution kernels increases from 64, the size of convolution kernels is 3×3, and the stride is 2; the last layer of convolution is used to map features to one-dimensional output, and the Sigmoid function is used as Activation function; batch normalization is performed after the first four layers of convolution, the activation function is LeakyReLU, and the value is 0.2.

在上述的基于条件生成对抗网络的SAR图像仿真方法中，步骤4的实现包括：In the above-mentioned SAR image simulation method based on conditional generative adversarial network, the realization of step 4 includes:

步骤4.1、按照下式的损失函数训练条件生成对抗网络：Step 4.1. Generate an adversarial network according to the loss function training condition of the following formula:

Figure BDA0002437477570000051

Figure BDA0002437477570000051

Figure BDA0002437477570000052

Figure BDA0002437477570000052

G'＝arg minG maxD LcGAN+λL1 (5)G'=arg min G max D L cGAN +λL 1 (5)

将真实SAR图像y作为约束条件，随机噪声记为z，x为输入的光学图像，其中x,y服从pdata(x,y)数据分布，随机噪声z服从pz(z)数据分布；LcGAN(G,D)表示生成器和判别器的对抗损失约束，

Figure BDA0002437477570000053

表示生成器的图像块与真实图像块之间的像素级约束，D(x,y)代表判别器对x、y的匹配预测，G(x,z)代表输入光学图像和噪声后生成器的输出图像，D(x,G(x,z))表示判别器对x、G(x,z)的匹配预测，λ表示引入的L1损失前的系数，设λ＝100；生成器训练使得LcGAN最小化，差别器训练使得LcGAN最大化；Taking the real SAR image y as the constraint condition, the random noise is denoted as z, and x is the input optical image, where x, y obey the data distribution of p data (x, y), and the random noise z obey the data distribution of p z (z); L cGAN (G, D) represents the adversarial loss constraints of the generator and discriminator,

Figure BDA0002437477570000053

represents the pixel-level constraints between the generator's image patch and the real image patch, D(x,y) represents the discriminator's matching prediction of x, y, and G(x,z) represents the input optical image and the post-noise generator's Output image, D(x, G(x, z)) represents the matching prediction of x, G(x, z) by the discriminator, λ represents the coefficient before the introduced L1 loss , set λ=100; the generator training makes L cGAN is minimized, and discriminator training maximizes L cGAN ;

步骤4.2、条件生成对抗网络训练步骤如下：Step 4.2. Conditional generative adversarial network training steps are as follows:

步骤4.2.1、初始化L1损失超参数λ，迭代的总次数t；Step 4.2.1. Initialize the L1 loss hyperparameter λ, the total number of iterations t;

步骤4.2.2、for i＝1,2,...,t do；Step 4.2.2, for i=1,2,...,t do;

步骤4.2.3、给出m对样本图像：Step 4.2.3, give m pairs of sample images:

步骤4.2.4、

Figure BDA0002437477570000054

Step 4.2.4,

Figure BDA0002437477570000054

步骤4.2.5、更新判别器D的参数并令下式最大化：Step 4.2.5, update the parameters of the discriminator D and maximize the following formula:

步骤4.2.6、

Figure BDA0002437477570000055

Step 4.2.6,

Figure BDA0002437477570000055

步骤4.2.7、更新生成器G的参数并令下式最小化：Step 4.2.7, update the parameters of the generator G and minimize the following formula:

步骤4.2.8、

Figure BDA0002437477570000056

Step 4.2.8,

Figure BDA0002437477570000056

步骤4.2.9、结束；Step 4.2.9, end;

其中，Io表示光学图像，Is表示对应的SAR图像，Ig表示生成的伪SAR图像；Among them, I o represents the optical image, Is represents the corresponding SAR image, and I g represents the generated pseudo SAR image;

步骤4.3、Adam算法对目标函数进行优化公式如下：Step 4.3. The Adam algorithm optimizes the objective function. The formula is as follows:

mt＝β1×mt-1+(1-β1)×gt (6)m t =β 1 ×m t-1 +(1-β 1 )×g t (6)

Figure BDA0002437477570000061

Figure BDA0002437477570000061

Figure BDA0002437477570000062

Figure BDA0002437477570000062

Figure BDA0002437477570000063

Figure BDA0002437477570000063

Figure BDA0002437477570000064

Figure BDA0002437477570000064

其中，mt和vt代表梯度一阶矩和二阶矩估计，β1和β2表示延长因子基数，gt是参数θ在t-1时刻时目标函数的梯度，

Figure BDA0002437477570000065

和

Figure BDA0002437477570000066

是mt，vt的修正，ε是一个小数值常数，η表示学习率；Among them, m t and v t represent the gradient first moment and second moment estimation, β 1 and β 2 represent the extension factor base, g t is the gradient of the objective function of the parameter θ at time t-1,

Figure BDA0002437477570000065

and

Figure BDA0002437477570000066

is the modification of m t , v t , ε is a small numerical constant, η represents the learning rate;

Adam算法流程如下：The Adam algorithm process is as follows:

步骤4.3.1、输入η,β1,β2,ε以及最大循环次数epoch参数；Step 4.3.1, input η, β 1 , β 2 , ε and the maximum number of cycles epoch parameter;

步骤4.3.2、在t＝0时，初始化参数θ0，并令一阶矩估计m0＝0，二阶矩估计v0＝0；Step 4.3.2. When t=0, initialize the parameter θ 0 , and make the first-order moment estimate m 0 =0, and the second-order moment estimate v 0 =0;

步骤4.3.3、更新迭代的次数：t＝t+1；Step 4.3.3, update the number of iterations: t=t+1;

步骤4.3.4、在训练样本集中选取m个样本{x(1),...,x(m)}，并且将其对应的目标样本记为y(i)，接着进行在θt-1时的梯度计算：

Figure BDA0002437477570000067

Step 4.3.4, select m samples {x (1) ,...,x (m) } in the training sample set, and record the corresponding target samples as y (i) , and then proceed to θ t-1 Gradient calculation when:

Figure BDA0002437477570000067

步骤4.3.5、更新mt：mt＝β1×mt-1+(1-β1)×gt；Step 4.3.5, update m t : m t =β 1 ×m t-1 +(1-β 1 )×g t ;

步骤4.3.6、更新vt：

Figure BDA0002437477570000071

Step 4.3.6, update v t :

Figure BDA0002437477570000071

步骤4.3.7、修正

Figure BDA0002437477570000072

Figure BDA0002437477570000073

Step 4.3.7, correction

Figure BDA0002437477570000072

Figure BDA0002437477570000073

步骤4.3.8、修正

Figure BDA0002437477570000074

Figure BDA0002437477570000075

Step 4.3.8, correction

Figure BDA0002437477570000074

Figure BDA0002437477570000075

步骤4.3.9、更新θt：

Figure BDA0002437477570000076

循环步骤4.4.3～步骤4.3.8，直到f(θ)收敛或是达到了预先设定的最大循环次数epoch后，返回f(θ)的最优解θt。Step 4.3.9, update θ t :

Figure BDA0002437477570000076

Repeat steps 4.4.3 to 4.3.8 until f(θ) converges or reaches the preset maximum number of cycles epoch, then return to the optimal solution θ t of f(θ).

在上述的基于条件生成对抗网络的SAR图像仿真方法中，Adam优化算法的学习率

Figure BDA0002437477570000077

的计算如下式：In the above-mentioned SAR image simulation method based on conditional generative adversarial network, the learning rate of Adam optimization algorithm

Figure BDA0002437477570000077

The calculation is as follows:

Figure BDA0002437477570000078

Figure BDA0002437477570000078

其中，η表示学习率初始值，epoch是代表迭代总次数，iter是目前的迭代次数，offset是训练过程中需要开始减小学习率

Figure BDA0002437477570000079

时的迭代次数；当iter小于offset时，先用一个预设的较大的η作为当前学习率，当iter达到offset后，逐渐的减小学习率。Among them, η represents the initial value of the learning rate, epoch represents the total number of iterations, iter is the current number of iterations, and offset is the need to start reducing the learning rate during the training process.

Figure BDA0002437477570000079

The number of iterations at time; when iter is less than offset, a preset larger η is used as the current learning rate, and when iter reaches offset, the learning rate is gradually reduced.

本发明的有益效果：(1)本发明采用滚动引导滤波算法去除了原始SAR图像中的相干斑噪声，在此基础上制作数据集，使得生成网络学习到训练SAR样本中更加真实的特征，减小虚假斑点特征带来的影响。The beneficial effects of the present invention are as follows: (1) The present invention adopts the rolling guide filtering algorithm to remove the speckle noise in the original SAR image, and creates a data set on this basis, so that the generating network can learn more real features in the training SAR samples, reducing the The effect of small false speckle features.

(2)采用了条件式生成对抗网络将光学图像转换为伪SAR图像，克服了由于异源图像之间的显著差异性导致的特征提取难度较大的问题。能在异源图像配准问题上有效的降低配准难度。(2) Conditional generative adversarial network is used to convert optical images into pseudo-SAR images, which overcomes the difficulty of feature extraction due to the significant differences between heterogeneous images. It can effectively reduce the difficulty of registration in the problem of heterogeneous image registration.

附图说明Description of drawings

图1为本发明一个实施例的流程图；1 is a flowchart of an embodiment of the present invention;

图2为本发明一个实施例中用于对SAR图像进行去噪的滚动引导滤波算法流程图；2 is a flowchart of a rolling-guided filtering algorithm for denoising a SAR image according to an embodiment of the present invention;

图3(a)为本发明一个实施例原始图像；Figure 3(a) is an original image of an embodiment of the present invention;

图3(b)为本发明一个实施例双边滤波算法结果示意图；FIG. 3(b) is a schematic diagram of the result of a bilateral filtering algorithm according to an embodiment of the present invention;

图3(c)为本发明一个实施例引导滤波算法结果示意图；FIG. 3(c) is a schematic diagram of the result of a guided filtering algorithm according to an embodiment of the present invention;

图3(d)为本发明一个实施例非线性扩散滤波算法结果示意图；FIG. 3(d) is a schematic diagram of the result of a nonlinear diffusion filtering algorithm according to an embodiment of the present invention;

图3(e)为本发明一个实施例滚动引导滤波算法结果示意图；FIG. 3(e) is a schematic diagram of the result of a scrolling guide filtering algorithm according to an embodiment of the present invention;

图4(a)为本发明一个实施例所用的光学图像；Figure 4(a) is an optical image used in an embodiment of the present invention;

图4(b)为本发明一个实施例所用SAR图像；Figure 4(b) is a SAR image used in an embodiment of the present invention;

图5(a)为本发明一个实施例中的生成器生成的第一光学图像效果图；Fig. 5(a) is a first optical image effect diagram generated by a generator in an embodiment of the present invention;

图5(b)为本发明一个实施例中的生成器生成的第二光学图像效果图；Fig. 5(b) is an effect diagram of the second optical image generated by the generator in an embodiment of the present invention;

图5(c)为本发明一个实施例中的生成器生成的第三光学图像效果图；Fig. 5(c) is an effect diagram of a third optical image generated by a generator in an embodiment of the present invention;

图5(d)为本发明一个实施例中的生成器生成的第一真实的SAR图像效果图；FIG. 5(d) is a first real SAR image effect diagram generated by a generator in an embodiment of the present invention;

图5(e)为本发明一个实施例中的生成器生成的第二真实的SAR图像效果图；Fig. 5(e) is a second real SAR image rendering effect diagram generated by a generator in an embodiment of the present invention;

图5(f)为本发明一个实施例中的生成器生成的第三真实的SAR图像效果图；Fig. 5(f) is a third real SAR image effect diagram generated by a generator in an embodiment of the present invention;

图5(g)为本发明一个实施例中的生成器生成的第一伪SAR图像效果图；Fig. 5 (g) is the first pseudo-SAR image effect diagram generated by the generator in one embodiment of the present invention;

图5(h)为本发明一个实施例中的生成器生成的第二伪SAR图像效果图；FIG. 5(h) is an effect diagram of a second pseudo-SAR image generated by a generator in an embodiment of the present invention;

图5(i)为本发明一个实施例中的生成器生成的第三伪SAR图像效果图。FIG. 5(i) is an effect diagram of a third pseudo-SAR image generated by a generator in an embodiment of the present invention.

具体实施方式Detailed ways

下面结合附图对本发明的实施方式进行详细描述。The embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

为了克服由异源图像之间的差异性造成的配准技术难度较大的问题，本实施例首先对需要用于制作数据集的SAR图像进行预处理，去除相干斑噪声后会使得生成网络学习到更加真实的特征，减小虚假斑点特征带来的影响。其次，利用已配准好的异源图像对来制作训练样本集。最后，搭建图像转换网络，并在数据集的基础上进行模型的训练和测试。具体步骤如下：In order to overcome the difficulty of registration technology caused by the difference between heterogenous images, this embodiment first preprocesses the SAR images that need to be used to create the data set, and after removing the speckle noise, the generation network will learn to more realistic features, reducing the impact of false speckle features. Second, use the registered heterologous image pairs to make a training sample set. Finally, the image conversion network is built, and the model is trained and tested on the basis of the dataset. Specific steps are as follows:

S100，利用滚动引导滤波方法对目标SAR图像进行去噪；S100, denoising the target SAR image by using the rolling guided filtering method;

S200，将光学图像与对应的预处理后的SAR图像对制作成训练样本集。包括以下具体步骤：S200, the optical image and the corresponding preprocessed SAR image pair are made into a training sample set. It includes the following specific steps:

S210，读入已配准的异源图像对；S210, read in the registered heterologous image pair;

S220，提取光学图像中的特征点，并在对应的SAR图像上以同一坐标位置的像素点为中心截取相同大小的图像块，待几组异源图像均处理完后，进行数据扩增，并将光学图像和SAR图像合并，最后划分训练集和测试集；S220, extracting the feature points in the optical image, and intercepting image blocks of the same size on the corresponding SAR image with the pixel points at the same coordinate position as the center, and after several groups of heterologous images are processed, perform data amplification, and Combine the optical image and SAR image, and finally divide the training set and test set;

S300，搭建基于条件生成对抗网络的图像转换网络结构。具体步骤如下：S300, build an image conversion network structure based on conditional generative adversarial network. Specific steps are as follows:

S310，以U-Net网络结构为基础，结合Res-Net的优点对生成网络进行优化；S310, based on the U-Net network structure, combined with the advantages of Res-Net to optimize the generation network;

S320，构建由五层卷积构成的判别器网络结构；S320, construct a discriminator network structure composed of five layers of convolution;

S400,以光学图像和SAR图像对组成的训练集作为输入，多次迭代训练模型，并使用Adam优化算法对目标函数进行优化；S400, take the training set composed of optical image and SAR image pair as input, train the model multiple times iteratively, and use the Adam optimization algorithm to optimize the objective function;

S500，在测试集上进行SAR效果图的转换和测试。S500, convert and test the SAR renderings on the test set.

进一步的，步骤S100的具体实现方式如下：Further, the specific implementation of step S100 is as follows:

对原始SAR图像，采用滚动引导滤波算法进行去噪处理。算法简要介绍如下：1)对原SAR图像进行高斯滤波，平滑掉图像中斑点噪声等复杂小区域；2)上述处理后的图像将作为引导图像并且在此基础上进行迭代滤波操作，对图像中大区域的物体边缘进行恢复。具体算法流程为：The original SAR image is denoised by rolling guided filtering algorithm. The algorithm is briefly introduced as follows: 1) Gaussian filtering is performed on the original SAR image to smooth out complex small areas such as speckle noise in the image; 2) The above-processed image will be used as a guide image and an iterative filtering operation will be performed on this basis. The edges of objects in large areas are restored. The specific algorithm flow is:

(1)首先对SAR图像进行高斯滤波，滤除小区域的斑点与细小结构，表达式为：(1) First, perform Gaussian filtering on the SAR image to filter out spots and small structures in small areas. The expression is:

Figure BDA0002437477570000101

Figure BDA0002437477570000101

在式(1)中，J1(p)表示像素点p经高斯滤波后的像素值大小，N(p)代表该点的周围用于滤波计算的邻域，q表示这一邻域中涉及到的所有像素点，kp是归一化系数，用于保持计算结果的范围。In formula (1), J 1 (p) represents the pixel value of pixel p after Gaussian filtering, N(p) represents the neighborhood around the point used for filtering calculation, and q represents the neighborhood involved in this neighborhood. To all the pixels, k p is the normalization coefficient used to keep the range of the calculation result.

(2)在上述基础上，采用引导的方式来做后续的迭代，加强图像的边缘结构，数学表达式为：(2) On the basis of the above, a guided method is used to do subsequent iterations to strengthen the edge structure of the image. The mathematical expression is:

Figure BDA0002437477570000102

Figure BDA0002437477570000102

式(2)中，Jt(p)表示像素点p经过第t次滤波后的像素值大小；Jt-1(p)、Jt-1(q)分别表示像素点p和q经过第t-1次滤波后的像素值大小；δs和δr分别表示空间尺度和距离尺度。In formula (2), J t (p) represents the pixel value of pixel p after the t-th filtering; J t-1 (p) and J t-1 (q) represent the pixel p and q after the The size of the pixel value after t-1 filtering; δ s and δ r represent the spatial scale and distance scale, respectively.

进一步的，步骤S200的具体实现方式如下：Further, the specific implementation of step S200 is as follows:

(1)读入已配准的异源图像对，采用SIFT方法对每一幅光学图像提取所有可能的SIFT特征点，并排除掉两个特征点之间的距离小于d的像素点，利用d来调整在图像中获取到的SIFT特征点的稠密度，本实施例中设置d＝20；(1) Read the registered heterologous image pair, use the SIFT method to extract all possible SIFT feature points for each optical image, and exclude the pixels whose distance between the two feature points is less than d, and use d to adjust the density of the SIFT feature points obtained in the image, in this embodiment, set d=20;

(2)进行筛选后，以每一个选中的特征点为中心，截取大小为256×256的图像块，若所选区域超出图像边界则丢弃该点。随后在对应的SAR图像上以同一坐标位置的像素点为中心截取相同大小的图像块。把光学图像和SAR图像以相同的命名分别保存到两个对应的文件夹中；(2) After screening, take each selected feature point as the center, intercept an image block with a size of 256 × 256, and discard the point if the selected area exceeds the image boundary. Then, on the corresponding SAR image, an image block of the same size is intercepted with the pixel point at the same coordinate position as the center. Save the optical image and SAR image to two corresponding folders with the same name;

(3)待几组异源图像都处理完后，对光学和SAR图像的两个文件夹中的所有图像进行旋转、镜像、翻转等操作以完成数据扩增；(3) After several sets of heterologous images are processed, all images in the two folders of optical and SAR images are rotated, mirrored, and flipped to complete data amplification;

(4)将光学和SAR图像两个文件夹中的图像合并为一张512×256的图像，并保存在另一个文件夹中；(4) Combine the images in the two folders of optical and SAR images into a 512×256 image and save it in another folder;

(5)将上述文件夹中的样本按照80％/20％的比例随机分为训练集和测试集，最后，本实施例中采用的训练样本数量为10464个，训练数据集为8372个，测试数据集为2092个；(5) The samples in the above folder are randomly divided into training set and test set according to the ratio of 80%/20%. Finally, the number of training samples used in this embodiment is 10,464, the training data set is 8,372, and the test set is 10,464. The dataset is 2092;

进一步的，步骤S310的具体实现方式如下：Further, the specific implementation of step S310 is as follows:

本实施例中的条件式生成对抗网络中的生成器是采用了U-Net网络结构的思想，网络结构包括了编码器模块和解码器模块，编码器模块包括了8层，每一层都为double(conv+bn+lrelu)+shortcut的结构，卷积核数目从64逐层倍增直到512后不变，bn是指批量归一化优化，lrelu是指激活函数为LeakyReLU，shortcut是指残差网络中的“捷径”。解码器模块也是8层，编解码器层数相同的单元具有相同的卷积核数目，区别在于解码器中每一个单元的结构为conv+bn+relu，relu表示激活函数使用的是ReLU函数，并且，对编解码模块对应的层进行拼接。此外，每一卷积操作的卷积核大小都为3×3，每一个单元之间都接有一层2*2的最大池化层。The generator in the conditional generative adversarial network in this embodiment adopts the idea of the U-Net network structure. The network structure includes an encoder module and a decoder module. The encoder module includes 8 layers, and each layer is The structure of double(conv+bn+lrelu)+shortcut, the number of convolution kernels is multiplied from 64 layer by layer until 512, which remains unchanged. bn refers to batch normalization optimization, lrelu refers to the activation function of LeakyReLU, and shortcut refers to residual error. "Shortcuts" in the web. The decoder module is also 8 layers. Units with the same number of encoder and decoder layers have the same number of convolution kernels. The difference is that the structure of each unit in the decoder is conv+bn+relu, and relu indicates that the activation function uses the ReLU function. And, the layers corresponding to the encoding and decoding modules are spliced. In addition, the size of the convolution kernel of each convolution operation is 3 × 3, and a 2 × 2 max pooling layer is connected between each unit.

进一步的，步骤S320的具体实现方式如下：Further, the specific implementation of step S320 is as follows:

本实施例中的判别器网络结构由五层卷积构成。前四层用于对样本进行特征提取，卷积核数目从64开始递增，卷积核大小为3×3，步长为2。最后一层卷积被用于将特征映射到一维输出，将Sigmoid函数作为激活函数。前四层卷积后都进行了批量归一化(BatchNormalization)处理，激活函数为LeakyReLU，取值为0.2。此外，判别器网络中采用了PatchGAN结构。PatchGAN的思想是将图像分为N×N个固定大小的图像块，判别器分别对每一块的真假性作判断，最后对一张图像中所得响应求均值后作为输出结果。利用PatchGAN可以更好判断图像的局部特征，在本实施例中，将patch size设置为70*70。The discriminator network structure in this embodiment consists of five layers of convolution. The first four layers are used for feature extraction of samples, the number of convolution kernels increases from 64, the size of convolution kernels is 3 × 3, and the stride is 2. The last layer of convolution is used to map the features to a one-dimensional output, using the sigmoid function as the activation function. Batch normalization is performed after the first four layers of convolution, and the activation function is LeakyReLU with a value of 0.2. In addition, the PatchGAN structure is adopted in the discriminator network. The idea of PatchGAN is to divide the image into N×N fixed-size image blocks, the discriminator judges the authenticity of each block separately, and finally averages the responses obtained in an image as the output result. Using PatchGAN can better determine the local features of the image. In this embodiment, the patch size is set to 70*70.

进一步的，步骤S400的具体实现方式如下：Further, the specific implementation of step S400 is as follows:

对于传统的GAN来说，输入一个随机噪声，则生成相应的输出数据。但在输入与输出之间没有约束条件，这使得生成的数据具有很大的不确定性，可能会偏离理想的生成目标。而条件生成对抗网络(CGAN)在GAN的基础上新增了一个额外的信息，将这一额外信息作为生成过程的约束条件，使得生成网络的输出数据满足预期要求。本实施例按照下式的损失函数训练生成对抗网络：For traditional GAN, input a random noise and generate corresponding output data. But there is no constraint between input and output, which makes the generated data with great uncertainty and may deviate from the ideal generation goal. The Conditional Generative Adversarial Network (CGAN) adds an additional information to the GAN, and uses this additional information as a constraint of the generation process, so that the output data of the generation network meets the expected requirements. This embodiment trains the generative adversarial network according to the loss function of the following formula:

Figure BDA0002437477570000121

Figure BDA0002437477570000121

Figure BDA0002437477570000122

Figure BDA0002437477570000122

G'＝arg minG maxD LcGAN+λL1 (5)′G'=arg min G max D L cGAN +λL 1 (5)′

将真实SAR图像y作为约束条件，随机噪声记为z，x为输入的光学图像，其中x,y服从pdata(x,y)数据分布，随机噪声z服从pz(z)数据分布。其中LcGAN(G,D)表示生成器和判别器的对抗损失约束，

Figure BDA0002437477570000123

表示生成器的图像块与真实图像块之间的像素级约束，D(x,y)代表了判别器对x、y的匹配预测，G(x,z)代表输入光学图像和噪声后生成器的输出图像，D(x,G(x,z))表示判别器对x、G(x,z)的匹配预测，λ表示引入的L1损失前的系数，本实施例中设置λ＝100。生成器训练目的是使得LcGAN最小化，判别器的训练目的是使得LcGAN最大化。通过将L1损失与cGAN的损失结合起来，同时关注到图像中的低频和高频特征，会有效的提高生成样本的质量。Taking the real SAR image y as the constraint condition, the random noise is denoted as z, and x is the input optical image, where x, y obey the data distribution of p data (x, y), and the random noise z obey the data distribution of p z (z). where L cGAN (G, D) represents the adversarial loss constraints of the generator and discriminator,

Figure BDA0002437477570000123

represents the pixel-level constraints between the generator's image patch and the real image patch, D(x, y) represents the discriminator's matching prediction for x, y, and G(x, z) represents the input optical image and the post-noise generator The output image of , D(x, G(x, z)) represents the matching prediction of x, G(x, z) by the discriminator, λ represents the coefficient before the introduced L 1 loss, in this example, λ=100 . The generator is trained to minimize LcGAN , and the discriminator is trained to maximize LcGAN . By combining the L1 loss with the cGAN loss, paying attention to both low-frequency and high-frequency features in the image, the quality of the generated samples is effectively improved.

对于本实施例中提出的生成对抗网络结构，假设光学图像表示为Io，对应的SAR图像为Is，生成的伪SAR图像为Ig本实施例条件生成对抗网络CGAN训练步骤如下：For the generative adversarial network structure proposed in this embodiment, it is assumed that the optical image is represented as I o , the corresponding SAR image is I s , and the generated pseudo-SAR image is I g . The training steps of the conditional generative adversarial network CGAN in this embodiment are as follows:

算法1条件生成对抗网络CGAN训练流程：Algorithm 1 Conditional Generative Adversarial Network CGAN Training Process:

1.初始化L1损失超参数λ，迭代的总次数t； 1. Initialize the L1 loss hyperparameter λ, the total number of iterations t;

2.for i＝1,2,...,t do；2. for i=1,2,...,t do;

3.给出m对样本图像：3. Given m pairs of sample images:

4.

Figure BDA0002437477570000131

4.

Figure BDA0002437477570000131

5.更新判别器D的参数并令下式最大化：5. Update the parameters of the discriminator D and maximize the following:

6.

Figure BDA0002437477570000132

6.

Figure BDA0002437477570000132

7.更新生成器G的参数并令下式最小化：7. Update the parameters of the generator G and minimize the following:

8.

Figure BDA0002437477570000133

8.

Figure BDA0002437477570000133

9.结束。9. End.

针对本实施例提出的目标函数，采用Adam优化算法进行优化。涉及到的公式有：For the objective function proposed in this embodiment, the Adam optimization algorithm is used for optimization. The formulas involved are:

mt＝β1×mt-1+(1-β1)×gt (6)′m t =β 1 ×m t-1 +(1-β 1 )×g t (6)′

Figure BDA0002437477570000134

Figure BDA0002437477570000134

Figure BDA0002437477570000135

Figure BDA0002437477570000135

Figure BDA0002437477570000136

Figure BDA0002437477570000136

Figure BDA0002437477570000141

Figure BDA0002437477570000141

其中，mt和vt代表梯度一阶矩和二阶矩估计，β1和β2表示延长因子基数，gt是参数θ在t-1时刻时目标函数的梯度，

Figure BDA0002437477570000142

和

Figure BDA0002437477570000143

是mt，vt的修正，ε是一个小数值常数，η表示学习率。对Adam算法流程的简要介绍如下：Among them, m t and v t represent the gradient first moment and second moment estimation, β 1 and β 2 represent the extension factor base, g t is the gradient of the objective function of the parameter θ at time t-1,

Figure BDA0002437477570000142

and

Figure BDA0002437477570000143

is the modification of m t , v t , ε is a small numerical constant, and η is the learning rate. A brief introduction to the Adam algorithm process is as follows:

1.输入η,β1,β2,ε以及最大循环次数epoch等参数；1. Input parameters such as η, β 1 , β 2 , ε and the maximum number of cycles epoch;

2.在t＝0时，初始化参数θ0，并令一阶矩估计m0＝0，二阶矩估计v0＝0；2. When t=0, initialize the parameter θ 0 , and make the first-order moment estimate m 0 =0, and the second-order moment estimate v 0 =0;

3.更新迭代的次数：t＝t+1；3. The number of update iterations: t=t+1;

4.在训练样本集中选取m个样本{x(1),...,x(m)}，并且将其对应的目标样本记为y(i)，接着进行在θt-1时的梯度计算：

Figure BDA0002437477570000144

4. Select m samples {x (1) ,...,x (m) } in the training sample set, and denote the corresponding target samples as y (i) , and then carry out the gradient at θ t-1 calculate:

Figure BDA0002437477570000144

5.更新mt：mt＝β1×mt-1+(1-β1)×gt；5. Update m t : m t =β 1 ×m t−1 +(1−β 1 )×g t ;

6.更新vt：

Figure BDA0002437477570000145

6. Update vt :

Figure BDA0002437477570000145

7.修正

Figure BDA0002437477570000146

Figure BDA0002437477570000147

7. Correction

Figure BDA0002437477570000146

Figure BDA0002437477570000147

8.修正

Figure BDA0002437477570000148

Figure BDA0002437477570000149

8. Correction

Figure BDA0002437477570000148

Figure BDA0002437477570000149

9.更新θt：

Figure BDA00024374775700001410

循环步骤3～步骤8，直到f(θ)收敛或是达到了预先设定的最大循环次数epoch后，返回f(θ)的最优解θt。9. Update θ t :

Figure BDA00024374775700001410

Repeat steps 3 to 8 until f(θ) converges or the preset maximum cycle times epoch is reached, then return to the optimal solution θ t of f(θ).

为了加速模型的收敛时间，本实施例在采用Adam优化算法的基础上，将固定的学习率改为动态调整。学习率

Figure BDA00024374775700001411

的计算如下式：In order to speed up the convergence time of the model, this embodiment changes the fixed learning rate to dynamic adjustment based on the Adam optimization algorithm. learning rate

Figure BDA00024374775700001411

The calculation is as follows:

Figure BDA00024374775700001412

Figure BDA00024374775700001412

其中，η表示学习率初始值，epoch是代表迭代总次数，iter是目前的迭代次数，offset是训练过程中需要开始减小学习率

Figure BDA0002437477570000151

时的迭代次数。当iter小于offset时，先用一个预设的较大的η作为当前学习率，这样可以使得目标函数较快的取得一个较好的解，当iter达到offset后，逐渐的减小学习率，避免最优解在极小值附近震荡。Among them, η represents the initial value of the learning rate, epoch represents the total number of iterations, iter is the current number of iterations, and offset is the need to start reducing the learning rate during the training process.

Figure BDA0002437477570000151

number of iterations. When iter is less than offset, first use a preset larger η as the current learning rate, so that the objective function can quickly obtain a better solution. When iter reaches the offset, gradually reduce the learning rate to avoid The optimal solution oscillates around the minimum.

具体实施时，如图1所示，本实施例采用的技术方案包括以下几个关键部分与技术：During specific implementation, as shown in FIG. 1 , the technical solution adopted in this embodiment includes the following key parts and technologies:

第一部分：SAR图像预处理。本实施例采用了一种适用于SAR图像的滚动引导滤波算法对SAR进行相干斑抑制。该方法流程如图2所示。该算法具体的流程为：Part I: SAR image preprocessing. In this embodiment, a rolling-guided filtering algorithm suitable for SAR images is used to suppress coherent speckle for SAR. The method flow is shown in Figure 2. The specific process of the algorithm is as follows:

(1)首先对SAR图像进行高斯滤波，以滤除小区域的斑点与细小结构，表达式为：(1) First, Gaussian filtering is performed on the SAR image to filter out spots and small structures in small areas. The expression is:

Figure BDA0002437477570000152

Figure BDA0002437477570000152

在式(12)中，J1(p)表示像素点p经过高斯滤波后的像素值大小，N(p)代表该像素点的周围将用于滤波计算的领域，q表示这一领域中涉及到的像素点，kp是归一化系数，用于保持计算结果的范围。In Equation (12), J 1 (p) represents the pixel value of pixel p after Gaussian filtering, N(p) represents the area around the pixel that will be used for filtering calculations, and q represents the area involved in this area. To the pixel point, k p is the normalization coefficient used to maintain the range of the calculation result.

(2)在上述基础上，采用引导的方式来做后续迭代，加强图像的边缘结构，表达式为：(2) On the basis of the above, a guided method is used to do subsequent iterations to strengthen the edge structure of the image. The expression is:

Figure BDA0002437477570000153

Figure BDA0002437477570000153

式(13)中，Jt(p)表示像素点p经过第t次滤波后的像素大小；Jt-1(p)、Jt-1(q)分别表示像素点p和q经过第t-1次滤波后的像素大小；δs和δr分别表示空间尺度与距离尺度。In formula (13), J t (p) represents the pixel size of the pixel p after the t-th filtering; J t-1 (p) and J t-1 (q) represent the pixel p and q after the t-th filter, respectively. - Pixel size after 1 filter; δ s and δ r represent spatial scale and distance scale, respectively.

图3(a)～图3(e)为经典滤波算法与滚动引导滤波算法结果示意图；图3(a)为原始图像，图3(b)为双边滤波图像，图3(c)为引导滤波图像，图3(d)为非线性扩散滤波图像，图3(e)为滚动引导滤波图像。Figures 3(a) to 3(e) are schematic diagrams of the results of the classical filtering algorithm and the rolling guided filtering algorithm; Figure 3(a) is the original image, Figure 3(b) is the bilateral filtered image, and Figure 3(c) is the guided filtering image, Fig. 3(d) is the nonlinear diffusion filtered image, and Fig. 3(e) is the scroll-guided filtered image.

第二部分：数据集制作，具体步骤如下：The second part: data set production, the specific steps are as follows:

(1)读入已配准的异源图像对，采用SIFT方法对每一幅光学图像提取所有可能的SIFT特征点，并排除掉两个特征点之间的距离小于d的像素点，利用d来调整在图像中获取到的SIFT特征点的稠密度，本实施例中设置d＝20；(1) Read the registered heterologous image pair, use the SIFT method to extract all possible SIFT feature points for each optical image, and exclude the pixels whose distance between the two feature points is less than d, and use d to adjust the density of the SIFT feature points obtained in the image, in this embodiment, set d=20;

(2)进行筛选后，以每一个选中的特征点为中心，截取大小为256×256的图像块，若所选区域超出图像边界则丢弃该点。随后在对应的SAR图像上以同一坐标位置的像素点为中心截取相同大小的图像块。把光学图像和SAR图像以相同的命名分别保存到两个对应的文件夹中；(2) After screening, take each selected feature point as the center, intercept an image block with a size of 256 × 256, and discard the point if the selected area exceeds the image boundary. Then, on the corresponding SAR image, an image block of the same size is intercepted with the pixel point at the same coordinate position as the center. Save the optical image and SAR image to two corresponding folders with the same name;

(3)待几组异源图像都处理完后，对光学和SAR图像的两个文件夹中的所有图像进行旋转、镜像、翻转等操作以完成数据扩增；(3) After several sets of heterologous images are processed, all images in the two folders of optical and SAR images are rotated, mirrored, and flipped to complete data amplification;

(4)将光学和SAR图像两个文件夹中的图像合并为一张512×256的图像，并保存在另一个文件夹中；(4) Combine the images in the two folders of optical and SAR images into a 512×256 image and save it in another folder;

(5)将上述文件夹中的样本按照80％/20％的比例随机分为训练集和测试集，最后，本实施例中采用的训练样本数量为10464个，训练数据集为8372个，测试数据集为2092个；(5) The samples in the above folder are randomly divided into training set and test set according to the ratio of 80%/20%. Finally, the number of training samples used in this embodiment is 10,464, the training data set is 8,372, and the test set is 10,464. The dataset is 2092;

第三部分：搭建图像转换网络框架，所述的图像转换网络，包括生成器网络和判别器网络，其中：The third part: Building an image conversion network framework, the image conversion network includes a generator network and a discriminator network, where:

(1)构建生成器网络：(1) Build the generator network:

本实施例中的条件式生成对抗网络中的生成器是采用了U-Net网络结构的思想，网络结构包括了编码器模块和解码器模块，编码器模块包括了8层，每一层都为double(conv+bn+lrelu)+shortcut的结构，卷积核数目从64逐层倍增直到512后不变，bn是指批量归一化优化，lrelu是指激活函数为LeakyReLU，shortcut是指残差网络中的“捷径”。解码器模块也是8层，编解码器层数相同的单元具有相同的卷积核数目，区别在于解码器中每一个单元的结构为conv+bn+relu，relu表示激活函数使用的是ReLU函数，并且，对编解码模块之间对应的层进行拼接。此外，每一卷积操作的卷积核大小都为3×3，每一个单元之间都接有一层2*2的最大池化层。The generator in the conditional generative adversarial network in this embodiment adopts the idea of the U-Net network structure. The network structure includes an encoder module and a decoder module. The encoder module includes 8 layers, and each layer is The structure of double(conv+bn+lrelu)+shortcut, the number of convolution kernels is multiplied from 64 layer by layer until 512, which remains unchanged. bn refers to batch normalization optimization, lrelu refers to the activation function of LeakyReLU, and shortcut refers to residual error. "Shortcuts" in the web. The decoder module is also 8 layers. Units with the same number of encoder and decoder layers have the same number of convolution kernels. The difference is that the structure of each unit in the decoder is conv+bn+relu, and relu indicates that the activation function uses the ReLU function. And, the corresponding layers between the codec modules are spliced. In addition, the size of the convolution kernel of each convolution operation is 3 × 3, and a 2 × 2 max pooling layer is connected between each unit.

(2)构建判别器网络：(2) Build the discriminator network:

本实施例中的判别器网络结构是由五层卷积构成。前四层卷积用于对样本进行特征提取，卷积核数目从64逐层递增，卷积核大小都为3×3，步长为2。最后一层卷积用于将特征映射到一维输出，将Sigmoid函数作为激活函数。并且在前四层卷积后都进行了批量归一化(Batch Normalization)处理，激活函数为LeakyReLU，取值为0.2。此外，判别器网络中还采用了PatchGAN结构。PatchGAN的总体思想是将图像划分为N×N个固定大小的图像块，判别器分别对每一个块的真假性作出判断，最后对一张图像中所得的响应求平均值后作为输出结果。利用PatchGAN可以更好的判断图像的局部特征，在本实施例中，将patch size设置为70*70。The discriminator network structure in this embodiment is composed of five layers of convolution. The first four layers of convolution are used for feature extraction of samples. The number of convolution kernels increases layer by layer from 64. The size of the convolution kernels is 3×3 and the stride is 2. The last layer of convolution is used to map the features to a one-dimensional output, with the sigmoid function as the activation function. And batch normalization is performed after the first four layers of convolution, the activation function is LeakyReLU, and the value is 0.2. In addition, the PatchGAN structure is also used in the discriminator network. The general idea of PatchGAN is to divide the image into N×N fixed-size image blocks, the discriminator judges the authenticity of each block separately, and finally averages the responses obtained in an image as the output result. The local features of the image can be better judged by using PatchGAN. In this embodiment, the patch size is set to 70*70.

第四部分：训练模型。将第二部分得到的生成对抗网络的训练样本对第三部分构建的图像转换网络进行训练。Part 4: Train the model. The training samples of the generative adversarial network obtained in the second part are used to train the image conversion network constructed in the third part.

本发明按照下式的损失函数训练生成对抗网络：The present invention trains the generative adversarial network according to the loss function of the following formula:

Figure BDA0002437477570000171

Figure BDA0002437477570000171

Figure BDA0002437477570000181

Figure BDA0002437477570000181

G'＝arg minG maxD LcGAN+λL1 (16)′G'=arg min G max D L cGAN +λL 1 (16)′

将真实SAR图像y作为约束条件，随机噪声记为z，x为输入的光学图像，其中x,y服从pdata(x,y)数据分布，随机噪声z服从pz(z)数据分布。其中LcGAN(G,D)表示生成器和判别器的对抗损失约束，

Figure BDA0002437477570000182

表示生成器的图像块与真实图像块之间的像素级约束，D(x,y)代表了判别器对x、y的匹配预测，G(x,z)代表输入光学图像和噪声后生成器的输出图像，D(x,G(x,z))表示判别器对x、G(x,z)的匹配预测，λ表示引入的L1损失前的系数，本实施例中设置λ＝100。生成器训练目的是使得LcGAN最小化，判别器的训练目的是使得LcGAN最大化。Taking the real SAR image y as the constraint condition, the random noise is denoted as z, and x is the input optical image, where x, y obey the data distribution of p data (x, y), and the random noise z obey the data distribution of p z (z). where L cGAN (G, D) represents the adversarial loss constraints of the generator and discriminator,

Figure BDA0002437477570000182

represents the pixel-level constraints between the generator's image patch and the real image patch, D(x, y) represents the discriminator's matching prediction for x, y, and G(x, z) represents the input optical image and the post-noise generator The output image of , D(x, G(x, z)) represents the matching prediction of x, G(x, z) by the discriminator, λ represents the coefficient before the introduced L 1 loss, in this example, λ=100 . The generator is trained to minimize LcGAN , and the discriminator is trained to maximize LcGAN .

进一步的，本实施例采用Adam优化算法对目标函数进行优化。涉及到的公式有：Further, in this embodiment, the Adam optimization algorithm is used to optimize the objective function. The formulas involved are:

mt＝β1×mt-1+(1-β1)×gt (17)′m t =β 1 ×m t-1 +(1-β 1 )×g t (17)′

Figure BDA0002437477570000183

Figure BDA0002437477570000183

Figure BDA0002437477570000184

Figure BDA0002437477570000184

Figure BDA0002437477570000185

Figure BDA0002437477570000185

Figure BDA0002437477570000186

Figure BDA0002437477570000186

其中，mt和vt代表梯度的一阶矩和二阶矩估计，β1和β2表示延长因子基数，gt是参数θ在t-1时刻时目标函数的梯度，

Figure BDA0002437477570000187

和

Figure BDA0002437477570000188

是mt，vt的修正，ε是一个小数值常数，η表示学习率。对Adam算法流程的简要介绍如下：Among them, m t and v t represent the first-order moment and second-order moment estimation of the gradient, β 1 and β 2 represent the extension factor base, g t is the gradient of the objective function of the parameter θ at time t-1,

Figure BDA0002437477570000187

and

Figure BDA0002437477570000188

is the modification of m t , v t , ε is a small numerical constant, and η is the learning rate. A brief introduction to the Adam algorithm process is as follows:

Figure BDA0002437477570000189

Figure BDA0002437477570000189

Figure BDA0002437477570000191

Figure BDA0002437477570000191

为了加速模型的收敛时间，本实施例在采用Adam优化算法的基础上，将固定的学习率改为动态调整。学习率

Figure BDA0002437477570000192

的计算如下式：In order to speed up the convergence time of the model, this embodiment changes the fixed learning rate to dynamic adjustment based on the Adam optimization algorithm. learning rate

Figure BDA0002437477570000192

The calculation is as follows:

Figure BDA0002437477570000193

Figure BDA0002437477570000193

其中，η表示学习率初始值，epoch是代表迭代总次数，iter是目前的迭代次数，offset是训练过程中需要开始减小学习率

Figure BDA0002437477570000194

时的迭代次数。当iter小于offset时，先用一个预设的较大的η作为当前学习率，这样可以使得目标函数较快的取得一个较好的解，当iter达到offset后，逐渐的减小学习率，避免最优解在极小值附近震荡。Among them, η represents the initial value of the learning rate, epoch represents the total number of iterations, iter is the current number of iterations, and offset is the need to start reducing the learning rate during the training process.

Figure BDA0002437477570000194

number of iterations. When iter is less than offset, first use a preset larger η as the current learning rate, so that the objective function can quickly obtain a better solution. When iter reaches the offset, gradually reduce the learning rate to avoid The optimal solution oscillates around the minimum.

第五部分：图像转换效果测试。下面结合仿真实验对本实施例的效果做进一步描述。The fifth part: Image conversion effect test. The effect of this embodiment will be further described below in conjunction with a simulation experiment.

1、仿真实验环境：1. Simulation experiment environment:

(1)计算机配置：(1) Computer configuration:

系统类型：Ubuntu 64位操作系统。System type: Ubuntu 64-bit operating system.

显卡：NVIDIA GEFORCE GTX 1050tiGraphics Card: NVIDIA GEFORCE GTX 1050ti

(2)实验环境及框架(2) Experimental environment and framework

框架：TensorFlow-1.7.0Framework: TensorFlow-1.7.0

Python版本：python3.5Python version: python3.5

2、实验内容与结果分析：2. Experiment content and result analysis:

图4(a)、图4(b)是对上海某区域的光学图像和SAR图像进行截取的图像，大小为1024×1024，两幅图像是已配准好的异源图像，数据集中包括了对上述图像进行切块的训练样本和测试样本。Figures 4(a) and 4(b) are the captured images of the optical image and SAR image of a certain area in Shanghai, with a size of 1024×1024. The two images are registered heterologous images. The data set includes The training samples and test samples for dicing the above images.

实验中将现有的图像转换的网络框架与本实施例进行比较，将包含未经预处理的SAR图像的数据集对pix2pix、cycleGAN以及本实施例进行训练，用相同测试集进行方法评价，分别记为A1,A2,A3，同时将包含经预处理的SAR图像的数据集对本实施例提出的模型进行训练，并用测试集进行评价，记为A4。下表为评价结果：In the experiment, the existing network framework for image conversion is compared with this example, and the data sets containing unpreprocessed SAR images are trained on pix2pix, cycleGAN and this example, and the same test set is used for method evaluation, respectively. It is denoted as A1, A2, and A3. At the same time, the model proposed in this embodiment is trained on the data set containing the preprocessed SAR images, and the test set is used for evaluation, which is denoted as A4. The following table shows the evaluation results:

表1仿真实验测试集的相似度评价Table 1. Similarity evaluation of test set of simulation experiment

Figure BDA0002437477570000201

Figure BDA0002437477570000201

Figure BDA0002437477570000211

Figure BDA0002437477570000211

从表1中可看出，本实施例提出的图像转换框架能有效的提高图像的SSIM结构相似度指数。本实施例的方法特征提取能力更强，网络收敛速度更快，在第200次左右就能生成较好效果的伪SAR图像，并且SSIM指数较高，这说明本实施例中提出的框架能有效的实现光学图像到伪SAR图像的转换。It can be seen from Table 1 that the image conversion framework proposed in this embodiment can effectively improve the SSIM structure similarity index of the image. The method of this embodiment has stronger feature extraction ability, faster network convergence speed, and can generate a pseudo-SAR image with better effect at about the 200th time, and the SSIM index is higher, which shows that the framework proposed in this embodiment can be effective. The realization of the conversion of optical image to pseudo SAR image.

图5(a)、图5(b)、图5(c)分别是第一、第二、第三光学图像，图5(d)、图5(e)、图5(f)分别是第一、第二、第三真实的SAR图像，图5(g)、图5(h)、图5(i)分别是生成器生成的第一、第二、第三伪SAR图像。Figures 5(a), 5(b), and 5(c) are the first, second, and third optical images, respectively, and Figures 5(d), 5(e), and 5(f) are the first, second, and third optical images, respectively. The first, second, and third real SAR images, Figure 5(g), Figure 5(h), and Figure 5(i) are the first, second, and third pseudo-SAR images generated by the generator, respectively.

应当理解的是，本说明书未详细阐述的部分均属于现有技术。It should be understood that the parts not described in detail in this specification belong to the prior art.

虽然以上结合附图描述了本发明的具体实施方式，但是本领域普通技术人员应当理解，这些仅是举例说明，可以对这些实施方式做出多种变形或修改，而不背离本发明的原理和实质。本发明的范围仅由所附权利要求书限定。Although the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, those skilled in the art should understand that these are only examples, and various modifications or changes may be made to these embodiments without departing from the principles and principles of the present invention and substance. The scope of the present invention is limited only by the appended claims.