CN114037771B - A few-photon imaging method based on deep learning - Google Patents

The application discloses a low-photon imaging method based on deep learning, which is a low-photon counting scattering reconstruction method. The training database under different low photon count conditions can be generated for training of the deep learning neural network by sampling the training target only once under the high photon count conditions and then using a simulation algorithm. After training, the network parameters are fixed, and the neural network can be used for reconstructing scattering images. And taking the acquired new scattered image with low photon count as the input of the deep learning network, and outputting the new scattered image with low photon count as a clear image. The method simplifies the complex sampling flow of the deep learning method in the field of weak light imaging, simultaneously reduces the requirement on the sensitivity of sampling equipment, and saves the equipment cost and the time cost. And the method does not need resampling to generate a training database aiming at different low photon count restored samples, so that the robustness of the deep learning method in the low photon count imaging field is improved.

Few-photon imaging method based on deep learning

Technical Field

The application relates to the field of computational imaging and image processing, in particular to a few-photon imaging method based on deep learning.

Background

The reconstruction of low photon count scatter imaging has important applications in real life, in various aspects such as night detection, astronomical remote sensing, and biomedical imaging.

Conventional imaging devices typically require 10 ¹² photons per pixel for acquisition of a high quality picture, and an average of 10 ⁵ photons per pixel for acquisition of a high quality array detector. However, in many practical low-light scenarios, the photon count value is difficult to achieve. Especially in very low light and exposure time limited situations, the number of photons available can be even only a few, and it is this low photon count scenario that the present application focuses on.

Under the condition of sufficient photon number, the signal detected by the common detector is analog quantity, and the optical signal contains a large number of photons, and the photons are overlapped to obtain the light intensity obtained by detection. The manner of acquiring an image by recording the light intensity of each position on the detection target is called an analog manner. However, as the light intensity of the detection target decays, the light intensity becomes a pulse signal gradually, and especially when the light intensity decays to a single photon condition, the signal becomes a discrete pulse signal with a small number of pulses. Whereas single photons are generally considered as the limit of detection, being the smallest unit of energy that cannot be segmented further. Thus, under low photon count conditions, the signal exhibits a particle characteristic. In this case, the light is recorded as a single photon, its spatial position is determined while the single photon is detected, and two-dimensional photon counting detection is performed, which is the basis of photon counting imaging.

A Single Photon Camera (SPC) is a two-dimensional array detector, each pixel of which corresponds to an independent point detector. The sensitivity thereof can break through the shot noise limit, and is therefore widely used for low photon count detection in recent years. In particular, such cameras do not directly capture a gray scale image, where each pixel records the number of photons detected, and the randomness of photon emission and detection is a major source of noise in the imaging system, with noise distribution conforming to poisson distribution under low photon count conditions. And normalizing the photon number to obtain the required gray level image. However, the packaging process of the single photon detector is difficult, the pixel spacing is large, and only a low-resolution image can be obtained. But this also corresponds to down-sampling the acquired high resolution map.

Conventional physics research problems have been combined with deep learning to take advantage, especially in image processing. Traditional scatter imaging only focuses on a single problem of scattering, but does not focus on the complex situation that both the low photon count and the scattering exist simultaneously, and the imaging is more difficult in the situation that information is lost due to the weakening of light intensity. And the traditional scattering imaging method needs to scan and detect the target for multiple times, meanwhile, the algorithm depends on a detection model, and different processing methods are needed for detection under different scenes, and especially for a micro-thick scattering medium, the traditional method is difficult to recover the image. The experimental operation and algorithm are complex, the detection is carried out for many times, and the image recovery quality is general. In recent years, a deep learning method is widely applied to the field of low photon count scattering imaging, and good imaging results are obtained. However, the deep learning method in the prior art has a plurality of defects, such as the need of sampling a large amount of training data for pre-training to have imaging capability. However, photon counting of the training data needs to be consistent with the scatter plot to be imaged, so that the deep learning network can correctly learn the features of the scatter pattern. When the photon count of the scatter plot to be imaged changes significantly, prior art deep learning methods have had to resample the training database of photon count matches to preserve the learning capabilities of the neural network. Therefore, the method has extremely high requirements on the sensitivity of the sampling equipment and the long-time stability of the ambient illumination condition, and has low efficiency, low robustness and no benefit to engineering. Accordingly, those skilled in the art have been working to develop a low-photon imaging method based on deep learning to solve the technical problems existing in the prior art.

Disclosure of Invention

In order to achieve the above purpose, the application provides a depth learning-based few-photon imaging method, which specifically comprises the following steps:

step1, adjusting a scattering imaging device to enable the scattering imaging device to acquire a scattering map;

step2, adjusting the light source intensity and the exposure time to obtain a scattering map with high photon count;

step 3, generating a scattering map with low photon count for training through a poisson algorithm;

Step 4, taking the plurality of low photon count scattering graphs generated in the step 3 as a training set for training the neural network, and determining the neural network for scattering imaging reconstruction by fixing parameters after training;

Step 5, adjusting the light source intensity and the exposure time, and obtaining a low photon counting scattering diagram to be recovered under weak light;

And 6, reconstructing and imaging the low photon counting scattering diagram to be recovered, which is obtained in the step 5, by adopting the neural network for scattering imaging reconstruction.

Further, in step 1, the sampling formula of the scattering map satisfies the following formula:

Wherein, Representing the average photon number per unit time, hν represents the energy of one photon. P (x, y) represents the scatter image sampled during the exposure time Δt. Possion [ ] represents the poisson process of photon detection.

Further, in step 1, the total photon number n _dj detected by each pixel in each scattering graph conforms to the poisson random distribution probability.

Further, the total photon number n _dj satisfies the following equation:

Wherein, Representing the average photon number at the exposure time Δt.

Further, in step 3, simulation is performed to generate a scattering map with the same photon count as the low photon count scattering map to be recovered by using the exposure time Δt of each pixel and the total photon number n _dj, so as to become the low photon count scattering map for training.

Further, the step 3 specifically includes the following steps:

Step 3.1, selecting a series of pictures N _i, i=1, 2, 3;

Step 3.2, sampling each of N _i under high photon count for exposure time t to obtain a scattering map S _i;

Step 3.3, counting photon number P _ij for each pixel point in the scattering map S _i;

Step 3.4 taking the photon number P _ij as n _dj, the exposure time t as Δt, and P _ij/t as Substitution into the formula of claim 4;

Step 3.5, narrowing Δt, randomly generating a plurality of random pixel values according to the formula as defined in claim 4;

step 3.6, forming a scattering diagram of the low photon count for training by using a matrix formed by all the random pixel values.

Further, in step 3.5, Δt is reduced to 0.001 times the original Δt.

Further, in step 1, the scattering imaging device includes a light source, a spatial light modulation device, a lens, frosted glass, a detector and a processor module, the modulated light is emitted from the light source, sequentially passes through the spatial light modulation device, the lens and the frosted glass, and reaches the detector, and the detector is in data connection with the processor module, so that the processor module performs imaging recovery on the data collected by the detector.

Further, the spatial light modulation device is used for loading the target image.

Further, the detector is a single photon camera with sensitivity capable of breaking the shot noise limit and used for counting photon numbers.

Compared with the prior art, the technical scheme of the application has at least the following technical advantages:

1. The method is suitable for the condition of low photon counting, and certain effective information can be obtained through counting imaging when the light is weak and the particle characteristics are presented;

2. The algorithm simplifies the flow of sampling training samples when the depth learning method carries out scatter imaging, and a large number of weak light scattering images can be generated rapidly by sampling a small number of high photon count scattering images;

3. The algorithm avoids the operation that the deep learning method needs to resample the training set when the photon count change of the scattering graph to be recovered is large, and the robustness is stronger;

4. the algorithm reduces the requirement on the sensitivity of the camera and the requirement that the ambient light is stable for a long time when the low photon count scattering imaging training sample is acquired;

5. The training samples are more, and the quality of the reconstructed image of the low photon count scattering imaging by the deep learning method is improved.

The conception, specific structure, and technical effects of the present application will be further described with reference to the accompanying drawings to fully understand the objects, features, and effects of the present application.

Drawings

FIG. 1 is a schematic diagram of a scatter imaging system according to an embodiment of the present application;

FIG. 2 is a diagram of one of the target originals in a training set, with pixels 32 x 32, and a high photon count condition of 350 photons per pixel on average, in accordance with one embodiment of the present application;

FIG. 3 is a scatter plot of the image of FIG. 2 obtained after scattering under high photon count conditions, with pixels 32 x 32;

FIG. 4 is a plot of low photon count scatter from the Poisson simulation of the plot of FIG. 3 using equation (2), with pixels 32 x 32, and low photon count conditions of 0.3 photons per pixel on average;

FIG. 5 is another low photon count scatter plot generated from the poisson simulation of the high photon count scatter plot of FIG. 3 using equation (2), with pixels 32 x 32, and low photon count conditions of an average of 1.5 photons per pixel;

FIG. 6 is a diagram of one of the target originals in a test set that does not intersect the training set, with pixels 32 x 32, in accordance with one embodiment of the present application;

FIG. 7 is a scatter plot of the image of FIG. 6 obtained by scattering an average of 0.3 photons per pixel at a low photon count condition, the pixels being 32X 32;

FIG. 8 is a scatter plot of the low photon count scatter image of FIG. 6 obtained by scattering an average of 1.5 photons per pixel at a low photon count condition, with pixels 32X 32;

FIG. 9 is a clear image of the low photon count scatter image of FIG. 7 reconstructed from a trained neural network, with pixels 32 x 32;

fig. 10 is a clear image of the low photon count scatter image of fig. 8 reconstructed from a trained neural network, with pixels 32 x 32.

Detailed Description

The following description of the preferred embodiments of the present application refers to the accompanying drawings, which make the technical contents thereof more clear and easy to understand. The present application may be embodied in many different forms of embodiments and the scope of the present application is not limited to only the embodiments described herein.

In the drawings, like structural elements are referred to by like reference numerals and components having similar structure or function are referred to by like reference numerals. The dimensions and thickness of each component shown in the drawings are arbitrarily shown, and the present application is not limited to the dimensions and thickness of each component. The thickness of the components is exaggerated in some places in the drawings for clarity of illustration.

Example 1

The embodiment provides a few-photon imaging method based on deep learning, which specifically comprises the following steps:

And step 1, adjusting the scattering imaging equipment to enable the scattering imaging equipment to acquire a scattering map.

And step2, adjusting the light source intensity and the exposure time, and obtaining a scattering graph with high photon count.

The scatter imaging apparatus used in this embodiment includes, as shown in fig. 1, a light source 1, a spatial light modulation device 2, a lens 3, ground glass 4, a detector 5, and a processor module 6. The modulated light is emitted from the light source 1, passes through the spatial light modulation device 2, the lens 3, and the frosted glass 4 in order, and reaches the detector 5. The lens 3 is used for converging the reflected light of the spatial light modulator 2, the frosted glass 4 is used for scattering the light converged by the lens 3, the detector 5 is used for detecting scattered light intensity and counting photon numbers, and the processor module 6 is used for carrying out data processing and scattered imaging recovery on the data acquired by the detector 5. Preferably, the detector 5 may perform light field intensity value acquisition at the same frequency as the spatial light modulation device 2. Preferably, the light intensity of the light source 1 is adjustable. Preferably, the detector 5 is a single photon camera for counting the number of photons whose sensitivity can break the shot noise limit.

The specific steps of the scattering imaging device for acquiring the scattering map are as follows:

(1) The spatial light modulation device 2 loads a target picture, and light emitted by the light source 1 irradiates the target picture of the spatial light modulator 2;

(2) The lens 3 condenses the reflected light of the object in the spatial light modulator 2;

(3) The ground glass 4 diffuses the light collected by the lens 3;

(4) The processor module 6 collects the photon number of the reflected light detected by the detector 5;

(5) The gray map is obtained by normalizing the number of photons obtained by the processor 6.

In this example, the SPC exposure time was adjusted to 1500 microseconds for a light intensity of 97 nanowatts from the light source, and 3000 raw clear images and high photon count scatter images were obtained.

And 3, generating a scattering map with low photon count for training through a poisson algorithm.

The sampling process under the condition of low photon count is a poisson process, and the light intensity distribution rule of each pixel point accords with poisson law. Although detection under conditions of low photon counts is random due to poisson noise, detection with sufficient photons is relatively stable. If any other noise is ignored, the sampling formula of the scatter image can be given by formula (1) as follows:

Wherein, Representing the average photon number per unit time, hν represents the energy of one photon. P (x, y) represents the scatter image sampled during the exposure time Δt. Possion [ ] represents the poisson process of photon detection.

If the total number of photons detected by each pixel of each image is expressed as an integer n _dj, the average number of photons per pixel under the condition of exposure time Δt can be expressed asSince n _dj meets the poisson random distribution probability, the distribution formula is given by the following formula (2):

Once the average photon number per unit time is known based on equation (2) And the exposure time deltat, any possible distribution of the photon count scattering pattern on the detector can be simulated. It is well known that in the higher illumination range, the variance introduced by photon shot noise is a small percentage of the total signal. Thus, the photon count value at each pixel point of the scatter plot can be approximated by using the measured scatter plot of high photon counts as the average photon count for that pixel point. Based on this property and the properties of the deep learning neural network (DL for short), there is no need to measure the real training samples at the same photon count level as the test set. Training data at a large number of different low photon count levels can be simulated by measuring only one scatter plot at high photon count for each training target.

In view of the problems existing in the prior art, the method provides a new idea for acquiring training data under low photon number. During the training phase of the network, only a small number of high photon count spots can be sampled. Since the sampling time Δt of the SPC is known, the total number of photons n _dj of the scatter plot can also be easily calculated. Thus, Δt can be flexibly adjusted, using a poisson simulation algorithm to simulate a large number of scatter plots with the same photon count as the test data. The method can skip the complex process of acquiring the training set under the condition of low photon number, reduces the requirements on the precision of sampling equipment and the stability of the sampling environment, and is more beneficial to practical application. On the other hand, after the scattering image of the target is obtained by detection of the single photon camera, the detected weak light scattering image is directly reconstructed through an algorithm, so that the complexity of experimental operation can be reduced, the algorithm is independent of a model, and for thin and thick scattering media, the same algorithm can be used for processing, and the generalization capability of post-processing is improved.

It is furthermore worth noting that imaging after passing through a scattering medium is a relatively complex one in the imaging scene. For imaging in other common scenes, there is a sophisticated physical model that allows its image to be calculated under high photon count conditions without the need for actual sampling. At the moment, the simulation algorithm is combined with the simulation algorithm, so that a large number of images with low photon count can be generated for training of the deep learning network under the condition of not actually sampling. This has very important engineering significance. Therefore, the technology of the application has important significance in the fields of military investigation, detection, remote sensing, biomedical imaging and the like.

In this embodiment, knowing the total exposure time Δt and the total photon number n _dj for each pixel, a scatter plot that is the same as the photon count for a large number of scatter plots to be recovered can be generated in a simulated manner as a training dataset. The method comprises the following specific steps:

Step 3.1, selecting a series of pictures N _i, i=1, 2, 3;

Step 3.2, sampling each of N _i under high photon count for exposure time t to obtain a scattering map S _i;

Step 3.3, counting photon number P _ij for each pixel point in the scattering map S _i;

Step 3.4 taking the photon number P _ij as n _dj, the exposure time t as Δt, and P _ij/t as Substituting into formula (2);

Step 3.5, reducing the delta t, and randomly generating a plurality of random pixel values according to the formula (2);

Step 3.6 matrix of all random pixel values constitutes a scatter plot of low photon counts for training.

In this embodiment, specifically, assuming that the photon number of the target low-light image is 0.001 times that of the high photon count, it is only necessary to reduce Δt in the formula (2) to 0.001 times that of the original one, and then randomly generate a value for each pixel point according to the probability distribution in the formula (2) as the pixel value generated by simulation. The matrix of all pixel values is the corresponding low photon count scatter plot value generated by the simulation. Because the value of each pixel point is randomly generated according to the probability, the same high photon count scattering map can be simulated for multiple times to generate a plurality of low photon count scattering maps, thereby avoiding the complex repeated sampling process. In addition, the photon count value of the low photon count generated by simulation can also be changed by adjusting Δt to generate, thereby increasing the practicability and the robustness of the method.

In this embodiment, in order to be able to construct a data set for neural network training, 3000 high photon count scatter images were acquired and 30000 low photon technique scatter images were generated by poisson algorithm simulation.

And 4, taking the plurality of low photon count scattering graphs generated in the step 3 as a training set for training the neural network, and determining the neural network for scattering imaging reconstruction by fixing parameters after training.

In this embodiment, the weak light scattering image is preferably used as an input of the neural network, and is output as a clear image reconstructed by the model, and parameters are automatically adjusted by minimizing a mean square error between the clear image reconstructed by the model and the original clear image, so as to optimize the neural network model.

And 5, adjusting the light source intensity and the exposure time, and obtaining a low photon counting scattering diagram to be recovered under weak light.

In particular, in this embodiment, the exposure time Δt is adjusted to 2 μs, and the intensity of the light emitted from the light source is reduced to obtain a low photon count scattering map to be recovered in weak light

And 6, reconstructing and imaging the low photon counting scattering diagram to be recovered obtained in the step 5 by adopting a neural network for scattering imaging reconstruction, and outputting the high-quality image to be reconstructed.

Fig. 2-10 are scatter plots of a few photon counting scatter image reconstruction using the method of the present embodiment, wherein,

Fig. 2 shows one of the target originals in the training set in this embodiment, the pixels are 32×32, and the high photon count condition is an average of 350 photons per pixel.

Fig. 3 is a scattering diagram of the image of fig. 2 obtained after scattering under high photon count conditions, with pixels 32 x 32.

Fig. 4 is a low photon count scatter plot generated by poisson simulation of the high photon count scatter plot of fig. 3 using equation (2), with pixels 32 x 32, and low photon count conditions of 0.3 photons per pixel on average.

Fig. 5 is another low photon count scatter plot generated by poisson simulation of the high photon count scatter plot of fig. 3 using equation (2), with pixels 32 x 32, and low photon count conditions of an average of 1.5 photons per pixel.

Fig. 6 shows one of the target originals in the test set that does not intersect the training set in this embodiment, and the pixels are 32×32.

Fig. 7 is a scatter plot of the image of fig. 6 obtained after scattering at a low photon count condition of 0.3 photons per pixel on average, with pixels 32 x 32.

Fig. 8 is a scatter plot of the low photon count scatter image of fig. 6 obtained after scattering at a low photon count condition of an average of 1.5 photons per pixel, with pixels 32 x 32.

Fig. 9 is a clear image of the low photon count scatter image of fig. 7 reconstructed from a trained neural network, with pixels 32 x 32.

Fig. 10 is a clear image of the low photon count scatter image of fig. 8 reconstructed from a trained neural network, with pixels 32 x 32.

The foregoing describes in detail preferred embodiments of the present application. It should be understood that numerous modifications and variations can be made in accordance with the concepts of the application without requiring creative effort by one of ordinary skill in the art. Therefore, all technical solutions which can be obtained by logic analysis, reasoning or limited experiments based on the prior art by the person skilled in the art according to the inventive concept shall be within the scope of protection defined by the claims.