CN112200726B - Urinary sediment visible component detection method and system based on lensless microscopic imaging - Google Patents

The invention discloses a urinary sediment visible component detection method and system based on lensless microscopic imaging, the method adopts lensless microscopic imaging to replace a traditional optical microscope, and simplifies lighting and imaging light paths, so that the size of microscopic equipment is miniaturized, the weight is light, the cost is low, the operation is simple, and the medical cost is further saved; the method adopts an artificial intelligence algorithm to carry out phase recovery and super-resolution reconstruction on the holographic intensity image acquired by the lens-free microscopic imaging, realizes the microscopic imaging with wide field of view and ultrahigh resolution, greatly shortens the time of the microscopic imaging, improves the imaging speed of the microscope, improves the imaging quality of the microscope, and can be applied to clinical medical examination. In addition, the invention adopts a deep neural network to concentrate the detection, the segmentation and the identification of the urinary sediment tangible components in one model for multi-task learning, and realizes the full-automatic end-to-end detection, the segmentation and the identification of the urinary sediment tangible components.

Urinary sediment visible component detection method and system based on lens-free microscopic imaging

Technical Field

The invention relates to the technical field of microscopic imaging and the technical field of medical image processing, in particular to a urinary sediment visible component detection method and system based on lensless microscopic imaging.

Background

The urine routine examination generally comprises examination of color, transparency, specific gravity, pH value, protein, sugar and urine sediment formed components of urine, wherein the examination of the urine sediment formed components is one of routine examination items in hospitals, and some components in the urine sediment have definite pathological significance, such as bacteria, red blood cells, white blood cells and the like, have very important diagnosis values, can help clinicians to know changes of all parts of the urinary system, and play an important role in assisting positioning diagnosis, differential diagnosis and prognosis judgment of urinary system diseases.

The conventional urinary sediment microscopic examination analysis method is to collect urinary sediment images through a microscope and then observe and manually identify the urinary sediment images. On the one hand, the workload of manual observation and analysis is large, the operation is complex, the influence of human subjective factors is easy to cause the problem of missed detection and false detection. On the other hand, in order to obtain a high-resolution, wide-field microscopic image, urinary sediment microscopy must employ a high-magnification microscopic imaging system, and the field of view and the depth of field are enlarged by mechanical scanning, image stitching and focal length adjustment, which not only complicates the imaging process, but also significantly increases the overall cost of the microscopic imaging system.

Disclosure of Invention

The invention provides a urinary sediment visible component detection method and system based on lens-free microscopic imaging, which are used for overcoming the defects of manual analysis, complicated imaging process, high cost and the like in the prior art.

In order to achieve the purpose, the invention provides a urinary sediment visible component detection method based on lens-free microscopic imaging, which comprises the following steps:

collecting a urinary sediment holographic intensity map by using a lens-free microscopic imaging mode;

inputting the holographic intensity map into a trained urinary sediment visible component detection model, wherein the urinary sediment visible component detection model comprises a first image conversion network, a second image conversion network and an image classification network;

carrying out phase recovery and automatic focusing on the holographic intensity image by using a first image conversion network to obtain an original microscopic image;

performing super-resolution reconstruction on the original microscopic image by using a second image conversion network to obtain a super-resolution microscopic image;

and obtaining the type and the quantity of the formed components of each urinary sediment by using an image classification network according to the super-resolution microscopic image.

In order to achieve the above object, the present invention further provides a urinary sediment visible component detection system based on lens-free microscopic imaging, comprising:

the image acquisition module is used for acquiring a urinary sediment holographic intensity map by using a lensless microscopic imaging mode;

the urine sediment visible component detection module is used for inputting the holographic intensity map into a trained urine sediment visible component detection model, and the urine sediment visible component detection model comprises a first image conversion network, a second image conversion network and an image classification network; carrying out phase recovery and automatic focusing on the holographic intensity image by using a first image conversion network to obtain an original microscopic image; performing super-resolution reconstruction on the original microscopic image by using a second image conversion network to obtain a super-resolution microscopic image; and obtaining the type and the quantity of the formed components of each urinary sediment by using an image classification network according to the super-resolution microscopic image.

To achieve the above object, the present invention further provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method when executing the computer program.

To achieve the above object, the present invention further proposes a computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method described above.

Compared with the prior art, the invention has the beneficial effects that:

according to the urine sediment visible component detection method based on lensless microscopic imaging, lensless microscopic imaging is adopted to replace a traditional optical microscope, an illumination and imaging light path is simplified, an expensive and heavy optical lens is abandoned, the size of microscopic equipment is miniaturized, the weight is light, the cost is low, the operation is simplified, and the medical cost is further saved. Meanwhile, the invention adopts an artificial intelligence algorithm to carry out phase recovery and super-resolution reconstruction on the holographic intensity image acquired by the lens-free microscopic imaging, realizes the microscopic imaging with wide field of view and ultrahigh resolution, does not need to rely on mechanical scanning, image splicing and focusing adjustment to enlarge the field of view and the depth of field, greatly shortens the time of the microscopic imaging, improves the imaging speed of the microscope, improves the imaging quality of the microscope, and can be applied to clinical medical examination. In addition, the invention adopts the deep neural network to concentrate the detection, the segmentation and the identification of the urinary sediment tangible components in one model for multi-task learning, realizes the end-to-end full-automatic detection, the segmentation and the identification of the urinary sediment tangible components, greatly improves the detection speed and the detection precision and lightens the workload of doctors.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the structures shown in the drawings without creative efforts.

FIG. 1 is a flow chart of a urinary sediment visible component detection method based on lensless microscopy imaging, provided by the invention;

FIG. 2 is a block diagram of a first image transformation network according to an embodiment of the present invention;

FIG. 3 is a block diagram of a second image conversion network according to an embodiment of the present invention;

fig. 4 is a structural diagram of an image classification network according to an embodiment of the present invention.

The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

In addition, the technical solutions in the embodiments of the present invention may be combined with each other, but it must be based on the realization of those skilled in the art, and when the technical solutions are contradictory or cannot be realized, such a combination of technical solutions should not be considered to exist, and is not within the protection scope of the present invention.

As shown in fig. 1, the present invention provides a method for detecting urinary sediment visible components based on lensless microscopy, comprising:

101: collecting a urinary sediment holographic intensity map by using a lens-free microscopic imaging mode;

102: inputting the holographic intensity map into a trained urinary sediment visible component detection model, wherein the urinary sediment visible component detection model comprises a first image conversion network, a second image conversion network and an image classification network;

the first image conversion network, the second image conversion network and the image classification network are all convolutional neural networks, but the network structures are different.

103: carrying out phase recovery and automatic focusing on the holographic intensity image by using a first image conversion network to obtain an original microscopic image;

104: performing super-resolution reconstruction on the original microscopic image by using a second image conversion network to obtain a super-resolution microscopic image;

105: and obtaining the type and the quantity of the formed components of each urinary sediment by using an image classification network according to the super-resolution microscopic image.

Types of urine sediment constituents include 7 types, bacteria, crystals, casts, mucus filaments, red blood cells, white blood cells, and squamous epithelial cells, respectively.

According to the urine sediment visible component detection method based on lensless microscopic imaging, lensless microscopic imaging is adopted to replace a traditional optical microscope, an illumination and imaging light path is simplified, an expensive and heavy optical lens is abandoned, the size of microscopic equipment is miniaturized, the weight is light, the cost is low, the operation is simplified, and the medical cost is further saved. Meanwhile, the invention adopts an artificial intelligence algorithm to carry out phase recovery and super-resolution reconstruction on the holographic intensity image acquired by the lens-free microscopic imaging, realizes the microscopic imaging with wide field of view and ultrahigh resolution, does not need to rely on mechanical scanning, image splicing and focusing adjustment to enlarge the field of view and the depth of field, greatly shortens the time of the microscopic imaging, improves the imaging speed of the microscope, improves the imaging quality of the microscope, and can be applied to clinical medical examination. In addition, the invention adopts the deep neural network to concentrate the detection, the segmentation and the identification of the urinary sediment tangible components in one model for multi-task learning, realizes the end-to-end full-automatic detection, the segmentation and the identification of the urinary sediment tangible components, greatly improves the detection speed and the detection precision and lightens the workload of doctors.

The working principle of the lens-free microscopic imaging system is as follows: light emitted by the light source is projected onto the sample, wherein the light transmitted through the sample is called reference light wave, the light scattered with the sample is called scattered light wave, the reference light wave and the scattered light wave emit interference on the imaging sensor to generate interference fringes, and the interference fringes are recorded by the imaging sensor to form a holographic intensity pattern.

The lens-free microscopic imaging system is a system for imaging without any imaging lens, and is based on a Gabor coaxial holographic principle, and utilizes an imaging sensor to sample interference images among transmitted light penetrating through a sample and perform holographic reconstruction through a digital image processing technology so as to obtain digital microscopic imaging of the sample. The lens-free microscopic imaging system breaks through the limitation of space bandwidth product brought by lenses of the traditional optical microscope, realizes high-resolution imaging in a large visual field range, and can provide rapid diagnosis and accurate detection of clinical samples in a large visual field.

The first image transformation network is shown in fig. 2 and comprises an encoder sub-network, a decoder sub-network and a frequency-time domain transformation sub-network;

the encoder sub-network is composed of 5 convolutional layers and is used for extracting high-level feature representations of an input initial phase image and an input initial amplitude image; the convolution kernel size of each convolution layer is 5 × 5 with a step size of 1.

The decoder sub-network consists of 5 deconvolution layers, and the 5 deconvolution layers are respectively connected with corresponding convolution layers in the encoder sub-network in a jump connection mode and are used for decoding to obtain a clear phase image and an amplitude image in a frequency domain after phase recovery and automatic focusing; the convolution kernel size of each deconvolution layer was 5 × 5, with a step size of 1.

The sub-network of frequency-time domain transformation comprises an inverse Fourier transform convolutional layer for converting the clear phase image and amplitude image from frequency domain to time domain by inverse Fourier transformation to obtain the original microscopic image of the urinary sediment.

In another embodiment, in lens-free microscopic imaging, due to the limited size of the pixel size of the imaging sensor, the resolution of the acquired original microscopic image of the sample is low, and in order to obtain a high-resolution microscopic image for detection and identification by a subsequent computer-aided analysis system, super-resolution image reconstruction is required. The present embodiment constructs the second image conversion network based on deep learning, a high resolution microscope image is reconstructed from the input low resolution microscope image.

The second image conversion network is shown in fig. 3 and comprises a feature extraction sub-network, a nonlinear mapping sub-network and an up-sampling sub-network;

the feature extraction sub-network comprises 4 convolutional layers, wherein the first three convolutional layers are used for carrying out high-dimensional feature extraction, and the fourth convolutional layer is used for carrying out feature dimension reduction; the convolution kernel size for the first three convolutional layers is 3 × 3, and the convolution kernel size for the fourth convolutional layer is 1 × 1.

The nonlinear mapping sub-network comprises 3 convolutional layers, wherein the first two convolutional layers are used for carrying out nonlinear processing on input features, and the third convolutional layer is used for carrying out feature dimension expansion; the convolution kernel sizes of the first two convolutional layers are 3 × 3, and the convolution kernel size of the third convolutional layer is 1 × 1.

The up-sampling sub-network comprises two deconvolution layers and is used for carrying out up-sampling operation on input features to obtain a super-resolution microscopic image. The convolution kernel size of the two deconvolution layers is 3 × 3, with a step size of 2.

In a certain embodiment, the image classification network is shown in fig. 4 and includes a feature extraction sub-network, a feature suggestion sub-network, a region of interest alignment sub-network, a mask generation sub-network, and a classification sub-network;

the feature extraction sub-network comprises a series of convolution layers and is used for extracting high-dimensional feature representation of an input image and outputting a feature map;

the feature proposing sub-network comprises a plurality of convolution layers and is used for distributing a fixed surrounding frame to each pixel position in the feature map, and performing binary classification and coordinate regression on pixels in the surrounding frame to obtain a proposing surrounding frame; the enclosure containing the visible components of the urinary sediment is the proposed enclosure.

The interested region alignment sub-network comprises a plurality of convolution layers and is used for mapping the proposed bounding box to the high-dimensional feature and obtaining the interested region feature with fixed size in a bilinear interpolation mode;

the mask generation sub-network adopts a full convolution network structure and is used for performing convolution and deconvolution operations on the characteristics of the region of interest, accurately classifying each pixel and generating an image segmentation mask;

the classification sub-network comprises a plurality of convolution layers and is used for carrying out full connection operation on the characteristics of the region of interest and outputting a final surrounding frame in a regression mode and the category of the urinary sediment visible components in the surrounding frame.

In the model training process, the mask generation sub-network and the classification sub-network are mutually supervised, and finally, the optimal condition is achieved. The image segmentation mask output by the mask generation sub-network optimizes the parameters of the classification sub-network, and meanwhile, the classification category and the bounding box output by the classification sub-network optimize the parameters of the mask generation sub-network, and finally, the parameters of the classification sub-network and the bounding box are optimal, so that the precision of the image classification network is higher.

The image classification network is a two-stage framework, the first stage carries out feature extraction on the input super-resolution microscopic image and generates an offer bounding box, and the second stage classifies the generated offer bounding box and obtains a corresponding segmentation mask and a corresponding bounding box. The number and position of each component in the urine sediment forming components can be obtained through the surrounding frame.

In one embodiment, in order to ensure that the image classification network correctly segments and identifies each tangible component in the super-resolution microscopic image, a loss function as shown below is used to guide the training process of the network:

L＝L _c +L _b +L _m

wherein L represents the total loss error; l is _c Representing a classification error; l is _b Indicating a detection error; l is a radical of an alcohol _m Indicating a segmentation error.

In a next embodiment, for step 103, phase recovery and auto-focusing of the holographic intensity map using the first image conversion network to obtain the original microscopic image, comprises:

301: reversely propagating the holographic intensity image for a distance by utilizing an angular spectrum reverse propagation mode in a free space to obtain a group of twin images of a frequency domain, wherein the twin images comprise an initial phase image and an initial amplitude image;

the distance of back propagation is very short, approximately 1mm.

302: and inputting the initial phase image and the initial amplitude image into a first image conversion network, performing phase recovery and automatic focusing to obtain a clear phase image and an amplitude image in a frequency domain, and converting the clear phase image and the clear amplitude image from the frequency domain to a time domain to obtain an original microscopic image of the urinary sediment.

In a certain embodiment, for step 301, reversely propagating the holographic intensity map by a distance using a backward propagation manner of angular spectrum in free space to obtain a set of twin images in the frequency domain, includes:

3011: calculating a light field E (x) obtained by reversely propagating the holographic intensity pattern I (x, y) by a distance-z by utilizing an angular spectrum reverse propagation mode in free space ₀ ,y ₀ )，

E(x ₀ ,y ₀ )＝f{I(x,y)；λ,-z}

Wherein f { } represents a transfer function, and λ represents a wavelength of the light source;

3012: according to said light field E (x) ₀ ,y ₀ ) And obtaining a set of twin images of the frequency domain.

The invention also provides a urinary sediment visible component detection system based on lensless microscopic imaging, which comprises:

the image acquisition module is used for acquiring a urinary sediment holographic intensity map by using a lensless microscopic imaging mode;

the urine sediment visible component detection module is used for inputting the holographic intensity map into a trained urine sediment visible component detection model, and the urine sediment visible component detection model comprises a first image conversion network, a second image conversion network and an image classification network; carrying out phase recovery and automatic focusing on the holographic intensity image by using a first image conversion network to obtain an original microscopic image; performing super-resolution reconstruction on the original microscopic image by using a second image conversion network to obtain a super-resolution microscopic image; and obtaining the type and the quantity of the formed components of each urinary sediment by using an image classification network according to the super-resolution microscopic image.

In one embodiment, the image acquisition module further comprises:

reversely propagating the holographic intensity image for a distance by utilizing an angular spectrum reverse propagation mode in a free space to obtain a group of twin images of a frequency domain, wherein the twin images comprise an initial phase image and an initial amplitude image;

and inputting the initial phase image and the initial amplitude image into a first image conversion network, performing phase recovery and automatic focusing to obtain a clear phase image and an amplitude image in a frequency domain, and converting the clear phase image and the clear amplitude image from the frequency domain to a time domain to obtain an original microscopic image of the urinary sediment.

In a next embodiment, the image acquisition module further comprises:

calculating a light field E (x) obtained by reversely propagating the holographic intensity pattern I (x, y) by a distance-z by utilizing an angular spectrum reverse propagation mode in free space ₀ ,y ₀ )，

E(x ₀ ,y ₀ )＝f{I(x,y)；λ,-z}

Wherein f { } represents a transfer function, and λ represents a wavelength of the light source;

from the light field E (x) ₀ ,y ₀ ) And obtaining a set of twin images of the frequency domain.

In another embodiment, for the urinary sediment visible component detection module, the first image conversion network includes an encoder subnetwork, a decoder subnetwork, and a frequency-time domain transform subnetwork;

the encoder sub-network is composed of 5 convolutional layers and is used for extracting high-level feature representations of an input initial phase image and an input initial amplitude image;

the decoder sub-network consists of 5 deconvolution layers, and the 5 deconvolution layers are respectively connected with corresponding convolution layers in the encoder sub-network in a jump connection mode and are used for decoding to obtain a clear phase image and an amplitude image in a frequency domain after phase recovery and automatic focusing;

the sub-network comprises an inverse Fourier transform convolutional layer for converting the clear phase image and the clear amplitude image from a frequency domain to a time domain through inverse Fourier transform to obtain an original microscopic image of the urinary sediment.

In a next embodiment, for the urinary sediment visible component detection module, the second image conversion network comprises a feature extraction sub-network, a nonlinear mapping sub-network, and an up-sampling sub-network;

the feature extraction sub-network comprises 4 convolutional layers, wherein the first three convolutional layers are used for carrying out high-dimensional feature extraction, and the fourth convolutional layer is used for carrying out feature dimension reduction;

the nonlinear mapping sub-network comprises 3 convolutional layers, the first two convolutional layers are used for carrying out nonlinear processing on input features, and the third convolutional layer is used for carrying out feature dimension expansion;

the up-sampling sub-network comprises two deconvolution layers and is used for performing up-sampling operation on input features to obtain a super-resolution microscopic image.

In another embodiment, for the urinary sediment tangible element detection module, the image classification network comprises a feature extraction sub-network, a feature proposal sub-network, a region of interest alignment sub-network, a mask generation sub-network and a classification sub-network;

the feature extraction sub-network comprises a series of convolution layers and is used for extracting high-dimensional feature representation of an input image and outputting a feature map;

the feature proposing sub-network comprises a plurality of convolution layers and is used for distributing a fixed surrounding frame to each pixel position in the feature map, and performing binary classification and coordinate regression on pixels in the surrounding frame to obtain a proposing surrounding frame;

the interested region alignment sub-network comprises a plurality of convolution layers and is used for mapping the proposed bounding box to the high-dimensional feature and obtaining the interested region feature with fixed size in a bilinear interpolation mode;

the mask generation sub-network adopts a full convolution network structure and is used for performing convolution and deconvolution operations on the characteristics of the region of interest and accurately classifying each pixel to generate an image segmentation mask;

the classification sub-network comprises a plurality of convolution layers and is used for carrying out full connection operation on the characteristics of the region of interest and outputting a final surrounding frame in a regression mode and the category of the urinary sediment visible components in the surrounding frame.

In one embodiment, for the urinary sediment formed component detection module, the image classification network uses a loss function of,

L＝L _c +L _b +L _m

wherein L represents the total loss error; l is _c Representing a classification error; l is _b Indicating a detection error; l is _m Indicating a segmentation error.

The invention further provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method when executing the computer program.

The invention also proposes a computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the method described above.

The above description is only a preferred embodiment of the present invention, and is not intended to limit the scope of the present invention, and all modifications and equivalents of the present invention, which are made by the contents of the present specification and the accompanying drawings, or directly/indirectly applied to other related technical fields, are included in the scope of the present invention.