CN110807732B - Panoramic stitching system and method for microscopic images - Google Patents

The application provides a panoramic stitching system and a panoramic stitching method for microscopic images, wherein the panoramic stitching system comprises the following steps: the device comprises a visual field sub-block matching module, a visual field position fitting module and a block extracting module; the view sub-block matching module is used for identifying overlapping areas among the images and judging adjacent position relations among the sub-images, so that the sub-images acquired by the micro scanning device are automatically arranged according to the splicing sequence of the images; the visual field position fitting module is used for finely adjusting the position according to the overlapping area among the sub-images so as to accurately splice the cell positions; the block extraction module is used for automatically extracting the spliced complete image. By adopting the scheme, the automatic splicing can be realized, the high-definition images can be obtained, the problem of ordering of clearly scanning pictures is not needed in the splicing process, the pictures in disordered arrangement can be adapted, the higher compatibility is realized, and the labor intensity is greatly reduced. The splicing process can be completed within a second range so as to adapt to the data processing intensity of large-scale cloud diagnosis.

Panoramic stitching system and method for microscopic images

Technical Field

The application relates to the field of intelligent cell identification by using a computer, in particular to a panoramic stitching system and method for microscopic images.

Background

Cytological assays are capable of rapidly distinguishing diseases based on the number and morphology of cells, and are particularly of great significance for distinguishing and detecting neoplastic diseases. However, manual examination is extremely inefficient in clinic, and a doctor with high experience can only examine about 50 samples per day. With the development of technology, fully automatic image analyzers have been used in cytopathology analysis work. The efficiency can be greatly improved. The high-magnification microscanning device can generate a plurality of images with independent focuses for one sample, the images with independent focuses are needed to be spliced together, in the prior art, a software-hardware combination mode is generally adopted for splicing, for example, after a corresponding scaling ratio is calculated through a displacement distance of hardware, the images are spliced into a complete sample image. However, this requires very high precision of the displacement driving mechanism of the micro scanning device, resulting in high hardware use cost. Chinese patent document CN100433060C describes a method for stitching, storing and browsing full-automatic microscopic images, and in the section of stitched images in the specific embodiment of the specification, "a stitching graph structure G with weights is constructed, each node in the stitching graph G represents each captured unit image to be stitched, and the weight on the edge between two adjacent nodes is the stitching probability value between the two unit images. And generating a maximum spanning tree T of the spliced graph by taking any node as a starting point, wherein the traversal sequence of the graph is the optimal splicing sequence. The maximum spanning tree is generated as follows: firstly, selecting any node as a current node; finding a node with the largest weight of the edge between the node and the current node from nodes outside the current tree, taking the edge between the current node and the node as an edge of a generated tree, adding the node into the current tree, and taking the node as the current node; repeating the steps until no node is available. The splicing method of the two adjacent unit images comprises the following steps: assuming that the unit image 1 and the unit image 2 have overlapping ranges, the overlapping range in fig. 1 is R1, the overlapping range in fig. 2 is R2, a picture PT of a region RT1 is selected as a template in R1, matching is performed on R2 by PT, and the best matching position RT2 is found, so that the spliced relative position between fig. 1 and 2 can be obtained according to RT1 and RT 2. The splicing method of all the unit images comprises the following steps: and (3) setting the position of the unit image of the first node as (0, 0), splicing two adjacent unit images according to the optimal splicing sequence, acquiring the relative position of the next unit image and the previous unit image, and calculating the absolute position of each unit image. The memory is not increased along with the increase of the size of the whole large image, so that the occupation of the memory can not influence the performance of the system all the time under the condition of the increase of the size of the large image. And finally, obtaining absolute positions of each unit image to be spliced in the target large image. In the scheme, a tree structure is required to be generated firstly, then automatic identification is difficult to realize by selecting a node scheme, hardware and software are required to cooperate, namely, parameters such as row and column values of the tree structure, approximate positions of image units in the tree structure and the like are required to be clearly known in the scheme. The lack of hardware support results in a significant increase in the effort of the solution. Such as some later manually displaced microscopes, or inconsistent setting of displacement direction and path by different models of microscanning devices, tends to result in the solution being only manually sequenced, resulting in a significant increase in effort. Furthermore, the spliced images in the scheme may have fine position differences, so that the precision of the images is affected.

Disclosure of Invention

The technical problem to be solved by the application is to provide a panoramic stitching system and a panoramic stitching method for microscopic images, which can further improve the intelligent degree of panoramic stitching of microscopic images, improve fault tolerance and be compatible with a plurality of types of microscopic scanning devices as much as possible.

In order to solve the technical problems, the application adopts the following technical scheme: a panoramic stitching system for microscopic images, comprising: the device comprises a visual field sub-block matching module, a visual field position fitting module and a block extracting module;

the view sub-block matching module is used for identifying overlapping areas among the images and judging adjacent position relations among the sub-images, so that the sub-images acquired by the micro scanning device are automatically arranged according to the splicing sequence of the images;

the visual field position fitting module is used for finely adjusting the position according to the overlapping area among the sub-images so as to accurately splice the cell positions;

the block extraction module is used for automatically extracting the spliced complete image.

Analyzing pixel values of overlapping positions, enabling the images to be automatically matched with corresponding positions through a view sub-block matching module, and calculating initial values of a two-dimensional transformation matrix from platform offset to pixel offset according to matched characteristic points in adjacent views to obtain splicing parameters; specifically, adjacent positions of all view sub-blocks relative to other view sub-blocks are determined;

cutting public parts of adjacent visual fields, dividing the public parts into a plurality of small blocks, adopting template matching, searching a public coincidence area, and selecting matching blocks with a matching threshold value larger than 0.9;

calculating the correlation of template matching of all visual fields;

after the position matching is successful, the cell position is accurately spliced through a visual field position fitting intelligent algorithm; after template matching, the approximate position of each view pixel can be obtained, and the maximum pixel deviation is calculated through the initial splicing parameters and the maximum displacement deviation of the platform;

filtering the point of the matching relation between each visual field and the adjacent visual field by using the maximum pixel deviation, removing the point with the deviation larger than the maximum pixel deviation, recalculating the splicing parameters by using the filtered point, recalculating the pixel position of the visual field by using the latest splicing parameters, and continuously updating and perfecting the picture position of the visual field by continuously iterative filtering and recalculating.

In a preferred scheme, the operation process of the view sub-block matching module is as follows:

sa01, inputting, and initializing a result set M;

sa02, setting the current field i as the first field;

sa03, solving all adjacent view sets J of the current view i;

sa04, setting the current adjacent view J as the first view in J;

sa05, finding possible overlapping areas Ri and Rj of the field i and the field j;

sa06, rasterizing the template region Ri into a template sub-block set Pi;

sa07, arranging the template sub-block sets Pi in descending order of the dynamic range of the sub-blocks;

sa08, setting the current template sub-block P as the first one of the template sub-block sets Pi;

sa09, finding the possible overlapping area s of the template sub-block P in the field of view J;

sa10, taking a template sub-block P as a template and s as a search area, and carrying out template matching search;

sa11, adding the best match M into the result set M;

sa12, finding all matching set vision sets N consistent with M in a result set M;

sa13, comparing and judging whether the sum of the weights in N is larger than a threshold v;

if not, setting the current template sub-block P as the next template sub-block set Pi, and returning to Sa09;

if yes, the next step is performed;

sa14, comparing and judging whether the view J is the last view in the view set J;

if not, setting the view J as the next view in the view set J, and returning to Sa05;

if yes, the next step is performed;

sa15, comparing and judging that the field i is the last field;

if not, setting i as the next view, and returning to Sa03;

if yes, outputting a result.

In a preferred scheme, the fitting process of the visual field position is as follows:

sa16, inputting, namely initializing all view positions Xi and Yi;

sa17, setting the current field i as the first field;

sa18, in the sub-block matching set M, obtaining a matching subset Mi containing a field of view i;

sa19, recalculating the positions Xi and Yi of the visual field i according to the matching subset Mi;

sa20, judging that all the visual field updating is completed;

if not, setting the field of view i as the next field of view;

if yes, the next step is performed;

sa21, calculating the average value L of the current wheel vision position and the upper wheel vision position deviation;

sa22, comparing and judging that the deviation average value L is smaller than a threshold value 1;

if not, returning to Sa17;

if yes, the next step is performed;

sa23, visual field position normalization adjustment;

all fields of view are output.

In a preferred scheme, the block extraction process is as follows:

sa24, extracting the size W and H of the full graph;

sa25, dividing the full graph into a set B of blocks according to the block size;

sa26, calculating the positions of all blocks B in the set B;

sa27, setting block B as the first block in set B;

sa28, calculating a set Fb of all fields of view overlapping with the block b;

sa29, setting the field f as the first field of view in Fb;

sa30, overlapping areas Rb and Rf of the field of view f and the block b;

sa31, copying the image in Rf to Rb;

sa32, judging that the field f is the last field in the set Fb;

if not, setting the field f as the next field in Fb, and returning to Sa29;

if yes, the next step is performed;

sa33, save the block b image;

sa34, judging that the block B is the last block in the set B;

if not, setting the block B as the first block in the set B, and returning to Sa28;

if yes, outputting a result.

A method for panoramic stitching of microscopic images, comprising the steps of:

s1, matching view sub-blocks; the view sub-block matching module is used for identifying overlapping areas among the images and judging adjacent position relations among the sub-images, so that the sub-images acquired by the micro scanning device are automatically arranged according to the splicing sequence of the images;

s2, fitting the visual field position; the visual field position fitting module is used for finely adjusting the position according to the overlapping area among the sub-images so as to accurately splice the cell positions;

s3, extracting blocks; the block extraction module is used for automatically extracting the spliced complete image.

In a preferred scheme, the operation process of field sub-block matching is as follows:

sa01, inputting, and initializing a result set M;

sa02, setting the current field i as the first field;

sa03, solving all adjacent view sets J of the current view i;

sa04, setting the current adjacent view J as the first view in J;

sa05, finding possible overlapping areas Ri and Rj of the field i and the field j;

sa06, rasterizing the template region Ri into a template sub-block set Pi;

sa07, arranging the template sub-block sets Pi in descending order of the dynamic range of the sub-blocks;

sa08, setting the current template sub-block P as the first one of the template sub-block sets Pi;

sa09, finding the possible overlapping area s of the template sub-block P in the field of view J;

sa10, taking a template sub-block P as a template and s as a search area, and carrying out template matching search;

sa11, adding the best match M into the result set M;

sa12, finding all matching set vision sets N consistent with M in a result set M;

sa13, comparing and judging whether the sum of the weights in N is larger than a threshold v;

if not, setting the current template sub-block P as the next template sub-block set Pi, and returning to Sa09;

if yes, the next step is performed;

sa14, comparing and judging whether the view J is the last view in the view set J;

if not, setting the view J as the next view in the view set J, and returning to Sa05;

if yes, the next step is performed;

sa15, comparing and judging that the field i is the last field;

if not, setting i as the next view, and returning to Sa03;

if yes, outputting a result.

In a preferred scheme, the fitting process of the visual field position is as follows:

sa16, inputting, namely initializing all view positions Xi and Yi;

sa17, setting the current field i as the first field;

sa18, in the sub-block matching set M, obtaining a matching subset Mi containing a field of view i;

sa19, recalculating the positions Xi and Yi of the visual field i according to the matching subset Mi;

sa20, judging that all the visual field updating is completed;

if not, setting the field of view i as the next field of view;

if yes, the next step is performed;

sa21, calculating the average value L of the current wheel vision position and the upper wheel vision position deviation;

sa22, comparing and judging that the deviation average value L is smaller than a threshold value 1;

if not, returning to Sa17;

if yes, the next step is performed;

sa23, visual field position normalization adjustment;

all fields of view are output.

In a preferred scheme, the block extraction process is as follows:

sa24, extracting the size W and H of the full graph;

sa25, dividing the full graph into a set B of blocks according to the block size;

sa26, calculating the positions of all blocks B in the set B;

sa27, setting block B as the first block in set B;

sa28, calculating a set Fb of all fields of view overlapping with the block b;

sa29, setting the field f as the first field of view in Fb;

sa30, overlapping areas Rb and Rf of the field of view f and the block b;

sa31, copying the image in Rf to Rb;

sa32, judging that the field f is the last field in the set Fb;

if not, setting the field f as the next field in Fb, and returning to Sa29;

if yes, the next step is performed;

sa33, save the block b image;

sa34, judging that the block B is the last block in the set B;

if not, setting the block B as the first block in the set B, and returning to Sa28;

if yes, outputting a result.

The application provides a panoramic stitching system and a panoramic stitching method for microscopic images, which can realize automatic stitching and obtain high-definition images by adopting the scheme, can adapt to images in disordered arrangement without definitely scanning the ordering problem of the images in the stitching process, realize higher compatibility and greatly reduce labor intensity. The scheme of the application can complete the splicing process within the range of seconds so as to adapt to the large-scale data processing intensity based on diagnosis on the cloud.

Drawings

The application is further illustrated by the following examples in conjunction with the accompanying drawings:

fig. 1 is a schematic flow chart of an image stitching process in the present application.

Fig. 2 is a schematic flow chart of field of view sub-block matching in the present application.

Fig. 3 is a schematic flow chart of field position fitting in the present application.

FIG. 4 is a flow chart of block extraction according to the present application.

Fig. 5 is a schematic diagram of a view field position fitting operation procedure in the present application.

FIG. 6 is a schematic diagram of a field of view sub-block matching operation in the present application.

Detailed Description

Example 1:

as shown in fig. 1, a panoramic stitching system for microscopic images, characterized in that it comprises: the device comprises a visual field sub-block matching module, a visual field position fitting module and a block extracting module;

the view sub-block matching module is used for identifying overlapping areas among the images and judging adjacent position relations among the sub-images, so that the sub-images acquired by the micro scanning device are automatically arranged according to the splicing sequence of the images;

the visual field position fitting module is used for finely adjusting the position according to the overlapping area among the sub-images so as to accurately splice the cell positions;

the block extraction module is used for automatically extracting the spliced complete image.

In a preferred scheme, the operation process of the view sub-block matching module is as follows:

sa01, inputting, and initializing a result set M;

sa02, setting the current field i as the first field;

sa03, solving all adjacent view sets J of the current view i;

sa04, setting the current adjacent view J as the first view in J;

sa05, finding possible overlapping areas Ri and Rj of the field i and the field j;

sa06, rasterizing the template region Ri into a template sub-block set Pi;

sa07, arranging the template sub-block sets Pi in descending order of the dynamic range of the sub-blocks;

sa08, setting the current template sub-block P as the first one of the template sub-block sets Pi;

sa09, finding the possible overlapping area s of the template sub-block P in the field of view J;

sa10, taking a template sub-block P as a template and s as a search area, and carrying out template matching search;

sa11, adding the best match M into the result set M;

sa12, finding all matching set vision sets N consistent with M in a result set M;

sa13, comparing and judging whether the sum of the weights in N is larger than a threshold v;

if not, setting the current template sub-block P as the next template sub-block set Pi, and returning to Sa09;

if yes, the next step is performed;

sa14, comparing and judging whether the view J is the last view in the view set J;

if not, setting the view J as the next view in the view set J, and returning to Sa05;

if yes, the next step is performed;

sa15, comparing and judging that the field i is the last field;

if not, setting i as the next view, and returning to Sa03;

if yes, outputting a result.

In a preferred scheme, the fitting process of the visual field position is as follows:

sa16, inputting, namely initializing all view positions Xi and Yi;

sa17, setting the current field i as the first field;

sa18, in the sub-block matching set M, obtaining a matching subset Mi containing a field of view i;

sa19, recalculating the positions Xi and Yi of the visual field i according to the matching subset Mi;

sa20, judging that all the visual field updating is completed;

if not, setting the field of view i as the next field of view;

if yes, the next step is performed;

sa21, calculating the average value L of the current wheel vision position and the upper wheel vision position deviation;

sa22, comparing and judging that the deviation average value L is smaller than a threshold value 1;

if not, returning to Sa17;

if yes, the next step is performed;

sa23, visual field position normalization adjustment;

all fields of view are output.

In a preferred scheme, the block extraction process is as follows:

sa24, extracting the size W and H of the full graph;

sa25, dividing the full graph into a set B of blocks according to the block size;

sa26, calculating the positions of all blocks B in the set B;

sa27, setting block B as the first block in set B;

sa28, calculating a set Fb of all fields of view overlapping with the block b;

sa29, setting the field f as the first field of view in Fb;

sa30, overlapping areas Rb and Rf of the field of view f and the block b;

sa31, copying the image in Rf to Rb;

sa32, judging that the field f is the last field in the set Fb;

if not, setting the field f as the next field in Fb, and returning to Sa29;

if yes, the next step is performed;

sa33, save the block b image;

sa34, judging that the block B is the last block in the set B;

if not, setting the block B as the first block in the set B, and returning to Sa28;

if yes, outputting a result.

Example 2:

a method for panoramic stitching of microscopic images, comprising the steps of:

s1, matching view sub-blocks; the view sub-block matching module is used for identifying overlapping areas among the images and judging adjacent position relations among the sub-images, so that the sub-images acquired by the micro scanning device are automatically arranged according to the splicing sequence of the images;

s2, fitting the visual field position; the visual field position fitting module is used for finely adjusting the position according to the overlapping area among the sub-images so as to accurately splice the cell positions;

s3, extracting blocks; the block extraction module is used for automatically extracting the spliced complete image.

6. The method for panoramic stitching of microscopic images according to claim 5, wherein: the operation process of the field of view sub-block matching is as follows:

sa01, inputting, and initializing a result set M;

sa02, setting the current field i as the first field;

sa03, solving all adjacent view sets J of the current view i;

sa04, setting the current adjacent view J as the first view in J;

sa05, finding possible overlapping areas Ri and Rj of the field i and the field j;

sa06, rasterizing the template region Ri into a template sub-block set Pi;

sa07, arranging the template sub-block sets Pi in descending order of the dynamic range of the sub-blocks;

sa08, setting the current template sub-block P as the first one of the template sub-block sets Pi;

sa09, finding the possible overlapping area s of the template sub-block P in the field of view J;

sa10, taking a template sub-block P as a template and s as a search area, and carrying out template matching search;

sa11, adding the best match M into the result set M;

sa12, finding all matching set vision sets N consistent with M in a result set M;

sa13, comparing and judging whether the sum of the weights in N is larger than a threshold v;

if not, setting the current template sub-block P as the next template sub-block set Pi, and returning to Sa09;

if yes, the next step is performed;

sa14, comparing and judging whether the view J is the last view in the view set J;

if not, setting the view J as the next view in the view set J, and returning to Sa05;

if yes, the next step is performed;

sa15, comparing and judging that the field i is the last field;

if not, setting i as the next view, and returning to Sa03;

if yes, outputting a result.

In a preferred scheme, the fitting process of the visual field position is as follows:

sa16, inputting, namely initializing all view positions Xi and Yi;

sa17, setting the current field i as the first field;

sa18, in the sub-block matching set M, obtaining a matching subset Mi containing a field of view i;

sa19, recalculating the positions Xi and Yi of the visual field i according to the matching subset Mi;

sa20, judging that all the visual field updating is completed;

if not, setting the field of view i as the next field of view;

if yes, the next step is performed;

sa21, calculating the average value L of the current wheel vision position and the upper wheel vision position deviation;

sa22, comparing and judging that the deviation average value L is smaller than a threshold value 1;

if not, returning to Sa17;

if yes, the next step is performed;

sa23, visual field position normalization adjustment;

all fields of view are output.

In a preferred scheme, the block extraction process is as follows:

sa24, extracting the size W and H of the full graph;

sa25, dividing the full graph into a set B of blocks according to the block size;

sa26, calculating the positions of all blocks B in the set B;

sa27, setting block B as the first block in set B;

sa28, calculating a set Fb of all fields of view overlapping with the block b;

sa29, setting the field f as the first field of view in Fb;

sa30, overlapping areas Rb and Rf of the field of view f and the block b;

sa31, copying the image in Rf to Rb;

sa32, judging that the field f is the last field in the set Fb;

if not, setting the field f as the next field in Fb, and returning to Sa29;

if yes, the next step is performed;

sa33, save the block b image;

sa34, judging that the block B is the last block in the set B;

if not, setting the block B as the first block in the set B, and returning to Sa28;

if yes, outputting a result.

Example 3:

based on examples 1-2, as shown in fig. 5-6, a certain cytopathology analysis example is taken as an example: the images automatically acquired from the microscanning device are, as shown in the upper diagram of fig. 5, ordered irregularly for each sub-image, depending on the microscanning device automated acquisition path, ensuring that each image has a position of overlap with each other during acquisition. And analyzing pixel values of overlapping positions, automatically matching the images to corresponding positions through a vision sub-block matching intelligent algorithm, and calculating initial values of a two-dimensional transformation matrix from platform offset to pixel offset according to the matched characteristic points in adjacent vision to obtain splicing parameters. In particular, the neighboring positions of the respective view sub-blocks, i.e. sub-images, with respect to the other sub-images are determined. Cutting the public part of the adjacent visual field, dividing the public part into a plurality of small blocks, adopting template matching, searching a public coincidence area, and selecting a matching block with a matching threshold value larger than 0.9. And calculating the correlation of template matching of all the fields of view. As shown in fig. 6, after the position matching is successful, the positions of the cells are slightly deviated, and the positions of the cells are accurately spliced through a visual field position fitting intelligent algorithm. Specifically, after template matching, the approximate position of each view pixel can be obtained, the maximum pixel deviation is calculated through the initial splicing parameter and the maximum displacement deviation of the platform, the point of the matching relation between each view and the adjacent view is filtered through the maximum pixel deviation, the point with the deviation larger than the maximum pixel deviation is removed, the splicing parameter is recalculated through the screened point, the latest splicing parameter is used, the pixel position of the view is recalculated, and the picture position of the view can be updated and perfected continuously through continuous iterative screening and recalculation, so that the error is smaller, and the splicing effect is more perfect. After the picture position of each view field is calculated, the brightness of each view field is updated through a background image by using the calculated background in the scanning process, the visual perception of a doctor for watching each view field image is improved, a perfect slide picture can be spliced, and the whole spliced image is extracted as a block. The large map is then cut as required to obtain the desired width and height of the picture, as the large map with all fields of view stitched would be large and unnecessary.

The above embodiments are merely preferred embodiments of the present application, and should not be construed as limiting the present application, and the embodiments and features of the embodiments of the present application may be arbitrarily combined with each other without collision. The protection scope of the present application is defined by the claims, and the protection scope includes equivalent alternatives to the technical features of the claims. I.e., equivalent replacement modifications within the scope of this application are also within the scope of the application.