CN113920439B - Extraction method and device for arrow point - Google Patents

The application relates to the technical field of satellite navigation, and discloses an extraction method and a device for an arrow point, wherein the method comprises the following steps: a style conversion model is constructed, and the style conversion model is used for converting satellite image styles into aerial image styles; constructing a super-resolution model, wherein the super-resolution model is used for converting a low-resolution aerial image into a high-resolution aerial image; acquiring a satellite image to be identified, converting the satellite image into a gray level image, extracting an arrow in the gray level image and judging the kind of the arrow; importing the gray level map into the style conversion model and the super-resolution model to generate a high-resolution aerial image style gray level map; converting the gray level diagram into a binary diagram, extracting the outline of the binary diagram, calculating the vertex and the center of the outline, and judging the position of the arrow point according to the positions of the vertex and the center.

Extraction method and device for arrow point

Technical Field

The application relates to the technical field of satellite navigation, in particular to a satellite image processing technology.

Background

In the processing problem of remote sensing images, the accurate positioning and identification of objects is a main application direction. One of the more typical application scenarios is the identification and precise positioning of printed matter, in particular arrows, on roads. When the positioning of the sharp point of the road arrow is performed in the satellite image, the accuracy of the image pixels of the satellite image is low, so that the obtained identification result is poor. In addition, because the brightness of the satellite image is affected by the angle and weather conditions of the acquired satellites, the brightness of the road printing arrows for different areas and the contrast with the surrounding road colors can be greatly changed, which further increases the difficulty in accurately identifying and locating the sharp points of the arrows on the roads in the image.

Currently, in applications involving ground prints in satellite images, manual selection is often used. Since the manual selection process is extremely dependent on subjective judgment of the selector, it is difficult to perform uniform criteria, resulting in uneven selection accuracy of different selectors. And because of low speed of manual selection and high labor cost, large-scale arrow point extraction is difficult. For the above two reasons, the image positioning accuracy of the ground print, particularly the arrow point, cannot be stably output. And large-scale work extraction cannot be performed.

On the other hand, the extraction scheme based on the traditional computer vision method relies more on the source image characteristics of satellite images, which makes the accuracy of the images unstable when the images are extracted for different satellite shooting angles and arrows in different areas. And the accuracy of the output value of the final arrow point is also lower due to the limitation of the pixels of the image. In view of the latter problem, one conventional solution is to perform processing of image super-resolution using an interpolation-based scheme, and then perform a sharp point extraction operation based on the processed image. However, since this super-resolution processing scheme is a solution based on the premise of sacrificing the sharpness of the image, it is practically impossible to bring about an improvement in the positioning accuracy of the arrow point. In contrast, due to the interpolation, the pixel change of the edge of the arrow is smoother, the edge characteristics are not obvious, and the failure of edge extraction or the unstable extraction process are more easily caused, so that the final arrow point positioning effect is affected.

Therefore, in practical application, a high-precision satellite image arrow point extraction method is needed, more ground arrow point positions in an image are extracted to the greatest extent possible on the premise of guaranteeing the extraction precision, and better image positioning output is provided for other applications based on the arrow positions.

Disclosure of Invention

The application aims to provide a satellite image arrow point extraction method and device, which can identify a road arrow point with higher efficiency and accuracy.

In one embodiment of the application, a method for extracting an arrow point is disclosed, which comprises the following steps:

a style conversion model is constructed, and the style conversion model is used for converting satellite image styles into aerial image styles;

Constructing a super-resolution model, wherein the super-resolution model is used for converting a low-resolution aerial image into a high-resolution aerial image;

acquiring a satellite image to be identified, converting the satellite image into a gray level image, extracting an arrow in the gray level image and judging the kind of the arrow;

importing the gray level map into the style conversion model and the super-resolution model to generate a high-resolution aerial image style gray level map;

converting the gray level diagram into a binary diagram, extracting the outline of the binary diagram, calculating the vertex and the center of the outline, and judging the position of the arrow point according to the positions of the vertex and the center.

In a preferred embodiment, the step of converting the gray scale map into a binary map and extracting a contour of the binary map, calculating a vertex and a center of the contour, and determining a position of the arrow point according to positions of the vertex and the center, further includes;

intercepting a center picture in the gray level map of the high-resolution aerial image style, wherein the center picture is one quarter of the gray level map;

extracting an OTSU threshold of the center picture as a set threshold, and converting the gray level map of the high-resolution aerial image style into a binary map according to the set threshold;

Extracting the outline of the binary image;

calculating four vertexes of the uppermost, lowermost, leftmost and rightmost of the outline;

calculating the center of the contour according to the four vertexes;

taking two vertexes far away from the center, recalculating the center of the outline by taking the center points of the two vertexes, and respectively making circles by taking the two center points between the two vertexes far away and the recalculated center as circle centers;

If the arrow is a straight arrow, one of the two farther vertexes corresponding to the more pixels in the two circles is taken as an arrow point, and if the arrow is a straight arrow with a turn, one of the two farther vertexes corresponding to the less pixels in the two circles is taken as an arrow point.

In a preferred embodiment, the step of extracting the contour of the binary image further includes: and calculating the area of the outline, and judging that the arrow is not included in the center picture when the area is larger than a first threshold value or smaller than a second threshold value.

In a preferred embodiment, the arrow types include straight arrows and straight arrows with turns.

In a preferred embodiment, the step of constructing the style conversion model further comprises:

acquiring a satellite style image and an aviation style image;

Capturing the same number of samples on the satellite style image and the aviation style image respectively, wherein the ratio of the size of the captured satellite style image to the size of the captured aviation style image is equal to the ratio of the resolution of the satellite style image to the resolution of the aviation style image;

The samples were model trained using CycleGAN algorithm, neural Style algorithm, or PixtoPix algorithm.

In a preferred embodiment, the step of constructing the super-resolution model further includes:

Adjusting the size of the intercepted aviation style image to be the same as the size of the intercepted satellite style image;

taking the intercepted aviation style image and the adjusted intercepted aviation style image as training pairs;

Model training is performed by adopting a bidirectional LSTM mode or an countermeasure network mode.

In a preferred embodiment, the ratio of the resolution of the satellite style image to the resolution of the aerial style image ranges from 2 to 5.

In a preferred embodiment, the steps of acquiring a satellite image to be identified and converting the satellite image into a gray scale, extracting an arrow in the gray scale and judging the kind of the arrow further comprise:

Cutting the satellite image to be identified and converting the satellite image into a gray level image;

Identifying and classifying arrows in the gray scale map by adopting a positioning model;

The arrow is verified using computer vision methods.

In a preferred embodiment, the step of cropping the satellite image to be identified and converting the satellite image to a gray scale image further includes: and clipping the satellite image to be identified into the same size as the intercepted satellite style image.

The application also discloses an extraction device of the arrow point, which comprises:

The style conversion model building unit is used for converting the satellite image style into the aerial image style;

the super-resolution model building unit is used for converting the low-resolution aerial image into a high-resolution aerial image;

an arrow type recognition unit configured to acquire a satellite image to be recognized and convert the satellite image into a gray scale map, extract an arrow in the gray scale map, and judge an arrow type;

a conversion unit configured to import the gray-scale map into the style conversion model and the super-resolution model, and generate a high-resolution aerial image style gray-scale map;

and the sharp point extraction unit is configured to convert the gray level diagram into a binary diagram and extract the outline of the binary diagram, calculate the vertex and the center of the outline, and judge the position of the arrow sharp point according to the positions of the vertex and the center.

This patent proposes a scheme of carrying out high accuracy calculation to the arrow point of satellite image road printed matter, has following beneficial effect:

1) According to different training examples, paired or unpaired aerial sheets and satellite images are used for training a style conversion model, so that satellite images are converted into aerial sheet styles with higher definition, the style conversion model obtained under the condition of paired training sets has higher definition and is not easy to generate common boundary distortion in style conversion, and the problem that objects with arrow levels are difficult to identify due to low resolution of satellite images is solved.

2) The super-resolution operation is carried out on the satellite image converted into the aerial lens style, and the precision of the sharp point extraction algorithm can be further improved under the condition that the processing precision of the original image is improved.

3) The arrow point points aiming at different arrow properties are obtained through the outline extraction, area filtering, boundary point extraction and center point secondary extraction schemes of the arrow characteristic design, so that the arrow point points are obtained with the highest precision under the condition of ensuring the confidence level.

The numerous technical features described in the description of the present application are distributed among the various technical solutions, which can make the description too lengthy if all possible combinations of technical features of the present application (i.e., technical solutions) are to be listed. In order to avoid this problem, the technical features disclosed in the above summary of the application, the technical features disclosed in the following embodiments and examples, and the technical features disclosed in the drawings may be freely combined with each other to constitute various new technical solutions (these technical solutions are regarded as already described in the present specification) unless such a combination of technical features is technically impossible. For example, in one example, feature a+b+c is disclosed, in another example, feature a+b+d+e is disclosed, and features C and D are equivalent technical means that perform the same function, technically only by alternative use, and may not be adopted simultaneously, feature E may be technically combined with feature C, and then the solution of a+b+c+d should not be considered as already described because of technical impossibility, and the solution of a+b+c+e should be considered as already described.

Drawings

Fig. 1 is a schematic flow chart of extraction of arrow points according to a first embodiment of the present application;

fig. 2 is a schematic structural view of an arrow point extraction apparatus according to a second embodiment of the present application.

Detailed Description

In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. It will be understood by those skilled in the art that the claimed application may be practiced without these specific details and with various changes and modifications from the embodiments that follow.

Description of the partial concepts:

style conversion: the body style (i.e., the distinctive feature) of the image changes or switches.

Super resolution: the super resolution is to increase the resolution of the original image by a hardware or software method, and the super resolution reconstruction is the process of obtaining a high resolution image by a series of low resolution images.

OTSU: the maximum inter-class variance method is proposed by Japanese scholars in 1979, is a method for determining a self-adaptive threshold value, is called as the Otsu method, is called as OTSU for short, is a global-based binarization algorithm, and divides an image into a foreground part and a background part according to the gray characteristic of the image. The difference between the two parts should be the largest when the optimal threshold is taken, and the criterion used in the OTSU algorithm to measure the difference is the more common maximum inter-class variance. If the inter-class variance between the foreground and the background is larger, the difference between the two parts forming the image is larger, when a part of targets are divided into the background by mistake or a part of the background is divided into the targets by mistake, the difference between the two parts is smaller, and when the division of the taken threshold value maximizes the inter-class variance, the probability of the wrong division is minimum.

Challenge network: a deep learning model, the model is built up of (at least) two modules: the mutual game learning of the generated model and the judging model generates a desired output result.

For the purpose of making the objects, technical solutions and advantages of the present application more apparent, embodiments of the present application will be described in further detail below with reference to the accompanying drawings.

The first embodiment of the application relates to a method for extracting an arrow point, the flow of which is shown in fig. 1, and the method comprises the following steps:

In step 110, a style conversion model is constructed, the style conversion model being used to convert the satellite image style into the aerial image style.

In one embodiment, the step of constructing the style conversion model 110 further comprises:

acquiring a satellite style image and an aviation style image;

Capturing the same number of samples on the satellite style image and the aviation style image respectively, wherein the ratio of the size of the captured satellite style image to the size of the captured aviation style image is equal to the ratio of the resolution of the satellite style image to the resolution of the aviation style image;

The samples were model trained using CycleGAN algorithm, neural Style algorithm, or PixtoPix algorithm.

In an embodiment, the ratio of the resolution of the satellite style image to the resolution of the aerial style image has a value ranging from 2 to 5, for example, 3, 4.

In step 120, a super-resolution model is constructed, which is used to convert the low-resolution aerial image into a high-resolution aerial image.

In one embodiment, the step of constructing the super-resolution model 120 further comprises:

Adjusting the size of the intercepted aviation style image to be the same as the size of the intercepted satellite style image;

taking the intercepted aviation style image and the adjusted intercepted aviation style image as training pairs;

model training is performed by adopting a two-way LSTM (Long Short-Term Memory network) mode or an antagonistic network mode.

It should be appreciated that in one embodiment, the step of building the stylistic transformation model and then building the super-resolution model may be performed first, in another embodiment, the super-resolution model may be built first and then the stylistic transformation model may be built, and in another embodiment, the stylistic transformation model and the super-resolution model may be built simultaneously.

And 130, acquiring a satellite image to be identified, converting the satellite image into a gray scale image, extracting an arrow in the gray scale image and judging the kind of the arrow. In one embodiment, the arrow types include straight arrows and straight arrows with turns. Wherein, the straight arrow with turning may include straight and turning left, straight and turning right.

And 140, importing the gray level map into the style conversion model and the super-resolution model to generate a high-resolution aerial image style gray level map.

In an embodiment, the steps of acquiring a satellite image to be identified and converting the satellite image into a gray scale, extracting an arrow in the gray scale and judging the kind of the arrow further comprise:

Cutting the satellite image to be identified and converting the satellite image into a gray level image;

Identifying and classifying arrows in the gray scale map by adopting a positioning model;

The arrow is verified using computer vision methods.

In an embodiment, the step of cropping the satellite image to be identified and converting the satellite image to a gray scale image further includes: and clipping the satellite image to be identified into the same size as the intercepted satellite style image.

And 150, converting the gray level map into a binary map, extracting the outline of the binary map, calculating the peak and the center of the outline, and judging the position of the pointed point of the arrow according to the positions of the peak and the center.

In one embodiment, the step of converting the gray scale map into a binary map and extracting a contour of the binary map, calculating a vertex and a center of the contour, and determining a position of the arrow point according to positions of the vertex and the center, further comprises;

Intercepting a center picture in the gray level picture of the high-resolution aerial image style, wherein the center picture is one quarter of the gray level picture, for example, intercepting 1/4 part and 3/4 part of each side, and then intercepting a square in the middle as a center image;

extracting an OTSU threshold of the center picture as a set threshold, and converting the gray level map of the high-resolution aerial image style into a binary map according to the set threshold;

Extracting the outline of the binary image;

calculating four vertexes of the uppermost, lowermost, leftmost and rightmost of the outline;

calculating the center of the contour according to the four vertexes;

Taking two vertexes far away from the center, recalculating the center of the contour by taking the center points of the two vertexes, and respectively making circles by taking the two center points between the two vertexes far away and the recalculated center as circle centers, wherein in one embodiment, the radius of the circles can be the distance between the circle centers and the recalculated center;

If the arrow is a straight arrow, one of the two farther vertexes corresponding to the more pixels in the two circles is taken as an arrow point, and if the arrow is a straight arrow with a turn, one of the two farther vertexes corresponding to the less pixels in the two circles is taken as an arrow point.

In one embodiment, the step of extracting the contour of the binary image further comprises: and calculating the area of the outline, and judging that the arrow is not included in the center picture when the area is larger than a first threshold value or smaller than a second threshold value.

In order to better understand the technical solutions of the present disclosure, the following description is given with reference to a specific example, in which details are listed mainly for the sake of understanding, and are not intended to limit the scope of protection of the present disclosure.

The extraction method of the arrow point is divided into a model training stage and an application stage in the present example.

Model training phase

As the size of the road arrow is smaller, the higher the resolution of the satellite image is, the better the positioning accuracy can be finally obtained, when the resolution of the satellite image is larger than 30 cm, the arrow is invisible or the definition is too low to be identified, so that the resolution required by the satellite image is 30 cm, and the resolution of the aerial photo (aerial image) is smaller than half of the resolution of the satellite image, thereby ensuring the best edge output effect in the super-resolution algorithm. The smaller the resolution of the aerial photograph, the better the edge definition of the output image that can be ultimately obtained.

A large number of square pictures are first taken on the satellite image and then the same number of square pictures are correspondingly taken on the aerial photo. It should be noted that the ratio of truncated aerial image to satellite image size is inversely proportional to their resolution (e.g., if the satellite image resolution is 30 cm and the aerial image resolution is 7.5 cm, then the truncated aerial image should be 30/7.5=4 times the satellite image size). In addition, to ensure the integrity of the arrows and surrounding features, the size of the truncated satellite image should not be less than 64 x 64. In order to ensure the speed and normal operation of the training process, the intercepted pictures are not too large. The size of a particular image may depend on the size of the computer memory used for training, the training speed requirements, and the complexity of the model of the style conversion and super resolution algorithms.

And constructing a training pair by using the intercepted aerial photos, adjusting each intercepted aerial photo to be the same as the satellite image, taking the adjusted aerial photo as a model input, taking the aerial photo before adjustment as a training pair for constructing a model by outputting the model, and training the super-resolution model to obtain the super-resolution model. Typical super-resolution implementations include constructing the image as a top-to-bottom, left-to-right sequence structure, and then learning and estimating the sequence using bi-directional LSTM, or modeling the sequence using LSTM combined with attention methods. Another is to train the super-resolution network as a generator in generating the countermeasure network, and then train the countermeasure network to obtain a higher-precision super-resolution network.

Different kinds of training pairs need to be built aiming at different kinds of style conversion models. In normal cases, no accurate corresponding pictures for satellite images and aerial photographs are available, and only style conversion models trained based on unpaired examples are available. Several more typical style conversion model selections include CycleGAN and Neural Style algorithms. However, when paired training pairs capable of providing satellite images and aerial photographs or other high-precision image sources (i.e., the images of the training pairs show the same region with the same resolution, and only the definition of the images is different), a style conversion model based on the paired training pairs is adopted, and generally, the model can obtain better output image definition, and a typical algorithm comprises PixtoPix style conversion algorithm.

(II) application phase

Firstly, determining the approximate position information of an arrow, taking the position as a center, intercepting pictures with the same size as that used by a training set, and converting the pictures into gray level images. The rough localization model trained to obtain the approximate location of the arrow may employ a deep learning model, while to ensure that the arrow does exist at that location, the extracted arrow and the type of arrow may be verified based on conventional computer vision methods, resulting in a higher confidence level. The processing method can also prevent the outputted intercepted image part from being blocked by the vehicle, thereby causing the positioning erroneous judgment of the subsequent arrow point.

And inputting the obtained gray level image into a style conversion model, and converting to obtain the model of the aerial photo style. After the process, the definition of the image can be found to be improved, and the ideal effect can be obtained under the condition that the definition of the arrow of the original satellite image is not very poor. If the definition of the original satellite image is poor, the contrast ratio between the original satellite image and the surrounding ground is small, the arrow generated after style conversion still can be blurred, and finally, the arrow can be filtered out by a rule-based positioning algorithm. One potential side effect of style conversion is the possibility of creating a distortion of the straight lines on the image. However, since these distortions rarely occur on the arrow and do not substantially change the arrow point position if at all, there is no effect on the final arrow point extraction result if a reasonable post-processing scheme is employed.

After converting the satellite image into the style of the aerial photo, super resolution of the image can be performed based on the picture. The resolution of the generated image is raised by a factor equal to the satellite image resolution/aerial image resolution used for training over the original satellite image. Because the deep learning scheme is adopted to extract the middle and high layer features in the super resolution, the problem of image blurring caused by the traditional super resolution based on interpolation can be effectively avoided. Meanwhile, due to the improvement of resolution, the subsequent sharp point extraction can obtain higher precision.

The extraction of the arrow points is performed on the super-resolution processed image by using a traditional scheme based on computer vision. There may be subtle differences in specific treatments for different arrow types.

1) Firstly, cutting out one fourth of the periphery of an image to obtain a center picture, wherein the size of the picture is one fourth of that of an original image, specifically, each side cuts out 1/4 part and 3/4 part, and then cuts out a square in the middle as a center image. And then binarizing the image which is not intercepted based on the threshold value to be output. And then extracting an OTSU threshold value based on the central image as a set threshold value, wherein in order to limit the threshold value range, the method can carry out value according to the arrow and the surrounding of the arrow, otherwise, the value of the threshold value is interfered. Then, based on the OTSU threshold value, binarizing the original image without intercepting the center image to be output;

2) Extracting contours from the generated binarized image, filtering the contours based on the contour area, and removing the too large or too small contours to ensure the effectiveness of output;

3) Selecting a candidate contour based on a relative relation with the position of the midpoint of the image, selecting a contour when the contour contains the midpoint, selecting the contour closest to the midpoint as the candidate contour if the condition is not satisfied, and outputting no effective sharp point when the distances between all contours and the center point are greater than a certain threshold (the threshold is determined by the size of the super-resolution image);

4) Calculating four boundary points (vertexes) of the contour at the uppermost, lowermost, leftmost and rightmost directions;

5) Calculating the distance between each boundary point and the center, and reserving two boundary points with the largest distance;

6) Calculating the arrow center again according to the rest two boundary points, for example, calculating the center point of the two boundary points to be the center of the recalculation, calculating a quarter intercept point and a three quarter intercept point of the arrow outline by taking the center and the two boundary points as endpoints, respectively making circles by taking the quarter intercept point and the three quarter intercept point as circle centers, and calculating the number of points of the outline in the two circles, wherein if the arrow is a straight arrow, the boundary point of one corresponding to the circle with more outline points in the circle is an arrow point; conversely, if the arrow is a straight arrow with a turn, the boundary point of the circle corresponding to the circle with the fewer outline points in the circle is the arrow point. The sharp point scheme can output the arrow sharp point position with maximum precision under the condition of ensuring no misjudgment.

A second embodiment of the present application relates to an arrow point extraction apparatus having a structure as shown in fig. 2, the arrow point extraction apparatus comprising: the system comprises a style conversion model construction unit, a super-resolution model construction unit, an arrow type identification unit, a conversion unit and a sharp point extraction unit, wherein:

The style conversion model building unit is used for converting the satellite image style into the aerial image style;

the super-resolution model building unit is used for converting the low-resolution aerial image into a high-resolution aerial image;

an arrow type recognition unit configured to acquire a satellite image to be recognized and convert the satellite image into a gray scale map, extract an arrow in the gray scale map, and judge an arrow type;

a conversion unit configured to import the gray-scale map into the style conversion model and the super-resolution model, and generate a high-resolution aerial image style gray-scale map;

and the sharp point extraction unit is configured to convert the gray level diagram into a binary diagram and extract the outline of the binary diagram, calculate the vertex and the center of the outline, and judge the position of the arrow sharp point according to the positions of the vertex and the center.

The first embodiment is a method embodiment corresponding to the present embodiment, and the technical details in the first embodiment can be applied to the present embodiment, and the technical details in the present embodiment can also be applied to the first embodiment.

It should be noted that, as will be understood by those skilled in the art, the implementation functions of the modules shown in the embodiments of the arrow point extraction device described above may be understood with reference to the foregoing description of the arrow point method. The functions of the modules shown in the embodiment of the arrow point extraction device described above may be implemented by a program (executable instructions) running on a processor, or by a specific logic circuit. The device for extracting the arrow point according to the embodiment of the present application may be stored in a computer readable storage medium if implemented in the form of a software function module and sold or used as a separate product. Based on such understanding, the technical solutions of the embodiments of the present application may be embodied in essence or a part contributing to the prior art in the form of a software product stored in a storage medium, including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read Only Memory (ROM), a magnetic disk, an optical disk, or other various media capable of storing program codes. Thus, embodiments of the application are not limited to any specific combination of hardware and software.

It should be noted that in the present patent application, relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element. In the present patent application, if it is mentioned that an action is performed according to an element, it means that the action is performed at least according to the element, and two cases are included: the act is performed solely on the basis of the element and is performed on the basis of the element and other elements. Multiple, etc. expressions include 2, 2 times, 2, and 2 or more, 2 or more times, 2 or more.

All references mentioned in this disclosure are to be considered as being included in the disclosure of the application in its entirety so that modifications may be made as necessary. Further, it is understood that various changes or modifications of the present application may be made by those skilled in the art after reading the above disclosure, and such equivalents are intended to fall within the scope of the application as claimed.