CN113112600A - Indoor scene three-dimensional modeling method based on structure - Google Patents

The invention discloses a structure-based indoor scene three-dimensional modeling method, which solves the problems of low modeling precision and the like of noise, cavities and small objects in indoor scene point cloud data by utilizing a planar structure and a polygonal structure. The method comprises the following steps: performing plane segmentation on the point cloud, and eliminating abnormal points and small objects with low modeling precision in the point cloud data according to plane information; and then carrying out polygon fitting on the plane boundary, automatically extracting the adjacency relation of the polygon set, and eliminating the gaps caused by the cavities and the plane segmentation in the data by applying the optimized polygon automatic bonding. The method provided by the invention solves the problems of noise, cavities and the like, ensures the consistency of the output result and the original point cloud, and can obtain a high-precision indoor scene model.

Indoor scene three-dimensional modeling method based on structure

Technical Field

The invention relates to the technical field of three-dimensional modeling, in particular to a structure-based indoor scene three-dimensional modeling method.

Background

Three-dimensional modeling of indoor scenes is the basis for many applications, such as positioning and navigation, building maintenance and renovation planning, emergency management, and generating game scenes in virtual reality. A person without experience in interior design may provide a three-dimensional digital representation of the housing to an expert or expert system for better furniture placement recommendations. The indoor model with rich semantics and high geometric precision provides key information (such as the position of a door for an exit, the opening direction of the door, the position and the topological relation of an indoor space, and the semantic attribute of the indoor space) for indoor navigation service. Three-dimensional modeling of indoor environments still faces particular challenges due to the complex layout of indoor structures, complex interactions between objects, clutter and occlusions. (1) During the data collection process, it is difficult to obtain data about walls, floors, and other structures of interest due to lack of view coverage, resulting in undesirable reconstruction results. (2) Restoring internal structures and the topological relationships between them (e.g., connectivity, containment, or adjacency) is difficult. (3) Weakly textured areas (such as featureless walls or floors) are often present in indoor environments, which leads to photo consistency measurement errors. (4) Sensor noise and outliers further complicate the modeling process. (5) The main appearance differences between different scenes, lighting and view variations also make the automatic and robust generation of an indoor model very challenging. The above points result in a number of defects in the point cloud model of the indoor scene generated by the data collected by the sensor: and a large amount of noise and abnormal points, cavities caused by shielding, materials and the like, low modeling precision of small objects and the like.

Disclosure of Invention

The invention aims to provide a structure-based indoor scene three-dimensional modeling method aiming at the defects of the prior art.

In order to achieve the purpose, the invention adopts the following technical scheme: a structure-based indoor scene three-dimensional modeling method comprises the following steps:

s1: removing abnormal points and small objects in the point cloud data of the indoor scene through plane segmentation;

s2: performing polygon fitting on a plane, including extracting plane boundaries, performing neighborhood estimation on each boundary point, calculating the normal direction of each boundary point, smoothing boundaries by moving the positions of the boundary points in the normal direction, and extracting polygons by using an angular point detection algorithm;

s3: searching the adjacency relation among points, lines and surfaces of the polygon set according to the self-adaptive search radius; establishing a graph structure according to the adjacency relation, and carrying out local pruning and global pruning;

s4: constructing an objective function according to the adjacency relation, the planarity and the orthogonality of the polygon set and the fitting of the input point cloud, solving the objective function to perform coordinate transformation on the polygon, and automatically bonding the polygon set;

s5: an indoor scene model is generated based on the polygons.

Further, the S1 includes the following sub-steps:

s11: constructing a KDTree, performing neighborhood search on each point in the point cloud data by using a nearest neighbor algorithm, and calculating the normal direction and curvature of each point by using a PCA algorithm;

s12: performing plane segmentation by adopting region growth based on distance and normal vector judgment;

s13: the plane equation is fitted to each plane using the PCA algorithm, removing outliers and small objects from the plane information (number of points, plane normal, etc.).

Further, in S2, projecting all points in the plane point set onto a plane according to a plane equation, converting the three-dimensional coordinates into two-dimensional coordinates, and extracting a two-dimensional point cloud boundary using an α -shape algorithm; a forward search method is applied to classify the locally smooth regions around each boundary point.

Further, in S2, calculating and optimizing the normal direction of each boundary point specifically includes: initializing a normal direction by using a PCA algorithm in a neighborhood, and optimizing the normal direction of the boundary point by using a least square method, wherein an optimization equation is as follows:

Further, in S2, the boundary is smoothed by moving the boundary point position in the normal direction, which can be expressed by the following equation:

p′＝p+t_pn_p

wherein t is_pThe distance of the boundary point moving in the normal direction is shown, and p' is the coordinate of the moved boundary point;

the new boundary point position is obtained by minimizing the energy function:

the first term smoothes the point cloud boundary by minimizing the dot product of the connecting line of q, p and the normal direction of q, p; the second term is used for preventing the point cloud boundary points from deviating from the initial positions thereof too much, and mu is a constraint term coefficient.

Further, in S3, searching for the adjacency relation between the points, lines, and surfaces of the polygon set according to the adaptive search radius includes:

s31: matching points with points;

S32: matching points with edges;

for a side e ═ p₀，p₁) Its search radius is defined as the minimum search radius of its two end points, r (e) min (r (p)₀)，r(p₁) ); if the orthogonal projection of vertex p on e is inside e, and the following formula is satisfied, then p matches e:

d(p，e)≤min(r(p)，r(e))

d(l，p)≤r(p)and d(l，e)≤r(e)

s33: for vertex p and face f, if the orthogonal projection of p on f is inside the polygon and the projection distance is less than r (p), then p and f match; two edges match if there are two vertex-vertex matches, or one vertex-vertex and one vertex-edge match, or two vertex-edge matches, for the endpoints of the two edges.

Further, in S3, a graph structure is established according to the adjacency relation, and local pruning is performed, specifically:

establishing graph structure G ═ V, E_M) To represent the matching relationship of the polygon set, all the vertexes and edges of the polygon set P form V, E_MContains all vertex-vertex/edge matches;

vertex-to-vertex matching typically produces stable results, correcting mismatching due to adding the closest vertex to the candidate set in the following pruning step: considering two or more vertices Q of a polygon Q_iMatching with the vertex P belonging to P, only keeping the matching pair with the nearest distance, and representing all single ring neighborhoods of the search G in the matching graph G;

intrinsic stability requires pruning vertex-edge matches where the vertices correspond to multiple non-adjacent edges of the polygon, which occurs due to overlap of search radii around the edges, which is compressed to the nearest vertex-edge match using graph G.

Further, in S3, constructing an expanded matching graph, and performing global pruning, specifically:

constructing an extended matching graph G_e＝(V，E_e) V contains all polygon elements, E_e＝E_M∪E_CContains all the matching relationships E_MAnd a constraint set E_C；E_CConnecting all element pairs in the V of the same polygon except the polygon vertex and the connecting edge thereof; according to the expanded matching graph, judging that the matching m of any vertex-vertex/edge is equal to (p, q) epsilon E_MWhether part of a detrimental cycle; find m cycles c (m), e (m, e) directly connected to another match on both sides₀，...，e_n)，e₀，...，e_n∈E_MIf such a cycle is found, the corresponding match is directly deleted;

for match m, search the actual match graph G for a matching sequence, check if all indirectly induced matches m_iAre also all at E_MFor being not in E_MPerforming geometric validation on each match to determine whether polygon degradation would result; and (3) projecting the matched vertex and/or edge endpoint to a common intersection line l of the plane where the polygon is located, and pruning m if the normal vector direction of the polygon corrected by vertex/edge projection is reversed or has self-intersection.

Further, the S4 specifically includes:

with adjacency relationships, data items are defined as the distance between matching elements:

to satisfy the ubiquitous orthogonality of indoor scenes, the following two orthogonal terms are used:

by minimizing the initial point p_iTo current vertex position p'_iThe sum of the squared distances to overcome the degradation problem that may lead to edges, the distance term is as follows:

the objective function is expressed as:

E＝λ_dataE_data+λ_consE_cons+λ_orth(E_orth1+E_orth2)+λ_curE_cur

wherein λ_data、λ_cons、λ_orth、λ_curAre all weight coefficients.

Further, on the basis of automatically creating and bonding polygons, an interactive editing tool is adopted for processing polygon errors or deletions caused by scene occlusion and material problems; automatically selecting a proper two-dimensional modeling space based on a partition plane in the point cloud, and simplifying all interactive operations into approximate two-dimensional operations; the interactive operation comprises polygon editing, polygon drawing, polygon boundary alignment to image boundary, polygon material giving and the like.

The invention has the beneficial effects that: the method processes the scene point cloud model based on the planar structure and the polygonal structure, successfully processes the problems of noise, cavities and the like in the original point cloud data, ensures the consistency of the output result and the original point cloud, and can obtain a high-precision indoor scene model. The method can be applied to visual positioning, virtual roaming and other applications.

Drawings

FIG. 1 is a flow chart of a method for three-dimensional modeling of an indoor scene based on a structure according to an embodiment of the present invention;

FIG. 2 shows an original point cloud (left) and a plane segmentation result (right) provided by an embodiment of the present invention;

FIG. 3 is a flow chart of polygon fitting provided by an embodiment of the present invention;

FIG. 4 is a schematic diagram of an error match provided by an embodiment of the present invention;

FIG. 5 shows a set V (left) and a constraint set E according to an embodiment of the present invention_C(right);

FIG. 6 is a diagram of a loop provided by an embodiment of the present invention that includes multiple constrained edges that may or may not result in polygon degradation (left);

FIG. 7 is a comparison of polygons before and after automatic bonding as provided by an embodiment of the present invention;

FIG. 8 is a schematic illustration of outliers and small object removal provided by an embodiment of the present invention;

FIG. 9 is a schematic diagram of hole repair provided by an embodiment of the present invention.

Detailed Description

The present invention will be described in further detail with reference to the following drawings and specific embodiments, it being understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

The core content of the three-dimensional modeling is to realize the digital expression of a three-dimensional physical world by comprehensively utilizing a sensor and a computing technology, and capture the three-dimensional shape and appearance with high reality sense of an object and a scene so as to simulate three-dimensional interaction and perception in a digital space.

Fig. 1 is a flowchart of a structure-based three-dimensional modeling method for an indoor scene provided in an embodiment of the present invention, where an implementation flow of the method is specifically as follows:

firstly, removing abnormal points and small objects in the point cloud data of the indoor scene through plane segmentation.

The first step is as follows: and performing neighborhood search and normal and curvature estimation on each point in the point cloud data. The method specifically comprises the following steps:

randomly selecting a point from the input point cloud, constructing a KDTree, searching a neighborhood for each point by using a nearest neighbor (KNN) algorithm, and calculating the normal direction and the curvature of each point by using a PCA algorithm.

The second step is that: and performing plane segmentation by adopting region growing based on distance and normal vector judgment. The method specifically comprises the following steps:

(1) selecting a point with the minimum curvature and no clustering, and adding the point to a current plane point list C; if all points are clustered, the traversal is stopped.

(2) Sequentially selecting points which are not accessed in C as seed points p_seed。

(3) Calculating p_seedEvery point p in the neighborhood_iTo p_seedEuclidean distance ED of_iAnd storing the data in an array EDs, and recording the median of the EDs as mean _ ED.

(4) Calculating p_seedEvery point p in the neighborhood_iTo p_seedIs orthogonal distance OD_iAnd storing the data in an array ODs, and recording the median of the ODs as mean _ OD. The median absolute difference MAD is calculated by:

where b is a constant and takes the value of 1.4826.

(5) For p_seedEvery point p in the neighborhood_iR is calculated by the following formula_zi，R_ziRepresents p_iAnd p_seedAnd (5) clustering weight scores.

If p is_iNot clustered and simultaneously satisfy R_ziLess than the weight threshold (usually set to 2), ED_i＜median_ED，p_iNormal vector of (a) and p_seedIs smaller than an angle threshold (typically set to 10-20 degrees), it is added to the current plane point list C.

(6) If the capacity of C increases, repeating steps (2) - (5). Otherwise, the current plane extraction is finished, C is stored in the segmentation result point list R, and the next plane extraction is carried out by turning to the step (1).

The third step: after plane segmentation is finished, fitting a plane equation to each plane by applying a PCA method, and removing abnormal points and small objects according to plane information;

because the point cloud data of the indoor scene is generally obtained through laser scanning, and the modeling precision of small objects in the point cloud data is low, and the small objects have the characteristics of easy displacement, the method removes the small objects from the scene. Small objects and abnormal points caused by walking of pedestrians are removed through the conditions of the number of vertexes of each plane, the normal direction and the like. As shown in fig. 2, the plane segmentation method adopted by the present invention can effectively segment the original point cloud. In the figure, the problem of the point cloud after segmentation can be visually seen: (1) holes and noise due to occlusion, scanning accuracy; (2) gaps appear between the divided planes and are not communicated. Therefore, the invention adopts an optimization-based polygon bonding algorithm to obtain a closed complete polygon model without holes and noise.

And secondly, performing polygon fitting on the plane, including extracting plane boundaries, performing neighborhood estimation on each boundary point, calculating the normal direction of each boundary point, smoothing the boundary by moving the positions of the boundary points in the normal direction, and extracting polygons by using an angular point detection algorithm. The method comprises the following concrete steps:

(1) and projecting all the points in the plane point set onto a plane according to a plane equation, converting the three-dimensional coordinates into two-dimensional coordinates, and extracting the boundary of the two-dimensional point cloud by using an alpha-shape algorithm of the CGAL library. As shown in fig. 3(a), the four-pointed star indicates the boundary.

(2) The local smooth region around each boundary point is classified using a forward search method that preserves local features and is robust to noise and outliers (as shown in fig. 3 (b)).

(3) The normal direction of each boundary point is calculated and optimized. The normal direction is initialized by using the PCA algorithm within the neighborhood, as in fig. 3 (c). In order to make the normal directions of the points on the same side consistent, as shown in fig. 3(d), the method optimizes the normal direction of the boundary point by using the least square method, and the optimization equation is as follows:

(4) Smoothing the boundary by moving the boundary point position in the normal direction can be represented by:

p′＝p+t_pn_p

wherein t is_pThe distance of the boundary point moving in the normal direction is shown, and p' is the coordinate of the moved boundary point;

the new boundary point position is obtained by minimizing the energy function:

the first term smoothes the point cloud boundary by minimizing the dot product of the connecting line of q, p and the normal to q, p. The second term is used to prevent the point cloud boundary points from deviating too much from their original positions, thereby avoiding the shrinkage of the point cloud boundary. The constraint term coefficient μ is set to 0.1 in the present embodiment. The resulting smoothed point cloud boundary is shown in fig. 3 (e).

(5) The polygons are extracted using a corner detection algorithm, as in fig. 3 (f). If the normal vector of a certain boundary point is almost parallel to the normal vectors of the succeeding boundary point and the preceding boundary point, respectively, it is classified as a point on a certain side of the polygon. Then, a straight line equation of the point set on each edge is fitted using a least square method. The intersection point of two continuous edges is the vertex of the polygon.

Thirdly, searching the adjacency relation among points, lines and surfaces of the polygon set according to the self-adaptive search radius; and establishing a graph structure according to the adjacency relation, and carrying out local pruning and global pruning. Specifically, the invention provides an automatic robust adjacency detection method based on stable vertex-vertex/edge/face and edge-edge matching, which comprises the following steps:

(1) matching of points to points.

||p-q||≤min(r_e(p)，r_e(q))

finally, in a later optimization phase the two matching vertices collapse to a point on the intersection line l of the planes they lie in. Therefore, the method further requires that the distance from a pair of matching points to l is:

d(l，p)≤r_e(p)and d(l，q)≤r_e(q)

(2) matching points with edges.

For a side e ═ p₀，p₁) Its search radius is defined as the minimum search radius of its two end points, r (e) min (r (p)₀)，r(p₁)). If the orthogonal projection of vertex p on e is inside e and satisfies the following two equations, p matches e:

d(p，e)≤min(r(p)，r(e))

d(l，p)≤r(p)and d(l，e)≤r(e)

(3) and (4) other matching.

For vertex p and face f, if the orthogonal projection of p on f is inside the polygon and the projection distance is less than r (p), then p and f match. Based on vertex-vertex and vertex-edge matching, an edge-edge match can be further derived: two edges are said to match if there are two vertex-vertex matches, or one vertex-vertex and one vertex-edge match, or two vertex-edge matches, for the endpoints of the two edges. Edge-to-edge matching is only used in the global pruning phase and not for optimization, since their contribution to optimization is implicitly included by vertex-vertex/edge matching.

(4) And (5) local pruning.

Vertex-to-vertex matching typically produces stable results, correcting mismatching due to adding the closest vertex to the candidate set in the following pruning step: considering two or more vertices Q of a polygon Q_iMatching with vertex P ∈ P. This obviously violates the intrinsic stability requirement of polygon Q, leaving only the nearest matching pairs. All single ring neighborhoods denoted as search G in the matching graph G.

Similar to vertex-vertex matching, intrinsic stability requires pruning those vertex-edge matches for which the vertices correspond to multiple non-adjacent edges of the polygon. This naturally occurs due to the overlap of search radii around the edges. This matching case is compressed to the nearest vertex-edge match using graph G.

(5) And (6) global pruning. As shown in fig. 4, the distances of the 3 polygons are close, and the matching of p and q causes the side e of the central polygon to collapse, which violates the intrinsic stability requirements of the polygons. This degradation occurs when certain polygon elements (vertices/edges) are connected by matching, forming a loop.

The method introduces an extended matching graph G_e＝(V，E_e). Like G, the set of points V contains all the polygon elements (vertices and edges). Edge set E_e＝E_M∪E_CContains all the matching relationships E_MAnd a constraint set E_C. As shown in FIG. 5, E_CAll element pairs in V that connect the same polygon (except for the polygon vertex and its connected edge). According to the expanded matching graph, the matching m of any vertex-vertex/edge can be judged to be (p, q) epsilon E_MWhether part of a harmful cycle. Since the degradation of the bounding edges of the polygons for p and q has already been dealt with in the local pruning stage, it is only necessary to find the loop c (m), m ═ and (m, e), which connects directly to the other match on both sides of m₀，..，e_n)，e₀，...，e_n∈E_M. If such a loop is found, the corresponding match can be directly deleted.

By pruning the matching loop with only one constraint edge, most of the degradation in the polygon model can be avoided. However, there are also cases involving multiple constraining edges, see FIG. 6, which may or may not result in polygon degradation. In order to solve the problem, the method provides a pruning method based on a search matching sequence. Matching sequences will cause further matches between the elements they connect. In a sense, these elements will also be connected during the optimization phase. Therefore, it is necessary to verify whether the indirectly induced matching will cause polygon degradation. For match m, the matching sequence is searched in the actual matching graph G. Then, check if all indirectly induced matches m_iAre also all at E_MIn (1). For being out of E_MNeeds to be geometrically validated to determine if polygon degradation would result. To geometrically verify whether an indirectly induced match causes any polygon degradation, the matched vertices and/or edge endpoints are projected onto a common intersection line/of the planes in which the polygons lie. If there is a flip or self-intersection in the normal vector direction of the polygon corrected by the vertex/edge projectionIn case (2), pruning is performed on m.

Fourthly, according to the adjacency relation, the planarity and the orthogonality of the polygon set and the fitting of the input point cloud, constructing an objective function, solving the objective function to perform coordinate transformation on the polygon, and automatically bonding the polygon set; the method specifically comprises the following steps:

using the adjacency found in the previous step, the method defines the data item as the distance between the matching elements:

wherein d is(p_i，p_j) Representing a vertex p_iTo the vertex p_jD (p) of_i，e_k) Representing a vertex p_iTo edge e_kD (p) of_i，P_l) Representing a vertex p_iTo plane P_lThe distance of (c).

in order to meet the ubiquitous orthogonality of indoor scenes, the method provides the following two orthogonal terms:

in summary, the objective function is expressed as:

E＝λ_dataE_data+λ_consE_cons+λ_orth(E_orth1+E_orth2)+λ_curE_cur

wherein λ_data、λ_cons、λ_orth、λ_curAre all weight coefficients. In this embodiment, the following settings are set: lambda [ alpha ]_data＝1，λ_cons＝0.5，λ_orth0.01 and λ_cur＝0.1。

Generating an indoor scene model based on the polygon, wherein the specific application comprises the following steps: giving materials to each polygon, and generating an indoor scene model for virtual reality; a point cloud model is obtained by sampling a polygon model, and then an indoor scene model with high reality is generated through meshing and texture mapping, so that the method can be applied to visual positioning, virtual roaming and other applications.

In addition, on the basis of automatically creating and bonding polygons, the method provides an interactive editing tool for processing polygon errors or deletions caused by scene occlusion, material and other problems. All interactive operations are simplified to approximate two-dimensional operations by automatically selecting a suitable two-dimensional modeling space based on the segmentation plane in the point cloud. One dimension is implicitly removed, which greatly reduces the complexity of interaction, thereby reducing the difficulty of the overall modeling work. The interactive operation comprises polygon editing, polygon drawing, polygon boundary alignment to image boundary, polygon material giving and the like.

The invention provides a point cloud processing method based on a plane structure and a polygon structure, aiming at the problems of noise, cavities and the like in a point cloud model acquired by a sensor. Firstly, a point cloud model is segmented, so that abnormal points and small objects in the point cloud model are eliminated. Next, polygonal bonding based on numerical optimization is performed to repair voids and gaps due to plane division caused by occlusion, material, and the like. As shown in fig. 8, due to a moving person or object, there are some outliers (represented by circles) in the original point cloud data, and the laser scanning has low acquisition accuracy for small objects, and the method adopts a plane segmentation method to remove them, as shown in fig. 8 (right). FIG. 9 shows that the method of the present invention can effectively repair the cavity in the point cloud data caused by the factors of angle, shielding, material, etc. Experiments show that the method can successfully process the problems of noise, cavities and the like, ensure the consistency of the output result and the original point cloud, and can obtain a high-precision indoor scene model.

In one embodiment, a computer device is provided, which includes a memory and a processor, the memory stores computer readable instructions, and the computer readable instructions, when executed by the processor, cause the processor to execute the steps of the structure-based indoor scene three-dimensional modeling method in the above embodiments.

In one embodiment, a storage medium is provided, in which computer readable instructions are stored, and when executed by one or more processors, the one or more processors perform the steps of the structure-based indoor scene three-dimensional modeling method in the embodiments. The storage medium may be a nonvolatile storage medium.

Those skilled in the art will appreciate that all or part of the steps in the methods of the above embodiments may be implemented by associated hardware instructed by a program, which may be stored in a computer-readable storage medium, and the storage medium may include: read Only Memory (ROM), Random Access Memory (RAM), magnetic or optical disks, and the like.

The above description is only for the purpose of illustrating the preferred embodiments of the one or more embodiments of the present disclosure, and is not intended to limit the scope of the one or more embodiments of the present disclosure, and any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the one or more embodiments of the present disclosure should be included in the scope of the one or more embodiments of the present disclosure.