TWI847355B - Reconstruction method of three dimensional model and computing apparatus - Google Patents

A reconstruction method of a three dimensional model and a computing apparatus are provided. In the method, multiple structures in the three-dimensional model are determined. An object in the three-dimensional model is separated from an original location, to form a hollow. The hollow is adjacent to at least two neighboring structures. The neighboring structure is extended to cover the hollow, and a similarity of the extended part covered on the hollow is determined. The similarity is a compared result between the extended part and the neighboring structures. The target structure of the hollow is determined according to the similarity and is used for compensating the hollow.

三維模型的重建方法及運算裝置Three-dimensional model reconstruction method and computing device

本發明是有關於一種空間建模技術，且特別是有關於一種三維模型的重建方法及運算裝置。The present invention relates to a spatial modeling technology, and in particular to a three-dimensional model reconstruction method and computing device.

為了模擬真實空間，可以對真實空間進行掃描以產生看起來像真實空間的模擬空間。模擬空間可實現在諸如遊戲、家居佈置、機器人移動等應用。值得注意的是，可移動或刪除模擬空間中的物件，但會在模擬空間中留下空洞。To simulate a real space, the real space can be scanned to produce a simulated space that looks like the real space. The simulated space can be used in applications such as games, home decoration, and robot movement. It is worth noting that objects in the simulated space can be moved or deleted, but holes will be left in the simulated space.

本發明實施例提供一種三維模型的重建方法及運算裝置，可補償空洞。The embodiment of the present invention provides a three-dimensional model reconstruction method and computing device, which can compensate for voids.

本發明實施例的三維模型的重建方法包括(但不僅限於)下列步驟：決定三維模型中的多個結構。三維模型中的物件自原位置分離而形成空洞，且空洞相鄰於那些結構中的至少二個鄰近結構。分別延伸那些鄰近結構至覆蓋空洞，並決定覆蓋至空洞的延伸部分的相似度，且相似度是與延伸部分的相鄰結構的比較結果。依據相似度決定空洞的目標結構，且目標結構用於補償空洞。The reconstruction method of the three-dimensional model of the embodiment of the present invention includes (but is not limited to) the following steps: determining multiple structures in the three-dimensional model. An object in the three-dimensional model is separated from its original position to form a hole, and the hole is adjacent to at least two neighboring structures among those structures. Extending those neighboring structures to cover the hole respectively, and determining the similarity of the extended parts covering the hole, and the similarity is the comparison result with the neighboring structures of the extended parts. Determining the target structure of the hole based on the similarity, and the target structure is used to compensate the hole.

本發明實施例的運算裝置包括記憶體及處理器。記憶體用以儲存程式碼。處理器耦接記憶體。處理器載入程式碼以執行決定三維模型中的多個結構，分別延伸那些鄰近結構至覆蓋空洞，決定覆蓋至空洞的延伸部分的相似度，且依據相似度決定空洞的目標結構。三維模型中的物件自原位置分離而形成空洞，且空洞相鄰於那些結構中的至少二個鄰近結構。相似度是與延伸部分的相鄰結構的比較結果。目標結構用於補償空洞。The computing device of the embodiment of the present invention includes a memory and a processor. The memory is used to store program codes. The processor is coupled to the memory. The processor loads the program code to execute the determination of multiple structures in the three-dimensional model, respectively extending those neighboring structures to cover the hole, determining the similarity of the extended parts covering the hole, and determining the target structure of the hole based on the similarity. Objects in the three-dimensional model are separated from their original positions to form a hole, and the hole is adjacent to at least two neighboring structures among those structures. The similarity is the comparison result with the neighboring structures of the extended parts. The target structure is used to compensate for the hole.

基於上述，依據本發明實施例的三維模型的重建方法及運算裝置，延伸相鄰結構至空洞，並依據延伸部分與相鄰結構之間的相似度決定空洞的目標結構。藉此，可將原始空間模型裡的空洞進行復原、衍伸或模型再創造，以構建全新完整的空間模型。Based on the above, according to the reconstruction method and computing device of the three-dimensional model of the embodiment of the present invention, the adjacent structure is extended to the void, and the target structure of the void is determined according to the similarity between the extended part and the adjacent structure. In this way, the void in the original spatial model can be restored, extended or re-modeled to construct a new and complete spatial model.

為讓本發明的上述特徵和優點能更明顯易懂，下文特舉實施例，並配合所附圖式作詳細說明如下。In order to make the above features and advantages of the present invention more clearly understood, embodiments are specifically cited below and described in detail with reference to the accompanying drawings.

圖1是依據本發明一實施例的運算裝置10的元件方塊圖。請參照圖1，運算裝置10可以是手機、平板電腦、桌上型電腦、筆記型電腦、伺服器或智能助理裝置。運算裝置10包括(但不僅限於)記憶體11及處理器12。FIG1 is a block diagram of a computing device 10 according to an embodiment of the present invention. Referring to FIG1 , the computing device 10 may be a mobile phone, a tablet computer, a desktop computer, a laptop computer, a server, or an intelligent assistant device. The computing device 10 includes (but is not limited to) a memory 11 and a processor 12.

記憶體11可以是任何型態的固定或可移動隨機存取記憶體(Radom Access Memory，RAM)、唯讀記憶體(Read Only Memory，ROM)、快閃記憶體(flash memory)、傳統硬碟(Hard Disk Drive，HDD)、固態硬碟(Solid-State Drive，SSD)或類似元件。在一實施例中，記憶體11用以儲存程式碼、軟體模組、資料(例如，位置、深度、或三維模型)或檔案，其詳細內容待後續實施例詳述。The memory 11 may be any type of fixed or removable random access memory (RAM), read only memory (ROM), flash memory, traditional hard disk drive (HDD), solid-state drive (SSD) or similar components. In one embodiment, the memory 11 is used to store program code, software modules, data (e.g., position, depth, or three-dimensional model) or files, and its details will be described in detail in subsequent embodiments.

處理器12耦接記憶體11。處理器12可以是中央處理單元(Central Processing Unit，CPU)，或是其他可程式化之一般用途或特殊用途的微處理器(Microprocessor)、數位信號處理器(Digital Signal Processor，DSP)、可程式化控制器、特殊應用積體電路(Application-Specific Integrated Circuit，ASIC)或其他類似元件或上述元件的組合。在一實施例中，處理器12用以執行運算裝置10的所有或部份作業，且可載入並執行記憶體11所儲存的程式碼、軟體模組、檔案及/或資料。在一實施例中，處理器12執行本發明實施例的所有或部分操作。在一些實施例中，記憶體11所儲存的那些軟體模組或程式碼也可能是實體電路所實現。The processor 12 is coupled to the memory 11. The processor 12 may be a central processing unit (CPU), or other programmable general-purpose or special-purpose microprocessor (Microprocessor), digital signal processor (Digital Signal Processor, DSP), programmable controller, application-specific integrated circuit (Application-Specific Integrated Circuit, ASIC) or other similar components or a combination of the above components. In one embodiment, the processor 12 is used to execute all or part of the operations of the computing device 10, and can load and execute the program code, software module, file and/or data stored in the memory 11. In one embodiment, the processor 12 executes all or part of the operations of the embodiment of the present invention. In some embodiments, the software modules or program codes stored in the memory 11 may also be implemented by physical circuits.

下文中，將搭配運算裝置10中的各項元件說明本發明實施例所述之方法。本方法的各個流程可依照實施情形而隨之調整，且並不僅限於此。Hereinafter, the method described in the embodiment of the present invention will be described in conjunction with various components in the computing device 10. The various processes of the method can be adjusted according to the implementation situation, and are not limited thereto.

圖2是依據本發明一實施例的三維模型的重建方法的流程圖。請參照圖2，處理器12決定三維模型中的多個結構(步驟S210)。具體而言，三維模型可以是透過感測器(例如，影像擷取裝置、光達(LiDAR)、飛行時間(Time-of-Flight，ToF)偵測器或慣性測量單元(Inertial Measurement Unit，IMU))掃描真實空間所得的模型，也可以是在模型編輯軟體創造或修改的模型。FIG2 is a flow chart of a method for reconstructing a three-dimensional model according to an embodiment of the present invention. Referring to FIG2 , the processor 12 determines a plurality of structures in the three-dimensional model (step S210). Specifically, the three-dimensional model may be a model obtained by scanning a real space through a sensor (e.g., an image capture device, a LiDAR, a Time-of-Flight (ToF) detector, or an Inertial Measurement Unit (IMU)), or a model created or modified in a model editing software.

三維模型可能包括(虛擬)物件及結構。物件例如是桌、椅、家電、或櫃架。結構例如是牆、地板、天花板、或梁柱。在一些實施例中，處理器12可預先定義結構的類型，並將結構以外的單元都視為物件。處理器12可基於神經網路的演算法(例如，YOLO(You only look once)、基於區域的卷積神經網路(Region Based Convolutional Neural Networks，R-CNN)、或快速R-CNN(Fast CNN))或是基於特徵匹配的演算法(例如，方向梯度直方圖(Histogram of Oriented Gradient，HOG)、尺度不變特徵轉換(Scale-Invariant Feature Transform，SIFT)、Harr、或加速穩健特徵(Speeded Up Robust Features，SURF)的特徵比對)辨識物件及/或結構的類型。The 3D model may include (virtual) objects and structures. Objects are, for example, tables, chairs, appliances, or shelves. Structures are, for example, walls, floors, ceilings, or beams. In some embodiments, the processor 12 may predefine the type of structure and treat all units other than structures as objects. The processor 12 can recognize the type of objects and/or structures based on a neural network algorithm (e.g., YOLO (You only look once), Region Based Convolutional Neural Networks (R-CNN), or Fast R-CNN) or a feature matching algorithm (e.g., Histogram of Oriented Gradient (HOG), Scale-Invariant Feature Transform (SIFT), Harr, or Speeded Up Robust Features (SURF) feature matching).

舉例而言，圖3A至圖3D是依據本發明一實施例的模型重建的示意圖。請參照圖3A，三維模型包括沙發O(即，物件)、牆S1、地板S2及牆S3(即，結構)。For example, Figures 3A to 3D are schematic diagrams of model reconstruction according to an embodiment of the present invention. Referring to Figure 3A, the three-dimensional model includes a sofa O (ie, an object), a wall S1, a floor S2, and a wall S3 (ie, a structure).

在一實施例中，三維模型中的物件自原位置分離而形成空洞。例如，輸入裝置(例如，滑鼠、鍵盤或觸控面板，圖未示)接收用於移動或拖移三維模型中的虛擬物件的操作。或者，物件選單包括多種物件類型以供選擇，並據以刪除所選的物件。舉例而言，請參照圖3A及圖3B，沙發O自原位置OL分離，使三維模型產生空洞H。空洞H可以是三維模型中未指派深度資訊及/或材質的部分。In one embodiment, an object in a three-dimensional model is separated from its original position to form a hole. For example, an input device (e.g., a mouse, keyboard, or touch panel, not shown) receives an operation for moving or dragging a virtual object in the three-dimensional model. Alternatively, an object menu includes a plurality of object types for selection, and the selected object is deleted accordingly. For example, referring to FIG. 3A and FIG. 3B , a sofa O is separated from its original position OL, resulting in a hole H in the three-dimensional model. The hole H may be a portion of the three-dimensional model to which depth information and/or material is not assigned.

在一實施例中，空洞相鄰於那些結構中的至少二個鄰近結構。也就是，空洞位於複數個結構的相連處。若空洞僅位於一個結構，則空洞屬於所處結構的可能性很高。然而，若空洞相鄰於兩個結構以上，則空洞可能是其中一個結構或兩個都是。以圖3B為例，假設以圖式所呈現的視角而言，空洞H遮擋部分的牆S1及地板S2(因直接相鄰於空洞H而作為鄰近結構)。因此，需要進一步確定空洞H的所屬結構。In one embodiment, the void is adjacent to at least two neighboring structures among those structures. That is, the void is located at the junction of multiple structures. If the void is located in only one structure, the possibility that the void belongs to the structure is high. However, if the void is adjacent to more than two structures, the void may be one of the structures or both. Taking FIG. 3B as an example, assuming that from the perspective presented in the figure, the void H blocks part of the wall S1 and the floor S2 (because they are directly adjacent to the void H and are considered as neighboring structures). Therefore, it is necessary to further determine the structure to which the void H belongs.

請參照圖2，處理器12分別延伸鄰近結構至覆蓋空洞，並決定覆蓋至空洞的延伸部分的相似度(步驟S220)。具體而言，處理器12可基於先驗知識分析三維模型中的空洞的初始結構類型。先驗知識例如是相鄰結構的類型。以圖3B為例，空洞H的初始結構類型可以是牆(對應於牆S1)或地板(對應於地板S2)。Referring to FIG. 2 , the processor 12 extends the neighboring structures to cover the voids, and determines the similarity of the extended portions covering the voids (step S220). Specifically, the processor 12 may analyze the initial structure type of the voids in the three-dimensional model based on prior knowledge. The prior knowledge may be, for example, the type of the neighboring structures. Taking FIG. 3B as an example, the initial structure type of the void H may be a wall (corresponding to the wall S1) or a floor (corresponding to the floor S2).

接著，處理器12可自鄰近結構與空洞的交界處朝空洞延伸鄰近結構，使鄰近結構的延伸部分覆蓋空洞。舉例而言，請參照圖3C，延伸部分EP是自位於空洞H與地板S2(即，鄰近結構)的交界處的原點L1延伸至延伸點L2，並據以覆蓋空洞H。而延伸結構的類型相同於延伸來源，即地板。請參照圖3D，延伸部分EP繼續延伸到延伸點L3。Next, the processor 12 may extend the neighboring structure from the boundary between the neighboring structure and the cavity toward the cavity, so that the extended portion of the neighboring structure covers the cavity. For example, referring to FIG. 3C , the extended portion EP extends from the origin L1 located at the boundary between the cavity H and the floor S2 (i.e., the neighboring structure) to the extension point L2, and covers the cavity H accordingly. The type of the extended structure is the same as the extension source, i.e., the floor. Referring to FIG. 3D , the extended portion EP continues to extend to the extension point L3.

在一實施例中，處理器12可依據延伸部分中的多個延伸點與原點之間的距離決定那些延伸點的比重。原點位於所延伸的鄰近結構與空洞的交界處。例如，圖3C的原點L1位於地板S2與空洞H的交界處。一般而言，延伸點與原點的距離越近，則延伸點所屬的結構較有可能屬於原點所屬的結構，且處理器12可定義較大的比重。另一方面，延伸點與原點的距離越遠，則延伸點所屬的結構較不可能屬於原點所屬的結構，且處理器12可定義較小的比重。In one embodiment, the processor 12 may determine the weight of multiple extension points in the extension portion based on the distance between those extension points and the origin. The origin is located at the junction of the extended neighboring structure and the cavity. For example, the origin L1 of FIG3C is located at the junction of the floor S2 and the cavity H. Generally speaking, the closer the distance between the extension point and the origin, the more likely the structure to which the extension point belongs belongs to the structure to which the origin belongs, and the processor 12 may define a larger weight. On the other hand, the farther the distance between the extension point and the origin, the less likely the structure to which the extension point belongs belongs to the structure to which the origin belongs, and the processor 12 may define a smaller weight.

在一實施例中，比重與延伸點與原點之間的距離的平方成反比。以圖3C及圖3D為例，原點L1與延伸點L2的距離為10公分，則比重為4；原點L1與延伸點L3的距離為20公分，則比重為1。In one embodiment, the specific gravity is inversely proportional to the square of the distance between the extension point and the origin. For example, in FIG. 3C and FIG. 3D , if the distance between the origin L1 and the extension point L2 is 10 cm, the specific gravity is 4; if the distance between the origin L1 and the extension point L3 is 20 cm, the specific gravity is 1.

處理器12可決定那些延伸點的比重及對應的相似度的加權運算。相似度是與延伸部分的相鄰結構的比較結果。例如，相鄰結構的影像特徵與衍伸點的影像特徵之間的誤差或特徵空間中的距離可以代表相似度。若誤差越小，則相似度越高；若誤差越大，則相似度越低。又例如，相似度是相同特徵的數量總和或數量加權總和。The processor 12 may determine the weight of those extension points and the corresponding weighted operation of similarity. The similarity is the result of the comparison with the adjacent structure of the extension. For example, the error between the image features of the adjacent structure and the image features of the extension point or the distance in the feature space may represent the similarity. If the error is smaller, the similarity is higher; if the error is larger, the similarity is lower. For another example, the similarity is the sum of the number of identical features or the weighted sum of the number.

請參照圖2，處理器12依據相似度決定空洞的一個或更多個目標結構(步驟S230)。具體而言，目標結構用於補償空洞。在一實施例中，處理器12可依據加權運算決定匹配函數。處理器12可透過最小平方法(least squares method)、均方誤差(Mean Square Error，MSE)、均方根誤差(Root-Mean-Square Error，RMSE)、最小均方誤差(Least-Mean-Square Error，LMSE)、餘弦相似性、餘弦距離或其他誤差相關演算法決定匹配函數。Referring to FIG. 2 , the processor 12 determines one or more target structures of the hole based on the similarity (step S230). Specifically, the target structure is used to compensate the hole. In one embodiment, the processor 12 may determine the matching function based on a weighted operation. The processor 12 may determine the matching function by using a least squares method, mean square error (MSE), root mean square error (RMSE), least mean square error (LMSE), cosine similarity, cosine distance, or other error-related algorithms.

接著，處理器12可依據這匹配函數最小化延伸部分與目標結構之間的誤差。處理器12可改變延伸點及/或延伸部分的結構類型，並決定對應的匹配函數。而針對各延伸點，可得出最小的誤差所對應的結構即是這延伸點的目標結構。因此，由空洞與鄰近結構的交界處延伸到最近的延伸點，處理器12可定義交界處至這延伸點的目標結構(即，相似度最高或誤差最小)。接著，延伸至下一個接近的延伸點，處理器12定義這個延伸點的目標結構，且其餘依此類推。處理器12可進一步定義延伸點的深度資訊。這深度資訊可基於相鄰結構的延伸線。Then, the processor 12 can minimize the error between the extended portion and the target structure based on the matching function. The processor 12 can change the structural type of the extension point and/or the extended portion and determine the corresponding matching function. For each extension point, the structure corresponding to the minimum error can be obtained, which is the target structure of this extension point. Therefore, extending from the junction of the void and the neighboring structure to the nearest extension point, the processor 12 can define the target structure from the junction to this extension point (i.e., the highest similarity or the smallest error). Then, extending to the next close extension point, the processor 12 defines the target structure of this extension point, and so on. The processor 12 can further define the depth information of the extension point. This depth information can be based on the extension line of the neighboring structure.

在一實施例中，處理器12依據至少二個鄰近結構的材質決定目標結構的材質。例如，處理器12可利用大數據(也就是，儲存多種結構與對應材質的資料庫)且依據空洞周遭的材質選擇原本受所分離的物件遮擋的部分的材質，並據以貼上所選擇的材質。所選擇的材質可能相同或相似於鄰近結構的材質。In one embodiment, the processor 12 determines the material of the target structure based on the materials of at least two neighboring structures. For example, the processor 12 may use big data (i.e., a database storing a variety of structures and corresponding materials) and select the material of the portion originally blocked by the separated object based on the material around the hole, and then attach the selected material accordingly. The selected material may be the same or similar to the material of the neighboring structure.

綜上所述，在本發明實施例的本發明實施例的三維模型的重建方法及運算裝置中，延伸相鄰結構到空洞，並據以決定目標結構。此外，參考空洞的周遭材質還原空洞的材質。藉此，可恢復完整成模型，並構建全新空間模型。In summary, in the reconstruction method and computing device of the three-dimensional model of the embodiment of the present invention, the adjacent structure is extended to the void, and the target structure is determined accordingly. In addition, the material of the void is restored with reference to the surrounding material of the void. In this way, the complete model can be restored and a new spatial model can be constructed.

雖然本發明已以實施例揭露如上，然其並非用以限定本發明，任何所屬技術領域中具有通常知識者，在不脫離本發明的精神和範圍內，當可作些許的更動與潤飾，故本發明的保護範圍當視後附的申請專利範圍所界定者為準。Although the present invention has been disclosed as above by the embodiments, they are not intended to limit the present invention. Any person with ordinary knowledge in the relevant technical field can make some changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be defined by the scope of the attached patent application.

10: 運算裝置 11: 記憶體 12: 處理器 S210~S230: 步驟 S1、S3: 牆 S2: 地板 O: 沙發 OL: 原位置 H: 空洞 L1: 原點 L2、L3: 延伸點 EP: 延伸部分 10: Computing device 11: Memory 12: Processor S210~S230: Steps S1, S3: Wall S2: Floor O: Sofa OL: Original position H: Hole L1: Origin L2, L3: Extension point EP: Extension part

圖1是依據本發明一實施例的運算裝置的元件方塊圖。 圖2是依據本發明一實施例的三維模型的重建方法的流程圖。 圖3A至圖3D是依據本發明一實施例的模型重建的示意圖。 FIG. 1 is a block diagram of components of a computing device according to an embodiment of the present invention. FIG. 2 is a flow chart of a method for reconstructing a three-dimensional model according to an embodiment of the present invention. FIG. 3A to FIG. 3D are schematic diagrams of model reconstruction according to an embodiment of the present invention.

S210~S230: 步驟S210~S230: Steps