CN114745010A - System and method relating to motion tracking - Google Patents

本发明提供一种相关于运动跟踪的系统和方法。系统包含可穿戴在用户的人体部位上的多个运动感测设备。在方法中，在第一持续时间内根据所述第一感测数据获得位置信息的第一部位，且在第二持续时间内根据第二感测数据获得位置信息的第二部位。第二感测数据是从图像传感器所捕获的图像所获得。第一感测数据不是从图像获得。将位置信息的第一部位和第二部位组合为组合位置信息。根据组合位置信息确定用户的运动信息。因此，可提高跟踪用户运动的准确性。

The present invention provides a system and method related to motion tracking. The system includes a plurality of motion sensing devices wearable on a user's body part. In the method, a first portion of location information is obtained from the first sensing data during a first duration, and a second portion of location information is obtained from second sensing data during a second duration. The second sensed data is obtained from an image captured by the image sensor. The first sensed data is not obtained from an image. The first part and the second part of the position information are combined into combined position information. The motion information of the user is determined according to the combined position information. Therefore, the accuracy of tracking the user's motion can be improved.

相关于运动跟踪的系统和方法Systems and methods related to motion tracking

技术领域technical field

本发明大体上涉及一种用于跟踪用户的运动的方法，具体地涉及一种相关于运动跟踪的系统和方法。The present invention generally relates to a method for tracking motion of a user, and in particular to a system and method related to motion tracking.

背景技术Background technique

为了在电子设备(诸如游戏机、计算机、智能手机、智能电器等)上提供直观的操作，可检测用户的运动，以根据用户的运动直接操作电子设备。In order to provide intuitive operation on electronic devices such as game consoles, computers, smart phones, smart appliances, etc., the user's motion may be detected to directly operate the electronic device according to the user's motion.

在传统技术中，一些电子设备可允许用户的人体部位(诸如手、腿、头等)来控制这些电子设备的操作，且可跟踪这些人体部位的运动。然而，这些电子设备仅提供一种同时检测多个人体部位的运动的方式。举例来说，虚拟现实(virtual reality；VR)产品可提供手持型控制器，且每个手持型控制器包含惯性测量单元(inertial measurement unit；IMU)以跟踪用户的手的运动。有时，仅一种运动跟踪方式可受其硬件或跟踪机制的限制，且导致跟踪结果异常或不准确。In conventional technology, some electronic devices may allow user's body parts (such as hands, legs, head, etc.) to control the operation of these electronic devices, and the movements of these body parts may be tracked. However, these electronic devices only provide a way to detect the motion of multiple body parts simultaneously. For example, virtual reality (VR) products may provide handheld controllers, and each handheld controller includes an inertial measurement unit (IMU) to track the movement of a user's hand. Sometimes only one motion tracking method can be limited by its hardware or tracking mechanism and result in abnormal or inaccurate tracking results.

发明内容SUMMARY OF THE INVENTION

有时候，只使用单一传感器的跟踪结果可能不准确。因此，本发明是针对一种相关于运动跟踪的系统和方法。Occasionally, tracking results using only a single sensor may be inaccurate. Accordingly, the present invention is directed to a system and method related to motion tracking.

在示例性实施例中的一个中，相关于运动跟踪的方法适用于系统，系统包含可穿戴在用户的人体部位上的第一运动感测设备、第二运动感测设备以及第三运动感测设备。相关于运动跟踪的方法包含(但不限于)以下步骤。基于安置在第一运动感测设备、第二运动感测设备以及第三运动感测设备上的运动传感器获得第一感测数据。基于在第一运动感测设备、第二运动感测设备以及第三运动感测设备之间发射的无线信号获得第二感测数据。根据包含第一感测数据和第二感测数据的决定因素确定用户的运动信息。In one of the exemplary embodiments, a method related to motion tracking is applied to a system comprising a first motion sensing device, a second motion sensing device, and a third motion sensing wearable on a user's body part equipment. Methods related to motion tracking include, but are not limited to, the following steps. The first sensing data is obtained based on motion sensors disposed on the first motion sensing device, the second motion sensing device, and the third motion sensing device. The second sensing data is obtained based on wireless signals transmitted between the first motion sensing device, the second motion sensing device, and the third motion sensing device. The motion information of the user is determined according to a decision factor including the first sensing data and the second sensing data.

在示例性实施例中的一个中，相关于运动跟踪的系统包含(但不限于)三个运动感测设备和处理器。运动感测设备可穿戴在用户的人体部位上。每个运动感测设备包含无线收发器和运动传感器。无线收发器用于发射或接收无线信号。运动传感器用于感测用户的一个人体部位的运动。处理器被配置以基于运动感测设备的运动传感器获得第一感测数据，且基于在所述三个运动感测设备之间发射的无线信号据获得第二感测数据，且由包含第一感测数据和第二感测数据的决定因素确定用户的运动信息。In one of the exemplary embodiments, a motion tracking related system includes, but is not limited to, three motion sensing devices and a processor. A motion sensing device can be worn on a user's body part. Each motion sensing device contains a wireless transceiver and a motion sensor. Wireless transceivers are used to transmit or receive wireless signals. The motion sensor is used to sense the motion of one body part of the user. The processor is configured to obtain first sensed data based on a motion sensor of the motion-sensing device and obtain second sensed data based on wireless signals transmitted between the three motion-sensing devices, and to obtain second sensed data based on a motion sensor comprising the first The sensing data and the determinants of the second sensing data determine motion information of the user.

基于上述，根据本发明实施例的相关于运动跟踪的系统及方法，可基于数种感测方式的感测数据来跟踪用户的单一操作部位的动作。藉此，可提供准确及可靠的跟踪机制。Based on the above, according to the system and method related to motion tracking according to the embodiments of the present invention, the motion of a single operating part of the user can be tracked based on the sensing data of several sensing methods. Thereby, an accurate and reliable tracking mechanism can be provided.

然而，应理解，此概述可能不含有本发明的所有方面和实施例，不意图以任何方式具有限制性或局限性，且如本文中所公开的发明由且将由本领域普通技术人员理解为涵盖对其所作的明显改进和修改。It should be understood, however, that this summary may not contain all aspects and embodiments of the invention, is not intended to be limiting or limiting in any way, and that the invention as disclosed herein is and will be understood by those of ordinary skill in the art to cover Significant improvements and modifications made to it.

附图说明Description of drawings

包含附图以提供对本发明的进一步理解，且附图并入本说明书中并构成本说明书的一部位。附图说明本发明的实施例，且与描述一起用于解释本发明的原理。The accompanying drawings are included to provide a further understanding of the invention, and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments of the invention, and together with the description serve to explain the principles of the invention.

图1是示出根据本发明的示例性实施例中的一个的运动跟踪系统的框图；1 is a block diagram illustrating a motion tracking system according to one of the exemplary embodiments of the present invention;

图2是示出根据本发明的示例性实施例中的一个的运动跟踪系统的示意图；2 is a schematic diagram illustrating a motion tracking system according to one of the exemplary embodiments of the present invention;

图3是示出根据本发明的示例性实施例中的一个的运动跟踪方法的流程图；3 is a flowchart illustrating a motion tracking method according to one of the exemplary embodiments of the present invention;

图4是示出根据本发明的示例性实施例中的一个的运动跟踪方法的示意图；4 is a schematic diagram illustrating a motion tracking method according to one of the exemplary embodiments of the present invention;

图5是示出根据本发明的示例性实施例中的一个的运动跟踪方法的示意图。FIG. 5 is a schematic diagram illustrating a motion tracking method according to one of the exemplary embodiments of the present invention.

附图标号说明Explanation of reference numerals

10：运动跟踪系统；10: Motion tracking system;

20：运动跟踪系统；20: motion tracking system;

100：运动感测设备；100: motion sensing device;

110：无线收发器；110: wireless transceiver;

130：运动传感器；130: motion sensor;

200：计算设备；200: computing equipment;

240：存储器；240: memory;

250：处理器；250: processor;

300：头戴式显示器；300: head-mounted display;

310：无线收发器；310: wireless transceiver;

360：图像传感器；360: image sensor;

B1、B2、B3、B4、B5：人体部位；B1, B2, B3, B4, B5: body parts;

S310、S330、S350：步骤；S310, S330, S350: steps;

FOV：视场。FOV: Field of View.

具体实施方式Detailed ways

现将详细参考本发明的优选实施例，其实例在附图中说明。只要可能，相同附图标号在附图和描述中用以指代相同或相似部件。Reference will now be made in detail to the preferred embodiments of the present invention, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numbers are used in the drawings and the description to refer to the same or like parts.

图1是示出根据本发明的示例性实施例中的一个的运动跟踪系统10的框图。参看图1，运动跟踪系统10包含(但不限于)三个或更多个多个运动感测设备100以及计算设备200。运动跟踪系统10可适用于VR、AR、MR、XR或其它现实相关的技术。FIG. 1 is a block diagram illustrating a motion tracking system 10 according to one of the exemplary embodiments of the present invention. Referring to FIG. 1 , a motion tracking system 10 includes, but is not limited to, three or more motion sensing devices 100 and a computing device 200 . The motion tracking system 10 may be applicable to VR, AR, MR, XR or other reality related technologies.

每个运动感测设备100包含(但不限于)无线收发器110和运动传感器130。运动感测设备100可以是手持型控制器或可穿戴设备，诸如可穿戴控制器、智能手表、脚踝传感器、腰带、头戴式显示器(head-mounted display；HMD)或类似物。在一个实施例中，每个运动感测设备100可穿戴在用户的一个人体部位上。人体部位可以是手、头、脚踝、腿、腰或其它部位。Each motion sensing device 100 includes, but is not limited to, a wireless transceiver 110 and a motion sensor 130 . The motion sensing device 100 may be a handheld controller or a wearable device such as a wearable controller, smart watch, ankle sensor, belt, head-mounted display (HMD), or the like. In one embodiment, each motion sensing device 100 is wearable on one body part of the user. The body part may be the hand, head, ankle, leg, waist or other parts.

无线收发器110可以是与蓝牙、Wi-Fi、IR、RFID或其它无线通信技术兼容的通信收发器。在一个实施例中，无线收发器110用于与其它运动感测设备100的无线收发器110一起发射和/或接收无线信号，且将基于在运动感测设备100之间发射的无线信号生成序列的第一感测数据。将在随后介绍用于生成序列的第一感测数据的详细流程。Wireless transceiver 110 may be a communication transceiver compatible with Bluetooth, Wi-Fi, IR, RFID, or other wireless communication technologies. In one embodiment, the wireless transceiver 110 is used to transmit and/or receive wireless signals with the wireless transceivers 110 of other motion sensing devices 100 and will generate sequences based on the wireless signals transmitted between the motion sensing devices 100 of the first sensing data. A detailed procedure for generating the first sensing data of the sequence will be described later.

运动传感器130可以是加速计、陀螺仪、磁力计、惯性测量单元(IMU)或前述传感器的任何组合。在实施例中，运动传感器130用于在一段时间内感测穿戴运动感测设备100的用户的对应人体部位的运动，以在时间段内的多个时间点从运动传感器130的感测结果(诸如加速度、旋转、磁力等)生成序列的第一感测数据。对于一个实例，第一感测数据包含3-自由度(3-degree of freedom；3-DoF)数据，且3-DoF数据与三维(three-dimensional；3D)空间中的人体部位的定向信息相关，诸如偏航、滚转以及俯仰的加速度。Motion sensor 130 may be an accelerometer, gyroscope, magnetometer, inertial measurement unit (IMU), or any combination of the foregoing. In an embodiment, the motion sensor 130 is used to sense the motion of the corresponding human body part of the user wearing the motion sensing device 100 within a period of time, so as to obtain information from the sensing results of the motion sensor 130 ( such as acceleration, rotation, magnetic force, etc.) to generate a sequence of first sensed data. For one example, the first sensed data includes 3-degree of freedom (3-DoF) data, and the 3-DoF data is related to orientation information of body parts in three-dimensional (3D) space , accelerations such as yaw, roll, and pitch.

计算设备200包含(但不限于)存储器240和处理器250。计算设备200可以是计算机、服务器、智能手机、平板电脑或运动感测设备100中的一个。Computing device 200 includes, but is not limited to, memory 240 and processor 250 . Computing device 200 may be one of a computer, server, smartphone, tablet, or motion sensing device 100 .

存储器240可以是任何类型的固定或可移动随机存取存储器(Random-AccessMemory；RAM)、只读存储器(read-only memory；ROM)、闪存存储器或类似装置或以上装置的组合。存储器240可用于存储程序代码、装置配置、缓冲器数据或永久性数据(诸如感测数据、运动信息、距离关系等)，且将在随后介绍这些数据。The memory 240 may be any type of fixed or removable random-access memory (RAM), read-only memory (ROM), flash memory, or the like or a combination of the above. The memory 240 may be used to store program code, device configuration, buffer data, or persistent data (such as sensory data, motion information, distance relationships, etc.), which will be described later.

处理器250连接到存储器240，且处理器250被配置以加载存储在存储器240中的程序代码，从而执行本发明的示例性实施例的程序。在一个实施例中，处理器150的功能通过使用诸如中央处理单元(central processing unit；CPU)、微处理器、微控制器、数字信号处理(digital signal processing；DSP)芯片、现场可编程门阵列(field programmablegate array；FPGA)等可编程单元来实施。在一些实施例中，处理器250的功能也可通过独立电子装置或集成电路(integrated circuit；IC)来实施，且处理器250的操作也可由软件来实施。The processor 250 is connected to the memory 240, and the processor 250 is configured to load program codes stored in the memory 240, thereby executing the programs of the exemplary embodiments of the present invention. In one embodiment, the functionality of the processor 150 is achieved through the use of devices such as central processing units (CPUs), microprocessors, microcontrollers, digital signal processing (DSP) chips, field programmable gate arrays (field programmablegate array; FPGA) and other programmable units to implement. In some embodiments, the functions of the processor 250 may also be implemented by a stand-alone electronic device or integrated circuit (IC), and the operations of the processor 250 may also be implemented by software.

应注意，处理器250可或可不与一个、部位或全部运动感测设备100安置在同一设备上。然而，分别装配有运动传感器130和处理器250的设备可还包含具有诸如蓝牙、Wi-Fi、IR或物理传输线路的兼容通信技术的通信收发器，以彼此发射/接收数据。It should be noted that the processor 250 may or may not be co-located on the same device as one, part or all of the motion sensing device 100 . However, the devices equipped with the motion sensor 130 and the processor 250, respectively, may also include a communication transceiver with compatible communication technologies such as Bluetooth, Wi-Fi, IR or physical transmission lines to transmit/receive data to each other.

在一个实施例中，运动跟踪系统10可还包含头戴式显示器(HMD)300。HMD 300可穿戴在用户的头部上。HMD 300包含(但不限于)无线收发器310和图像传感器360。In one embodiment, the motion tracking system 10 may also include a head mounted display (HMD) 300 . The HMD 300 is wearable on the user's head. HMD 300 includes, but is not limited to, wireless transceiver 310 and image sensor 360 .

无线收发器310的描述可指无线收发器110的描述，且将省略。这意味着HMD 300可通过无线收发器310与运动感测设备100通信。The description of the wireless transceiver 310 may refer to the description of the wireless transceiver 110 and will be omitted. This means that the HMD 300 can communicate with the motion sensing device 100 through the wireless transceiver 310 .

图像传感器360可以是相机，如单色相机或彩色相机、深度相机、录像机或能够捕获图像的其它传感器。Image sensor 360 may be a camera, such as a monochrome or color camera, a depth camera, a video recorder, or other sensor capable of capturing images.

图2是示出根据本发明的示例性实施例中的一个的运动跟踪系统20的示意图。参看图2，运动跟踪系统20包含HMD 300和四个运动感测设备100(所述四个设备是穿戴在人体部位B1和人体部位B2(即，两个脚踝)上的两个脚踝传感器和穿戴在人体部位B3和人体部位B4(即，两只手)上的两个手持型控制器)。在一些实施例中，HMD 300可还包含另一运动传感器130(未示出)，以获得人体部位B5(即，头部)的定向信息。处理器250嵌入在HMD 300中。FIG. 2 is a schematic diagram illustrating a motion tracking system 20 according to one of the exemplary embodiments of the present invention. 2, the motion tracking system 20 includes an HMD 300 and four motion sensing devices 100 (the four devices are two ankle sensors and wear Two handheld controllers on body part B3 and body part B4 (ie, both hands). In some embodiments, HMD 300 may also include another motion sensor 130 (not shown) to obtain orientation information for body part B5 (ie, the head). The processor 250 is embedded in the HMD 300 .

应注意，运动跟踪系统20仅是实例以示出运动感测设备100、HMD 300以及处理器250的安置方式。然而，仍存在行为理解系统10的许多其它实施方式，且本发明不限于此。It should be noted that the motion tracking system 20 is merely an example to illustrate the arrangement of the motion sensing device 100 , the HMD 300 and the processor 250 . However, many other implementations of the behavior understanding system 10 exist, and the invention is not limited thereto.

为了更好地理解在本发明的一或多个实施例中提供的操作流程，下文将例示若干实施例以详细解释运动跟踪系统10或运动跟踪系统20的操作流程。运动跟踪系统10或运动跟踪系统20中的装置和模块在以下实施例应用以解释本文所提供的控制方法。可根据实际实施情况调整控制方法的每个步骤，且不应限于本文描述的内容。In order to better understand the operational flow provided in one or more embodiments of the present invention, several embodiments will be illustrated below to explain the operational flow of the motion tracking system 10 or the motion tracking system 20 in detail. The devices and modules in the motion tracking system 10 or the motion tracking system 20 are used in the following embodiments to explain the control methods provided herein. Each step of the control method can be adjusted according to the actual implementation, and should not be limited to the content described herein.

图3是示出根据本发明的示例性实施例中的一个的运动跟踪方法的流程图。参看图3，处理器250可基于安置在三个运动感测设备100上的运动传感器130获得第一感测数据(步骤S310)。确切地说，关于不同类型的运动传感器130，可获得2D/3D空间中对应人体部位的运动的加速度、旋转、磁力、定向和/或3-DoF/6-DoF，且运动传感器130的一或多个感测结果将成为人体部位的序列第一感测数据。FIG. 3 is a flowchart illustrating a motion tracking method according to one of the exemplary embodiments of the present invention. Referring to FIG. 3, the processor 250 may obtain first sensing data based on the motion sensors 130 disposed on the three motion sensing devices 100 (step S310). Specifically, with respect to different types of motion sensors 130 , the acceleration, rotation, magnetic force, orientation and/or 3-DoF/6-DoF of the motion of the corresponding human body part in 2D/3D space can be obtained, and one of the motion sensors 130 or The plurality of sensing results will become the sequence first sensing data of the human body part.

另一方面，处理器250可基于在三个运动感测设备100之间发射的无线信号获得第二感测数据(步骤S330)。在一个实施例中，处理器250可在多个时间点从三个或更多个运动感测设备100获得无线信号的信号强度，且每个信号强度将与其对应的发射器和接收器一起记录在存储器240中。信号强度可以是接收的信号强度指示(received signal strengthindication；RSSI)、接收信道功率参数(received channel power indicator；RCPI)、参考信号接收功率(reference signal received power；RSRP)或类似物。在一个实施例中，运动感测设备100可监测所有可检测的无线信号的信号强度，且每个无线信号包含发射器和/或接收器的特定标识符。运动感测设备100可进一步将具有对应标识符的信号强度反馈给计算设备200。在另一实施例中，计算设备200可监测所有可检测的无线信号的信号强度，且处理器250将信号强度与发射器的对应标识符记录在存储器240中。将记录信号强度一段时间以生成序列的第二感测数据。这意味着第二感测数据包含按时间排列的序列的信号强度。On the other hand, the processor 250 may obtain second sensing data based on wireless signals transmitted between the three motion sensing devices 100 (step S330). In one embodiment, the processor 250 may obtain signal strengths of wireless signals from three or more motion sensing devices 100 at multiple points in time, and each signal strength will be recorded with its corresponding transmitter and receiver in memory 240. The signal strength may be a received signal strength indication (RSSI), a received channel power indicator (RCPI), a reference signal received power (RSRP), or the like. In one embodiment, motion sensing device 100 may monitor the signal strength of all detectable wireless signals, and each wireless signal includes a specific identifier for the transmitter and/or receiver. The motion sensing device 100 may further feed back the signal strength with the corresponding identifier to the computing device 200 . In another embodiment, computing device 200 may monitor the signal strength of all detectable wireless signals, and processor 250 records the signal strength and the corresponding identifier of the transmitter in memory 240 . The signal strength will be recorded for a period of time to generate a sequence of second sensed data. This means that the second sensed data contains the signal intensities of a time-arranged sequence.

在一些实施例中，处理器250可进一步基于从图像传感器360所捕获的图像获得第三感测数据。第三感测数据可以是序列的图像和/或图像中的像素的感测结果(诸如亮度、色彩、深度等)。In some embodiments, the processor 250 may further obtain third sensing data based on images captured from the image sensor 360 . The third sensing data may be a sequence of images and/or sensing results (such as brightness, color, depth, etc.) of pixels in the images.

接着，处理器250可由包含第一感测数据和第二感测数据的决定因素确定用户的运动信息(步骤S350)。在一个实施例中，运动信息可包含位置信息和定向信息。首先关于位置信息，在一个实施例中，处理器250可根据第一感测数据确定用户的位置信息。在此实施例中，决定因素包含第一感测数据。对应人体部位的位移可通过对人体部位在三条轴线上的检测到的加速度(即，第二感测数据)二重积分来估计，以基于位移进一步确定位置信息。举例来说，位置信息可以是两条或三条轴线上的坐标，相对于参考的位置等。Next, the processor 250 may determine the motion information of the user from the decision factor including the first sensing data and the second sensing data (step S350). In one embodiment, the motion information may include location information and orientation information. First, regarding the location information, in one embodiment, the processor 250 may determine the location information of the user according to the first sensing data. In this embodiment, the determining factor includes the first sensing data. The displacement of the corresponding body part may be estimated by double integrating the detected acceleration of the body part in the three axes (ie, the second sensed data) to further determine position information based on the displacement. For example, the location information can be coordinates on two or three axes, a location relative to a reference, and the like.

在另一实施例中，处理器250可根据基于三个运动感测设备100之间的无线信号的第二感测数据获得位置信息。在此实施例中，决定因素包含第二感测数据，应注意，无线信号的信号强度与两个运动感测设备100之间的相对距离相关。另外，基于三边测量，三个点之间的三个距离可用于确定三个点的相对位置信息。假定三个运动感测设备100作为前述三个点，那么处理器250可将每两个运动感测设备100之间的相对距离确定为运动感测设备100之间的距离关系。接着，处理器250可基于距离关系和三边测量生成跟踪设备的位置信息。In another embodiment, the processor 250 may obtain location information according to second sensing data based on wireless signals between the three motion sensing devices 100 . In this embodiment, the determining factor includes the second sensing data, and it should be noted that the signal strength of the wireless signal is related to the relative distance between the two motion sensing devices 100 . In addition, based on trilateration, the three distances between the three points can be used to determine the relative position information of the three points. Assuming three motion sensing apparatuses 100 as the aforementioned three points, the processor 250 may determine a relative distance between every two motion sensing apparatuses 100 as a distance relationship between the motion sensing apparatuses 100 . Next, the processor 250 may generate position information for the tracking device based on the distance relationship and trilateration.

以运动跟踪系统20为实例，处理器250可获得从用于人体部位B3的运动感测设备100到用于人体部位B5的HMD 300(在此实施例中是运动感测设备100中的一个)的无线信号、从用于人体部位B4的运动感测设备100到用于人体部位B5的HMD 300的无线信号以及从用于人体部位B3的运动感测设备100到用于人体部位B4的运动感测设备100的无线信号的信号强度。处理器250可根据信号强度确定其距离关系，且接着基于距离关系生成人体部位B3的位置信息。位置信息可以是坐标或相对位置。Taking the motion tracking system 20 as an example, the processor 250 can obtain from the motion sensing device 100 for the body part B3 to the HMD 300 (in this embodiment one of the motion sensing devices 100 ) for the body part B5 The wireless signal from the motion sensing device 100 for the human body part B4 to the HMD 300 for the human body part B5 and the wireless signal from the motion sensing device 100 for the human body part B3 to the motion sensing device for the human body part B4 The signal strength of the wireless signal of the device 100 is measured. The processor 250 may determine the distance relationship thereof according to the signal strength, and then generate the position information of the human body part B3 based on the distance relationship. The location information can be coordinates or relative locations.

应注意，实施例不限制选定三个运动感测设备100。举例来说，从用于人体部位B2的运动感测设备100到用于人体部位B3的运动感测设备100的无线信号、从用于人体部位B3的运动感测设备100到用于人体部位B1的运动感测设备100的无线信号以及从用于人体部位B2的运动感测设备100到用于人体部位B1的运动感测设备100的无线信号的信号强度可用于估计人体部位B1的位置信息。运动感测设备100的组合可根据需要变化。It should be noted that embodiments do not limit the selection of three motion sensing devices 100 . For example, a wireless signal from the motion sensing device 100 for the body part B2 to the motion sensing device 100 for the body part B3, from the motion sensing device 100 for the body part B3 to the motion sensing device 100 for the body part B1 The wireless signal of the motion sensing device 100 and the signal strength of the wireless signal from the motion sensing device 100 for the human body part B2 to the motion sensing device 100 for the human body part B1 can be used to estimate the position information of the human body part B1. The combination of motion sensing devices 100 may vary as desired.

在另一实施例中，处理器250可根据第三感测数据确定用户的位置信息。在此实施例中，决定因素包含第三感测数据。图像中的人体部位的位置和位移可用于确定真实环境中的位置信息。以图2为实例，对应于图像中的人体部位B4的感测强度和像素位置可用于估计人体部位B4的深度信息(即，相对于HMD 300的距离)和估计人体部位B4在平行于图像传感器360的平面上的2D位置。In another embodiment, the processor 250 may determine the location information of the user according to the third sensing data. In this embodiment, the determining factor includes the third sensing data. The position and displacement of human body parts in the image can be used to determine position information in the real environment. Taking FIG. 2 as an example, the sensed intensities and pixel positions corresponding to the human body part B4 in the image can be used to estimate the depth information of the human body part B4 (ie, the distance relative to the HMD 300) and to estimate that the human body part B4 is parallel to the image sensor. 2D position on the plane of the 360.

应注意，基于仅一种感测方式(例如，是基于无线收发器110、运动传感器130以及图像传感器360中的一个)的位置信息的准确度可以是不同的。因此，两种或更多种感测方式可用于确定对应人体部位的位置信息。It should be noted that the accuracy of location information based on only one sensing means (eg, based on one of wireless transceiver 110, motion sensor 130, and image sensor 360) may be different. Therefore, two or more sensing methods can be used to determine the position information of the corresponding body part.

在一个实施例中，处理器250可根据第一感测数据获得第一位置信息，根据第二感测数据获得第二位置信息，以及根据第一位置信息和第二位置信息获得调整后的位置信息。在此实施例中，决定因素包含第一感测数据和第二感测数据。处理器250可根据第一位置信息和第二位置信息的组合确定位置信息。在一些实施例中，组合是加权组合。根据加权的第一位置信息和加权的第二位置信息确定调整后的位置信息。In one embodiment, the processor 250 may obtain the first position information according to the first sensing data, obtain the second position information according to the second sensing data, and obtain the adjusted position according to the first position information and the second position information information. In this embodiment, the determining factor includes the first sensing data and the second sensing data. The processor 250 may determine the location information according to a combination of the first location information and the second location information. In some embodiments, the combination is a weighted combination. The adjusted position information is determined according to the weighted first position information and the weighted second position information.

在一个实施例中，第一位置信息和第二位置信息的加权组合的权重可以是固定的。在另一实施例中，第一位置信息和第二位置信息的加权组合的权重可以是变化的。第一位置信息的权重可以是从0到100％的值，且第二位置信息的权重可以是从0到100％的值。然而，第一位置信息和第二位置信息的权重都不能同时是0。In one embodiment, the weight of the weighted combination of the first location information and the second location information may be fixed. In another embodiment, the weight of the weighted combination of the first location information and the second location information may vary. The weight of the first location information may be a value from 0 to 100%, and the weight of the second location information may be a value from 0 to 100%. However, the weights of both the first position information and the second position information cannot be 0 at the same time.

应注意，在一些实施例中，基于由图像传感器360的图像生成的第三感测数据而确定的位置信息可比基于无线收发器110和/或运动传感器130而确定的位置信息更准确。因此，在一个实施例中，决定因素可包含第二感测数据和第三感测数据。处理器250可根据基于第一感测数据、第二感测数据以及第三感测数据获得的位置信息的组合确定位置信息。It should be noted that, in some embodiments, the location information determined based on the third sensed data generated from the images of the image sensor 360 may be more accurate than the location information determined based on the wireless transceiver 110 and/or the motion sensor 130 . Therefore, in one embodiment, the determining factor may include the second sensing data and the third sensing data. The processor 250 may determine the location information according to a combination of location information obtained based on the first sensing data, the second sensing data, and the third sensing data.

在一个实施例中，处理器250可在第一持续时间内根据第二感测数据获得位置信息的第一部位，在第二持续时间内根据第三感测数据获得位置信息的第二部位，以及将位置信息的第一部位和第二部位组合为组合位置信息。检测人体部位的第三感测数据可用于在第一持续时间和第二持续时间内基于第二感测数据来校正位置信息。处理器250可在不同持续时间内基于位置信息的第一部位和第二部位来确定组合位置信息。举例来说，在第一持续时间基于第二感测数据确定位置(1，1)，在第二持续时间基于第三感测数据确定另一位置(2，1)，且组合位置信息可以是从位置(1，1)到位置(2，1)的位移。In one embodiment, the processor 250 may obtain the first part of the position information according to the second sensing data during the first duration, and obtain the second part of the position information according to the third sensing data during the second duration, and combining the first part and the second part of the position information into combined position information. The third sensed data detecting the body part may be used to correct the position information based on the second sensed data for the first duration and the second duration. The processor 250 may determine the combined position information based on the first and second portions of the position information at different time durations. For example, the location (1, 1) is determined based on the second sensing data for a first duration, another location (2, 1) is determined based on the third sensing data for a second duration, and the combined location information may be Displacement from position (1, 1) to position (2, 1).

在一些实施例中，处理器250可根据第二位置信息和第三位置信息的加权组合确定位置信息。基于实际情况，第二位置信息和第三位置信息的权重可以是变化的或固定的。举例来说，第三位置信息的权重可大于第二位置信息的权重。在另一实施例中，如果人体部位存在于第三感测数据中，那么位置信息是加权组合，且如果人体部位不存在于第三感测数据中，那么位置是第二位置信息。In some embodiments, the processor 250 may determine the location information according to a weighted combination of the second location information and the third location information. Based on the actual situation, the weights of the second position information and the third position information may be changed or fixed. For example, the weight of the third location information may be greater than the weight of the second location information. In another embodiment, the position information is the weighted combination if the body part is present in the third sensing data, and the position is the second position information if the body part is not present in the third sensing data.

在一个实施例中，图像传感器360可设计有特定视场(specific field of view；FOV)。如果一个人体部位定位于图像传感器360的视场的外部，那么处理器250可能不能够仅使用第三感测数据来确定此人体部位的运动信息，且应考虑第一感测数据或第二感测数据。In one embodiment, the image sensor 360 may be designed with a specific field of view (FOV). If a body part is positioned outside the field of view of the image sensor 360, the processor 250 may not be able to determine motion information for this body part using only the third sensed data, and should consider either the first sensed data or the second sensed data measurement data.

在一个实施例中，处理器250可确定用户的一个人体部位是否存在于序列的第三感测数据中，且根据人体部位的存在的确定的结果来确定是否使用三个运动感测设备100之间的距离关系，以基于三边测量确定位置信息。处理器250可使用机器学习技术(诸如深度学习、人工神经网络(artificial neural network；ANN)或支持向量机(support vectormachine；SVM)等)，以识别第三感测数据中的目标人体部位。In one embodiment, the processor 250 may determine whether one body part of the user is present in the third sensing data of the sequence, and determine whether to use one of the three motion sensing devices 100 according to the result of the determination of the presence of the body part distance relationship between to determine location information based on trilateration. The processor 250 may use machine learning techniques (such as deep learning, artificial neural network (ANN), or support vector machine (SVM), etc.) to identify the target body part in the third sensing data.

图4是示出根据本发明的示例性实施例中的一个的运动跟踪方法的示意图。参看图4，假定人体部位B4的运动感测设备100是目标设备。在此图中，人体部位B4存在于HMD300的视场FOV中(即，人体部位B4存在于第三感测数据中)。FIG. 4 is a schematic diagram illustrating a motion tracking method according to one of the exemplary embodiments of the present invention. Referring to FIG. 4, it is assumed that the motion sensing device 100 of the human body part B4 is the target device. In this figure, the human body part B4 exists in the field of view FOV of the HMD 300 (ie, the human body part B4 exists in the third sensing data).

图5是示出根据本发明的示例性实施例中的一个的运动跟踪方法的示意图。参看图5，假定人体部位B3的运动感测设备100是目标设备。在此图中，人体部位B5不存在于HMD300的视场FOV中(即，人体部位B3不存在于第三感测数据中)。FIG. 5 is a schematic diagram illustrating a motion tracking method according to one of the exemplary embodiments of the present invention. Referring to FIG. 5, it is assumed that the motion sensing device 100 of the human body part B3 is the target device. In this figure, the human body part B5 does not exist in the field of view FOV of the HMD 300 (ie, the human body part B3 does not exist in the third sensing data).

应注意，图4和图5中示出的视场的大小和形状仅是实例，且可基于实际需要修改。It should be noted that the size and shape of the fields of view shown in FIGS. 4 and 5 are only examples and may be modified based on actual needs.

因此，图像传感器360的视场用于确定人体部位是否存在于第三感测数据中。在一个实施例中，假定人体部位在第一持续时间位于视场外(即，不存在于第三感测数据中)，且人体部位在第二持续时间位于图像传感器360的视场内(即，存在于第三感测数据中)。在一些实施例中，假定人体部位在第一持续时间和第二持续时间定位于图像传感器360的视场的内部。Therefore, the field of view of the image sensor 360 is used to determine whether a body part is present in the third sensed data. In one embodiment, it is assumed that the body part is outside the field of view (ie, not present in the third sensed data) for the first duration, and the body part is within the field of view of the image sensor 360 for the second duration (ie, not present in the third sensed data) , present in the third sensing data). In some embodiments, it is assumed that the body part is positioned inside the field of view of image sensor 360 for the first duration and the second duration.

在另一实施例中，处理器250可根据第一感测数据获得第一位置信息，根据第二感测数据获得第二位置信息，根据第三感测数据获得第三位置信息，且根据第一位置信息、第二位置信息以及第三位置信息获得调整后的位置信息。在此实施例中，决定因素包含第一感测数据、第二感测数据以及第三感测数据。处理器250可根据第一运动信息、第二运动信息以及第三位置信息的组合确定调整后的位置信息。In another embodiment, the processor 250 may obtain the first position information according to the first sensing data, obtain the second position information according to the second sensing data, obtain the third position information according to the third sensing data, and obtain the third position information according to the third sensing data. The first position information, the second position information and the third position information obtain the adjusted position information. In this embodiment, the determining factors include the first sensing data, the second sensing data and the third sensing data. The processor 250 may determine the adjusted position information according to a combination of the first motion information, the second motion information, and the third position information.

在一个实施例中，组合是加权组合。处理器250可根据第三位置信息确定第一位置信息的第一权重和第二位置信息的第二权重。在一个实施例中，第一权重和第二权重反复变化。在人体部位存在于第三感测数据中的持续时间内，第三位置信息将被视为校正位置信息，且将根据第三位置信息调整具有第一权重和第二权重的第一位置信息和第二位置信息的加权组合。应注意，处理器250可通过将第一权重与第一位置信息相乘获得第一参数，通过将第二权重与第二位置信息相乘获得第二参数，且通过将第一参数与第二参数相加获得调整后的位置信息，从而以获得加权组合。In one embodiment, the combination is a weighted combination. The processor 250 may determine the first weight of the first location information and the second weight of the second location information according to the third location information. In one embodiment, the first weight and the second weight vary iteratively. During the duration that the body part is present in the third sensing data, the third position information will be regarded as corrected position information, and the first position information and the second weight with the first weight and the second weight will be adjusted according to the third position information A weighted combination of the second location information. It should be noted that the processor 250 may obtain the first parameter by multiplying the first weight by the first position information, obtain the second parameter by multiplying the second weight by the second position information, and obtain the first parameter by multiplying the first parameter by the second position information The parameters are added to obtain the adjusted position information to obtain a weighted combination.

在一个实施例中，可基于第三位置信息等于先前时间点的第一位置信息和第二位置信息的加权组合的等式而确定后续时间点的第一权重和第二权重。举例来说，在第三时间点，在3维坐标系统中，第一权重是0.5且第二权重是0.5，第一位置信息是(6，6，6)且第二位置信息是(10，10，10)，且调整后的位置信息将是(8，8，8)。如果第三位置信息是(7，7，7)，那么在第四时间点的第一权重和第二权重将分别确定为0.75和0.25。接着，在第四时间点，如果在3维坐标系统中第一位置信息是(7，6，6)且第二位置信息是(12，10，10)，那么调整后的位置信息将是(8.25，7，7)。In one embodiment, the first weight and the second weight for the subsequent point in time may be determined based on an equation in which the third position information is equal to a weighted combination of the first position information and the second position information at the previous point in time. For example, at the third time point, in the 3-dimensional coordinate system, the first weight is 0.5 and the second weight is 0.5, the first position information is (6, 6, 6) and the second position information is (10, 10, 10), and the adjusted position information will be (8, 8, 8). If the third position information is (7, 7, 7), the first weight and the second weight at the fourth time point will be determined to be 0.75 and 0.25, respectively. Then, at the fourth time point, if the first position information is (7, 6, 6) and the second position information is (12, 10, 10) in the 3-dimensional coordinate system, then the adjusted position information will be ( 8.25, 7, 7).

在另一实施例中，可基于第三位置信息等于当前时间点的第一位置信息和第二位置信息的加权组合的等式而确定当前时间点的第一权重和第二权重。举例来说，在第二时间点，在3维坐标系统中，第一位置信息是(6，6，6)且第二位置信息是(10，10，10)。如果第三位置信息是(7，7，7)，那么在第二时间点的第一权重和第二权重将分别确定为0.75和0.25。接着，在第二时间点的调整后的位置信息将确定为(7，7，7)。In another embodiment, the first weight and the second weight of the current time point may be determined based on an equation in which the third position information is equal to a weighted combination of the first position information and the second position information of the current time point. For example, at the second time point, in the 3-dimensional coordinate system, the first position information is (6, 6, 6) and the second position information is (10, 10, 10). If the third position information is (7, 7, 7), then the first weight and the second weight at the second time point will be determined to be 0.75 and 0.25, respectively. Next, the adjusted position information at the second time point will be determined as (7, 7, 7).

在一些实施例中，如果用户的人体部位不存在于第三感测数据中，那么第一权重和第二权重是固定的。如果人体部位位于图像传感器360的视场外，那么当用户的人体部位仍存在于第三感测数据中时，第三权重和第二权重将与先前时间点的先前第一权重和第二权重相同。举例来说，在第一时间点，人体部位位于图像传感器360的视场内，且第一权重是0.5且第二权重是0.5。接着，在第二时间点，人体部位位于图像传感器360的视场外。与在第一时间点的第一权重和第二权重相同，在第二时间点，第一权重将是0.5且第二权重将是0.5。直到用户的人体部位存在于第三感测数据中，第一权重和第二权重将根据第三感测数据而变化。In some embodiments, if the user's body part is not present in the third sensed data, the first weight and the second weight are fixed. If the body part is located outside the field of view of the image sensor 360, when the user's body part is still present in the third sensed data, the third and second weights will be the same as the previous first and second weights at the previous point in time same. For example, at the first point in time, the body part is within the field of view of the image sensor 360, and the first weight is 0.5 and the second weight is 0.5. Next, at a second point in time, the body part is located outside the field of view of the image sensor 360 . As with the first and second weights at the first point in time, at the second point in time, the first weight will be 0.5 and the second weight will be 0.5. Until the user's body part exists in the third sensing data, the first weight and the second weight will vary according to the third sensing data.

在另一实施例中，处理器250可根据第一位置信息、第二位置信息以及第三位置信息的加权组合确定调整后的位置信息。根据加权的第一位置信息、加权的第二位置信息以及加权的第三位置信息的总和确定调整后的位置信息。基于实际情况，三条位置信息的权重可以是变化的或固定的。In another embodiment, the processor 250 may determine the adjusted location information according to a weighted combination of the first location information, the second location information, and the third location information. The adjusted position information is determined according to the sum of the weighted first position information, the weighted second position information, and the weighted third position information. Based on the actual situation, the weights of the three pieces of location information can be changed or fixed.

另一方面，关于定向信息，在一个实施例中，处理器250可将序列的第一感测数据直接用作定向信息。举例来说，定向信息可以是加速度、三轴中的角速度、定向、3-DoF信息和/或6-DoF信息。On the other hand, with regard to orientation information, in one embodiment, the processor 250 may directly use the first sensed data of the sequence as the orientation information. For example, the orientation information may be acceleration, angular velocity in three axes, orientation, 3-DoF information, and/or 6-DoF information.

在另一实施例中，处理器250可根据第三感测数据确定定向信息。以图4为实例，图像中的人体部位B4在不同时间点的两个姿势可用于估计定向信息。In another embodiment, the processor 250 may determine orientation information according to the third sensing data. Taking FIG. 4 as an example, the two poses of the human body part B4 in the image at different time points can be used to estimate the orientation information.

在一些实施例中，处理器250可根据第一感测数据和第三感测数据确定定向信息。定向信息可以是第一感测数据和第三感测数据的加权组合。举例来说，根据基于运动传感器130的加权的第一定向信息和基于图像传感器360的加权的第二定向信息的总和确定位置信息。In some embodiments, the processor 250 may determine orientation information according to the first sensing data and the third sensing data. The orientation information may be a weighted combination of the first sensed data and the third sensed data. For example, the location information is determined from the sum of the first orientation information based on the weighting of the motion sensor 130 and the second orientation information based on the weighting of the image sensor 360 .

在一个实施例中，图像传感器360的视场将是关于是否根据第三感测数据使用定向信息的条件。如果人体部位存在于第三感测数据中，那么可根据第一感测数据和第三感测数据来确定定向信息。如果人体部位不存在于第三感测数据中，那么可仅根据第一感测数据来确定定向信息。In one embodiment, the field of view of the image sensor 360 will be a condition on whether to use the orientation information according to the third sensed data. If the body part exists in the third sensing data, the orientation information may be determined according to the first sensing data and the third sensing data. If the human body part is not present in the third sensing data, the orientation information may be determined only from the first sensing data.

在一个实施例中，处理器250可根据定向信息和位置信息确定用户的运动信息。可基于第一感测数据、第三感测数据或如上所述的第一感测数据和第三感测数据的组合生成定向信息。可基于如上所述的第一感测数据、第二感测数据以及第三感测数据生成位置信息。以图2中的人体部位B1或人体部位B2为实例，运动信息可与抬起、点、踢、踩或跳跃运动相关。In one embodiment, the processor 250 may determine the motion information of the user according to the orientation information and the location information. The orientation information may be generated based on the first sensed data, the third sensed data, or a combination of the first sensed data and the third sensed data as described above. The location information may be generated based on the first sensing data, the second sensing data, and the third sensing data as described above. Taking the human body part B1 or the human body part B2 in FIG. 2 as an example, the motion information may be related to a lift, point, kick, step or jump motion.

在另一实施例中，处理器250可根据基于第一感测数据的定向信息和基于第一位置信息和第二位置信息的调整后的位置信息确定用户的运动信息。无论人体部位是否存在于第三感测数据中，处理器250可预测用户的运动。In another embodiment, the processor 250 may determine the motion information of the user according to the orientation information based on the first sensing data and the adjusted location information based on the first location information and the second location information. Regardless of whether the body part exists in the third sensing data, the processor 250 may predict the motion of the user.

在另一实施例中，处理器250可根据基于第一感测数据的定向信息和基于第二感测数据和第三感测数据的组合位置信息确定用户的运动信息。这意味着，当人体部位存在及不存在于第三感测数据中时，可在两个持续时间内基于定向信息和组合位置信息确定运动信息。In another embodiment, the processor 250 may determine the motion information of the user according to the orientation information based on the first sensing data and the combined location information based on the second sensing data and the third sensing data. This means that the motion information can be determined based on the orientation information and the combined position information for two durations when the body part is present and not present in the third sensed data.

以图4和图5为实例，在图4中确定人体部位B4的举手运动，且在图5中确定放下手运动。接着，确定人体部位B4的从上到下的摆动运动。Taking FIG. 4 and FIG. 5 as an example, in FIG. 4 , the raising-hand motion of the human body part B4 is determined, and in FIG. 5 , the lowering-hand motion is determined. Next, the top-to-bottom swing motion of the human body part B4 is determined.

在一个实施例中，处理器250可仅根据基于第二感测数据的位置信息确定用户的运动信息。在另一实施例中，处理器250可仅根据基于第二感测数据和第三感测数据的组合位置信息确定用户的运动信息。在一些实施例中，如果人体部位不存在于第三感测数据中，那么处理器250可仅根据基于第二感测数据的位置信息确定用户的运动信息，且如果人体部位存在于第三感测数据中，那么处理器250可仅根据基于第三感测数据的位置信息或组合位置信息确定用户的运动信息。In one embodiment, the processor 250 may determine the motion information of the user only according to the location information based on the second sensing data. In another embodiment, the processor 250 may determine the motion information of the user only according to the combined location information based on the second sensing data and the third sensing data. In some embodiments, if the human body part does not exist in the third sensing data, the processor 250 may only determine the motion information of the user according to the position information based on the second sensing data, and if the human body part exists in the third sensing data In the sensing data, the processor 250 may determine the motion information of the user only according to the position information or the combined position information based on the third sensing data.

可跟踪人体部位的位移或轨迹，且可基于位移或轨迹确定运动信息。以图4和图5为实例，人体部位B3从上到下移动，且确定人体部位B4从上到下摆动运动。The displacement or trajectory of the body part can be tracked, and motion information can be determined based on the displacement or trajectory. Taking FIG. 4 and FIG. 5 as an example, the human body part B3 moves from top to bottom, and it is determined that the human body part B4 swings from top to bottom.

综上所述，在本发明实施例针对用户数个操作部位的相关于运动跟踪的系统及方法中，可基于信号强度、运动传感器的感测结果和/或相机图像追踪人体部位的动作。若依据相机图像的检测结果得出追踪的人体部位未存在于FOV内，则运动感测设备之间的信号强度可用于补偿使用运动传感器所得的感测结果在位置信息决定上的准确性。此外，若跟踪的人体部位存在于FOV内，则相机图像可用于修正基于信号强度的位置估测结果。藉此，数个追踪方法可用于不同条件，并据以改善跟踪结果的准确性。To sum up, in the system and method related to motion tracking for several operating parts of the user according to the embodiments of the present invention, the motion of human body parts can be tracked based on signal strength, sensing results of motion sensors and/or camera images. If the body part to be tracked does not exist in the FOV according to the detection result of the camera image, the signal strength between the motion sensing devices can be used to compensate the accuracy of the position information determination of the sensing result obtained by using the motion sensor. Additionally, if the tracked body part is present within the FOV, the camera image can be used to correct the signal strength-based position estimate. Thereby, several tracking methods can be used for different conditions, and accordingly, the accuracy of the tracking results can be improved.

本领域的技术人员将显而易见，可在不脱离本发明的范围或精神的情况下对本发明的结构作出各种修改和变化。鉴于以上内容，希望本发明涵盖本发明的修改和变化，前提为所述修改和变化落入所附权利要求和其等效物的范围内。It will be apparent to those skilled in the art that various modifications and changes can be made in the structure of the present invention without departing from the scope or spirit of the invention. In view of the foregoing, it is intended that the present invention covers modifications and variations of this invention provided that they fall within the scope of the appended claims and their equivalents.