CN119002693A - Interactive method, device, electronic equipment, storage medium, system and computer program product executed by electronic equipment - Google Patents

Detailed Description

In order to enable those skilled in the art to better understand the technical solutions of the present disclosure, the technical solutions of the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

It should be noted that the terms "first," "second," and the like in the description and claims of the present disclosure and in the foregoing figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the disclosure described herein may be capable of operation in sequences other than those illustrated or described herein. The embodiments described in the examples below are not representative of all embodiments consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present disclosure as detailed in the accompanying claims.

It should be noted that, in this disclosure, "at least one of the items" refers to a case where three types of juxtaposition including "any one of the items", "a combination of any of the items", "an entirety of the items" are included. For example, "including at least one of a and B" includes three cases side by side as follows: (1) comprises A; (2) comprising B; (3) includes A and B. For example, "at least one of the first and second steps is executed", that is, three cases are juxtaposed as follows: (1) performing step one; (2) executing the second step; (3) executing the first step and the second step.

Fig. 1 shows a schematic diagram of inputting a password through a remote controller on a smart television in the related art.

Referring to fig. 1, since the remote controller has a small size and fewer keys, whether the remote controller moves the selection frame in a direction to select from the keyboard displayed on the smart tv to realize input or directly input through the keyboard on the remote controller, the remote controller needs to be relied on as an auxiliary physical device, and the process is slow and tedious and cannot achieve the effects of high efficiency and convenience.

In addition, gesture recognition is a research and development area in computer science and language technology, involving the recognition and interpretation of human gestures. The gesture recognition technology in the related art is mostly based on a machine learning algorithm, and can mark 21 key points of a hand in an image as shown in fig. 2, that is, the coordinates of each key point in the image can be given.

Fig. 2 shows a schematic diagram of 21 key points for recognizing a gesture in the related art, and positions, labels, definitions of labels (for example, "0" means Wrist (WRIST)), and the like of the 21 key points of the gesture may refer to fig. 2.

Fig. 3 shows a schematic diagram of recognizing a gesture through a key point in the related art, referring to fig. 3, the related art may infer a current gesture through calculation according to coordinates of a key point of a hand, for example, "good (OK)", "1", "Yere (YEAH)", "spider knight-errant (SPIDERMAN)", "FIST (FIST)", "FIVE (FIVE)", "THREE (THREE)", "ROCK)", etc. may be sequentially expressed from top to bottom and from left to right in fig. 3, so that a smart television control command based on the gesture may be designed, thereby providing a more natural and intuitive interaction experience for a user.

Fig. 4 shows a schematic diagram of interaction with a smart television through gestures in the related art, and referring to fig. 4, corresponding interaction control on the smart television may be implemented through the following gestures in sequence: for example, "play/pause" may be achieved by "lifting the palm," up "may be achieved by" raising the index finger, "down" may be achieved by "thumb," mute/unmute "may be achieved by" fist, "return" may be achieved by "thumb" to the left, "enter" into the next page "may be achieved by" thumb up.

However, the following problems still remain with the interaction method based on gesture recognition:

(1) The learning cost is increased, and a user needs to learn a specific gesture;

(2) In the interaction process, a user needs to make a specific gesture in the air, which is inconvenient.

In the related art, an air mouse (also called an air mouse) is generally based on a wireless mouse or a physical remote controller, and a gyroscope, a UWB (Ultra-Wideband) module and the like are added, so that the change of direction and speed can be perceived, and a user can realize cursor movement and clicking on the intelligent television by only moving in the air and clicking the remote controller.

Fig. 5 shows a schematic diagram of interaction with a smart tv through an air mouse and a remote controller in the related art.

Referring to fig. 5 (a), in order to schematically illustrate interaction with the smart tv through the air mouse, a mouse arrow 502 corresponding to a remote controller 501 (air mouse) may be displayed on a screen of the smart tv; referring to (b) of fig. 5, in order to schematically interact with the smart tv through the remote controller, a screen of the smart tv may be pointed through the remote controller 503, and a pointing identifier 504 corresponding to the pointing of the remote controller 503 may be displayed in the screen.

However, this interaction method still has the following problems:

(1) The interaction requirement of the intelligent television cannot be completely adapted, because the mouse is designed for a computer initially, a user is required to watch a screen at a short distance, and the watching distance of the intelligent television is relatively long;

(2) There remains a need for a handheld physical device that does not differ much in the interactive experience from a traditional physical remote control.

Fig. 6 shows a schematic diagram of interaction with a smart tv through touch projection in the related art.

Referring to fig. 6, touch projection generally employs infrared or laser technology to implement touch functions, and generally includes a projection module 601 and a sensor module 602. The projection module can project the content of the smart television onto a certain physical plane, for example, plane 603, and the sensor module 602 can sense the gesture and operation of the user in the projection area 604 and convert the gesture and operation into signals which can be identified by the smart television, so that interaction similar to a touch screen is realized, and interaction efficiency and interaction performance are improved.

However, this interaction method still has the following problems:

(1) The user sight needs to be frequently switched between the projection area and the intelligent television screen, the interaction flow is complex, and the user experience is poor;

(2) To ensure the effect, the area of the projection area cannot be too small, otherwise the projected content is not clearly seen, which may limit the use of the scene.

Research finds that most users are more accustomed to a mouse interaction mode on a personal computer or a touch screen interaction mode on mobile equipment, and if the two interaction modes can be applied to the interaction process of electronic equipment such as intelligent televisions, the interaction experience of the users is greatly improved. Meanwhile, the watching distance of the intelligent television is far compared with that of a personal computer and mobile equipment, and the intelligent television can only be operated in a space. Therefore, the interaction between the space and the intelligent television in a touch screen or mouse mode can be realized through the novel interaction method.

In order to solve the above-mentioned problems, the present disclosure provides an interaction method, an apparatus, an electronic device, a storage medium, a system and a computer program product executed by an electronic device, which can enable a user to interact with the electronic device through a body part such as a hand without depending on any auxiliary physical device, and the interaction experience is consistent with the traditional touch screen or mouse interaction mode, so that the interaction experience of the user on the electronic device, especially the electronic device needing to interact with the space can be significantly improved.

The interactive method, apparatus, electronic device, storage medium, system and computer program product performed by the electronic device of the present disclosure will be described in detail below with reference to fig. 7 to 19.

The interaction method performed by the electronic device described in the exemplary embodiments of the present disclosure may be applied to various scenes (e.g., a smart tv, an in-vehicle center screen, etc.) requiring interaction with a screen, and the following description will mainly be given by taking a scene applied to smart tv interaction as an example.

Fig. 7 illustrates a scene graph applied to smart television interactions in an exemplary embodiment of the present disclosure.

Referring to fig. 7, a smart tv to which an interaction method according to an exemplary embodiment of the present disclosure is applied may detect a state of a user's hand to acquire a control instruction from the user, and the user may control the smart tv directly through a motion condition of the hand (e.g., various gestures) and a contact condition of the hand with a surface of an object (e.g., a desktop, etc.).

Fig. 8 shows a flowchart of an interaction method performed by an electronic device in an exemplary embodiment of the present disclosure.

Referring to fig. 8, in step 801, a virtual object corresponding to a user's hand is displayed on a screen of an electronic device in a first mode of operation of the electronic device.

According to an exemplary embodiment of the present disclosure, the execution subject of each step of the present disclosure may be an electronic device having a display function, such as a smart tv, and the electronic device may interact with a user. An instruction to turn on the unassisted device mode of operation may be issued by a user to enter the first mode of operation. For example, the user may enter the first operation mode by operating a remote control of the smart television, or may enter the first operation mode by voice control, which may not be limiting of the present disclosure. After entering the first mode of operation, the user may be prompted by way of displaying a virtual object on the screen that the first mode of operation has been entered currently. The virtual object may be used to indicate a screen position corresponding to where the user's hand is located and a state of the user's hand. The initial position of displaying the virtual object may be a preset position, such as a screen center, etc., in a display form including, but not limited to, a mouse cursor, a virtual palm, a highlighted selection box, etc., a virtual icon for prompting visual focus, or other possible forms.

At step 802, a state of a user's hand may be detected, where the state of the hand may include a motion of the hand and a contact of the hand with a surface of an object. The motion of the hand may include speed of movement of the hand, direction, gesture, etc. The object contacted by the user's hand may be any object that is currently in surface contact with the user's hand, such as a desktop, a dedicated touch pad, a touch ball, or the like. The contact condition of the hand with the surface of the object may include whether the hand is in contact with the object, the size of the contact area, the contact location, etc.

In accordance with exemplary embodiments of the present disclosure, for human body sensing and gesture recognition techniques, more advanced techniques may also occur in the future based on millimeter wave radar, machine vision, wearable devices, etc., which human body sensing and gesture recognition techniques may not be used to detect the state of a user's hand.

According to the exemplary embodiment of the disclosure, a millimeter wave radar module can be built in the smart television to detect the state of the hand of the user. The millimeter wave radar is a radar working in the millimeter wave band, can measure the relative distance, speed and direction of a measured object, can be widely applied to the fields of human body detection, gesture recognition and the like, and can detect the movement of a few tenths of millimeters, for example, the millimeter wave radar with the working frequency of 76-81GHz (the corresponding wavelength is about 4 mm). Millimeter wave radars can sense the surrounding environment by emitting millimeter waves and receiving their reflected signals, which can be converted into a set of discrete point clouds, each of which can represent the location and properties of a reflection source, which can include the three-dimensional coordinates of an object, the speed of movement, the direction of movement, etc.

At step 803, a corresponding control instruction may be determined based on the detected state of the hand.

According to the exemplary embodiment of the disclosure, the control instruction of the smart television can be determined according to the detected action condition of the hand and the contact condition of the hand and the object surface.

According to an exemplary embodiment of the present disclosure, determining a corresponding control instruction based on the detected state of the hand may include, but is not limited to, the following steps: determining a position of the hand in contact with the object surface based on the detected contact condition of the hand with the object surface; when it is determined that the preset portion of the hand is in contact with the object surface, a control instruction corresponding to the motion situation of the hand may be determined based on the detected motion situation of the hand.

According to the exemplary embodiment of the disclosure, the preset portion of the hand may be, for example, a wrist, an elbow, etc., and when it is determined that the preset portion of the hand is in contact with the surface of the object, a corresponding control command may be determined according to the action condition of the hand, so as to prevent false touch and improve control efficiency.

According to an exemplary embodiment of the present disclosure, determining a location of a hand in contact with a surface of an object based on a detected contact condition of the hand with the surface of the object may include, but is not limited to, the steps of: the point cloud of the hand and the object in the preset range under the preset coordinate system can be detected in real time through the millimeter wave radar; an object having a portion overlapping with the point cloud of the hand may be determined as an object with which the hand is currently in contact; a set of locations of the hand overlying the portion may be identified to determine where the hand is in contact with the object surface.

According to an exemplary embodiment of the present disclosure, when detecting the motion condition of a user's hand and the contact condition of the hand with the surface of an object in real time, the following steps may be adopted:

(1) The point cloud of the user hand under a preset coordinate system can be detected in real time through the millimeter wave radar module;

(2) Judging whether the point cloud of one object is overlapped with the point cloud of the hand of the user or not according to the point cloud, wherein the specific method comprises traversing the three-dimensional coordinates of each point in the point cloud, judging whether the three-dimensional coordinates of the point cloud of the other object are consistent with the three-dimensional coordinates or the distance is smaller than a certain threshold value, if so, executing the step (3), otherwise, returning to the step (1);

(3) Recognizing a user gesture and a part set of a user hand corresponding to the overlapping part through a gesture recognition algorithm based on millimeter wave radar, wherein the user gesture can comprise but is not limited to clicking, sliding up and down and left and right, and the like, and the part set of the user hand can comprise but is not limited to an index finger, a middle finger, a wrist, an elbow, and the like;

(4) Judging whether the above-mentioned user gesture and the position set of the user hand meet the preset condition, if yes, executing step (5), otherwise returning to step (1), where the preset condition may be used to prevent the user from misoperation, for example, one possible preset condition is that the position set of the user hand needs to contain the wrist, that is, only when the wrist of the user contacts with the object, the operation action of the user is identified;

(5) The gesture and the set of parts of the user hand may be mapped into corresponding operation actions according to a preset mapping relationship, where the operation actions may include, but are not limited to, clicking with an index finger, clicking with a middle finger, sliding up and down, left and right, etc., the mapping relationship may refer to the mapping relationship between the gesture and the set of parts of the user hand and the operation actions, for example, if the gesture is clicking, and the set of parts of the user hand includes an index finger, the corresponding operation action may be clicking with an index finger.

FIG. 9 shows a flow diagram of gesture recognition in an exemplary embodiment of the present disclosure.

Referring to fig. 9, a method of recognizing a user gesture by a millimeter wave radar according to an exemplary embodiment of the present disclosure may be as follows:

[1] The two-dimensional fast Fourier transform is adopted to carry out distance and Doppler analysis on the scene:

Two-dimensional fast fourier transform (Two-Dimensional Fast Fourier Transform, 2D-FFT) operations may be performed on analog-to-digital converted (Analog to Digital Converter, ADC) data for each receive channel, resulting in a Two-dimensional fast fourier transform matrix (2D-FFT matrix) comprising a distance dimension (Range) and a Doppler dimension (Doppler);

[2] 2D-FFT Non-coherent accumulation across multiple channels (Non-coherent accumulation) to create a range-doppler heat map:

Incoherent accumulation can be performed on the two-dimensional fast fourier transform matrix of each receive channel to obtain a Range-Doppler Heat map (Range-Doppler Heat map);

[3] A plurality of features may be extracted in the range-doppler heat map, each feature generating a time series; specifically, the time sequence may be acquired according to the frame index;

[4] The extracted features are then used for gesture recognition:

The features extracted in the specific time window can be classified by a machine learning algorithm, and the corresponding gesture category is output, for example, the features extracted in the specific time window can be sent to the machine learning algorithm such as an artificial neural network (ARTIFICIAL NEURAL NETWORK, ANN).

According to an exemplary embodiment of the present disclosure, the state of the hand may further include a distance between the hand and the electronic device; wherein the corresponding control instruction is determined based on the detected hand state, which may include, but is not limited to, the following steps: the corresponding control instruction may be determined to adjust the size of the virtual object displayed on the screen based on the detected distance between the hand and the electronic device.

According to the exemplary embodiment of the disclosure, the three-dimensional coordinates of the user hand and the smart television under the preset coordinate system with the position of the user as the origin can be detected in real time through the millimeter wave radar module, and the distance between the user hand and the smart television can be calculated according to the three-dimensional coordinates. For example, if the smart tv has a coordinate of (x ₀,y₀,z₀) and the user's hand has a coordinate of (x ₁,y₁,z₁), the distance may beThe distance between the user's hand and the smart television can represent the distance between the user and the smart television, when the distance is larger, the virtual object displayed on the screen can be enlarged so as to reduce the visual threshold, and when the distance is smaller, the virtual object displayed on the screen can be reduced so as to avoid shielding the picture.

According to the exemplary embodiment of the disclosure, the three-dimensional coordinates of the user hand under the preset coordinate system with the position of the user as the origin can be detected in real time through the millimeter wave radar module, and the moving direction and the moving distance of the user hand are calculated according to the change of the three-dimensional coordinates. For example, if the user's hand moves from coordinates (x ₁,y₁,z₁) to coordinates (x ₂,y₂,z₂), the movement vector may be expressed as (x ₂-x₁,y₂-y₁,z₂-z₁) = (a, b, c), and the movement distance may be expressed asThe direction of movement can be expressed as

According to the exemplary embodiments of the present disclosure, it is possible to determine whether a user's hand is in contact with the surface of an object and whether a contact portion satisfies a preset condition, and perform different interactive operations according to different determination results.

According to an exemplary embodiment of the present disclosure, when the preset portion in contact with the object surface includes the first portion, determining a control instruction corresponding to the motion condition of the hand based on the detected motion condition of the hand may include, but is not limited to, one of the following: when the index finger is detected to click the surface of the object, the corresponding control instruction can be determined to be a first click instruction; when the middle finger is detected to click on the surface of the object, the corresponding control instruction can be determined to be a second click instruction; when the sliding of the index finger on the surface of the object is detected, a corresponding control instruction can be determined to be a sliding current page; when the index finger is detected to move on the surface of the object relative to the user, the corresponding control instruction can be determined to be the position of the virtual object displayed on the mobile screen.

According to the exemplary embodiment of the disclosure, the first portion may be a wrist, or may be another portion, taking the first portion as an example of the wrist, when the preset portion contacted with the surface of the object includes the wrist, the action condition of the hand of the user contacting the surface of the object may be simulated as a space to perform mouse operation on the smart television; the first click command can simulate that the left button of the mouse is clicked and is used for realizing basic operations such as clicking, selecting, dragging and the like; the second click command can simulate that the right button of the mouse is clicked and is used for opening a menu, viewing attributes, shortcuts and other operations; the operation of the mouse wheel can be simulated by sliding the index finger up and down on the surface of the object, and the corresponding control instruction can be determined to be sliding the current page.

According to an exemplary embodiment of the present disclosure, when the preset portion in contact with the object surface includes the second portion, determining a control instruction corresponding to the motion condition of the hand based on the detected motion condition of the hand may include, but is not limited to, one of the following: when the hand is detected to slide on the surface of the object, a corresponding control instruction can be determined to be a current sliding page; when the finger clicking on the surface of the object is detected, the corresponding control instruction can be determined to be a third clicking instruction; when a hand movement is detected with respect to the user, a corresponding control instruction may be determined as a position of a virtual object displayed on the mobile screen.

According to the exemplary embodiment of the present disclosure, the second portion may be an elbow, or may be another portion, taking the second portion as an elbow as an example, when the preset portion that contacts the surface of the object includes an elbow, the action condition of the user's hand contacting the surface of the object may be simulated as a space-free touch operation on the smart television; the third click command can simulate the click of the user finger on the element in the screen in a touch screen mode; at this time, the user can control the position where the virtual object is displayed in the screen by the movement of the hand in the air.

It should be understood that the above mapping relation of the actions of the user's hand and the control instructions is merely an example, and the actions, the control instructions, and the mapping relation thereof according to the embodiments of the present disclosure are not limited thereto. In addition, new actions and control instructions can be adjusted or added according to the needs of the user. In addition, the portion where the hand of the user contacts the object surface is not limited to the wrist or elbow, and may be a portion of another hand such as a finger.

Referring back to fig. 8, at step 804, the virtual object may be controlled according to the control instructions to perform interactions of the user with the electronic device.

According to an exemplary embodiment of the present disclosure, the virtual object may be controlled according to a control instruction to perform an element click, an element selection, a page swipe, a page zoom, etc. interaction between the user and the electronic device. Here, the element may refer to an icon, a control, or the like displayed on the electronic device, and the page may refer to a displayed home screen page, an application page, or the like of the electronic device. It should be understood that the type of interaction is not limited thereto.

According to the exemplary embodiment of the disclosure, the control instruction is executed by the man-machine interaction module built in the intelligent television in a manner of embedding the man-machine interaction module in the intelligent television so as to interact with the user. The man-machine interaction module can dynamically execute corresponding control instructions for the intelligent television according to a preset mapping relation according to the operation action of the user hand on the surface of a certain object (the object currently contacted by the user hand), wherein the mapping relation refers to the mapping relation between the operation action of the user hand on the surface of the certain object and the control instructions for the intelligent television.

According to the exemplary embodiment of the disclosure, the size of the virtual object can be dynamically adjusted according to the real-time detection result, the virtual object is moved, the corresponding virtual object animation effect is displayed, and the corresponding control instruction for the intelligent television is executed.

For example, when the user hand performs index finger clicking operation on the object surface, the smart television can execute a control instruction for clicking the current element; when the user hand performs left-right sliding operation on the surface of the object, the intelligent television can execute a control instruction for sliding the current page left and right.

For example, the position of the virtual object on the smart television screen can be dynamically updated according to a preset mapping relation by the man-machine interaction module according to the moving direction and the moving distance of the user hand relative to the user, wherein the mapping relation can refer to the mapping relation between the moving direction and the moving distance of the user hand relative to the user and the moving direction and the moving distance of the virtual object on the smart television screen. For example, if the user's hand moves left and right with respect to the user, the virtual object may move left and right with respect to the screen, and if the user's hand moves back and forth with respect to the user, the virtual object may move up and down with respect to the screen; a possible mapping relationship of the movement distance is that if the movement distance of the user hand in the corresponding movement direction is x and the preset sensitivity coefficient is a, the movement distance of the virtual object relative to the screen may be x×a.

According to example embodiments of the present disclosure, mapping movements of a user's hand to corresponding positional movements of virtual objects displayed on an electronic device may be implemented.

According to the exemplary embodiment of the disclosure, the animation effect of the corresponding virtual object can be dynamically displayed according to a preset mapping relation through the human-computer interaction module according to the operation action of the user hand on the surface of a certain object. The mapping relation here may refer to a mapping relation between an operation action of a user hand on a surface of a certain object and a preset animation effect of a virtual object; for example, when a user's hand performs an index finger clicking operation on the object surface, a virtual object on the screen may display an animation effect of the index finger clicking; when the user's hand performs a sliding operation on the object surface, the virtual object on the screen can display an animation effect of sliding left and right.

According to an exemplary embodiment of the present disclosure, the operation actions of the user's hand on the object surface may be mapped to respective animation effects of the virtual object and respective interactive operations on the electronic device.

Fig. 10 illustrates a flow chart for resizing virtual objects in an exemplary embodiment of the present disclosure.

Referring to fig. 10, according to an exemplary embodiment of the present disclosure, a human hand may be detected in real time, and the size of a virtual object may be adjusted according to the real-time detection result.

For example, the size of the virtual object can be dynamically adjusted according to a preset mapping relation by the man-machine interaction module according to the distance between the user hand and the intelligent television, so that the user can still obtain better interaction experience when the user is far or near to the intelligent television. The mapping relationship may be a mapping relationship between a distance value and a virtual object size, or a mapping relationship between a distance range and a virtual object size. For example, assuming that the preset standard distance is X ₀, the preset virtual object initial size is Y ₀, the current distance is X, and the current virtual object size is Y, then: y=y ₀*(X/X₀).

According to the exemplary embodiment of the disclosure, the size of the virtual object on the smart television can be dynamically adjusted according to the distance between the hand of the user and the electronic device.

Fig. 11 shows a frame diagram of an interaction method performed by an electronic device in an exemplary embodiment of the present disclosure.

Fig. 12 shows a flow diagram of an interaction method performed by an electronic device in an exemplary embodiment of the present disclosure.

Referring to fig. 11, according to an exemplary embodiment of the present disclosure, a millimeter wave radar module and a man-machine interaction module may be built in a smart tv, and referring to fig. 12, after a first operation mode, i.e., an operation mode without auxiliary devices, is turned on, a distance between a user's hand and the smart tv, a movement of the user with respect to the smart tv, an operation action on a surface of an object, etc. may be detected in real time (through the millimeter wave radar module); the size of the virtual object can be dynamically adjusted (through the man-machine interaction module) according to the real-time detection result, the virtual object is moved, the corresponding virtual object animation effect is displayed, the corresponding control instruction for the intelligent television is executed, and the like.

Fig. 13 shows a flowchart of implementing a blank touch screen operation using an interaction method performed by an electronic device in an exemplary embodiment of the present disclosure.

Referring to fig. 13, according to an exemplary embodiment of the present disclosure, a user may perform a touch screen operation on a smart tv screen through a hand space, so that an interactive experience consistent with the touch screen interaction can be obtained without touching the screen. The specific steps can be as follows: the millimeter wave radar module and the man-machine interaction module can be built in the intelligent television; after a user opens the intelligent television and opens a touch screen interaction mode (one of the interaction modes without auxiliary equipment), a virtual hand can be displayed on the screen of the intelligent television, and the initial position can be positioned at the right center of the screen of the intelligent television; the millimeter wave radar module can detect the movement of the hand of the user relative to the user in real time and the operation action on the surface of a certain object; when the user's elbow contacts with a certain object, the man-machine interaction module can execute corresponding control instructions to the smart television according to the real-time detection result, for example: when the user hand slides left and right on the surface of the current object, the intelligent television can execute a control instruction for sliding the current page left and right; when the user hand slides up and down on the surface of the current object, the intelligent television can execute a control instruction for sliding the current page up and down; fig. 14 is a schematic diagram illustrating implementation of a blank touch screen operation using an interaction method performed by an electronic device in an exemplary embodiment of the present disclosure, and referring to (a) in fig. 14, when a user's finger clicks a current object surface, a smart tv may execute a control instruction to click a current element. When the user's elbow contacts an object, the human-computer interaction module may move the virtual hand according to the real-time detection result, for example: referring to (b) of fig. 14, when the user's hand moves left and right with respect to the user, the virtual hand may move left and right with respect to the screen; when the user hand moves back and forth relative to the user, the virtual hand may move up and down relative to the screen, i.e., the user hand may move in a space in the touch screen mode to control the movement of the virtual hand in the screen.

Fig. 15 shows a flowchart of implementing a blank mouse operation using an interaction method performed by an electronic device in an exemplary embodiment of the present disclosure.

Referring to fig. 15, according to an exemplary embodiment of the present disclosure, a user may perform a mouse operation on a smart tv screen through a hand space, so that an interactive experience consistent with a mouse interaction can be obtained without depending on a physical mouse. The method comprises the following specific steps: the millimeter wave radar module and the man-machine interaction module can be built in the intelligent television; after the user opens the smart tv and opens the mouse interaction mode (one of the first operation modes without auxiliary device interaction), a mouse cursor may be displayed on the screen of the smart tv, and the initial position may be located at the very center of the screen of the smart tv. The millimeter wave radar module can be used for detecting the movement of the hand of the user relative to the user in real time and the operation action on the surface of a certain object; when the wrist of the user contacts with a certain object, a corresponding control instruction for the intelligent television can be executed through the human-computer interaction module according to the real-time detection result, for example: FIG. 16 is a schematic diagram showing a method of implementing a space-saving mouse operation using an interaction method performed by an electronic device in an exemplary embodiment of the present disclosure, and referring to (a) in FIG. 16, when a user index finger clicks a current object surface, a control instruction for clicking a left mouse button may be executed by the smart television; when the middle finger of the user clicks the surface of the current object, the intelligent television can execute a control instruction for clicking the right button of the mouse; when the index finger of the user slides up and down on the surface of the current object, the intelligent television can execute the mouse wheel rolling operation. When the wrist of the user contacts with a certain object, the human-computer interaction module can move the mouse cursor according to the real-time detection result, for example: referring to (b) of fig. 16, when a user's index finger moves left and right with respect to a user on the surface of the current object, the mouse cursor may move left and right with respect to the screen; the mouse cursor may move up and down relative to the screen as the user's index finger moves back and forth relative to the user on the surface of the current object.

According to an exemplary embodiment of the present disclosure, the size of the virtual object may be dynamically adjusted according to the distance between the user's hand and the smart tv, and the specific steps may be as follows: the millimeter wave radar module and the man-machine interaction module can be built in the intelligent television; after a user opens the intelligent television and opens a first operation mode without auxiliary equipment, virtual objects can be displayed on the screen of the intelligent television, and the initial size can be a default size; the distance between the user hand and the intelligent television can be detected in real time through the millimeter wave radar module; the size of the virtual object may be dynamically adjusted according to the real-time detection result through the human-computer interaction module, for example, fig. 17 shows a schematic diagram of adjusting the size of the virtual object in an exemplary embodiment of the present disclosure, and referring to (a) in fig. 17, when the user is far from the smart tv, a larger virtual object may be displayed; referring to (b) of fig. 17, as the user is closer to the smart tv, a smaller virtual object may be displayed.

According to an exemplary embodiment of the present disclosure, a new interaction method performed by an electronic device may be proposed based on millimeter wave radar technology, to realize interaction with the electronic device in a touch screen or mouse manner at intervals.

Fig. 18 shows an application scenario diagram of an interaction method performed by an electronic device in an exemplary embodiment of the present disclosure.

According to an exemplary embodiment of the present disclosure, as a new manner of spaced interaction, the present disclosure may be applied to any scenario requiring spaced interaction, and implementation steps may be substantially consistent. For example, referring to (a) in fig. 18, the present disclosure may be applied to a screen-wise input of a password or the like on a smart television, a screen-wise zoom view of a photograph on an electronic device, a screen-wise operation of a vehicle-mounted center screen on an automobile steering wheel, or the like.

In addition, application scenarios of the interaction method performed by the electronic device in the exemplary embodiments of the present disclosure may also include, but are not limited to, digital signage, projectors, in-vehicle center control screens, VR (Virtual Reality)/AR (Augmented Reality), MR (Mixed Reality) glasses, and any other scenario requiring isolated interactions.

The interaction method executed by the electronic device provided by the exemplary embodiment of the disclosure can enable the user to interact with the electronic device without depending on any auxiliary physical device through hands, and the interaction experience is consistent with the traditional touch screen or mouse interaction mode, so that the interaction experience of the user on the electronic device, particularly the electronic device needing to interact with the electronic device at intervals, can be remarkably improved.

Compared to the gesture recognition-based interaction method in the related art, the exemplary embodiments of the present disclosure have the following advantages: the learning cost is low, the user does not need to learn a specific gesture, and any user used to a touch screen or mouse interaction mode can quickly get up; the interaction process is easier, and the user does not need to make a specific gesture in the air, and only needs to operate like touching a screen or operating a mouse on the surface of an object.

Compared to the air mouse-based interaction method in the related art, the exemplary embodiments of the present disclosure have the following advantages: the interaction requirement of the intelligent television can be completely adapted, and a user can more easily identify a mouse cursor on the screen of the intelligent television; less dependence on physical devices and no need to hold any physical devices.

Compared to the interaction method based on touch projection in the related art, the exemplary embodiments of the present disclosure have the following advantages: the interaction process is easier, and the user sight does not need to be frequently switched between the projection area and the screen of the electronic equipment; the physical plane has low requirement, the user can complete interaction on any object surface within the identifiable range of the millimeter wave radar, and the user can move the position and reselect the physical plane for interaction at will in the interaction process.

Fig. 19 shows a block diagram of an interaction device executed by an electronic device in an exemplary embodiment of the disclosure.

Referring to fig. 19, exemplary embodiments of the present disclosure also provide an interactive apparatus 1900 executed by an electronic device, which may include, but is not limited to, an object display module 1901, a state detection module 1902, an instruction determination module 1903, and an interactive execution module 1904.

The object display module 1901 may display a virtual object corresponding to a user's hand on a screen of the electronic device in a first mode of operation of the electronic device.

The state detection module 1902 may detect a state of a hand of a user, where the state of the hand includes a motion condition of the hand and a contact condition of the hand with a surface of an object.

The instruction determination module 1903 may determine a corresponding control instruction based on the detected state of the hand.

The interaction execution module 1904 may control the virtual object according to the control instruction to execute the interaction of the user and the electronic device.

According to an example embodiment of the present disclosure, the instruction determination module 1903 may determine a location where the hand is in contact with the object surface based on the detected contact condition of the hand with the object surface; when it is determined that the preset portion of the hand is in contact with the object surface, a control instruction corresponding to the motion situation of the hand is determined based on the detected motion situation of the hand.

According to an exemplary embodiment of the present disclosure, when the preset portion in contact with the object surface includes the first portion, the instruction determination module 1903 may execute at least one of the following including, but not limited to: when the index finger is detected to click the surface of the object, determining the corresponding control instruction as a first click instruction; when the middle finger is detected to click the surface of the object, determining the corresponding control instruction as a second click instruction; when the sliding of the index finger on the surface of the object is detected, determining a corresponding control instruction as a sliding current page; when the index finger is detected to move on the surface of the object relative to the user, the corresponding control instruction is determined to be the position of the virtual object displayed on the mobile screen.

According to an exemplary embodiment of the present disclosure, when the preset portion in contact with the object surface includes the second portion, the instruction determination module 1903 may execute at least one of the following including, but not limited to: when the hand is detected to slide on the surface of the object, determining a corresponding control instruction as a current sliding page; when the fact that the finger clicks the surface of the object is detected, determining that the corresponding control instruction is a third click instruction; when the movement of the hand relative to the user is detected, the corresponding control instruction is determined as the position of the virtual object displayed on the mobile screen.

According to an exemplary embodiment of the present disclosure, the state of the hand may further include a distance between the hand and the electronic device; wherein the instruction determination module 1903 may determine the corresponding control instruction to adjust the size of the virtual object displayed on the screen based on the detected distance between the hand and the electronic device.

According to an exemplary embodiment of the present disclosure, the instruction determination module 1903 may detect a point cloud of a hand and an object within a preset range under a preset coordinate system in real time by a millimeter wave radar; determining an object with a superposition part with the point cloud of the hand as the object currently contacted by the hand; a set of locations of the hand overlying the portion is identified to determine locations of the hand in contact with the object surface.

It can be appreciated that in the above exemplary embodiment of the interaction device 1900 executed by the electronic apparatus, the specific implementation process is substantially the same as that in the above exemplary embodiment of the interaction method executed by the electronic apparatus, and will not be described herein. The interaction means 1900 executed by the electronic device may each be configured as software, hardware, firmware, or any combination of the above for performing the specified functions. For example, these means may correspond to application specific integrated circuits, to pure software code, or to modules of software in combination with hardware. Furthermore, one or more functions implemented by these means may also be performed uniformly by components in a physical entity apparatus (e.g., a processor, a client, a server, or the like).

Fig. 20 shows a block diagram of an electronic device of an exemplary embodiment of the present disclosure.

Referring to fig. 20, an electronic device 2000 includes at least one memory 2001 and at least one processor 2002, the at least one memory 2001 having stored therein a set of computer-executable instructions that, when executed by the at least one processor 2002, perform an interaction method performed by the electronic device according to an exemplary embodiment of the present disclosure.

By way of example, the electronic device 2000 may be a PC computer, tablet device, personal digital assistant, smart phone, or other device capable of executing the above-described set of instructions. Here, the electronic device 2000 is not necessarily a single electronic device, but may be any apparatus or a collection of circuits capable of executing the above-described instructions (or instruction set) individually or in combination. The electronic device 2000 may also be part of an integrated control system or system manager or may be a portable electronic device configured to interface with itself locally or remotely (e.g., via wireless transmission).

In electronic device 2000, processor 2002 may include a Central Processing Unit (CPU), a Graphics Processor (GPU), a programmable logic device, a special purpose processor system, a microcontroller, or a microprocessor. By way of example, and not limitation, processors may also include analog processors, digital processors, microprocessors, multi-core processors, processor arrays, network processors, and the like.

The processor 2002 may execute instructions or code stored in the memory 2001, wherein the memory 2001 may also store data. The instructions and data may also be transmitted and received over a network via a network interface device, which may employ any known transmission protocol.

The memory 2001 may be integrated with the processor 2002, for example, by arranging RAM or flash memory within an integrated circuit microprocessor or the like. In addition, memory 2001 may include a stand-alone device, such as an external disk drive, a storage array, or any other storage device usable by a database system. The memory 2001 and the processor 2002 may be operatively coupled or may communicate with each other, for example, through an I/O port, a network connection, etc., such that the processor 2002 is able to read files stored in the memory.

In addition, the electronic device 2000 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, mouse, touch input device, etc.). All components of the electronic device 2000 may be connected to each other via buses and/or networks.

According to an exemplary embodiment of the present disclosure, a computer-readable storage medium storing instructions may also be provided, wherein the instructions, when executed by at least one computing device, cause the at least one computing device to perform the above-described interaction method performed by an electronic apparatus.

Examples of the computer readable storage medium herein include: read-only memory (ROM), random-access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random-access memory (RAM), dynamic random-access memory (DRAM), static random-access memory (SRAM), flash memory, nonvolatile memory, CD-ROM, CD-R, CD + R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD + R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, blu-ray or optical disk storage, hard Disk Drives (HDD), solid State Disks (SSD), card-type memories (such as multimedia cards, secure Digital (SD) cards or ultra-fast digital (XD) cards), magnetic tapes, floppy disks, magneto-optical data storage devices, hard disks, solid state disks, and any other devices configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and to provide the computer programs and any associated data, data files and data structures to a processor or computer to enable the processor or computer to execute the programs. The computer programs in the computer readable storage media described above can be run in an environment deployed in a computer device, such as a client, host, proxy device, server, etc., and further, in one example, the computer programs and any associated data, data files, and data structures are distributed across networked computer systems such that the computer programs and any associated data, data files, and data structures are stored, accessed, and executed in a distributed fashion by one or more processors or computers. It should be noted that the instructions may also be used to perform additional steps than or more specific processes when performing the above steps, the contents of which have been mentioned in the description of the relevant method, and thus will not be repeated here.

Another embodiment of the present disclosure is directed to a system comprising at least one computing device and at least one storage device storing instructions that, when executed by the at least one computing device, cause the at least one computing device to perform the above-described interaction method performed by an electronic device.

It should be noted that a system according to an exemplary embodiment of the present disclosure may rely entirely on the execution of a computer program or instruction, i.e., the individual elements correspond to the individual steps in the functional architecture of the computer program so that the entire system is invoked by a dedicated software package (e.g., lib library) to achieve the corresponding functionality.

On the other hand, when the above-described system is implemented in software, firmware, middleware or microcode, the program code or code segments to perform the corresponding operations may be stored in a computer-readable medium, such as a storage medium, so that at least one processor or at least one computing device can perform the corresponding operations by reading and executing the corresponding program code or code segments.

According to exemplary embodiments of the present disclosure, the storage device may be integrated with the computing device, for example, RAM or flash memory disposed within an integrated circuit microprocessor or the like. Further, the storage devices may include stand-alone devices, such as external disk drives, storage arrays, or other storage devices usable by any database system. The storage device and the computing device may be operatively coupled or may communicate with each other, such as through an I/O port, network connection, or the like, such that the computing device is capable of reading instructions stored in the storage device.

Another embodiment of the present disclosure relates to a computer program product comprising a computer program/instructions which, when executed by a processor, implement the interaction method performed by an electronic device as described in any of the above.

According to the interaction method, the device, the electronic equipment, the storage medium, the system and the computer program product for executing the electronic equipment, the user can interact with the target equipment through the body parts such as hands without depending on any auxiliary physical equipment, the interaction experience is consistent with the traditional touch screen or mouse interaction mode, and the interaction experience of the user on the target equipment, particularly the target equipment needing to interact with the space can be remarkably improved.

In addition, the control instruction corresponding to the gesture of the user is recognized only when the specific preset part contacts the surface of the object, so that false contact can be prevented, and interaction efficiency is improved.

In addition, when the wrist and the like contact the surface of the object, the operation of the hand of the user on the surface of the object can be simulated as mouse operation so as to control the electronic equipment at intervals, and the interaction efficiency and the operation convenience are high.

In addition, when the elbow and the like contact the surface of the object, the action condition of the hand of the user and the contact condition of the surface of the contact object can be simulated into touch screen operation so as to control the electronic equipment at intervals, and the electronic equipment has high interaction efficiency and high operation convenience.

In addition, the learning cost is low, the user does not need to learn a specific gesture, and any user used to a touch screen or mouse interaction mode can quickly get up; moreover, the interaction process is easy, the user sight can be switched frequently between auxiliary equipment such as a projection area and the screen of electronic equipment such as the intelligent television, a specific gesture can be made without being in the air, and the electronic equipment such as the intelligent television can be operated just like touching the screen or operating a mouse on the surface of an object.

In addition, the distance between the user and the electronic equipment is detected, and the display size of the virtual object is adjusted accordingly, so that the visual requirement of the user can be met, the interaction efficiency is improved, and the interaction experience is optimized.

In addition, the dynamic conditions of the user and surrounding objects are detected through the millimeter wave radar, whether the body part of the user is contacted with a certain object or not and which parts are contacted are judged through whether the point clouds are overlapped, and the corresponding purpose can be achieved efficiently, conveniently, rapidly and with low consumption.

In addition, the physical plane of the contact object has low requirement, the user can complete interaction on any object surface in the recognizable range of the millimeter wave radar, and the user can move the position and reselect the physical plane for interaction at will in the interaction process.

The foregoing description of exemplary embodiments of the present disclosure has been presented only to be understood as illustrative and not exhaustive, and the present disclosure is not limited to the exemplary embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the disclosure. Accordingly, the scope of the present disclosure should be determined by the scope of the claims.