patents.google.com

CN112511741A - Image processing method, mobile terminal and computer storage medium - Google Patents

️Tue Mar 16 2021

Detailed Description

It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. In the following description, suffixes such as "module", "component", or "unit" used to denote elements are used only for facilitating the explanation of the present invention, and have no specific meaning in itself. Thus, "module", "component" or "unit" may be used mixedly.

The terminal may be implemented in various forms. For example, the terminal described in the present invention may include a mobile terminal such as a mobile phone, a tablet computer, a notebook computer, a palmtop computer, a Personal Digital Assistant (PDA), a Portable Media Player (PMP), a navigation device, a wearable device, a smart band, a pedometer, and the like, and a fixed terminal such as a Digital TV, a desktop computer, and the like. The following description will be given by way of example of a mobile terminal, and it will be understood by those skilled in the art that the construction according to the embodiment of the present invention can be applied to a fixed type terminal, in addition to elements particularly used for mobile purposes.

Referring to fig. 1, which is a schematic diagram of a hardware structure of a mobile terminal for implementing various embodiments of the present invention, the

mobile terminal

100 may include: RF (Radio Frequency)

unit

101,

WiFi module

102,

audio output unit

103, a/V (audio/video)

input unit

104,

sensor

105,

display unit

106,

user input unit

107,

interface unit

108,

memory

109,

processor

110, and

power supply

111. Those skilled in the art will appreciate that the mobile terminal architecture shown in fig. 1 is not intended to be limiting of mobile terminals, which may include more or fewer components than those shown, or some components may be combined, or a different arrangement of components.

The following describes each component of the mobile terminal in detail with reference to fig. 1:

the

radio frequency unit

101 may be configured to receive and transmit signals during information transmission and reception or during a call, and specifically, receive downlink information of a base station and then process the downlink information to the

processor

110; in addition, the uplink data is transmitted to the base station. Typically,

radio frequency unit

101 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier, a duplexer, and the like. In addition, the

radio frequency unit

101 can also communicate with a network and other devices through wireless communication. The wireless communication may use any communication standard or protocol, including but not limited to GSM (Global System for Mobile communications), GPRS (General Packet Radio Service), CDMA2000(Code Division Multiple Access 2000), WCDMA (Wideband Code Division Multiple Access), TD-SCDMA (Time Division-Synchronous Code Division Multiple Access), FDD-LTE (Frequency Division duplex Long Term Evolution), and TDD-LTE (Time Division duplex Long Term Evolution).

WiFi belongs to short-distance wireless transmission technology, and the mobile terminal can help a user to receive and send e-mails, browse webpages, access streaming media and the like through the

WiFi module

102, and provides wireless broadband internet access for the user. Although fig. 1 shows the

WiFi module

102, it is understood that it does not belong to the essential constitution of the mobile terminal, and may be omitted entirely as needed within the scope not changing the essence of the invention.

The

audio output unit

103 may convert audio data received by the

radio frequency unit

101 or the

WiFi module

102 or stored in the

memory

109 into an audio signal and output as sound when the

mobile terminal

100 is in a call signal reception mode, a call mode, a recording mode, a voice recognition mode, a broadcast reception mode, or the like. Also, the

audio output unit

103 may also provide audio output related to a specific function performed by the mobile terminal 100 (e.g., a call signal reception sound, a message reception sound, etc.). The

audio output unit

103 may include a speaker, a buzzer, and the like.

The a/

V input unit

104 is used to receive audio or video signals. The a/

V input Unit

104 may include a Graphics Processing Unit (GPU) 1041 and a

microphone

1042, the

Graphics processor

1041 Processing image data of still pictures or video obtained by an image capturing device (e.g., a camera) in a video capturing mode or an image capturing mode. The processed image frames may be displayed on the

display unit

106. The image frames processed by the

graphic processor

1041 may be stored in the memory 109 (or other storage medium) or transmitted via the

radio frequency unit

101 or the

WiFi module

102. The

microphone

1042 may receive sounds (audio data) via the

microphone

1042 in a phone call mode, a recording mode, a voice recognition mode, or the like, and may be capable of processing such sounds into audio data. The processed audio (voice) data may be converted into a format output transmittable to a mobile communication base station via the

radio frequency unit

101 in case of a phone call mode. The

microphone

1042 may implement various types of noise cancellation (or suppression) algorithms to cancel (or suppress) noise or interference generated in the course of receiving and transmitting audio signals.

The

mobile terminal

100 also includes at least one

sensor

105, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor includes an ambient light sensor that can adjust the brightness of the

display panel

1061 according to the brightness of ambient light, and a proximity sensor that can turn off the

display panel

1061 and/or a backlight when the

mobile terminal

100 is moved to the ear. As one of the motion sensors, the accelerometer sensor can detect the magnitude of acceleration in each direction (generally, three axes), can detect the magnitude and direction of gravity when stationary, and can be used for applications of recognizing the posture of a mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer and tapping), and the like; as for other sensors such as a fingerprint sensor, a pressure sensor, an iris sensor, a molecular sensor, a gyroscope, a barometer, a hygrometer, a thermometer, and an infrared sensor, which can be configured on the mobile phone, further description is omitted here.

The

display unit

106 is used to display information input by a user or information provided to the user. The

Display unit

106 may include a

Display panel

1061, and the

Display panel

1061 may be configured in the form of a Liquid Crystal Display (LCD), an Organic Light-Emitting Diode (OLED), or the like.

The

user input unit

107 may be used to receive input numeric or character information and generate key signal inputs related to user settings and function control of the mobile terminal. Specifically, the

user input unit

107 may include a

touch panel

1071 and

other input devices

1072. The

touch panel

1071, also referred to as a touch screen, may collect a touch operation performed by a user on or near the touch panel 1071 (e.g., an operation performed by the user on or near the

touch panel

1071 using a finger, a stylus, or any other suitable object or accessory), and drive a corresponding connection device according to a predetermined program. The

touch panel

1071 may include two parts of a touch detection device and a touch controller. The touch detection device detects the touch direction of a user, detects a signal brought by touch operation and transmits the signal to the touch controller; the touch controller receives touch information from the touch sensing device, converts the touch information into touch point coordinates, sends the touch point coordinates to the

processor

110, and can receive and execute commands sent by the

processor

110. In addition, the

touch panel

1071 may be implemented in various types, such as a resistive type, a capacitive type, an infrared ray, and a surface acoustic wave. In addition to the

touch panel

1071, the

user input unit

107 may include

other input devices

1072. In particular,

other input devices

1072 may include, but are not limited to, one or more of a physical keyboard, function keys (e.g., volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like, and are not limited to these specific examples.

Further, the

touch panel

1071 may cover the

display panel

1061, and when the

touch panel

1071 detects a touch operation thereon or nearby, the

touch panel

1071 transmits the touch operation to the

processor

110 to determine the type of the touch event, and then the

processor

110 provides a corresponding visual output on the

display panel

1061 according to the type of the touch event. Although the

touch panel

1071 and the

display panel

1061 are shown in fig. 1 as two separate components to implement the input and output functions of the mobile terminal, in some embodiments, the

touch panel

1071 and the

display panel

1061 may be integrated to implement the input and output functions of the mobile terminal, and is not limited herein.

The

interface unit

108 serves as an interface through which at least one external device is connected to the

mobile terminal

100. For example, the external device may include a wired or wireless headset port, an external power supply (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device having an identification module, an audio input/output (I/O) port, a video I/O port, an earphone port, and the like. The

interface unit

108 may be used to receive input (e.g., data information, power, etc.) from external devices and transmit the received input to one or more elements within the

mobile terminal

100 or may be used to transmit data between the

mobile terminal

100 and external devices.

The

memory

109 may be used to store software programs as well as various data. The

memory

109 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; the storage data area may store data (such as audio data, a phonebook, etc.) created according to the use of the cellular phone, and the like. Further, the

memory

109 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device.

The

processor

110 is a control center of the mobile terminal, connects various parts of the entire mobile terminal using various interfaces and lines, and performs various functions of the mobile terminal and processes data by operating or executing software programs and/or modules stored in the

memory

109 and calling data stored in the

memory

109, thereby performing overall monitoring of the mobile terminal.

Processor

110 may include one or more processing units; preferably, the

processor

110 may integrate an application processor, which mainly handles operating systems, user interfaces, application programs, etc., and a modem processor, which mainly handles wireless communications. It will be appreciated that the modem processor described above may not be integrated into the

processor

110.

The

mobile terminal

100 may further include a power supply 111 (e.g., a battery) for supplying power to various components, and preferably, the

power supply

111 may be logically connected to the

processor

110 via a power management system, so as to manage charging, discharging, and power consumption management functions via the power management system.

Although not shown in fig. 1, the

mobile terminal

100 may further include a bluetooth module or the like, which is not described in detail herein.

In order to facilitate understanding of the embodiments of the present invention, a communication network system on which the mobile terminal of the present invention is based is described below.

Referring to fig. 2, fig. 2 is an architecture diagram of a communication Network system according to an embodiment of the present invention, where the communication Network system is an LTE system of a universal mobile telecommunications technology, and the LTE system includes a UE (User Equipment) 201, an E-UTRAN (Evolved UMTS Terrestrial Radio Access Network) 202, an EPC (Evolved Packet Core) 203, and an

IP service

204 of an operator, which are in communication connection in sequence.

Specifically, the UE201 may be the terminal 100 described above, and is not described herein again.

The E-UTRAN202 includes eNodeB2021 and

other eNodeBs

2022, among others. Among them, the eNodeB2021 may be connected with other eNodeB2022 through backhaul (e.g., X2 interface), the eNodeB2021 is connected to the EPC203, and the eNodeB2021 may provide the UE201 access to the

EPC

203.

The EPC203 may include an MME (Mobility Management Entity) 2031, an HSS (Home Subscriber Server) 2032,

other MMEs

2033, an SGW (Serving gateway) 2034, a PGW (PDN gateway) 2035, and a PCRF (Policy and Charging Rules Function) 2036, and the like. The MME2031 is a control node that handles signaling between the UE201 and the EPC203, and provides bearer and connection management. HSS2032 is used to provide registers to manage functions such as home location register (not shown) and holds subscriber specific information about service characteristics, data rates, etc. All user data may be sent through SGW2034, PGW2035 may provide IP address assignment for UE201 and other functions, and PCRF2036 is a policy and charging control policy decision point for traffic data flow and IP bearer resources, which selects and provides available policy and charging control decisions for a policy and charging enforcement function (not shown).

The

IP services

204 may include the internet, intranets, IMS (IP Multimedia Subsystem), or other IP services, among others.

Although the LTE system is described as an example, it should be understood by those skilled in the art that the present invention is not limited to the LTE system, but may also be applied to other wireless communication systems, such as GSM, CDMA2000, WCDMA, TD-SCDMA, and future new network systems.

Based on the above mobile terminal hardware structure and communication network system, the present invention provides various embodiments of the method.

Fig. 3 is a flowchart of an image processing method according to an embodiment of the present invention. In this embodiment, the image processing method may be applied to the mobile terminal shown in fig. 1 or fig. 2, and it is understood that the image processing method may also be applied to electronic devices such as a tablet, an electronic book, and the like. The image processing method comprises the following steps:

step S310, determining a background image;

step S320, acquiring a continuously acquired video image;

and S330, synthesizing the acquired video image and the background image in real time to obtain a target video.

Through the embodiment, after the image acquisition application program is started, the background image is determined in the unified graphical user interface provided by the application program, the background image is displayed through the image preview interface of the application program, the video image continuously acquired by the image acquisition device is controlled by receiving the video acquisition instruction in the current graphical user interface, then the target video is obtained by real-time synthesis according to the acquired video image and the background image, and the real-time synthesized target video is displayed on the image preview interface, so that a user can know the currently synthesized target video in real time.

The above steps will be specifically described with reference to specific examples.

In step S310, a background image is determined.

Specifically, the background image includes at least one of: at least one background static picture; and a background video image. The background still picture refers to a still picture, which may be one or more, and is not particularly limited. A background video image refers to a segment of a video file. The background image may be an image that has been shot previously, or may be an image that is temporarily captured in the current state.

In an alternative embodiment, step S310 may be performed by:

step S3101, providing an image preview interface and a first interactive control through a graphical user interface;

step S3102, determining the background image in response to the touch operation for the first interactive control.

Specifically, the method and the device can realize the synthesis with the background image while acquiring the video image through a single application program, and avoid the complicated flow caused by the separate operation of the steps of acquiring the video and synthesizing the background image in the prior art. In step S3101, the gui is an exchange interface provided by the mobile terminal, and the image preview interface is an interface provided by the present application for displaying the synthesized video image in real time, for example, if the image processing function provided by the present application is provided by a camera application of the mobile terminal, the image preview interface is a preview interface after the camera application is started, and the content displayed in the preview interface is an image acquired by an image acquisition device of the mobile terminal. It should be noted that one of the important points of the present application lies in the selection of the background image and the subsequent acquisition of the video image, which can be displayed in real time through the image preview interface.

The first interactive control can be a specific position of the setting and image preview interface, so that a user can set a background image on the current interactive interface without switching to other application programs.

In an alternative embodiment, step S3102 may include the steps of:

step S31021, displaying a material selection interface including an existing picture or video in the material selection interface in response to the touch operation of the first interactive control;

step S31022, responding to the selection operation on the material selection interface, and according to the target picture or video corresponding to the selection operation;

step S31023, the target picture or video is determined to be a background image.

Specifically, through the above embodiment, the user can select the background image from the pictures or videos already saved in the mobile terminal. In this embodiment, the touch operation of the first interactive control is a click operation, a slide operation, or other operations that can trigger the first interactive control. The material selection interface can be overlapped in the current image preview interface in a pop-up window mode, and thumbnails generated according to pictures or videos stored in the mobile terminal are displayed in the material selection interface. Through the embodiment, the user can select the image which is required to be used as the background from the saved pictures or videos before the video image is acquired.

In an alternative embodiment, step S3102 may include the steps of:

step S31024, in response to the touch operation for the first interactive control, acquiring an image through an image acquisition device, wherein the image may be a picture or a video;

in step S31025, the captured image is determined as the background image.

Specifically, the touch operation for the first interactive control is the same as the content described above, and therefore, the description thereof is omitted. Through touch-control first interactive control, can control the camera and shoot or record the video to current environment. By the embodiment, the user can acquire the background image for subsequent synthesis without switching to other photographing programs.

Step S3103, displaying the determined background image through the image preview interface.

Specifically, before shooting a video, a user may select a picture or a video from the mobile terminal, and then display the selected picture or video in the image preview interface. At this time, the background image and the current camera preview image are displayed in an overlapping manner. If the video is selected, the video and the preview picture of the current camera are subjected to overlapping and circulating playing.

In step S320, continuously captured video images are acquired.

Specifically, after a background image is displayed in the current image preview interface, the image capture device (e.g., a camera) is controlled to continuously capture video images by receiving a video image capture instruction. In an optional embodiment, other areas in the current graphical user interface providing the image preview interface provide video capture controls, and the video capture controls are triggered to send video capture instructions to the camera. In other embodiments, the video capture control may also be disposed in the image preview interface area.

Through the implementation mode, a user can set the background image and the video acquisition to be realized in the same interactive interface, and the video acquisition is not required to be carried out firstly in the existing mode, and then the acquired video and the background image are synthesized.

In step S330, a real-time synthesis is performed according to the captured video image and the background image to obtain a target video.

Specifically, the video recording process comprises video decoding, the decoded video is sent to a video superposition module, the video collected by a camera and a background image are synthesized by the video superposition module, the synthesized data are respectively drawn to a screen area to be displayed in real time, and meanwhile, the synthesized data are sent to an encoder to generate a video file. In an alternative embodiment, step S330 may include the following steps:

step S3301, displaying the background image through the image preview interface;

s3302, synthesizing the video image collected in real time with the background image to obtain target video data;

and step S3303, drawing the target video data to the image preview interface to display the target video in real time.

Specifically, in a specific implementation, the background image and the acquired video image are respectively drawn into a screen in a process of synthesizing, a current synthesized video effect is displayed through an image preview interface, and simultaneously, synthesized video data is sent to an encoder to generate a target video file in real time.

Through the implementation mode, the user can record the video and display the video effect synthesized with the self-defined background image in an interactive interface or a flow, and simultaneously, the synthesized video can be generated when the video recording is finished.

Further, the method provided by the present application further includes:

and responding to the selection operation of the superposition mode, and determining the superposition mode of the background image and the video image. The superposition mode refers to a pixel superposition mode of a background image and a video screen image frame. In this embodiment, in step S3302, the captured video image and the background image are synthesized in real time according to the overlay method to obtain the target video.

Wherein, the superposition mode comprises at least the following modes: transparency stacking, bright portion stacking, dark portion stacking, and top-bottom stacking. Specifically, the transparency overlap is obtained by performing weighted average on the video image and the background image according to the transparency selected by the user to obtain an overlapped image. And (3) bright part superposition: and comparing the pixels of the video image and the background image one by one, reserving brighter pixels, and forming a superposed image by the pixels. Dark part superposition: and comparing pixels of the video image and the background image one by one, reserving darker pixels, and forming a superposed image by the pixels. And (3) stacking up and down: the upper half or lower half of the video image and the lower half or upper half of the background image are used to form the overlay image, as selected by the user.

Through the embodiment, the user determines the composite effect in advance before the mobile terminal collects the video image, and then the video image and the background image are directly composited according to the composite mode determined by the user in the video collecting process and are displayed through the image preview interface.

Fig. 4 is a schematic structural component diagram of the

mobile terminal

100 according to an embodiment of the present application, where the

mobile terminal

100 includes: a

touch panel

1071; a

processor

110; a

memory

109 connected to the

processor

110, wherein the

memory

109 contains a control instruction, and when the

processor

110 reads the control instruction, the

mobile terminal

100 is controlled to implement the following steps:

determining a background image;

acquiring a continuously acquired video image;

and synthesizing the acquired video image and the background image in real time to obtain a target video.

Optionally, the background image comprises at least one of:

at least one background static picture; and

a background video image.

Optionally, the step of determining a background image includes:

providing an image preview interface and a first interaction control through a graphical user interface;

responding to touch operation aiming at the first interactive control, and determining the background image;

and displaying the determined background image through the image preview interface.

Optionally, the step of determining the background image in response to the touch operation of the first interaction control includes:

the response is to the touch operation of the first interactive control, and a material selection interface is displayed, wherein the material selection interface comprises existing pictures or videos;

responding to the selection operation aiming at the material selection interface, and according to the target picture or video corresponding to the selection operation;

and determining the target picture or video as a background image.

Optionally, the step of determining the background image in response to the touch operation of the first interaction control includes:

responding to touch operation aiming at the first interactive control, and collecting an image through image collecting equipment, wherein the image can be a picture or a video;

determining the acquired image as the background image.

Optionally, the step of synthesizing the acquired video image and the background image in real time to obtain the target video includes:

displaying the background image through the image preview interface;

synthesizing the video image acquired in real time with the background image to obtain target video data;

and drawing the target video data to the image preview interface to display the target video in real time.

Optionally, the method further comprises:

responding to the selection operation of the superposition mode, and determining the superposition mode of the background image and the video image;

and the step of synthesizing the acquired video image and the background image in real time to obtain the target video comprises the step of synthesizing the acquired video image and the background image in real time to obtain the target video according to the superposition mode.

Optionally, the superimposing manner includes at least the following manner: transparency stacking, bright portion stacking, dark portion stacking, and top-bottom stacking.

The mobile terminal in the embodiment of the invention determines the background image in the unified graphical user interface provided by the application program after the image acquisition application program is started, displays the background image through the image preview interface of the application program, controls the image acquisition device to continuously acquire the video image by receiving the video acquisition instruction in the current graphical user interface, then synthesizes the acquired video image and the background image in real time to obtain the target video, and displays the target video synthesized in real time on the image preview interface, so that a user can know the currently synthesized target video in real time.

An embodiment of the present invention further provides a computer storage medium, where an executable program is stored in the computer storage medium, and when the executable program is executed, the following steps are implemented: