CN115514883B - Cross-equipment collaborative shooting method, related device and system - Google Patents

As shown in fig. 1B, devices a and B may perform video call through an instant messaging server, for example

And providing video call. Specifically, the device a may collect an image through a camera, and then send the image to the device B through an instant messaging server. The device B can directly display the image sent by the device A on the video call interface, and also can display the self adoption of the device BImages of the collection, thereby enabling collaborative photographing across devices.

The system provided in the embodiments of the present application is first described below. Fig. 2A illustrates the structure of

10.

As shown, the

10 includes:

100,

200. Wherein the number of

200 may be one or more, one

200 is illustrated in fig. 2A.

The

100 and the

200 are each electronic devices configured with cameras. The number of cameras that the

100 and the

200 have is not limited in the embodiments of the present application. For example, the

200 may be configured with five cameras (2 front cameras and 3 rear cameras).

Electronic devices include, but are not limited to, smartphones, tablet computers, personal digital assistants (personal digital assistant, PDAs), wearable electronic devices with wireless communication capabilities (e.g., smartwatches, smart glasses), augmented reality (augmented reality, AR) devices, virtual Reality (VR) devices, and the like. Exemplary embodiments of electronic devices include, but are not limited to, piggybacking

Linux, or other operating system. The electronic device may also be other portable electronic devices such as a Laptop computer (Laptop) or the like. It should also be appreciated that in other embodiments, the electronic device described above may be a desktop computer or the like instead of a portable electronic device.

A communication connection is established between the

100 and the

200, which may be a wired connection, a wireless connection.

In some embodiments, the wireless connection may be a high fidelity wireless communication (Wi-Fi) connection, a bluetooth connection, an infrared connection, an NFC connection, a ZigBee connection, or the like. The

100 may directly transmit a control command for adjusting the photographing effect to the

200 through the close-range connection. The

200 may respond to the control command issued by the

100 and transmit the adjusted image back to the

100. Thereafter, the

100 may display the image returned from the

200. In addition, the

100 may also complete the tasks of recording, photographing, and forwarding by using the images. Here, specific implementations of the

100 sending a control command for adjusting a photographing effect to the

200, the

200 adjusting an image according to the control command, and the like may refer to the detailed description of the subsequent method embodiments, which are not described herein in detail.

Optionally, the

10 shown in fig. 2A may further include a server 300, where the master device and the slave device may log into the same user account (e.g., the "home" account) and then remotely connect through the server 300 (e.g., the "home" multi-device collaborative photographing server). The server 300 may be used for data transmission of the

100 and the

200. I.e. the

100 may send control commands to the

200 via the server 300. Likewise, the

200 may transmit an image to the

100 through the server 300.

Fig. 2B is a schematic structural diagram of an electronic device 400 according to an embodiment of the present application. Electronic device 400 may be

100 or

200 in

10 shown in fig. 2A.

The electronic device 400 may include:

110,

120, internal memory 121, universal serial bus (universal serial bus, USB) interface 130,

140,

141,

142, antenna 1,

2,

150, wireless communication module 160,

170, speaker 170A, receiver 170B,

170C, headset interface 170D, sensor module 180, keys 190, motor 191, indicator 192, camera 193, display 194, and subscriber identity module (subscriber identification module, SIM)

195, etc. The sensor module 180 may include a pressure sensor 180A, a gyro sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, and the like.

It should be understood that the illustrated structure of the embodiment of the present invention does not constitute a specific limitation on the

100. In other embodiments of the present application,

100 may include more or fewer components than shown, or certain components may be combined, or certain components may be split, or different arrangements of components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

The

110 may include one or more processing units, such as: the

110 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a controller, a memory, a video codec, a digital signal processor (digital signal processor, DSP), a baseband processor, and/or a neural network processor (neural-network processing unit, NPU), etc. Wherein the different processing units may be separate devices or may be integrated in one or more processors.

A memory may also be provided in the

110 for storing commands and data. In some embodiments, the memory in the

110 is a cache memory. The memory may hold commands or data that the

110 has just used or recycled. If the

110 needs to reuse the command or data, it can be called directly from the memory. Repeated accesses are avoided and the latency of the

110 is reduced, thereby improving the efficiency of the system.

The wireless communication function of the electronic device 400 may be implemented by the antenna 1, the

2, the

150, the wireless communication module 160, a modem processor, a baseband processor, and the like.

The

1 and 2 are used for transmitting and detecting electromagnetic wave signals. Each antenna in electronic device 400 may be used to cover a single or multiple communication bands. Different antennas may also be multiplexed to improve the utilization of the antennas. For example: the antenna 1 may be multiplexed into a diversity antenna of a wireless local area network. In other embodiments, the antenna may be used in conjunction with a tuning switch. The

150 may provide a solution for wireless communication including 2G/3G/4G/5G, etc., applied to the electronic device 400. The

150 may include at least one filter, switch, power amplifier, low noise amplifier (low noise amplifier, LNA), etc. The

150 may detect electromagnetic waves by the antenna 1, perform processes such as filtering, amplifying, and the like on the detected electromagnetic waves, and transmit the processed electromagnetic waves to the modem processor for demodulation. The

150 can amplify the signal modulated by the modem processor, and convert the signal into electromagnetic waves through the antenna 1 to radiate. In some embodiments, at least some of the functional modules of the

150 may be disposed in the

110. In some embodiments, at least some of the functional modules of the

150 may be provided in the same device as at least some of the modules of the

110.

The modem processor may include a modulator and a demodulator. The modulator is used for modulating the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used for demodulating the detected electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low frequency baseband signal to the baseband processor for processing. The low frequency baseband signal is processed by the baseband processor and then transferred to the application processor. The application processor outputs sound signals through an audio output device (not limited to the speaker 170A, the receiver 170B, etc.), or displays images or video through the display screen 194. In some embodiments, the modem processor may be a stand-alone device. In other embodiments, the modem processor may be provided in the same device as the

150 or other functional module, independent of the

110.

The wireless communication module 160 may provide solutions for wireless communication including wireless local area network (wireless local area networks, WLAN) (e.g., wireless fidelity (wireless fidelity, wi-Fi) network), bluetooth (BT), global navigation satellite system (global navigation satellite system, GNSS), frequency modulation (frequency modulation, FM), near field wireless communication technology (near field communication, NFC), infrared technology (IR), etc., as applied to the electronic device 400. The wireless communication module 160 may be one or more devices that integrate at least one communication processing module. The wireless communication module 160 detects an electromagnetic wave via the

2, modulates the electromagnetic wave signal, filters the electromagnetic wave signal, and transmits the processed signal to the

110. The wireless communication module 160 may also detect a signal to be transmitted from the

110, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation via the

2. Illustratively, the wireless communication module 160 may include a Bluetooth module, a Wi-Fi module, or the like.

In some embodiments, antenna 1 and

150 of electronic device 400 are coupled, and

2 and wireless communication module 160 are coupled, such that electronic device 400 may communicate with a network and other devices via wireless communication techniques. The wireless communication techniques may include the Global System for Mobile communications (global system for mobile communications, GSM), general packet radio service (general packet radio service, GPRS), code division multiple access (code division multiple access, CDMA), wideband code division multiple access (wideband code division multiple access, WCDMA), time division code division multiple access (time-division code division multiple access, TD-SCDMA), long term evolution (long term evolution, LTE), 5G and subsequent standard protocols, BT, GNSS, WLAN, NFC, FM, and/or IR techniques, among others. The GNSS may include a global satellite positioning system (global positioning system, GPS), a global navigation satellite system (global navigation satellite system, GLONASS), a beidou satellite navigation system (beidou navigation satellite system, BDS), a quasi zenith satellite system (quasi-zenith satellite system, QZSS) and/or a satellite based augmentation system (satellite based augmentation systems, SBAS).

The electronic device 400 implements display functions through a GPU, a display screen 194, an application processor, and the like. The GPU is a microprocessor for image processing, and is connected to the display 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering.

110 may include one or more GPUs that execute program commands to generate or change display information.

When the electronic device 400 shown in figure 2B is the

100 in figure 2A,

the

150 and the wireless communication module 160 may be used to provide communication services for the

100. Specifically, in embodiments of the present application, the

100 may establish a communication connection with other electronic devices (i.e., slave devices 200) having cameras 193 through the

150 or the wireless communication module 160. Further, through the above connection, the

100 may transmit a control command to the

200 and receive an image transmitted back from the

200.

ISP, camera 193, video codec, GPU, display 194, application processor, etc. provide the

100 with the ability to capture and display images. When the

100 turns on the camera 193, the

100 may obtain optical imaging collected by the camera 193 and convert the optical signal into an electrical signal through the ISP. In the experiment for controlling the shooting effect in the embodiment of the application, the ISP can also adjust shooting parameters such as exposure, color temperature and the like of a shooting scene and optimize image processing parameters for noise, brightness and skin color of an image.

In cross-device collaborative photography, a video codec may be used for digital video compression or decompression. The

100 may encode the photographing file photographed in cooperation across devices into video files of various formats through a video codec.

With a GPU, a display screen 194, and an application processor, etc., the

100 may implement display functionality. Specifically, the electronic device may display an image or the like acquired by the camera 193 through the display screen 194. In addition, the image received by the

100 and transmitted from the

200 may be displayed by the display 194 or the like. In some embodiments, the display 194 may also display only images sent from the

200. At the same time, the

100 may respond to user operations acting on user interface controls via the touch sensor 180K, i.e., a "touch panel".

When the electronic device 400 shown in figure 2B is the

200 in figure 2A,

the

150 and the wireless communication module 160 may be used to provide communication services for the

200. Specifically, the

200 may establish a communication connection with the

100 through the

150 or the wireless communication module 160. Through the above connection, the

200 receives a control command for controlling a photographing effect transmitted from the

100, and may photograph an image in response to the control command, and transmit the image acquired and processed by the camera 193 to the

100.

As with the

100, the ISP, the camera 193, and the video codec may provide the function of capturing and transmitting images for the

200. In cross-device collaborative photography, a video codec may be used to compress or decompress digital video when the

200 sends images to the

100. The

200 may encode the photographing file photographed in cooperation with each other across devices into an image stream through a video codec and then transmit it to the

100.

In some embodiments, the

200 may also display images captured by its own camera 193. At this time, display functions may be implemented for the

200 by the GPU, the display screen 194, and the application processor, etc.

While

200 may respond to user operations acting on user interface controls via touch sensor 180K.

The software systems of the

100 and the

200 may each employ a layered architecture, an event driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. Taking an Android system with a layered architecture as an example, the embodiment of the present invention illustrates software structures of the

100 and the

200. Of course, in other operating systems (e.g., hong-mong system, linux system, etc.), the solution of the present application can be implemented as long as the functions implemented by the respective functional modules are similar to those of the embodiments of the present application.

Fig. 3 is a software configuration block diagram of the

100 according to the embodiment of the present invention.

As shown in fig. 3, the software architecture block diagram of the

100 may include an application layer, a framework layer, a service layer, and a hardware abstraction layer (hardware abstraction layer, HAL). The framework layer may also include, among other things, a device virtualization suite (device virtualkit, DVKit) and a device virtualization platform (distributedmobile sensing development platform, DMSDP).

DVkit is a software development kit (software development kit, SDK). The DVkit may provide a capability interface to the application layer. Through the above interface, the application layer may invoke services and capabilities provided in the DVkit, e.g., discovery of slave devices, etc. DMSDP is a framework layer service. When the DVkit initiates the connection of the slave device, the DVkit may pull up the DMSDP service, and then the DMSDP may implement a control session, a transmission of a data session, and the like during the connection of the slave device. The application layer may include a series of application packages. Examples may include cameras, gallery, calendar, conversation, map, navigation, WLAN, bluetooth, music, video, short message, etc. applications. In the embodiment of the application, the application layer includes various applications using a camera, such as a camera application, a live broadcast application, a video call application, and the like. The video call application refers to an application having both voice call and video call, such as an instant messaging application

(not shown in fig. 3), and so forth. The camera applications may include native camera applications, and third party camera applications. The application layer may request the frame layer for shooting capabilities using the camera.

Among other things, the camera kit (camera kit) may include a mode management module that may be used to adjust the shooting mode used when the

100 runs various types of applications that use cameras. The shooting mode may include, but is not limited to, preview mode, photo mode, video mode, cross-device mode, and so forth. Wherein, after the

100 enters the cross-device mode, a device virtualization kit (DVKit) may be used to discover the

200. These shooting modes can be realized by calling a Camera interface (Camera API).

Among them, camera management (camera manager) may be used to manage photographing capabilities of the

100 and photographing capabilities of the

200 connected to the

100.

Table 1 exemplarily shows the respective photographing capabilities of the

100 and the

200.

As shown in table 1, camera management (camera manager) may record the numbers of the cameras of the

100 and the cameras of the

200. For example, three cameras of the

100 may be numbered 1, 2, 3, respectively. The three cameras of the

200 may be numbered 1001, 1002, 1003, respectively. The number is used to uniquely identify the camera. In particular, the numbering of the cameras of the

200 may be done by the virtual camera HAL of the

100. Specific numbering rules may refer to the subsequent description of the virtual HAL and will not be described in detail herein.

The camera device (camera device) may also cooperate with a device virtualization suite (DVKit) and a device virtualization platform (DMSDP) to establish a communication connection between the

100 and the

200, and bind the

200 and the

100.

Specifically, a device virtualization suite (DVKit) may be used to discover the

200, and a device virtualization platform (DMSDP) may be used to establish a session channel with the discovered

200. The DMSDP may include a control session (control session) and a data session (data session). Wherein the control session is used to transmit control commands between the

100 and the slave device 200 (e.g., a request to use a camera of the

200, a photographing command, a video recording command, a control command to adjust a photographing effect, etc.). The data session is used to transfer streams returned from the

200, such as preview streams, photo streams, video streams, and so forth.

The camel providermanager may be used to obtain virtual camera HAL information including the shooting capabilities of the

200. A detailed description of the shooting capability of the

200 may refer to table 1.

Device co-management may be used to control the latency of the respective pictures of the

100 and the

200. When the

100 and the

200 perform cross-device collaborative shooting, since a certain time is required for the

200 to transmit the shot image to the

100, preview images of both parties displayed by the

100 may not be the same time and have a time delay. The device collaborative management module may copy a portion of frames of the frames acquired by the

100 by adding a buffer (buffer), so that preview frames of both sides are acquired at a time point with little difference, and thus, a time delay generated by the preview frames of both sides may be controlled within a visual perception range of a user, without affecting user experience.

The dynamic pipeline may be used to generate a pending command queue in response to a stream creation command issued by a camera device (camera device). In particular, the dynamic pipeline may generate one or more pending command queues depending on the type of stream creation command, the device being acted upon. The type of stream creation command may include, for example, a preview command, a video command, a photographing command, and the like. The devices on which the stream creation command acts may include the

100, the

200, and may even be further refined to the specific cameras of the devices. The detailed workflow of the dynamic pipeline will be described in detail in the following method embodiments, which are not described in detail herein.

The dynamic pipeline may also be used to distribute the commands acting on the

200 in the pending command queue to the stream processing module, and distribute the commands acting on the

200 to the local camera HAL module of the HAL layer, which will be described in detail with reference to fig. 13.

The lifecycle management module may be used to monitor the entire lifecycle of the flow. When a camera device (camera device) transmits a command to create a stream to a stream process, lifecycle management can record information of the stream, such as a time stamp requesting creation of the stream, whether the

200 responds to create the stream, and the like. When the

100 turns off the camera or ends running the application, the lifecycle of the corresponding stream stops and the lifecycle management module can record the end time of the stream.

The post-processing module is used to process the image stream returned from the

200. The post-processing module may include a smart split and multi-stream output module. The smart cut may be used to expand the stream returned from the

200 into a stream consistent with the type and number of streams requested by the camera device (camera device). For example, the camera device (camera device) requests a 1080P preview stream and a 720P analysis stream. The control command requesting two streams is multiplexed into a control command requesting one 1080P preview stream by multiplexing control. By executing the control command requesting a 1080P preview stream,

200 may transmit a 1080P preview stream back to

100. When the stream processing receives the 1080P preview stream, the stream processing may restore the 1080P preview stream to a 1080P preview stream and a 720P analysis stream through the intelligent branching module.

The frame synchronization module can be used for performing frame synchronization processing on the image frames during photographing. Specifically, cross-device transmissions may cause a delay in the instruction sent by the

100 when reaching the

200. I.e.

200 will receive this same instruction later than

100. Therefore, the execution result obtained by the

200 when executing the above-described instruction may also be different from the result expected by the

100. For example, the

100 may issue a control command at a first time, the

200 may not return the image stream in response to the control command at a second time, and the

100 may wish to receive the image stream from the

200 at the first time. Thus, frame synchronization may shift the result at the second time back from the

200 result forward to get a result (user expected result) closer to the first time, thereby reducing the impact of network latency.

The local camera HAL may include a camera session (camera session) and a camera provider module. A camera session (camera session) may be used for the

100 to issue control commands to the hardware. The camera provider module may be used to manage the shooting capabilities of the camera of the

100. The shooting capability of the

100 can be referred to table 1 above. Here, the camera provider may manage only the photographing capability of the local camera of the

100.

The virtual camera HAL also includes a camera session (camera session) and a camera provider module. Among them, a camera session (camera session) may also be used to register the

200, which establishes a communication connection with the

100, to the home, to feed back the connection state of the

200 to the DVkit, and to transmit a control command issued by the

100 to the

200. The camera provider module is responsible for management of the shooting capability of the

200. Also, the shooting capability of the

200 may refer to table 1, and will not be described here.

In addition, the camera virtual HAL also provides a function of numbering the cameras of the registered

200. When the virtual camera HAL of the master device acquires the shooting capability of the slave device, the virtual camera HAL may acquire the number of cameras that the slave device has and establish an ID for each camera. The ID may be used by the

100 to distinguish between multiple cameras of the

200. The virtual camera HAL may use a different numbering method from the

100 itself in numbering the cameras. For example, when the cameras of the

100 itself are numbered from 1, the cameras of the

200 may be numbered from 1000.

It will be appreciated that the configuration of the master device to 100 illustrated in fig. 3 does not constitute a specific limitation on the

200. In other embodiments of the present application, the

200 may include more or fewer modules than shown, or may combine certain modules, or split certain modules, or may be arranged in different modules.

Fig. 4 illustrates a system frame diagram of the

200.

As shown in FIG. 4, the

200 may include an application layer, a framework layer, a service layer, and a Hardware Abstraction Layer (HAL).

Among other things, pipe control may be used to establish communication connections (including establishing control sessions and data sessions), transfer control commands, and image streaming with the

100.

The multi-stream adaptation may be used for the camera proxy service to return stream data according to configuration information of the streams sent by the

100. With reference to the description of the foregoing embodiments, the

100 may configure a desired stream for a camera device session in response to a request to create the stream. Performing the above configuration procedure may generate corresponding flow configuration information. The camera proxy service may generate a corresponding stream creation command according to the configuration information. Thus, in response to the above-described control command to create a stream, the underlying service of the

200 may create a stream that matches the above-described command.

Where camera management (camera manager), camera device (camera device), camera service (camera service), cameraDeviceClient, camera3Device, cameraProviderManager have the same function as corresponding respective modules in the

100, reference is made to the description of fig. 3 above.

The local camera HAL of the

200 may refer to the local camera HAL of the

100. The embodiments of the present application are not described herein. The camera HAL layer of the

200 may also include a camera session (camera session) and a camera provider module. A camera session (camera session) may be used for communication between control commands and hardware. The camera provider module may be used to manage shooting capabilities of the

200. The photographing capability of the

200 may refer to the above table 1, and likewise, the camera provider of the

200 may manage only the photographing capability of the local camera of the

200.

It will be appreciated that the configuration of the

200 illustrated in fig. 4 does not constitute a specific limitation of the

200. In other embodiments of the present application, the

200 may include more or fewer modules than shown, or may combine certain modules, or split certain modules, or may be arranged in different modules.

Based on the foregoing software and hardware structures of the

10, the

100, and the

200, the cross-device collaborative shooting method provided in the embodiments of the present application is described in detail below.

In a live broadcast scene, the

100 may be connected with the

200, the

100 and the

200 may be in different angles to shoot the same object or different objects, the

200 may send the shot image to the

100, and the

100 may upload the images of both parties to a live broadcast server and distribute the images to more users for viewing by the live broadcast server. In this process, the

100 may control the photographing effect of the

200, for example, parameters such as a focal length of a camera of the

200 or a filter used may be adjusted by the

100.

By using the cross-device collaborative shooting method provided by the embodiment of the application in the live scene, a host who initiates live broadcast can conveniently control the shooting effect of the

200 on the

100, and a user who watches live broadcast can see a plurality of images displayed at different angles for the same object.

After the

100 starts the camera application, it may connect with the

200 and take images at different angles, and the

200 may send the taken images to the

100. The

100 may control the photographing effect of the

200, may display photographed images of both sides at the same time, and may perform processes such as preview, photographing, video recording, and the like on the photographed images. The camera application may be a native camera application or a third party camera application.

Thus, functions of cross-equipment double-view video, multi-view video and the like can be realized, more and more free visual angles can be provided for users, and the users can conveniently control the shooting effect of the

200 on the

100, so that the shooting interestingness is increased.

The

100 may be a cell phone and the

200 may be a large screen television. The mobile phone can carry out video call with other devices, in the process, the mobile phone can be connected with the large-screen television and control the camera of the large-screen television to shoot, the large-screen television can send the shot image to the mobile phone, and then the mobile phone sends the image to the other end device of the video call. Therefore, the mobile phone can realize video call through the large-screen television, a user does not need to hold the mobile phone to shoot images at a specific angle, and more convenient video call experience can be provided for the user.

As shown in fig. 5, the live scene may include a

100, a

200, an object a, and an object B.

The

100 and the

200 establish a communication connection, and the

100 and the

200 may be in different positions or angles. The

100 photographs the object a, and the

200 photographs the object B. Then, the

200 may display the photographed image and transmit the image to the

100 after processing. The

100 may simultaneously display the image transmitted from the

200 and the image obtained by photographing the object a itself.

During the live broadcast, the

100 may also upload the two images displayed to the live broadcast server. Further, the server may distribute the two images to devices of other users entering the live room.

Based on the live scenario described above, some User Interfaces (UIs) on the

100 and the

200 provided in the embodiments of the present application are described below. Fig. 6A-6D, 7A-7B, 8A-8B, 9A-9D, and 10A-10C illustrate some user interfaces implemented on the

100 and the

200 in a live scenario.

Fig. 6A-6C illustrate one manner in which the

100 and the

200 establish a communication connection. Fig. 6A-6C are user interfaces implemented on the

100, and fig. 6D is a user interface implemented on the

200.

FIG. 6A illustrates an

60 on the

100 for exposing installed applications. The

60 is displayed with: status bars, calendar indicators, weather indicators, trays with commonly used application icons, navigation bars,

601 for live class applications,

602 for camera applications, icons for other applications, and so forth. Wherein the status bar may include: one or more signal strength indicators of mobile communication signals (also may be referred to as cellular signals), carrier names (e.g., "chinese mobile"), one or more signal strength indicators of Wi-Fi signals, battery status indicators, time indicators, etc. The navigation bar may include system navigation keys such as a return key, a home screen key, a multi-tasking key, and the like. In some embodiments, the

60 exemplarily shown in fig. 6A may be a Home screen.

As shown in fig. 6A, the

100 may detect a user operation of an

601 acting on the live class application, and display the user interface 61 shown in fig. 6B in response to the user operation.

The user interface 61 may be a main interface provided by a live-type application. The user interface 61 may include:

611,

612,

613, add

614, set control 615.

611 may be used to present some information of the host, such as head portraits, live time length, number of viewers, and live account numbers, among others.

The

612 may be used to display messages sent by the anchor or viewer during the live broadcast process, or system messages generated by interactive operations such as "like" and "like".

The

613 may be used to display images captured and processed in real time by the camera of the

100. The

100 may refresh the display content therein in real time so that the user previews the image captured and processed by the camera of the

100 in real time. The camera may be a rear camera of the

100 or a front camera.

The settings control 615 may be used to adjust the photographic effect of the

100. When a user operation (e.g., a click operation, a touch operation, etc.) is detected on the set control 615, the

100 may display: options for adjusting the photographing parameters of the

100 and/or the image processing parameters. These options may be referred to in the following description of the user interface and are not described here in detail.

The

614 may be used to find the

200. When detecting a user operation on the

614, the

100 may discover other nearby electronic devices using the aforementioned short-range communication technologies such as bluetooth, wi-Fi, NFC, etc., or may discover other remote electronic devices using the aforementioned long-range communication technologies, and query whether the discovered other electronic devices have a camera.

Upon receiving responses from other electronic devices, the

100 may display the discovered camera-equipped electronic devices on the user interface 62. For example, referring to fig. 6C, the

100 may display a

622, the

622 may include information for two electronic devices, including respective: icons, names, distances, locations, etc. of the electronic device.

623 may be used to show the type of electronic device. For example, the first slave device displayed by the

100 may be a tablet computer. The user can quickly and easily preliminarily recognize through the

623 whether the slave device is a device to which he/she wants to connect.

Name 624 may be used to display the name of the slave device. In some embodiments, the

624 may be the model number of the slave device. In other embodiments, the name may also be a user-defined name for the slave device. The names may also be a combination of device model number and user-defined names. The present application is not limited in this regard. It is understood that the list of user interfaces 62, such as "pad C1", "Phone P40-LouS", etc., are exemplary names.

As shown in fig. 6C, the

100 may detect a user operation acting on the

623 of the electronic device, and in response to the user operation, the

100 transmits a request to establish a communication connection to the electronic device (slave device 200) corresponding to the

623.

Referring to fig. 6D, fig. 6D shows the user interface 63 displayed by the electronic device (slave device 200) after the

200 receives the request for establishing the communication connection sent by the

100. As shown in fig. 6D, the user interface 63 includes:

631, a

632, and a cancel

633 of the

100.

The

631 may be used to present identity information of the

100. I.e. the user can determine the information of the

100 that issued the above request through the

631. When a user is able to determine the information of the

100 and trust the host device through the

631, the user may agree to the

100 to use the camera of the electronic device through the

632. The electronic device may detect an operation on the

632, in response to which the

200 may agree to the

100 to use its own camera, i.e., the

100 may establish a communication connection with the electronic device.

The communication connection may be the wired connection or the wireless connection. For example, the master device and the slave device may log into the same user account (e.g., hua as an account) and then make a remote connection through a server (e.g., hua as a provided multi-device collaborative photographing server). The embodiments of the present application are not limited in this regard. It should be understood that the electronic device corresponds to a slave device of the

100.

The user interface 63 also includes a cancel

633. When the

100 cannot be determined by the

631 or when the

100 is not trusted, the user can reject a request to use the electronic device camera sent by the

100 through the cancel

633. The electronic device may detect an operation on cancel

633 and in response to the user operation, the electronic device may refuse the

100 to use its own camera, i.e., the electronic device does not agree to establish a communication connection with the

100.

Fig. 7A-7B illustrate another manner in which the

100 and the

200 establish a communication connection. Wherein fig. 7A is a user interface 71 implemented on a

200 and fig. 7B is a user interface 72 implemented on a

100.

When the

100 detects an operation acting on the

623 in the user interface 62, in response to the user operation, the

100 transmits a request to establish a communication connection to the electronic device (slave device 200) corresponding to the

623.

The verification code 712 may be used for connection confirmation of the

100 and the

200. In response to the above request sent by the

100, the

200 may generate a verification code 712. In some embodiments, the verification code 712 may also be generated by the server 300 and then transmitted to the

200 over the wireless network. The

200 may then display the verification code on the user interface 71.

The cancel control 714 may be used to reject a request sent by the

100 to use the

200 camera. The

200 may detect a user operation on the cancel control 714, in response to which the

100 may close the

711.

Fig. 7B illustrates the user interface 72 for the

100 to enter a passcode. While the

200 displays the user interface 71, the

100 may display the user interface 72. The user interface 72 may display a

721.

721 may include a

7211, a

7212. The

7211 may represent an authentication code of the user

100. The

100 may detect an operation on the

7212. In response to the user operation, the

100 may transmit the

7211 to the

200. The user operation is, for example, a click operation, a long press operation, or the like.

The

200 may check whether the received

7211 is identical to the authentication code 712 displayed by itself. If the two verification codes are the same, the

200 agrees to the

100 to use its own camera. Further, the

200 may turn on its own camera and transmit the image collected by the camera and processed by the ISP to the

100. Conversely, the

200 may reject the request of the

100 to use the

200 camera.

In some embodiments, when

100 enters a

7211 that is different from verification code 712 displayed by

200,

200 may maintain display of verification code 712 and wait for

100 to enter a new verification code. The

200 may also agree to use its own camera by the

100 when the new authentication code received by the

200 coincides with the authentication code 712.

In other embodiments, when the

7211 transmitted by the

100 is different from the authentication code 712 displayed by the

200, the

200 may regenerate another authentication code M, and the

100 may retrieve the authentication code M. When the verification code N acquired by the

100 is consistent with the verification code M, the

200 may also agree that the

100 uses its own camera.

The communication connection may be established in other manners, for example, using near field communication (near field communication, NFC) technology, and the

100 and the

200 may perform authentication by a user operation such as a bump, not limited to the 2 described manners of establishing communication connection shown in fig. 6D to 7B. The authentication method of the present application is not limited to the authentication method of the above-mentioned 2.

After the connection is established, the

100 and the

200 may display hint information, respectively. The prompt may prompt the user that the

100 and the

200 have established a communication connection. As shown in fig. 8A, the user interface 81 shows a user interface that displays prompt information from the

200.

After the authorization shown in fig. 6C (or the authorization shown in fig. 7A-7B) is completed, the

200 may display the user interface 81. The user interface 81 may include a

811 and a

812.

812 may be used to display images captured from the camera of

200.

811 may be used to display prompt information. The above-mentioned hint information is, for example, "the camera of the slave device is being used by the

100".

When the

200 grants the

100 the use of the

200's camera, the

200 may turn on its own camera. The

200 may then display the pictures acquired and processed by its own camera in

812. Above the display layer of

812, a

811 may be displayed by

200.

In some embodiments, the

200 may also display images acquired and processed by its own camera through a floating window. Specifically, the

200 may display a floating window in the upper right corner of the

60 as shown in FIG. 6A. The floating window may display images acquired and processed from the camera of the

200.

When the

200 displays the user interface 81, the

100 may display the user interface 82 as shown in fig. 8B. The user interface 82 may include a

821, a

822, and a

823.

The

821 may be used to display prompt information. The above-mentioned hint information is, for example, "the

100 has connected the

200".

822 may be used to display images acquired and processed from the camera of

200. The

823 may display images captured and processed by the camera of the

100.

When the

200 agrees that the

100 uses the

200 camera, the

100 may obtain an image acquired and processed by the

200 camera from the

200. The

100 may then display the image on the

822. Meanwhile, the

100 may also display a

821. The user can understand that the

100 is connected to the

200 through the prompt contents displayed in the

821.

In addition, the user interface 82 may also add a

824. The setup control may be used to display the shooting capability options of the

200. Reference should be made to the following examples for specific description, which are omitted here for brevity.

In some embodiments, the

100 may also exchange content displayed by

823 and

822. Specifically, the

100 may detect a user operation acting on the

822, and in response to the user operation, the

100 may display an image acquired and processed by the camera of the

200 in the

823. At the same time, the

100 may display images captured and processed by the camera of the

100 at

822. The user operation may be a click operation, a left slide operation, or the like.

In some embodiments, the host device may also divide the

823 into two separate portions. One for displaying images captured and processed by the camera of the

100 and the other for displaying images captured and processed by the camera of the

200. The present application does not limit the arrangement of the preview images of the

100 and the

200 displayed on the

100.

Fig. 6A-8B illustrate a set of user interfaces where the

100 establishes a communication connection with the

200 and displays images captured and processed by the cameras of the

200. After that, the

100 may acquire the ability of the

200 to control the photographing effect, and may transmit a command to the

200 to control the photographing effect.

Fig. 9A to 9D exemplarily show a set of user interfaces of the

100 controlling the photographing effect of the

200. 9A-9C are user interfaces on the

100 and 9D are user interfaces on the

200.

After the

821, which displays prompt contents, shown in fig. 8B is closed, the

100 may display the user interface 91 as shown in fig. 9A. The user interface 91 may include a

911, a

912, a

913, a

915, and a

916.

911 may display images captured and processed by the camera of

100.

912 may display images acquired and processed from the camera of

200.

913 may be used to close

912.

913 may be used to close

912. The

100 may detect a user operation on the

913, in response to which the

100 may close the

912.

The

915 may be used to display a capability option for the

100 to control a photographic effect. The

916 may be used to display capability options for controlling the photographic effect from the

200. A prompt, such as "click to adjust remote screen," may also be displayed alongside the

916. The

100 may detect a user operation acting on the

916, and in response to the user operation, the

100 may display a capability option of the

200 to control a photographing effect, referring to fig. 9B.

In some embodiments, the user interface 91 may also include a

913, an add control 914. The

913 may be used to close one or more windows in the user interface 91, e.g.,

911,

912. The add control 914 may be used to find connections to other slave devices. After the

100 detects a user operation on the add control 914, the

100 may display the query results shown in

622 in FIG. 6C.

When the

100 detects an action on the other slave devices displayed in the

622, the

100 may send a request to the slave device to use the camera. Similarly, the other slave devices may agree to the request sent by the

100, and then the slave device may enable its own camera to collect and process the image according to the default shooting parameters, and further send the processed image to the master device. Meanwhile, the

100 may add a window to display the above image.

By implementing the method, the

100 can display images sent by a plurality of slave devices, so that a richer shooting experience is provided for a user.

In some embodiments, the

100 may also multiplex the

915 and the

916. A particular user interface 91 may display a general setup control. When the image captured and processed by the

100 is displayed in the

911, the setting control may display an ability option of the

100 to control the photographing effect. When images captured and processed from the

200 are displayed in the

911, the setting control may display a capability option of the

200 to control the photographing effect.

Fig. 9B illustrates the user interface 92 of the

100 displaying the capability options of the

200 for controlling the shooting effect. The user interface 92 may include a

921 from the

200. The

921 may display various capability options that the

200 has to control photographing effects, such as an aperture, a flash, intelligent follow,

922, ISO sensitivity, a zoom range, beauty, a filter, and the like.

The present application specifically describes a user interface in which the

100 transmits control commands to the

200 and the

200 executes the control commands described above, taking the

922 adjustment as an example. White balance can be used to calibrate the color temperature bias of the camera.

922 may include daylight mode, incandescent

923, fluorescent mode, cloudy day mode, shadow mode. The

100 may detect a user operation acting in any of the modes described above. When the

100 detects a user operation acting on the

923, the

100 may issue a control command to the

200 to change the white balance mode to the incandescent lamp mode in response to the user operation. The

200 receiving the above command may change the

922 to the

923. The

100 may then receive and display the image of the change

923 sent from the

200. Refer to fig. 9C. Meanwhile, the display image from the viewfinder of the

200 may also be adjusted to an image after the white balance mode is changed. Referring to FIG. 9D

In some embodiments, the

100 may also set a dedicated page to display the capability option of the

200 to control the shooting effect. I.e., the

100 may use a capability option in a separate

921. The embodiments of the present application are not limited in this regard.

As shown in fig. 9C, the user interface 93 may include a

931.

931 may be used to display images acquired and processed from the camera of

200. When the

100 can issue a control command to the

200 to change the white balance mode to the incandescent lamp mode, the

100 can receive an image from the

200 to change the white balance mode to the incandescent lamp mode. The

931 may display the above-described image.

While the user interface 93 displays the image after the white balance adjustment, the

200 may also display the user interface 94 after the white balance adjustment. Refer to fig. 9D. The user interface 94 is a user interface displayed on the

200. The user interface 94 may include a

941.

The

941 may be used to display images captured and processed from the camera of the

200. Upon receiving a control command issued from the

100 to the

200 to change the white balance mode to the incandescent lamp mode, the

200 may change the white balance mode to the incandescent lamp mode. The

941 may then display the image captured and processed from the camera of the

200 in incandescent mode. Meanwhile, the

200 may transmit the above-described image to the

100. The

100 may display the above-described image as shown in fig. 9C.

In some embodiments, the

200 may have some or all of the above-described capabilities in the list, or may also have other capabilities not mentioned in the

921 to control the shooting effect. The present application is not limited in this regard.

The

100 may also acquire the capability of the

100 to control the photographing effect, and issue a control instruction to itself. Fig. 10A to 10C show a set of user interfaces for controlling photographing effects by the

100 down to itself. Fig. 10A illustrates a user interface 101 in which the

100 acquires the ability to control a photographing effect itself. The user interface 101 may include a

1011.

The

1011 may be used to display a capability option for the

100 to control the shooting effect. A prompt may also be displayed next to the

1011, such as "click adjust home screen". The

100 may detect a user operation on the

1011, and in response to the user operation, the

100 may display a shooting effect capability list of the

100, as in the user interface 102 shown in fig. 10B.

The user interface 102 may include a

1021. The window may display options for the ability of the

100 to control the shooting effect, such as

1022, flash, intelligent follow, beauty, filters, etc. The present embodiment describes, taking an aperture as an example, that the

100 transmits a control command for adjusting the photographing effect to the

100.

The

1022 is adjustable in size by a

1024. The

100 may detect a user operation on the

1024, and in response to the user operation, the

100 may issue a control command to the

100 to adjust the aperture.

Specifically, the

1024 may initially scale "f/8". The user can slide the float on the

1024 to the aperture scale of "f/17" by a right-slide operation. The

100 may detect this user operation, in response to which the camera of the

100 may replace the aperture with "f/8" with "f/17". The aperture change to "f/17" may result in a shallower depth of field, and accordingly, the window displaying the image captured by the

100 may display an image having a shallower depth of field captured by the

100. As shown by

1031 in fig. 10C.

Fig. 10C illustrates the

103 with the

100 preview window depth of view reduced. The

103 may include a

1031. The

1031 may be used to display images captured and processed by the camera of the

100.

Upon receiving a control command issued by the

100 to replace the aperture "f/8" with "f/17", the

100 may replace the aperture "f/8" with "f/17". The

1031 may then display the image captured and processed by the

100 having an aperture size of "f/17".

Fig. 6A-6D, fig. 7A-7B, fig. 8A-8B, fig. 9A-9D, fig. 10A-10C, fig. 11A-11C depict a series of user interfaces for the

100 to establish a communication connection with the

200 and to control the effects of capturing images from the

200 in a live scene. The above-described method of controlling the photographing effect of the

200 may also be used in photographing scenes. A series of user interfaces for the

100 to establish a communication connection with the

200 and to control the photographing effect of the

200 in the photographing application scenario will be described.

Fig. 11A illustrates the

100 displaying a user interface 111 that adds slave devices. User interface 111 can include add

1112,

1113.

100 may detect a user operation on

1112, in response to which

100 may query an electronic device having a camera. Upon receiving the message with the camera transmitted back from the electronic device, the

100 may display a

1113.

1113 may be used to present information for an electronic device having a camera. For example,

1113 illustrates information for two electronic devices with cameras (

1114, electronic device 1115). Similarly, the above information is information such as the name and location of the slave device.

Taking the

1115 as an example, the

100 may detect a user operation acting on the

1115, and in response to the user operation, the

100 may send a request to the

200 to use the camera of the

1115. The

1115 may detect an operation in which the user agrees to the

100 to use the own camera, and in response to the operation, the

1115 may agree to the

100 to use the camera of the

1115. The user interface of the above-described process in which the

1115 grants the use authority to the

100 may refer to the user interface in the live scene. As shown in fig. 6C-6D, or fig. 7A-7B. This is not repeated here.

After the user of the

200 agrees that the

100 uses the camera of the

200, the

200 may display the user interface 112, referring to fig. 11B. While the

100 may display a user interface 113, as shown in fig. 11C.

User interface 112 is a user interface on

200. User interface 112 illustratively shows a user interface for displaying prompts from

200. The user interface 112 may include a

1121 and a

1122.

When the

100 is granted access to the

200, the

200 may turn on its own camera, and further, the

1122 may display the image captured and processed by the

200 camera. The user interface 112 may also display a

1121. The

1121 prompts the user that the camera of the device has been used by other devices. For example, "the current picture is being used by LISA".

As shown in fig. 11C, the user interface 113 is a user interface on the

100. User interface 113 illustrates a user interface in which

100 displays a prompt.

When the

200 displays the user interface 112, the

100 may display the user interface 113. The user interface 113 may include a

1131, a

1132, and a

1134.

The

1131 may be used to display images captured and processed by the camera of the

100.

1132 may display images acquired and processed from the camera of

200. When receiving an image transmitted from the

200, the

1132 may display the image. The

1134 may be used to display prompt information, such as "connected camera: phone P40-LouS). The above-described hint information may be used to hint to the user that the

100 has connected to the

200.

In some embodiments,

1131 and

1132 may also exchange display content. I.e.,

1131 may display images acquired and processed from the camera of

200.

1132 may be used to display images captured and processed by the camera of

100. In particular, when the

1131 may display an image collected and processed by the camera of the

200, the

100 may be provided with the above-described delete key 1133 in the

1131. In response to a user operation acting on delete key 1133,

100 may close

1131.

In some embodiments,

1132 may be displayed over

1131 in the form of a floating window in user interface 113. In another embodiment,

1132 may also be tiled with

1131. Refer to fig. 11D.

Fig. 11D illustrates another user interface 114 in which the

100 displays a prompt. User interface 114 may include

1141,

1142,

1143, setting

1147, setting

1148.

1141 may be used to display images captured and processed by the camera of

100.

1142 may be used to display images captured and processed from the camera of

200. Likewise,

1141 and

1142 may also exchange display content. I.e.,

1141 may display an image captured and processed from the camera of

200.

1142 may be used to display images captured and processed by the camera of

100. The present application is not limited in this regard. The

1143 is used to display prompt information.

The

1147 may be used to display a camera capability option of the camera of the

100.

1148 may be used to display a capture capability option from the camera of

200.

The

100 may detect a user operation on the

1148, in response to which the

100 may display a shooting capability option of the

200. The shooting capability options described above may refer to shooting capability options displayed in a

921 in the user interface 92 shown in fig. 9B. This is not repeated here.

Further, the

100 may detect a user operation acting on a certain shooting capability option, and in response to the user operation, the

100 may transmit a control command to adjust the shooting effect to the

200.

Also taking the user interface 92 shown in fig. 9B as an example, in response to a user operation of the

923 acting in the above-described

921, the

100 may transmit a control command to adjust the white balance mode to the incandescent lamp mode to the

200. In response to the control command described above, the

200 may perform color temperature calibration of the incandescent lamp mode on the image captured by the camera. The

200 may then transmit an image for color temperature calibration using the incandescent lamp mode to the

100.

In response to the image after the photographing parameters are adjusted, which is transmitted from the

200, the

100 may display the image.

In some embodiments, the user interface 114 may also include a

1144, an

1145, and a

1146.

1144 may be used to close

1142. The

1145 may be used for the

100 to discover other electronic devices with cameras.

The

100 can detect a user operation on the

1146. In response to the user operation, the

100 may store the contents displayed in the

1131, 1132 as pictures or videos. In a live or video call scenario, the live application/video call application may also obtain the above-mentioned picture or video and send it to the server providing the live/video call.

In some embodiments, the number of slave devices may also not be limited to one. For example, the

100 may detect a user operation of an add control, such as add control 914 of user interface 91, add

1145 of user interface 114, and the like. In response to the user operation, the

100 may establish a connection with other electronic devices with cameras. Further, the

100 may establish connection with a plurality of slave devices and use images transmitted from the plurality of slave devices.

In some embodiments, the

100 may also display only images sent by the slave device. For example, after the

100 establishes a communication connection with the

200, the

100 may turn off its own camera and display only the image transmitted from the

200.

s101: the

100 establishes a communication connection with the

200.

In a specific implementation, the communication connection established between the

100 and the

200 may be a wired connection or a wireless connection as described above.

In some embodiments, the

100 may first discover other electronic devices with cameras in response to a received user operation (e.g., a user operation acting on the

614 as shown in fig. 6B), and then send a connection request to the electronic device selected by the user. After the electronic device responds to the user operation granting the request (e.g., the user operation acting on the

632 shown in fig. 6D), the

100 and the electronic device successfully establish a communication connection.

In other embodiments, the

100 may scan the two-dimensional code of the

200 and establish a connection with the

200. Specifically, the electronic device can display the two-dimensional code using the camera of the electronic device. The

100 may acquire the two-dimensional code, and in response to the operation, the

100 may transmit a use request to the electronic device and obtain approval of the electronic device.

Without being limited to the above method, the

100 may also establish a communication connection in other ways. Such as a bump-and-bump operation based on NFC technology. The embodiments of the present application are not limited in this regard.

Specifically, the

100 may search for other electronic devices with cameras through a device virtualization kit (DVKit), send a request for establishing communication connection to the discovered electronic device, and after the DVKit receives a message that agrees to establish communication connection fed back by the other electronic devices, establish communication connection with the electronic device. Further, the DVKit establishes a communication connection with the electronic device through a distributed device virtualization platform (DMSDP), and the DMSDP is specifically configured to establish a session with the electronic device. The sessions include control sessions and data sessions.

At this time, the above-described electronic device that establishes a session with the

100 may be referred to as a

200 of the

100.

S102: the

100 acquires shooting capability information of the

200 through a communication connection.

DMSDP may register the slave device to the virtual camera HAL based on the session channel established by the

100 and the

200. Meanwhile, the DMSDP may request photographing capability of the

200 from the

200.

Specifically, the

200 may acquire its shooting capability from a camera service module (camera service) module itself. The shooting capability includes the hardware capability of the camera and the software capability of the image processing module such as ISP, GPU and the like, and some shooting capabilities combining the hardware capability and the software capability can be specifically described with reference to table 1 in FIG. 3.

The

200 may then transmit the shooting capability described above to the

100. Upon receiving the shooting capability transmitted from the

200, the DMSDP may transmit the shooting capability information to the HAL layer virtual HAL module. Further, the virtual camera HAL may also send the shooting capabilities described above to camera management (camera manager).

Table 1 shows the shooting functions that the

200 may include in hardware capabilities, combined with hardware capabilities and software capabilities. Such as the number of cameras, camera IDs, pixels, aperture sizes, zoom ranges, filters, beauty, and various shooting modes, which the

200 has. In particular, the shooting modes such as the night view mode, the portrait mode, and the like include not only hardware capabilities of the camera of the

200 but also image-point capabilities of the

200.

The device co-management and dynamic pipeline may learn that the

200 is currently registered in the virtual camera HAL.

When the

200 is registered in the virtual camera HAL, the device collaborative management needs to perform collaborative management on the

200. Specifically, when the

100 displays images from the

100 and the

200 at the same time, the device cooperation management can distinguish which camera of which device the above image comes from, for example, the camera 1 of the master device, the camera 1001 of the slave device, according to the IDs of the cameras. The device co-management may then cause the time delay for displaying the images from the

100 and the

200 to be within acceptable range to the human eye by repeating or buffering the images of the

100.

Similarly, the dynamic pipeline may distinguish between control commands flowing to the

100 or the

200 according to information such as the ID of the camera, and further send the control commands sent to the

200 to the virtual camera HAL.

After registration is completed, the virtual HAL may send a notification to the DVKit to change the connection state. In particular, the virtual HAL may inform the DVKit to change the unconnected state to the connected state. The unconnected state refers to that the

100 does not establish a connection with other electronic devices using the camera of the electronic device. Accordingly, the above-mentioned connected state means that the

100 has established a connection with other electronic devices using the camera of the electronic device.

S103: the

200 collects and processes images and then transmits the collected and processed images to the

100.

After the

200 and the

100 successfully establish a communication connection, the

200 may automatically start capturing and processing images.

Reference is made to the user interfaces shown in fig. 6D and 8A. When the

200 agrees to connect with the

100, the

200 may open its own camera application. The user interface of the camera application is shown in fig. 8A. At the same time, the

200 may display images captured and processed by its own camera, see

812 of fig. 8A.

The

200 may then transmit the image to the

100. The

100 may display the above-described image. As shown in fig. 8B, the

100 may add a

822 in the user interface 82. The preview window may display images captured and processed from the

200. Thus, the

100 can realize collaborative shooting across devices and simultaneously display images acquired and located by a plurality of cameras on a display screen.

In some embodiments, the

100 may also only control and use the cameras of the

200, i.e. the

100 only displays images of the

200, and not captured and processed images of the

100.

In other embodiments, after the

100 and the

200 establish a communication connection, the

100 may transmit a control command to turn on the camera to the

200 in response to an operation of turning on the camera of the

200 by the user. The

200 may respond to the above command and turn on the camera.

For example, after the user interface 63 shown in fig. 6D, the

100 may also display a prompt window. The prompt window may ask the user whether to open the camera of the

200. When the

100 detects a user operation for turning on the

200, the

100 may transmit a control command for turning on the camera to the

200. In response to the control command, the

100 may turn on the camera and obtain an image acquired and processed by the camera.

Further, the

200 may transmit the above-described image to the

100. The

100 may display the image described above, such as

822 shown in fig. 8B. Meanwhile, the

200 may display the above-described image on its own display screen. Such as user interface 81 shown in fig. 8A.

In some embodiments, the

200 may not display the image acquired and processed by its own camera.

In the two schemes, the shooting parameters used by the camera of the

200 may be default, for example, the

200 may default to use a rear-mounted common camera for acquisition, the camera uses a double focal length, the color temperature calibration uses a default daylight mode, and the aperture size is f/1. 6. Optical anti-shake, flash off, shutter time 1/60, ISO sensitivity 400, pixels 8192 x 6144, crop box size 3:4, off face/body algorithms, no filter, no decal, etc.

Then, the

100 may transmit a series of control commands for collaborative photographing, such as photographing, video recording, or control commands for adjusting photographing effects, such as changing a filter, to the

200. The

200 may adjust the acquisition and processing of the image according to the above-described command.

S104: the

100 transmits a command for controlling a photographing effect to the

200 in response to the received user operation.

The control command carrying the photographing parameters may be transmitted to the

200 through the communication connection established by the

100 and the

200. In particular, the virtual camera HAL may send the created control commands to a device virtualization platform (DMSDP), which may include a data session channel and a control session channel of the

100 and the

200. The control command may be transmitted to the

200 through the control session channel.

For example, the "set white balance mode of the

200 to incandescent lamp mode" control command may include: the stream creation command may be a preview command, with the shooting parameters modified (white balance=incandescent lamp mode). The control commands may also include default shooting parameters such as one-time focus, no filter, off beauty, etc.

As shown by

921 shown in user interface 92. According to the photographing capability information of the

200 acquired in S102, the

100 may display photographing capability options corresponding to the photographing capability on the display screen. The

100 may detect a user operation acting on a certain shooting capability option. In response to the user operation, the

100 may transmit a command to control the photographing effect to the

200.

Taking the control command of "set white balance mode of

200 to incandescent lamp mode" as an example,

100 may detect a user operation acting on the incandescent lamp mode, for example, a user operation acting on

923 in user interface 92, and in response to the user operation,

100 may transmit the above-described control command to

200 for controlling

200 to set white balance mode to incandescent lamp mode.

In creating and transmitting the control command, the

100 may also perform various processes on the command, such as dynamic pipelining, multi-stream configuration, and multiplexing control.

When the application program generates a control command to set the white balance mode of the

200 to the incandescent lamp mode, the

131 may generate a control command with the preview control 1311 and the adjustment of the photographing

1312. The control command corresponds to a repeated

1314. Repeated frame control may indicate that the control command is acting on multiple frames. The repeated frame control may include fields Cmd and Surfaces. The Cmd field may be used to represent a control command. In some embodiments, the number of the control command may also be included in the cmd. The Surfaces may be used to receive a view of the rendered frame and send the resulting effect to a surfacefinger for image composition display to a screen.

The dynamic pipeline may tag the control commands described above. The tags may include an on-demand tag, a stream tag, a repeat frame tag. The camera service 132 may include a pending

1321. When the above-mentioned repeated frame control command reaches the pending

1321, the repeated frame control command may replace a basic command 1322 (for example, "cmd+streamids+buffer") in the original pending

1321, where the basic command further includes other default parameters for controlling the shooting effect, for example, default mode of white balance: daylight mode, default filter: no filter, etc.

After replacement, an on-demand repeat frame control command 1324 (e.g., "cmd+streamids+buffer+incandescent lamp mode+isnew+repeat") is added to the pending

1321. The repeat

1324 may add two fields IsNew and repeat. IsNew may be used to indicate that the command is an on-demand control issued for an application. Repetition may be used to indicate that the command is a repeated frame control. Meanwhile, the repeated

1324 may replace the original default white balance mode (e.g., the above-described daylight mode) with the incandescent lamp mode (i.e., white balance=incandescent lamp mode).

In addition, the dynamic pipeline may also flag the flow direction tag that sends the control command to the

200. For example, the dynamic pipeline may add a flow tag Device to the control command. The Device may represent the object to which the control command acts by a camera number (ID). Referring to table 1, when device=1, the control command may represent a control command to the front lens of the

100. The local camera HAL of the

100 may receive the control command described above. When device=1002, the above control command may represent a control command to the rear-mounted normal lens of the

200. The virtual camera HAL of the

100 may receive the control command described above. Accordingly, a Device according to a control command to set the white balance mode of the

200 to the incandescent lamp mode may be set to 1002.

The multi-stream configuration in the stream processing module may then add stream configuration information for the control commands described above. The control commands described above include the preview control 1311 and the

1312 of "set white balance mode to incandescent lamp mode". That is, the multi-stream configuration may configure one preview stream (1080P) and one analysis stream (720P) for preview control 1311. The specific configuration rules of the multi-stream configuration may be described with reference to table 2 in fig. 3, and will not be described herein.

The multiplexing control may multiplex the plurality of streams configured by the multi-stream configuration module. The number of streams transmitted back from the

200 to the

100 can be reduced through multiplexing control, thereby reducing network load and improving transmission efficiency. In particular, multiplexing control may cover low quality streams with high quality streams. For example, a split stream of

720P may multiplex a preview stream of 1080P. Thus, for control commands requiring a preview stream (1080P) and an analysis stream (720P),

200 may only transmit back a preview stream of 1080P.

Further, according to the flow direction label device=1002, the control command may be sent to the virtual camera HAL module of the HAL layer. As shown in fig. 13, the virtual camera HAL may filter out control commands 1324 issued on demand. The filtered on-demand control commands may be stored in a

1331.

100 may send control commands in a

1331 to

200.

When

141 sends a photographing command,

142 module may configure a preview stream (1080P), an analysis stream (720P), a video stream (4K), a photographing stream (4K) for the command. Then, the multiplexing

143 module may multiplex the above information configuring the four streams into configuration of two streams: one preview stream (1080P), one video stream (4K). Finally, the

100 transmits the photographing control command, which is divided and multiplexed, to the

200.

In the process in which the

100 transmits a command for controlling a photographing effect to the

200 in response to a received user operation, the

100 may multiplex an image stream (stream) required in the control command. In response to the multiplexed control command, the

200 may transmit only the multiplexed image stream, thereby reducing the image stream transmitted in the network, reducing the network load, and further improving the transmission efficiency.

S105: a command for controlling a photographing effect is received from the

200, and in response to the command, the

200 may acquire and process an image.

The camera proxy service of the

200 may receive a control command for adjusting a photographing effect transmitted from the

100. In particular, DMSDP may have established a data session channel and a control session channel between the

100 and the

200. The camera proxy service module of the

200 may receive the control command transmitted by the control session channel. In response to the control command described above, the

200 can acquire and process an image according to the photographing parameters carried in the control command.

Specifically, the

200 may perform a corresponding operation according to the photographing parameter carried in the control command. When the control command carries a hardware parameter, the

200 may collect an image according to the hardware parameter, which may include, but is not limited to: the

200 captures an image using the front or rear camera, the optical zoom range, the optical image anti-shake on, the flash on, the light supplement on, the frame rate of 30fps indicated in the control command. When the control command carries a software parameter, the

200 may process the acquired image according to the software parameter, which may include, but is not limited to: cutting, color temperature calibration, noise reduction, filter effect addition and the like are performed on the acquired image.

Also taking the control command of "setting the white balance mode of the

200 to the incandescent lamp mode" as an example, after receiving the command, the slave device may adapt and parse the command, and then send the command to the local camera HAL of the

200, and finally obtain the processing result.

Fig. 13 is a diagram illustrating a processing procedure of the

200 in part by the control instruction described above.

The

134 may include a receive

1341, a table of surfaces maps 1344, and a

1342, and the receive

1341 may receive control commands sent by the

100 to set the white balance mode of the

200 to the incandescent mode. The control command may refer to command 1343 (i.e., "cmd+streamids+repeating+incandescent lamp mode+1002, one 1080P preview stream"). Wherein the incandescent lamp mode may represent setting the white balance mode of the

200 to the incandescent lamp mode, 1002 may identify the object to which the command is sent as the camera 1002 of the

200.

First, the

134 can convert StreamIds of control commands in the received command queue into a Surface corresponding to the

200 according to the Surface mapping table 1344. In the camera service, there is a one-to-one correspondence between streamId and surfaceId, i.e., one streamId corresponds to one surfaceId. surface may be used to identify surfaces. The surfaces described above may act as a carrier to display images of the stream generated from the

200. Referring to the description of the foregoing embodiments, streamId may indicate a particular stream and surfaces may indicate a carrier on which a particular display image is rendered. The

200 needs to generate a corresponding stream according to the request sent by the

100, and therefore, the

200 needs to generate the streams ids according to the mapping table 1344 to generate surfaces corresponding thereto after receiving the control command.

The pending

1351 may then replace the base command in the original queue with the repeated frame control described above, resulting in a pending

1352. The

200 may send the above command to the

1361 of the camera 1002 according to device=1002. In response to the above command, the ISP of the camera 1002 may set the white balance mode to the incandescent lamp mode, so that the ISP of the camera 1002 may perform the incandescent lamp color temperature calibration on the image collected by the camera 1002. The ISP may be an Image Video Processor (IVP)/natural processing unit (natural processing unit, NPU)/digital signal processing (digital signal processing, DSP), etc., which is not limited in this application.

After the white balance mode is adjusted, a new image is available from the

200.

Referring to the user interface 91 shown in fig. 9A, before responding to the above-described control command to set the white balance mode of the

200 to the incandescent lamp mode, the image acquired and processed from the camera 1002 of the

200 may display contents with reference to the

912. As shown in

912, in a default white balance option (e.g., daylight mode), the captured and processed image by camera 1002 may be less bright and the image may be grayed out. After responding to a control command to set the white balance mode of the

200 to the incandescent lamp mode, the image acquired and processed by the camera 1002 may be shown with reference to the

931 in the user interface 93. At this time, the brightness of the image is higher, and the overall tone of the picture is closer to the color observed by human eyes.

It is to be understood that the above description of the image effects before and after the

200 responds to the control command for adjusting the photographing effect is an example. The above description should not be taken as limiting the embodiments of the present application.

S106: the

200 displays the processed image.

This step is optional. As described in S103, in some embodiments, after the

200 uses the own camera at the same time as the

100, the

200 may display the image acquired and processed by the own camera on the display screen of the present device. Reference is made to the user interface 81 shown in fig. 8A. Accordingly, the

200 may also display the updated image upon receiving a control command in response to adjusting the photographing effect. Such as user interface 94 shown in fig. 9D.

In other embodiments, the

200 may not display the images acquired and processed by its own camera when the camera is turned on. Therefore, after the

200 responds to the control command for adjusting the photographing effect transmitted from the

100, the

200 may not display the adjusted image. That is, the slave device only transmits the acquired image to the

100 for use, and the display screen of the slave device does not realize the image acquired and processed by the camera of the slave device.

S107: the

200 transmits the processed image to the

100.

In response to a control command to adjust the shooting effect transmitted from the

100, the

200 can obtain a new image. In response to the type and number of streams required by the

100, the

200 may generate streams required by the

100 accordingly. In some embodiments,

200 may stream capture a set of image streams. The group of image streams is the group with the highest image quality among the types and numbers of streams required by the

100. The highest image quality may be the highest resolution, etc. The

200 may then copy the acquired set of image streams into a plurality of streams, and compress, adjust, etc. the copied streams according to the type and number of streams required by the

100, for example, copy a set of 720P analysis streams from the acquired set of 1080P video streams.

Also taking the control command of "set white balance mode of

200 to incandescent lamp mode" as an example, in the foregoing description, the multiplexing control module may multiplex two streams configured by the multi-stream configuration module into one stream. Thus, the

100 finally transmits a control command requiring one 1080P preview stream to the

200. Reference is made specifically to the description of S103.

In response to the control commands sent by

100,

200's local camera HAL may generate a 1080P preview stream. The preview stream described above has set the white balance mode to the incandescent lamp mode. The local camera HAL may then send the stream to the camera proxy service of the

200, which may further transmit the stream back to the

100.

In some embodiments, the streams returned from the

200 to the

100 are greater than 1 stream, such as the multiple stream scenario illustrated in fig. 14. When the

100 transmits the photographing control command to the

200, the multi-stream configuration and multiplexing control finally configures one preview stream (1080P) and one video stream (4K) for the photographing control command, and the description of S103 may be referred to.

In response to the photographing control command described above, the

200 may generate one 1080P preview stream (stream 1), one 4K video stream (stream 2), and one photographed image stream (stream 3). In an alternative embodiment,

200 may capture a set of image streams according to

2. From

2, the

200 may replicate two other sets of streams (stream 1', stream 2'), and from the

100, the

200 may process streams 1 'and 2' to obtain

1 and 2, e.g., compress stream 1 'of 4K to obtain stream 1 (1080P), obtain a photographed image from stream 2' of 4K, and so on, as required by the

100 for

1 and 2. The camera proxy service of the

200 may then transmit the 3 streams to the

100.

S108: the

100 displays an image transmitted from the

200.

The

100 may restore the streams returned from the

200 to the number and type of streams actually required by the

100. The analysis stream may be used for image processing, the preview stream may be used for display, etc. The

100 may send each stream to a corresponding module for processing and utilization according to the type of the recovered stream. With the

200 transmitting the preview stream, the

100 can realize a function of displaying an image.

The post-processing module of the

100 may perform intelligent splitting and multi-stream output to the

200.

First, the intelligent diversion module can record the number and types of flows configured by the multi-flow configuration module. Upon receipt of the multiplexed stream returned from the

200, the intelligent splitting module may replicate the received stream into a plurality of streams according to the records described above. The replicated multiple streams may be the same as or slightly lower than the received stream, so as to obtain multiple streams with the same number and type of streams configured by the multi-stream configuration module. The split streams may then be sent to the corresponding modules for processing and utilization.

For example, in an example in which the white balance mode of

200 is set to the incandescent lamp mode,

100 may receive a 1080P preview stream returned from

200. At this time, the intelligent branching module may restore the preview stream to one preview stream (1080P) and one analysis stream (720P) configured by the multi-stream configuration module.

Fig. 14 also shows another example of smart diversion. The figure shows a split process where the

100 receives multiple streams back from the

200.

As shown, the

100 may receive a 1080P stream (stream 1), a 4K stream (stream 2), and a picture stream (stream 3). The intelligent splitting module may split stream 1 into a 1080P preview stream and a 720P analysis stream.

2 may be divided into a 1080P stream (stream 1) and a 4K video stream, and the 1080P stream (stream 1) may be split further, with reference to the splitting of stream 1. Stream 3 may be split into a 4K photo stream.

The splitting process can restore the multiplexed stream to the stream required by the original

100, thereby reducing network load and improving transmission efficiency caused by multiplexing, and meeting the original requirement of the application program, so that the normal use of the application program is not affected.

The multiple streams obtained after the splitting can be respectively sent to corresponding modules of the application layer for application programs to use. Based on this, the application may display the preview stream. In some embodiments, the

100 may only display the image returned from the

200. In other embodiments, the

100 may display both the image returned from the

200 and the image captured and processed by its own camera, referring to the user interface 93 shown in fig. 9C.

As shown in fig. 9C, the

100 displays a form in which an image returned from the

200 can pass through a floating window, referring to a

931. In some embodiments, the

100 may detect a user operation on the

931, in response to which the

100 may move the position of the floating window to any position of the user interface 93. The operation described above is, for example, a long press drag operation. In some embodiments, the

100 may detect another user operation on the

931, in response to which the

100 may exchange content displayed in the floating window and the preview window. The operation is, for example, a click operation. In addition, the

100 may also adjust the size of the floating window, etc. in response to other operations of the user.

In other embodiments, the

100 may also divide into two tiled window displays, such as the user interface 114, when displaying both the image returned from the

200 and the image captured and processed by the own camera.

It will be appreciated that when the

100 connects a plurality of slave devices, the

100 may display images returned by the plurality of slave devices. Likewise, the

100 may be displayed through a floating window, may be tiled, and is not limited in this embodiment.

In some embodiments, when the control command is a parameter-adjusted photographing command, the frame synchronization module may synchronize the received stream returned from the

200 before the post-processing shunts the stream returned from the

200. Frame synchronization may reduce delay errors due to network transmissions. Particularly for the photographing flow, the photographing result obtained by the photographing flow with the frame synchronization can be closer to the requirement of the user.

As shown in fig. 15, the frame sync may include three parts. The

151 and the

153 may represent processes occurring on the

100. The

152 may represent a process that occurs from the

200. Taking a stream of 7 frames and a photographing command as an example, the

100 performs frame synchronization on the photographing command.

At the

1511, the

100 may create a

1512. The

100 may then send the command to the

200 via a wired or wireless connection. The

200 receives the above command at the

1521. Thus, the

200 takes the image of the

1521 as a result of executing the photographing

1512. The

200 having processed the photographing command may transmit the processing result back to the

100. The processing result received by the

100 is a

1521 image. At this time, the

100 may perform synchronization processing on the above result.

In a photo scene, frame synchronization may advance the processing results. For example, the processing result received by the

100 is a

1521 image. The

100 may advance by one frame with the

1511 as the result of the synchronized processing. It will be appreciated that the receipt of a command from the

200 one frame time later is an exemplary network delay. That is, in some embodiments, the time at which the command is received from the

200 may also be a fourth frame, a fifth frame, etc. The time delay for receiving a command from the

200 varies depending on the actual network communication quality. The

100 advances one frame as a synchronization result is also exemplified.

In some embodiments, the

100 may not have obtained the

1511 as a result of the synchronized processing. For example, in the second frame, the

100 may create a photographing command. Then, the

200 receives the above command at the fourth frame. Accordingly, the

200 takes the image of the fourth frame as a result of executing the photographing command. The

100 performs forward synchronization processing on the received fourth frame image to obtain a third frame image.

When the control command is a command for adjusting only parameters or a video recording command, for example, the control command for controlling the photographing effect of the above-described "white balance mode is set to incandescent lamp mode", the

100 may not perform frame synchronization on the received image.

S109: the

100 performs processing such as photographing, video recording, and forwarding on the displayed image.

When the application of the

100 obtains the image transmitted from the

200, the

100 may further utilize the image.

As shown in fig. 9C, in a live application scenario, the live application of the

100 may forward the obtained image. Specifically, the

100 may forward the image to a live server. The server may distribute the image to the user devices viewing the live broadcast.

As shown in fig. 11D, in an application scenario of photographing or video recording, the

100 may detect a user operation acting on the photographing

1146, and in response to the operation, the

100 may perform photographing storage or video recording storage on an obtained image acquired from the camera of the

200.

in the above-described method S103, before receiving the control command for adjusting the photographing effect transmitted by the master device, the image acquired and processed by the slave device may be referred to as a first image, for example, an image shown by a

912 in the user interface 91.

In the above-described method S108, after the master device transmits a control command for adjusting the photographing effect to the slave device, the image collected and processed by the slave device according to the control command may be referred to as a second image. Such as the image shown by

921 in user interface 93.

The image acquired and processed by the host device according to the default shooting parameters of its own camera may be referred to as a third image, for example, an image displayed in the

911 in the user interface 91.

The user can adjust the shooting effect of the main equipment, and the main equipment can control the camera of the main equipment to adjust shooting parameters in response to the adjustment operation. The main device acquires and processes the new image according to the adjusted shooting parameters, and the new image can be called a fourth image. Such as an image displayed by

1031 in

103.

Another possible software structure of the

100 is described below.