CN118171572A - Unmanned plane cluster evolution type simulation training method, system, medium and equipment - Google Patents

The invention relates to a simulation training method, a system, a medium and equipment for unmanned aerial vehicle cluster evolution, wherein the method comprises the following steps: designing and modeling an unmanned aerial vehicle air combat scene; modeling an unmanned aerial vehicle air combat equipment model; constructing an unmanned aerial vehicle air combat autonomous decision algorithm; performing simulation iterative training on an unmanned aerial vehicle air combat autonomous decision algorithm; performing intelligent algorithm man-machine countermeasure verification; and carrying out multiple iterations on the unmanned aerial vehicle air combat autonomous decision algorithm, and repeating the following four steps. According to the invention, simulation of the unmanned aerial vehicle air combat scene is realized by a simulation means, the algorithm evolution is advanced by adopting a parallel simulation node mode, and the algorithm multi-round iterative training is realized.

Unmanned plane cluster evolution type simulation training method, system, medium and equipment

Technical Field

The invention belongs to the technical field of unmanned aerial vehicle cluster air combat decision making, and particularly relates to an unmanned aerial vehicle cluster evolution type simulation training method, system, medium and equipment.

Background

Since the first world war appears unmanned plane, the unmanned plane plays an increasingly important role in war by virtue of the characteristics of low cost, consumption, flexibility, multiple purposes and the like. In an air battle field, the unmanned aerial vehicle carries out the tasks such as approaching investigation, close-range ammunition and the like, extends as the capability of a man-machine battle system, executes the high-risk battle task, can greatly improve the survival probability of the man-machine and improve the battle efficiency of the system.

The unmanned aerial vehicle has the problems of high human dependence of task decisions, large cluster control difficulty, limited operational radius due to links and insufficient onboard processing capacity, poor processing capacity of emergency battlefield events and the like. The traditional autonomous decision algorithm is limited by priori knowledge of developers, and has the conditions of poor robustness, low emergency processing capability and the like, and the algorithm has certain defects.

An evolution type simulation training method for the unmanned aerial vehicle cluster collaborative air combat autonomous decision algorithm is constructed, so that the air combat unmanned aerial vehicle cluster can conduct rapid autonomous learning and iteration of the algorithm in a digital environment, the algorithm research and development cost is reduced, the robustness and environmental adaptability of the algorithm are improved, and the intelligent degree and reliability of the algorithm are stably improved.

Disclosure of Invention

In order to overcome the problems in the prior art, the invention provides an unmanned aerial vehicle cluster evolution simulation training method, system, medium and equipment, which are used for solving the technical problems in the prior art.

An unmanned aerial vehicle cluster evolution-type simulation training method, the method comprising:

S1, designing and modeling an unmanned aerial vehicle air combat scene;

S2, modeling an unmanned aerial vehicle air combat equipment model;

s3, constructing an unmanned aerial vehicle air combat autonomous decision algorithm;

S4, performing simulation iterative training on an unmanned aerial vehicle air combat autonomous decision algorithm;

S5, performing intelligent algorithm man-machine countermeasure verification;

S6, carrying out multiple iterations on the unmanned aerial vehicle air combat autonomous decision algorithm, and repeating the steps S3 to S6.

In the aspect and any possible implementation manner as described above, there is further provided an implementation manner, where the S1 specifically includes: aiming at the operation mode of the unmanned aerial vehicle cluster, the fight site of unmanned aerial vehicle fight, equipment configuration of both sides, fight purposes of both sides and fight task elements in different stages are designed; and (3) carrying out design modeling on environmental elements influencing the fight result at the fight site, and setting up the fight limit conditions of the two parties.

In the aspect and any possible implementation manner as described above, there is further provided an implementation manner, where the S2 includes: and carrying out parameter modeling on unmanned planes, manned fighters, reconnaissance early warning machines and carried air-to-air missile weapons and detection loads which directly participate in the fight in the scene, and carrying out configuration on fight rules on other models in the scene.

In the aspect and any possible implementation manner as described above, there is further provided an implementation manner, where the S3 includes: and (3) establishing an unmanned aerial vehicle cluster control algorithm, driving the unmanned aerial vehicle and an enemy to perform air combat in a simulation environment, setting simulation data acquisition points and loss functions, and establishing an algorithm iterative learning rewarding mechanism.

In the foregoing aspect and any possible implementation manner, there is further provided an implementation manner, where the S4 specifically includes: and constructing an intelligent algorithm training hardware cluster and an algorithm and simulation parallel training frame, carrying out task allocation on the intelligent algorithm training hardware cluster after carrying out multiple instantiations on training tasks, and carrying out overall algorithm parallel training tasks.

In the foregoing aspect and any possible implementation manner, there is further provided an implementation manner, where the S5 specifically includes: and performing air combat fight comparison in a simulation scene through a hardware-controlled man-machine simulation model and an algorithm-controlled unmanned plane model, and testing the combat effectiveness of the algorithm under different opponents and different combat conditions.

In the foregoing aspect and any possible implementation manner, there is further provided an implementation manner, where the S6 specifically includes: correcting and adjusting the intelligent algorithm of the S3 by adopting the results of the S4 and the S5, and then repeating the training of the algorithm of the S4 to finish algorithm convergence; and S5, verifying and evaluating the operational effectiveness of the algorithm is repeated until the algorithm meets the design requirement.

The invention also provides a simulation training system for unmanned aerial vehicle cluster evolution, which is used for realizing the method and specifically comprises the following steps:

the first modeling module is used for designing and modeling an unmanned aerial vehicle air combat scene;

the second modeling module is used for modeling an unmanned aerial vehicle air combat equipment model;

the construction module is used for constructing an unmanned aerial vehicle air combat autonomous decision algorithm;

The simulation iteration training module is used for carrying out simulation iteration training on the unmanned aerial vehicle air combat autonomous decision algorithm;

the verification module is used for performing intelligent algorithm man-machine countermeasure verification;

and the iteration module is used for carrying out multiple iterations on the unmanned aerial vehicle air combat autonomous decision algorithm.

The invention also provides a computer storage medium having stored thereon a computer program for execution by a processor to perform the method.

The invention also provides an electronic device, which comprises:

A memory storing executable instructions;

A processor, the processor

Executing the executable instructions in the memory to implement the method.

The beneficial effects of the invention are that

Compared with the prior art, the invention has the following beneficial effects:

The unmanned aerial vehicle cluster evolution simulation training method comprises the following steps: designing and modeling an unmanned aerial vehicle air combat scene; modeling an unmanned aerial vehicle air combat equipment model; constructing an unmanned aerial vehicle air combat autonomous decision algorithm; performing simulation iterative training on an unmanned aerial vehicle air combat autonomous decision algorithm; performing intelligent algorithm man-machine countermeasure verification; and carrying out multiple iterations on the unmanned aerial vehicle air combat autonomous decision algorithm, and repeating the following four steps. According to the invention, simulation of the unmanned aerial vehicle air combat scene is realized by a simulation means, the algorithm evolution is advanced by adopting a parallel simulation node mode, and the algorithm multi-round iterative training is realized. Taking the current mainstream 30-series and 40-series deep learning operation servers as an example, a typical air combat 2V2 countermeasure scene of 20 minutes is operated, 8-24 simulated countermeasures can be simultaneously carried out in a simulation training environment, tens of thousands of countermeasure combat data can be generated in one day, the whole round of optimization, comparison and verification of an algorithm can be completed in 3-4 hours, and under the condition that the algorithm optimization technology is mature, intelligent algorithm development of an unmanned aerial vehicle or the combat scene can be completed in one week.

Drawings

FIG. 1 is a flow chart of simulation evolution training and verification of an unmanned cluster cooperative air combat autonomous decision algorithm of the invention;

fig. 2 is a training flow chart of an unmanned cluster cooperative air combat autonomous decision algorithm.

Detailed Description

For a better understanding of the present invention, the present disclosure includes, but is not limited to, the following detailed description, and similar techniques and methods should be considered as falling within the scope of the present protection. In order to make the technical problems, technical solutions and advantages to be solved more apparent, the following detailed description will be given with reference to the accompanying drawings and specific embodiments.

It should be understood that the described embodiments of the invention are only some, but not all, embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

The terminology used in the embodiments of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in this application and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

The method of the invention comprises the following steps: step one, designing and modeling an unmanned aerial vehicle air combat scene, which specifically comprises the following steps:

And designing combat design elements such as combat sites of unmanned aerial vehicle combat, equipment configuration of both sides, combat purposes of both sides and combat tasks of different stages according to the operation mode of the unmanned aerial vehicle cluster. The design modeling is performed for environmental elements such as meteorological environment, geographic environment, electromagnetic field environment and the like which may have influence on the air combat result in the combat place. And setting the limit conditions of the two parties, such as the area range of the air combat, the winning conditions of the two parties, an intelligent algorithm evaluation mechanism and the like.

Modeling the unmanned aerial vehicle air combat equipment model, specifically:

And respectively carrying out parameter modeling on detection loads of unmanned aerial vehicles, manned fighters, reconnaissance early warning machines, carried air-to-air missiles and the like, photoelectricity, radars and the like which directly participate in fight in a scene, configuring fight rules for models except for unmanned aerial vehicles and self-carried weapons and loads, simulating command control systems, and driving equipment to act in a simulation environment.

Thirdly, constructing an unmanned aerial vehicle air combat autonomous decision algorithm, which specifically comprises the following steps:

the unmanned aerial vehicle cluster control algorithm is established by adopting deep learning, reinforcement learning, game countermeasure and other architectures, the unmanned aerial vehicle and the enemy are driven to perform air combat in a simulation environment, and a reward mechanism for algorithm iterative learning is established by setting proper simulation data acquisition points and loss functions.

Developing algorithm simulation iterative training, which specifically comprises the following steps:

And constructing an intelligent algorithm training hardware cluster by constructing a deep learning cloud computing server, adopting a container technology and the like to construct an algorithm and simulation parallel training framework, carrying out multiple instantiations on training tasks, carrying out task allocation on the intelligent algorithm training hardware cluster through hardware resource management software, and carrying out overall algorithm parallel training tasks.

Step five, developing intelligent algorithm man-machine countermeasure verification, specifically:

The unmanned aerial vehicle model controlled by the unmanned aerial vehicle simulation model and the algorithm controlled by the flight simulator, the controller and other hardware is used for performing air combat fight in a simulation scene, or an intelligent control module is added for the blue square model to perform simulated countermeasure, and the combat effectiveness of the algorithm under different opponents and different combat conditions is tested.

Step six, carrying out algorithm multi-round iteration, and repeating the step three to the step six, wherein the method specifically comprises the following steps:

Correcting the intelligent algorithm in the third step through the results in the fourth step and the fifth step, and adjusting the algorithm structure, algorithm parameters and the like; then repeating the fourth step to train the algorithm, and completing algorithm convergence; and step five, verifying and evaluating the operational effectiveness of the algorithm until the algorithm meets the design requirement.

Specific examples are given below for illustration: an evolution type simulation training method for an unmanned aerial vehicle cluster collaborative air combat autonomous decision algorithm is shown in fig. 1, and comprises the following steps:

Step one, designing and modeling an unmanned aerial vehicle air combat scene, which specifically comprises the following steps:

Aiming at the fight requirement of an advanced manned fighter plane, the fighter scene design is developed. The direct participation combat equipment in the set scene is the detection loads of weapons such as red unmanned plane, blue advanced fighter, red reconnaissance early warning machine, air-to-air missile and the like, photoelectricity, radar and the like carried by the red advanced fighter and the red reconnaissance early warning machine. Setting the number of blue advanced fighters as X frames, and setting the number of red certain empty unmanned aerial vehicles as Y frames according to the fighter performance of the unmanned aerial vehicles and the basic parameters of the unmanned aerial vehicles. According to the mission task of the red and blue party, the battle site is set as a sea area. Based on the intention of both red and blue, the load detection capability, and the like, the area engagement, the engagement distance, the combat altitude, and the like are confirmed. And setting equipment configuration, combat tasks, basic rules and initial positions of the two parties in simulation software according to combat background. The blue party fight victory condition is set to break through the red party limit, and the air striking of a certain target or the complete fight of the red party unmanned aerial vehicle is completed. The red party fight victory condition is set as the blue party is blocked to break through by man, the blue party is completely killed or the blue party is promoted to achieve the return condition (fuel oil condition, ammunition condition and the like). Modeling the meteorological environment near the combat area, and inputting the modeled meteorological environment into a simulation system for simulating the effect of the atmosphere on the maneuvering performance of the aircraft after loading the aerodynamic characteristics of the unmanned aerial vehicle platform. And carrying out the design development of the scene design result to form a basic design with the capability of simulation operation and deduction.

Step two: all the equipment involved in the design and its components are modeled, in particular:

(1) Flight platform modeling

The unmanned plane model comprises an unmanned plane platform, an advanced fighter plane platform, an early warning plane platform and the like, wherein the model comprises dynamic characteristics, pneumatic characteristics, a navigation model, a flight control model and the like, a model frame is built through Matlab and Simulink software, a six-degree-of-freedom motion equation of an airplane is built, and model data are filled through real flight test data, wind tunnel test data, simulation test and the like.

(2) Weapon modeling

The method comprises the steps of modeling an air-air missile carried by an advanced unmanned aerial vehicle and an unmanned aerial vehicle, wherein the model comprises a fire control model, a weapon motion model, a weapon emission model, a weapon guidance model, a weapon damage model and the like, is used for carrying out simulation on each stage of weapon emission, flight, tracking, damage and the like, constructing a basic weapon model by adopting a simulation platform general air-air weapon frame, and modifying weapon characteristics, such as an emission mode, a preparation mode, an emission track and the like, according to the characteristics of each weapon.

(3) Sensor modeling

The intelligent sensor system comprises an onboard radar, an infrared sensor, a visible light sensor, a laser radar and the like of an advanced unmanned aerial vehicle, an unmanned aerial vehicle and a reconnaissance early warning machine, is used for simulating the detection capability of the unmanned aerial vehicle and the advanced fighter in the fight process, influencing information acquisition, fight strategies and the like of both red and blue parties, constructing a basic sensor model by adopting a simulation platform general sensor frame, and respectively correcting according to the working mode, a holder structure, an identification algorithm, imaging precision and the like of each type of sensor.

(4) Communication link modeling

The simulation platform universal communication link framework is used for constructing a basic model, and parameter modification is carried out according to the real-machine communication link mode and characteristics.

(5) Combat rule modeling

The configuration of the combat rules, including the remote hitting strategy outside the visual range, the close dog fight strategy, the return strategy, the weapon firing strategy and the like, is needed for the models except the combat unit platform and the self carried weapons, loads and links, and the behavior and action control is carried out on equipment except the unmanned aerial vehicle in the simulation through the strategies. The rules include the relative attributes of task orders, such as type, time, target, requirement, etc., and the task is divided into one or more actions of executing equipment with different time sequences according to the battlefield environment, self state, such as weather condition, topography condition, etc. and the specific mode of completing the task according to the corresponding task attributes, and then the actions are defined according to the rules to form a series of action orders corresponding to the executing equipment

Step three: the unmanned aerial vehicle air combat autonomous decision algorithm is constructed, and specifically comprises the following steps:

(1) Flight control algorithm construction

And constructing a flight control algorithm, and dividing the complete flight process of the air combat unmanned aerial vehicle into a take-off stage, a conventional flight stage, a maneuvering flight stage and a recovery stage. A flight process design comprising: conventional flight phase design, along a planned three-dimensional track, reaching a specified position at a specified time; in the maneuver flight stage, maneuver flight comprises various maneuver actions such as turning, spiraling, jumping, diving, fighting and semi-fighting. And controlling the mode switching, controlling transient responses generated during the mode switching of different flight phases, and completing the conversion of the control modes by adopting a synchronous tracking conversion desalination device, so that the output of the controller is not jumped. Finally, inputting a flight digital simulation system, and constructing the digital simulation system in an MATLAB environment according to the unmanned aerial vehicle model and the control structure

(2) Intelligent decision algorithm construction

And the target allocation algorithm allocates one or more enemy robots or target points for each unmanned aerial vehicle in the unmanned aerial vehicle cluster, so that all enemy targets can be attacked and the overall attack efficiency reaches the global optimum.

And (3) strategy and path planning of the emergent prevention approaching tactics, namely planning a strategy of cooperative approaching of a plurality of unmanned aerial vehicles to an enemy target and the emergent prevention path based on a target distribution result, realizing tactical maneuver such as approaching of the unmanned aerial vehicles to the target, avoiding and the like, and realizing cooperative tactics such as detouring, hunting and ammunition trapping.

The multi-agent reinforcement learning is to model the unmanned aerial vehicle cluster game countermeasure problem into a multi-agent game, and a centralized training distributed execution framework is adopted to effectively solve the problem of multi-agent reinforcement learning combined action space combination explosion.

Step four: the algorithm simulation iterative training is carried out, specifically:

As shown in fig. 2, an intelligent algorithm training hardware cluster is constructed by constructing a deep learning cloud computing server, and a plurality of containers are automatically organized according to training tasks in a manner of container management based on Kubernetes and the like to form an abstract resource set, so that resources such as computation, storage and the like are provided for the training tasks. And (3) adopting a container technology based on a Docker to container, deploy and schedule the simulation environment, and realizing parallel super-real-time operation of a plurality of simulation nodes. And each algorithm iteration step length realizes the complete operation of a simulation scene, the algorithm is converged for one round through a simulation result, and the algorithm training iteration process is repeated until the loss function is converged. Meanwhile, an evaluation component is arranged to evaluate and monitor the algorithm learning process, and a data management component is arranged to store and record the results and events of the algorithm iterative process so as to support the analysis and playback of the subsequent training process.

Step five: developing intelligent algorithm man-machine countermeasure verification, specifically:

And verifying the operational efficiency of the algorithm on a real battlefield by adopting a man-machine countermeasure mode: an advanced fighter plane flight simulator is built. The retired pilot with air combat experience is used for operating the simulation simulator, the unmanned aerial vehicle driven by the man-machine model and the intelligent algorithm in the simulation scene is driven to fight against the air combat, the combat efficiency of the intelligent decision algorithm of the unmanned aerial vehicle is tested by trying different combat modes and tactical combat methods of the advanced unmanned aerial vehicle, and meanwhile, the combat process is visually displayed through situation display software, so that the combat situation is conveniently and rapidly analyzed. After the challenge verification is completed, the operational effectiveness of the algorithm is systematically evaluated.

Step six: carrying out algorithm multi-round iteration, and repeating the third to sixth steps, wherein the method specifically comprises the following steps:

And correcting the intelligent algorithm in the third step through the results in the fourth step and the fifth step, wherein the model algorithm of the system adopts algorithm model libraries with different granularities such as a neural network model library, an intelligent algorithm library, a game training method library and the like, and utilizes different network structures such as a full-connection network (MLP), a long-short-term memory network (LSTM), a Convolutional Neural Network (CNN), a cyclic neural network (RNN), a feature conversion network (Transformer), a attention mechanism and the like provided in the neural network model library to construct a neural network model of the algorithm to realize the algorithm design with high degree of freedom according to algorithm requirements. Different algorithms such as a depth Q value network learning algorithm (DQN), a distributed priority experience playback algorithm (APE-X), a depth deterministic strategy gradient algorithm (DDPG), a synchronous near-end optimization algorithm (PPO), an asynchronous near-end optimization algorithm (A-PPO), a multi-agent monotonic value function decomposition algorithm (QMIX) and the like can also be directly utilized. And then repeating the fourth step to train the algorithm, and completing algorithm convergence. And step five, verifying and evaluating the operational effectiveness of the algorithm until the algorithm meets the design requirement.

As an embodiment of the disclosure, the present invention further provides an unmanned aerial vehicle cluster evolution simulation training system, where the system is configured to implement the method, and specifically includes:

the first modeling module is used for designing and modeling an unmanned aerial vehicle air combat scene;

the second modeling module is used for modeling an unmanned aerial vehicle air combat equipment model;

the construction module is used for constructing an unmanned aerial vehicle air combat autonomous decision algorithm;

The simulation iteration training module is used for carrying out simulation iteration training on the unmanned aerial vehicle air combat autonomous decision algorithm;

the verification module is used for performing intelligent algorithm man-machine countermeasure verification;

and the iteration module is used for carrying out multiple iterations on the unmanned aerial vehicle air combat autonomous decision algorithm.

As an embodiment of the present disclosure, the present disclosure further provides a computer storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the method.

As an embodiment of the present disclosure, the present disclosure further provides an electronic device, including:

A memory storing executable instructions;

A processor, the processor

Executing the executable instructions in the memory to implement the method.

While the foregoing description illustrates and describes the preferred embodiments of the present invention, it is to be understood that the invention is not limited to the forms disclosed herein, but is not to be construed as limited to other embodiments, and is capable of numerous other combinations, modifications and environments and is capable of changes or modifications within the scope of the inventive concept as expressed herein, either as a result of the foregoing teachings or as a result of the knowledge or technology of the relevant art. And that modifications and variations which do not depart from the spirit and scope of the invention are intended to be within the scope of the appended claims.