CN107918564B - Data transmission exception handling method and device, electronic equipment and storage medium - Google Patents

Detailed Description

Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The same reference numerals denote the same or similar parts in the drawings, and thus, a repetitive description thereof will be omitted.

Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to provide a thorough understanding of embodiments of the invention. One skilled in the relevant art will recognize, however, that the invention may be practiced without one or more of the specific details, or with other methods, components, materials, devices, steps, and so forth. In other instances, well-known structures, methods, devices, implementations, materials, or operations are not shown or described in detail to avoid obscuring aspects of embodiments of the invention.

The block diagrams shown in the figures are functional entities only and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in the form of software, or in one or more software-hardened modules, or in different networks and/or processor devices and/or microcontroller devices.

In this example embodiment, first, a data transmission exception handling method is provided, which may be applied to a plurality of business systems for performing data export and import operations through a database environment, and as shown in fig. 1, the method may include:

in step S110, exporting target data in a first server at regular time by an export script file in the first server, and determining whether the export is abnormal;

in step S120, when it is determined that the export is abnormal, a temporary export script file stored in the first server and a temporary import script file stored in a second server are generated;

in step S130, exporting the target data again through the temporary export script file, and generating a temporary file according to the exported target data;

in step S140, when it is determined that the generation of the temporary file is completed, the target data is imported to the second server through the temporary import script file.

According to the data transmission exception handling method provided by the embodiment of the invention, on one hand, the data with the exception of transmission is handled by generating the temporary export script and the temporary import script, so that the workload is reduced, and the processing efficiency is further improved; on the other hand, the condition that errors occur when the abnormity is processed manually is avoided, and the accuracy is improved.

Next, the steps of the data transmission abnormality method in the embodiment of the present invention will be explained and explained in detail with reference to fig. 1 to 4.

In step S110, the target data in the first server is exported at regular time by the export script file in the first server, and whether the export is abnormal is determined.

In this example, the first server may be understood as a data source server for storing target data to be exported, and there may be a plurality of the first servers. The target data may be stored in the first server in the form of a data table or other form. The target data may be, for example, all sales data of each day, video download data within a certain number of hours of each day, etc., and may be specifically determined according to actual needs, and in this example, all sales data of each day are taken as an example for explanation. The export script file may include, for example, SQL (Structured Query Language) script, Insert script, and the like, and the target data in the first server may be exported periodically by executing the original data export script file in the first server. The timing here may be understood as, for example, deriving the target data at the same preset time of day, which may be, for example, 3 o' clock of day or any time of day. In addition, the target data exported from the first server may be generated into a text file stored in the target server, so as to transmit or import the target data into a database of a required target server for storage or organization.

When the target data in the first server is exported at regular time through the export script file in the first server, the data transmission state between the data servers can be monitored in real time every day through the monitoring module or the monitoring device, so as to judge whether the export process of the target data is abnormal. Derived exceptions are herein understood to mean that the fetch is empty due to batch processing delays of the traffic system, adjustments of the traffic nodes, network exceptions, and other exceptional conditions. The monitoring module may comprise, for example, a monitoring program or function. For example, the target data normally derived may be in a text format in the form of characters or in a compressed format in the form of binary data; in an abnormal situation, no data in any form is present in the exported TXT file. In step S120, when it is determined that the export is abnormal, a temporary export script file stored in the first server and a temporary import script file stored in the second server are generated.

In this example, when the monitoring module determines that the target data export process is abnormal, a temporary export script file may be generated by remotely generating a T-SQL temporary file via the SSH protocol at once, where the temporary export script file is stored in the first server. Further, the fetch range of the T-SQL temporary file may be modified to modify the fetch range of the temporary export script file. The temporary export script file is in the same form as the export script file, and is obtained by calling the T-SQL temporary file or the T-SQL file to the database server and stored in the second server. In addition, the timed task program originally used for executing the export script file and the import script file can be copied to generate a temporary timed task program, and the time stamp of the temporary timed task program is modified to the time point of the abnormal condition. After that, the temporary import script file and the temporary export script file are executed by the newly generated temporary timed task program. Through the modification process, the access time of the temporary export script file and the temporary import script file can be set as the time of the abnormal situation. The second server is understood herein as a target server that needs to import target data in the first server.

In step S130, the target data is exported again by the temporary export script file, and a temporary file is generated according to the exported target data.

After the timestamp of the temporary timed task program is modified, the temporary export script file can be executed by the temporary timed task program to export the target data again. Meanwhile, whether the export process is abnormal or not can be continuously judged through the monitoring module, if the export process is not abnormal, a temporary file can be generated in the first server according to the exported target data, and then the temporary file is transmitted to the second server. The temporary file may here be used for storing the target data exported from the first server and may for example be in TXT format.

If the export process is abnormal, the method in step S120 may be repeated, the temporary export script file stored in the first server, the temporary import script file stored in the second server, and the temporary timing task program are generated again, then the timestamp for modifying the temporary timing task program is modified to the time when the abnormality occurs, then the temporary export script file is executed by the temporary timing task program to export the target data again, and the temporary file is generated again according to the exported target data.

In the process of monitoring data transmission in real time, if a data export abnormality is found, the step in step S120 is executed again immediately, a temporary export script file and a temporary import script file are generated, and the target data is exported again until no abnormality occurs in the export process. The specific steps of monitoring for anomalies and repeatedly executing the derivation process can be performed by a function or a written program, which is not particularly limited in this example. By the method for exporting the target data by repeatedly generating the temporary export script file and the temporary import script file when the data transmission is abnormal, the accuracy of the data can be ensured to the maximum extent and the reliability of the data transmission can be improved.

Next, in step S140, when it is determined that the generation of the temporary file is completed, the target data is imported to the second server by the temporary import script file.

When the temporary file is generated, whether the temporary file is generated or not can be judged in real time through software or a scanning tool, and the scanning tool can be a file scanner. When the temporary file is judged to be generated completely, the temporary import script file stored in the second server can be executed, and the temporary file is imported into the second server database. Whether the temporary file is generated or not is judged, so that the problem that data cannot be completely transmitted due to the fact that the temporary file is incomplete can be solved, and the reliability of the data transmission process can be improved.

In detail, after the T-SQL temporary file is generated, the timestamp of the temporary timed task program may be modified to be the time of the occurrence of the exception, then the access range of the T-SQL temporary file may be modified, and further the temporary import script file is executed by the temporary timed task program to import the target data again.

Fig. 2 is a schematic diagram illustrating generation of a temporary export script file and a temporary import script file according to an embodiment of the present invention. With reference to fig. 2, step S120 may exemplarily include:

in step S210, the export script file is copied to generate the temporary export script file, and the import script file is copied to generate the temporary import script file.

The export script file may first be copied and stored in the first server to generate a temporary export script file, and the import script file in the second server may be copied and stored in the second server to generate a temporary import script file. In the copying process, the T-SQL file called by the export script file is also copied, and a T-SQL temporary file is generated. The T-SQL file records access logic and an access range and is used for executing main tasks of exporting the script file.

And subsequently, modifying the access range of the T-SQL temporary file in the temporary export script file, thereby modifying the access range of the temporary export script file.

In step S220, the copy timed task program generates a temporary timed task program, and modifies the timestamp of the temporary timed task program to the occurrence time of the abnormality.

The data transmission process is realized by executing the export script file and the import script file in a timing mode through a timing task program. The time stamp of the timed task program controls the start time of the data transfer process. Therefore, the time of the next access when the transmission exception occurs can be modified by generating a temporary timing task program. For example, normally, the timestamp of the timed task program is 3: 00, starting the export script file in the first server to export target data at 3 am every day. If export abnormity is found at 3 o 'clock and 10 o' clock, a timed task program can be immediately copied to generate a new temporary timed task program, and the time stamp of the temporary timed task program is automatically modified by 3 o 'clock and 10 o' clock, namely, when abnormity occurs, the temporary timed task program is immediately generated and executed, so that the restart time of the temporary export script file and the temporary import script file is modified.

By automatically modifying the starting time of the temporary export script file and the temporary import script file into the time of the export exception, the exception handling efficiency can be improved, and the integrity of data can be ensured to the maximum extent.

Fig. 3 schematically shows a schematic diagram of determining whether the generation of the temporary file is completed. Referring to fig. 3, the step S140 may exemplarily include a step S310 and a step S320, wherein:

in step S310, the time of the temporary file generation and the time of the occurrence of the export exception are compared.

Specifically, if the monitoring module monitors that the data transmission is abnormal, the abnormal time can be immediately transmitted to the file scanner in the second server, and the file scanner is executed to judge the generation time of the temporary file. The file scanner may be, for example, a separate code, which may be part of the import script file. In this example, the file scanner searches the export file generated by the temporary export script on the second server in a full text manner, and obtains attribute information of the file, for example, the generation time of the temporary file and the occurrence time of the export exception may be compared to determine whether the generation time of the temporary file and the occurrence time of the export exception are consistent.

In step S320, when the time of generating the temporary file is consistent with the time of the occurrence of the export exception and the size of the temporary file remains unchanged within a preset time period, it is determined that the generation of the temporary file is completed.

When the generation time of the temporary file is judged to be consistent with the time of the export abnormity, whether the generation of the temporary file is finished or not can be further judged by combining the change condition of the temporary file within the preset time length. The change of the temporary file may include, for example, a change in the size of the temporary file or a change in the number of temporary files. For example, when it is determined that the generation time of the temporary file matches the generation time of the export exception, the generation time of the export exception is 3: 10 minutes, and the generation time of the temporary file is 3: 10 minutes. If the number or the size of the temporary files in the preset time length is kept unchanged, for example, the size of the temporary files is 200M in the preset time length, it can be considered that no new data is imported in the preset time length, and at this time, it can be determined that the generation of the temporary files is finished. In this example, the preset time period may be, for example, 1 minute or 5 minutes, and may be determined according to an empirical value or other conditions.

In this example, the generation of the temporary file is determined by determining whether the generation time of the temporary file is consistent with the time of occurrence of the export exception and determining the generation of the temporary file according to the change condition of the temporary file, so that the integrity of the data can be ensured to the maximum extent.

In addition, in order to facilitate the transmission of the target data among the plurality of servers, before monitoring the export of the target data in the first server by the export script file timing in the first server, the method may further include:

and acquiring the authority of accessing the first server through a public key and a private key.

In general, a public key is typically used to encrypt a session key, verify a digital signature, or encrypt data that may be decrypted with a corresponding private key. The key pair derived by such an algorithm can be guaranteed to be unique worldwide. When using this key pair, if one of the keys is used to encrypt a piece of data, the other key must be used to decrypt the piece of data. For example, encrypting data with a public key necessitates decryption with the private key, and if encrypting with the private key, also must decrypt with the public key, otherwise decryption will not succeed.

In this example, a user or a program developer may first establish a public key and a private key corresponding to the second server, and further, the second server may implement data import according to the established public key and private key, so as to ensure security of data transmission. Specifically, the obtaining of the right to access the first server through the public key and the private key may include:

and sharing the public key to the first server after the public key and the private key corresponding to the second server are newly established.

Firstly, developers can open an SSH (struts + spring + hibernate) integrated service framework between the second server and each first server, so that the second server can perform password-free access between the servers through public keys and private keys between the servers. SSH is a network protocol in which public key encryption is employed and may be used for encrypted login between computers. If a user logs on to another remote computer from a local computer using the SSH protocol, we can consider that the login is secure and the password will not be revealed even if it is intercepted halfway. The whole process can be described as follows: the remote host receives a login request of a user and sends a public key of the remote host to the user; the user uses the public key to encrypt the login password and send the login password back; the remote host decrypts the login password by using the private key of the remote host, and if the password is correct, the remote host agrees to login of the user.

In particular, the ssh-keygen command may be executed on the second server to recreate the public and private keys of the second server. And after the new establishment is finished, the user can access the ssh under the user root directory to check whether the new establishment of the public key and the private key is finished. If there is no exception, there will be two files: id _ rsa and id _ rsa.pub, where id _ rsa is the private key and id _ rsa.pub is the public key. Next, the second server may use the scp command in Linix to perform file replication, so as to share the newly-created public key id _ rsa.

Through the steps, the second server can log in the plurality of first servers without passwords, and can remotely execute the script files corresponding to the first servers and check the file information without passwords, so that the data security is ensured, the data source server can be quickly accessed, and the automatic processing method for the data transmission abnormity is realized through the precondition.

On this basis, importing the target data into the second server in step S140 may include:

and executing the temporary import script file through the public key so as to import the target data of the first server into the second server.

After the temporary file is judged to be generated completely, the second server can remotely and automatically execute the generated temporary import script file through the shared public key, and then import the temporary file into a database of the second server so as to realize the import of the target data. By sharing the public key to execute the temporary import script file, the automatic processing can be realized, the cost is saved, and the system safety is improved.

In addition, after the target data is imported to the second server by the temporary import script file, the method may further include:

deleting the temporary export script file and the temporary import script file; and

and backing up and deleting the temporary file.

After the export and import of the target data are completed, the temporary export script file and the temporary import script file which are generated when the transmission is abnormal can be automatically deleted through the temporary task script recycling station, so that the temporary script files are prevented from occupying a large amount of space; meanwhile, the generated temporary files can be backed up to protect the original files from being influenced, and after the temporary files are backed up, the temporary files can be automatically deleted to avoid the temporary files occupying a large amount of space.

It should be noted that, if no exception occurs in data export after the temporary export script file and the temporary import script file are generated for the first time, the temporary export script file and the temporary import script file can be directly deleted; if it is detected that data export is abnormal after the temporary export script file and the temporary import script file are generated for the first time, the temporary export script file and the temporary import script file may be deleted and then regenerated according to the method in step S120.

Fig. 4 schematically shows a flow chart of the data transmission exception handling method. Referring to fig. 4, the data transmission exception handling method may specifically include the following steps:

in step S410, the database server a remotely logs in each data source server without a password, and remotely executes a corresponding script file and view file information in the data source server. The specific process can include:

s411, a developer opens SSH authority between a database server A and a data source server B1;

s412, establishing a public key and a private key of the database server A;

s413, the database server A shares the public key thereof through the scp command;

s414, the database server a accesses the data source server B1. The data source server B1, Bn is the first server in the above step, and the database server a is the second server in the above step.

In step S420, a timed import data task is triggered, for example, when the database server a accesses any one or more of the data source servers B1.

In step S430, whether the data transmission is abnormal is monitored in real time, for example, whether the derived target data is empty can be monitored in real time by a monitoring module including a monitoring program or function, so as to determine whether the timing derivation process is abnormal.

Next, in step S440, when the transmission of the monitoring data is abnormal, the time for deriving the abnormal is transmitted to the file scanner through the trigger, and the file generation time is determined by executing the file scanner; the file scanner may be stored in the second server, for example, as part of the import script file, or may be a separate applet.

In step S450, the database server a is triggered to execute the data transfer exception handling method. The specific process may include:

s451, text full-text search is conducted based on the POI library, for example, the database server A remotely executes an export script file of each data source server through SSH service, and the export script file searches a T-SQL file used for exporting data according to a feature of interest (POI) full-text. The interesting features can also be called a set of mark points, and can be used for describing parts needing to be modified and replaced in an original export script file, for example, a time stamp in a T-SQL file needs to be replaced by the time of the export exception of the current day to generate a T-SQL temporary file.

S452, generating a temporary script file; for example, the temporary export script file is generated by copying according to the rules configured in the script and generating a correct T-SQL temporary file. And simultaneously, a timing task program originally used for executing the export script file and the import script file can be copied to generate a temporary timing task program, and the time stamp of the temporary timing task program is modified to the time point of the abnormal condition, so that the starting time of exporting data again and importing data is determined. After that, the newly generated temporary import script file and temporary export script file are executed by the newly generated temporary timed task program, and the target data is exported and imported. Through the above modification process, the fetch time of the temporary import script can be set to the time when the abnormal situation occurs for executing the export and import processes of the target data again.

In this example, generating a new T-SQL temporary file and a temporary script file can protect the integrity of the original file and the original data, and can prevent risks caused by modifying the file, thereby improving the security of the file.

S453, the fetch task is triggered, i.e., database server a may remotely execute each of the temporary export script files of data source server B1. For example, when the target data is successfully exported by executing or starting the original export script file in the data source server at 3 o 'clock on the first day, and export abnormality is found at 3 o' clock and 10 min on the second day, all data between 3 o 'clock on the first day and 3 o' clock and 10 min on the second day needs to be exported.

In combination with the above steps, the time for next access when the transmission is abnormal can be modified by generating a temporary timing task program. For example, normally, the timestamp of the timed task program is 3: 00, starting the export script file in the first server to export target data at 3 am every day. If export abnormity is found at 3 point and 10 minutes, the timed task program can be immediately copied to generate a new temporary timed task program, and the time stamp of the temporary timed task program is automatically modified by 3 point and 10 minutes, namely, the temporary timed task program is immediately generated and executed when the abnormity occurs, so that the starting time of the temporary export script and the temporary import script is modified, and the target data is exported and imported again.

S454, exporting the data to the temporary file, wherein the temporary file can define and modify the access range, so that the data can be ensured to be exported correctly.

And then the data export process and the import process of the target data are completed by executing S414, namely the database server accesses the data source server through the public key and triggering the timed import data task S420.

And S455, automatically deleting the exported data file through the temporary timing task script recycle bin, and avoiding that the generated temporary script file and the temporary file occupy a large amount of storage space.

Fig. 4 illustrates a process of handling a data transmission exception when target data in a data source server is imported to a database server. It should be noted that the data in the multiple data source servers may also be exported at the same time, and all the exporting methods, the transmission exception monitoring methods, and the transmission exception handling methods are the same, and all the exporting methods, the transmission exception monitoring methods, and the transmission exception handling methods may be implemented by using programs written in Python language, Shell language, or other languages in the Linux system, and the specific processes are not described herein again.

In summary, the data transmission exception handling method provided in the embodiment of the present invention improves the efficiency of data transmission exception handling by generating the temporary export script file, the temporary import script file, and the temporary file to handle the data of the transmission exception; by automatically modifying the temporary export script file and the temporary import script file, the condition of errors when abnormity is processed manually is avoided, and the accuracy is improved; in addition, in the process of monitoring data transmission in real time, as long as data export abnormity is found, the temporary export script file and the temporary import script file are immediately regenerated, and target data is exported again until abnormity does not occur in the export process, so that the reliability of data transmission can be improved; the generation of the temporary file is judged by judging whether the generation time of the temporary file is consistent with the time of the export abnormity and the change condition of the temporary file, so that the integrity of the data can be ensured to the maximum extent.

It should be noted that, the specific details of each module in the data transmission exception handling apparatus have been described in detail in the corresponding data transmission exception handling method, and therefore are not described herein again.

In an exemplary embodiment of the present invention, an electronic device capable of implementing the above method is also provided.

As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, method or program product. Thus, various aspects of the invention may be embodied in the form of: an entirely hardware embodiment, an entirely software embodiment (including firmware, microcode, etc.) or an embodiment combining hardware and software aspects that may all generally be referred to herein as a "circuit," module "or" system.

Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein may be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solution according to the embodiment of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a usb disk, a removable hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiment of the present invention.

In an exemplary embodiment of the embodiments of the present invention, there is also provided a computer-readable storage medium having stored thereon a program product capable of implementing the above-described method of the present specification. In some possible embodiments, aspects of the invention may also be implemented in the form of a program product comprising program code means for causing a terminal device to carry out the steps according to various exemplary embodiments of the invention described in the above section "exemplary methods" of the present description, when said program product is run on the terminal device.

The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

A computer readable signal medium may include a propagated data signal with readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable signal medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

Program code embodied on a readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).

Furthermore, the above-described figures are merely schematic illustrations of processes involved in methods according to exemplary embodiments of the invention, and are not intended to be limiting. It will be readily understood that the processes shown in the above figures are not intended to indicate or limit the chronological order of the processes. In addition, it is also readily understood that these processes may be performed synchronously or asynchronously, e.g., in multiple modules.

It should be noted that although in the above detailed description several modules or units of the device for action execution are mentioned, such a division is not mandatory. Indeed, the features and functionality of two or more modules or units described above may be embodied in one module or unit, according to an implementation of an embodiment of the invention. Conversely, the features and functions of one module or unit described above may be further divided into embodiments by a plurality of modules or units.

Other embodiments of the invention will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the embodiments of the invention following, in general, the principles of the embodiments of the invention and including such departures from the present disclosure as come within known or customary practice within the art to which the embodiments of the invention pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the embodiments of the invention being indicated by the following claims.

It is to be understood that the embodiments of the present invention are not limited to the precise arrangements described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of embodiments of the invention is limited only by the appended claims.