CN114090366A - Method, device and system for monitoring data - Google Patents

The invention discloses a method, a device and a system for monitoring data, and relates to the technical field of computers. A specific implementation mode of the method comprises the steps that a server receives index data collected by a client, and the read-write operation of a third-party time sequence database is executed by utilizing a uniform interface so as to monitor the storage and the reading of the index data, so that the defect that the existing system writes the third-party time sequence database in a one-way mode is overcome, the efficiency of monitoring the data is improved, and the fluidity and the utilization rate of the index data are improved; by determining the global service label, the problem of poor correlation of the collected index data caused by non-standard service labels is solved; and the received index data is put into a message queue, so that the problem of high concurrency of mass data is solved.

Method, device and system for monitoring data

Technical Field

The present invention relates to the field of computer technologies, and in particular, to a method, an apparatus, and a system for monitoring data.

Background

The Prometheus system is used as a set of monitoring system to provide a complete data monitoring solution, and is widely deployed and applied in the field of data monitoring due to the ecological openness and the flexibility of multiple components.

In the process of implementing the invention, the inventor finds that at least the following problems exist in the prior art:

the metrics data received by the Prometheus system is stored in a local storage medium, which makes mass data storage challenging due to the limitation of storage space. When the index data is stored in the third-party database, only writing in the third-party database is supported, but data of the third-party database is not directly read, when a plurality of third-party databases exist, the complexity of monitoring reading and writing of the third-party database is increased, and meanwhile, the problems of poor data mobility and low data utilization rate are caused.

The Prometheus system has complex and various modes for acquiring the index data, for example, data of a target data source is acquired, or index data stored in a data gateway is acquired, or the found index data is acquired through service discovery, so that the problem of relevance loss among data caused by non-standard monitoring data labels is brought, and difficulty is brought to later-stage data relevance analysis. When mass data are collected, the Prometheus system has the defect of high concurrent data collection, the problem of data loss exists in the collection process of mass data, and meanwhile, the centralized processing of the mass data causes the failure risk of the Prometheus system.

Disclosure of Invention

In view of this, embodiments of the present invention provide a method, an apparatus, and a system for monitoring data, in which a server receives index data collected by a client, and performs a read-write operation on a third-party time series database by using a unified interface to monitor storage and reading of the index data, so as to overcome a defect that an existing system writes the third-party time series database in a one-way manner, improve efficiency of monitoring data, and improve fluidity and utilization rate of the index data; by determining the global service label, the problem of poor correlation of the collected index data caused by non-standard service labels is solved; and the received index data is put into a message queue, so that the problem of high concurrency of mass data is solved.

To achieve the above object, according to an aspect of an embodiment of the present invention, there is provided a method for monitoring data, applied to a Prometheus system, including: receiving index data, determining a global service label corresponding to the index data according to the category of the index data, and adding the global service label and corresponding content to each index data; forming target index data and storing the target index data in a message queue; acquiring the target index data from the message queue by using a Prometous server, monitoring the target index data according to a set monitoring strategy, and storing the target index data in a time sequence database so as to monitor the storage of the index data; receiving a query request, and determining a time sequence database corresponding to target index data according to the target index data in the query request; and acquiring the target index data from the time sequence database through a Prometheus server to monitor the reading of the index data.

Optionally, the method of monitoring data, wherein,

and converting the data format of the index data based on the second format into the first format according to the format rule of the first format.

Optionally, the method of monitoring data, wherein,

and when the index data contains non-numerical values, converting the non-numerical values into corresponding numbers according to a predefined corresponding relation between the non-numerical values and the numerical values.

Optionally, the method of monitoring data, wherein,

and acquiring a query request of the target index data, and converting operators contained in the query request according to grammar rules of a time sequence database.

Optionally, the method of monitoring data, wherein,

and storing the index data to the time sequence database based on a remote process calling model, and reading the index data of the time sequence database.

To achieve the above object, according to a second aspect of an embodiment of the present invention, there is provided a method of monitoring data, including: acquiring index data, and sending the index data and the global service label to the network address according to the configured global service label and the network address and a set period.

Optionally, the method of monitoring data, wherein,

and acquiring index data by using a data acquisition software package.

Optionally, the method of monitoring data, wherein,

and adding a user-defined index by using a registration method contained in the index data acquisition software package, and acquiring index data corresponding to the user-defined index by using the data acquisition software package.

Optionally, the method of monitoring data, wherein,

and acquiring the index data by using the index data acquisition script.

To achieve the above object, according to a third aspect of the embodiments of the present invention, there is provided an apparatus for monitoring data, which is applied to a Prometheus system, including: the data processing module and the data reading and writing module; wherein,

the data processing module is used for receiving index data, determining a global service tag corresponding to the index data according to the category of the index data, and adding the global service tag and corresponding content to each index data; forming target index data and storing the target index data in a message queue;

the data reading and writing module is used for acquiring the target index data from the message queue by using a Prometous server, monitoring the target index data according to a set monitoring strategy, and storing the target index data in a time sequence database so as to monitor the storage of the index data; receiving a query request, and determining a time sequence database corresponding to target index data according to the target index data in the query request; and acquiring the target index data from the time sequence database through a Prometheus server to monitor the reading of the index data.

Optionally, the apparatus for monitoring data is characterized in that,

and converting the data format of the index data based on the second format into the first format according to the format rule of the first format.

Optionally, the apparatus for monitoring data is characterized in that,

and when the index data contains non-numerical values, converting the non-numerical values into corresponding numbers according to a predefined corresponding relation between the non-numerical values and the numerical values.

Optionally, the apparatus for monitoring data is characterized in that,

and acquiring a query request of the target index data, and converting operators contained in the query request according to grammar rules of a time sequence database.

Optionally, the apparatus for monitoring data is characterized in that,

and storing the index data to the time sequence database based on a remote process calling model, and reading the index data of the time sequence database.

In order to achieve the above object, according to a fourth aspect of the embodiments of the present invention, there is provided an apparatus for monitoring data, comprising: a data acquisition module; the data acquisition module is used for acquiring index data and sending the index data and the global service label to the network address according to a set period and the configured global service label and the network address.

Optionally, the apparatus for monitoring data is characterized in that,

and acquiring index data by using a data acquisition software package.

Optionally, the apparatus for monitoring data is characterized in that,

and adding a user-defined index by using a registration method contained in the index data acquisition software package, and acquiring index data corresponding to the user-defined index by using the data acquisition software package.

Optionally, the apparatus for monitoring data is characterized in that,

and acquiring the index data by using the index data acquisition script.

To achieve the above object, according to a sixth aspect of the embodiments of the present invention, there is provided an electronic device for monitoring data, including: one or more processors; a storage device for storing one or more programs which, when executed by the one or more processors, cause the one or more processors to implement a method as in any one of the above methods of monitoring data.

To achieve the above object, according to a seventh aspect of the embodiments of the present invention, there is provided a computer readable medium having a computer program stored thereon, characterized in that the program, when executed by a processor, implements the method as any one of the above-mentioned methods of monitoring data.

One embodiment of the above invention has the following advantages or benefits: the server receives the index data collected by the client, and performs read-write operation on the third-party time sequence database by using the unified interface so as to monitor storage and reading of the index data, so that the defect that the existing system writes the third-party time sequence database in a one-way manner is overcome, the efficiency of monitoring the data is improved, and the fluidity and the utilization rate of the index data are improved; by determining the global service label, the problem of poor correlation of the collected index data caused by non-standard service labels is solved; and the received index data is put into a message queue, so that the problem of high concurrency of mass data is solved.

Further effects of the above-mentioned non-conventional alternatives will be described below in connection with the embodiments.

Drawings

The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:

FIG. 1 is a flow chart illustrating a method for monitoring data according to an embodiment of the present invention;

FIG. 2 is a schematic flow chart of a method for collecting index data according to an embodiment of the present invention;

FIG. 3 is a flow chart of monitoring data according to an embodiment of the present invention;

FIG. 4 is a schematic diagram of a prior art Prometheus system;

FIG. 5 is a schematic diagram of an improved prior art Prometoeus system provided by one embodiment of the present invention;

FIG. 6 is a schematic diagram of an apparatus for monitoring data according to an embodiment of the present invention;

fig. 7 is a schematic structural diagram of an apparatus for collecting index data according to an embodiment of the present invention;

FIG. 8 is a block diagram of a system for monitoring data according to an embodiment of the present invention;

FIG. 9 is an exemplary system architecture diagram in which embodiments of the present invention may be employed;

fig. 10 is a schematic block diagram of a computer system suitable for use in implementing a terminal device or server according to an embodiment of the present invention.

Detailed Description

Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, in which various details of embodiments of the invention are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

As shown in fig. 1, an embodiment of the present invention provides a method for a server to monitor data, where the method may include the following steps:

step S101: receiving index data, determining a global service label corresponding to the index data according to the category of the index data, and adding the global service label to each index data; forming target index data and storing the target index data in a message queue;

specifically, the index data is data monitored by a Prometheus monitoring system, for example: the system comprises index data related to a physical machine (such as the temperature, hardware fault information and the like of a router, a switch and a server), system index data operated on the physical machine (such as the CPU utilization rate, the memory utilization rate, the hard disk utilization rate, the network card flow, the TCP state, the process number and the like), and service index data (such as the operation index data of services such as Nginx, Tomcat, PHP, MySQL, Redis and the like used by the system); service index data (index data generated by a specific service scenario, such as e.g. e-commerce website, how many orders are generated per minute, etc.), it can be understood that the present invention is an improvement based on the component framework of the Prometheus system, and the range of monitored index data is similar to the range of data processed by the Prometheus system; the format of the metric data contained in the Prometheus system is as follows:

metric{tagk＝tagv,tagk1＝tagv1,…}value；

wherein, metric is the identifier of the monitored index, and tagk1 are the names of the parameters associated with the monitored index; tagv, tagv1 are values corresponding to parameters tagk, tagk 1; value is the value corresponding to metric.

Further, determining a global service label corresponding to the index data according to the category of the index data; the types of the index data are to distinguish the source, the service type and the like of the index data, for example, the index data of different clusters are divided into different types; dividing index data from different network address ranges into different categories; dividing index data of different services into different categories (for example, dividing index data of logistics and e-commerce into different categories); further, according to the category of the index data, determining a global service label corresponding to the index data, for example, determining that a global service label corresponding to the index data from the cluster1 is "cluster 11111"; the invention does not limit the specific content of the index data category and the specific content of the global service label.

Further, adding the global service label to each index data; for example: the received index data is identified as finishJobAvgTime, the identification may be originated from different services, such as Hbase, Hadoop, Spark, etc., and the source of the collected index data may be distinguished according to the category and under the condition that the index data identifications are the same through the global service tag. Therefore, the received index data has uniqueness and relevance by utilizing the global service tag, and the problem of low data relevance caused by non-standard index data tag is solved; for example, setting a global service tag serviceID (123456 abcdef) corresponding to index data of HBase service, setting a global service tag serviceID (789010 abcdef) corresponding to index data of Hadoop service, and adding the global service tag to each index data in a corresponding batch of index data to form target index data; the problem of poor data association degree caused by non-uniform index data labels is solved through the steps; preferably, according to the type of the index data, determining a global service label according to the received global service label; or according to a self-defined global service label configuration rule, converting the received global service label into a global service label matched with the configuration rule. Meanwhile, if the client sends the global service tag and other service tags, the server may add other service tags to each index data in a batch of index data.

Further, the target index data is stored in the message queue. Specifically, the existing Prometheus system acquires data of a target network unit or a data gateway according to a set period, which cannot complete high-concurrency data acquisition, and obtains the following example data through a pressure test: the data volume of the related data generated by 4 ten thousand data processing units can be 219 GB/day, and the generated index number is: 150 hundred million/day, about 17.3 million indicators per second, unscheduled mass data may cause the stability of the promemeus server to be degraded, so before the mass indicator data is sent to the promemeus server, the indicator data is processed by using a message queue (e.g. kafka), and a mechanism of message queue caching is utilized, so that the problem of concurrent processing of mass data is solved. For example: one embodiment of the present invention is that the data gateway pushes the received index data to the message queue by setting a topic partition (for example, setting topic1 to indicate that the index data is related to HBase, and setting topic2 to indicate that the index data is related to Hadoop), it can be understood that reading and writing of data in the message queue (kafka) of different services can be isolated by setting the topic partition (topic), and preferably, the Prometheus server can obtain the index data from the message queue according to a set period, or push the index data Prometheus server by the message queue according to a set period; therefore, the problem of task blocking or poor server stability caused by directly pushing mass data to the Prometheus server is partially solved.

Still further, the data format of the index data based on the second format is converted into the first format according to a format rule of the first format. Specifically, the first format is a data format supported by the existing Prometheus system, and the second format is a data format inconsistent with the data format supported by the existing Prometheus system, for example: the second format is any data format that can be converted into the first format, and the specific format of the second format is not limited in the present invention.

The data format of the metric data may be based on a data format defined by the Prometheus system (i.e., a first format), examples of which are as follows:

metric { tagk ═ tagv, tagk1 ═ tagv1, … } value; wherein, metric is the identifier of the monitored index, and tagk1 are the names of the parameters associated with the monitored index; tagv, tagv1 are values corresponding to parameters tagk, tagk 1; value is a value corresponding to metric; further, the data format of the received indicator data may also be a second format, and the second format is exemplified as a data format defined based on the OpenTSDB database, and the second format is exemplified as follows:

where, metric is the monitored indicator identifier, in the above example, metric is "query _ info _ 12345", tags includes the name of the parameter associated with the indicator, for example, parameter 1 is "status" and the value is "failed"; converting the index data format based on the second format into the first format according to the format rule of the first format; for example, converting the above example of metric data based on the second format into the first format is:

query_info_12345{status＝”failed”，cluster＝”AAAA”}200

it can be understood that, according to the format rule of the first format, the received index data in the second format is correspondingly converted into the first format according to the corresponding tag or content, and further the index data in the first format is further monitored and analyzed, so that the range of the monitored index data is expanded, and the complexity of data processing is reduced due to the unification of the data formats.

Step S102: acquiring the target index data from the message queue by using a Prometous server, monitoring the target index data according to a set monitoring strategy, and storing the target index data in a time sequence database so as to monitor the storage of the index data; receiving a query request, and determining a time sequence database corresponding to target index data according to the target index data in the query request; and acquiring the target index data from the time sequence database through a Prometheus server to monitor the reading of the index data.

Specifically, the target index data is acquired from the message queue by using a Prometheus server (i.e., a monitoring server), and preferably, the index data may be acquired from the message queue at a set period, for example, the set period is set to 30 seconds; it will be appreciated that the set period is determined according to the particular service or service scenario, and the frequency and granularity of the user monitoring data. For example, for index data with high real-time requirement, a time with a short period may be set, for example: may be set to 1 second. Monitoring the target index data according to a set monitoring strategy, and monitoring the received index data by the Prometous server according to the set monitoring strategy, wherein the monitoring strategy is set according to a monitored service scene and an index, and comprises setting a triggering alarm rule, setting a threshold value, setting a monitoring period, setting monitored index data and the like; the function of the Prometheus server is monitored according to the set monitoring strategy, and the function of the Prometheus server is not further discussed.

Further, according to the description in step S101, the received data is processed to form target index data with a consistent format, and the Prometheus server stores the target index data in a time series database to monitor the storage of the index data; among them, using the HBase-based time series database OpenTSDB as the time series database, it is understood that the index data may be stored in a plurality of time series databases according to the category of the index data, each time series database may be a third party database, by storing the data in the plurality of third party time series databases, the problem of capacity limitation of local storage is solved, and by this step, the storage of the index data in the plurality of time series databases may be monitored.

Further, according to the rule of the time series database (OpenTSDB), if the index data with a non-numeric value is not supported to be stored, the value of the index data is converted into a corresponding number according to a predefined corresponding relationship between the non-numeric value and the numeric value. For example, a numerical value "active" indicating a service state corresponds to a numeral 1, and "standby" corresponds to a numeral 0; that is, when the index data includes a non-numeric value, the non-numeric value is converted into a corresponding number according to a predefined correspondence relationship between the non-numeric value and the number.

Further, receiving a query request, and determining a time sequence database corresponding to target index data according to the target index data in the query request; and acquiring the target index data from the time sequence database to monitor the reading of the index data. Specifically, a query request for target index data is received, a time sequence database corresponding to the target index data is determined, the target index data is obtained, and therefore reading of the index data is monitored; compared with the existing Prometheus system, the method has the advantages that reading of the target time sequence database is monitored by the Prometheus server, index data in the time sequence database are obtained according to the request, mobility and utilization rate of mass data are increased, and the index data can be analyzed more conveniently to generate index data analysis information. Meanwhile, the Prometheus server is used for directly acquiring index data in the time sequence database, so that the monitoring and management efficiency of the third-party time sequence database is improved, and the defect that the Prometheus server cannot read data from the third-party database in the prior art is overcome.

Since there are parts where the query syntax of the time series database (OpenTSDB) is inconsistent with the query syntax of the data format of the promemeus system, for example: the data format of Prometheus system includes "═ and"! "-", "! "four operation symbols, and the data format of OpenTSDB does not contain the operation symbols; then, the query request of the target index data is obtained, and the conversion is performed according to the grammar rule of the time series database, for example: the "-" and "!in the query request may be set! "filter condition operator" primitive _ or "in syntax rules that can be converted into OpenTSDB format; and the fuzzy matching operand "-" and "! Preferably, a return result corresponding to a query request containing a fuzzy matching operator symbol is obtained, and then, based on the return result, an or operator is used to convert the return result into a grammar rule of a time sequence database defined by the OpenTSDB, and further query operation is performed. It is understood that after the Prometheus server obtains the target metric data from OpenTSDB, further data calculation and processing are performed in the Prometheus server. Namely, a query request of the target index data is obtained, and operators contained in the query request are converted according to grammar rules of a time sequence database.

And further, storing the index data to the target database and reading the index data of the target database based on a remote process call model. Specifically, a monitoring server (Prometheus server) executes read-write operation on a time sequence database (such as OpenTSDB) through a remote process call model (such as gRPC); and monitoring the read-write operation of the time series database, wherein the gPC can define an interface through a structural data serialization method (for example, protobuf), and the structural data serialization method can serialize data into binary coding and compress the data, thereby reducing the data transmission amount and improving the data transmission performance.

As shown in fig. 2, an embodiment of the present invention provides a method for a client to collect index data, where the method may include the following steps:

step S201: acquiring index data, and sending the index data and the global service label to the network address according to the configured global service label and the network address and a set period.

Two methods for acquiring index data by the client are as follows: the client is a server or a computer used for acquiring index data; the invention does not limit the specific equipment to which the client belongs.

The first method comprises the following steps: and acquiring index data by using a data acquisition software package.

Specifically, an embodiment of the present invention is that a client collects index data by using an application developed by Java language, the client is completed by using a data collection software package (for example, javagent. jar), and the client may add the following parameters to parameters of a Java virtual machine by using the following commands, javagent: { IP: Port }, labels ═ { serviceId: abcd }, file ═ a.yml }, jobName ═ abc }; specifically, the parameter description about the present example is shown in table 1;

TABLE 1 Java class application Collection index data parameters

Specifically, the data collection software package includes the following functions: supporting the increase of global service labels through starting parameters; and directly pushing the index data to the data gateway by configuring the network address of the data gateway. Further, the client scans and collects data through a probe technology possessed by Java after the application is started by configuring a global service tag and dynamically configuring a data gateway network address. And sending the collected index data to a configured data gateway through a hypertext transfer protocol, further, the data gateway pushes the received index data to a message queue (kafka) through topic information, and the Prometheus server can obtain the index data from the message queue.

Further, when the client needs to send the index data of the user-defined index based on the business logic of the client, the data can be reported by using a registration method included in a data acquisition software package (jar), so that the data acquisition is completed. For example: the custom index metricregistration.register (name, metricType) can be registered using the method shown below, where metricregistration.register is the method used for registration, and metricType is the index type, and includes five types, respectively: gauge, Counter, Meter, Timer, Histogram; wherein, Gauge: recording instantaneous values of indexes, such as service conditions of the current Java virtual machine, including memory utilization rate, central processing unit utilization rate, thread use state and the like; counter: is a counter that, by incrementing and decrementing operations, forms an accumulated index, such as: the sum of the number of submitted tasks in one cluster, and the like; a Meter: for counting the frequency of occurrence of events, for example: counting the network flow of the latest 1 minute, 5 minutes and 15 minutes for the aggregation calculation of the indexes; timer: for statistical distribution, for example: counting request frequency and time-consuming data of an interface; histopram: numerical distribution for statistical index data, for example: the minimum, maximum, mean, median, 75 quantiles, 90 quantiles, etc. of the statistical values. The client can define the index and place the index at the position where the logic code throws the exception, and when the exception occurs, the logic code of the exception throwing part is triggered to send the index data corresponding to the index to the data gateway, so that the client can call the registration method included in the data acquisition software package to acquire the index data corresponding to the user-defined index at the logic code throwing part of the exception, namely, the registration method included in the index data acquisition software package is utilized to increase the user-defined index, and the data acquisition software package is utilized to acquire the index data corresponding to the user-defined index.

The second method comprises the following steps: and acquiring the index data by using the index data acquisition script.

Specifically, for the application developed by the non-java language, the client can use the index data acquisition script to acquire the index data, send the index data and the global service label to the configured network address (for example, the network address of the data gateway) by using the hypertext transfer protocol according to the configured global service label and the network address and the customized index according to the data format set by the data gateway, and the language used for developing the index data acquisition script can be Phython, Perl and the like.

As shown in fig. 3, an embodiment of the present invention provides a flow chart of monitoring data, where the method may include the following steps:

step S301: and the client acquires index data.

Specifically, the description of the client acquiring the index data by using the data acquisition software package or the data acquisition script is consistent with step S201, and is not described herein again. The method comprises the steps of acquiring index data, and sending the index data and a global service label to a network address according to a configured global service label and the network address and a set period. Further, index data is collected by using a data collection software package. And adding a user-defined index by using a registration method contained in the index data acquisition software package, and acquiring index data corresponding to the user-defined index by using the data acquisition software package. And acquiring index data corresponding to the service identifier by using the index data acquisition script.

Step S302: the data gateway receives the metric data from the client.

Specifically, a data gateway receives index data, receives the index data, determines a global service tag corresponding to the index data according to the category of the index data, and adds the global service tag to each index data; forming target index data and storing the target index data in a message queue;

the description of receiving the index data and processing the index data is consistent with step S101, and is not repeated here.

Step S303: and the data gateway puts the target index data into a message queue.

Specifically, the description about the data gateway placing the target index data into the message queue is consistent with step S101, and is not described herein again.

Step S304-step S305: acquiring the target index data from the message queue by using a Prometous server, monitoring the target index data according to a set monitoring strategy, and storing the target index data in a time sequence database so as to monitor the storage of the index data; receiving a query request, and determining a time sequence database corresponding to target index data according to the target index data in the query request; and acquiring the target index data from the time sequence database through a Prometheus server to monitor the reading of the index data.

The description about the Prometheus server obtaining data from the message queue and monitoring the storage and reading of the time series database is consistent with step S102, and will not be described herein.

FIG. 4 shows a schematic diagram of a prior art Prometheus system;

FIG. 5 shows a schematic diagram of an improved Prometheus system provided by one embodiment of the present invention;

an embodiment of the present invention will be described below by comparing fig. 4 and 5.

1) In the existing Prometheus system, index data are stored locally, and the problem of capacity limitation exists in mass data storage. Although the index data may be stored in a third-party database (such as the TSDB shown in fig. 4), the interaction with the third-party database is unidirectional and only supports unidirectional writing, that is, the index data stored in the third-party database is static, and has poor fluidity and low data utilization rate, thereby causing the problems of data association loss and possible incompatibility of data formats.

Comparing with fig. 4, as shown in fig. 5, the present invention implements bidirectional operations of storing and reading time series databases by the Prometheus server and monitors the storing and reading for storing and reading index data based on monitoring of a plurality of third party time series databases. For example, a Prometheus server for monitoring stores and reads index data of a third-party database (e.g., TSDB shown in fig. 5, which is an OpenTSDB described in the present invention), monitors the storage and reading of the third-party database, and performs read-write operation on the third-party database by using a unified interface of the Prometheus server, so that the efficiency of monitoring the index data is improved, the mobility of the index data and the utilization rate of the index data are improved, and the problem of Prometheus performance caused by mass data stored in a local hard disk is solved.

The method comprises the steps that a Prometous server is used for obtaining target index data from a message queue, the target index data are monitored according to a set monitoring strategy, and the target index data are stored in a time sequence database so as to monitor storage of the index data; receiving a query request, and determining a time sequence database corresponding to target index data according to the target index data in the query request; and acquiring the target index data from the time sequence database through a Prometheus server to monitor the reading of the index data.

2) As shown in fig. 4, in the existing Prometheus system, Prometheus periodically obtains index data of static configuration monitoring targets (targets) or data gateways (Pushgateway), and may also obtain index data of data containers kubernets (k8s) through Service discovery (Service discovery). Therefore, the data acquisition modes are complex and various, and the data islanding (namely, the data lack correlation) caused by the possible irregular condition of the index data label used for acquisition causes problems in the later data correlation analysis; and as can be seen from fig. 4, the k8s machine is coupled to the monitoring service to a higher degree.

As shown in fig. 5, the Prometheus server can periodically obtain data (i.e. target index data) in the message queue, and in contrast to the existing Prometheus system, the manner of collecting data is converted from multiple types to single type, and the target index data of the message queue (kafka) is obtained by processing the received index data through a data gateway (Pushgateway), and the data gateway determines a global service tag according to the type of the received index data and adds a global service table to fill in the index data.

The method comprises the steps of receiving index data, determining a global service label corresponding to the index data according to the type of the index data, and adding the global service label to each index data; forming target index data and storing the target index data in a message queue;

3) from the description of 2), it can be seen that, in one embodiment of the present invention, a message queue (kafka) is used to solve the problem of collecting high-concurrency mass data, and this technical solution is to add a new component to the present invention based on the existing Prometheus system. By using a message queue caching mechanism, the problems caused by high throughput and low delay of the monitoring server can be partially solved, and meanwhile, the fault tolerance is good under the condition of high concurrency.

As shown in fig. 8, an embodiment of the present invention provides a system for monitoring data, including: such as the means for monitoring data shown in fig. 6, and the means for monitoring data shown in fig. 7.

An embodiment of the present invention further provides an electronic device for monitoring data, including: one or more processors; the storage device is used for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors are enabled to realize the method provided by any one of the above embodiments.

Embodiments of the present invention further provide a computer-readable medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the method provided in any of the above embodiments.

It should be understood that the number of terminal devices, networks, and servers in fig. 9 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.

It should be noted that the computer readable medium shown in the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The modules and/or units described in the embodiments of the present invention may be implemented by software, and may also be implemented by hardware. The described modules and/or units may also be provided in a processor, and may be described as: a processor comprises a data processing module, a data reading and writing module and a data acquisition module. The names of the modules do not limit the modules themselves in some cases, for example, the data processing module may also be described as a module that receives the index data, converts the data format of the index data into the first format, and adds a global service tag to the index data.

As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be separate and not incorporated into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to comprise: receiving index data, determining a global service label corresponding to the index data according to the category of the index data, and adding the global service label to each index data; forming target index data and storing the target index data in a message queue; acquiring the target index data from the message queue by using a Prometous server, monitoring the target index data according to a set monitoring strategy, and storing the target index data in a time sequence database so as to monitor the storage of the index data; receiving a query request, and determining a time sequence database corresponding to target index data according to the target index data in the query request; and acquiring the target index data from the time sequence database through a Prometheus server to monitor the reading of the index data. Acquiring index data, and sending the index data and the global service label to the network address according to the configured global service label and the network address and a set period.

According to the technical scheme of the embodiment of the invention, the server receives the index data collected by the client and executes the read-write operation on the third-party time sequence database by using the unified interface so as to monitor the storage and reading of the index data, thereby overcoming the defect that the existing system stores the third-party time sequence database in a one-way manner, improving the efficiency of monitoring the data and improving the mobility and the utilization rate of the index data; by determining the global service label, the problem of poor correlation of the collected index data caused by non-standard service labels is solved; and the received index data is put into a message queue, so that the problem of high concurrency of mass data is solved.

The above-described embodiments should not be construed as limiting the scope of the invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions can occur, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.