CN114359666B - Multi-mode fused lung cancer patient curative effect prediction method, system, device and medium - Google Patents

The invention discloses a multi-mode fusion lung cancer patient curative effect prediction method, a system, a device and a medium, wherein the method comprises the following steps: acquiring first patient information of a first lung cancer patient; performing data preprocessing on the first patient information to obtain second patient information; respectively inputting the second patient information into a plurality of preset neural network models for feature extraction to obtain facial features, CT image features, pathological features, pulse features and clinical features of the first lung cancer patient; feature fusion is carried out on the face features, the CT image features, the pathological features, the pulse features and the clinical features according to preset weights, so that multi-mode fusion features are obtained, and then the multi-mode fusion features are input into a pre-trained lung cancer patient curative effect prediction model, so that a curative effect prediction result of the first lung cancer patient is obtained. The invention improves the comprehensiveness of the prediction of the curative effect of the lung cancer patients and the accuracy of the prognosis life time of the lung cancer patients, and can be widely applied to the technical field of artificial intelligence.

Multi-mode fused lung cancer patient curative effect prediction method, system, device and medium

Technical Field

The invention relates to the technical field of artificial intelligence, in particular to a method, a system, a device and a medium for predicting curative effect of a multi-mode fused lung cancer patient.

Background

One implementation mode of traditional Chinese medicine modernization is to integrate and extract characteristics and fuse multi-mode clinical data such as CT, pathological images, medical record data, vital sign data, traditional Chinese medicine 'pulse disease treatment' data and the like which are involved in the traditional Chinese medicine lung cancer prevention and treatment process by means of new technical means such as artificial intelligence and the like and by combining an image histology technology, so that dominant crowd characteristics are mined, and a curative effect prediction model is constructed.

In the prior art, modeling prediction is mostly carried out on a certain data mode of a lung cancer patient, and a method which contains multi-mode information as much as possible and can accurately predict the prognosis lifetime of the lung cancer patient is lacked.

Disclosure of Invention

The present invention aims to solve at least one of the technical problems existing in the prior art to a certain extent.

Therefore, an object of the embodiments of the present invention is to provide a method for predicting the curative effect of a lung cancer patient by multi-modal fusion, which extracts facial features, CT image features, pathological features, pulse features and clinical features of the lung cancer patient and performs feature fusion to obtain multi-modal fusion features, wherein the multi-modal fusion features include feature information of each aspect of the lung cancer patient, and training and predicting the curative effect prediction model of the lung cancer patient are performed based on the multi-modal fusion features, so that the comprehensiveness of curative effect prediction and the accuracy of prognosis survival time of the lung cancer patient are improved.

Another object of the embodiments of the present invention is to provide a multi-modal fusion lung cancer patient efficacy prediction system.

In order to achieve the technical purpose, the technical scheme adopted by the embodiment of the invention comprises the following steps:

in a first aspect, an embodiment of the present invention provides a method for predicting curative effects of a multi-modal fused lung cancer patient, including the steps of:

Acquiring first patient information of a first lung cancer patient, wherein the first patient information comprises first facial information, first CT image information, first pathology slide information, first pulse wave information and first clinical information;

Performing data preprocessing on the first patient information to obtain second patient information;

The second patient information is respectively input into a plurality of preset neural network models for feature extraction, and the face features, CT image features, pathological features, pulse features and clinical features of the first lung cancer patient are obtained;

And carrying out feature fusion on the face feature, the CT image feature, the pathological feature, the pulse feature and the clinical feature according to preset weights to obtain multi-modal fusion features, and further inputting the multi-modal fusion features into a pre-trained lung cancer patient curative effect prediction model to obtain a curative effect prediction result of the first lung cancer patient.

Further, in one embodiment of the present invention, the step of acquiring the first patient information of the first lung cancer patient specifically includes:

acquiring first facial information of the first lung cancer patient before treatment;

Acquiring first CT image information of the first lung cancer patient before treatment;

Acquiring a lung pathology slide of the first lung cancer patient, and digitally scanning the lung pathology slide through a digital slide scanner to obtain first pathology slide information;

Acquiring first pulse wave information of the first lung cancer patient before treatment;

First clinical information of the first lung cancer patient is obtained, the first clinical information including patient age, patient gender, patient blood pressure, patient weight, and treatment regimen.

Further, in an embodiment of the present invention, the second patient information includes second facial information, second CT image information, second pathology slide information, second pulse wave information, and second clinical information, and the step of performing data preprocessing on the first patient information to obtain second patient information specifically includes:

Carrying out image normalization processing on the first facial information to obtain the second facial information;

obtaining a CT image according to the first CT image information, and performing image normalization processing on the CT image to obtain the second CT image information;

Sliding window sampling is carried out on the first pathological slide information according to a preset window width to obtain slide sample information, and image normalization is carried out on the slide sample information to obtain the second pathological slide information;

generating a pulse wave image according to the first pulse wave information, and carrying out image normalization on the pulse wave image to obtain the second pulse wave information;

And generating text information according to the first clinical information, and performing single-hot coding on the text information to obtain the second clinical information.

Further, in an embodiment of the present invention, the step of inputting the second patient information into a plurality of preset neural network models to perform feature extraction to obtain facial features, CT image features, pathological features, pulse features and clinical features of the first lung cancer patient specifically includes:

Inputting the second facial information into a preset first convolutional neural network model for feature extraction to obtain the facial features;

Inputting the second CT image information into a preset second convolutional neural network model for feature extraction to obtain CT image features;

Inputting the second pathological slide information into a preset third convolutional neural network model for feature extraction to obtain the pathological features;

Inputting the second pulse wave information into a preset long-short-period memory neural network model for feature extraction to obtain the pulse features;

and inputting the second clinical information into a preset fully-connected neural network model for feature extraction to obtain the clinical features.

Further, in an embodiment of the present invention, the step of performing feature fusion on the face feature, the CT image feature, the pathological feature, the pulse feature and the clinical feature according to a preset weight to obtain a multi-mode fusion feature specifically includes:

Vectorizing the face feature, the CT image feature, the pathological feature, the pulse feature and the clinical feature to obtain a face feature vector, a CT image feature vector, a pathological feature vector, a pulse feature vector and a clinical feature vector;

And carrying out weighted summation on the face feature vector, the CT image feature vector, the pathological feature vector, the pulse feature vector and the clinical feature vector according to preset weights to obtain the multi-mode fusion feature.

Further, in one embodiment of the present invention, the method for predicting the efficacy of a lung cancer patient further includes a step of pre-training a model for predicting the efficacy of the lung cancer patient, which specifically includes:

Acquiring third patient information of a second lung cancer patient, and acquiring a second fusion characteristic according to the third patient information;

Generating a label according to the curative effect of the second lung cancer patient, and generating a training data set according to the second fusion characteristic and the corresponding label;

And inputting the training data set into a pre-constructed lung cancer patient curative effect prediction model for model training to obtain a trained lung cancer patient curative effect prediction model.

Further, in one embodiment of the present invention, the step of inputting the training data set into a pre-constructed lung cancer patient efficacy prediction model for model training to obtain a trained lung cancer patient efficacy prediction model specifically includes:

inputting the training data set into a pre-constructed lung cancer patient curative effect prediction model to obtain a first prediction result;

determining a loss value of the lung cancer patient curative effect prediction model according to the first prediction result and the label;

Updating parameters of the lung cancer patient curative effect prediction model through a back propagation algorithm according to the loss value;

And stopping training when the loss value reaches a preset first threshold value or the iteration number reaches a preset second threshold value or the test precision reaches a preset third threshold value, and obtaining a trained lung cancer patient curative effect prediction model.

In a second aspect, an embodiment of the present invention provides a multi-modal fusion lung cancer patient efficacy prediction system, including:

The system comprises a patient information acquisition module, a first clinical information acquisition module and a first information processing module, wherein the patient information acquisition module is used for acquiring first patient information of a first lung cancer patient, and the first patient information comprises first facial information, first CT image information, first pathology slide information, first pulse wave information and first clinical information;

The data preprocessing module is used for preprocessing the data of the first patient information to obtain second patient information;

the feature extraction module is used for respectively inputting the second patient information into a plurality of preset neural network models to perform feature extraction to obtain face features, CT image features, pathological features, pulse features and clinical features of the first lung cancer patient;

The feature fusion and prediction module is used for carrying out feature fusion on the face feature, the CT image feature, the pathological feature, the pulse feature and the clinical feature according to preset weights to obtain a multi-mode fusion feature, and further inputting the multi-mode fusion feature into a pre-trained lung cancer patient curative effect prediction model to obtain a curative effect prediction result of the first lung cancer patient.

In a third aspect, an embodiment of the present invention provides a multi-modal fusion lung cancer patient efficacy prediction apparatus, including:

at least one processor;

at least one memory for storing at least one program;

The at least one program, when executed by the at least one processor, causes the at least one processor to implement a multi-modal fusion lung cancer patient efficacy prediction method as described above.

In a fourth aspect, embodiments of the present invention also provide a computer readable storage medium having stored therein a processor executable program which when executed by a processor is configured to perform a multi-modal fusion lung cancer patient efficacy prediction method as described above.

The advantages and benefits of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.

According to the embodiment of the invention, the facial features, CT image features, pathological features, pulse features and clinical features of the lung cancer patient are extracted and feature fusion is carried out to obtain the multi-mode fusion features, the multi-mode fusion features comprise feature information of all aspects of the lung cancer patient, training and prediction of the lung cancer patient curative effect prediction model are carried out based on the multi-mode fusion features, and the comprehensiveness of curative effect prediction and the accuracy of prognosis survival time of the lung cancer patient are improved.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following description will refer to the drawings that are needed in the embodiments of the present invention, and it should be understood that the drawings in the following description are only for convenience and clarity to describe some embodiments in the technical solutions of the present invention, and other drawings may be obtained according to these drawings without any inventive effort for those skilled in the art.

FIG. 1 is a flowchart showing steps of a method for predicting curative effect of a multi-modal fusion lung cancer patient according to an embodiment of the present invention;

FIG. 2 is a schematic diagram of feature extraction and multimodal fusion provided by an embodiment of the present invention;

FIG. 3 is a block diagram of a multi-modal fusion lung cancer patient efficacy prediction system according to an embodiment of the present invention;

Fig. 4 is a block diagram of a device for predicting curative effect of a lung cancer patient by multi-mode fusion according to an embodiment of the present invention.

Detailed Description

Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to like or similar elements or elements having like or similar functions throughout. The embodiments described below by referring to the drawings are illustrative only and are not to be construed as limiting the invention. The step numbers in the following embodiments are set for convenience of illustration only, and the order between the steps is not limited in any way, and the execution order of the steps in the embodiments may be adaptively adjusted according to the understanding of those skilled in the art.

In the description of the present invention, the plurality means two or more, and if the description is made to the first and second for the purpose of distinguishing technical features, it should not be construed as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the precedence of the indicated technical features. Furthermore, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art.

Referring to fig. 1, the embodiment of the invention provides a multi-mode fusion lung cancer patient curative effect prediction method, which specifically comprises the following steps:

s101, acquiring first patient information of a first lung cancer patient, wherein the first patient information comprises first facial information, first CT image information, first pathology slide information, first pulse wave information and first clinical information.

Specifically, the embodiment of the invention models feature information of five aspects of a lung cancer patient, including face information, CT image information, pathology slide information, pulse wave information and clinical information. The step S101 specifically includes the following steps:

s1011, acquiring first facial information of a first lung cancer patient before treatment;

S1012, acquiring first CT image information of a first lung cancer patient before treatment;

S1013, acquiring a lung pathology slide of a first lung cancer patient, and digitally scanning the lung pathology slide through a digital slide scanner to obtain first pathology slide information;

s1014, acquiring first pulse wave information of a first lung cancer patient before treatment;

s1015, acquiring first clinical information of a first lung cancer patient, wherein the first clinical information comprises patient age, patient gender, patient blood pressure, patient weight and treatment scheme.

Specifically, (1) face information: collecting the positive facial makeup photos of the patient in two weeks before treatment, wherein the resolution is more than 256 x 256, the illumination is required to be normal, bareheaded, glasses are taken off, and other requirements are not required; (2) CT image information: collecting DICOM data of lung CT slices in two weeks before treatment of a patient, wherein the model, the layer thickness, the scanning resolution and the like of CT equipment are not required; (3) pathological slide information: digitally scanning a patient lung pathology slide under an objective lens 40X by using a digital slide scanner to generate digital pathology (WSI), wherein the WSI is generally one hundred thousand X hundred thousand resolution, and the size is generally larger than 1GB; (4) pulse wave information: measuring pulse waves at any wrist of the patient in a calm state for 30 seconds to obtain pulse wave information; (5) clinical information: patient hospitalization routine information is collected including age, sex, blood pressure, weight, treatment regimen, etc.

S102, performing data preprocessing on the first patient information to obtain second patient information.

Specifically, the second patient information includes second facial information, second CT image information, second pathology slide information, second pulse wave information, and second clinical information. The step S102 specifically includes the following steps:

s1021, performing image normalization processing on the first facial information to obtain second facial information;

s1022, obtaining a CT image according to the first CT image information, and performing image normalization processing on the CT image to obtain second CT image information;

S1023, sliding window sampling is carried out on the first pathological slide information according to a preset window width to obtain slide sample information, and image normalization is carried out on the slide sample information to obtain second pathological slide information;

S1024, generating a pulse wave image according to the first pulse wave information, and carrying out image normalization on the pulse wave image to obtain second pulse wave information;

S1025, generating text information according to the first clinical information, and performing single-hot encoding on the text information to obtain second clinical information.

Specifically, (1) face information: firstly, adjusting the resolution of a face image to 256 x 256 pixels, then carrying out image normalization, compressing pixel values from (0-255) to (0-1), and then storing the pixel values as Python Numpy arrays; (2) CT image information: opening CT images through RadiAnt DICOM Viewer software, storing the CT images as jpg or other image formats one by one, carrying out normalization processing on the stored images, uniformly adjusting the resolution to 512 x 512 pixels, and storing the images as Python Numpy arrays; (3) pathological slide information: sampling WSI (digital pathology slide) by using openslide software with an open source according to a window width of 256 x 256, and then storing the sample as Python Numpy array after normalization treatment; (4) pulse wave information: reading by using a Padas library of Python, then normalizing, and storing into Python Numpy arrays; (5) medical record data: and carrying out classified statistics on the text data, then carrying out single-heat encoding, and storing the text data into Python Numpy arrays.

S103, respectively inputting the second patient information into a plurality of preset neural network models for feature extraction to obtain the face features, CT image features, pathological features, pulse features and clinical features of the first lung cancer patient.

Specifically, in the embodiment of the invention, a plurality of neural network models are preset for extracting various features. Step S103 specifically includes the following steps:

S1031, inputting second facial information into a preset first convolutional neural network model for feature extraction to obtain facial features;

s1032, inputting the second CT image information into a preset second convolutional neural network model for feature extraction to obtain CT image features;

S1033, inputting the second pathological slide information into a preset third convolutional neural network model for feature extraction to obtain pathological features;

s1034, inputting the second pulse wave information into a preset long-short-period memory neural network model for feature extraction to obtain pulse features;

s1035, inputting the second clinical information into a preset fully-connected neural network model for feature extraction, and obtaining clinical features.

Specifically, (1) for face images, tensorflow2.0 is adopted, a convolutional neural network architecture model EFFICIENTNET library is introduced into the input, and then the corresponding EFFICIENTNET weight is introduced into the model, and the weight is obtained through full training on an ImageNet dataset and is used for CNN migration learning. Next, since the last layer of the model carrying the weight is 1000 output neurons, the embodiment of the present invention removes the last layer, and adds a fully connected layer containing only 1 neuron as the output feature layer, the layer does not add any activation function, and sets the learning rate to 0.0008, the loss function to mean square error, and the optimizer to Adam.

(2) For CT images, tensorflow2.0 is adopted, a convolutional neural network architecture model EFFICIENTNET library is introduced at the input, and then the corresponding EFFICIENTNET weight is introduced into the model, and the weight is obtained by fully training on an ImageNet dataset and is used for CNN migration learning. Next, since the last layer of the model carrying the weight is 1000 output neurons, the embodiment of the present invention removes the last layer, and adds a fully connected layer containing only 1 neuron as the output feature layer, the layer does not add any activation function, and sets the learning rate to 0.0008, the loss function to mean square error, and the optimizer to Adam.

(3) For pathological images, tensorflow2.0 is adopted, a convolutional neural network model EFFICIENTNET library is introduced at the input, and then the corresponding EFFICIENTNET weight is introduced into the model, wherein the weight is obtained by fully training on an ImageNet dataset and is used for CNN migration learning. Next, since the last layer of the model carrying the weight is 1000 output neurons, the embodiment of the present invention removes the last layer, and adds a fully connected layer containing only 1 neuron as the output feature layer, the layer does not add any activation function, and sets the learning rate to 0.0008, the loss function to mean square error, and the optimizer to Adam.

(4) For pulse wave data, tensorflow2.0 is adopted, a long-short-period memory neural network (or other circulating neural networks) is introduced into the input, a layer containing 50 neurons is arranged, the learning rate is 0.0008, the loss function is mean square error, the optimizer is Adam, a fully-connected layer containing only 1 neuron is added as an output characteristic layer, the layer is not added with any activation function, the learning rate is 0.0008, the loss function is mean square error, and the optimizer is Adam.

(5) For clinical information, tensorflow2.0 is adopted, a layer of full-connection network is introduced at the input, the full-connection network comprises 20 neurons, the activation function is RELU, a full-connection layer only comprising 1 neuron is added as an output characteristic layer, the layer is not added with any activation function, the learning rate is set to be 0.0008, the loss function is the mean square error, and the optimizer is Adam.

As shown in fig. 2, the facial features F1, CT image features F2, pathological features F3, pulse features F4 and clinical features F5 of the lung cancer patient can be obtained after the feature extraction processing, and then multi-modal feature fusion can be performed.

S104, carrying out feature fusion on the face feature, the CT image feature, the pathological feature, the pulse feature and the clinical feature according to preset weights to obtain multi-mode fusion features, and further inputting the multi-mode fusion features into a pre-trained lung cancer patient curative effect prediction model to obtain a curative effect prediction result of the first lung cancer patient.

Specifically, in the embodiment of the invention, five characteristics (F1 to F5) are obtained after five-mode data are input into the corresponding models, and then the five characteristics are fused according to preset weights to obtain multi-mode fusion characteristics, so that the multi-mode fusion characteristics can be input into a trained lung cancer patient curative effect prediction model to predict the prognosis survival time.

Further as an optional implementation manner, the step of performing feature fusion on the facial features, the CT image features, the pathological features, the pulse features and the clinical features according to preset weights to obtain multi-mode fusion features specifically includes:

A1, carrying out vectorization processing on facial features, CT image features, pathological features, pulse features and clinical features to obtain facial feature vectors, CT image feature vectors, pathological feature vectors, pulse feature vectors and clinical feature vectors;

And A2, carrying out weighted summation on the face feature vector, the CT image feature vector, the pathological feature vector, the pulse feature vector and the clinical feature vector according to preset weights to obtain the multi-mode fusion feature.

Further as an optional embodiment, the lung cancer patient efficacy prediction method further includes a step of pre-training a lung cancer patient efficacy prediction model, which specifically includes:

B1, acquiring third patient information of a second lung cancer patient, and acquiring a second fusion characteristic according to the third patient information;

B2, generating a label according to the curative effect of the second lung cancer patient, and generating a training data set according to the second fusion characteristic and the corresponding label;

and B3, inputting the training data set into a pre-constructed lung cancer patient curative effect prediction model for model training, and obtaining a trained lung cancer patient curative effect prediction model.

Specifically, in the stage of training the lung cancer patient curative effect prediction model, patient information of a second lung cancer patient with known curative effect (namely, prognosis lifetime) is obtained, and a second fusion characteristic is obtained by adopting a method similar to the characteristic extraction and fusion, and the specific process is not repeated; meanwhile, determining labels according to the actual curative effect of the second lung cancer patient, wherein the labels are in one-to-one correspondence with the second fusion characteristics, and constructing a training data set according to the labels and the second fusion characteristics; and finally, inputting the training data set into an initialized lung cancer patient curative effect prediction model for model training.

Further as an optional implementation manner, the step B3 of inputting the training data set into a pre-constructed lung cancer patient curative effect prediction model to perform model training, and obtaining a trained lung cancer patient curative effect prediction model specifically includes:

b31, inputting the training data set into a pre-constructed lung cancer patient curative effect prediction model to obtain a first prediction result;

b32, determining a loss value of the curative effect prediction model of the lung cancer patient according to the first prediction result and the label;

B33, updating parameters of a lung cancer patient curative effect prediction model through a back propagation algorithm according to the loss value;

and B34, stopping training when the loss value reaches a preset first threshold value or the iteration number reaches a preset second threshold value or the test precision reaches a preset third threshold value, and obtaining a trained lung cancer patient curative effect prediction model.

Specifically, the lung cancer patient curative effect prediction model of the embodiment of the invention can be built based on a convolutional neural network, after data in a training data set is input into the initialized lung cancer patient curative effect prediction model, a prediction result output by the model can be obtained, and the accuracy of the lung cancer patient curative effect prediction model can be evaluated by using the prediction result and the label, so that parameters of the model are updated. For a model for predicting the efficacy of a lung cancer patient, the accuracy of the model prediction can be measured by a Loss Function (Loss Function) defined on a single training data, for measuring the prediction error of one training data, specifically determining the Loss value of the training data through the label of the single training data and the model for the prediction result of the training data. In actual training, one training data set has a lot of training data, so that a Cost Function (Cost Function) is generally adopted to measure the overall error of the training data set, and the Cost Function is defined on the whole training data set and is used for calculating the average value of the prediction errors of all the training data, so that the prediction effect of the model can be better measured. For a general machine learning model, based on the cost function, a regular term for measuring the complexity of the model can be used as a training objective function, and based on the objective function, the loss value of the whole training data set can be obtained. There are many kinds of common loss functions, such as 0-1 loss function, square loss function, absolute loss function, logarithmic loss function, cross entropy loss function, etc., which can be used as the loss function of the machine learning model, and will not be described in detail herein. In the embodiment of the invention, one loss function can be selected to determine the loss value of training. Based on the trained loss value, updating the parameters of the model by adopting a back propagation algorithm, and iterating for several rounds to obtain the trained point cloud pole tower identification model. Specifically, the number of iteration rounds may be preset, or training may be considered complete when the test set meets the accuracy requirements.

The method steps of the embodiments of the present invention are described above. It should be appreciated that in the prior art, no artificial intelligence model capable of predicting the prognosis lifetime of a lung cancer patient based on multi-modal data is available, and in particular, special facial data of traditional Chinese medicine can be processed.

According to the embodiment of the invention, the facial features, CT image features, pathological features, pulse features and clinical features of the lung cancer patient are extracted and feature fusion is carried out to obtain the multi-mode fusion features, the multi-mode fusion features comprise feature information of all aspects of the lung cancer patient, training and prediction of the lung cancer patient curative effect prediction model are carried out based on the multi-mode fusion features, and the comprehensiveness of curative effect prediction and the accuracy of prognosis survival time of the lung cancer patient are improved.

In addition, the embodiment of the invention has the following advantages:

(1) The model of the embodiment of the invention supports mode selection according to clinical actual data, can provide complete 5 modes, can only provide several modes of data, even only provide any one mode, and can perform modeling prediction in a buffet mode.

(2) The later application of the embodiment of the invention has the prompting function of assisting a clinician in revealing macroscopic features of a lung cancer patient and finding out the dominant population selection of a treatment scheme, and belongs to a novel marker.

(3) The embodiment of the invention can predict the curative effect by only providing any one or more data such as CT/pathology/face/pulse/clinical information and the like of routine examination, thereby being convenient to use, beneficial to popularization and free from increasing the burden of patients and medical institutions.

Referring to fig. 3, an embodiment of the present invention provides a multi-modal fusion lung cancer patient efficacy prediction system, including:

the patient information acquisition module is used for acquiring first patient information of a first lung cancer patient, wherein the first patient information comprises first facial information, first CT image information, first pathology slide information, first pulse wave information and first clinical information;

the data preprocessing module is used for preprocessing the data of the first patient information to obtain second patient information;

the feature extraction module is used for respectively inputting second patient information into a plurality of preset neural network models to perform feature extraction to obtain face features, CT image features, pathological features, pulse features and clinical features of the first lung cancer patient;

The feature fusion and prediction module is used for carrying out feature fusion on the face feature, the CT image feature, the pathological feature, the pulse feature and the clinical feature according to preset weights to obtain multi-mode fusion features, and further inputting the multi-mode fusion features into a pre-trained lung cancer patient curative effect prediction model to obtain a curative effect prediction result of the first lung cancer patient.

The content in the method embodiment is applicable to the system embodiment, the functions specifically realized by the system embodiment are the same as those of the method embodiment, and the achieved beneficial effects are the same as those of the method embodiment.

Referring to fig. 4, an embodiment of the present invention provides a multi-modal fusion lung cancer patient efficacy prediction apparatus, including:

at least one processor;

at least one memory for storing at least one program;

The at least one program, when executed by the at least one processor, causes the at least one processor to implement the multi-modal fusion lung cancer patient efficacy prediction method.

The content in the method embodiment is applicable to the embodiment of the device, and the functions specifically realized by the embodiment of the device are the same as those of the method embodiment, and the obtained beneficial effects are the same as those of the method embodiment.

The embodiment of the invention also provides a computer readable storage medium, in which a processor executable program is stored, the processor executable program being used for executing the above-mentioned multi-modal fusion lung cancer patient curative effect prediction method when being executed by a processor.

The computer readable storage medium of the embodiment of the invention can execute the multi-mode fusion lung cancer patient curative effect prediction method provided by the embodiment of the method of the invention, and can execute the implementation steps of any combination of the embodiment of the method, thereby having the corresponding functions and beneficial effects of the method.

Embodiments of the present invention also disclose a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The computer instructions may be read from a computer-readable storage medium by a processor of a computer device, and executed by the processor, to cause the computer device to perform the method shown in fig. 1.

In some alternative embodiments, the functions/acts noted in the block diagrams may occur out of the order noted in the operational illustrations. For example, two blocks shown in succession may in fact be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality/acts involved. Furthermore, the embodiments presented and described in the flowcharts of the present invention are provided by way of example in order to provide a more thorough understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed, and in which sub-operations described as part of a larger operation are performed independently.

Furthermore, while the present invention has been described in the context of functional modules, it should be appreciated that, unless otherwise indicated, one or more of the functions and/or features described above may be integrated in a single physical device and/or software module or one or more of the functions and/or features may be implemented in separate physical devices or software modules. It will also be appreciated that a detailed discussion of the actual implementation of each module is not necessary to an understanding of the present invention. Rather, the actual implementation of the various functional modules in the apparatus disclosed herein will be apparent to those skilled in the art from consideration of their attributes, functions and internal relationships. Accordingly, one of ordinary skill in the art can implement the invention as set forth in the claims without undue experimentation. It is also to be understood that the specific concepts disclosed are merely illustrative and are not intended to be limiting upon the scope of the invention, which is to be defined in the appended claims and their full scope of equivalents.

The above functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on this understanding, the technical solution of the present invention may be embodied in essence or a part contributing to the prior art or a part of the technical solution in the form of a software product stored in a storage medium, including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the above-described method of the various embodiments of the present invention. And the aforementioned storage medium includes: a usb disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.

Logic and/or steps represented in the flowcharts or otherwise described herein, e.g., a ordered listing of executable instructions for implementing logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.

More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium upon which the program described above is printed, as the program described above may be electronically captured, via, for instance, optical scanning of the paper or other medium, then compiled, interpreted, or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory.

It is to be understood that portions of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, the various steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, may be implemented using any one or combination of the following techniques, as is well known in the art: discrete logic circuits having logic gates for implementing logic functions on data signals, application specific integrated circuits having suitable combinational logic gates, programmable Gate Arrays (PGAs), field Programmable Gate Arrays (FPGAs), and the like.

In the foregoing description of the present specification, reference has been made to the terms "one embodiment/example", "another embodiment/example", "certain embodiments/examples", and the like, means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiments or examples. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

While embodiments of the present invention have been shown and described, it will be understood by those of ordinary skill in the art that: many changes, modifications, substitutions and variations may be made to the embodiments without departing from the spirit and principles of the invention, the scope of which is defined by the claims and their equivalents.

While the preferred embodiment of the present application has been described in detail, the present application is not limited to the above embodiments, and various equivalent modifications and substitutions can be made by those skilled in the art without departing from the spirit of the present application, and these equivalent modifications and substitutions are intended to be included in the scope of the present application as defined in the appended claims.