CN112084911A - Human face feature point positioning method and system based on global attention - Google Patents

The invention discloses a human face feature point positioning method and a system based on global attention, which comprises the following steps: acquiring a local image of a human face; inputting the acquired image into a human face feature point positioning model based on global attention trained in advance, and directly outputting the position of a human face feature point after the forward operation of the human face feature point positioning model based on the global attention for the input human face local image; the method and the system for positioning the face feature points based on the global attention acquire the fusion characteristics of the face image with the global semantic information and the local semantic information by utilizing a residual error network mechanism and a global attention fusion mechanism based on the deep learning technology, so that a deep neural network model can take the global information and the local information of the face image into account, the position of the face feature points is accurately calculated, the face feature points are positioned more accurately, and the robustness is higher.

Human face feature point positioning method and system based on global attention

Technical Field

The invention relates to the technical field of face recognition, in particular to a face feature point positioning method and system based on global attention.

Background

The positioning of the human face feature points refers to the positioning of key feature point positions of the face on a human face image through a machine vision technology, wherein the key feature points comprise organ positions such as a mouth corner, an eye corner and a nose tip and positions such as a face contour. The positioning of the face feature points is a technical basis of the application fields of a face recognition system, an expression recognition system, a face attribute analysis system and the like, and the reliability and the accuracy of subsequent work can be directly influenced by the quality of the positioning of the face feature points.

In recent 20 years, the face feature point positioning algorithm is always a research hotspot in the field of machine vision, and a plurality of classical algorithms emerge, and the specific algorithms can be divided into the following categories:

(1) a face feature point positioning algorithm based on the traditional technology is mainly based on a face statistical shape model method and a cascade regression method, such as a classical algorithm: ASM, AAM, SDM, LBF, and the like. The algorithm is characterized in that the geometric position relation of human face organs is utilized, a statistical method and a cascade optimization method are adopted to obtain the final position of the human face characteristic points, the expression capability of the algorithm for extracting the human face characteristics is limited, the shape constraint between the human face characteristic points is not considered, and the positioning accuracy error of the characteristic points of the algorithm is large.

(2) In recent years, Deep Learning technology can simulate a human brain neural network, accurate nonlinear prediction can be performed, various fields are widely concerned and applied, and a group of classical human face feature point positioning network frameworks such as MDM (Mnemunic surface method), PFLD (PFLD: A Practical Facial Landmark Detector), TCDCN (Facial Landmark Detection by Deep Multi-task Learning) and the like appear. The algorithm is characterized in that a convolutional neural network model is used for capturing deep semantic features of the human face, and the final positions of the feature points of the human face are obtained by using the deep semantic features, or based on a multi-branch task training mode, or based on a plurality of cascaded neural network models for iterative optimization training. Compared with a human face feature point positioning algorithm of the traditional technology, the human face feature point positioning accuracy is greatly improved, but local semantic information of a human face is mainly used for positioning the feature points, and the local semantic information cannot comprehensively utilize the overall geometric information of human face organs, so that certain errors exist in positioning of the human face feature points.

Disclosure of Invention

The invention provides a human face feature point positioning method and system based on global attention, which can solve the problems in the background technology.

In order to achieve the purpose, the invention adopts the following technical scheme:

a face feature point positioning method based on global attention comprises the following steps:

acquiring a local image of a human face;

inputting the acquired image into a human face feature point positioning model based on global attention trained in advance, and directly outputting the position of a human face feature point after the forward operation of the human face feature point positioning model based on the global attention for the input human face local image;

wherein,

the network structure of the human face feature point positioning model based on global attention comprises:

the conv0 layer is a convolutional layer with a core size of 7 × 7 and a span of 2 × 2;

the maxpool0 layer is a maximum pooling layer with a kernel size of 2 × 2 and a span of 2 × 2;

the conv0 layer and the maxpool0 layer jointly form a feature map resolution rapid reduction network;

resblock0, resblock1, resblock2 and resblock3 are resblock residual modules of a resnet network;

GFM0, GFM1, GFM2, GFM3 are all global attention fusion modules;

the ave-pool layer is a global mean pooling layer; the fc layer is a fully connected layer with 2xN dimension output characteristic, and N represents the number of human face characteristic points.

Further, the specific network structure of the resblock residual module includes:

the rconv2 layer is a convolutional layer with a core size of 1x1 and a span of 2x2, the rconv0 layer is a convolutional layer with a core size of 3x3 and a span of 2x2, the rconv1 layer, the rconv3 layer and the rconv4 layer are convolutional layers with a core size of 3x3 and a span of 1x1, and the eltsum0 layer and the eltsum1 layer are merging layers and are used for merging a plurality of input feature maps into an output feature map according to corresponding element additions.

Further, the specific network structure of the global attention fusion module includes:

gfmconv0, gfmconv1 and gfmconv2 are convolution layers with the kernel size of 1 × 1 and the span of 1 × 1, and reshape0, reshape1, reshape2 and reshape3 are feature size conversion layers and are used for adjusting the size of an input feature to meet the requirement of subsequent feature layer operation;

globalavodiol 0 is a global mean pooling layer based on the feature map channel dimension, globalmaxpool0 is a global maximum pooling layer based on the feature map channel dimension; splicing the output characteristic diagram of the globavapool 0 layer and the output characteristic diagram of the globalmaxpool0 layer according to the channel dimension; gfmconv is a convolution layer with a kernel size of 7 × 7 and a span of 1 × 1, and is used for extracting importance degree weights of each pixel position on the input feature map;

the sigmod layer is an activation function of the sigmod type; the scale layer is a pixel weighting layer, and is used for weighting the input feature graph one by one according to pixel positions, wherein the weighting calculation process is as shown in formula (1), and a space attention mechanism module is formed by globavapoool 0, globalmaxpool0, gfmconv, sigmod and scale; the softmax layer is used for performing softmax type activation operation according to the 2 nd dimension of the input feature map so as to obtain a probability distribution value of the input feature vector;

matmul0 and matmul1 are both feature map multiplication operation layers and follow a general matrix multiplication rule; matsum is a feature map addition operation layer which is used for adding and combining two input feature maps into an output feature map according to corresponding elements;

O_c(x,y)＝w(x,y)*I_c(x,y) (1)

wherein, O_c(x, y) represents a numerical value at the (x, y) th channel of the output weighted feature map, w (x, y) represents an importance level weight value at the (x, y) position of the input feature map, I_c(x, y) represents the value at the (x, y) th position of the c-th channel of the input feature map.

Further, the training step of the human face feature point location model based on global attention is as follows:

s21, acquiring training sample images, namely collecting face images under various scenes, various light rays and various angles, acquiring a local area image of each face through the existing face detection algorithm, then labeling the positions of N feature points on each face local image, and recording the position information of the feature points;

s22, designing a target loss function of the deep neural network model;

and S23, training the deep neural network model, namely, sending the labeled face sample image set into the well-defined deep neural network model, and learning related model parameters.

Further, the target loss function adopts a mean square error loss function.

In another aspect, the present invention provides a system for locating facial feature points based on global attention, including the following units:

the image acquisition unit is used for acquiring a local image of a human face;

and the face feature point positioning unit is used for inputting the acquired image into a pre-trained face feature point positioning model based on global attention, and directly outputting the position of the face feature point after the forward operation of the face feature point positioning model based on the global attention on the input face local image.

Further, the device also comprises the following sub-units,

the training sample acquisition unit is used for acquiring training sample images, namely collecting face images under various scenes, various light rays and various angles, acquiring a local area image of each face through the existing face detection algorithm, then labeling the positions of N characteristic points on each face local image, and recording the position information of the characteristic points;

the target loss function design unit is used for designing a target loss function of the deep neural network model;

and the deep neural network model training unit is used for sending the labeled human face sample image set into the well-defined deep neural network model and learning related model parameters.

According to the technical scheme, the human face feature point positioning method and system based on the global attention acquire the fusion feature of the human face image with the global semantic information and the local semantic information by utilizing a residual error network mechanism and a global attention fusion mechanism based on the deep learning technology, so that the deep neural network model can give consideration to the global information and the local information of the human face image, the position of the human face feature point is accurately calculated, the positioning of the human face feature point is more accurate, and the robustness is higher.

Drawings

FIG. 1 is a flow chart of a method of the present invention;

FIG. 2 is a model building flow diagram of the present invention;

figure 3 is a diagram of a deep neural network model architecture,

FIG. 4 is a block diagram of a resblock residual module;

FIG. 5 is a block diagram of a global attention fusion module;

wherein the alphanumeric next to each module graphic represents the output feature map size of the current module, namely: the feature map height x feature map width x number of feature map channels.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention.

As shown in fig. 1, the method for positioning human face feature points based on global attention in this embodiment includes:

this embodiment is performed on the premise that a single face partial image has been acquired.

A method for positioning human face feature points based on global attention is disclosed, as shown in FIG. 1, and includes the following steps:

acquiring a local image of a human face;

inputting the acquired image into a human face feature point positioning model based on global attention trained in advance, and directly outputting the position of a human face feature point after the forward operation of the human face feature point positioning model based on the global attention for the input human face local image;

the following is a detailed description:

as shown in fig. 2: the construction steps of the human face feature point positioning model trained in advance based on the global attention are as follows:

s1, designing a deep neural network model, wherein the deep neural network model designed by the invention mainly has the main function of fusing local semantic information of a face image and global semantic information of the face image by means of a well-designed deep neural network model, extracting fusion characteristics of the face image and the global semantic information, and accurately calculating the position of a face feature point. The present invention uses a Convolutional Neural Network (CNN), which defines some terms for convenience of describing the present invention: feature resolution refers to feature height x feature width, feature size refers to feature height x feature width x number of feature channels, kernel size refers to kernel width x kernel height, and span refers to width span x height span, and each convolutional layer is followed by a bulk normalization layer and a nonlinear activation layer. The method specifically comprises the following steps of designing a deep neural network model:

s11, designing an input image of the deep neural network model, wherein the input image adopted by the invention is a 3-channel RGB image with the resolution of 224 x 224, and the larger the size of the input image is, the more details are contained, and the more the accurate positioning of the human face characteristic points is facilitated.

S12, designing a main network of the deep neural network model, wherein the main network is mainly used for acquiring local semantic information of the face image under the condition that the deep neural network model can see the global semantic information of the face image, extracting the fusion characteristics of the face image with the global information and the local information, and the extraction quality of the fusion characteristics of the face directly influences the positioning accuracy of the subsequent face characteristic points. As can be seen from step S11, the size of the input image used in the present invention is large, which is not favorable for the fast operation of the deep neural network model, and therefore, an efficient network capable of fast extracting the features of the input face image is required. As shown in fig. 3, the present invention adopts an improved classical resnet network structure as a model main network, wherein, the conv0 layer is a convolutional layer with a core size of 7 × 7 and a span of 2 × 2; the maxpool0 layer is a maximum pooling layer with a kernel size of 2 × 2 and a span of 2 × 2; the conv0 layer and the maxpool0 layer jointly form a feature map resolution rapid reduction network, and the main function is to rapidly reduce the feature map resolution and reduce the calculation amount of subsequent operations while keeping more image details; resblock0, resblock1, resblock2 and resblock3 are resblock residual modules of a resnet network; GFM0, GFM1, GFM2, GFM3 are all Global attention fusion modules (Global Fuse modules); the ave-pool layer is a global mean pooling layer; the fc layer is a fully connected layer with 2xN dimension output characteristic, and N represents the number of human face characteristic points. The specific network structure of the reslock residual module is shown in fig. 4, the rconv2 layer is a convolutional layer with a core size of 1x1 and a span of 2x2, the rconv0 layer is a convolutional layer with a core size of 3x3 and a span of 2x2, the rconv1 layer, the rconv3 layer and the rconv4 layer are convolutional layers with a core size of 3x3 and a span of 1x1, and the eltsum0 layer and the eltsum1 layer are merging layers and are used for merging a plurality of input feature maps into an output feature map according to corresponding elements; the specific network structure of the global attention fusion module GFM is shown in fig. 5, wherein gfmconv0, gfmconv1, gfmconv2 are convolutional layers with a core size of 1 × 1 and a span of 1 × 1, and reshape0, reshape1, reshape2, and reshape3 are feature size conversion layers, and mainly function to adjust the input feature size to meet the requirement of subsequent feature layer operation; globalavodiol 0 is a global mean pooling layer based on the feature map channel dimension, globalmaxpool0 is a global maximum pooling layer based on the feature map channel dimension; splicing the output characteristic diagram of the globavapool 0 layer and the output characteristic diagram of the globalmaxpool0 layer according to the channel dimension; gfmconv is a convolution layer with a kernel size of 7 × 7 and a span of 1 × 1, and is mainly used for extracting importance degree weights of each pixel position on an input feature map; the sigmod layer is an activation function of the sigmod type; the scale layer is a pixel weighting layer, and is used for weighting the input feature graph one by one according to pixel positions, wherein the weighting calculation process is as shown in formula (1), and a space attention mechanism module is formed by globavapoool 0, globalmaxpool0, gfmconv, sigmod and scale; the softmax layer is mainly used for performing softmax type activation operation according to the 2 nd dimension of the input feature map so as to acquire the probability distribution value of the input feature vector. matmul0 and matmul1 are both feature map multiplication operation layers and follow a general matrix multiplication rule; matsum is a feature map addition operation layer, and is mainly used for adding and combining two input feature maps into one output feature map according to corresponding elements.

O_c(x,y)＝w(x,y)*I_c(x,y) (1)

Wherein, O_c(x, y) represents a numerical value at the (x, y) th channel of the output weighted feature map, w (x, y) represents an importance level weight value at the (x, y) position of the input feature map, I_c(x, y) represents the value at the (x, y) th position of the c-th channel of the input feature map.

S2, training the deep neural network model, optimizing parameters of the deep neural network model mainly through a large amount of labeled training sample data, so that the deep neural network model can accurately position the positions of the characteristic points of the human face, and the specific steps are as follows:

s21, acquiring training sample images, mainly collecting face images under various scenes, various light rays and various angles, acquiring a local area image of each face through the existing face detection algorithm, then labeling the positions of N feature points on each face local image, and recording the position information of the feature points;

s22, designing a target loss function of the deep neural network model, wherein the target loss function is a Mean Square Error (MSE) loss function.

S23, training a deep neural network model, mainly sending a labeled face sample image set into the well-defined deep neural network model, and learning related model parameters;

and S3, directly outputting the positions of the characteristic points of the human face after forward operation of the depth neural network model for any given local image of the human face by using the depth neural network model.

In summary, the method and system for positioning the face feature points based on the global attention of the present invention obtain the fusion features of the face image with the global semantic information and the local semantic information by using the residual network mechanism and the global attention fusion mechanism based on the deep learning technology, so that the deep neural network model can give consideration to the global information and the local information of the face image, accurately calculate the position of the face feature points, and the face feature points are more accurately positioned and have higher robustness.

In another aspect, the present invention provides a system for locating facial feature points based on global attention, including the following units:

the image acquisition unit is used for acquiring a local image of a human face;

and the face feature point positioning unit is used for inputting the acquired image into a pre-trained face feature point positioning model based on global attention, and directly outputting the position of the face feature point after the forward operation of the face feature point positioning model based on the global attention on the input face local image.

Further, the device also comprises the following sub-units,

the training sample acquisition unit is used for acquiring training sample images, namely collecting face images under various scenes, various light rays and various angles, acquiring a local area image of each face through the existing face detection algorithm, then labeling the positions of N characteristic points on each face local image, and recording the position information of the characteristic points;

the target loss function design unit is used for designing a target loss function of the deep neural network model;

and the deep neural network model training unit is used for sending the labeled human face sample image set into the well-defined deep neural network model and learning related model parameters.

In a third aspect, the present invention also discloses a computer readable storage medium storing a computer program which, when executed by a processor, causes the processor to perform the steps of the method as described above.

It is understood that the system provided by the embodiment of the present invention corresponds to the method provided by the embodiment of the present invention, and the explanation, the example and the beneficial effects of the related contents can refer to the corresponding parts in the method.

As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

The above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.