METHODS AND SYSTEMS FOR DETERMINING RELEVANCE OF DOCUMENTS
A method for screening documents. The order of the text in the documents can be changed in order to create additional documents. A vocabulary of words from the text in the documents can be built. The words can be coded as vectors. Vector relationships can be defined. The documents can be classified using the vector relationships.
Latest The Boston Consulting Group, Inc. Patents:
This application is a continuation application to U.S. application Ser. No. 16/891,396 filed Jun. 3, 2020, which is a continuation application to U.S. application Ser. No. 16/203,102 filed Nov. 28, 2018, which claims the benefit of U.S. Provisional Application No. 62/621,404 filed Jan. 24, 2018, which are incorporated by reference in their entireties.
BRIEF DESCRIPTION OF THE DRAWINGSThe following description illustrates example embodiments for screening documents. Although resumes are used throughout this application as example embodiments, those of ordinary skill in the art will see that many other types of documents may also be utilized.
Display device 106 may be any known display technology, including but not limited to display devices using Liquid Crystal Display (LCD) or Light Emitting Diode (LED) technology. Processor(s) 102 may use any known processor technology, including but not limited to graphics processors and multi-core processors. Input device 104 may be any known input device technology, including but not limited to a keyboard (including a virtual keyboard), mouse, track ball, and touch-sensitive pad or display. Bus 112 may be any known internal or external bus technology, including but not limited to ISA, EISA, PCI, PCI Express, NuBus, USB, Serial ATA or FireWire. Computer-readable medium 110 may be any medium that participates in providing instructions to processor(s) 102 for execution, including without limitation, non-volatile storage media (e.g., optical disks, magnetic disks, flash drives, etc.), or volatile media (e.g., SDRAM, ROM, etc.).
Computer-readable medium 110 may include various instructions 114 for implementing an operating system (e.g., Mac OS®, Windows®, Linux). The operating system may be multi-user, multiprocessing, multitasking, multithreading, real-time, and the like. The operating system may perform basic tasks, including but not limited to: recognizing input from input device 104; sending output to display device 106; keeping track of files and directories on computer-readable medium 110; controlling peripheral devices (e.g., disk drives, printers, etc.) which can be controlled directly or through an I/O controller; and managing traffic on bus 112. Network communications instructions 116 may establish and maintain network connections (e.g., software for implementing communication protocols, such as TCP/IP, HTTP, Ethernet, etc.).
The screening process can be utilized to determine the probability that a resume is of interest. For example, if 1000 resumes are submitted for a particular position, the screening process can provide a probability of relevance for each resume and rank the order of the resumes according to the probability of relevance rankings. In this way, a hiring person can determine almost immediately which resumes are the most important. This may be valuable, for example, when it is important to make an early contact for a prime candidate for a position, rather than waiting several weeks or months to get to that particular resume in a list or pile of resumes. This process can also, for example, cut down significantly on the amount of time it takes to review resumes. For example, when a probability is given for each resume, a hiring person can start reviewing the resumes with the highest probabilities first in order to maximize the chances that the most relevant or desirable hires are reviewed and interviewed first. The screening process can also, for example, cut down on the time it takes to interview a candidate. For example, if a resume is determined to be a very good candidate, the first pre-screening interview can focus on language skills and personality, instead of qualifications, which can result in a significant reduction of the time spent in the first interview.
In some embodiments, resumes that are determined to have a low probability of relevance can be reviewed, and if it is determined that they should have a higher probability of relevance, these can be fed into the system in order to better train the system. In a similar way, resumes that are determined to have a high probability of relevance can be reviewed, and if it is determined that they should have a lower probability of relevance, these can be fed into the system in order to better train the system.
With respect to the model training, two folders of PDF documents can be created, one with examples of good candidates or resumes, and one with examples of bad candidates or resumes. The PDF documents can be converted to text. Sentences in the resumes can be pre-processed. The words can be mapped to embeddings. The data can then be augmented. A model code and performance metrics can be output with accuracy, recall (e.g., true positives divided by the sum of true positives and false negatives) and precision (e.g., true positives divided by the sum of the true positives and the false positives) information.
With respect to candidate predictions, three set of data may be input: resumes (in PDF format) to be scored, the relevant vocabulary list, and the model. Each PDF document can be run through the model and a probability of being a good candidate can be assigned. The output can be a prioritized list of candidates to interview.
Example EmbodimentsVocabulary Building. In some embodiments, the following process can be used to build the vocabulary.
-
- 1. Import resume file names into a dataframe.
- 2. For each line of the dataframe.
- a. Read the resume file name.
- b. Import the PDF file.
- c. Transform the PDF into text and append it as a column of the dataframe.
- d. Process the text and append the list of words as a column of the dataframe.
- e. Calculate the number of words in each resume and append it as a column to the dataframe.
- 3. Create a set including all the words in the resumes.
- 4. Count how many times each word appears.
- 5. Sort words by count in descending order.
- 6. Create a dictionary of vocabulary words using number to word matching based on previous sorting.
PDF to Text Transformation. In some embodiments, the following process can be used to transform the pdf document to text.
-
- 1. Create a list of words (e.g., English, French, any language) present in all the resumes we are going to examine. Each word can be called a “token”.
- 2. The resumes used as input can be stored in PDF, which can be the format candidates use to submit the resume in order to apply to a job. Because the PDF format may not allow immediate token identification and manipulation, we can transform the PDF documents into text documents using the example function set forth in
FIG. 5 which takes a PDF document as input and returns a text version of the document as output. The function inFIG. 5 can be re-used at any stage of the process to transform a PDF into text. For example, this process can be used on a batch of resumes being used to train the model. This function can also be used when a new pdf resume is received and checked to see if it is good (e.g., a desirable candidate). In both these cases pdf files need to be transformed into text. Of course, if a resume is received in a format other than PDF, it can be easily converted into PDF using standard PDF technology and then the PDF document can be transformed into text.
Tokenization.
-
- 1. Once the PDF documents have been transformed into text, the tokens (e.g., words) can be extracted.
FIG. 6A illustrates an example function where a text file is input, and a set of tokens are output. The text can be split into words, sentences, and/or paragraphs using a set of regular expressions and character based rules (e.g., using Regex). This function can be re-used at any stage of the process to tokenize text. For example, this function can be used on a batch of resumes being used to train the model, but can also be used when a new pdf resume is received and is being checked for the probability of the resume being a good one. In both cases, the text can be tokenized. - 2. Once we have the tokens, we can count how many times each word appears in the corpus (e.g., the set of documents considered) as shown in
FIG. 6B . - 3. When we have the count of how many times each word appears in the corpus (e.g., the body of resumes), we can sort those words in decreasing order (e.g., the words with the highest count appear first) as shown in
FIG. 6C and we can assign each word their ranking number (e.g., index) in this special sorted list. If you have two words appearing the exact same number of times, in some embodiments, they can be given subsequent numbers in alphabetical order.
- 1. Once the PDF documents have been transformed into text, the tokens (e.g., words) can be extracted.
Training. The training procedure can assume there is a set of resume PDF documents that have been stored in two different folders. Folder “yes” can include candidates that have been deemed to be successful in the past after a pre-screening hiring round by a hiring team of person(s). Folder “no” can include candidates who did not pass the pre-screening hiring round by the hiring team of person(s). In some embodiment, the following process can be used, For example, using Python as the coding language, with a Keras library and a TensorFlow backend.
-
- 1. Import resume file names from the “yes'” folder into a dataframe.
- 2. Create a column “proceed”.
- 3. For each resume append the value 1 in the column “proceed”.
- 4. Import resume file names from the “no” folder into a dataframe.
- 5. For each resume append the value 0 in the column “proceed”.
- 6. For each line of the dataframe:
- a. Read the resume file name.
- b. Import the PDF file.
- c. Transform the PDF into text and append it as a column of the dataframe.
- d. Process the text and append the list of words as a column of the dataframe.
- e. Transform list of word into a list of indexes using the pre-built vocabulary and append it to the dataframe.
- 7. Split dataset into train and validation randomly (e.g., using known machine learning techniques).
- 8. Augment data by using a sentence and/or paragraph shuffling procedure. This can be done only for the train dataset in some embodiments.
- 9. Append the additional resumes created in the previous step to the training dataset.
- 10. Pre-process data to feed Deep Neural networks.
- 11. Train Deep Neural networks.
Resume Import and Processing. The example function of
Data Augmentation. Because machines learn from examples, it is quite helpful to have more examples (e.g., hundreds or thousands or more). By using the fact that sentences resumes are quite independent, we can create “new” example resumes by shuffling sentences belonging to the resume around. This technique can improve the prediction power of the model. For example, in
Deep Neural Network. We are now ready to do some pre-processing of the encoded resumes before feeding them to our Deep Neural Network.
A deep neural network (DNN) is an ANN (Artificial Neural Network) with multiple hidden layers between the input and output layers.
https://en.wikipedia.org/wiki/Deep_learning#Deep_neural_networks.
The following code represents a Deep Neural Network written in Keras
Each row is a layer of the network gets input from the previous layer, applies the layer specific transformations and provides its output as the following layer input.
Transforms the output of dropout into 1 positive between 0 and 1 which represents the probability of the resume to be a ‘good’ one.
Stochastic gradient descent (often shortened to SGD), also known as incremental gradient descent, is a stochastic approximation of the gradient descent optimization and iterative method for minimizing an objective function that is written as a sum of differentiable functions.
https://en.wikipedia.org/wiki/Stochastic_gradient_descent. When we are finished training the model, we can save both the embeddings value and the model multipliers into a file.
Predicting Candidate “Goodness”. To predict how a new candidate will perform during pre-screening steps, we can load the model file and pass to it the PDF to get a prediction. The prediction process set forth below can be called scoring and for each resume it can return the predicted probability of the candidate passing an HR department's pre-screening tests.
-
- 1. Import resume into a dataframe
- 2. For each line of the dataframe.
- a. Read the resume file name.
- b. Import the PDF file.
- c. Transform the PDF into text and append it as a column of the dataframe.
- d. Process the text and append the list of words as a column of the dataframe.
- e. Transform list of word into a list of indexes using the pre-built vocabulary and append it to the dataframe.
- f. Apply same pre-processing steps as was done in the Deep Neural Network section.
- 3. Load model.
- 4. Feed resume to model.
- 5. Get prediction.
- 6. Output prediction to .csv file
Once the PDF has been transformed into a sequence of indexes and pre-processed as described above, the scoring procedure comes from multiple transformations of the word vectors using model parameters found during training for the neural network functions. In very simplified terms, this transformation can be represented as y(x)=f(x), where: y can be a predicted probability of a candidate being “good”; f can be the set of parametrized transformations happening as result of optimized neural network parameters determined by the Deep Neural Network algorithm during the model training procedure; and x can be the encoded resume (which can be translated into embeddings during the scoring procedure).
predictions=NN.predict(X_cand, verbose=1)
The output can then reshaped to a nice .cvs format as illustrated in
While various embodiments have been described above, it should be understood that they have been presented by way of example and not limitation. It will be apparent to persons skilled in the relevant art(s) that various changes in form and detail can be made therein without departing from the spirit and scope. In fact, after reading the above description, it will be apparent to one skilled in the relevant art(s) how to implement alternative embodiments. For example, other steps may be provided, or steps may be eliminated, from the described flows, and other components may be added to, or removed from, the described systems. Accordingly, other implementations are within the scope of the following claims.
In addition, it should be understood that any figures which highlight the functionality and advantages are presented for example purposes only. The disclosed methodology and system are each sufficiently flexible and configurable such that they may be utilized in ways other than that shown.
Although the term “at least one” may often be used in the specification, claims and drawings, the terms “a”, “an”, “the”, “said”, etc. also signify “at least one” or “the at least one” in the specification, claims and drawings.
Finally, it is the applicant's intent that only claims that include the express language “means for” or “step for” be interpreted under 35 U.S.C. 112(f). Claims that do not expressly include the phrase “means for” or “step for” are not to be interpreted under 35 U.S.C. 112(f).
Claims
1. A method for screening documents, comprising:
- changing the order of the text in the documents in order to create additional documents;
- building a vocabulary of words from the text in the documents;
- coding the words as vectors;
- defining vector relationships; and
- classifying the documents utilizing the vector relationships.
Type: Application
Filed: Oct 10, 2022
Publication Date: Mar 2, 2023
Applicant: The Boston Consulting Group, Inc. (Boston, MA)
Inventor: Emilio Alfredo LAPIELLO (BOSTON, MA)
Application Number: 18/045,308