Abstract: Embodiments can provide a computer implemented method, in a data processing system comprising a processor and a memory comprising instructions which are executed by the processor to cause the processor to implement an improved search query generation system, the method comprising inputting a natural language question; parsing the natural language question into a parse tree; identifying argument positions comprising one or more argument position terms, wherein each argument position term is a single word; for each argument position: comparing a head term's discriminator score against a threshold discriminator score; and if the head term surpasses the threshold discriminator score, adding the head term as a required term to an improved search query; and outputting the improved search query.

Abstract: Embodiments can provide a computer implemented method, in a data processing system comprising a processor and a memory comprising instructions which are executed by the processor to cause the processor to implement an improved search query generation system, the method comprising inputting a natural language question; parsing the natural language question into a parse tree; identifying argument positions comprising one or more argument position terms; for each argument position: comparing a head term's discriminator score against a threshold discriminator score; and if the head term surpasses the threshold discriminator score, adding the head term as a required term to an improved search query; and outputting the improved search query.

Abstract: A mechanism is provided in a data processing system for identifying nonsense passages. An annotator in a natural language processing pipeline configured to execute in the data processing system annotates an input passage in a corpus with linguistic features to form an annotated passage. A domain-specific policy is associated with a domain of the corpus. A metric counters component in the natural language processing pipeline counts a number of instances of each type of linguistic feature in the annotated passage to form a set of feature counts. The metric counters component of the natural language processing pipeline determines a value for a metric based on the set of feature counts. The metric is specified in the domain-specific policy. A comparator component of the natural language processing pipeline compares the value for the metric to a predetermined model threshold. The threshold is specified in the domain-specific policy.

Abstract: Embodiments can provide a computer implemented method, in a data processing system comprising a processor and a memory comprising instructions which are executed by the processor to cause the processor to implement an improved search query generation system, the method comprising inputting a natural language question; parsing the natural language question into a parse tree; identifying argument positions comprising one or more argument position terms; for each argument position: comparing a head term's discriminator score against a threshold discriminator score; and if the head term surpasses the threshold discriminator score, adding the head term as a required term to an improved search query; and outputting the improved search query.

Abstract: Embodiments can provide a computer implemented method, in a data processing system comprising a processor and a memory comprising instructions which are executed by the processor to cause the processor to implement an improved search query generation system, the method comprising inputting a natural language question; parsing the natural language question into a parse tree; identifying argument positions comprising one or more argument position terms; for each argument position: comparing a head term's discriminator score against a threshold discriminator score; and if the head term surpasses the threshold discriminator score, adding the head term as a required term to an improved search query; and outputting the improved search query.

Abstract: A mechanism is provided in a data processing system for identifying nonsense passages. The mechanism annotates an input passage with linguistic features to form an annotated passage. The mechanism counts a number of instances of each type of linguistic feature in the annotated passage to form a set of feature counts. The mechanism determines a value for a metric based on the set of feature counts and compares the value for the metric to a predetermined model threshold. The mechanism identifies whether the input passage is a nonsense passage based on a result of the comparison.

Abstract: Embodiments can provide a computer implemented method, in a data processing system comprising a processor and a memory comprising instructions which are executed by the processor to cause the processor to implement an improved search query generation system, the method comprising inputting a natural language question; parsing the natural language question into a parse tree; identifying argument positions comprising one or more argument position terms; for each argument position: comparing a head term's discriminator score against a threshold discriminator score; and if the head term surpasses the threshold discriminator score, adding the head term as a required term to an improved search query; and outputting the improved search query.

Abstract: A mechanism is provided in a data processing system for identifying nonsense passages in documents being ingested into a corpus. A natural language processing pipeline configured to execute in the data processing system receives an input document to be ingested into a corpus. The natural language processing pipeline divides the input document into a plurality of input passages. A filter component of the natural language processing pipeline identifies whether each input passage is a nonsense passage based on a value of a metric determined according to a set of feature counts. The natural language processing pipeline filters each input passage in the plurality of input passages based on whether the input passage is identified as a nonsense passage or not identified as a nonsense passage to form a filtered plurality of input passages. The natural language processing pipeline adds the filtered plurality of input passages into the corpus.

Abstract: A mechanism is provided in a data processing system for identifying nonsense passages in documents being ingested into a corpus. A natural language processing pipeline configured to execute in the data processing system receives an input document to be ingested into a corpus. The natural language processing pipeline divides the input document into a plurality of input passages. A filter component of the natural language processing pipeline identifies whether each input passage is a nonsense passage based on a value of a metric determined according to a set of feature counts. The natural language processing pipeline filters each input passage in the plurality of input passages based on whether the input passage is identified as a nonsense passage or not identified as a nonsense passage to form a filtered plurality of input passages. The natural language processing pipeline adds the filtered plurality of input passages into the corpus.

Abstract: A mechanism is provided in a data processing system for identifying nonsense passages. The mechanism annotates an input passage with linguistic features to form an annotated passage. The mechanism counts a number of instances of each type of linguistic feature in the annotated passage to form a set of feature counts. The mechanism determines a value for a metric based on the set of feature counts and compares the value for the metric to a predetermined model threshold. The mechanism identifies whether the input passage is a nonsense passage based on a result of the comparison.

Abstract: A mechanism is provided in a data processing system for identifying nonsense passages. An annotator in a natural language processing pipeline configured to execute in the data processing system annotates an input passage in a corpus with linguistic features to form an annotated passage. A domain-specific policy is associated with a domain of the corpus. A metric counters component in the natural language processing pipeline counts a number of instances of each type of linguistic feature in the annotated passage to form a set of feature counts. The metric counters component of the natural language processing pipeline determines a value for a metric based on the set of feature counts. The metric is specified in the domain-specific policy. A comparator component of the natural language processing pipeline compares the value for the metric to a predetermined model threshold. The threshold is specified in the domain-specific policy.