423,723 research outputs found
Multilingual Sentence Categorization according to Language
In this paper, we describe an approach to sentence categorization which has
the originality to be based on natural properties of languages with no training
set dependency. The implementation is fast, small, robust and textual errors
tolerant. Tested for french, english, spanish and german discrimination, the
system gives very interesting results, achieving in one test 99.4% correct
assignments on real sentences.
The resolution power is based on grammatical words (not the most common
words) and alphabet. Having the grammatical words and the alphabet of each
language at its disposal, the system computes for each of them its likelihood
to be selected. The name of the language having the optimum likelihood will tag
the sentence --- but non resolved ambiguities will be maintained. We will
discuss the reasons which lead us to use these linguistic facts and present
several directions to improve the system's classification performance.
Categorization sentences with linguistic properties shows that difficult
problems have sometimes simple solutions.Comment: 4 pages --- LaTe
PTE: Predictive Text Embedding through Large-scale Heterogeneous Text Networks
Unsupervised text embedding methods, such as Skip-gram and Paragraph Vector,
have been attracting increasing attention due to their simplicity, scalability,
and effectiveness. However, comparing to sophisticated deep learning
architectures such as convolutional neural networks, these methods usually
yield inferior results when applied to particular machine learning tasks. One
possible reason is that these text embedding methods learn the representation
of text in a fully unsupervised way, without leveraging the labeled information
available for the task. Although the low dimensional representations learned
are applicable to many different tasks, they are not particularly tuned for any
task. In this paper, we fill this gap by proposing a semi-supervised
representation learning method for text data, which we call the
\textit{predictive text embedding} (PTE). Predictive text embedding utilizes
both labeled and unlabeled data to learn the embedding of text. The labeled
information and different levels of word co-occurrence information are first
represented as a large-scale heterogeneous text network, which is then embedded
into a low dimensional space through a principled and efficient algorithm. This
low dimensional embedding not only preserves the semantic closeness of words
and documents, but also has a strong predictive power for the particular task.
Compared to recent supervised approaches based on convolutional neural
networks, predictive text embedding is comparable or more effective, much more
efficient, and has fewer parameters to tune.Comment: KDD 201
Text categorization and similarity analysis: similarity measure, architecture and design
This research looks at the most appropriate similarity measure to use for a document classification problem. The goal is to find a method that is accurate in finding both semantically and version related documents. A necessary requirement is that the method is efficient in its speed and disk usage. Simhash is found to be the measure best suited to the application and it can be combined with other software to increase the accuracy. Pingar have provided an API that will extract the entities from a document and create a taxonomy displaying the relationships and this extra information can be used to accurately classify input documents. Two algorithms are designed incorporating the Pingar API and then finally an efficient comparison algorithm is introduced to cut down the comparisons required
- …