Publication Type : Conference Paper
Publisher : Advances in Intelligent Systems and Computing
Source : Advances in Intelligent Systems and Computing, Springer Verlag, Volume 750, p.519-528 (2019)
ISBN : 9789811318818
Keywords : Artificial intelligence, Big data, Cloud computing, De duplications, It focus, Learning algorithms, Learning systems, Linguistics, Natural language processing systems, Natural languages, Paraphrase, Paraphrase corpus, Quora, Social networking (online), Supervised learning, Supervised machine learning, Twitter
Campus : Coimbatore
School : School of Engineering
Center : Computational Engineering and Networking
Department : Electronics and Communication
Year : 2019
Abstract : Detection of duplicate sentences from a corpus containing a pair of sentences deals with identifying whether two sentences in the pair convey the same meaning or not. This detection of duplicates helps in deduplication, a process in which duplicates are removed. Traditional natural language processing techniques are less accurate in identifying similarity between sentences, such similar sentences can also be referred as paraphrases. Using Quora and Twitter paraphrase corpus, we explored various approaches including several machine learning algorithms to obtain a liable approach that can identify the duplicate sentences given a pair of sentences. This paper discusses the performance of six supervised machine learning algorithms in two different paraphrase corpus, and it focuses on analyzing how accurately the algorithms classify sentences present in the corpus as duplicates and non-duplicates. © 2019, Springer Nature Singapore Pte Ltd.
Cite this Research Publication : S. Viswanathan, Damodaran, N., Simon, A., George, A., M. Kumar, A., and Dr. Soman K. P., “Detection of Duplicates in Quora and Twitter Corpus”, in Advances in Intelligent Systems and Computing, 2019, vol. 750, pp. 519-528.