2021 |
Chen, Shuguang; Neves, Leonardo; Solorio, Thamar Mitigating Temporal-Drift: A Simple Approach to Keep NER Models Crisp Conference Ninth International Workshop on Natural Language Processing for Social Media (SocialNLP @ NAACL 2021), 2021. Abstract | Links | BibTeX | Tags: NER, Social Media @conference{Chen2021, title = {Mitigating Temporal-Drift: A Simple Approach to Keep NER Models Crisp}, author = {Shuguang Chen and Leonardo Neves and Thamar Solorio}, url = {https://arxiv.org/abs/2104.09742 https://github.com/RiTUAL-UH/trending_NER}, year = {2021}, date = {2021-04-19}, publisher = {Ninth International Workshop on Natural Language Processing for Social Media (SocialNLP @ NAACL 2021)}, abstract = {Performance of neural models for named entity recognition degrades over time, becoming stale. This degradation is due to temporal drift, the change in our target variables’ statistical properties over time. This issue is especially problematic for social media data, where topics change rapidly. In order to mitigate the problem, data annotation and retraining of models is common. Despite its usefulness, this process is expensive and time-consuming, which motivates new research on efficient model updating. In this paper, we propose an intuitive approach to measure the potential trendiness of tweets and use this metric to select the most informative instances to use for training. We conduct experiments on three state-of-the-art models on the Temporal Twitter Dataset. Our approach shows larger increases in prediction accuracy with less training data than the alternatives, making it an attractive, practical solution.}, keywords = {NER, Social Media}, pubstate = {published}, tppubtype = {conference} } Performance of neural models for named entity recognition degrades over time, becoming stale. This degradation is due to temporal drift, the change in our target variables’ statistical properties over time. This issue is especially problematic for social media data, where topics change rapidly. In order to mitigate the problem, data annotation and retraining of models is common. Despite its usefulness, this process is expensive and time-consuming, which motivates new research on efficient model updating. In this paper, we propose an intuitive approach to measure the potential trendiness of tweets and use this metric to select the most informative instances to use for training. We conduct experiments on three state-of-the-art models on the Temporal Twitter Dataset. Our approach shows larger increases in prediction accuracy with less training data than the alternatives, making it an attractive, practical solution. |
2018 |
Aguilar, Gustavo; AlGhamdi, Fahad; Soto, Victor; Diab, Mona; Hirschberg, Julia; Solorio, Thamar Named Entity Recognition on Code-Switched Data: Overview of the CALCS 2018 Shared Task Inproceedings for Linguistics, Association Computational (Ed.): Proceedings of the Third Workshop on Computational Approaches to Linguistic Code-Switching, Association for Computational Linguistics, Melbourne, Australia, 2018. Abstract | Links | BibTeX | Tags: Code-Switching, English-Spanish, Modern Standard Arabic-Egyptian, NER, shared task, Social Media @inproceedings{aguilar@calcs2018, title = {Named Entity Recognition on Code-Switched Data: Overview of the CALCS 2018 Shared Task}, author = {Gustavo Aguilar and Fahad AlGhamdi and Victor Soto and Mona Diab and Julia Hirschberg and Thamar Solorio}, editor = {Association for Computational Linguistics }, url = {http://www.aclweb.org/anthology/W18-3219}, year = {2018}, date = {2018-07-15}, booktitle = {Proceedings of the Third Workshop on Computational Approaches to Linguistic Code-Switching}, publisher = {Association for Computational Linguistics}, address = {Melbourne, Australia}, abstract = {In the third shared task of the Computational Approaches to Linguistic CodeSwitching (CALCS) workshop, we focus on Named Entity Recognition (NER) on code-switched social-media data. We divide the shared task into two competitions based on the English-Spanish (ENG-SPA) and Modern Standard Arabic-Egyptian (MSA-EGY) language pairs. We use Twitter data and 9 entity types to establish a new dataset for code-switched NER benchmarks. In addition to the CS phenomenon, the diversity of the entities and the social media challenges make the task considerably hard to process. As a result, the best scores of the competitions are 63.76% and 71.61% for ENG-SPA and MSA-EGY, respectively. We present the scores of 9 participants and discuss the most common challenges among submissions.}, keywords = {Code-Switching, English-Spanish, Modern Standard Arabic-Egyptian, NER, shared task, Social Media}, pubstate = {published}, tppubtype = {inproceedings} } In the third shared task of the Computational Approaches to Linguistic CodeSwitching (CALCS) workshop, we focus on Named Entity Recognition (NER) on code-switched social-media data. We divide the shared task into two competitions based on the English-Spanish (ENG-SPA) and Modern Standard Arabic-Egyptian (MSA-EGY) language pairs. We use Twitter data and 9 entity types to establish a new dataset for code-switched NER benchmarks. In addition to the CS phenomenon, the diversity of the entities and the social media challenges make the task considerably hard to process. As a result, the best scores of the competitions are 63.76% and 71.61% for ENG-SPA and MSA-EGY, respectively. We present the scores of 9 participants and discuss the most common challenges among submissions. |
Aguilar, Gustavo; Monroy, Pastor López A; Gonzalez, Fabio A; Solorio, Thamar Modeling Noisiness to Recognize Named Entities using Multitask Neural Networks on Social Media Inproceedings for Linguistics, Association Computational (Ed.): Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Association for Computational Linguistics, New Orleans, Louisiana, 2018. Abstract | Links | BibTeX | Tags: CRF, Multitask, NER, Phonetics, Phonology, Social Media @inproceedings{gaguilar2018, title = {Modeling Noisiness to Recognize Named Entities using Multitask Neural Networks on Social Media}, author = {Gustavo Aguilar and A. Pastor López Monroy and Fabio A. Gonzalez and Thamar Solorio}, editor = {Association for Computational Linguistics }, url = {http://www.aclweb.org/anthology/N18-1127}, year = {2018}, date = {2018-06-01}, booktitle = {Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies}, publisher = {Association for Computational Linguistics}, address = {New Orleans, Louisiana}, abstract = {Recognizing named entities in a document is a key task in many NLP applications. Although current state-of-the-art approaches to this task reach a high performance on clean text (e.g. newswire genres), those algorithms dramatically degrade when they are moved to noisy environments such as social media domains. We present two systems that address the challenges of processing social media data using character-level phonetics and phonology, word embeddings, and Part-of-Speech tags as features. The first model is a multitask end-to-end Bidirectional Long Short-Term Memory (BLSTM)-Conditional Random Field (CRF) network whose output layer contains two CRF classifiers. The second model uses a multitask BLSTM network as feature extractor that transfers the learning to a CRF classifier for the final prediction. Our systems outperform the current F1 scores from state-of-the-art on the Workshop on Noisy User-generated Text 2017 dataset by 2.45% and 3.69%, establishing a more suitable approach for social media environments. }, keywords = {CRF, Multitask, NER, Phonetics, Phonology, Social Media}, pubstate = {published}, tppubtype = {inproceedings} } Recognizing named entities in a document is a key task in many NLP applications. Although current state-of-the-art approaches to this task reach a high performance on clean text (e.g. newswire genres), those algorithms dramatically degrade when they are moved to noisy environments such as social media domains. We present two systems that address the challenges of processing social media data using character-level phonetics and phonology, word embeddings, and Part-of-Speech tags as features. The first model is a multitask end-to-end Bidirectional Long Short-Term Memory (BLSTM)-Conditional Random Field (CRF) network whose output layer contains two CRF classifiers. The second model uses a multitask BLSTM network as feature extractor that transfers the learning to a CRF classifier for the final prediction. Our systems outperform the current F1 scores from state-of-the-art on the Workshop on Noisy User-generated Text 2017 dataset by 2.45% and 3.69%, establishing a more suitable approach for social media environments. |
2017 |
Gustavo Aguilar Suraj Maharjan, Pastor López Monroy Thamar Solorio A A Multi-task Approach for Named Entity Recognition on Social Media Data Inproceedings Proceedings of 3rd Workshop on Noisy User-generated Text, WNUT 2017., 2017, (Ranked 1st place in the two evaluation metrics). Abstract | Links | BibTeX | Tags: CRF, Deeplearning, Multitask, NER @inproceedings{aguilar-EtAl:2017:WNUT, title = {A Multi-task Approach for Named Entity Recognition on Social Media Data}, author = {Gustavo Aguilar, Suraj Maharjan, A. Pastor López Monroy, Thamar Solorio}, url = {http://www.aclweb.org/anthology/W17-4419}, year = {2017}, date = {2017-09-07}, publisher = {Proceedings of 3rd Workshop on Noisy User-generated Text, WNUT 2017.}, abstract = {Named Entity Recognition for social media data is challenging because of its inherent noisiness. In addition to improper grammatical structures, it contains spelling inconsistencies and numerous informal abbreviations. We propose a novel multi-task approach by employing a more general secondary task of Named Entity (NE) segmentation together with the primary task of fine-grained NE categorization. The multi-task neural network architecture learns higher order feature representations from word and character sequences along with basic Part-of-Speech tags and gazetteer information. This neural network acts as a feature extractor to feed a Conditional Random Fields classifier. We were able to obtain the first position in the 3rd Workshop on Noisy User-generated Text (WNUT-2017) with a 41.86% entity F1-score and a 40.24% surface F1-score.}, note = {Ranked 1st place in the two evaluation metrics}, keywords = {CRF, Deeplearning, Multitask, NER}, pubstate = {published}, tppubtype = {inproceedings} } Named Entity Recognition for social media data is challenging because of its inherent noisiness. In addition to improper grammatical structures, it contains spelling inconsistencies and numerous informal abbreviations. We propose a novel multi-task approach by employing a more general secondary task of Named Entity (NE) segmentation together with the primary task of fine-grained NE categorization. The multi-task neural network architecture learns higher order feature representations from word and character sequences along with basic Part-of-Speech tags and gazetteer information. This neural network acts as a feature extractor to feed a Conditional Random Fields classifier. We were able to obtain the first position in the 3rd Workshop on Noisy User-generated Text (WNUT-2017) with a 41.86% entity F1-score and a 40.24% surface F1-score. |