2021
Chen, Shuguang; Neves, Leonardo; Solorio, Thamar
Mitigating Temporal-Drift: A Simple Approach to Keep NER Models Crisp Conference
Ninth International Workshop on Natural Language Processing for Social Media (SocialNLP @ NAACL 2021), 2021.
Abstract | Links | BibTeX | Tags: NER, Social Media
@conference{Chen2021,
title = {Mitigating Temporal-Drift: A Simple Approach to Keep NER Models Crisp},
author = {Shuguang Chen and Leonardo Neves and Thamar Solorio},
url = {https://arxiv.org/abs/2104.09742
https://github.com/RiTUAL-UH/trending_NER},
year = {2021},
date = {2021-04-19},
publisher = {Ninth International Workshop on Natural Language Processing for Social Media (SocialNLP @ NAACL 2021)},
abstract = {Performance of neural models for named entity recognition degrades over time, becoming stale. This degradation is due to temporal drift, the change in our target variables' statistical properties over time. This issue is especially problematic for social media data, where topics change rapidly. In order to mitigate the problem, data annotation and retraining of models is common. Despite its usefulness, this process is expensive and time-consuming, which motivates new research on efficient model updating. In this paper, we propose an intuitive approach to measure the potential trendiness of tweets and use this metric to select the most informative instances to use for training. We conduct experiments on three state-of-the-art models on the Temporal Twitter Dataset. Our approach shows larger increases in prediction accuracy with less training data than the alternatives, making it an attractive, practical solution.},
keywords = {NER, Social Media},
pubstate = {published},
tppubtype = {conference}
}
Performance of neural models for named entity recognition degrades over time, becoming stale. This degradation is due to temporal drift, the change in our target variables’ statistical properties over time. This issue is especially problematic for social media data, where topics change rapidly. In order to mitigate the problem, data annotation and retraining of models is common. Despite its usefulness, this process is expensive and time-consuming, which motivates new research on efficient model updating. In this paper, we propose an intuitive approach to measure the potential trendiness of tweets and use this metric to select the most informative instances to use for training. We conduct experiments on three state-of-the-art models on the Temporal Twitter Dataset. Our approach shows larger increases in prediction accuracy with less training data than the alternatives, making it an attractive, practical solution.
2018
Aguilar, Gustavo; AlGhamdi, Fahad; Soto, Victor; Diab, Mona; Hirschberg, Julia; Solorio, Thamar
Named Entity Recognition on Code-Switched Data: Overview of the CALCS 2018 Shared Task Inproceedings
In: for Computational Linguistics, Association (Ed.): Proceedings of the Third Workshop on Computational Approaches to Linguistic Code-Switching, Association for Computational Linguistics, Melbourne, Australia, 2018.
Abstract | Links | BibTeX | Tags: Code-Switching, English-Spanish, Modern Standard Arabic-Egyptian, NER, shared task, Social Media
@inproceedings{aguilar@calcs2018,
title = {Named Entity Recognition on Code-Switched Data: Overview of the CALCS 2018 Shared Task},
author = {Gustavo Aguilar and Fahad AlGhamdi and Victor Soto and Mona Diab and Julia Hirschberg and Thamar Solorio},
editor = {Association for Computational Linguistics },
url = {http://www.aclweb.org/anthology/W18-3219},
year = {2018},
date = {2018-07-15},
booktitle = {Proceedings of the Third Workshop on Computational Approaches to Linguistic Code-Switching},
publisher = {Association for Computational Linguistics},
address = {Melbourne, Australia},
abstract = {In the third shared task of the Computational Approaches to Linguistic CodeSwitching (CALCS) workshop, we focus on Named Entity Recognition (NER) on code-switched social-media data. We divide the shared task into two competitions based on the English-Spanish (ENG-SPA) and Modern Standard Arabic-Egyptian (MSA-EGY) language pairs. We use Twitter data and 9 entity types to establish a new dataset for code-switched NER benchmarks. In addition to the CS phenomenon, the diversity of the entities and the social media challenges make the task considerably hard to process. As a result, the best scores of the competitions are 63.76% and 71.61% for ENG-SPA and MSA-EGY, respectively. We present the scores of 9 participants and discuss the most common challenges among submissions.},
keywords = {Code-Switching, English-Spanish, Modern Standard Arabic-Egyptian, NER, shared task, Social Media},
pubstate = {published},
tppubtype = {inproceedings}
}
In the third shared task of the Computational Approaches to Linguistic CodeSwitching (CALCS) workshop, we focus on Named Entity Recognition (NER) on code-switched social-media data. We divide the shared task into two competitions based on the English-Spanish (ENG-SPA) and Modern Standard Arabic-Egyptian (MSA-EGY) language pairs. We use Twitter data and 9 entity types to establish a new dataset for code-switched NER benchmarks. In addition to the CS phenomenon, the diversity of the entities and the social media challenges make the task considerably hard to process. As a result, the best scores of the competitions are 63.76% and 71.61% for ENG-SPA and MSA-EGY, respectively. We present the scores of 9 participants and discuss the most common challenges among submissions.
Aguilar, Gustavo; Monroy, A. Pastor López; Gonzalez, Fabio A.; Solorio, Thamar
Modeling Noisiness to Recognize Named Entities using Multitask Neural Networks on Social Media Inproceedings
In: for Computational Linguistics, Association (Ed.): Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Association for Computational Linguistics, New Orleans, Louisiana, 2018.
Abstract | Links | BibTeX | Tags: CRF, Multitask, NER, Phonetics, Phonology, Social Media
@inproceedings{gaguilar2018,
title = {Modeling Noisiness to Recognize Named Entities using Multitask Neural Networks on Social Media},
author = {Gustavo Aguilar and A. Pastor López Monroy and Fabio A. Gonzalez and Thamar Solorio},
editor = {Association for Computational Linguistics },
url = {http://www.aclweb.org/anthology/N18-1127},
year = {2018},
date = {2018-06-01},
booktitle = {Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
publisher = {Association for Computational Linguistics},
address = {New Orleans, Louisiana},
abstract = {Recognizing named entities in a document is a key task in many NLP applications. Although current state-of-the-art approaches to this task reach a high performance on clean text (e.g. newswire genres), those algorithms dramatically degrade when they are moved to noisy environments such as social media domains. We present two systems that address the challenges of processing social media data using character-level phonetics and phonology, word embeddings, and Part-of-Speech tags as features. The first model is a multitask end-to-end Bidirectional Long Short-Term Memory (BLSTM)-Conditional Random Field (CRF) network whose output layer contains two CRF classifiers. The second model uses a multitask BLSTM network as feature extractor that transfers the learning to a CRF classifier for the final prediction. Our systems outperform the current F1 scores from state-of-the-art on the Workshop on Noisy User-generated Text 2017 dataset by 2.45% and 3.69%, establishing a more suitable approach for social media environments. },
keywords = {CRF, Multitask, NER, Phonetics, Phonology, Social Media},
pubstate = {published},
tppubtype = {inproceedings}
}
Recognizing named entities in a document is a key task in many NLP applications. Although current state-of-the-art approaches to this task reach a high performance on clean text (e.g. newswire genres), those algorithms dramatically degrade when they are moved to noisy environments such as social media domains. We present two systems that address the challenges of processing social media data using character-level phonetics and phonology, word embeddings, and Part-of-Speech tags as features. The first model is a multitask end-to-end Bidirectional Long Short-Term Memory (BLSTM)-Conditional Random Field (CRF) network whose output layer contains two CRF classifiers. The second model uses a multitask BLSTM network as feature extractor that transfers the learning to a CRF classifier for the final prediction. Our systems outperform the current F1 scores from state-of-the-art on the Workshop on Noisy User-generated Text 2017 dataset by 2.45% and 3.69%, establishing a more suitable approach for social media environments.