Fancy Man Lauches Zippo at WNUT 2020 Shared Task-1: A Bert Case Model for Wet Lab Entity Extraction

Named entity recognition is one of downstream tasks of Bert. Since it has been trained on a large scale corpus, the recommended learning rate is generally small, such as $2e-5$, $3e-5$, $5e-5$ (Devlin et al., 2018). But this needs to be considering with the specific application scenarios. By using the default Bert model: cased without and uncased with, we trained on both 2060s and 2080 with different learning rates.

As shown in table 3, all recommended learning rates did not perform well in this task, and we thought that the dataset provided this time are not common in daily life, so we were ought to increase the learning rate appropriately. At the same time, we noticed that the uncased model is better than the cased model on 2060s, but it is opposite in 2080. Although the best model is obtained by training on 2080, the model training on 2060s is more stable.

Generally speaking, the uncased model is better than the cased model, however, the cased model performs better when there are obvious case differences in specific aspects such as named entity recognition. But we also noticed that we could train an uncased model after processing the text in lowercase.

During testing, we also found that different versions of classes imported will lead to differing results. To better explore the influence of case sensitivity and version, we further trained three possible casing methods, including: cased without, cased with and uncased with (by the way, uncased without should perform the worst, because it can’t actually distinguish case information, and the experimental results are exactly the same, so we omit this possible combination) and two different versions of the combination model. Record the performance of each model and take the top two to get table 4.

Except for some models with slight performance decrease, in most cases, the model can be further improved by using pytorch-transformers to import the required classes. Moreover, we noticed that uncased with is the best model on both 2060s and 2080 when considering the use of previous versions of classes (uncased with V), however, when only the latest version of transformers is used, the cased model works best. But the relationship between whether to use lowercase processing and the final performance is not obvious from our experiments.

Theoretically, words with different case could represent the same named entity, while using lowercase processing can increase the number of training samples but reduce the number of types. So we suggest that when using the updated transformers training, please use cased without, and when using the previous version training, consider using uncased with. What’s more, cased with is also an option worth considering.

We used 0.005 as our default weight decay value before, which based on several simple attempts. Here, we selected the best four models on 2060s and 2080 respectively to adjust the weight decay under the condition of using previous version classes or not, and sorted them out as figure 5.

In theory, the increase of weight decay will make the performance of the model increase to the maximum firstly and then decrease. This is because at the beginning, the model performs poorly in the test set due to over fitting. With the increase of penalty term, the performance of the model is improved and the best value is generated. If the penalty term continues to increase, the model will tend to be more simple model, so the performance of the model drops. However, the actual situation in the evaluation is that the performance of the model firstly increases and then decreases with the increase of weight decay, and then increases to the maximum value and then decreases according to the theory. It is worth noting that the f1-score of case without when weight decay value is 0.009 is the same as that is 0.005, which is difficult to explain according to the classical theory (more specifically, the recall of the former is higher, but the precision of the latter is higher). Though it’s true that the best performance of most models is achieved at 0.005, the final selected model in this experiment is uncased with V on 2080 when the weight decay value is 0.007. The specific index comparison table between this Base case model and baseline is shown in table 5.

Bert case based on Transformer has achieved better results in the recognition of most entity categories than Linear CRF combined with traditional feature engineering. It can be seen that using a large corpus for pre-training combined with downstream task fine-tuning strategies is a very effective and operational relatively simple model paradigm. However, for the output results of Bert case without CRF standardizing, we can find that many unreasonable annotation results are generated in the test set, and on some indicators, the model does not perform as well as the baseline, especially in the prediction results on Temperature and Measure. All in all, Bert can be used as a relatively good benchmark, but there is still much room for improvement. We have planned to replace it with other pre-trained models such as Roberta, Albert, XLNet, and connected it with CRF to observe the effect of the experiment, but it finally failed to achieve due to limited time and capacity.