This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Zhihang Ren, University of California, Berkeley and these authors contributed equally to this work (Email: peter.zhren@berkeley.edu);
(2) Jefferson Ortega, University of California, Berkeley and these authors contributed equally to this work (Email: jefferson_ortega@berkeley.edu);
(3) Yifan Wang, University of California, Berkeley and these authors contributed equally to this work (Email: wyf020803@berkeley.edu);
(4) Zhimin Chen, University of California, Berkeley (Email: zhimin@berkeley.edu);
(5) Yunhui Guo, University of Texas at Dallas (Email: yunhui.guo@utdallas.edu);
(6) Stella X. Yu, University of California, Berkeley and University of Michigan, Ann Arbor (Email: stellayu@umich.edu);
(7) David Whitney, University of California, Berkeley (Email: dwhitney@berkeley.edu).
Table of Links
- Abstract and Intro
- Related Wok
- VEATIC Dataset
- Experiments
- Discussion
- Conclusion
- More About Stimuli
- Annotation Details
- Outlier Processing
- Subject Agreement Across Videos
- Familiarity and Enjoyment Ratings and References
9. Outlier Processing
We assessed whether there were any noisy annotators in our dataset by computing each individual annotator’s agreement with the consensus. This was done by calculating the Pearson correlation between each annotator and the leaveone-out consensus (aggregate of responses except for the current annotator) for each video. Only one observer in our dataset had a correlation smaller than .2 with the leave-oneout consensus rating across videos. We chose .2 as a threshold because it is often used as an indicator of a weak correlation in psychological research. Importantly, if we compare the correlations between the consensus of each video and a consensus that removes the one annotator who shows weak agreement, we get a very high correlation (r = 0.999) indicating that leaving out that subject does not significantly influence the consensus response in our dataset. Thus, we decided to keep the annotator with weak agreement in the dataset in order to avoid removing any important alternative annotations to the videos.
This paper is available on arxiv under CC 4.0 license.