Abstract
Observer agreement is often regarded as the sine qua non of observational research. Cohen’s κ ?is a widely used index and is appropriate when discrete entities—such as a turn-of-talk or a demarcated time interval—are presented to pairs of observers to code. κ-like statistics and agreement matrices are also used for the timed-event sequential data produced when observers first segment and then code events detected in the stream of behavior, noting onset and offset times. Such κs are of two kinds: time-based and event-based. Available for download is a computer program (OASTES; Observer Agreement for Simulated Timed Event Sequences) that simulates the coding of observers of a stated accuracy and then computes agreement statistics for two time-based κs (with and without tolerance) and three event-based κs (one implemented in The Observer, one in INTERACT, and one in GSEQ). On the basis of simulation results presented here, and due to the somewhat different information provided by each, the reporting of both a time-based and an event-based κ?is recommended.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Adamson, L. B., & Bakeman, R. (2006). Development of displaced speech in early mother-child conversations. Child Development, 77, 186–200. doi:10.1111/j.1467-8624.2006.00864.x
Adamson, L. B., Bakeman, R., & Deckner, D. F. (2004). The development of symbol-infused joint engagement. Child Development, 75, 1171–1187. doi:10.1111/j.1467-8624.2004.00732.x
Bakeman, R., Adamson, L. B., Konner, M., & Barr, R. G. (1990). !Kung infancy: The social context of object exploration. Child Development, 61, 794–809. doi:10.2307/1130964
Bakeman, R., & Gottman, J. M. (1997). Observing interaction: An introduction to sequential analysis (2nd ed.). New York: Cambridge University Press.
Bakeman, R., & Quera, V. (1995). Analyzing interaction: Sequential analysis with SDIS and GSEQ. New York: Cambridge University Press.
Bakeman, R., Quera, V., McArthur, D., & Robinson, B. F. (1997). Detecting sequential patterns and determining their reliability with fallible observers. Psychological Methods, 2, 357–370. doi:10.1037/1082-989X.2.4.357
Bakeman, R., & Robinson, B. F. (1994). Understanding log-linear analysis with ILOG: An interactive approach. Hillsdale, NJ: Erlbaum.
Cohen, J. A. (1960). A coeff icient of agreement for nominal scales. Educational & Psychological Measurement, 20, 37–46. doi:10.1177/001316446002000104
Galisson, F. (2000, August). Introduction to computational sequence analysis. Tutorial presented at the Eighth International Conference on Intelligent Systems for Molecular Biology, San Diego, CA.
Haccou, P., & Meelis, E. (1992). Statistical analysis of behavioral data: An approach based on time-structured models. Oxford: Oxford University Press.
Hollenbeck, A. R. (1978). Problems of reliability in observational data. In G. P. Sackett (Ed.), Observing behavior: Vol. 2. Data collection and analysis methods (pp. 79–88). Baltimore: University Park Press.
Jansen, R. G., Wiertz, L. F., Meyer, E. S., & Noldus, L. P. J. J. (2003). Reliability analyis of observational data: Problems, solutions, and software implementation. Behavior Research Methods, Instruments, & Computers, 35, 391–399.
Konner, M. J. (1978). Maternal care, infant behavior, and development among the !Kung. In R. B. Lee & I. DeVore (Eds.), Kalahari huntergatherers: Studies of the !Kung Sari and their neighbors (pp. 218–245). Cambridge, MA: Harvard University Press.
Mangold, P. (2006, September). Getting better results in less time: When using audio/video recordings in research applications makes sense. Paper presented at the Third Congress of the European Society on Family Relations, Darmstadt, Germany.
Mannila, H., & Ronkainen, P. (1997, May). Similarity of event sequences. In Proceedings of the Fourth International Workshop on Temporal Representation and Reasoning (pp. 136–139). Daytona Beach, FL.
Needleman, S. B., & Wunsch, C. D. (1970). A general method applicable to the search for similarities in the amino acid sequence of two proteins. Journal of Molecular Biology, 48, 443–453.
Noldus Information Technology (2003). The Observer: Professional system for collection, analysis, presentation and management of observational data [Reference Manual, Version 5.0]. Wageningen, The Netherlands: Author.
Quera, V., Bakeman, R., & Gnisci, A. (2007). Observer agreement for event sequences: Methods and software for sequence alignment and reliability estimates. Behavior Research Methods, 39, 39–49.
Sankoff, D., & Kruskal, J. (Eds.) (1999). Time warps, string edits, and macromolecules: The theory and practice of sequence comparison (2nd ed.). Stanford, CA: CSLI Publications.
Suen, H. K. (1988). Agreement, reliability, accuracy, and validity: Toward a clarification. Behavioral Assessment, 10, 343–366.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Bakeman, R., Quera, V. & Gnisci, A. Observer agreement for timed-event sequential data: A comparison of time-based and event-based algorithms. Behavior Research Methods 41, 137–147 (2009). https://doi.org/10.3758/BRM.41.1.137
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.3758/BRM.41.1.137