A Mixed Method Program Evaluation of Annual Inspections Conducted in Childcare Programs in Washington State


  • Sonya Stevens Research and Methodology Manager – Licensing Division, Washington Department of Children, Youth, and Families https://orcid.org/0000-0002-6897-4667




Licensing, Child Care, Inter-rater Reliability


This mixed method study used a program evaluation to assess the reliability and social validity of the focused childcare monitoring checklist used in Washington State, as well as its social validity in maintaining quality programming in licensed childcare centers. The focused monitoring checklist and interview responses were used to answer two specific research questions: (1) How do stakeholders describe the value, usefulness, and effects of state administrated focused monitoring?; and (2) What is the inter-rater reliability of the focused monitoring tool used to assess the foundational health and safety issues that must be met by state licensed early childhood programs? The study found that licensors and providers found the focused monitoring tool as more efficient and informative than the current differential monitoring system. The use of a checklist focusing on real time compliance increased the value placed on the relevance of the inspection with respect to meeting licensor and provider needs. The results also showed that even with a controlled tool, performance of onsite inspections can vary greatly along a continuum of reliability and objectivity due to licensor rater drift and individual perceptions of licensing procedures. Licensing agencies should consider further evaluation of the monitoring process and the reliability of the checklist tool as the process is implemented statewide, concentrating on the training content and training methods provided to licensors.



Alkon, A., Rose, R., Wolff, M., Kotch, J. B., & Aronson, S. S. (2015). Health and Safety Checklist for Early Care and Education Programs to Assess Key National Health and Safety Standards. Maternal and Child Health Journal, 20, 114-127. https://www.doi.org/10.1007/s10995-015-1809-8

Bradshaw, C. P., Milam, A. J., Furr-Holden, C. D. M., & Lindstrom-Johnson, S. (2015). The School Assessment for Environmental Typology (SAfETy): An Observational Measure of the School Environment. American Journal of Community Psychology, 56(3-4), 280-292. https://www.doi.org/10.1007/s10464-015-9743-x

Bromer, J., McCabe, L. A., & Porter, T. (2013). Special section on understanding and improving quality in family childcare: Introduction and commentary. Early Childhood Research Quarterly, 28(4), 875-878. https://www.doi.org/10.1016/j.ecresq.2013.08.003

Cassidy, D. J., Hestenes, L. L., Hegde, A., Hestenes, S., & Mims, S. (2005). Measurement of quality in preschool child care classrooms: An exploratory and confirmatory factor analysis of the early childhood environment rating scale-revised. Early Childhood Research Quarterly, 20(3), 345-360. https://www.doi.org/10.1016/j.ecresq.2005.07.005

Crocker, L., & Algina, J. (1986). Introduction to Classical and Modern Test Theory. Holt, Rinehart, and Winston.

Crowley, A. A., Sangchoon, J., & Rosenthal, M. S. (2013). Health and Safety of Child Care Centers: An Analysis of Licensing Specialists' Reports of Routine, Unannounced Inspections. American Journal of Public Health, 103(10), e52-e58. https://www.doi.org/10.2105/AJPH.2013.301298

Fiene, R. (2013). Differential monitoring logic model and algorithm (DMLMA)™: A new early childhood program quality indicator model (ECPQIM4)© for early care and education regulatory agencies. Research Institute for Key Indicators. Retrieved from http://www.researchconnections.org/childcare/resources/24950/pdf

Fiene, R. (2014). Georgia childcare licensing study: Validating the core rule differential monitoring system. Bright From the Start. Retrieved from http://www.researchconnections.org/childcare/resources/30431

Fiene, R. (n.d.). The relationship of accreditation and licensing standards, training, and child development outcomes. Pennsylvania State University Capital Area Early Childhood Training Institute. Retrieved from https://www.researchconnections.org/childcare/resources/9067

Fiene, R., & Carl, B. (2010). 13 Key Indicators of Child Care quality: Child Care Quality Indicators (CCQ1 – CDPES2 PC1) Scale. Retrieved from https://harrisburg.psu.edu/files/pdf/371/2017/02/20/13keyindicatorsofchildcarequalitychildcarequalityindicatorsccqi-cdpes2pc1scale.pdf

Fiene, R., & Nixon, M. (1985). The Instrument Based Program Monitoring Information System and the Indicator Checklist for child care. Childcare Quarterly, 14(3), 198-214. https://www.doi.org/10.1007/BF01113748

Gall, M. D., Gall, J. P., & Borg, W. R. (2007). Educational Research: An Introduction, 8th Edition. Pearson.

Huston, A. C., Bobbitt, K. C., & Bentley, A. (2015). Time spent in childcare: How and why does it affect social development? Developmental Psychology, 51(5), 621-634. https://www.doi.org/10.1037/a0038951

Hudson, G., Miller, G. G., & Seikel, K. (2014). Regulations, Policies, and Guidelines Addressing Environmental Exposures in Early Learning Environments: A Review. Journal of Environmental Health, 76(7), 24-34.

Hyde, B. (2011). Childcare licensing in Washington: A report to the Washington State Legislature. Washington State Department of Early Learning. Retrieved from https://app.leg.wa.gov/ReportsToTheLegislature/Home/GetPDF?fileName=LicensingProvisoReport11511_6bd0983a-3c05-4657-a8ba-5c6476f0d19f.pdf

Kayira, T. (2016). Fundamentals for effective monitoring of child-care institutions. Commonwealth Law Bulletin, 42(1), 105-124. https://www.doi.org/10.1080/03050718.2016.1143380

Kelley, K. S., & Littenberg, B. (2015). Inter-rater reliability for scoring children's dichotic words test responses. International Journal of Audiology, 54(10), 751-757. https://www.doi.org/10.3109/14992027.2015.1052103

Kimberlin, C. L., & Winterstein, A. G. (2008). Validity and Reliability of Measurement Instruments Used in Research. American Journal of Health-System Pharmacy, 65(23), 2276-2284. https://www.doi.org/10.2146/ajhp070364

Lapp-Payne, A. & Testa, M. S. (2014). Protecting Children's Health and Safety in Every Setting. Exchange Press.

Ledford, J. R., Hall, E., Conder, E., & Lane, J. D. (2016). Research for young children with autism spectrum disorders: Evidence of social and ecological validity. Topics in Early Childhood Special Education, 35(4), 223-233. https://www.doi.org/10.1177/0271121415585956

Meloy, M. E., Lipscomb, S. T., & Baron, M. J. (2015). Linking state child care and child welfare policies and populations: Implications for children, families, and policymakers. Children and Youth Services Review, 57(C), 30-39. https://www.doi.org/10.1016/j.childyouth.2015.07.008

Miller, A. (2013). Potential Dangers of Unlicensed Daycares. Canadian Medical Association Journal, 185(18), 1566. https://www.doi.org/10.1503/cmaj.109-4643

Moloney, M. (2016). Childcare regulations: Regulatory enforcement in Ireland. What happens when the inspector calls? Journal of Early Childhood Research, 14(1), 84-97. https://www.doi.org/10.1177/1476718X14536717

National Center on Child Care Quality Improvement. (2013). Research Brief #1: Trends in Child Care Center Licensing Requirements and Policies for 2011. Retrieved from https://childcareta.acf.hhs.gov/sites/default/files/public/065_1304_center_licensing_trends_brief_final.pdf

Rosenthal, M. S., Jeon, S., & Crowley, A. A. (2016). Health and Safety in Family Day Care Homes: Association Between Regulatory Non-compliance and Lower Median Income. Maternal & Child Health Journal, 20(5), 984-992. https://www.doi.org/10.1007/s10995-015-1883-y

Strain, P. S., Barton, E. E., & Dunlap, G. (2012). Lessons learned about the utility of social validity. Education & Treatment of Children, 35(2), 183-200. https://www.doi.org/10.1353/etc.2012.0007

Stufflebeam, D. L., & Shinkfield, A. J. (2007). Evaluation Theory, Models, and Applications. Jossey-Bass, an imprint of Wiley.

Washington State Department of Children, Youth & Families (formerly Department of Early Learning). (n.d.). Become a Licensed Child Care & Early Learning Provider. Retrieved from https://www.dcyf.wa.gov/services/early-learning-providers/licensed-provider (formerly http://www.del.wa.gov/care/about/Default.aspx)

Wolf, M. M. (1978). Social validity: The case for subjective measurement or how applied behavior analysis is finding its heart. Journal of Applied Behavior Analysis, 11(2), 203-214. https://www.doi.org/10.1901/jaba.1978.11-203






Scientific Articles

How to Cite

A Mixed Method Program Evaluation of Annual Inspections Conducted in Childcare Programs in Washington State. (2020). Journal of Regulatory Science, 8, 1-11. https://doi.org/10.21423/JRS-V08STEVENS