A Mixed Method Program Evaluation of Annual Inspections Conducted in Childcare Programs in Washington State
DOI:
https://doi.org/10.21423/JRS-V08STEVENSKeywords:
Licensing, Child Care, Inter-rater ReliabilityAbstract
This mixed method study used a program evaluation to assess the reliability and social validity of the focused childcare monitoring checklist used in Washington State, as well as its social validity in maintaining quality programming in licensed childcare centers. The focused monitoring checklist and interview responses were used to answer two specific research questions: (1) How do stakeholders describe the value, usefulness, and effects of state administrated focused monitoring?; and (2) What is the inter-rater reliability of the focused monitoring tool used to assess the foundational health and safety issues that must be met by state licensed early childhood programs? The study found that licensors and providers found the focused monitoring tool as more efficient and informative than the current differential monitoring system. The use of a checklist focusing on real time compliance increased the value placed on the relevance of the inspection with respect to meeting licensor and provider needs. The results also showed that even with a controlled tool, performance of onsite inspections can vary greatly along a continuum of reliability and objectivity due to licensor rater drift and individual perceptions of licensing procedures. Licensing agencies should consider further evaluation of the monitoring process and the reliability of the checklist tool as the process is implemented statewide, concentrating on the training content and training methods provided to licensors.
References
Alkon, A., Rose, R., Wolff, M., Kotch, J. B., & Aronson, S. S. (2015). Health and Safety Checklist for Early Care and Education Programs to Assess Key National Health and Safety Standards. Maternal and Child Health Journal, 20, 114-127. https://www.doi.org/10.1007/s10995-015-1809-8
Bradshaw, C. P., Milam, A. J., Furr-Holden, C. D. M., & Lindstrom-Johnson, S. (2015). The School Assessment for Environmental Typology (SAfETy): An Observational Measure of the School Environment. American Journal of Community Psychology, 56(3-4), 280-292. https://www.doi.org/10.1007/s10464-015-9743-x
Bromer, J., McCabe, L. A., & Porter, T. (2013). Special section on understanding and improving quality in family childcare: Introduction and commentary. Early Childhood Research Quarterly, 28(4), 875-878. https://www.doi.org/10.1016/j.ecresq.2013.08.003
Cassidy, D. J., Hestenes, L. L., Hegde, A., Hestenes, S., & Mims, S. (2005). Measurement of quality in preschool child care classrooms: An exploratory and confirmatory factor analysis of the early childhood environment rating scale-revised. Early Childhood Research Quarterly, 20(3), 345-360. https://www.doi.org/10.1016/j.ecresq.2005.07.005
Crocker, L., & Algina, J. (1986). Introduction to Classical and Modern Test Theory. Holt, Rinehart, and Winston.
Crowley, A. A., Sangchoon, J., & Rosenthal, M. S. (2013). Health and Safety of Child Care Centers: An Analysis of Licensing Specialists' Reports of Routine, Unannounced Inspections. American Journal of Public Health, 103(10), e52-e58. https://www.doi.org/10.2105/AJPH.2013.301298
Fiene, R. (2013). Differential monitoring logic model and algorithm (DMLMA)™: A new early childhood program quality indicator model (ECPQIM4)© for early care and education regulatory agencies. Research Institute for Key Indicators. Retrieved from http://www.researchconnections.org/childcare/resources/24950/pdf
Fiene, R. (2014). Georgia childcare licensing study: Validating the core rule differential monitoring system. Bright From the Start. Retrieved from http://www.researchconnections.org/childcare/resources/30431
Fiene, R. (n.d.). The relationship of accreditation and licensing standards, training, and child development outcomes. Pennsylvania State University Capital Area Early Childhood Training Institute. Retrieved from https://www.researchconnections.org/childcare/resources/9067
Fiene, R., & Carl, B. (2010). 13 Key Indicators of Child Care quality: Child Care Quality Indicators (CCQ1 – CDPES2 PC1) Scale. Retrieved from https://harrisburg.psu.edu/files/pdf/371/2017/02/20/13keyindicatorsofchildcarequalitychildcarequalityindicatorsccqi-cdpes2pc1scale.pdf
Fiene, R., & Nixon, M. (1985). The Instrument Based Program Monitoring Information System and the Indicator Checklist for child care. Childcare Quarterly, 14(3), 198-214. https://www.doi.org/10.1007/BF01113748
Gall, M. D., Gall, J. P., & Borg, W. R. (2007). Educational Research: An Introduction, 8th Edition. Pearson.
Huston, A. C., Bobbitt, K. C., & Bentley, A. (2015). Time spent in childcare: How and why does it affect social development? Developmental Psychology, 51(5), 621-634. https://www.doi.org/10.1037/a0038951
Hudson, G., Miller, G. G., & Seikel, K. (2014). Regulations, Policies, and Guidelines Addressing Environmental Exposures in Early Learning Environments: A Review. Journal of Environmental Health, 76(7), 24-34.
Hyde, B. (2011). Childcare licensing in Washington: A report to the Washington State Legislature. Washington State Department of Early Learning. Retrieved from https://app.leg.wa.gov/ReportsToTheLegislature/Home/GetPDF?fileName=LicensingProvisoReport11511_6bd0983a-3c05-4657-a8ba-5c6476f0d19f.pdf
Kayira, T. (2016). Fundamentals for effective monitoring of child-care institutions. Commonwealth Law Bulletin, 42(1), 105-124. https://www.doi.org/10.1080/03050718.2016.1143380
Kelley, K. S., & Littenberg, B. (2015). Inter-rater reliability for scoring children's dichotic words test responses. International Journal of Audiology, 54(10), 751-757. https://www.doi.org/10.3109/14992027.2015.1052103
Kimberlin, C. L., & Winterstein, A. G. (2008). Validity and Reliability of Measurement Instruments Used in Research. American Journal of Health-System Pharmacy, 65(23), 2276-2284. https://www.doi.org/10.2146/ajhp070364
Lapp-Payne, A. & Testa, M. S. (2014). Protecting Children's Health and Safety in Every Setting. Exchange Press.
Ledford, J. R., Hall, E., Conder, E., & Lane, J. D. (2016). Research for young children with autism spectrum disorders: Evidence of social and ecological validity. Topics in Early Childhood Special Education, 35(4), 223-233. https://www.doi.org/10.1177/0271121415585956
Meloy, M. E., Lipscomb, S. T., & Baron, M. J. (2015). Linking state child care and child welfare policies and populations: Implications for children, families, and policymakers. Children and Youth Services Review, 57(C), 30-39. https://www.doi.org/10.1016/j.childyouth.2015.07.008
Miller, A. (2013). Potential Dangers of Unlicensed Daycares. Canadian Medical Association Journal, 185(18), 1566. https://www.doi.org/10.1503/cmaj.109-4643
Moloney, M. (2016). Childcare regulations: Regulatory enforcement in Ireland. What happens when the inspector calls? Journal of Early Childhood Research, 14(1), 84-97. https://www.doi.org/10.1177/1476718X14536717
National Center on Child Care Quality Improvement. (2013). Research Brief #1: Trends in Child Care Center Licensing Requirements and Policies for 2011. Retrieved from https://childcareta.acf.hhs.gov/sites/default/files/public/065_1304_center_licensing_trends_brief_final.pdf
Rosenthal, M. S., Jeon, S., & Crowley, A. A. (2016). Health and Safety in Family Day Care Homes: Association Between Regulatory Non-compliance and Lower Median Income. Maternal & Child Health Journal, 20(5), 984-992. https://www.doi.org/10.1007/s10995-015-1883-y
Strain, P. S., Barton, E. E., & Dunlap, G. (2012). Lessons learned about the utility of social validity. Education & Treatment of Children, 35(2), 183-200. https://www.doi.org/10.1353/etc.2012.0007
Stufflebeam, D. L., & Shinkfield, A. J. (2007). Evaluation Theory, Models, and Applications. Jossey-Bass, an imprint of Wiley.
Washington State Department of Children, Youth & Families (formerly Department of Early Learning). (n.d.). Become a Licensed Child Care & Early Learning Provider. Retrieved from https://www.dcyf.wa.gov/services/early-learning-providers/licensed-provider (formerly http://www.del.wa.gov/care/about/Default.aspx)
Wolf, M. M. (1978). Social validity: The case for subjective measurement or how applied behavior analysis is finding its heart. Journal of Applied Behavior Analysis, 11(2), 203-214. https://www.doi.org/10.1901/jaba.1978.11-203
Downloads
Published
Issue
Section
License
Copyright (c) 2020 Sonya Stevens
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
By submitting content to the Journal of Regulatory Science (JRS), authors agree to the following terms:
- Authors retain copyright and grant the JRS the right of first publication. Authors retain patent, trademark, and other intellectual property rights (including research data) and grant third parties the right to use, reproduce, and share the article according to the Creative Commons — Attribution-NonCommercial 4.0 International — CC BY-NC 4.0 license agreement. The JRS is an open access journal and, as a result, articles are free to use with proper acknowledgment of the work's authorship and initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process.
- If excerpts from other copyrighted works are included, the author(s) must obtain written permission from the copyright owners and credit the source(s) in the article.
- The publication of the submission has been approved by all co-authors and responsible authorities at the institute or organization where the work has been carried out.
- Copyright has not been breached in seeking publication of the submission.
