Social media as a recruitment and data collection tool: Experimental evidence on the relative effectiveness of web surveys and chatbots

Abstract

Online technologies enable lower-cost, rapid data collection, but concerns about sample composition biases and mode-specific measurement error impede their use. I conduct a randomized experiment in the Philippines to test the effectiveness of web-form and chatbot surveys of K–12 teachers recruited through social media and compare their effectiveness with phone surveys of teachers recruited from a pre-existing frame. Chatbot surveys yield higher response rates and higher-quality data than web-form surveys in terms of missed question and item differentiation. The results suggest that chatbot responses match CATI responses on multiple dimensions of quality. Relative to CATI, online methods also yield higher rates or information disclosure on potentially sensitive topics, revealing substantially higher levels of distress among teachers. I show that social media-based recruitment can be an attractive alternative for targeted sampling and that online surveys can be implemented effectively at a fraction of the cost of phone surveys.