Replication Data for: Automated Quality Assessment for LLM-Based Complex Qualitative Coding: A Confidence-Diversity Frameworkhttps://doi.org/10.7910/DVN/GM8T8Qzhao, zhilongHarvard Dataverse2025-08-262025-08-28T01:38:22ZThis replication package contains all code and data necessary to reproduce the results presented in "Cross-Domain Quality Assessment for Complex Qualitative Analysis: Validating Confidence-Entropy Signals Across Legal, Political, and Medical Tasks". Research Context: This study extends beyond accessible coding tasks to validate automated quality assessment for complex qualitative analysis requiring domain expertise and interpretive judgment across legal, political, and medical domains. Package Contents: - Core Scripts: reproduce_all_results.py (main reproduction script), generate_synthetic_data.py (data generator), validate_reproduction.py (result validation) - Data Files: Synthetic datasets matching paper statistics for SCOTUS legal reasoning (390 cases), Hyperpartisan political analysis (644 cases), and MTSamples medical classification (1,000 cases) - Expected Outputs: All LaTeX tables (Table 1-5), validation reports, and cross-domain statistical analyses Key Findings Reproduced: - Cross-domain signal effectiveness (Table 1): Perfect correlation reproduction across all domains (±0.005 accuracy) - Dual-signal weight optimization (Table 2): 6.6-113.7% improvements over single-signal baselines - Cross-domain transferability (Table 3): 88.9% success rate for weight transfer across domains - Intelligent triage efficiency (Table 5): 45.4% vs 44.6% effort reduction (0.8% difference) - Domain-specific patterns: Confidence signals are stronger in legal contexts, and entropy signals are more reliable in political/medical domains Validation Status: Successfully reproduces all core findings with statistical significance maintained across complex analytical tasks. Demonstrates automated quality assessment viability for scaling complex qualitative research beyond accessible coding tasks. Usage: Run ./run_complete_reproduction.sh for complete reproduction, or python3 reproduce_all_results.py for individual table generation. All dependencies included.Computer and Information ScienceSocial Sciences2025-08-26zhao, zhilong2025-08-26CC0 1.0