LongDS-Bench — why agents fail at multi-turn data analysis
arXiv paper introduces LongDS, a benchmark for long-horizon agentic data analysis. Existing eval suites test isolated tasks; LongDS requires agents to maintain, update, and compose analytical state across 2,225 turns.
• 68 tasks from real Kaggle notebooks across Geoscience, Business, Education
• State-evolution patterns: counterfactual perturbation, rollback, multi-state composition
• Average dependency span of 11.3 turns per task
• Five SOTA models evaluated; all show degradation on long-horizon sequences