Conditional hypothesis
Missed errors and invalid calls in evaluation counts → changes → Measured failure rate
If an earlier evaluation omitted errors inside HTTP 200 responses or discarded invalid calls, fixes that actually bring those events into the counts may raise the measured failure rate. This conditional measurement effect is not evidence that execution became worse, nor has it been experimentally established here.
Read the evidence
If the old evaluation counted only status codes, or only call events that were ultimately retained, while missing errors within responses or invalid calls that were discarded, then the relevant fixes may increase the measured failure rate. The prerequisite is that the fixes actually bring these events into the statistics.
Background sources for the article's reasoning: [S6] CrewAI [S9] LangChain
Read the full research