The Quest for Agent Data
Comparison articles about AI agent frameworks often present opinions rather than hard data. To rectify this, a developer built the same agent three times — using Strands, LangGraph, and CrewAI — and logged every LLM call. This rigorous approach, involving 27 executions and a custom local proxy to standardize log formats, provides a unique, data-driven look at how these frameworks actually perform and differ.
The core objective was to move beyond subjective reviews and quantify the trade-offs inherent in different agent orchestration approaches. By routing all LLM interactions through a single recording proxy, the developer ensured that the resulting logs were directly comparable, regardless of the framework’s native logging capabilities. This method revealed significant differences in log formats, trace structures, and overall output consistency across the three platforms.
LangGraph's Determinism Comes at a Price
The headline finding from this empirical study is that LangGraph’s explicit verify/revise loop dramatically improves output consistency. This structured approach reduced output variance by an impressive 77%. Specifically, the word-count spread across executions narrowed from 13 down to just 3. This level of determinism is highly desirable for applications requiring predictable and reliable agent behavior.
However, this enhanced control is not without its costs. The data shows that LangGraph’s explicit loop resulted in approximately 2.5 times more tokens being consumed and a corresponding 2.5x increase in execution latency compared to the other frameworks. This highlights a fundamental trade-off: increased control and predictability come at the expense of computational resources and response time. For developers prioritizing reliability above all else, LangGraph presents a compelling, albeit resource-intensive, option.
Strands and CrewAI: Speed vs. Variance
In contrast to LangGraph, both Strands and CrewAI exhibited higher output variance but offered a more token-efficient and faster execution. While the specific numbers for Strands and CrewAI were not detailed in the excerpt beyond the initial variance spread, the implication is that they represent a different point on the control-vs-cost spectrum. These frameworks likely employ less rigid, more emergent strategies for achieving agent goals, which can lead to a wider range of outputs for the same prompt or task.
This difference in approach is crucial for developers to understand. If an application can tolerate a degree of variability in agent responses and prioritizes speed and cost-effectiveness, Strands or CrewAI might be the more suitable choice. For instance, creative generation tasks or less critical decision-making processes might benefit from the agility of these frameworks. The choice then becomes a direct function of the application’s tolerance for output deviation versus its sensitivity to latency and token expenditure.
The Data Collection Methodology
The ingenuity of this comparison lies in its methodological rigor. The developer implemented a custom, one-file proxy that sat in front of all LLM calls, regardless of which framework was being used. This proxy captured and standardized the logs from each interaction. This meticulous data collection strategy is what enables a direct, apples-to-apples comparison of the underlying LLM usage and output patterns of Strands, LangGraph, and CrewAI.
Without such a standardized logging mechanism, comparing these frameworks would be nearly impossible due to their disparate internal tracing and logging formats. The proxy acts as a universal translator, converting the unique outputs of each framework into a common language that can then be analyzed for variance, token count, and latency. This technique is a powerful example of how developers can overcome framework-specific idiosyncrasies to gather objective performance data.
Unanswered Questions and Future Implications
While this study provides invaluable empirical data, it naturally raises further questions. What is the precise breakdown of token usage and latency for Strands and CrewAI individually? How do these frameworks perform with different LLM models, or with more complex agentic tasks beyond the scope of this experiment? The current analysis provides a strong signal, but a more comprehensive benchmark would involve varying these parameters.
Furthermore, the developer's choice to use an explicit verify/revise loop in LangGraph is a significant architectural decision. It begs the question: can LangGraph achieve similar levels of determinism with more optimized prompting strategies or by selectively applying these loops only where strictly necessary, thereby mitigating some of the token and latency overhead? The future of agent development may lie in hybrid approaches that leverage the strengths of each framework, selectively applying rigorous control only when required.
Conclusion: Data-Driven Framework Selection
This experiment underscores the critical need for empirical data in evaluating AI agent frameworks. The findings clearly illustrate that choosing between Strands, LangGraph, and CrewAI involves a direct trade-off between output consistency and resource consumption. LangGraph offers superior determinism at a higher cost, while Strands and CrewAI provide faster, more economical, but less predictable results.
Developers must weigh these factors against their specific application requirements. The ability to systematically record and analyze LLM calls, as demonstrated by the proxy method, offers a path to making informed decisions. As the agent landscape continues to evolve, such data-driven comparisons will become increasingly vital for selecting the right tools for the job.