The Bar We Set
On July 10, 2026, a plan was approved with an acceptance test that seemed ambitious, perhaps even unreachable. The goal was to evaluate an AI system's capability to generate analyst memos based on Nigerian economic data. The benchmark involved six drafted queries, each scored on a scale of 0-10 across five critical dimensions: named-entity density, citation quality, sector-specific detail, honest-gap acknowledgement, and decision-usefulness. The strict pass criterion was unforgiving: every query's mean score across three separate runs, each with a temperature setting of 0.2, had to be 8 or higher. Furthermore, no single query could score below a 6 in any individual run.
At the time of approval, the system's aggregate performance hovered around 30 out of a possible 60 across all six queries. This indicated a system capable of producing grounded, albeit generic, answers. It could refuse requests competently but not always effectively. The gap between its current performance and the stringent acceptance bar was significant and undeniable. The team allocated four to five weeks to bridge this gap.
What We Shipped: Phase 1
The initial phase, termed 'retrieval breadth,' focused on diversifying the top-K result set. The objective was to prevent queries from collapsing into a narrow set of sources. For instance, a query about starting a fintech company should not solely return 12 circulars from the Central Bank of Nigeria (CBN). Instead, the system was engineered to pull relevant information from a wider array of institutions and entities, including the Bank of Industry (BOI), Nigerian Export-Import Bank (NEXIM), and prominent fintech players like PayStack and Flutterwave, alongside international organizations like the World Bank. This phase aimed to broaden the informational landscape the AI could draw from.
The Unmet Target
Despite the efforts in Phase 1 to enhance retrieval breadth, the core performance metrics remained stubbornly below the target. The acceptance test, with its requirement for a mean score of 8 across all queries and runs, was not met. The system continued to struggle with consistently delivering the depth and specificity demanded by the acceptance criteria. While the answers were grounded and the refusals competent, they lacked the nuanced detail and actionable insights required for decision-making. The system produced what could be described as a competent starting point, but not the finished, high-value output anticipated.
Why Fine-Tuning Wasn't the Silver Bullet
A surprising element of this evaluation was the limited impact of fine-tuning. Typically, fine-tuning a large language model can significantly boost performance on specific tasks. However, in this case, the rigorous testing revealed that the fundamental issues were not solely related to the model's parametric knowledge or its ability to generate fluent text. The challenges lay deeper within the retrieval and synthesis processes. Even with fine-tuning, the system failed to consistently hit the critical thresholds for named-entity density, citation quality, and decision-usefulness. This suggests that the bottleneck wasn't just in *what* the model knew or *how* it phrased things, but in its ability to accurately and comprehensively extract, verify, and synthesize the most relevant information from the retrieved documents.
The Broader Implications
This outcome raises pointed questions about the efficacy of standard fine-tuning approaches when faced with complex, multi-dimensional evaluation criteria. It highlights that achieving high performance in specialized domains, particularly those requiring deep analytical rigor and precise data synthesis, demands more than just parameter adjustments. Architectures that heavily rely on retrieval-augmented generation (RAG) may require more sophisticated re-ranking mechanisms, better grounding techniques, or entirely novel approaches to information synthesis. The gap between a system that can produce a passable answer and one that can deliver a truly decision-useful, high-quality analytical output remains a significant challenge in the field. The experience underscores that the 'last mile' in AI performance is often the hardest, and it requires a deeper understanding of the entire pipeline, not just the generative component.