Prompt engineering
Financial-domain priors and a structured relevance scaffold guide ranking. P₄ standardizes output discipline; longer prompts are not inherently better.
EMNLP 2026 · Industry Track · Accepted
PRISMPrompt-Refined In-Context System Modeling
for Financial Retrieval
1 AI Lens, Kuala Lumpur, Malaysia
2 Hong Kong Baptist University, Hong Kong
* Equal contribution
PRISM studies how to retrieve financial evidence without fine-tuning, composing prompt engineering, selective in-context learning, and candidate refinement. Its empirical focus is when each component earns its accuracy, cost, and latency overhead.
01 / Research question
Abstract
Deploying LLMs for financial information retrieval demands accuracy, cost efficiency, and reproducibility, yet most approaches require expensive fine-tuning. PRISM is a training-free framework that composes three deployable modules (prompt engineering, in-context learning, and a candidate refinement module) with a boundary-condition study of multi-agent coordination for financial document retrieval. On FiQA-2018, stacked reranking and k-reciprocal encoding raise Recall@100 from 0.7783 to 0.8326. On FinAgentBench, a Cohere-refined top-50 pool yields a higher observed end-to-end NDCG@5, below this evaluation's minimum detectable effect, while cutting per-query cost by about 35%. Added complexity does not always pay off: k-reciprocal encoding stops helping once the upstream stage is already high-recall, and multi-agent coordination is not a default for fine-grained chunk ranking. Matched pairs that hold prompt, model, and exemplars fixed lose 0.0702 and 0.0681 combined NDCG@5 to their single-path controls, and the whole gap sits at the chunk stage. We release an expert-annotated FinAgentBench evaluation set, together with a label-robustness sweep over 16 alternative ground-truth constructions, restoring a reproducible evaluation path after the original evaluator went offline. Without fine-tuning, PRISM placed third on the official FinAgentBench private leaderboard (leaderboard basis, NDCG@5 0.71181), reaches NDCG@10 0.6197 on FiQA-2018, and 98% accuracy on FinanceBench. We report per-query latency, token, and dollar accounting across roughly 40 configurations, present the quality–cost–latency Pareto frontier, and introduce FACET, a transparent SLA-specific scalarization for choosing among non-dominated configurations.
02 / Method
Adapt the context. Refine the candidates.
Selective in-context learning
Retrieve demonstrations from a separate training-split FAISS store. These examples add context without entering the candidate pool; document and chunk tasks need different choices.
Candidate refinement
Retrieve, rerank, and optionally apply k-reciprocal encoding before LLM ranking. Unlike exemplar retrieval, this path removes candidates and compresses context.
When coordination does not pay off
Multi-agent coordination is an optional replacement for the ranking stage. In two matched A₄ comparisons holding the prompt, model and exemplars fixed, the single-path controls lead by 0.0702 and 0.0681 combined NDCG@5, with the loss at the chunk stage. This finding is scoped to the tested workflow, not every multi-agent architecture.
03 / Evidence
Main results
| Evaluation | Metric | Paper result | Scope |
|---|---|---|---|
| FinAgentBench · official private leaderboard | Combined NDCG@5 | 0.71181 | Third place; historical leaderboard basis |
| FinAgentBench · re-annotated | Combined NDCG@5 | 0.83721 | Run 39, grok-4-20-reasoning; 200 document / 199 chunk scoring queries |
| FiQA-2018 · LLM ranking | NDCG@10 | 0.6197 | P₃ / GPT-5 / ICL-5 |
| FiQA-2018 · candidate refinement | Recall@100 | 0.8326 | TE3-S → Cohere → k-reciprocal; baseline 0.7783 |
| FinanceBench · Oracle QA | Accuracy | 98% (147/150) | P₁ / GPT-5; relevant passages provided, not retrieved |
FiQA: refine a shared candidate pool
| Pipeline | Recall@100 | NDCG@10 |
|---|---|---|
| TE3-S | 0.77830 | 0.54988 |
| TE3-S → Cohere | 0.81150 | 0.62580 |
| TE3-S → Cohere → GPT-5 | 0.83066 | 0.62426 |
| TE3-S → Cohere → k-reciprocal | 0.83260 | 0.63333 |
The refinement stack and the 0.6197 LLM-ranking headline are distinct configurations. The stack reaches 0.63333 NDCG@10 without an LLM ranking stage.
FinAgentBench: lower cost, longer latency
| Chunk pipeline | NDCG@5 | USD/query | Latency |
|---|---|---|---|
| No refinement | 0.61278 | $0.278 | 136.7 s |
| TE3-S → Cohere, top 50 | 0.63408 | $0.181 | 146.3 s |
FinAgentBench chunk ranking, re-annotated basis. Refinement reduces reported cost by about 35%, but increases latency. The observed +0.0213 NDCG@5 is below the evaluation’s 0.057 minimum detectable effect; this single-run comparison does not establish a significant quality improvement. These chunk-pipeline costs use a different denominator from the combined document–chunk accounting in the Pareto figure.
Choose an operating point, not a universal winner
Six of eleven measured configurations are non-dominated across the three axes. FACET summarizes fit to a stated service-level agreement, with raw quality, cost and latency alongside.
The confidence intervals of the grok-4 and DeepSeek quality anchors overlap. The paper does not establish a strict quality ordering between them.
A boundary case for refinement
On FinAgentBench’s smaller, already refined pools, adding k-reciprocal encoding after Cohere lowers reranker-only NDCG@5 from 0.5464 to 0.2829. The decision depends on remaining retrieval headroom, not simply on adding more stages.
04 / Open research
Resources
The research implementation is maintained by AI Lens under AGPL-3.0. This companion repository contains the project overview, figures, citation and website.
05 / Citation
BibTeX
@inproceedings{ng-etal-2026-prism,
title = {{PRISM}: Prompt-Refined In-Context System Modeling for Financial Retrieval},
author = {Ng, Chun Chet and Lim, Jia Yu and Chu, Zhen Hao and Zhou, Yixi and Low, Wei Zeng},
booktitle = {Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Industry Track},
year = {2026},
address = {Budapest, Hungary},
url = {https://arxiv.org/abs/2511.14130}
}