# PIXIU: open financial LLMs, instruction data and an evaluation benchmark > PIXIU is a research release that bundles three things for the financial domain: FinBen, an evaluation benchmark for financial understanding and prediction, FIT, a multi-task and multi-modal instruction dataset, and FinMA, a financial LLM fine-tuned on FIT. Everything is published on Hugging Face with a leaderboard, and the papers appeared at NeurIPS 2023 and IJCAI. **The-FinAI/PIXIU** — This repository introduces PIXIU, an open-source resource featuring the first financial large language models (LLMs), instruction tuning data, and evaluation benchmarks to holistically assess financial LLMs. Our goal is to continually push forward the open-source development of financial artificial intelligence (AI). - Repository: https://github.com/The-FinAI/PIXIU - Website: https://huggingface.co/collections/TheFinAI/finben-and-pixiu-english-financial-evaluation-658f515911f68f12ea193194 - Stars: 895 · Forks: 126 - Language: Jupyter Notebook - License: MIT - Published: 2026-10-20 · Updated: 2026-10-20 · Language: en - Canonical page: https://hysenlabs.com/projects/the-finai-pixiu ## Three artefacts released together, not one model PIXIU is described as an open-source resource featuring the first financial large language models, instruction tuning data, and evaluation benchmarks to holistically assess financial LLMs, with the stated goal of continually pushing forward the open-source development of financial artificial intelligence. The structure of the repository is the important part, and the readme spells out three components that each serve a distinct purpose in a financial NLP pipeline. FinBen is the Financial Language Understanding and Prediction Evaluation Benchmark, and it serves as the evaluation suite for financial LLMs with a focus on understanding and prediction tasks across various financial contexts. FIT is the Financial Instruction Dataset, a multi-task and multi-modal instruction dataset tailored for financial tasks, described as the training ground for fine-tuning LLMs. FinMA is the Financial Large Language Model, the core of the project, providing the learning and prediction power for the financial tasks. So this is a complete research pipeline rather than a model release: data to train on, a model trained on it, and a benchmark to evaluate against. The readme states that all models, instruction data and evaluation datasets are openly released on Hugging Face, and the key features section repeats that open resources are provided. The primary language of the repository is Jupyter Notebook, which is consistent with a project whose artefacts are datasets, model checkpoints and evaluation scripts rather than a Python package with a clean public API. ## Two papers, a NeurIPS track and an IJCAI challenge The scholarly framing is unusually visible for a repository, and it tells you what kind of work this is. There is an arXiv link for PIXIU at 2306.05443 and one for FinBen at 2402.12659, and a badge marking the PIXIU paper as a NeurIPS 2023 Datasets and Benchmarks track entry. The repository also carries a README.ijcai_challenge.md, which points to an IJCAI challenge built around this work. The two papers are separate things. PIXIU is the earlier one and covers the model and the instruction dataset; FinBen is the later one and covers the benchmark. That ordering matches the way the components are described: FinBen as the evaluation suite, FIT as the training ground, FinMA as the model produced from it. For a reader evaluating whether to use this, the split matters in a practical way. The model and its training data are the things you fine-tune on or adapt. The benchmark is the thing you measure against, and it is published separately with its own paper, which means you can use the benchmark without adopting the model at all. The affiliation list is long, with thirteen numbered institutions including The Fin AI, Wuhan University, the University of Manchester, the University of Florida, Columbia University, the Chinese University of Hong Kong Shenzhen, Sichuan University, Yunnan University, Stevens Institute of Technology, Stony Brook University, Nanjing Audit University, Jiangxi Normal University and Southwest Jiaotong University, and the corresponding authors are marked with asterisks. That is the profile of a multi-institution academic collaboration rather than a company project, and it is the reason the readme is structured around papers and data releases rather than around installation. ## A leaderboard that turns the benchmark into a comparison The readme links a Hugging Face Space labelled as a leaderboard, TheFinAI/Open-FinLLM-Leaderboard, alongside a Hugging Face collection for the data and a model link for finma-7b-full. A leaderboard is what turns a benchmark from a dataset into a measurement instrument. A benchmark you can only run yourself tells you how your model compares to what you remember of your last model. A leaderboard with several models on it, evaluated under the same protocol, tells you something you could not otherwise know. The practical question for an evaluator is what the leaderboard measures. The topics attached to the repository give a good indication of the coverage: question answering, sentiment analysis, text classification, named entity recognition and stock price prediction sit alongside the general categories of chatgpt, gpt-4, llama, large language models, machine learning, natural language processing and financial technology. Those task names are the useful part. A financial benchmark that only measures chat style tells you very little, whereas one that separates sentiment analysis from named entity recognition from stock prediction tells you which capability is missing. It is worth being careful about one thing. The readme does not describe the evaluation protocol, the prompting setup, or the model list in the visible documentation, and those details determine whether a leaderboard position means anything. Benchmark results in this field are sensitive to the prompt, and the arXiv papers are the place to look for how the numbers were obtained. ## What is in the tree, and the unusual entries The top level entries give a good picture of the working parts. There is src/ for the library code, notebooks/ for the analysis work, scripts/ for the pipeline scripts, and docker/ for the container setup. The .gitmodules file and .cache/ directory indicate that some components, most likely the spaCy model used for evaluation, are pulled in as submodules rather than vendored. Two entries stand out. FinMem-LLM-StockTrading is a separate subproject in the tree, suggesting an extension of the work into memory for stock trading scenarios. And a static/ directory holds assets for a web presentation or demo. The requirements file shows the stack, and it is a research stack rather than a deployment one. transformers and peft for the model work, sentence_transformers for embeddings, vllm pinned at 0.2.7 for inference, gradio for the interactive components, rank_bm25 for a lexical baseline, and evaluate and seqeval for scoring. scikit-learn, pandas and numpy for the data work, spacy for language processing, and a specific spaCy English model tarball listed directly as a path. Two of those deserve a comment. The spaCy model being pinned as a tarball in requirements.txt rather than a pip install of the package is a deliberate choice to make the evaluation reproducible, since spaCy model versions change and change the scores. And vllm pinned at 0.2.7 alongside a much newer transformers is the kind of combination that tells you this stack was assembled for a specific set of experiments and has not been re-pinned since. The openai dependency in the list is there for API calls, and the fact that it sits next to a local vllm suggests the evaluation compares closed and open models on the same tasks. ## Where PIXIU is the wrong tool Three cases, and the first one is about what these artefacts are for. PIXIU is not a trading system and not a financial assistant. FinMA is a seven billion parameter model fine-tuned on FIT, published as a research artefact, and the repository's own framing is an evaluation benchmark and a dataset. If you need to analyse a portfolio, or answer a question about a specific instrument with current data, none of that is here, and a fine-tuned model trained on instruction data is the wrong shape for live market data regardless of how good it is. The second is the reproducibility problem, which is specific and worth naming. The dependency set pins vllm at 0.2.7 and a spaCy model tarball, and the repository has no GitHub releases, so there is no tag to point at and no published version of the environment. The papers document the setups used, but reproducing a leaderboard number from this repository alone is a research task rather than a build step. The third is the general problem with financial benchmarks. A benchmark measures what it was designed to measure, and a benchmark designed for financial language understanding is not a benchmark for investment skill. Sentiment analysis accuracy on financial text does not tell you whether a model can value a bond. If your goal is to know how a model performs on a trading decision, you need an evaluation designed for that, and you should read the FinBen paper's task descriptions rather than inferring coverage from the topic list. The alternatives differ in what they assume. General financial datasets on Hugging Face give you training data without the benchmark, and general LLM leaderboards give you comparison without domain-specific tasks. The value of PIXIU specifically is that the data, the model and the measurement were built together, so the benchmark measures something the model was actually trained for. ## Conclusion PIXIU suits a researcher building or evaluating a financial language model who needs both training data and a way to measure the result, because the instruction set and the benchmark are released together rather than as separate artefacts you have to reconcile. It does not suit a team looking for a turnkey financial assistant, since these are research models and datasets rather than a product, and the benchmark measures general financial language ability rather than trading performance. Before using the numbers, read the papers and check the leaderboard for which models were evaluated under which prompts, because benchmark results in this area move with the prompting setup. ## FAQ ### What is the PIXIU project? It is an open-source resource for the financial domain that brings together FinBen, an evaluation benchmark for financial understanding and prediction, FIT, a multi-task and multi-modal financial instruction dataset, and FinMA, a financial LLM fine-tuned on FIT. All models, instruction data and evaluation datasets are openly released on Hugging Face. ### What are FinBen, FIT and FinMA? FinBen is the Financial Language Understanding and Prediction Evaluation Benchmark and serves as the evaluation suite for financial LLMs. FIT is the Financial Instruction Dataset, a multi-task and multi-modal instruction dataset for financial tasks and the training ground for fine-tuning. FinMA is the Financial Large Language Model, the model produced from that data. ### Where are the PIXIU models and datasets published? On Hugging Face, with a collection linked for the data and models, a link to finma-7b-full for the model, and a Hugging Face Space leaderboard called Open-FinLLM-Leaderboard for the comparison results. ### Which papers describe PIXIU and FinBen? PIXIU is described in arXiv 2306.05443 and was a NeurIPS 2023 Datasets and Benchmarks track entry, while FinBen is described in arXiv 2402.12659. The repository also carries a README for an IJCAI challenge built on the work. ### What tools does the PIXIU repository depend on? The requirements file lists transformers, peft, gradio, tqdm, scikit-learn, sentencepiece, pandas, seqeval, numpy, evaluate, openai, rank_bm25, spacy, sentence_transformers, a pinned spaCy English model tarball and vllm pinned at 0.2.7. The primary language of the repository is Jupyter Notebook and there are no GitHub releases, so the environment is pinned through requirements.txt rather than through tags. ## Sources - [Issues](https://github.com/The-FinAI/PIXIU/issues) - [License: MIT](https://github.com/The-FinAI/PIXIU/blob/main/LICENSE) - [Project website](https://huggingface.co/collections/TheFinAI/finben-and-pixiu-english-financial-evaluation-658f515911f68f12ea193194) - [README](https://github.com/The-FinAI/PIXIU/blob/main/README.md) - [The-FinAI/PIXIU on GitHub](https://github.com/The-FinAI/PIXIU) --- Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/the-finai-pixiu