## Introduction
This repository contains the replication package for the paper "No Resource, No Benchmarks, No Problem? Evaluating and Improving LLMs for Code Generation in No Resource Languages". In this study, we investigate the code generation performance of modern Large Language Models on popular (Python, Java), low-resource (R, Lua, Haskell, Julia, Racket), and no-resource programming languages (Gleam, MoonBit).
We introduce a more challenging benchmark for code generation, McEval Hard, available in 9 programming languages of different popularity (Python, Java, R, Lua, Haskell, Julia, Racket, Gleam, and MoonBit). This benchmark is based on 'difficult' tasks from the original [McEval benchmark](https://mceval.github.io/).
Finally, we show that the **fine-tuning reuse** [[1, 2]](#references) technique can outperform other strategies (zero-shot, few-shot, RAG, pre-training, and fine-tuning) for improving code generation performance in no-resource programming languages.
The repository includes code and instructions to run the benchmarks, datasets and materials used in the paper, and the scripts to replicate the results.
## Benchmark Usage
In this section, we provide instructions on how to use McEval Hard benchmark and the translated versions of HumanEval and MBPP for no-resource languages.
### Prerequisites
Before running the following scripts, make sure to install the requirements below:
* Python (tested with version: 3.10.14)
* Docker (tested with version: 24.0.7)
Ensure that Docker is running and you have the necessary permissions to run Docker commands. We use Docker to evaluate LLMs completion in an isolated environment with the required dependencies.
Then, proceed with the installation of Python packages as follows:
```sh
# Create and activate a virtual environment
python3 -m venv .venv
source .venv/bin/activate
# Install the dependencies
pip3 install -r requirements.txt
```
### Usage
#### McEval Hard
You can find the scripts to run McEval Hard under the `evaluation/mceval_hard` directory.
This benchmark includes 227 challenging code tasks translated from the original McEval benchmark to 9 programming languages.
We rely on the [LiteLLM](https://www.litellm.ai/) library to run the benchmark. For more information about the available models and how to run them, please refer to the [LiteLLM documentation](https://docs.litellm.ai/docs/providers).
**Arguments:**
- `--model`: the name of the model to evaluate. The name should match the LiteLLM format (e.g., `hosted_vllm/` for models hosted with vLLM).
- `--language`: one of: Python, Java, R, Lua, Haskell, Julia, Racket, Gleam, MoonBit
- `--iterations`: the number of iterations to run the benchmark (default: 1)
- `--output_dir`: the directory to save the results (default: `./output`)
- `--verbose`: whether to print the results to the console (default: True)
- `--timeout`: the timeout for code evaluation in seconds (default: 60)
- `extra arguments`: additional arguments to pass to the model, such as `--temperature`, `--max_tokens`, etc. These arguments depend on the model and the provider you are using.
Below is an example of how to run the benchmark with the Qwen3-32B model (self-hosted with [vLLM](https://docs.vllm.ai/en/latest/)) for Python language.
```sh
# Go to the benchmark directory
cd evaluation/mceval_hard
# Run the benchmark with the proper arguments
python3 -u pipeline.py \
--model "hosted_vllm/Qwen/Qwen3-32B" \
--language "Python" \
--iterations 10 \
--output_dir "results" \
--temperature 0.2 \
--max_tokens 4096 \
--api_base "http://localhost:8000/v1"
```
According to the model you are using, you may need to specify additional arguments, such as `--api_base` for the API endpoint, or exporting an environment variable containing the API key (e.g., `export OPENAI_API_KEY="your_api_key"` for OpenAI models).
Under the `evaluation/mceval_hard` directory, you can find the scripts used in the paper to run the benchmarks and adapt them to your needs.
#### No-Resource benchmarks
In addition to McEval Hard, we provide the translated versions of the HumanEval and MBPP benchmarks for no-resource languages (Gleam and MoonBit). These benchmarks are available in the `evaluation/no_resource_bench` directory.
The arguments to run these benchmarks are similar to those of McEval Hard, with the addition of `--benchmark`, `--run_single_tests`, and `--strategy` arguments:
**Arguments:**
- `--model`: the name of the model to evaluate. The name should match the LiteLLM format (e.g., `hosted_vllm/` for models hosted with vLLM).
- `--language`: one of Gleam and MoonBit
- `--benchmark`: one of 'humaneval', 'mbpp', 'mceval_hard'.
- `--strategy_name`: one of 'zeroshot', 'fewshot', 'rag', 'manual' (default: 'zeroshot')
- `--override_cache`: whether to override the strategy cache (default: False)
- `--remove_thinking`: remove the thinking tokens from the output (default: False)
- `--run_single_tests`: whether to run each test case separately to compute the `passed%` metric (default: False)
- `--iterations`: the number of iterations to run the benchmark (default: 1)
- `--output_dir`: the directory to save the results (default: `./output`)
- `--verbose`: whether to print the results to the console (default: True)
- `--timeout`: the timeout for code evaluation in seconds (default: 60)
- `extra arguments`: additional arguments to pass to the model, such as `--temperature`, `--max_tokens`, etc. These arguments depend on the model and the provider you are using.
Below is an example of how to run o3-mini model on the HumanEval benchmark for Gleam language using the zero-shot strategy:
```sh
export OPENAI_API_KEY="your_api_key"
cd evaluation/no_resource_bench
python3 -u pipeline.py \
--model "o3-mini-2025-01-31" \
--language "Gleam" \
--benchmark "humaneval" \
--strategy_name "zeroshot" \
--iterations 10 \
--output_dir "results" \
--max_completion_tokens 32768
```
### Fine-tuning reuse
Our experiments show that fine-tuning reuse can significantly improve the code generation performance for no-resource languages.
Below, we explain the step-by-step process experimented in the paper.
#### Prerequisites
- An open-source model with a `base` and `instruct` versions publicly available. For example:
- Qwen2.5 Coder 32B (`base`): https://huggingface.co/Qwen/Qwen2.5-Coder-32B
- Qwen2.5 Coder 32B Instruct (`instruct`): https://huggingface.co/Qwen/Qwen2.5-Coder-32B-Instruct
- A training dataset for the no-resource language. This may include code files, API documentation, and other relevant materials.
#### Procedure
Below, we explain the two-step procedure to perform fine-tuning reuse for no-resource languages.
1. **Pre-training**: We continuously pre-train the `base` model on the training dataset via Causal Language Modeling (CLM) objective. This step will return a pre-trained model specialized for the no-resource language of interest. You can find the scripts used in this paper under the `experiments/3_training/pretraining` directory.
2. **Weights diff**: We compute the weights difference between the `instruct` and `base` models. This step will return a `diff` vector containing the instruction-following capabilities acquired by the `instruct` model after its post-training procedure. Then, we add this `diff` vector to the resulting pre-trained model from the previous step. As a result, we obtain a new model that is specialized for the no-resource language and has the instruction-following capabilities of the `instruct` model. You can find an example of such script under `experiments/4_diff` directory.
### Datasets and materials
All benchmarks, model generations, and other materials used in this paper are available in our [Zenodo](https://doi.org/10.5281/zenodo.19366887) repository.
In particular, you can find the following directories:
- **benchmarks**: we provide all the benchmarks used in this paper in the `benchmarks` directory. Each benchmark is stored in a JSONL file, containing the data from the original benchmark and the prompt/tests translation stored in the `translated_prompt` and `translated_tests` fields, respectively. The benchmarks are:
- `mceval_hard`: the McEval Hard benchmark for all the nine programming languages.
- `noresource`: HumanEval, MBPP, and McEval Hard benchmarks translated to Gleam and MoonBit.
- **generations**: containing model generations across all the research questions (`rq1`, `rq2`, and `rq3`) and results of statistical tests (`rq2/stat_tests` and `rq3/stat_tests`). Models generations are stored in JSONL files, where each line contains the task provided to the model and its 10 completions, along with the evaluation results. In `passed_percentage_single_tests`, instead, we provide the evaluation results for each single test case, used to compute the `passed%` metric, and performed in RQ2 and RQ3.
- **prompts**: list of prompts used in the experiments. More details are provided in the **Prompts** section below.
- **scripts**: all code used to create the datasets, run the experiments, and perform the analyses.
- **additional_results**: extra materials commented in the paper but not included in the main results. In particular:
- `errors_analysis`: tables containing the error analysis performed on the model generations for RQ1, RQ2, and RQ3.
- `passed_percentage_results`: tables with the results of the `passed%` metric for RQ2 and RQ3 experiments.
- `losses`: loss recorded during the pre-training and fine-tuning procedures
- `manuals`: the Gleam- and MoonBit-specific manuals used as static prompting strategy.
- `moonbit_image.tar`: the Docker image used to run the benchmarks for MoonBit language, for reproducibility purposes.
In this GitHub repository, we also provide the two manual annotations performed to assess the benchmarks quality and fine-tuning datasets quality, under the `experiments/6_manual_annotations` directory. Tables with error analyses and `passed%` results are also available under the `experiments/7_additional_results` directory.
### Replication
Under the `experiments` directory, you can find the scripts used in this work to replicate the results.
In particular, we provide the following directories:
1. **multiple_evaluation**: containing the scripts used to run the [MultiPL-E](https://github.com/nuprl/MultiPL-E) benchmark on the nine programming languages. Since at the time of our experiments MultiPL-E was mainly designed for base and not instruct models, we adapted the original scripts using the [LiteLLM](https://www.litellm.ai/) library. The zero-shot evaluation is performed using an adapted version of the [Reflexion](https://github.com/noahshinn/reflexion) system prompt, as presented below. Later we will further discuss the prompts used in our experiments.
For better reproducibility, we used a fixed snapshot for gpt-4o (`gpt-4o-2024-11-20`) and o3-mini (`o3-mini-2025-01-31`) models, as well as a fixed seed at inference time (for all the experiments in this paper).
```
You are an AI that only responds with {language} code, NOT ENGLISH. You will be given a function signature and its docstring by the user. Write your full implementation. You always return the signature and anything that came before it in the input prompt (such as the docstring, libraries, imports, and so on) along with the full implementation of the function. Write the output in a markdown code block. For example:```\n\n```
```
2. **mining**: scripts used to fetch and process the two languages documentation (`fetch_language_doc`), retrieve Gleam and MoonBit repositories (`fetch_repositories` directory), mine these repositories and generate the datasets used for the various experiments (`mine_repositories`).
3. **training**: contains the scripts to pre-train and fine-tune the models on the no-resource languages. The fine-tuning directory contains, for each model, the official scripts adapted to our context. Please, before running these scripts, create a separate virtual environment and install the required dependencies from the specific `requirements.txt` file in each directory. Under these directories, you can also find the `losses` directory, which contains the losses during the pre-training and fine-tuning procedures.
4. **diff**: contains the scripts used to transfer the instruction-following capabilities from the instruct model to the pre-trained model specialized one the target no-resource language, as explained in section [Fine-tuning reuse](#fine-tuning-reuse).
5. **stats**: scripts used to perform statistical tests to answer the research questions of this paper. [R](https://www.r-project.org/) is required to run these scripts.
Scripts to run the McEval Hard benchmarks and the no-resource benchmarks (HumanEval and MBPP) are already presented in section [Benchmark Usage](#benchmark-usage). For reproducibility purposes, we recommend to run MoonBit evaluations using the provided Docker image (see `additional_results/moonbit_image.tar`), as it contains the fixed MoonBit version used in the paper.
**Prompts:**
As mentioned before, we adapted the Reflexion system prompt to run the MultiPL-E benchmark on the subject LLMs. The same system prompt is used in all the experiments of this paper, including the McEval Hard benchmark (see [pipeline.py](evaluation/mceval_hard/pipeline.py)) and the no-resource benchmarks (HumanEval and MBPP).
This prompt was slightly modified for the retrieval-augmented generation (RAG) strategy, to include the retrieved documents in the prompt.
For the RAG strategy, we use two additional prompts:
- **Retrieval prompt**: this prompt is used to generate a step-by-step plan, along with documentation queries to retrieve relevant documentation snippets.
- **Documents summarization prompt**: this prompt is used to summarize the retrieved documentation snippets.
All prompts used in this paper are available in the `extra/prompts` directory from the Zenodo repository under the names `default_system_prompt.txt` (the adapted Reflexion system prompt), `rag_system_prompt.txt` (used for the RAG strategy), `retrieval_prompt.txt` (used to generate the retrieval queries), `document_summarization_prompt.txt` (used to summarize the retrieved documentation snippets), and `manual_summarization_prompt.txt` (used to generate the language-specific manuals for the static prompting strategy).
### Citation
You can cite our paper as follows:
```bibtex
@article{giagnorio2026noresource,
title={No Resource, No Benchmarks, No Problem? Evaluating and Improving LLMs for Code Generation in No-Resource Languages},
author={Giagnorio, Alessandro and Martin-Lopez, Alberto and Bavota, Gabriele},
journal={IEEE Transactions on Software Engineering},
year={2026},
publisher={IEEE}
}
```
### References
[1] Ishan Jindal, Chandana Badrinath, Pranjal Bharti, Lakkidi Vinay, and Sachin Dev Sharma. 2024. Balancing Continuous Pre-Training and Instruction Fine-Tuning: Optimizing Instruction-Following in LLMs.
[2] Pin-Jie Lin, Rishab Balasubramanian, Fengyuan Liu, Nikhil Kandpal, and Tu Vu.
2025. Efficient Model Development through Fine-tuning Transfer
