# ANEMLL ports Hugging Face LLMs to the Apple Neural Engine, chunk by chunk > ANEMLL is a Python pipeline that converts Hugging Face language models into CoreML for the Apple Neural Engine, with Swift inference code and a chat app on top. The conversion path is well documented, the quantization quality is not, and the package metadata disagrees with itself on version and dependencies. **Anemll/Anemll** — Artificial Neural Engine Machine Learning Library - Repository: https://github.com/Anemll/Anemll - Stars: 1,706 · Forks: 88 - Language: Python - License: not declared - Published: 2026-10-16 · Updated: 2026-10-16 · Language: en - Canonical page: https://hysenlabs.com/projects/anemll-anemll ## The pipeline is the product: HF weights in, CoreML out ANEMLL exists to move a language model onto the Apple Neural Engine, and its centre of gravity is the conversion step rather than the runtime API. The stated goal is a fully open-source path from model conversion to inference for common LLM architectures, with on-device examples in Swift and C/C++. The supported families are given concretely in the README: Gemma 3 in 270M, 1B and 4B QAT variants with sliding window plus global attention and up to 4K context; LLaMA 3.1 and 3.2 at 1B and 8B; Qwen 3 at 0.6B and 1.7B; Qwen 2.5 at 0.5B; DeepSeek R1 8B distilled and DeepHermes 3B and 8B through the LLaMA converter. Pre-converted versions of several of these are published so you can skip conversion entirely for a first run. Version 0.3.5 added monolithic models, meaning a single-file conversion and inference path for LLaMA, Qwen, Qwen 2.5 and Gemma 3, alongside the earlier chunked approach. The package name is anemll, it requires Python 3.9 or newer, and its declared dependencies are coremltools>=9.0, numpy>=1.24.0, tqdm>=4.66.0 and transformers>=4.36.0. Development Status is recorded as 3 - Alpha. ## In-model argmax moves the bottleneck off the ANE bus One of the more interesting changes in 0.3.5 is the --argmax flag, which moves the argmax operation into the CoreML language model head. Instead of shipping full logits from the neural engine back to the host, the model returns a per-chunk winner index and value. That matters because on Apple hardware the transfer between the ANE and the host is where token generation stalls. The README states the effect as a drastic reduction in ANE-to-host data transfer, and notes the design is extensible to top-k sampling. The setting is recorded in meta.yaml as argmax_in_model: true, so a converted model carries the fact about itself rather than leaving the runtime to guess. The CLI also grew a set of debugging switches in the same release: --st for single-token prefill, --cpu, --debug-argmax, --mem-report, --split-rotate and --sliding-window. Stop-token detection became architecture aware, which is a detail that matters when the failure mode is an infinite generation loop rather than a wrong token. ## ANEMLL-Dedup trades a second pass for a smaller model Weights are deduplicated surgically across multifunction CoreML models, and the README puts the saving at roughly 50 percent. The mechanism is straightforward in principle: multifunction models built for the neural engine often carry the same weights in several components, and collapsing them shrinks the artefact. The 0.3.5 release ties this to monolithic models, since a single-file model with deduplicated weights is where the saving is easiest to collect. Documentation for the tool is kept separately at docs/anemll-dedup.md. The claim is about file size, not accuracy. Nothing in the release notes states how deduplication interacts with numerical output, so if you enable it, compare against a non-deduplicated conversion on your own prompts rather than assuming equivalence. ## LUT4 quantization is the acknowledged weak point The README is unusually candid here. A note states that quantization should be improved and that LUT4 quality is fairly low due to a lack of Block Quantization on the Apple Neural Engine. That is the project's own assessment of its quantization fidelity, and it applies to the neural engine's lookup-table mode specifically rather than to every conversion path. Anyone evaluating ANEMLL for an application that cannot tolerate degraded output needs to test LUT4 against their workload before committing, because the documented alternative, block quantization, is not available on the hardware the project targets. There is also a numeric parity table comparing ANEMLL-FP16 against HuggingFace FP16 on the MPS backend across arc_challenge, arc_easy, boolq, piqa and winogrande. The average difference is listed as +0.71 percent, and the note explains that positive values mean ANEMLL scores higher. That table covers FP16, not LUT4, so it should not be read as evidence about the low-quality path. ## Setting up: uv, a Python 3.9 env, and a conversion smoke test The quick start in the README runs through three steps, starting with environment setup. uv is the recommended tool and create_uv_env.sh builds an environment named env-anemll pinned to Python 3.9: ```bash brew install uv # one-time ./create_uv_env.sh # creates env-anemll with Python 3.9 source env-anemll/bin/activate ./install_dependencies.sh ``` The helper scripts activate a project virtual environment automatically, and you can point them elsewhere with ANEMLL_VENV or turn the behaviour off with ANEMLL_AUTO_VENV=0. Before converting your own weights, run one of the bundled tests to confirm the pipeline works end to end: ```bash python tests/test_gemma3_model.py python tests/test_qwen_model.py python tests/test_llama_model.py ``` These exercise Gemma 3 270M in monolithic plus argmax mode, Qwen 3 and LLaMA respectively. A generic entry point also exists for arbitrary Hugging Face models: ```bash ./tests/conv/test_hf_model.sh [model_name] [output_dir] [chunks] ``` Conversion itself goes through convert_model.sh with a model path and an output directory. The README does not document what a successful conversion prints, so treat the test scripts rather than log text as your pass signal. ## Runtime pieces: Swift CLI, chat app, profiler, evaluation harness Around the converter sits a set of consumer tools. The Swift reference implementation provides optimized inference code plus a sample CLI application in anemll-swift-cli. Python examples include a basic chat interface in chat.py and a fuller conversation manager in chat_full.py. ANEMLL Chat, the reference app, was rebuilt for iOS, macOS and visionOS in 0.3.5 with voice input, AirDrop model sharing, local model import and linking, Markdown rendering and a thinking mode. It pulls models from HuggingFace or imports them locally, and manages conversations with streaming and performance metrics. A TestFlight link is published for it. Profiling does not require Xcode. The ANE Profiler analyzes a compute plan, benchmarks individual units and produces compatibility reports, and it needs CoreMLTools 9.0 or newer on macOS 15 or newer. That is the tool to reach for when you need to know why a model is not landing on the neural engine. Model quality checking goes through lm-evaluation-harness support added in 0.3.4, with a script at run_ane_evaluations.sh and documentation under evaluate/ane. ## Version drift and dependency conflicts you will hit on install The metadata is not consistent, and it will cost you time. pyproject.toml reports version 0.3.5 and transformers>=4.36.0, while setup.py still reports version 0.3.3 with the same transformers floor. Building from the wrong path installs the wrong version string. requirements.txt disagrees with both. It pins torch==2.5.0, requires transformers>=4.39.0 rather than >=4.36.0, and caps scikit-learn at <=1.5.1 with the comment that this is the maximum for LUT quantization in CoreML. It also pulls sentencepiece>=0.2.0 for the Gemma tokenizer and psutil>=5.9.0 for system monitoring. lm-evaluation-harness>=0.4.9 is present but commented out as temporarily skipped because it is not on a package registry, even though the 0.3.4 notes announce harness support. Install with install_dependencies.sh so you land on the requirements.txt view, since that is where the scikit-learn ceiling lives. Breaking the cap is how you get a conversion failure that looks like a CoreML bug. On maintenance: the repository is not archived and the last push was on 2026-09-18, with 0.3.5 Beta released on 2026-02-14. Earlier tags 0.3.4 Alpha and 0.3.3 Alpha date from 2025. The MIT licence is recorded in both pyproject.toml and setup.py. Note also that 0.3.4 fixed a RoPE tensor size bug and says pre-0.3.4 models should be re-converted, which is the clearest signal of how fast these conversions go stale. ## Where ANEMLL is the wrong choice It is Apple-only by construction. The target is the Apple Neural Engine, the OS classifier is MacOS X, and the ANE Profiler requires macOS 15 or newer. Running the conversion step on Linux to produce a model for Apple hardware is not something the README describes, and CoreML tooling is itself macOS oriented. It is also not a general inference runtime. If you want to serve a model on a server, or across hardware, the extra layer here buys you nothing. The value is specific: keeping a model resident on Apple silicon for offline or privacy-sensitive use. The alternative approach is to convert models with Apple's own tooling directly and accept whatever ANE coverage it yields, keeping full control over the compute plan. ANEMLL's case against that is breadth of architecture support, in-model argmax, deduplication, and the profiling and evaluation tooling that comes with it. The case against ANEMLL is the quantization ceiling and the metadata drift, both of which you meet during setup rather than after. For benchmarking, ANEMLL-BENCH lives in a separate repository and covers performance testing, optimization metrics and hardware-specific benchmarks. ## Conclusion Use ANEMLL if you need an LLM running on Apple silicon without a network round trip and you are prepared to re-convert models after each release, because the RoPE fix in 0.3.4 alone invalidated pre-0.3.4 conversions. Do not adopt it if quantization fidelity matters more than ANE residency, since the project itself flags LUT4 quality as low. First read requirements.txt rather than pyproject.toml: the two disagree on torch, scikit-learn and transformers, and requirements.txt is where the LUT ceiling on scikit-learn is recorded. ## FAQ ### How do I convert a Hugging Face model to run on the Apple Neural Engine with ANEMLL? Create the environment with create_uv_env.sh, run install_dependencies.sh, then invoke the converter with a model path and an output directory: ./anemll/utils/convert_model.sh --model --output . The README also offers a generic test path at ./tests/conv/test_hf_model.sh for arbitrary models. ### Does ANEMLL support on-device profiling without Xcode? Yes. The ANE Profiler analyzes the compute plan, benchmarks units and produces compatibility reports without Xcode, and requires CoreMLTools 9.0 or newer plus macOS 15 or newer. ### Which architectures can ANEMLL convert? The README lists Gemma 3 at 270M, 1B and 4B QAT, LLaMA 3.1 and 3.2 at 1B and 8B, Qwen 3 at 0.6B and 1.7B, Qwen 2.5 at 0.5B, and DeepSeek R1 8B distilled and DeepHermes through the LLaMA converter. ### What is the quality caveat for LUT4 quantization in ANEMLL? The README states that quantization should be improved and that LUT4 quality is fairly low because the Apple Neural Engine lacks block quantization. The published accuracy comparison between ANEMLL-FP16 and HuggingFace FP16 covers FP16 only. ## Sources - [Anemll/Anemll on GitHub](https://github.com/Anemll/Anemll) - [Issues](https://github.com/Anemll/Anemll/issues) - [README](https://github.com/Anemll/Anemll/blob/main/README.md) - [Releases](https://github.com/Anemll/Anemll/releases) --- Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/anemll-anemll