HTTPoison: an Elixir HTTP client, and the atom leak in its own example

The dependency floor is OTP 27

HTTPoison describes itself, in the most Elixir way available, as yet another HTTP client, and the README is direct about where it sits in the stack: an HTTP client for Elixir based on HTTPotion, powered by hackney.

The requirement is the first thing to check. HTTPoison 3.x depends on hackney 4.0, which requires Erlang/OTP 27 or later, and therefore Elixir 1.17 or later. The README states plainly that older OTP releases are not supported.

That is a significant floor for a library most people reach for as a default. If your production runtime is on OTP 26, HTTPoison 3 is not available to you and you are looking at the 2.x line, which the README documents under its own heading for upgrading.

The install itself is a Mix dependency and a fetch.

def deps do
  [
    {:httpoison, "~> 3.0"}
  ]
end

followed by `$ mix deps.get`. Documentation is hosted on hexdocs.pm, and the package itself is on hex.pm.

The lineage is worth knowing because it explains some of the API. HTTPoison is based on HTTPotion, a separate project, which itself carries the same core ideas. The GitHub API example in the README calls `GitHub.get!("/users/myfreeweb")`, and myfreeweb is the author behind HTTPotion, so the example was carried across the fork rather than rewritten.

Two error shapes, and the bang tells you which one you have

The API surface is small and the convention is the interesting part. There is a bang function and a non-bang function for each verb, and the difference is whether an error becomes an exception or a return value.

HTTPoison.get! "https://postman-echo.com/get"
%HTTPoison.Response{status_code: 200, body: "...", headers: [...]}

HTTPoison.get "http://localhost:1"
{:error, %HTTPoison.Error{id: nil, reason: :econnrefused}}

The success struct carries `status_code`, `body` and `headers`, with `json` present when the server returned JSON. The error struct carries an `id`, which is `nil` when no request was ever created, and a `reason` that is the POSIX error code, `:econnrefused` there. The README links Erlang's `inet` documentation for the full list of possible reasons, which is the reference to bookmark.

Because both shapes are plain structs, the intended usage is pattern matching rather than inspecting fields.

case HTTPoison.get(url) do
  {:ok, %HTTPoison.Response{status_code: 200, body: body}} ->
    IO.puts body
  {:ok, %HTTPoison.Response{status_code: 404}} ->
    IO.puts "Not found :("
  {:error, %HTTPoison.Error{reason: reason}} ->
    IO.inspect reason
end

The one thing to notice is that a 404 and a 200 are both `{:ok, ...}`. Status code handling is yours to write, which is correct for a client library and is also the most common place an application gets its error handling wrong.

Options, and the 2.x change that will bite a merge

Request options are passed as the third argument, and the README calls out that they are not to be confused with the HTTP options method.

The most commonly needed one is `:recv_timeout`, which sets a timeout on receiving a response with a default of 5000ms. Five seconds is short for a slow API and long for a dead socket, which means it is the option you will end up overriding in nearly every project that talks to someone else's server.

The `:ssl` option accepts anything the Erlang SSL module accepts. A request to an API needing a bearer token looks like this.

options = [ssl: [{:versions, [:'tlsv1.2']}], recv_timeout: 500]
{:ok, response} = HTTPoison.get(url, headers, options)

Client certificate authentication works the same way, with `ssl: [certfile: "certs/client.crt"]`.

Then the breaking change, which the README devotes a section to because it is silent when it goes wrong. In 2.x the `ssl` option now merges with the default options, where previously it would override them. A new `ssl_override` option was added for people who want the old behaviour, and the README notes it is more explicit now.

The consequence is that code written against 1.x may silently lose its TLS settings after an upgrade. Merging means your partial SSL list is combined with hackney's defaults rather than replacing them, and if you had been overriding a default deliberately, that override quietly stopped happening. The default options are documented by pointing at hackney's own source, since HTTPoison uses `:hackney_connections.merge_ssl_opts/2` directly. Pull request 466 has more context.

HTTPoison.Base, and an atom leak in the README's own example

`HTTPoison.Base` is the feature that turns a client into an API client. You `use HTTPoison.Base` in a module and override whichever hooks you need, from `process_request_url` and `process_request_body` through `process_response_body`, `process_response_chunk`, `process_response_headers` and `process_response_status_code`. That is a proper behaviour with a default implementation per callback, so you override only what changes.

The README's worked example builds a GitHub client: `process_request_url` prefixes `https://api.github.com`, and `process_response_body` decodes JSON, takes a fixed list of fields, and converts the remaining keys to atoms.

That last line is the problem, and it is the example the README tells you to copy.

|> Enum.map(fn({k, v}) -> {String.to_atom(k), v} end)

`String.to_atom/1` creates a new atom for every distinct key, and BEAM atoms are never garbage collected. A long-running system decoding responses from an API you do not control will accumulate atoms permanently, one per unique key per response. The atom table is fixed at boot in older releases and growable but never shrinking in current ones, so the failure mode is a slow memory climb rather than a crash, which makes it worse to diagnose.

The safe version is `String.to_existing_atom/1`, which raises on an unknown key instead of inventing one, or a map with string keys. Either way you have to change the example, so it is worth knowing before you copy it.

The example has a second, smaller problem. `@expected_fields` is a hardcoded list of GitHub user fields, and `Map.take` silently drops anything new, so your client quietly returns less than the API sends the day GitHub adds an attribute.

Async streaming, and the warning attached to it

HTTPoison can stream a response into a process instead of buffering it, and the shape is a series of tagged structs delivered to whatever process you name.

iex> HTTPoison.get! "https://github.com/", %{}, stream_to: self
%HTTPoison.AsyncResponse{id: #Reference<0.0.0.1654>}
iex> flush
%HTTPoison.AsyncStatus{code: 200, id: #Reference<0.0.0.1654>}
%HTTPoison.AsyncHeaders{headers: %{"Connection" => "keep-alive", ...}, id: #Reference<0.0.0.1654>}
%HTTPoison.AsyncChunk{chunk: "<!DOCTYPE html>...", id: #Reference<0.0.0.1654>}
%HTTPoison.AsyncEnd{id: #Reference<0.0.0.1654>}
:ok

Every message carries the same `id` reference, which is how you correlate a stream's events, and the lifecycle is status, then headers, then any number of chunks, then end.

Then the warning, which is the whole paragraph: this option can flood a receiver in messages. With a large response, chunks arrive faster than a BEAM process can reduce them, and the mailbox grows until memory does.

The fix is the `async: :once` option. Instead of pushing everything, it sends a single chunk at a time and waits for the receiver to say it can handle more by calling `HTTPoison.stream_next/1`. That turns an unbounded mailbox into a pull-based flow control, which is what you want for a large download or a slow consumer.

The README also documents cookie support, and the section on it is cut off in the copy available here, so what it permits is not something to guess at.

The sample output predates the hackney dependency it needs

In the usage examples, the response echoed back by postman-echo includes a `user-agent` of `hackney/1.18.1`.

That is a hackney 1.x user agent, in a README whose requirements section says version 3.x depends on hackney 4.0. The examples were not re-run when the dependency floor moved by three major versions, which means the exact values in the sample output, down to the trace IDs and the content-length, are not what you will see today.

That is not a functional problem. Nothing depends on the user agent, and the struct shapes are still accurate. It matters for two other reasons. Anyone reading the README to determine which hackney is underneath will draw the wrong conclusion, and it is a signal about how carefully the surrounding documentation is maintained, which is worth weighing against the parts that are maintained well, like the detailed 2.x upgrade note.

The repository itself is a conventional Elixir package: `mix.exs`, `mix.lock`, `config/`, `lib/`, `test/`, a `CHANGELOG.md`, a `LICENSE` and a `.formatter.exs` for `mix format`. There is no vendored copy of hackney, which is the right arrangement for an Elixir library and means the HTTP behaviour you inherit is whatever the resolved hackney release does.

Where HTTPoison is the wrong tool

Three cases are ruled out by the project's own documentation, and two more follow from what it is built on.

If you cannot move to OTP 27, this is 2.x. The README says older OTP releases are not supported for 3.x, and the 2.x upgrade note describes behaviour changes you would be installing deliberately rather than by accident.

If you need request or response instrumentation, this is thin. The library offers a bang and a non-bang call, a behaviour with hooks, and streaming. There is no documented tracing, metrics or structured logging hook in what the README describes, so an HTTP client in a system with an OpenTelemetry requirement needs its own wrapper around the behaviour callbacks.

If you are streaming very large bodies, the default async mode is the wrong choice and you must remember `async: :once`. The README warns about the mailbox flood rather than making the safe path the default, so that is a decision every caller has to make deliberately.

The two that follow from the architecture: HTTPoison is a thin wrapper, so any hackney limitation is yours, and its SSL defaults are hackney's defaults, read from a specific commit of a different repository. Pinning hackney yourself and reading its configuration is part of running this in production.

HTTPoison against calling hackney directly

The alternative is not another Elixir library. It is to skip the wrapper and depend on hackney itself, which HTTPoison is explicitly built on.

The difference in approach is entirely about the surface you get. hackney is an Erlang HTTP client, so you would work with its own pool configuration, its own request records and its own SSL options, reading the defaults from the same source the README points you at. You would also lose the Elixir-flavoured conveniences: the `HTTPoison.Response` and `HTTPoison.Error` structs, the bang and non-bang pairing, `HTTPoison.Base` with its per-callback hooks, and the async structs with `stream_next/1`.

In practice, teams that already have a wrapper layer around hackney often stop at hackney, because the wrapper is thin enough that maintaining it costs less than depending on someone else's version of it. HTTPoison earns its place when you want that behaviour implemented once, correctly, with the 2.x SSL merge handled and the async backpressure available.

The deciding factor is your OTP version and how much Elixir-specific structure you want in your client code. On OTP 27 with a mix of projects, take HTTPoison. On OTP 26, or in a codebase that has already committed to hackney, the dependency is one you have anyway.

Editorial conclusion

Adopt HTTPoison when you are on OTP 27 or later and want bang and non-bang calls, a Base behaviour and chunked async without writing them, and set recv_timeout explicitly rather than accepting the 5000ms default. Do not copy the README's Base example verbatim, because String.to_atom on response keys never frees its atoms. Verify first that no code depends on the ssl option overriding hackney's defaults, since 2.x merges them instead and ssl_override is the documented way to get the old behaviour back.

Frequently asked questions

Official sources

1. edgurgel/httpoison on GitHub
2. License: MIT
3. Project website
4. README
5. Releases