charlock_holmes: encoding detection that leans on ICU

The detection work is ICU's, not the gem's

charlock_holmes is a character encoding detection library for Ruby that uses ICU under the hood. The project description is blunt about the arrangement: the detection is brought to you by ICU, and the gem's job is to expose that to Ruby code and to add the transcoding step on top.

That division shapes everything about the library. ICU is a large, mature C and C++ library maintained outside this project, and it is what actually looks at the bytes and decides whether they are UTF-8, ISO-8859-1, something else, or not text at all. The gem is a native extension, with the C++ code under ext/ and the Ruby side under lib/, plus a gemspec, a Rakefile, a Gemfile and a benchmark/ directory. The top-level entries also include a .travis.yml, which dates the continuous integration setup to an era when that was the default choice.

For a reader deciding whether to use it, the practical question is not whether the detection algorithm is good, since you are borrowing ICU's, but whether you are willing to depend on ICU at all. If your data is only ever UTF-8, this library is machinery you do not need. If your data arrives as arbitrary bytes from a legacy system, the heuristics you would write yourself are exactly the thing ICU is better at.

The licence is MIT, the default branch is master, and the repository is not archived.

One call, or a detector you reuse

The simplest use is a single call on a string. You require the library and hand the content to the detector:

require 'charlock_holmes'

contents = File.read('test.xml')
detection = CharlockHolmes::EncodingDetector.detect(contents)
# => {:encoding => 'UTF-8', :confidence => 100, :type => :text}

The result is a hash with the detected encoding, a confidence value and a type. The type is :text for ordinary content and :binary for binary content, which is a distinction worth acting on, since a binary blob is not a text file in a bad encoding. A :language key may also be present, though the documentation notes it comes back mostly only for legacy encodings such as ISO-8859-1.

The failure mode is documented in a single line and it matters: the detect method returns nil when it is unable to find an encoding. That is not an exception, so code that assumes a hash will get a NoMethodError somewhere else in the pipeline, far from the cause. Handling the nil is not optional in production use.

The documentation also points out that it is more efficient to reuse one detector instance rather than calling the class method repeatedly:

detector = CharlockHolmes::EncodingDetector.new

detection1 = detector.detect(File.read('test.xml'))
detection2 = detector.detect(File.read('test2.json'))

Since the heavy lifting happens inside ICU, the question is how much state the detector holds between calls. The documentation's advice is to reuse one, and for a pipeline that inspects many files in a loop that is the shape to write.

detect_encoding! and the monkey patch route

There is a second way in, and it is the one many Ruby applications end up using because it needs almost no ceremony. Instead of a detector object, the library can patch the String class with a method that returns the detection:

require 'charlock_holmes/string'

contents = File.read('test.xml')

detection = contents.detect_encoding

The string variant returns the same kind of detection hash, so you can use it as a drop-in when you only have the bytes and no detector instance to hand.

There is also a bang form, detect_encoding!, which detects the encoding and then sets the encoding flag on the string, returning the string itself. The documentation flags it as a Ruby 1.9 specific method, and it only exists on Ruby 1.9 and later.

require 'charlock_holmes/string'

contents = File.read('test.xml')

# this will detect and set the encoding of `contents`, then return self
contents.detect_encoding!

This is the interesting one for legacy pipelines, because it does not require you to restructure the call site. Code that already has a string and already calls force_encoding on it can be pointed at detect_encoding! instead and get a detected value rather than a guess. The trade-off is that a global monkey patch on String is a decision about the whole process, and it only takes effect when the patch file is explicitly required, so a codebase can easily end up with some call sites using it and others not.

Transcoding is the step you probably came for

The library is direct about this: being able to detect the encoding of arbitrary content is nice, but what you probably want is to be able to transcode that content into an encoding your application is using.

The converter takes three parameters, and the documentation spells out each one. The first is the content to transcode, the second is the source encoding, meaning the encoding the content is assumed to be in, and the third is the destination encoding.

content = File.read('test2.txt')
detection = CharlockHolmes::EncodingDetector.detect(content)
utf8_encoded_content = CharlockHolmes::Converter.convert content, detection[:encoding], 'UTF-8'

The pattern in the example is the one to copy. You detect first, then you feed the detected encoding back in as the source, and you name UTF-8 as the destination. Passing the detection result straight into the converter is what keeps the two steps consistent, because a hardcoded source encoding and a detected one can disagree, and the disagreement shows up as mojibake rather than as an error.

One thing to keep in mind when reading the converter: the source encoding you pass is the encoding the content is assumed to be in. If the detector was wrong, the converter is not going to notice, and it will happily transcode garbage into a confidently labelled UTF-8 string.

Installing means finding your ICU4C first

This is where the friction lives, and the documentation spends more space on installation than on any single API. The gem needs ICU, and if the traditional gem install charlock_holmes does not work, the fix is to tell the gem where your ICU installation lives, using the --with-icu-dir option, either directly during the gem install or through Bundler.

For Bundler, the configuration command sets the arguments to use every time the gem is built:

bundle config build.charlock_holmes --with-icu-dir=/path/to/installed/icu4c

And installing with Gem directly, without Bundler:

gem install charlock_holmes -- --with-icu-dir=/path/to/installed/icu4c

If you hit a compile time error that looks like error: delegating constructors are permitted only in C++11, or something else related to C++11, you need to set the --with-cxxflags=-std=c++11 option as well. Bundler and Gem take the same pair of flags:

bundle config build.charlock_holmes --with-icu-dir=/path/to/installed/icu4c --with-cxxflags=-std=c++11

gem install charlock_holmes -- --with-icu-dir=/path/to/installed/icu4c --with-cxxflags=-std=c++11

On macOS the documentation points at Homebrew as the easiest way to install ICU, with a warning that follows. The Homebrew install is keg-only, which means RubyGems will not find it when installing without specifying --with-icu-dir. The install is brew install icu4c, and the argument to pass becomes the Homebrew prefix path:

bundle config build.charlock_holmes --with-icu-dir=/usr/local/opt/icu4c

gem install charlock_holmes -- --with-icu-dir=/usr/local/opt/icu4c

An old release, an unarchived repository, and where to look next

The maintenance picture is the part to weigh carefully. The repository is not archived, and the last push was on 2026-07-01, so the code is not frozen. The release history, however, shows the most recent tagged version, v0.7.9, dated 2024-07-10. That gap between the last commit and the last release is the honest summary of this project: patches land on master, and a new gem gets cut on its own schedule.

For a library whose job is to wrap ICU, that is a defensible position. The detection algorithm does not change when ICU changes, and the Ruby surface area of this gem is small enough that most of the maintenance is ICU's problem. But if you pin to the released gem you are pinning to code from 2024, and if you need a fix you are tracking master.

There is also no homepage listed for the repository, so the README and the API in the source are the documentation. The benchmark/ directory suggests the author cared about the cost of detection, and the.travis.yml continuous integration configuration is another sign of when the project was last actively shaped.

The alternative to this gem is to call ICU directly, or to use a detector that ships as a pure Ruby library with no native build. The difference in approach is straightforward: charlock_holmes gives you ICU's detection plus transcoding in a few lines, at the price of a C++ extension, an ICU dependency and a build step that needs the right flags. A pure Ruby detector removes the build step entirely and is the better fit for a container image that must build reproducibly on a platform where ICU is awkward. For a service that must handle whatever encoding arrives, and where a wrong guess is expensive, borrowing ICU's implementation is the better trade, provided you can accept the dependency and the nil return path.

Editorial conclusion

Adopt charlock_holmes if you inherit legacy byte streams with mixed encodings and want detection that leans on ICU rather than a hand-written heuristic, and if you are prepared to manage a native extension and an ICU dependency. Do not adopt it for a greenfield service that only ever handles UTF-8, where the build friction buys you nothing, and do not adopt it expecting an actively released gem, because the last release v0.7.9 dates from 2024-07-10 even though the last push was on 2026-07-01. Before you start, confirm your system has ICU4C and whether it is keg-only, because the --with-icu-dir argument is the difference between a clean install and a failed one, and read the nil return path, since a detector that cannot decide returns nil rather than raising.

Frequently asked questions

Official sources

1. brianmario/charlock_holmes on GitHub
2. Issues
3. License: MIT
4. README
5. Releases