lua-resty-websocket: WebSocket handling inside the nginx request loop

A WebSocket library that runs inside an nginx location block

The library's stated job is WebSocket support for the ngx_lua module, and the whole design follows from where it runs. The README describes it as a Lua library that implements a WebSocket server and client on top of ngx_lua, and it explains the mechanism plainly: the library takes advantage of the cosocket API, which ensures 100% nonblocking behavior.

That single sentence is the entire selling point for the target user. A WebSocket connection in nginx is a hijacked HTTP request, and once the upgrade completes, the request is still tied to a Lua coroutine. With cosockets, the connection is a Lua object with nonblocking read and write operations, so a handler can loop over frames without ever blocking the worker process or tying up a thread.

The audience is therefore narrow and clear: engineers running OpenResty or nginx with ngx_lua who want to speak WebSocket without bolting on a separate process. The server module handles the upgrade inside nginx and the client module lets an nginx worker open an outbound WebSocket connection to another service, using the same socket primitives as the rest of the ngx_lua ecosystem.

Only RFC 6455 is on the table

The README is explicit about protocol scope: only RFC 6455 is supported, and earlier protocol revisions such as hybi-10, hybi-07 and hybi-00 are not and will not be considered.

This matters more than it might appear at first glance, because the WebSocket handshake changed shape several times during its history. Modern browsers negotiate RFC 6455, so a browser client is fine. A client library or embedded device still speaking an old draft is not, and the project has closed that door deliberately rather than by accident.

The same sentence also tells you what is not here. There is no mention of any WebSocket extension in the documentation, which in practice means you should not expect permessage-deflate support, and you should assume the payload limit the library enforces applies to compressed or uncompressed data alike. If your use case is pushing large text through a browser connection, that is a boundary to check early, because the way out of it is to send smaller frames rather than to raise a limit.

The README describes the status of the library as production ready. Whether that description matches your expectations depends on what you are asking it to carry.

server:new does the handshake and returns a frame sender

The server module is loaded with a plain require, and the object is created with a single constructor that also performs the handshake:

    local server = require "resty.websocket.server"

    local wb, err = server:new{
        timeout = 5000,  -- in milliseconds
        max_payload_len = 65535,
    }
    if not wb then
        ngx.log(ngx.ERR, "failed to new websocket: ", err)
        return ngx.exit(444)
    end

    local data, typ, err = wb:recv_frame()

On success you get a websocket object and no error. On failure you get nil and a string describing what went wrong, and the documented way to end the request in that case is ngx.exit(444), which closes the connection without a response body.

The options table is where the interesting limits live. max_payload_len sets the maximum payload length allowed when sending and receiving frames and defaults to 65535. max_recv_len and max_send_len each default to the value of max_payload_len, so a single number governs both directions unless you split them. send_masked controls whether outgoing frames are masked and defaults to false, which matches what a server should do and what a client should not.

The timeout option is in milliseconds and can be changed later through set_timeout(ms). The documentation attaches one important condition to it, covered further down.

Frame types are handled explicitly rather than by a generic read loop

The example in the README is a full request lifecycle rather than a snippet, and it shows how each frame type is meant to be treated. recv_frame() returns the payload, the frame type and an error string, and the caller branches on the type: a close frame is answered with send_close(1000, "enough, enough!"), a ping frame is answered with send_pong(data), a pong frame is discarded, and anything else is logged with its payload.

The send side is symmetric: send_text, send_binary, send_ping, send_pong, send_close and the lower level send_frame, plus recv_frame for reading. The send methods return the number of bytes actually sent at the TCP level, and return nil plus an error string on failure. That distinction between a WebSocket payload and TCP bytes matters when you are writing to a client that disconnects mid-write.

Two details in the example are worth copying. The timeout error is detected with a string match on "timeout" rather than by comparing error codes, so the pattern is: log and exit only when the error is not a timeout. And the close handler reads the status code out of the error field, since for a close frame the README says the error contains the status code.

The lower level send_frame and recv_frame calls are also exposed through a separate resty.websocket.protocol module, which handles protocol.recv_frame, protocol.build_frame and protocol.send_frame for code that needs to work on frames directly.

The timeout option does not cover the handshake response

This is the sharpest documented trap in the library. The README states that the timeout setting does not affect the HTTP response header sending process for the websocket handshake, and that you need to configure the nginx send_timeout directive at the same time.

In other words, there are two separate clocks. The library timeout is in milliseconds and governs network operations on the established connection. The handshake response, the headers that tell the client the upgrade succeeded, is written by nginx itself, and that write is bounded by the send_timeout directive in the nginx core module. If a client opens a connection and stops reading, a library timeout will not save you from a stalled header write.

The fix is administrative rather than clever: set send_timeout in the nginx configuration alongside the Lua timeout, and remember that the library's timeout can be changed mid-connection through set_timeout(ms) while the nginx directive is a configuration-level value. A configuration that sets the library timeout to a second and leaves send_timeout at its default gives you a handshake path with a much longer failure mode than the connection path.

This is also the reason the frame example shortens the timeout to 1000 milliseconds right before it starts sending, an explicit change of the network timeout on a live object.

The client module talks to another server from inside nginx

resty.websocket.client covers the outbound direction, and it has its own method set: client:new, client:connect, client:close, client:set_keepalive, client:set_timeout, the send family and recv_frame.

The set_keepalive method deserves a mention because it is the part that differs from the server side. A cosocket in nginx can be returned to a keepalive pool for reuse, and set_keepalive governs that. The idea is the one the rest of ngx_lua uses: instead of opening a new upstream connection per request, you let nginx manage a pool, and an idle connection can be reused for a later call.

Whether that works well for a long-lived WebSocket depends on what the remote end allows, and the README does not discuss pooling behaviour for WebSocket connections specifically. The documented method set is the guide here: connect, exchange frames, then either close the connection or hand it back with set_keepalive.

For a developer, the client module matters in two situations: proxying or fanning out to a WebSocket backend from an nginx worker, and writing a service that originates WebSocket connections from the same Lua code that serves HTTP.

Installing means copying a directory of Lua files into the Lua path

There is no package manager step. The repository ships a Makefile whose install target copies the library files into the resty/websocket subdirectory of the Lua library path, and whose defaults point at a standard OpenResty prefix:

OPENRESTY_PREFIX=/usr/local/openresty

PREFIX ?=          /usr/local
LUA_INCLUDE_DIR ?= $(PREFIX)/include
LUA_LIB_DIR ?=     $(PREFIX)/lib/lua/$(LUA_VERSION)

The install target itself is two copy commands:

install: all
	$(INSTALL) -d $(DESTDIR)/$(LUA_LIB_DIR)/resty/websocket
	$(INSTALL) lib/resty/websocket/*.lua $(DESTDIR)/$(LUA_LIB_DIR)/resty/websocket/

So the source directory is lib/resty/websocket and the files land under $(LUA_LIB_DIR)/resty/websocket. If your OpenResty lives somewhere other than /usr/local/openresty, or your Lua version differs, the Makefile variables are where you change that; DESTDIR is available for packaging.

The same Makefile has a test target that runs the suite under prove with the nginx binary directory prepended to PATH:

test: all
	PATH=$(OPENRESTY_PREFIX)/nginx/sbin:$$PATH prove -I../test-nginx/lib -r t

Note the -I../test-nginx/lib path. The tests depend on a sibling checkout of the test-nginx harness, so a standalone clone cannot run the t/ suite as it stands. The top-level listing also shows a valgrind.suppress file and a dist.ini, so the project carries both memory-checking support and Perl distribution metadata alongside the Lua code.

Where this approach stops and a standalone server begins

The alternative to embedding WebSocket handling in nginx is to run a dedicated WebSocket process and put nginx in front of it purely as a reverse proxy. The difference in approach is not subtle.

With lua-resty-websocket, the connection never leaves the nginx worker. The upgrade, the frame loop, the upstream call and the logging all share one configuration and one Lua runtime, so a handler can read request headers, apply access control and route to an upstream without a second protocol hop. The cost is that you inherit nginx's concurrency model, its cosocket pool, its timeout semantics as described above, and the fact that a long-lived connection occupies a worker.

A dedicated server gives you its own connection handling, its own scaling story and its own ecosystem of client libraries, and the reverse proxy in front becomes a simple pass-through. What it costs is an extra hop, a second deployment and the loss of the single-process model.

The practical rule: if WebSocket is one feature among many inside an existing OpenResty deployment, this library fits and a separate service does not. If WebSocket is the product, a standalone implementation fits better.

Editorial conclusion

lua-resty-websocket is the right choice for teams that already run OpenResty and want WebSocket traffic handled inside nginx, with access control, logging and upstream routing configured in the same place as the rest of the site. It is the wrong choice for a standalone service, for a browser that needs permessage-deflate, or for anyone expecting a complete WebSocket implementation in the general sense. Verify first that your build of nginx and ngx_lua exposes the cosocket API, that your nginx send_timeout is set alongside the library timeout, and that the t/ suite passes against your own test-nginx checkout.

Frequently asked questions

Official sources

1. Issues
2. openresty/lua-resty-websocket on GitHub
3. README