Downloads, caches and loads Tree-sitter parsers.
A language is a parser WASM plus the queries that drive highlighting, released
together as a @lumis-sh/wasm-* package with its size and SHA-256. Nothing
here needs calling for normal use: Lumis.highlight/2 fetches and loads what a
document turns out to need, including languages injected inside it.
Reach for it in two situations.
Load ahead of the first request, so a download does not land on a user. From
an application's start/2, use async_load/1, which returns before the
network work rather than holding up the boot:
Lumis.Languages.async_load(["elixir", "html"])
:ok = Lumis.Languages.load(["elixir", "html"])Or prepare a directory a different process will read, validating parsers and persisting their compiled modules without retaining any language in memory:
{:ok, _paths} = Lumis.Languages.cache(["elixir", "html"])Where parsers come from
Each is taken from $LUMIS_DATA_DIR/parsers if it is there, and the CDN
otherwise. Bytes are checked against the size and digest their package
declares before use, and anything that fails is discarded rather than trusted,
so a corrupted store repairs itself.
Concurrent requests for the same uncached language wait for the first rather than each downloading it, and loading is global to the VM: the process that pays for a language pays once, for every process after it.
Summary
Functions
Runs load/1 in the background and returns immediately.
The languages each :bundle_* name covers.
Downloads, validates and compiles languages without loading them permanently.
Looks one language up by id or alias, or returns default.
Resolves a name, path or source to a language id, the way highlighting does.
Loads one language, a list of them, or a bundle, verifying each parser first.
Types
Functions
@spec async_load(bundle() | String.t() | atom() | [String.t() | atom()]) :: DynamicSupervisor.on_start_child()
Runs load/1 in the background and returns immediately.
Written for start/2, where the alternative is holding the whole boot on a
download. Highlighting loads on demand anyway, so an application that starts
before its parsers are warm serves correctly the entire time; it only pays for
a language on the first request that names one the warm-up has not reached.
def start(_type, _args) do
Lumis.Languages.async_load(~w(markdown elixir javascript))
Supervisor.start_link(children(), strategy: :one_for_one, name: MyApp.Supervisor)
endFailures are logged rather than returned, because nothing is waiting on them.
Nothing this function does can stop an application from booting: the work runs
under a :temporary child of Lumis's own supervisor, so it is never retried
and never escalates. Use load/1 when the caller does need the result.
Returns {:ok, pid}; the docs above ignore it deliberately, since matching on
it is how a warm-up ends up able to break a boot after all.
The languages each :bundle_* name covers.
These are the same sets the @lumis-sh/wasm-bundle-* packages ship, so naming
a bundle means the same thing in every runtime.
@spec cache( [String.t() | atom()], keyword() ) :: {:ok, [String.t()]} | {:error, String.t() | {:unknown_bundle, String.t()} | %{required(String.t()) => String.t()}}
Downloads, validates and compiles languages without loading them permanently.
Takes the same names load/1 does, including :bundle_*. Returns the paths
written. Already verified parser bytes and compiled modules are reused on
later calls.
This fills a directory for a process that has not started yet, so it validates
and compiles without keeping anything: a release step preparing a volume, or a
task run before the VM that will serve. To warm the VM you are already in, use
async_load/1 or load/1, which do the same work and keep the result, so no
request reloads what this would have discarded.
Names are downloaded and compiled concurrently. Every one is attempted: a
bundle reports each language it could not prepare rather than stopping at the
first, the same way load/1 does.
{:ok, paths} = Lumis.Languages.cache(["elixir", "html"])
{:error, %{"css" => reason}} = Lumis.Languages.cache(["elixir", "css"])Options
:force— resolve the compatible package range again and replace a verified parser
@spec get(String.t() | atom(), any()) :: Lumis.language_info() | any()
Looks one language up by id or alias, or returns default.
The same record Lumis.available_languages/0 returns one of, without scanning
the catalog, and resolving aliases the way highlighting does: "js" finds
JavaScript.
iex> Lumis.Languages.get("js").id
"javascript"
iex> Lumis.Languages.get("not-a-language")
nil
Resolves a name, path or source to a language id, the way highlighting does.
name can be a language id, an alias, a file name or a path. When it does not
resolve, source is checked for an Emacs mode header, a shebang, an HTML
doctype or an XML declaration. Falls back to "plaintext".
The counterpart of Language::guess in Rust and guessLanguage() in
JavaScript. Lumis.highlight/2 already calls this when no language is given;
reach for it directly to label a snippet before deciding what to do with it.
"elixir" = Lumis.Languages.guess("lib/app.ex")
"bash" = Lumis.Languages.guess(nil, "#!/usr/bin/env bash")
"plaintext" = Lumis.Languages.guess(nil, "")
@spec load(bundle() | String.t() | atom() | [String.t() | atom()]) :: :ok | {:error, :unknown_bundle | %{required(String.t()) => failure()}}
Loads one language, a list of them, or a bundle, verifying each parser first.
Highlighting loads on demand, so this is an optimization rather than a requirement: call it at startup to move the download off the first request. Already-loaded languages return immediately.
:ok = Lumis.Languages.load("elixir")
:ok = Lumis.Languages.load(["elixir", :html])
:ok = Lumis.Languages.load(:bundle_web)A :bundle_* atom names the same set of languages as the @lumis-sh/wasm-bundle-*
package of that name, so every runtime means the same thing by it. :bundle_full
is every language in the catalog, which is rarely what a deployment wants; prefer
a narrower bundle, or name the languages a document can contain.
Failures
A list loads every name in it and reports the ones that failed, rather than stopping at the first. One unpublished parser in a bundle should not cost the rest, the same way one bad block does not cost a document.
{:error, %{"css" => :failed_to_load_parser}} = Lumis.Languages.load(["elixir", "css"]):unknown_language— the name is not in the catalog:failed_to_load_parser— it is, but its parser could not be obtained or verified:unknown_bundle— no bundle by that name