Lumis.Languages (Lumis v0.10.0)

Copy Markdown View Source

Loads Tree-sitter parsers this project depends on.

A language is a parser WASM plus the queries that drive highlighting, shipped together as a lumis_wasm_* dependency. Nothing here needs calling for normal use: Lumis.highlight/2 loads what a document turns out to need, including languages injected inside it — as long as the project depends on them. One it does not answers :not_installed.

Reach for it to load ahead of the first request, so the compile does not land on a user. From an application's start/2, use async_load/1, which returns without holding up the boot:

Lumis.Languages.async_load(["elixir", "html"])
:ok = Lumis.Languages.load(["elixir", "html"])

Where parsers come from

From the priv/parsers of each installed lumis_wasm_* dependency, or a directory named by config :lumis, :parser_dirs. A language no dependency supplies is not fetched — it answers :not_installed. The bytes are loaded as Hex delivered them, or as the project put them there.

Concurrent requests for the same unloaded language wait for the first rather than each compiling it, and loading is global to the VM: the process that pays for a language pays once, for every process after it.

Summary

Types

A language set shared with the @lumis-sh/wasm-bundle-* packages.

Functions

Runs load/1 in the background and returns immediately.

The languages each :bundle_* name covers.

Looks one language up by id or alias, or returns default.

Resolves a name, path or source to a language id, the way highlighting does.

Loads one language, a list of them, or a bundle.

Types

bundle()

@type bundle() ::
  :bundle_web
  | :bundle_web_extra
  | :bundle_system
  | :bundle_backend
  | :bundle_full

A language set shared with the @lumis-sh/wasm-bundle-* packages.

failure()

@type failure() ::
  :unknown_language
  | :not_installed
  | :failed_to_load_parser
  | Lumis.ParserError.t()

Functions

async_load(names)

@spec async_load(bundle() | String.t() | atom() | [String.t() | atom()]) ::
  DynamicSupervisor.on_start_child()

Runs load/1 in the background and returns immediately.

Written for start/2, where the alternative is holding the whole boot on a compile. Highlighting loads on demand anyway, so an application that starts before its parsers are warm serves correctly the entire time; it only pays for a language on the first request that names one the warm-up has not reached.

def start(_type, _args) do
  Lumis.Languages.async_load(~w(markdown elixir javascript))
  Supervisor.start_link(children(), strategy: :one_for_one, name: MyApp.Supervisor)
end

Failures are logged rather than returned, because nothing is waiting on them. Nothing this function does can stop an application from booting: the work runs under a :temporary child of Lumis's own supervisor, so it is never retried and never escalates. Use load/1 when the caller does need the result.

Returns {:ok, pid}; the docs above ignore it deliberately, since matching on it is how a warm-up ends up able to break a boot after all.

bundles()

@spec bundles() :: %{required(bundle()) => [String.t()]}

The languages each :bundle_* name covers.

These are the same sets the @lumis-sh/wasm-bundle-* packages ship, so naming a bundle means the same thing in every runtime.

get(name, default \\ nil)

@spec get(String.t() | atom(), any()) :: Lumis.language_info() | any()

Looks one language up by id or alias, or returns default.

The same record Lumis.available_languages/0 returns one of, without scanning the catalog, and resolving aliases the way highlighting does: "js" finds JavaScript.

iex> Lumis.Languages.get("js").id
"javascript"

iex> Lumis.Languages.get("not-a-language")
nil

guess(name, source \\ "")

@spec guess(String.t() | atom() | nil, String.t()) :: String.t()

Resolves a name, path or source to a language id, the way highlighting does.

name can be a language id, an alias, a file name or a path. When it does not resolve, source is checked for an Emacs mode header, a shebang, an HTML doctype or an XML declaration. Falls back to "plaintext".

The counterpart of Language::guess in Rust and guessLanguage() in JavaScript. Lumis.highlight/2 already calls this when no language is given; reach for it directly to label a snippet before deciding what to do with it.

"elixir" = Lumis.Languages.guess("lib/app.ex")
"bash" = Lumis.Languages.guess(nil, "#!/usr/bin/env bash")
"plaintext" = Lumis.Languages.guess(nil, "")

load(names)

@spec load(bundle() | String.t() | atom() | [String.t() | atom()]) ::
  :ok
  | {:error, :unknown_bundle | failure()}
  | {:error, %{required(String.t()) => :unknown_bundle | failure()}}

Loads one language, a list of them, or a bundle.

Highlighting loads on demand, so this is an optimization rather than a requirement: call it at startup to move the compile off the first request. Already-loaded languages return immediately.

:ok = Lumis.Languages.load("elixir")
:ok = Lumis.Languages.load(["elixir", :html])
:ok = Lumis.Languages.load(:bundle_web)

A :bundle_* atom names the same set of languages as the lumis_wasm_bundle_* package of that name, so every runtime means the same thing by it. :bundle_full is every language in the catalog, which is rarely what a deployment wants; prefer a narrower bundle, or name the languages a document can contain.

Loading only ever reaches parsers this project depends on, because a parser is an ordinary dependency:

# mix.exs
{:lumis_wasm_elixir, "~> 0.26.0"}

Failures

A list loads every name in it and reports the ones that failed, rather than stopping at the first. One unpublished parser in a bundle should not cost the rest, the same way one bad block does not cost a document.

{:error, %{"css" => :failed_to_load_parser}} = Lumis.Languages.load(["elixir", "css"])
  • :unknown_language — the name is not in the catalog
  • :not_installed — it is, but this project does not depend on its parser
  • :failed_to_load_parser — the parser could not be read or loaded
  • :unknown_bundle — no bundle by that name
  • %Lumis.ParserError{reason: :store_full} — the parser is fine, and this process has no room for another one

The last is the exception Lumis.highlight/2 answers with, rather than an atom, because it is the one failure here that is not about the language named: it loads on its own, retrying it cannot help, and Exception.message/1 says what does. See Lumis.ParserError for the reason itself.

A single name answers with the reason itself; a list answers with a map from name to reason, so one failure never hides the others.