uts58

A JavaScript implementation of UTS58, the Unicode spec for finding links in running text. Given a chunk of text, it returns the URLs in it along with their UTF-16 offsets. This is a port of the Ruby uts58 gem; the test suite is the same, with very minor differences.

Tested extensively on relevant OSes:

Install

npm install @agulbra/uts58

ESM-only; Node 18+ (uses Unicode property escapes and lookbehinds). Starting with 0.2.4 this ought to work on browsers too.

Usage

import { extractUrls, extractUrlsWithIndices } from '@agulbra/uts58';

extractUrlsWithIndices('see https://example.com/ for details');
// => [{ url: 'https://example.com/', indices: [4, 24] }]

extractUrls('see https://example.com/ for details');
// => ['https://example.com/']

The API mirrors twitter-text's Extractor#extractUrlsWithIndices closely; it was written to provide what Mastodon-style consumers need. The two top-level functions above also strip partly overlapping matches via Extractor#removeOverlappingEntities: candidates are sorted by start offset and anything that begins inside an earlier survivor's span is dropped. Length doesn't enter into it; the earlier-starting candidate wins even if a later one is longer.

Import Extractor directly if you'd rather merge with other extractors (mentions, hashtags, …) and resolve overlap across all of them yourself.

Input without a scheme is recognised, and https:// is prepended in the returned url:

extractUrlsWithIndices('blogspot.com is still a thing');
// => [{ url: 'https://blogspot.com', indices: [0, 12] }]

As in twitter-text, you can pass { extractUrlsWithoutProtocol: false } to find only links that start with http:// or https://. The default is true, matching UTS58.

IDNs are decoded to use UTF-8 in the output, for better readability:

extractUrls('xn-----ctdbabcfhu9c2b9l1acccr4c.xn--mgbah1a3hjkrd')[0];
// => 'https://تجربة-القبول-الشامل.موريتانيا'

(Admittedly that output isn't very readable if you can't read Arabic. But the input wasn't readable to anyone, no matter what languages they can read.)

Trailing punctuation, balanced brackets, ports, paths, queries and fragments are handled per the spec. Indices in the output are UTF-16 code unit offsets (the units used by String.prototype.slice, String#length, the DOM, and text editors) so text.slice(start, end) returns the matched substring directly, even across characters outside the BMP. (This is different from the Ruby version. Ruby tends to use codepoint offsets instead, and the Ruby gem follows that convention. example.com/🐪/#camel is a good test of the difference, since the emoji is one codepoint but two UTF-16 units.)

Email addresses

import {
  extractEmailAddresses,
  extractEmailAddressesWithIndices,
  extractEntities,
  extractEntitiesWithIndices,
} from '@agulbra/uts58';

extractEmailAddressesWithIndices('contact info@grå.org today');
// => [{ email: 'info@grå.org', url: 'mailto:info@grå.org', indices: [8, 20] }]

extractEmailAddresses('contact info@grå.org today');
// => ['info@grå.org']

Each result carries both the bare email and a mailto: url, so it drops straight into anything that already renders a url entity. The domain is IDN-decoded the same way as in extractUrls, and a leading mailto: in the input is absorbed into indices per UTS58 5.2.

A plain address overlaps the bare domain that extractUrls would find after the @. extractEntitiesWithIndices runs both extractors, sorts by start offset, and removes overlaps. The email address starts earlier and wins over the domain inside it:

extractEntities('mail arnt@grå.org or see blogspot.com');
// => ['mailto:arnt@grå.org', 'https://blogspot.com']

Choosing the public-suffix check

To decide whether something.example is a plausible link, the extractor checks the host against a public-suffix table. Which table is a bundle-time choice:

All three expose the same API. The two tables are about the same size and agree on nearly every host. The difference is how strict the check is for the handful of TLDs that only register at the second level. @agulbra/uts58/iana asks only "is the rightmost label a real TLD": enough to tell blogspot.jp from blogspot.exe and to reject typos like example.cmo, but it treats a bare foo.za as plausible. The default @agulbra/uts58 carries the PSL, which knows South Africa registers under co.za / org.za and so rejects a bare foo.za. If that distinction doesn't matter to you, the tables are interchangeable.

Neither reproduces the PSL's wildcard/exception rules: the question here is "could this be a link", not "where is the exact registrable boundary", so a flat membership test is all it does. The PSL table is folded accordingly, that's why the gzip table is only a few bytes bigger than the IANA list.

@agulbra/uts58/core bundles no table at all, and you insert your own check like this::

import { Extractor } from '@agulbra/uts58/core';
import { parse } from 'tldts';

const ex = new Extractor({
  isPlausibleHost: (host) => {
    const p = parse(host);
    return !!p.domain && p.isIcann && p.publicSuffix !== 'invalid';
  },
});
ex.extractUrls('see example.com here');

That route is also how you get exact PSL semantics back, at the cost of a dependency you choose rather than one this package forces on you. (@agulbra/uts58 itself depends only on punycode.)

Suggested test cases and notable behaviour

There are a few sharp edges that you probably should cover in your own tests.

The href and the visible text are not the same string. For see example.com here, the indices span example.com, but url is the longer https://example.com. Use url for the href attribute, slice the original text by indices for the visible content. A test that compares text.slice(start, end) === url will fail on every scheme-less input, and on every IDN where A-labels were decoded.

No autoLink. This package extracts; it doesn't render. There's no equivalent of twitter-text's autoLink, autoLinkUrlsCustom, htmlEscape, etc. — building HTML is the caller's job. This code does not decide on escaping.

No mentions, hashtags, cashtags, replies. UTS58 doesn't define them, so this package doesn't either. If you need them, run twitter-text (or another extractor) for those alongside this one and merge with Extractor#removeOverlappingEntities.

Overlap resolution is start-wins, not longest-wins. This needs a test when you merge entities from multiple extractors. ask [email protected]/02074960909 for details shows why. The raw extractors find both email [email protected] at [4, 21] and url https://example.com/02074960909 at [10, 33]. Start-wins keeps [email protected], which is what a human reader probably considers right, at least one who reads 02074960909 as a phone number. Longest-wins would keep the longer https://example.com/02074960909.

maxLength measures the matched input span. Not the returned URL. IF you pass 12-codepoint cap, the code keeps blogspot.com (12) and drops https://example.com, even though the input span and url happen to be identical there. The asymmetry shows up the other way for example.com (11 input cp, 19 in url), where a cap of 12 lets the code detect a link.

Even though most of the API counts in terms of UTF-16, like the String class, maxLength uses codepoints. The reason is that most of the API is for software developers, but maxLength is for end-users, and codepoints are closer to what end-users see. "☺" and "😀" are both one codepoint long. (This isn't quite perfect: "é" may be either one or two codepoints long and "🇳🇴" is two. Life is hard.)

mailto: is absorbed into indices. Per UTS58 5.2, the input mailto:[email protected] returns an entity whose span covers the whole 23-codepoint run, not just the address. The email field still holds the bare address. If your link-rendering code assumes the span starts at the local-part, mailto inputs will surprise it.

What's not here

• Link validation. Recognised URLs are not fetched, normalised beyond IDN decoding, or their hostnames checked in the DNS. There is no attempt at checking for possible attacks (Мышκин.рф and Мышкин.рф are both detected, note the Greek kappa in the middle of the prince's name).

Regenerating the generated tables

src/constants.js (the link-termination and bracket tables) is packed from the Ruby reference's constants.rb:

npm run maketables

That reads ../ruby/lib/uts58/constants.rb by default; pass a path to point it elsewhere. constants.rb is itself generated on the Ruby side, where tools/maketables.rb downloads the UTS58 data files (LinkTerm.txt and LinkBracket.txt) straight from unicode.org, so the source of truth is Unicode's published data.

src/tlds-iana.js and src/suffixes-psl.js are the public-suffix tables. With no arguments they're fetched from their canonical sources (IANA, PSL). Pass local paths to regenerate offline:

npm run maketlds
npm run maketlds -- /path/to/tlds-alpha-by-domain.txt /path/to/public_suffix_list.dat

License

BSD-2-Clause. See LICENSE.

FWIW, I wrote this as part of my work at ICANN and will maintain it as part of the same work. (I resolve problems relating to Unicode in domains, email addresses and similar, so more people, more communities, can use the internet in the way they prefer.)