@fabi-play/protocol

The messages between the phone, the screen and the games of Fabi Play, with their checks: the body landmarks (P, Body), players, age bands, the game manifest (fabi-play.json, parseManifest, and its JSON Schema), and the parsers every side runs on what it receives.

npm install @fabi-play/protocol

import { parseManifest, AGE_RANGES, type GameManifest } from '@fabi-play/protocol'

const result = parseManifest(JSON.parse(text))
if ('errors' in result) console.error(result.errors)

The manifest's JSON Schema: @fabi-play/protocol/fabi-play.schema.json (also served next to the SDK on https://cdn.fabiplay.app). ESM only, types included, no dependencies. Docs: https://docs.fabiplay.app.

The game intro

A manifest's optional intro (GameIntro: 1 to 4 steps, each a sentence over an image or a move) is what Fabi says on the TV before calibration; see "Fabi explains your game" in the @fabi-play/sdk README. parseManifest reports what is wrong with it; parseIntro gives the intro or null (the TV skips a bad one, unknown fields included), introText the sentence for a locale and age.

• Sentences: up to 90 characters per language, counted as Unicode code points of the NFC form (an emoji is one); no white space at either end; no control characters, lone surrogates or bidirectional controls (U+202A–U+202E, U+2066–U+2069). The JSON Schema says the same, except lone surrogates and the NFC count, which portable JSON Schema regexes can't express: parseManifest is the reference.
• Paths (entry, a step's image): relative to the game's address, no scheme, no leading /, no empty, . or .. segments, up to 200 characters; segments of ASCII letters, digits, _ and - (dots inside, not first); extensions in lowercase: .html for entry, .jpg or .png for images (intro/track.jpg, not intro/Track.PNG). No WebP: old TV browsers (WebKit-based Tizen 2.x, webOS 1–2) may not decode it. JPG (quality about 80) for screenshots, PNG only for flat art.
• Voice clips: a sentence's clip is at voiceClipUrl(base, locale, voiceKeySync(text, locale)), that is ${base}/voice/${FABI_VOICE}/${locale}/${key}.mp3, one folder per language (voiceKey is the same as a promise; voiceClipUrl throws on a locale other than it or en). The key is the first 32 lowercase hex characters of SHA-256 of the UTF-8 bytes of `${FABI_VOICE}\n${locale}\n${t}` (FABI_VOICE is fabi-v1), where t is the text in Unicode NFC with JavaScript's trim() applied; a valid sentence has no white space around it, so for a generator in any language t is just the NFC text. So voiceKeySync('Salta!', 'it') is cbd16ba39816bdc5afd34a3e84082331 and its clip https://cdn.fabiplay.app/voice/fabi-v1/it/cbd16ba39816bdc5afd34a3e84082331.mp3. voiceUrl(base, key), the old flat layout without the locale folder, is deprecated. The SHA-256 is plain JavaScript, synchronous: no crypto.subtle, so it works on TV pages served over http, old TV browsers and Node 18 (the package's engines).
• The phone's buttons: during the intro the phone sends { type: 'intro', action }, with action next, repeat, skip, or replay (from the pause menu: the explanation again); the screen tells it what Fabi is saying with { type: 'intro', step, steps, text, voice }. Its phase message carries intro: true when the game has an intro: only then does the phone offer replay.

The cover

A manifest's optional cover is the game's card in the catalogue: a 16:9 image (1280×720 recommended) of the game's own files, cover.jpg. parseManifest checks the path (isCoverPath: relative, no scheme, no leading /, no backslash, no empty, . or .. segments, .jpg, .jpeg, .png or .webp in lowercase, up to COVER_MAX_PATH, 128, characters). The file itself is checked by Fabi Play's review, with the constants exported here: COVER_MAX_BYTES (150 000), COVER_ASPECT (16 / 9) and COVER_ASPECT_TOLERANCE (0.02, relative), with isCoverAspect(width, height); COVER_EXTENSIONS and COVER_RECOMMENDED (1280×720) too.

Orientations

A manifest's optional orientations lists the screen orientations a game can be played in (Orientation: 'landscape' | 'portrait', all of them in ORIENTATIONS): non-empty, without repeats; absent means ['landscape'], which gameOrientations(manifest) applies. A phone alone («Gioca qui / Play here») is held upright and offers only games that list 'portrait'; the TV and the tablet are landscape. The screen says the orientation a game starts in with init.orientation (orientationOf(width, height): width ≥ height is 'landscape'); absent from older hosts, it means 'landscape'. It is informational: a game lays out from its real viewport size.

Aiming. aimPoint(body | frame, side, { orientation, view }) is the one rule for where a hand aims on the screen (what it touches), in fractions: in landscape toScreen(view, wrist); in portrait, where the camera image is small at the bottom of a phone, the wrist stretched around the shoulders to the whole screen (HAND_SPACE, the stretch of PlayerFrame.leftHand, clamped to 0–1; with the hips unseen the torso is at least TORSO_PER_SHOULDER_WIDTH shoulder widths). null when the wrist (in portrait, the shoulders too) isn't seen (AIM_VISIBLE). AimHold keeps a lost hand where it was for AIM_HOLD_MS (300). The SDK's play.aim and the engine's hands() apply it.

Safe area. init.safeArea (optional) is what a phone's notch, rounded corners and home bar cover, in fractions of the game's viewport: { top, bottom, left, right } (SafeArea). parseSafeArea clamps each side to 0–0.25 (MAX_SAFE_INSET), 0 when missing; NO_SAFE_AREA is all zeros, what hosts that don't send it mean.

Local poses. init.localPoses (optional) is true when the bodies in the frames come from a camera on the same device as the screen (a phone or a tablet playing alone), not from a phone over the network. The SDK then smooths and interpolates the avatars lighter by default (less latency: there is little jitter to absorb), unless the game gives a smooth amount of its own. Absent (older hosts, a phone sending to a TV) means false.

The carousel on two screens

The phone can show the TV's carousel of games, and both move together. While the grown-up swipes, the phone sends { type: 'focus', game } (ControllerMessage) for the card it centred: a preview only, nothing starts (pick starts a game). The screen centres the same card and says the game's name with its voice (the screen owns the voice; the phone stays silent). When the screen's carousel moves for any other reason (the remote, the hand cursor, another phone), it sends { type: 'focus', game } (ScreenMessage) and the phones follow. Both are checked like pick (isGameId); a side that doesn't have the game in its catalogue ignores the message.

Poses on the wire

The phone streams { type: 'pose', t, bodies } about 30 times a second over an unordered, no-retransmit data channel. A screen that lists POSE_CODEC_VERSION in its welcome (binaryPose: [1], see acceptsBinaryPose) gets them in binary, encodePose on the phone and decodePose on the screen; any other screen gets JSON, and a screen reads both. Version 1, little-endian:

So 12 + 132 × bodies bytes (540 for 4 bodies). Values are the thousandths toPlayerSpace rounds to, so a decoded pose holds exactly the numbers the JSON one would. decodePose returns null on anything else (wrong length, magic, version or kind, more than 4 bodies, a value over 1000, t not finite) and never throws.

Most games need only @fabi-play/sdk, which includes what it uses from this package.

License

Fabi Play Kit License: free to use, for making games and tools for Fabi Play only. See LICENSE.

The TV remote

A manifest's optional inputs (GameInput[], from GAME_INPUTS: body, remote; absent ["body"], see gameInputs) says whether the game can be played with the TV remote when there is no phone; parseManifest requires body in it, no repeats and no unknown values. The screen tells the game how a session is played in init.input (absent 'body') and, with 'remote', whether the remote points (init.pointer, LG's Magic Remote: plain pointer and mouse events in the game's page). The screen turns keys into gestures (↑ or OK jump, ↓ held squat/stand, ← → lanes, OK twice clap, Back the platform's pause) and plays one child at a time: init's players is the child playing now (one id). Several children take turns only with games whose manifest says "turns": true (valid only with remote in inputs; it means the game follows the turn contract; without it, one child only): init.turns lists every child in turn order (parseTurnOrder; absent: players). The game plays a round per child; when a child's round ends (a crash too) and turns remain, it asks break('turn'), and the screen always answers break-start (its turn screen), { type: 'turn', players: [2] } (exactly one id, the next in the order; parseTurnPlayers checks its shape) and break-done, the next child's puppet recentred in the middle lane. After the last round the game ends with every child's scores. level and gameover breaks never swap players; hosts before turns answer turn as an unknown reason, at once. Games keep scores by player id. See "Playing with the TV remote" in the @fabi-play/sdk README.

The arrows as a cursor

A manifest's optional "remoteCursor": true (valid only with remote in inputs) asks the screen, in remote mode, to turn the arrows into a free cursor instead of gestures. The screen keeps which arrows are held and runs KeyCursor every animation frame (update(held, dtMs) → { x, y }, 0–1 of the game's frame: slow at first, faster the longer an arrow is held, opposite arrows cancel, diagonals no faster, clamped to the edges, a long frame counted as 100 ms; reset() at a turn), draws the cursor above the game, and sends { type: 'cursor', x, y, down } (parseCursor) when it moves and when OK goes down or up. init says pointer: true, as for a Magic Remote. The SDK turns the messages into pointer and mouse events in the game's page, so the game sees a mouse either way. Back stays the platform's pause; OK is the cursor's button, not a jump.

Pictures to the phone

A game can save a picture on the family's phone. The path, and the messages:

1. The screen tells the game whether it can: init.pictures and, when it changes, { type: 'capabilities', pictures }. It is true while a phone whose hello said pictures: true is linked (the app; the web phone doesn't say it).
2. The game sends { type: 'picture', data, title?, id? } (GameMessage): data a base64 data URL, data:image/jpeg or data:image/png, at most MAX_PICTURE_BYTES (3 MiB of image), whose bytes really start as that type (checkPicture; parseGameMessage drops anything else); title up to MAX_PICTURE_TITLE (60) characters, cleaned by parsePictureTitle.
3. The screen sends it to the phone over their reliable, ordered data channel, in pieces, because data channels don't carry multi-megabyte messages (16 KiB is what every browser takes): chunkPicture(dataUrl, id, title) gives { type: 'picture-start', id, mime, size, title? } (size in bytes of the image), then { type: 'picture-chunk', id, seq, data } (PICTURE_CHUNK_CHARS, 16 000 base64 characters, each but the last, seq from 0; each JSON message stays under 16 KiB), then { type: 'picture-end', id } (ScreenMessage).
4. The phone feeds them to a PictureAssembler (chunks in any order; a missing chunk, a picture over the limit, a chunk past its size, bytes that are not the type the start said, all fail with a reason) and answers { type: 'picture-saved', id } or { type: 'picture-failed', id } (ControllerMessage).
5. The screen answers the game { type: 'picture-result', id, result } (parsePictureResult): 'sent', 'unavailable', 'too-big' or 'error'.

Privacy: the picture travels only TV → phone over the direct link (WebRTC, encrypted), never through or to a server, and no copy is kept anywhere else. Games can send only a JPEG or PNG image this way.