Vision runtime

Use the vision runtime to ask a question about a photo, isolate a named object, follow it through a video, read a document, recover metric depth and camera intrinsics from one frame, or reconstruct a physically based rendering (PBR) mesh. The runtime provides 19 commands that run on your machine.

Commands

Understand and read

Command	What it does
mere.run vision inspect	Describe or answer questions about an image using Qwen3-VL.
mere.run vision caption	Generate training-friendly captions for images.
mere.run vision ocr	Extract text from images using LightOnOCR, GLM-OCR, or Infinity-Parser2.

Find, isolate, and follow

Command	What it does
mere.run vision ground	Ground text queries in an image with the native Falcon Perception runtime.
mere.run vision segment	Segment prompted objects in an image with the native SAM 3.1 runtime.
mere.run vision track	Track prompted objects through a video with the native SAM 3.1 runtime.
mere.run vision track-live	Capture from a camera and track text-prompted objects with native SAM 3.1.

Faces

Command	What it does
mere.run vision face detect	Detect faces and five-point landmarks in an image.
mere.run vision face embed	Create a normalized ArcFace embedding for one face in an image.
mere.run vision face compare	Compare one face from each of two images with cosine similarity.
mere.run vision face batch	Analyze many images with one warm detector/recognizer session and emit JSONL.

Measure, reconstruct, and move

Command	What it does
mere.run vision pose	Detect body, hand, and face landmarks with the native platform runtime.
mere.run vision flow	Generate dense optical flow between two equal-size images.
mere.run vision depth	Estimate affine-invariant depth for a still image with native Marigold V2.
mere.run vision depth-video	Generate temporally consistent relative or metric video depth with native VDA-S.
mere.run vision geometry	Generate metric depth, normals, camera intrinsics, and a point cloud with native MoGe-2.
mere.run vision geometry-multiview	Solve native DA3-Small multi-view relative geometry, confidence, and cameras.
mere.run vision image-to-3d	Reconstruct a colored object mesh from one image with native TripoSR.
mere.run vision image-to-3d-trellis2	Reconstruct a 512-resolution PBR O-Voxel mesh with native MLX TRELLIS.2.
mere.run vision image-to-3d-multiview	VFX alias for native 4/6-view InstantMesh reconstruction.

The macOS Studio app exposes the same surface through its shared capability contract, as the ten tasks of the Vision domain. Read, Find, Segment, and Track show the answer beside the input: boxes drawn over the picture, masks composited on it, track spans under the video's scrubber, and the raw result document one segment away. Depth, Pose, Faces, Flow, Geometry, and Live cover video depth, native pose and optical flow, Buffalo-L detection, embedding, comparison and batch analysis, single- and multi-view geometry, and live camera tracking; they draw native face and pose overlays and direction-colored flow vectors, play annotated and depth video, embed orbitable point clouds, report request-scoped progress, and preserve structured and visual sidecars in the Library.

The Command view (⌥⌘C) and the Command Console show and run the raw command for any of them. Image-to-3D commands live in the 3D domain so the app has one canonical reconstruction flow.

Model families

• vision-ocr-lighton
• vision-ocr-lighton-3-0.8b
• vision-ocr-lighton-3-4b
• vision-ground-falcon-perception
• vision-face-buffalo-l
• vision-segment-sam31
• vision-geometry-moge2-small
• vision-depth-vda-small
• vision-depth-vda-small-metric
• vision-geometry-da3-small
• vision-depth-marigold-v2
• image-3d-triposr
• image-3d-trellis2-4b
• image-3d-instantmesh-base

Captioning and inspect flows also depend on vision-language support code in MereRunCore.

Grounding runs natively through the Swift/MLX Falcon Perception stack in MereRunCore.

Segmentation and tracking run natively through the Swift/MLX SAM 3.1 stack in MereRunCore.

Face analysis runs the Buffalo-L detector and ArcFace R50 recognizer locally through ONNX Runtime. The default auto provider uses CPU for this model based on the repository's Apple Silicon parity run; --execution-provider coreml and --execution-provider cpu remain explicit controls.

The detector is useful beyond identity search: it provides face boxes and five-point landmarks for alignment, crops, counts, redaction, face-aware reframing, thumbnail selection, and dataset quality gates. The embedding model also supports verification, unnamed-person clustering, similarity search, representative prototype selection, and identity-outlier review. Neither model claims emotion, demographic attributes, liveness, deepfake detection, or general image understanding.

SAM 3.1 scope

• vision segment supports text prompts and geometry prompts with boxes and points.
• vision track supports text, box, and point prompts on the initial frame. It then propagates tracked objects through later frames.
• vision track-live records a camera clip, searches a short warm-up window for seed objects, and then runs the same native tracking path over the saved recording.
• The managed vision-segment-sam31 package supports segmentation and tracking.

Implementation notes

• Still-image text prompting uses the native detector path.
• Still-image box and point prompting use the native interactive SAM prompt path. Boxes and points combine by label: a labeled point refines the box with the same label that contains it (else the first with that label), unlabeled points refine the one unlabeled box when there is exactly one, and points that match no box form one object per label and must include a positive point.
• Offline video tracking uses native prompt propagation built on the image segmenter instead of a full SAM memory-bank tracker.
• Live capture supports text-prompt seeding in the CLI.

Fused attention policy

Supported SAM 3.1, LightOn OCR, and selected vision-encoder attention shapes use MLX fused scaled-dot-product attention by default. Unsupported shapes retain their portable implementation. Set MERERUN_FUSED_SDPA=0 for an emergency compatibility fallback or a controlled A/B.

The installed-model gate on 2026-07-11 used release binaries on an M4 Max with 128 GB unified memory. A SAM 3.1 text-prompt segmentation run was about 3% faster (1.36 seconds to 1.32 seconds) and reduced peak footprint by 13.2%; all 11 exported masks were bit-exact, with only negligible score/box deltas. A warm LightOn OCR run improved from 2.87 seconds to 1.97 seconds (1.46x throughput) and reduced peak footprint by 55%, with byte-identical text. Process RSS was effectively flat in both comparisons, so these measurements are not presented as general RSS savings or as guarantees for other models, prompts, or shapes.

Typical workflows

Caption an image

swift run mere.run vision caption ./image.png

For dataset captioning, use a domain prompt file and focus terms when the generic captioner would miss the training objective:

swift run mere.run vision caption ./cards/*.jpg \
  --output-dir ./captions \
  --prompt-file ./card-caption-prompt.txt \
  --focus "full card border" "printed title text" "visible gag" \
  --trigger-token cardstyle \
  --temperature 0.1

Inspect an image with a question

swift run mere.run vision inspect ./image.png "What objects are visible?"

Detect, embed, and compare faces

swift run mere.run model pull vision-face-buffalo-l --accept-model-license
swift run mere.run vision face detect ./group.jpg --json
swift run mere.run vision face embed ./reference.jpg --json
swift run mere.run vision face compare ./reference.jpg ./candidate.jpg --json

detect --include-embeddings emits one normalized 512-dimensional embedding per detected face. embed selects the largest face by default or accepts --face-index; compare returns cosine similarity. Use the companion mere-face-tools plugin for resumable folder indexing, SQLite search, and review/export workflows.

For folder-scale processing, face batch keeps the detector and embedding sessions warm across many images:

swift run mere.run vision face batch --input-list ./images.txt --jsonl-output ./faces.jsonl

--input-list reads one image path per line (positional image paths also work), --jsonl-output writes one durable JSONL record per image instead of stdout, and --fail-fast stops at the first unreadable or invalid image.

Buffalo-L pretrained weights are provided by InsightFace for non-commercial research use. The pull command requires --accept-model-license to acknowledge the upstream restriction and prints the authoritative license URL before the download starts. The weights are not bundled with mere.run.

Segment an image with SAM 3.1

swift run mere.run model pull vision-segment-sam31 --accept-model-license
swift run mere.run vision segment ./image.png --prompt "a person"

Ground objects with Falcon Perception

swift run mere.run model pull vision-ground-falcon-perception
swift run mere.run vision ground ./image.png --query "cat" "person in red" \
  --mask-output-dir ./masks

--query (alias --prompt) accepts one or more grounding expressions in a single run; at least one is required. If --model/-m is omitted, the command resolves the managed vision-ground-falcon-perception package from the local model store; it also accepts a local Falcon Perception model root directory. The annotated image defaults to <stem>_grounded.<ext> (--output/-o overrides it), JSON metadata defaults to <stem>_grounded.json (--json-output overrides it), and --mask-output-dir exports one PNG mask per detection. --preflight --json validates the image, installed model, queries, and output plan without loading the model; --quiet prints only the annotated image path.

For local model roots, preflight applies the same manifest, checkpoint-file, and tokenizer-layout validation as execution. Missing or invalid manifests fail before the command reports readiness. These checks do not load tensor data or establish model accuracy.

Coordinate decoding follows the Falcon reference's repeated-location policy in both direct grounding and the batch endpoint. Each query keeps its own coordinate history. If both predicted axes are within a strict 1% of an earlier coordinate, the decoder suppresses both selected bins and tries again, for at most 100 attempts. The selected coordinate feeds subsequent token generation; this is not postprocessing of final boxes. The last candidate is retained when the attempt limit is reached, so duplicates are still possible. Token budgets and image preprocessing remain separate settings.

Direct grounding and batched vision serving stop each query when the model emits its configured end-of-sequence token or the tokenizer's <|end_of_query|> token. Other batch queries continue until they stop or reach the requested token limit.

Portable graphs expose the same runtime as the built-in vision.ground node. The node accepts an image plus a JSON array of queries and produces a verified annotated image, structured detections JSON, and a portable masks directory containing per-detection PNG masks. These outputs are candidate geometry, not authoritative evidence.

For repeated binary frames, keep Falcon Perception resident behind the generic loopback vision service:

mere.run vision serve --port 8091

curl http://127.0.0.1:8091/v1/vision/ground \
  -F 'stream_id=camera-1' \
  -F 'frame_id=frame-000042' \
  -F 'query=vehicle' \
  -F 'query=person' \
  -F '[email protected];type=image/webp'

The service accepts PNG, JPEG, or WebP bytes and returns normalized detections, the input image SHA-256, request identifiers, and inference timing. It does not assign application semantics or preserve application-specific state; clients own cadence, temporal association, and policy. Loopback is the default, and non-loopback binds require --api-key.

Use the bounded batch endpoint to ground several images in one native MLX prefill/decode pass:

curl http://127.0.0.1:8091/v1/vision/ground-batch \
  -F 'stream_id=batch-0001' \
  -F 'frame_id[]=frame-000042' \
  -F 'frame_id[]=frame-000043' \
  -F 'query=target' \
  -F 'image[][email protected];type=image/webp' \
  -F 'image[][email protected];type=image/webp'

frame_id[] is optional; when present it must provide one identifier per image. Every query applies to every image, and --max-batch-size limits the flattened number of image-query pairs (8 by default, at most 32). Individual images remain subject to --max-frame-bytes; their combined encoded bytes are bounded by --max-batch-bytes. The batch response contains ordered per-image hashes and normalized detections. It intentionally omits masks and annotated images; use the single-image endpoint when those artifacts are required.

Track objects through a video

swift run mere.run model pull vision-segment-sam31 --accept-model-license
swift run mere.run vision track ./clip.mp4 --prompt "a dog" --init-frame 12

Track a recorded live camera session

swift run mere.run vision track-live --output ./live.mp4 --prompt "a person"

Extract pose landmarks

swift run mere.run vision pose ./person.png \
  --json-output ./person-pose.json \
  --minimum-confidence 0.2

The native pose result contains body, hand, and face subjects. Landmark coordinates use a normalized bottom-left coordinate system and retain per-point confidence for downstream temporal filtering and motion export.

Generate a dense motion pass

swift run mere.run vision flow ./frame-001.png ./frame-002.png \
  --output ./frame-001-to-002.flo \
  --accuracy high

The two images must have equal dimensions. Output vectors use the Middlebury .flo format and preserve full-resolution 32-bit horizontal and vertical motion components.

Estimate depth for a single image

swift run mere.run model pull vision-depth-marigold-v2
swift run mere.run vision depth ./photo.jpg --output ./photo-depth

Treat this runtime as experimental. Accuracy parity with the reference has not been established.

Native Marigold V2 writes a depth EXR, a preview PNG, and a manifest JSON into the output directory (default <stem>-depth next to the input). Inference is a single rectified-flow step at a fixed timestep on a frozen, 4-bit quantized Qwen-Image-Edit-2509 transformer with rank-128 adapters installed on top, so there is no step count or guidance scale to set.

Output is affine-invariant: depth is recovered up to an unknown scale and shift per image. Values are normalized per image so that larger means farther, and the manifest records the mapping that was applied along with the raw range. No camera intrinsics or point cloud are written, because the model does not estimate them; use vision geometry when you need metric units or a camera.

--max-edge bounds the longest inference edge, rounded to a multiple of 16 (default 1024); --native runs at the source resolution instead. Memory grows with resolution, so raise the cap only with headroom to spare. --checkpoint selects a published variant — log-stage2 (default) is the paper model, the *-layered variants predict see-through depth behind glass, and the disparity-* and uniform-* variants change the depth parameterization. Variants other than the default are not part of the managed install and need a local Marigold repository root passed to --model. --dry-run and --json behave as in geometry; --receipt prints the final result line naming the depth EXR, the preview PNG, and the manifest, and cannot be combined with --dry-run.

vision depth is available from the CLI and Studio's Image depth task only. There is no API route: the depth API serves video depth, and the managed catalog declares vision-depth-marigold-v2 CLI-only so API discovery never lists it. Machine admission treats every run as a large job from the catalog's 64 GB minimum, because the BF16 base is loaded before quantization even though the download itself is smaller than the large-download threshold; two depth runs therefore do not share the machine.

The native runtime uses MLX affine 4-bit weights and deterministic VAE encoding. It preserves the first image-modulation projection in full precision; quantizing that projection destroys depth structure. Prequantized checkpoints must preserve this projection too. The reference uses bitsandbytes NF4 and samples the VAE posterior. Accuracy parity between these implementations has not been established. Loading also requires the BF16 base before quantization, so the reference's quantized inference memory figures do not describe native loading requirements.

Generate temporally consistent video depth

swift run mere.run model pull vision-depth-vda-small
swift run mere.run vision depth-video ./clip.mp4 --output ./clip-depth

Native Video Depth Anything Small writes per-frame depth EXRs, preview PNGs, a review MP4, and a depth-sequence manifest JSON into the output directory (default <stem>-depth next to the input). --input-size bounds the longest network edge before aspect-ratio adjustment (default 518) and --max-frames bounds decoded source frames (default 240). The default model is relative depth; --model vision-depth-vda-small-metric switches to the metric variant. --dry-run hashes and decodes the bounded input, verifies media/network limits and the checkpoint, then prints the plan without inference; --json prints the structured result on stdout.

Recover metric geometry from a single image

swift run mere.run model pull vision-geometry-moge2-small
swift run mere.run vision geometry ./photo.jpg --output ./photo-geometry

Native MoGe-2 emits metric depth and normal EXRs with preview PNGs, a validity mask, camera intrinsics JSON, a point-cloud PLY, and a manifest (default directory <stem>-geometry). --resolution-level selects quality 0 through 9 (default 9), --token-count overrides the DINO base-token count (1 to 3600), and --max-points caps the PLY point count. --dry-run and --json behave as in depth-video.

Solve multi-view geometry and cameras

swift run mere.run model pull vision-geometry-da3-small
swift run mere.run vision geometry-multiview ./view-01.jpg ./view-02.jpg ./view-03.jpg \
  --output ./scene

Native DA3-Small solves relative depth, per-view confidence, and cameras across the ordered views, exporting per-view depth/confidence EXRs and preview PNGs, camera JSON, colored point clouds (PLY and GLB), a Nerfstudio/3DGS initialization handoff, and a scene manifest (default directory <first-stem>-da3-scene). --cameras supplies one calibrated W2C camera per image as JSON; --process-resolution bounds the longest processed side (default 504); --reference-view picks first, middle, saddle-balanced (default), or saddle-similarity-range; --confidence-percentile (default 40) discards low-confidence points and --max-points caps scene exports. --dry-run and --json behave as in depth-video.

Reconstruct a PBR object with TRELLIS.2

Accept the DINOv3 checkpoint license on Hugging Face before the first pull, then run the native 512-resolution pipeline:

swift run mere.run model pull image-3d-trellis2-4b --accept-model-license
swift run mere.run vision image-to-3d-trellis2 ./object.png \
  --output ./object-trellis2 \
  --seed 42

Transparent alpha is required by default. --already-framed explicitly opts an opaque, isolated object into black-background conditioning. The result contains canonical colored OBJ/PLY/GLB meshes and a hashed .pbrvox sidecar that preserves base color, metallic, roughness, and alpha.

Output artifacts

vision ground

• annotated image written to <stem>_grounded.<ext> unless --output is provided
• JSON metadata written to <stem>_grounded.json unless --json-output is provided
• optional mask PNGs written to --mask-output-dir

The JSON includes:

• schemaVersion
• model and input/output paths
• query list
• detections with query, normalized xy, normalized hw, derived box, optional score, and optional maskPath

vision face

• detect emits image dimensions, elapsed inference time, face scores, pixel boxes, and five landmarks; --include-embeddings adds identity vectors
• embed emits one selected face and its normalized 512-value embedding
• compare emits the selected face indexes and cosine similarity
• batch keeps the sessions warm and emits one durable JSONL result per image; --include-embeddings enables recognition/search vectors
• --json keeps stdout machine-readable; --json-output writes the same sorted payload atomically

vision segment

• annotated image written to <stem>_segmented.<ext> unless --output is provided
• JSON metadata written to <stem>_segmented.json unless --json-output is provided
• optional mask PNGs written to --mask-output-dir

The JSON includes:

• schemaVersion
• model and input/output paths
• prompts, threshold, and resolution
• detections with label, score, box, maskAreaPixels, and optional objectID, promptKind, maskPath, and candidateIndex

vision track and vision track-live

• annotated video written to <stem>_tracked.mp4 for vision track
• JSON metadata written to <stem>_tracked.json for vision track
• vision track-live requires an explicit output video path
• vision track-live defaults to frame 0 but searches a short warm-up window when that frame yields no seed objects
• optional per-frame mask PNGs written under frame-named subdirectories when --mask-output-dir is set on vision track

The tracking JSON includes:

• schemaVersion
• model and input/output paths
• fps, frame size, init frame, and dropped frame count
• stable tracked object metadata
• per-frame detections with objectID, label, score, visible, box, maskAreaPixels, and optional maskPath

vision pose

• JSON metadata written to <stem>_pose.json unless --json-output is provided
• typed body, hand, and face subjects with stable point names
• normalized coordinates, image dimensions, and confidence values
• no separately installed model package on Apple platforms; inference is owned by the native platform runtime in MereRunCore

vision flow

• dense full-resolution two-component optical-flow vectors
• standard Middlebury .flo output plus typed JSON metadata
• selectable native accuracy and magnitude statistics
• explicit equal-dimension validation

OCR

swift run mere.run vision ocr ./page.png --backend lighton
swift run mere.run vision ocr ./page.png --backend infinity --infinity-task doc2md

--backend, --infinity-runtime, and --compare pick the runtime; each reads only its own options and prints a warning for the rest. For an external Infinity-Parser2 parity evaluation against an already-running vLLM server, run:

swift run mere.run vision ocr ./page.png \
  --backend infinity \
  --infinity-runtime external \
  --infinity-api-url http://127.0.0.1:8000/v1/chat/completions

Runtime entrypoints

CLI

• Sources/MereRunCLI/Commands/VisionCaptionCommand.swift
• Sources/MereRunCLI/Commands/VisionInspectCommand.swift
• Sources/MereRunCLI/Commands/VisionFaceCommand.swift
• Sources/MereRunCLI/Commands/VisionGroundCommand.swift
• Sources/MereRunCLI/Commands/VisionSegmentCommand.swift
• Sources/MereRunCLI/Commands/VisionTrackCommand.swift
• Sources/MereRunCLI/Commands/VisionTrackLiveCommand.swift
• Sources/MereRunCLI/Commands/VisionPoseCommand.swift
• Sources/MereRunCLI/Commands/VisionFlowCommand.swift
• Sources/MereRunCLI/Commands/VisionDepthVideoCommand.swift
• Sources/MereRunCLI/Commands/VisionGeometryCommand.swift
• Sources/MereRunCLI/Commands/VisionGeometryMultiViewCommand.swift
• Sources/MereRunCLI/Commands/VisionOCRCommand.swift

Pose runtime

• Sources/MereRunCore/Pose/NativePoseDetector.swift

Optical-flow runtime

• Sources/MereRunCore/OpticalFlow/NativeOpticalFlowGenerator.swift

OCR runtime

• Sources/MereRunCore/LightOnOCR/LightOnOCRGenerator.swift
• Sources/MereRunCore/LightOnOCR/LightOnOCRGenerator+Loading.swift
• Sources/MereRunCore/LightOnOCR/LightOnOCRGenerator+Inference.swift
• Sources/MereRunCore/LightOnOCR/LightOnOCRSupport.swift
• Sources/MereRunCore/Q35/Q35Generator.swift
• Sources/MereRunQwenModel/Q35Model.swift
• Sources/MereRunCore/Q35/Q35VisionTower.swift

Vision-language support

• Sources/MereRunCore/VLM/
• Sources/MereRunCore/QwenVLCaptioner.swift
• Sources/MereRunCore/Qwen25VLEncoder.swift
• Sources/MereRunCore/QwenVisionAttention.swift

Falcon grounding runtime

• Sources/MereRunCore/FalconPerception/FalconPerceptionConfig.swift
• Sources/MereRunCore/FalconPerception/FalconPerceptionResources.swift
• Sources/MereRunCore/FalconPerception/FalconPerceptionTokenizer.swift
• Sources/MereRunCore/FalconPerception/FalconPerceptionProcessor.swift
• Sources/MereRunCore/FalconPerception/FalconPerceptionModel.swift
• Sources/MereRunCore/FalconPerception/FalconPerceptionAnyUp.swift
• Sources/MereRunCore/FalconPerception/FalconPerceptionGrounder.swift

SAM 3.1 runtime

• Sources/MereRunCore/SAM3/SAM31Config.swift
• Sources/MereRunCore/SAM3/SAM31Resources.swift
• Sources/MereRunCore/SAM3/SAM31Tokenizer.swift
• Sources/MereRunCore/SAM3/SAM31Model.swift
• Sources/MereRunCore/SAM3/SAM31InteractiveSAM.swift
• Sources/MereRunCore/SAM3/SAM31Prompts.swift
• Sources/MereRunCore/SAM3/SAM31ImageSegmenter.swift
• Sources/MereRunCore/SAM3/SAM31VideoIO.swift
• Sources/MereRunCore/SAM3/SAM31VideoTracker.swift
• Sources/MereRunCore/SAM3/SAM31CameraCapture.swift

How the OCR path works

1. The CLI resolves the OCR model.
2. The OCR runtime loads the required components.
3. The runtime normalizes the input image into the expected tensor form.
4. OCR inference runs.
5. The command emits text on standard output without internal bring-up logs.

LightOnOCR uses the dedicated LightOn runtime and remains the default vision ocr backend. Native Infinity-Parser2 uses the Q35 text runtime plus the Qwen-family vision tower, with vision-ocr-infinity-pro-int8 as the default native Infinity model and vision-ocr-infinity-pro as the full BF16 heavyweight compatibility target.

GLM-OCR remains an external CLI adapter that shells out to glmocr. Infinity can also run as an external parity adapter through --infinity-runtime external, which shells out to the parser executable from infinity_parser2 and can target an upstream Transformers, vLLM engine, or vLLM server run.

How segmentation and tracking work

1. The CLI resolves vision-segment-sam31 from the model store or uses the local root passed with --model.
2. The native SAM 3.1 runtime validates the root, loads the tokenizer, configuration, and weights, and preprocesses the input image or video frames.
3. Still-image text prompts run the detector once and then run text, Detection Transformer (DETR), and mask decoding for each prompt.
4. Geometry prompts use the interactive SAM path. Video tracking reuses those prompts frame to frame after the seed frame.
5. Native postprocessing applies thresholding, mask resizing, score ordering, non-maximum suppression (NMS), and optional mask export.
6. The runtime writes annotated media and structured JSON metadata.

How grounding works

1. The CLI resolves vision-ground-falcon-perception from the model store or uses the local root passed with --model.
2. The native Falcon runtime validates the root, loads the configuration, tokenizer, and weights, and preprocesses the image and text query.
3. The model autoregressively emits grounded detections, including coordinate and size tokens, and decodes optional segmentation masks.
4. Native postprocessing derives normalized centers, sizes, bounding boxes, and optional exported mask artifacts.
5. The runtime writes an annotated image and structured JSON metadata for downstream agents.

How caption and inspect differ

• caption is a direct descriptive task.
• inspect is a question-driven vision-language path.

They share some of the same underlying vision support code, but they are presented as separate public tasks because your intent differs.

LightOnOCR 3

vision-ocr-lighton now downloads the pinned official LightOnOCR-3-1B checkpoint. The 1B model uses the native Pixtral/Qwen runtime; vision-ocr-lighton-3-0.8b and vision-ocr-lighton-3-4b use the existing native Qwen3.5 vision runtime. The 4B variant is LightOn's quality recommendation; 0.8B is the smaller lane. The default stays 1B for compatibility with existing workflows.

mere.run model pull vision-ocr-lighton-3-0.8b
mere.run vision ocr ./page.png --model vision-ocr-lighton-3-0.8b --lighton-mode grounding

--lighton-mode plain (default) sends an empty image-only prompt and returns Markdown transcription. grounding sends the exact trained instruction and preserves raw labels and boxes such as ![text](80,175,920,225), followed by content. Coordinates are normalized to 0–1000. Image regions can contain short descriptions; chart regions can contain HTML tables with estimated values. The runtime does not validate those estimates or convert the output to JSON. The flag applies to the LightOn side of --compare as well.

Qwen variants disable thinking and process aspect-preserving images within 65,536–5,000,000 pixels, with further resizing when required by the 16K context budget. Pixtral keeps the 1540px long-edge processor convention. Render PDF pages to upright images before calling the command.

Existing vision-ocr-lighton installations are preserved and may still contain LightOnOCR 2. To intentionally replace an existing install with 3-1B, back it up and run mere.run model pull vision-ocr-lighton --force, or download 3-1B to a separate directory and pass that directory with --model.

All three released BF16 checkpoints completed the bounded native qualification. 1B and 4B passed all ten checked output cases each; 0.8B passed eight and failed the handwritten-word crop in both modes. The generated invoice rows, chart values, and table/chart grounding regions passed for every variant. This is a small image-input qualification; full transcription accuracy, upstream numerical parity, and comparative throughput remain unqualified. See the LightOnOCR 3 release and 1B model card.