How PRISM works

The pipeline in one paragraph

Every input becomes a scene graph: a JSON object with a type, a title, axes and data series (for graphs), objects with properties and geometry, relationships between objects, an optional table, a confidence score and a list of honest notes. All five outputs — description, tree, accessible SVG, audio and data — are generated from that one scene graph, which is why changing a parameter or refining a label updates all of them together.

{
  type: "graph" | "geometry" | "circuit" | "diagram" | "table" | "flowchart" | "math" | "chemistry" | "unknown",
  title, source: "sample" | "svg" | "raster", confidence: 0–1, notes: [ … ],
  axes:      { x: { label, unit, min, max, ticks, px0, px1 }, y: { … } },
  series:    [ { name, kind: "line" | "bar" | "scatter", color, points: [[x, y], …] } ],
  objects:   [ { id, kind, label, desc, props: { "Base": "3 cm", … }, shapes: [ … ] } ],
  relations: [ { from, to, type, label } ],
  table:     { headers, rows }
}

SVG inputs and the sample library

The fourteen samples are written as structured JSON (axes, series, objects with properties, relationships, tables) and rendered into SVG by samples.js. That SVG is semantic: every axis, series, object, relationship and table cell is a group with data-* attributes and a <title>/<desc> pair. The converter does not shortcut from the JSON — it hands the SVG text to the parser, which rebuilds the scene graph from the markup, inverting pixel coordinates back to data values through the axis scales. An uploaded SVG carrying the same annotations gets exactly the same treatment.

An SVG without PRISM annotations is parsed generically: shapes are classified by element type (polygon with 3 points → triangle, and so on), text elements are read literally (SVG text is real text, so that part is honest), two long perpendicular lines meeting at a corner are treated as axes, a regular grid of lines as a table. Confidence is capped at about 70% because meaning, units and relationships are not inferred.

Raster heuristics (PNG, JPG, GIF, WebP)

A raster image contains no structure — only pixels. PRISM applies the following steps in raster-analyzer.js, in this order, on a copy downscaled to at most 360 px wide:

Grayscale and background. Luminance and saturation for every pixel; the most common luminance bin is the background. Dark backgrounds are handled by comparing against that background rather than against white.
Ink mask. A pixel is “ink” when it differs from the background by more than 55 luminance levels or has saturation above 70 (out of 255).
Long lines. Runs of non-coloured ink longer than 35% of the width (rows) or height (columns) become horizontal/vertical lines; adjacent rows are merged into one line with a thickness.
Axes. A vertical line in the left half meeting a horizontal line in the lower half within 5% of the image size is treated as a pair of axes. Their pixel extents define the plot area and a 0–1 scale on each axis.
Table grid. Three or more long lines in each direction, spanning more than half the image with roughly regular spacing, are a grid; cells are (rows − 1) × (columns − 1). Cell contents are not read.
Coloured series. Saturated pixels inside the plot area are binned by hue (12 bins, adjacent bins merged). Each dominant colour becomes a series. For every image column the mean y of that colour is recorded. Wide, tall, filled column-runs are classified as bars; several small separated blobs as scatter; otherwise a line, sampled at about 24 points across the plot. Sample positions are converted to fractions of the axis length (0–1) until you enter the real range in Refine.
Connected components. A flood fill over the ink mask (excluding long lines and, when axes exist, the coloured curves) yields components with a bounding box, pixel count and boundary pixels.
Text-like regions. Small components (height ≤ 7% of the image and ≤ 28 px at analysis scale) are clustered with horizontal neighbours that overlap vertically. Clusters of two or more glyphs, or one wide glyph, are reported as “text region (not read)”. PRISM never guesses what they say.
Shapes. Larger components get a convex hull, simplified with Douglas–Peucker (tolerance 4.5% of the size). Three vertices → triangle (with approximate angles and a right-angle check); four → rectangle or quadrilateral; a hull whose points are at near-constant distance from the centroid (coefficient of variation < 7%) → circle; five to eight vertices → polygon; otherwise “irregular shape” or “line drawing”. Filled versus outline is judged from pixel count over hull area. Because the hull is convex, concave outlines (an L-shape, a star) are reported by their hull.
Classification and confidence. Grid → table; axes + series → graph; axes alone → graph at lower confidence; shapes → diagram; otherwise unknown. Confidence starts at a base for the type and rises with the amount of supporting evidence, capped at 95%, and is cut to 30% if more than 60% of the image is non-background (photographs).

The Refine panel lets you type the title, axis labels, units, the value at the origin and at the axis end, and series names. Data values are then rescaled from their pixel positions, and the description, tree, audio and data all use your labels. The notes on the result say when that has happened.

What it can and cannot detect

Capabilities by input type
Input	Detected reliably	Not detected
PRISM sample or annotated SVG	Everything in the scene graph: labels, units, values, properties, relationships, tables, equations.	Nothing missing — this is the ground truth.
Other SVG	Shape types and sizes, literal text, axes by geometry, grids.	Which text labels which shape; units; data values; relationships.
Raster graph	Axes, plot area, distinct coloured series and their shape, bar heights, scatter positions, trend.	Axis labels, tick values, legends, black-on-white curves (they cannot be separated from axes by colour), overlapping same-colour series.
Raster table	Grid geometry: number of rows and columns.	Cell contents; tables without ruled lines.
Raster diagram	Convex shapes (triangle, rectangle, circle, polygon), their size and position, text-like regions.	What the text says, arrows and leader lines as connections, concave or overlapping shapes, meaning of the diagram.
Photographs, handwriting, equations as images	Usually only that the image is dense or text-like, with low confidence.	Everything else. PRISM says so rather than guessing.

The five outputs

Description (describe.js): type-specific prose at short, medium and detailed levels. Graph statistics — trend, linear fit, minimum, maximum, intercepts, correlation, turning points — are computed from the extracted points, never typed in.
Structure (tree.js): an ARIA tree (role="tree", aria-expanded, aria-level) with Up/Down, Left/Right, Home, End, Enter and * to expand all, plus a text outline.
Accessible SVG (accessible-svg.js): each component is a focusable <g tabindex="0" role="img" aria-label> with <title>/<desc>. Arrow keys move between components, Enter announces “What is this?” in a live region, and the current component is outlined. High-contrast and large-label modes are CSS toggles; the download button serialises the DOM as an SVG file.
Audio (audio.js): Web Speech API with voice and rate controls and per-component buttons; long texts are chunked by sentence so engines that cut off after ~15 s finish the job. Sonification uses the Web Audio API: a triangle-wave oscillator whose pitch maps the y-value across two octaves (220–880 Hz) while time sweeps the x-axis; bars and scatter points play as discrete tones.
Data (data.js): graphs become an x column plus one column per series; tables are shown as-is; cells are editable and “Apply” pushes them back into the scene graph; CSV and JSON downloads.

Privacy

Images are decoded onto an in-memory canvas and never transmitted. The only persistence is the prism:v1 key in your browser's localStorage: display settings and the last ten conversions (a small JPEG thumbnail for raster uploads, or the sample id and parameters). “Clear history” removes them. The pages load two font families from Google Fonts; that is the only external request, and it carries no data about what you convert.

PRISM is built to WCAG 2.2 AA as the baseline; the items below describe what has been implemented and checked by hand, not an external audit.

Landmarks (header, nav, main, sections with headings, footer) and a skip link on every page.
All text at least 4.5:1 against white; headings in near-black (#111318, 17.6:1); muted text #4A5060 (7.6:1).
Body text 1.2 rem in Atkinson Hyperlegible; A−/A+ scale the root font from 85% to 160%; layouts survive 200% zoom.
“Wide spacing” toggle adds letter, word and line spacing for readers with dyslexia.
Every control reachable and operable by keyboard; 4 px focus ring; touch targets at least 44 × 44 px.
Tabs follow the WAI-ARIA tabs pattern; “Show all outputs at once” keeps every panel in the DOM for screen readers.
Dynamic results use aria-live="polite" status regions; the “What is this?” answer is announced.
Colour is never the only carrier: series are also dashed/solid and named; badges carry text.
prefers-reduced-motion disables all transitions; there is no autoplaying media.
Spoken use only: description, tree, per-component speech, sonification and data are all usable without looking at the picture.

Roadmap — and what we chose not to fake

The obvious next step is reading text: an on-device OCR model (for example a small Tesseract build) would turn “text region (not read)” into axis labels, tick values and legends, and a layout model could link labels to shapes and follow arrows. A learned classifier would beat the hand-written rules on photographs and on black-on-white curves. We did not include any of that in this version because we could not ship it honestly inside a static, no-server hackathon page: a half-working OCR that silently misreads “10” as “16” is worse for a blind student than a clear “not read”. Everything PRISM tells you today is either read from markup or computed from pixels, and every guess is labelled as one.

Other planned additions: MathML output for equations, braille-ready tables, tactile-graphic export (line-only SVG), and a shareable link that carries the scene graph rather than the image.

Convert a visual