What Is Set-of-Mark Prompting?

Scrapeless Scraping Browser provides a managed browser runtime that can supply rendered pages and screenshots to visual browser-agent workflows.

Set-of-Mark Prompting Defined

Set-of-Mark prompting is a visual prompting method that places distinct, speakable marks—usually numbers or letters—over candidate regions in an image. The multimodal model sees the annotated image and answers with the mark that corresponds to the requested object. The method was introduced in the paper Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V, where off-the-shelf segmentation models partition an image and the overlay makes each region easy to name.

The key problem is visual grounding. A model may understand the sentence “click the account menu” and may recognize the menu in a screenshot, yet still need a reliable way to express where the target sits. Natural language such as “the icon near the upper-right corner” becomes ambiguous when several icons are close together. A marker creates a temporary identifier: the model can return “17,” and the controller can associate 17 with the stored region geometry.

SoM does not change the underlying page, train a new model, or define an automation protocol. It changes the observation shown to the model. That distinction matters because teams can add or remove the overlay at inference time. The original research presents SoM as a training-free prompting method, and the public Microsoft SoM implementation shows the separate stages used to create masks and render marks.

A marked screenshot is therefore a small interface between computer vision and language reasoning. The vision side proposes selectable regions. The rendering side assigns unique labels. The language model reasons over the picture and those labels. The action layer converts the selected label back into coordinates or a structured element reference. Each stage can be evaluated separately instead of treating the entire agent as one opaque result.

How Set-of-Mark Prompting Works

The first stage creates candidate regions. On a natural image, a segmentation model may produce object masks at several levels of detail. On a web page, candidate regions may come from detected controls, OCR boxes, accessibility nodes, DOM geometry, or a combination. The choice determines what the model is allowed to select. If the region generator misses a small disclosure arrow, no marker prompt can recover that target later.

The second stage filters and orders the regions. Overlapping masks can create several labels for the same object, while tiny decorative shapes can flood the screenshot with irrelevant numbers. Practical systems remove regions below a size threshold, merge duplicates, prioritize likely controls, and keep labels stable for the life of one observation. Stability reduces mistakes when the model refers to a label in a later reasoning step.

The third stage draws a high-contrast label near or inside each region. The label must remain legible without covering the very evidence needed to choose the object. A checkout button, for example, may contain price or state text that distinguishes it from another button. Placing a marker over that text trades spatial clarity for semantic loss. Good renderers choose an empty area, offset the label, or provide both original and marked images.

The final stage asks the model for a constrained answer and validates it. A controller might accept one marker identifier plus an action type, rather than an unrestricted paragraph. The selected identifier is checked against the current observation before any click occurs. If the page changed after the screenshot, the controller should take a fresh observation; a correct label from an old frame can point to the wrong control in the new state.

SoM Compared With Other Grounding Methods

Set-of-Mark prompting is easiest to understand when its output, dependencies, and failure modes are separated from neighboring techniques.

Where Set-of-Mark Prompting Helps

SoM earns its place when a task needs precise reference to visual regions and the page structure alone does not express enough meaning.

Designing a Reliable SoM Pipeline

Start with an explicit action inventory. A browsing system may need click, type, scroll, hover, and inspect actions, but not every marked region supports each action. Store the allowed actions beside the region identifier. A text field can accept typing; a paragraph may only support inspection. This simple contract prevents the model from selecting a visually plausible region for an impossible operation.

Keep the original observation alongside the marked version. The marked frame is useful for reference, while the original frame preserves typography, color, boundaries, and small status text. A two-view prompt lets the model reason from clean evidence and answer with the marked identifier. It also makes debugging easier because a reviewer can see whether the overlay obscured the target.

Connect every mark to provenance. For a web interface, the record may include screenshot bounds, viewport size, scroll offset, DOM node identity when available, accessible name, and the observation timestamp. The WebArena benchmark demonstrates why stateful browser tasks need reproducible environments and functional evaluation rather than attractive screenshots alone. Provenance lets a controller check that the selected region still belongs to the current page state.

Evaluate the pipeline by stage. Measure candidate recall before model choice accuracy: did the region generator mark the needed control at all? Then measure label legibility, model selection accuracy, action execution, and end-task correctness. A final success rate can hide two very different problems—poor visual grounding and correct grounding followed by a faulty action.

Limitations and Failure Modes

Dense pages can exceed the useful label budget. Hundreds of marks compete with page text and with one another. Filtering only by area is rarely enough because a small icon may be more actionable than a large decorative image. Interface role, visibility, occlusion, and likely task relevance provide better filters. Hierarchical marking is another option: select a panel first, then mark the controls inside it.

SoM inherits errors from the region proposal system. Segmentation can divide one control into several pieces or merge neighboring objects. DOM rectangles can include invisible layers. OCR can miss stylized text. Accessibility trees can omit canvas content. Combining sources improves coverage, but duplicate resolution must remain deterministic so the same visible control does not receive several competing labels.

A marker is valid only for the frame that produced it. Animation, responsive layout, lazy loading, and scrolling can move the target after observation. Browser-agent systems should bind a marker set to the page URL, viewport, scroll position, and a short-lived state identifier. High-impact actions deserve a fresh observation and an approval boundary even when the visual selection looks certain.

The technique also introduces accessibility and security questions. Overlays should not become part of the user-facing site, and sensitive page regions should be redacted before screenshots leave an approved boundary. Marker prompts are observations for a model, not authorization to submit a form, disclose credentials, purchase an item, or change account state.

How to Evaluate Set-of-Mark Prompting

Build an evaluation set with varied density, text size, themes, viewport widths, scroll positions, and interaction types. Record whether the target was proposed, whether its label remained readable, whether the model selected it, and whether the resulting action had the intended effect. This layered record shows where an improvement actually lands.

Compare SoM against a semantic baseline. If an accessibility-tree reference selects the same control with fewer tokens and fewer mistakes, the semantic path should win. If a canvas, chart, or image contains no dependable semantic node, the marked visual path may be the better observation. A hybrid agent can select the cheapest reliable representation per step.

Watch for label-position bias. Models may prefer early numbers, central regions, or visually prominent colors. Shuffle marker assignments across repeated examples while holding the screenshot fixed. A stable result should follow the object, not the identifier. Also test near-duplicate targets such as repeated product cards or identical toolbar icons, where spatial grounding does most of the work.

Report task impact separately from grounding accuracy. A higher selection score matters only if it reduces wrong actions or shortens the path to completion. Log the number of observations, model calls, corrected selections, stale-frame rejections, and human approvals. Those measures turn SoM from a clever overlay into an accountable component of an agent system.

Conclusion

Set-of-Mark prompting gives a multimodal model a compact language for pointing. It works by labeling proposed visual regions, asking the model to choose a label, and mapping that answer back to stored geometry. The technique is useful because it can be added at inference time and because its grounding decision is easy to inspect.

Its limits are equally concrete: missing regions stay missing, crowded labels hide evidence, and every mark expires when the screen changes. Teams get the best results by keeping clean and marked observations together, validating selections against current state, and comparing SoM with semantic locators instead of assuming a visual method must control every step.