Charts, tables and documents
Ask about what an image of a chart, a table or a page shows.
Any image can go into the state: a chart, a table, a scanned page, a photo. The model does not transcribe it; it answers questions about it, with the answers you supply.
The examples use the image() and ask() helpers from Reading screens.
Read a chart
answers = ask(
state={"chart": image("revenue_by_region.png")},
questions={
"largest": {
"type": "choice",
"instructions": "Which region has the highest revenue in the chart?",
"criteria": {"north": "North", "south": "South", "east": "East", "west": "West"},
},
"growing": {
"type": "noul",
"instructions": "Does total revenue increase from the first period to the last?",
},
},
)Compare two images
answers = ask(
state={
"this_week": image("dashboard_this_week.png"),
"last_week": image("dashboard_last_week.png"),
},
questions={
"worse": {
"type": "noul",
"instructions": "Is the error rate in `this_week` visibly higher than in `last_week`?",
},
},
)Look something up in a table
answers = ask(
state={"price_list": image("price_list.png"), "plan": "Team", "seats": 12},
questions={
"tier": {
"type": "choice",
"instructions": "Which row of `price_list` applies to this plan and number of seats?",
"criteria": {
"row_1": "Team, 1 to 5 seats",
"row_2": "Team, 6 to 20 seats",
"row_3": "Team, more than 20 seats",
"none": "No row applies.",
},
},
},
)Check a document
A page as an image, and the facts you want to verify as questions.
answers = ask(
state={
"invoice": image("invoice_page_1.jpg", "image/jpeg"),
"expected_vendor": "Northwind Traders",
},
questions={
"is_invoice": {"type": "noul", "instructions": "Is this document an invoice?"},
"vendor_matches": {"type": "noul", "instructions": "Is the invoice issued by `expected_vendor`?"},
"signed": {"type": "noul", "instructions": "Does the page carry a signature or a stamp?"},
"kind": {
"type": "choice",
"instructions": "What kind of document is this?",
"criteria": {
"invoice": "A request for payment with line items and a total.",
"receipt": "A confirmation that a payment was made.",
"contract": "An agreement with terms and signatures.",
"other": "Anything else.",
},
},
},
)
needs_review = (
answers["is_invoice"]["noul"] < 0.9
or answers["vendor_matches"]["noul"] < 0.9
or answers["kind"]["confidence"] < 0.8
)What this is not
- Not OCR. The model cannot return the text of a page or the value of a cell. If you need the invoice total as
a number, extract it with an OCR or a generative model; Instinct One can then check it ("Is the total on the
page
total?"). - Not a calculator. "Is the sum of the column above 10,000?" needs arithmetic the model is not reliable at. Ask what can be seen.
- Multi-page documents: up to 8 pages fit into one state as 8 images. Past that, ask per page and combine the answers.
Small print needs pixels. If answers about fine detail are unsure, crop to the relevant part of the page instead of sending the whole page scaled down.