language:
- en
license: apache-2.0
library_name: devision
pipeline_tag: visual-question-answering
base_model:
- convaiinnovations/laya
- google/siglip2-base-patch16-256
datasets:
- HuggingFaceM4/VQAv2
- lmms-lab/GQA
- HuggingFaceM4/A-OKVQA
- derek-thomas/ScienceQA
- HuggingFaceM4/the_cauldron
- cambridgeltl/vsr_random
- jxu124/objects365
- J1mb0o/e-SNLI-VE
- Vision-Flan/vision-flan_191-task_1k
- ryokamoi/VisOnlyQA_Train
- RyanWW/Super-CLEVR
- allenai/pixmo-count
tags:
- devision
- visual-decisions
- calibrated-probabilities
- jev
- system-one
- rlcd
- cpu
model-index:
- name: deVision v0.3
results:
- task:
type: visual-question-answering
dataset:
name: COCO object presence
type: test_exist
metrics:
- type: accuracy
value: 0.9473
- type: ece
value: 0.0341
- task:
type: visual-question-answering
dataset:
name: VQAv2 multiple choice
type: test_vqa_choice
metrics:
- type: accuracy
value: 0.9092
- type: ece
value: 0.0384
- task:
type: visual-question-answering
dataset:
name: COCO size
type: test_size
metrics:
- type: accuracy
value: 0.8718
- type: ece
value: 0.0295
- task:
type: visual-question-answering
dataset:
name: POPE, project filtered set
type: bench_pope
metrics:
- type: accuracy
value: 0.8496
- type: ece
value: 0.0393
- task:
type: visual-question-answering
dataset:
name: GQA val subset
type: test_gqa
metrics:
- type: accuracy
value: 0.7903
- type: ece
value: 0.0396
- task:
type: visual-question-answering
dataset:
name: COCO position
type: test_position
metrics:
- type: accuracy
value: 0.8846
- type: ece
value: 0.029
- task:
type: visual-question-answering
dataset:
name: VQAv2 yes/no subset
type: test_vqa_yesno
metrics:
- type: accuracy
value: 0.704
- type: ece
value: 0.0265
- task:
type: visual-question-answering
dataset:
name: COCO relative position
type: test_relation
metrics:
- type: accuracy
value: 0.7549
- type: ece
value: 0.0384
- task:
type: visual-question-answering
dataset:
name: VSR, project held-out split
type: test_vsr
metrics:
- type: accuracy
value: 0.6615
- type: ece
value: 0.0465
- task:
type: visual-question-answering
dataset:
name: Visual7W, project held-out split
type: test_v7w
metrics:
- type: accuracy
value: 0.721
- type: ece
value: 0.0411
- task:
type: visual-question-answering
dataset:
name: Fresh counting test (unseen pictures)
type: test_count_fresh
metrics:
- type: accuracy
value: 0.7633
- type: ece
value: 0.054
- task:
type: visual-question-answering
dataset:
name: CountBench, 0–10 choices
type: bench_countbench
metrics:
- type: accuracy
value: 0.3564
- type: ece
value: 0.0523
- task:
type: visual-question-answering
dataset:
name: COCO mirrored position
type: test_position_flip
metrics:
- type: accuracy
value: 0.8469
- type: ece
value: 0.0444
- task:
type: visual-question-answering
dataset:
name: COCO mirrored relative position
type: test_relation_flip
metrics:
- type: accuracy
value: 0.6977
- type: ece
value: 0.03
- task:
type: visual-question-answering
dataset:
name: GQA binding test
type: test_bind_gqa
metrics:
- type: accuracy
value: 0.6765
- type: ece
value: 0.0277
- task:
type: visual-question-answering
dataset:
name: What'sUp
type: bench_whatsup
metrics:
- type: accuracy
value: 0.2866
- type: ece
value: 0.2447
deVision v0.3
Image + English questions → structured answers with calibrated probabilities. deVision pairs a SigLIP2 vision encoder with Laya's ModernBERT decision model. It scores the options of yes/no (noul) and multiple-choice (choice) questions instead of generating text, follows the Jev answer format, and runs without a GPU. Several questions about one image share a single image encoding.
Usage
pip install devision
import devision
model = devision.load("lukbit/devision") # latest release; device defaults to auto
result = model.predict(
image="photo.jpg", # a path, an http(s) URL, a data URI, base64, bytes or a PIL image
state="", # text context; "" when there is none
questions={
"has_fork": {"type": "noul", "instructions": "Is there a fork in the image?"},
"room": {"type": "choice", "instructions": "Which room is this?",
"criteria": {"kitchen": None, "bathroom": None, "bedroom": None}},
},
)
print(result["answers"]["has_fork"]["noul"]) # probability of yes
print(result["answers"]["room"]["probabilities"]) # one probability per option, summing to 1
HTTP server with a browser demo at http://127.0.0.1:8000:
devision-serve --checkpoint lukbit/devision --port 8000
curl -s http://127.0.0.1:8000/v1/systemone -H 'Content-Type: application/json' \
-d '{"image": "https://example.com/photo.jpg", "state": "", "questions": {"has_fork": {"type": "noul", "instructions": "Is there a fork in the image?"}}}'
Over HTTP, image is an http(s) URL, a data URI or base64; the server never reads its own files. Responses follow Jev; confidence for choice is (n * p_max - 1) / (n - 1). Invalid requests return 422 (devision.InvalidRequest in Python).
Good for: whether something is present and how many (ask counts as a choice), what kind of thing or scene, colours and materials, up/down and left/right in photos (as a choice). Not for: comparing two places on a diagram, chart or map; relative-position yes/no questions; small text; other languages or several images. Use the probabilities as thresholds and send uncertain cases to a person or a stronger model, after checking the thresholds on your own data.
Evaluation
Accuracy through decide with the fitted temperatures. Mismatched: the same questions with every picture swapped for an unrelated one. These are project evaluation subsets, not official leaderboard scores. Although test pictures are excluded from training, these sets have informed successive experiments and release decisions; they are not a fresh independent test. See the release limitations below.
| Set | Questions | Accuracy | Mismatched | NLL | ECE |
|---|---|---|---|---|---|
| COCO object presence | 1,120 | 0.947 | 0.504 | 0.163 | 0.034 |
| VQAv2 multiple choice | 1,420 | 0.909 | 0.380 | 0.271 | 0.038 |
| COCO size | 1,100 | 0.872 | 0.511 | 0.336 | 0.029 |
| POPE, project filtered set | 8,676 | 0.850 | 0.527 | 0.368 | 0.039 |
| GQA val subset | 992 | 0.790 | 0.550 | 0.410 | 0.040 |
| COCO position | 1,274 | 0.885 | 0.493 | 0.288 | 0.029 |
| VQAv2 yes/no subset | 1,000 | 0.704 | 0.512 | 0.574 | 0.026 |
| COCO relative position | 1,950 | 0.755 | 0.496 | 0.490 | 0.038 |
| VSR, project held-out split | 904 | 0.661 | 0.487 | 0.620 | 0.046 |
| Visual7W, project held-out split | 1,000 | 0.721 | 0.419 | 0.673 | 0.041 |
| Fresh counting test (unseen pictures) | 600 | 0.763 | 0.405 | 0.574 | 0.054 |
| CountBench, 0–10 choices | 491 | 0.356 | 0.104 | 1.602 | 0.052 |
| COCO mirrored position | 640 | 0.847 | 0.477 | 0.359 | 0.044 |
| COCO mirrored relative position | 1,492 | 0.698 | 0.507 | 0.545 | 0.030 |
| GQA binding test | 1,484 | 0.676 | 0.498 | 0.559 | 0.028 |
| What'sUp | 820 | 0.287 | 0.245 | 1.491 | 0.245 |
Against Laya Vision 201M on the same questions (its published per-question predictions; difference in points, 95% interval from paired resampling by picture):
| Set | Questions | Laya Vision | deVision | Difference |
|---|---|---|---|---|
| VQAv2 yes/no | 4,887 | 0.717 | 0.725 | +0.8 [-0.8, +2.4] |
| A-OKVQA | 1,138 | 0.598 | 0.613 | +1.4 [-1.8, +4.7] |
| ScienceQA with images | 2,097 | 0.824 | 0.789 | -3.5 [-5.6, -1.4] |
| ScienceQA natural science, needs the picture | 323 | 0.700 | 0.495 | -20.4 [-27.2, -13.3] |
On the full POPE random / popular / adversarial sets (3,000 questions each) deVision scores 0.880 / 0.859 / 0.805, against Laya Vision's published 0.836 / 0.819 / 0.777 (aggregate scores only, not paired).
CPU latency: P50 175 ms, P95 190 ms (Darwin arm64, 4 threads, FP32, one question per request, warm-up excluded).
Model
| Vision | SigLIP2-B/16, 256 × 256 letterboxed input, 64 visual tokens after a 2 × 2 merge and an MLP projector; training policy described below |
| Decision | Laya-initialised ModernBERT-large and Laya's decision head; each option is scored at its [MASK], then a temperature-scaled softmax |
| Size | About 518M parameters, FP32, 2.07 GB (LoRA merged) |
This release continues v0.2 with 81,875 questions of counting, object presence, mirrored spatial questions, VSR and broad replay, for one 10,235-step schedule (seed 1). The first 2,000 steps were run as exp19; r15 resumes that exact training state. SigLIP2's top four encoder layers and final LayerNorm were fine-tuned at 5e-6 alongside the projector, decision head and ModernBERT LoRA. LoRA is merged into the released weights. The input remains 256×256 with 64 visual tokens; S2 question conditioning is not enabled.
Temperatures were fitted on the designated calibration set, by question type and option-count bucket. Missing buckets fall back to the question-type temperature:
| Bucket | Temperature |
|---|---|
| Choice, 11+ options | 1.2012 |
| Choice, 2 options | 1.9488 |
| Choice, 3–5 options | 1.7575 |
| Noul (yes / no) | 1.8258 |
| Choice, any other count | 1.5922 |
The owner adopted v0.3 as a trade-off: counting, relations and ScienceQA improved, while four old-capability protection conditions were not met. The original automated verdict remains candidate=false. See the adoption decision and v0.2 comparison, original acceptance report and training configuration.
Limitations
- English, one image, yes/no and choice questions only.
scoreis not implemented. At 256×256, small text and fine diagram details are hard to read. - CountBench improved by 16.7 points over v0.2, but absolute accuracy is only 35.6%. Answer-order sensitivity increased (39.5% change their answer when options are reversed, versus 16.9%); this is not a general-purpose accurate counter.
- VSR accuracy is 1.8 points lower and A-OKVQA 1.3 points lower than v0.2; their overall intervals contain zero, but non-inferiority was not established. VSR positive recall fell by 5.3 points. A-OKVQA and What'sUp have worse probability quality than v0.2. No prewritten NLL warning was triggered; that does not mean all probabilities improved.
- Diagram, chart and map comparisons and question-to-region binding remain weak. On natural-science ScienceQA questions that require the picture, accuracy is 49.5% versus Laya Vision's 70.0%.
- Calibration reflects the calibration distribution, not every deployment domain. Validate thresholds on your own examples.
- Test sets have repeatedly informed experiment and release decisions. This is one training seed, not a fresh independent confirmation. Gains cannot be attributed solely to unfreezing the vision encoder.
Licence
Weights and code: Apache-2.0, like the three base models (Laya, SigLIP2, ModernBERT). The training datasets (listed in this card's metadata) have their own terms, some non-commercial (e.g. ScienceQA, CC BY-NC-SA 4.0); check them against your use.
Links
Code and training records (this release: tag model-v0.3) · evaluation details · provenance · Laya Vision