Simulate,evaluate, and learn
A controllable and interactive simulation engine built for vision researchers.
What is LychSim
LychSim is a highly controllable, interactive simulation framework built on Unreal Engine 5, designed to lower the technical barrier of using a modern game engine for computer vision research.
Python and
Agentic Integration
One scene, three entry points. Script experiments through the Python API, hand the wheel to an LLM via MCP, or manage running instances from the CLI.
Streamlined Python API
A single LychSim class composes Camera, Object, and Data mixins. Capture RGB, depth, normals, segmentation, and point maps; query and manipulate actors; pause and resume the engine — all from native Python, no boilerplate.
from lychsim.api import LychSim
sim = LychSim(server_name="localhost", port=9000)
rgb = sim.get_cam_lit(cam_id=0)
depth = sim.get_cam_depth(cam_id=0)
annots = sim.get_obj_annots()
0+
Scenes annotated with procedural rules
0+
Objects with category,
size, color, and material annotations
Case Studies
How LychSim powers real research — from RL-driven model stress-testing to LLM-controlled scene composition.
Adversarial Examiners
Standard datasets cover only a narrow slice of the real-world parameter space. LychSim's controllable simulation lets an RL policy explore 3D camera viewpoints and scene configurations to surface vision-model failures — for example, minimizing Segment Anything's IoU on a target object exposes weaknesses on common objects in simple environments.
Interactive Scene Planning
The MCP integration turns LychSim into a closed-loop playground for language-driven scene layout. Agentic LLMs query scene state, plan layouts from natural language, place actors, and verify the result — all through the same standardized tool calls.
Public Release
We release the LychSim simulation framework, two Hugging Face datasets (object pose alignments and scene-level procedural rules), and the technical report describing the full system.
Works Built on LychSim
Research projects that build on LychSim's simulation framework.