Zhejiang University

AI4GC Lab

We build efficient multimodal models and computer-use agents—and the systems work to run them outside the lab.

AI4GC Lab mark

Who we are

Welcome to AI4GC Lab at Zhejiang University, led by Shengyu Zhang. We build efficient, deployable multimodal AI—compact multimodal LLMs and computer-use agents, accelerated image and video generation, and the device–cloud systems that carry them out of the datacenter and onto real phones and computers. Today's foundation models are remarkably capable yet costly in computation, memory, and energy, which keeps much of their power locked inside the cloud.

Our research closes that gap from both ends. We fine-tune small multimodal models and GUI agents to reason and act reliably under tight budgets; we attack the redundancy inside generation and the KV cache to make inference faster without sacrificing quality; and we design large–small collaboration so that powerful cloud models and constrained on-device models share intelligence and adapt to each user in real time. Alongside this, we take agent safety and evaluation seriously, stress-testing systems before they are trusted in the real world. We publish at leading venues including CVPR, ICML, ICLR, and ACL, and work with industry partners to put these ideas into production—aiming to make multimodal AI faster, more trustworthy, and accessible everywhere it is needed.

Research directions

  • Fine-tuning Small MLLMs / Agents

    How small can a multimodal agent be and still reason, reflect, and act reliably on real interfaces? Our InfiGUI line turns compact GUI agents from reactive responders into deliberative reasoners — InfiGUIAgent is a generalist GUI agent with native reasoning and reflection, InfiGUI-R1 learns step-level planning and recovery through reinforcement learning, and InfiGUI-G1 sharpens grounding via adaptive exploration. We also let small models improve themselves without collapsing, through adaptive policy optimization (AdaPO) and proposer–critic co-evolution (Propose-then-Critic).

  • Efficient Text-to-Image / Video

    Visual generation is expensive — can we make it faster without visibly trading away quality? We target the redundancy inside the generation process rather than retraining the model: CIAR decodes images collaboratively across intervals, and NaviCache caches and self-calibrates video features at test time. We pair this with the data side of controllable editing through the reasoning-enriched UnicEdit-10M benchmark.

  • MLLM KV-cache Optimization

    Long multimodal contexts make the KV cache the real memory bottleneck — but not every token deserves to stay. Our idea is to decide what to evict by reading modality and uncertainty, not raw attention scores: MadaKV is modality-aware, AccKV cross-calibrates across audio and video, and RetentiveKV adds a state-space memory for uncertainty-aware eviction.

  • On-device Computer Use

    We want agents that genuinely operate phones and computers under on-device limits — and a way to know when to trust them. EcoAgent divides work between device and cloud to stay efficient, while Graph2Eval and AEIA stress-test such agents at scale, including against active environmental injection attacks.

  • Large-small (Device-cloud) Model Collaboration

    How should a powerful cloud model and a constrained on-device model share intelligence — and keep up with each user in real time? We migrate knowledge across otherwise-incompatible architectures (MergeNet, CKI) and adapt on-device recommenders to drift and personalization (Forward-OFA, Persona), letting large and small models correct one another (CoCorrRec, LSC4Rec).

News

Agentic Visual Generation: A Survey is now available on Preprints

Agentic Visual Generation: A Survey is now available as a public preprint.

Invited to serve as an Area Chair for ICLR 2027

FlowMM framework for information-flow-guided merging and sensitivity-adaptive token matchingPaper

FlowMM accepted to EMNLP 2026 Main

FlowMM adapts KV-cache merging to layer-wise cross-modal information flow and token sensitivity, reducing KV-cache memory by 80–95% and delivering 1.3–1.8× decoding speedup while preserving multimodal context performance.

Projects

  • Agentic visual generation architecture with planning, tools, memory, perception, action, and self-improvement

    Agentic Visual Generation: A Survey

    2026–present

    A public survey on agentic visual generation, with the full manuscript hosted on Preprints and the source repository kept in sync with the lab website.

  • GUI-Anything sidecar observer for long Claude Code sessions

    GUI-Anything

    2026–present

    A sidecar Flow Observer for long Claude Code sessions. Claude Code stays native in the left pane while the right pane maps session streams into timelines, flowcharts, summaries, and reusable local wiki memory.

  • paper2anything — one paper PDF turned into slides, a poster, a webpage, and social-media covers

    paper2anything

    2026–present

    An agent skills pack that turns an academic paper PDF into a presentation deck, conference poster, project homepage, Xiaohongshu post, or WeChat article — each generated end-to-end and ready to share.