Zhejiang University

AI4GC Lab

We build efficient multimodal models and computer-use agents—and the systems work to run them outside the lab.

AI4GC Lab mark

Who we are

Welcome to AI4GC Lab at Zhejiang University, led by Shengyu Zhang. We build efficient, deployable multimodal AI—compact multimodal LLMs and computer-use agents, accelerated image and video generation, and the device–cloud systems that carry them out of the datacenter and onto real phones and computers. Today's foundation models are remarkably capable yet costly in computation, memory, and energy, which keeps much of their power locked inside the cloud.

Our research closes that gap from both ends. We fine-tune small multimodal models and GUI agents to reason and act reliably under tight budgets; we attack the redundancy inside generation and the KV cache to make inference faster without sacrificing quality; and we design large–small collaboration so that powerful cloud models and constrained on-device models share intelligence and adapt to each user in real time. Alongside this, we take agent safety and evaluation seriously, stress-testing systems before they are trusted in the real world. We publish at leading venues including CVPR, ICML, ICLR, and ACL, and work with industry partners to put these ideas into production—aiming to make multimodal AI faster, more trustworthy, and accessible everywhere it is needed.

Research directions

  • Fine-tuning Small MLLMs / Agents

    How small can a multimodal agent be and still reason, reflect, and act reliably on real interfaces? Our InfiGUI line turns compact GUI agents from reactive responders into deliberative reasoners — InfiGUIAgent is a generalist GUI agent with native reasoning and reflection, InfiGUI-R1 learns step-level planning and recovery through reinforcement learning, and InfiGUI-G1 sharpens grounding via adaptive exploration. We also let small models improve themselves without collapsing, through adaptive policy optimization (AdaPO) and proposer–critic co-evolution (Propose-then-Critic).

  • Efficient Text-to-Image / Video

    Visual generation is expensive — can we make it faster without visibly trading away quality? We target the redundancy inside the generation process rather than retraining the model: CIAR decodes images collaboratively across intervals, and NaviCache caches and self-calibrates video features at test time. We pair this with the data side of controllable editing through the reasoning-enriched UnicEdit-10M benchmark.

  • MLLM KV-cache Optimization

    Long multimodal contexts make the KV cache the real memory bottleneck — but not every token deserves to stay. Our idea is to decide what to evict by reading modality and uncertainty, not raw attention scores: MadaKV is modality-aware, AccKV cross-calibrates across audio and video, and RetentiveKV adds a state-space memory for uncertainty-aware eviction.

  • On-device Computer Use

    We want agents that genuinely operate phones and computers under on-device limits — and a way to know when to trust them. EcoAgent divides work between device and cloud to stay efficient, while Graph2Eval and AEIA stress-test such agents at scale, including against active environmental injection attacks.

  • Large-small (Device-cloud) Model Collaboration

    How should a powerful cloud model and a constrained on-device model share intelligence — and keep up with each user in real time? We migrate knowledge across otherwise-incompatible architectures (MergeNet, CKI) and adapt on-device recommenders to drift and personalization (Forward-OFA, Persona), letting large and small models correct one another (CoCorrRec, LSC4Rec).

News

ACM Multimedia 2026 logoPaper

Three papers accepted to ACM MM 2026

VertiKV, PruneVAR, and SPECTRA are accepted to ACM MM 2026 — on vertical-integrity KV-cache compression for efficient multimodal long-context inference, training-free acceleration of visual autoregressive generation, and autonomous edge–cloud GUI grounding on device, respectively.

GUI-Anything sidecar observer for long Claude Code sessionsOpen Source

GUI-Anything is open-source — a flight recorder for long Claude Code sessions

We open-sourced GUI-Anything, a sidecar Flow Observer for Claude Code that watches live session streams, visualizes explorations as timelines and flowcharts, and brings local wiki context back on demand without interrupting the coding flow.

DeskCraft desktop GUI agent benchmark overviewOpen Source

DeskCraft is open-source — a benchmark for desktop GUI agents

DeskCraft puts desktop GUI agents to work on real professional workflows — 538 executable tasks spanning office, design, video, audio, 3D, and multi-app software, with human-in-the-loop collaboration. Code and paper are out now.

paper2anything — one paper PDF turned into slides, a poster, a webpage, and social-media coversOpen Source

paper2anything is open-source — turn any paper PDF into slides, a poster, a webpage, or social posts

We open-sourced paper2anything, an agent skills pack that turns an academic paper PDF into a presentation deck, conference poster, project homepage, Xiaohongshu post, or WeChat article — each generated end-to-end and ready to share.

Projects

  • GUI-Anything sidecar observer for long Claude Code sessions

    GUI-Anything

    2026–present

    A sidecar Flow Observer for long Claude Code sessions. Claude Code stays native in the left pane while the right pane maps session streams into timelines, flowcharts, summaries, and reusable local wiki memory.

  • paper2anything — one paper PDF turned into slides, a poster, a webpage, and social-media covers

    paper2anything

    2026–present

    An agent skills pack that turns an academic paper PDF into a presentation deck, conference poster, project homepage, Xiaohongshu post, or WeChat article — each generated end-to-end and ready to share.

  • Repo-Docs skill for readable, understandable project code

    Repo-Docs

    2026–present

    A coding-agent skill that makes fast-moving repos easier to read and understand. It writes and syncs repo-docs/ with the codebase—walkthroughs, concept pages, glossary, change log, and agent rules—so readers can follow one real path through the code and grasp how it works.