About
Shiyu Hu (胡世宇)
AI is moving from answering single questions to carrying out sustained tasks in open environments: multimodal large models are asked to understand ever longer videos, LLM-based agents to act as people and assist them over long interactions, and unmanned systems to keep perceiving under real changes in lighting and motion. An increasingly discussed bottleneck is that such systems often get the current step right without necessarily keeping the state they should keep. In my work, this shows up in several ways: a tracker often struggles to find its target again after the target leaves the view; a video model can answer correctly while its opening description of the scene no longer holds; and a virtual student played by a large language model may sound like a learner without behaving like one. My research studies the dynamic states of humans and AI in open environments: what a system should keep, update, or leave open as conditions change, time passes, and people take part. Using human performance and judgment as reference, I evaluate where these states hold and where they break, model systems that maintain them better, and test them in open-environment visual perception (such as UAV and nighttime tracking), long and streaming video understanding, and education settings where people and AI work together.
News
2026.10 One paper (Hyper-Rationality) has been accepted by the IEEE Transactions on Network Science and Engineering (TNSE, JCR Q1, IF 7.3).
2026.10 One paper (GHASS) has been accepted by Chaos, Solitons & Fractals (CSF, JCR Q1, IF 5.7).
2026.09 I am honored to be invited to give a talk, From Object Tracking to Process Understanding: State Modeling in Dynamic Vision, at the International Conference on Image and Graphics (ICIG 2026) (CCF-C Conference) on 4 October 2026 in Singapore, as part of the forum Visual Intelligence in Transition: From Physical Perception to Psychological Cognition.
2026.09 I am honored to be invited to give a talk, Studying the Human Mind with AI: From Behavioral Analysis to Role Simulation, in the CLA Invited Academic Talks series at the College of Liberal Arts, Wenzhou-Kean University, in November 2026 in Wenzhou, China.
2026.09 The second edition of our Electronics Special Issue on Techniques and Applications of Multimodal Data Fusion is now open. Thank you to everyone who supported the first edition.
2026.09 One paper (META-PAP) has been accepted by the 40th Conference on Neural Information Processing Systems (NeurIPS, CCF-A Conference).
2026.09 I am co-hosting a Virtual Collection on Control and Decision Intelligence for Unmanned Systems in The Innovation Informatics. Submissions are welcome until 30 June 2027.
2026.09 I am pleased to serve as an Area Chair for the 15th International Conference on Learning Representations (ICLR). I look forward to contributing to the community and supporting a thoughtful peer-review process.
2026.08 I will serve as a Publicity Chair for the 2026 CSIG Annual Conference on Video and Image Security, to be held on 21 November 2026 in Xiong’an, China. Further information will be shared as it becomes available.
2026.08 Two papers (FNC and ViEBench) have been accepted by EMNLP 2026 (CCF-B Conference), including one main-conference paper and one Findings paper.
2026.06 One paper (DASTrack) has been accepted by the 2026 European Conference on Computer Vision (ECCV, CCF-B Conference).
2026.05 I am glad to have the opportunity to give a talk at the Chinese Congress on Image and Graphics (CCIG 2026) in Guangzhou, China. Many thanks to the special session Intelligent Evolution of Video and Image Security: Perception, Reasoning, and Adversarial Challenges. I sincerely look forward to exchanging ideas with everyone and hearing your valuable suggestions.
2026.05 I received a Reviewer Award from the 43rd International Conference on Machine Learning (ICML 2026).
2026.04 One paper (RGRL) has been accepted by the main conference of the 64th Annual Meeting of the Association for Computational Linguistics (ACL, CCF-A Conference).
2026.04 One research paper (Retinal OCT) has been accepted by the 48th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC, CAAI-B Conference).
2026.04 One paper (COAL) has been accepted by the 35th International Joint Conference on Artificial Intelligence (IJCAI, CCF-B Conference).
2026.03 One paper (TBDQ) has been accepted by the Pattern Recognition (PR, CCF-B Journal, JCR Q1, IF 9.1).
2026.02 One paper (EARL) has been accepted by the 2026 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR, CCF-A Conference).
2026.02 One paper (MATrack) has been accepted by the 2026 IEEE International Conference on Robotics & Automation (ICRA, CCF-B Conference).
2026.02 One paper (DAAWBench) has been accepted by the IEEE Transactions on Circuits and Systems for Video Technology (TCSVT, CCF-B Journal, JCR Q1, IF 10.8).
2026.02 One review paper (Video intelligence) has been accepted by the IEEE Transactions on Network Science and Engineering (TNSE, JCR Q1, IF 7.3).
2026.01 I am honored to have participated in Humanity’s Last Exam (HLE), published in Nature (JCR Q1, IF 56.1), by submitting expert-level questions related to AI as a member of the HLE Contributors Consortium.
2026.01 We presented two main-conference Oral papers (CausalStep Slides and VerifyBench Slides) and three workshop papers (SOEI Slides, EduVerse Slides, and EduPersona Slides) at AAAI 2026. Thanks to everyone who visited us at the Singapore EXPO for the discussions.
2026.01 One paper (NarrLV) has been accepted by the 14th International Conference on Learning Representations (ICLR, CCF-A Conference).
2025.12 I was invited to share my research experience as a UCAS alum at the Artificial Intelligence Sub-forum of the University of Chinese Academy of Sciences 2025 Graduate Academic Forum (13th December, 2025, Beijing).
2025.12 We will conduct a Mini-Symposium (topic: Complex Network Systems and Large Language Models) on NODYCON 2026 (The Fifth International Nonlinear Dynamics Conference), more information will be released soon.
2025.11 Three papers have been accepted by the AI for Education Workshop in the 40th Annual AAAI Conference on Artificial Intelligence (AAAIW, Workshop in CCF-A Conference, Oral).
2025.11 Two papers (CausalStep and VerifyBench) have been accepted by the 40th Annual AAAI Conference on Artificial Intelligence (AAAI, CCF-A Conference, Oral).
2025.10 We have conducted a tutorial at 28th European Conference on Artificial Intelligence (ECAI) (26th October, 2025, Bologna, Italy).
2025.10 We have conducted a tutorial at 2025 IEEE International Conference on Systems, Man, and Cybernetics (SMC) (5th October, 2025, Vienna, Austria).
2025.08 I gave an invited talk, Human-Centric Multimodal Visual Intelligence: From Benchmarks to Human-Like Modeling, at the Visual Intelligence International Postdoctoral (Doctoral) Academic Frontier Seminar (30th August, 2025, online).
2025.08 We have conducted a tutorial at 34th International Joint Conference on Artificial Intelligence (IJCAI) (18th August, 2025, Montreal, Canada).
2025.07 Obtain IEEE SMCS TEAM Program Award.
2025.06 One paper (ATCTrack) has been accepted by International Conference on Computer Vision (ICCV, CCF-A Conference, Highlight).
2025.05 One review paper (ALP) has been accepted by Computers and Education: Artificial Intelligence (C&E:AI, JCR Q1, IF 23.4).
2025.05 Our new work FIOVA is now online! We introduce a multi-annotator benchmark for human-aligned video captioning, supporting semantic diversity and cognitive-aware evaluation. Check out the project page and arXiv paper for more details.
2025.05 Our updated work SOEI now available! Building upon our previous framework, this version introduces interactive multi-turn simulation to model open-ended educational dialogues with cognitively plausible virtual students. We further validate the framework’s effectiveness through behavioral analysis, personality recognition, and teacher-student reflection. Read more in the arXiv paper.
2025.05 We will present our work (SOTVerse) at IJCV2024 during the VALSE2025 poster session (June 2025, Zhuhai, China).
2025.05 One paper (CSTrack) has been accepted by International Conference on Machine Learning (ICML, CCF-A Conference).
2025.05 One paper (DARTer) has been accepted by International Conference on Multimedia Retrieval (ICMR, CCF-B Conference).
2025.05 One paper (MSAD) has been accepted by IET Computer Vision (IET-CVI, CCF-C Journal, JCR Q4, IF 1.4).
2025.04 We will conduct a tutorial at 28th European Conference on Artificial Intelligence (ECAI) (25th-30th October, 2025, Bologna, Italy).
2025.03 We will conduct a tutorial at 2025 IEEE International Conference on Systems, Man, and Cybernetics (SMC) (5th-8th October, 2025, Vienna, Austria).
2025.02 The book Visual Object Tracking: An Evaluation Perspective is online.
2025.01 One paper (CTVLT) has been accepted by IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP, CCF-B Conference).
2025.01 A special issue (Techniques and Applications of Multimodal Data Fusion) in Electronics has been announced, all papers related to this topic are welcomed for submission!
2024.12 We have conducted a tutorial at Asian Conference on Computer Vision (ACCV) (Dec. 9th 2024, Hanoi, Vietnam).
2024.12 We have prepared a tutorial at International Conference on Pattern Recognition (ICPR) (Dec. 1st 2024, Kolkata, India).
2024.10 We have conducted a tutorial at IEEE International Conference on Image Processing (ICIP) (Oct. 27th 2024, Abu Dhabi, United Arab Emirates).
2024.09 Two papers (MemVLT and CPDTrack) have been accepted by Conference on Neural Information Processing Systems (NeurIPS, CCF-A Conference).
2024.08 One tutorial proposal has been accepted by Asian Conference on Computer Vision (ACCV), the tutorial will be conducted in Dec. 2024 (Hanoi, Vietnam).
2024.08 Start my work as a Research Fellow in Nanyang Technological University (NTU), Singapore.
2024.07 One tutorial proposal has been accepted by International Conference on Pattern Recognition (ICPR), the tutorial will be conducted in Dec. 2024 (Kolkata, India).
2024.06 One paper (VS-LLM) has been accepted by Chinese Conference on Pattern Recognition and Computer Vision (PRCV, CCF-C Conference).
2024.06 One paper (IGBA) has been accepted by Chinese Mental Health Journal (《中国心理卫生杂志》).
2024.05 Obtain Beijing Outstanding Graduates (北京市优秀毕业生, top 5%).
2024.05 We have presented our work (GIT) at TPAMI2023 during the VALSE2024 poster session (May 2024, Chongqing, China, see our Poster for more information).
2024.04 One paper (DTLLM-VLT) has been accepted by the 3rd Workshop on Vision Datasets Understanding and DataCV Challenge in CVPR 2024 (CVPRW, Workshop in CCF-A Conference, Oral, Best Paper Honorable Mention).
2024.04 One paper (AWCV-100K) has been accepted by IEEE Transactions on Circuits and Systems for Video Technology (TCSVT, CCF-B Journal, JCR Q1, IF 10.8).
2024.01 One project about human-computer interaction in intelligent education has been funded by the 2023 Intelligent Education PhD Research Fund, supported by the Institute of AI Education Shanghai and East China Normal University.
2024.01 Got my Ph.D. degree at Institute of Automation, Chinese Academy of Sciences (CASIA) and University of Chinese Academy of Sciences (UCAS).
2023.12 One paper (Cryo-EM) has been accepted by the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP, CCF-B Conference).
2023.11 One paper (SOI2023) has been accepted by International Conference on Computer Science and Artificial Intelligence (CSAI, EI Conference, Oral).
2023.10 Obtain China National Scholarship (国家奖学金, top 1%, only 8 Ph.D. students in main campus of UCAS win this scholarship).
2023.10 Obtain First Prize of Climbing Scholarship (攀登一等奖学金, only 6 students in CASIA win this scholarship).
2023.10 One paper (BioDrone) has been accepted by International Journal of Computer Vision (IJCV, CCF-A Journal, JCR Q1, IF 10.3).
2023.09 One survey (SOT survey) has been accepted by Journal of Image and Graphics (《中国图象图形学报》).
2023.09 One paper (MGIT) has been accepted by Conference on Neural Information Processing Systems (NeurIPS, CCF-A Conference).
2023.09 One paper (SOTVerse) has been accepted by International Journal of Computer Vision (IJCV, CCF-A Journal, JCR Q1, IF 10.3).
2023.08 One paper (HIST) has been accepted by Chinese Conference on Pattern Recognition and Computer Vision (PRCV, CCF-C Conference).
2022.06 Obtain merit student of University of Chinese Academy of Sciences.
2022.06 One paper (INS) has been accepted by Neurocomputing (CCF-C Journal, JCR Q1, IF 6.7).
2022.02 One paper (GIT) has been accepted by IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI, CCF-A Journal, JCR Q1, IF 20.4).
2021.06 One survey (Visual Turing) has been accepted by Journal of Graphics (《图学学报》).
Research
The dynamic states of humans and AI in open environments. From giving an answer to keeping the right state.
Research Framework
In open environments, a system's state is challenged in three ways: conditions change, time passes, and people take part. Each gives one line of work, and each line runs from evaluation to modeling to application.
PaperSurvey or perspectiveBookDashed border: preprint
Open environments
Across Conditions
When the conditions change, does the capability still hold?
A tracker that works well by day may fail at night or on a shaking drone, and an average score does not show where. I break tasks and environments into conditions that can be recombined, add human baselines, and test whether a capability measured under some conditions predicts behavior under others.
SOTVerseBioDroneVLTVerseSTEMVerseSOT surveyMATrackDARTerMonographTBDQHLEINSDAAWBenchAWCV-100KMSADRetinal OCTCryo-EM
Sustained interaction
Over Time
When the answer is right, has the system kept the right state?
In long and streaming video, and in the multimodal models that now process it, a target can leave the view and return, a description given at the start can go stale, and similar objects can pull identity away. I describe state by three rules: keep what still holds, update what has changed, and leave open what the evidence cannot support.
GITMGITATCTrackSOIStreaming surveyEARLRGRLCausalStepVerifyBenchViEBenchNarrLVMemVLTCSTrackCTVLTDTVLTVLT-MIDTLLM-VLTCOALDASTrackVideo intelligenceSOI2023
Human–AI interaction
Humans and AI
When people disagree, or work with AI, what counts as correct?
When people disagree, a single ground truth is often not the right reference. I treat human judgments as a reference distribution, compare people and models on the same tasks and the same scale, and model LLM-based agents that simulate learners, asking whether they behave like learners rather than only sound like them.
FIOVANearing or SurpassingEduPersonaSOEICPDTrackVisual TuringEduVerseHyper-RationalityCalcTutorALPAGSTrustworthy AIEdGHASSIGBAVS-LLMHISTFNCMETA-PAP
In every line, against human reference
- Evaluatewhere does the state hold, and where and why does it break?
- Modelhow can a system keep, update, or leave it open?
- Applydoes it hold up in open-environment visual perception, long and streaming video understanding, and human–AI learning?
Where It Started
My research began with visual object tracking and machine vision evaluation. Three questions from that work still shape it: how a task should be defined, how an evaluation space should be built, and how machines compare with people.
Task definition: from local tracking to global instance tracking
Visual object tracking gave a concrete task for studying dynamic visual ability. Global Instance Tracking (GIT) removes the assumption of continuous motion and asks a system to find the same target across a long video, while Multi-modal GIT (MGIT) adds hierarchical semantics and spatiotemporal-causal relations.
Evaluation space: from fixed benchmarks to conditions
Benchmarks usually test a tracker under one fixed mix of conditions. VideoCube organizes long videos through narrative structure, SOTVerse turns challenge factors into a user-defined task space built on the 3E framework (environment, evaluation, executor), and BioDrone examines perception under physical disturbance.
Human-machine comparison: people as the reference
Building on the idea of a visual Turing test, I placed people and trackers on the same dynamic visual tasks (human–machine dynamic vision) to see where conventional metrics hide the difference between them.
Each step asked what a system carries forward when something changes. The three lines above follow this question across conditions, over time, and between humans and AI.
Publications
Research Monograph
Visual Object Tracking: An Evaluation Perspective
X. Zhao, Shiyu Hu, X. Yin
Springer, Part of the book series: Advances in Computer Vision and Pattern Recognition (ACVPR)
Visual Object Tracking · Task–Environment–Executor Framework · Visual Turing Test
📘 Book
Peer-Reviewed Publications
Journal metrics: JCR 2026 quartiles and 2025 impact factors for every journal indexed in JCR; CCF-T1/T2/T3 are the CCF 2025 tiers for Chinese journals.
Lead or Corresponding Author
Global Instance Tracking: Locating Target More Like Humans
Shiyu Hu, X. Zhao, L. Huang, K. Huang
IEEE Transactions on Pattern Analysis and Machine Intelligence (CCF-A Journal, JCR Q1, IF 20.4)
Global Instance Tracking · Target Identity Across Shots · Human Visual Baselines
📃 Paper 📑 PDF 🪧 Poster 🌐 Platform 🔧 Toolkit 💾 Dataset
SOTVerse: A User-defined Task Space of Single Object Tracking
Shiyu Hu, X. Zhao, K. Huang
International Journal of Computer Vision (CCF-A Journal, JCR Q1, IF 10.3)
User-Defined Task Space · Challenge-Factor Conditions · Diagnostic Evaluation
📃 Paper 📑 PDF 🪧 Poster 🌐 Platform
BioDrone: A Bionic Drone-based Single Object Tracking Benchmark for Robust Vision
X. Zhao, Shiyu Hu✉️, Y. Wang, J. Zhang, Y. Hu, R. Liu, H. Lin, Y. Li, R. Li, K. Liu, J. Li
International Journal of Computer Vision (CCF-A Journal, JCR Q1, IF 10.3)
Bionic UAV Vision · Real Physical Disturbance · Robustness Benchmark
📃 Paper 🌐 Platform 📑 PDF 🔧 Toolkit 💾 Dataset
A Multi-modal Global Instance Tracking Benchmark (MGIT): Better Locating Target in Complex Spatio-temporal and causal Relationship
Shiyu Hu, D. Zhang, M. Wu, X. Feng, X. Li, X. Zhao, K. Huang
Conference on Neural Information Processing Systems (CCF-A Conference)
Long-Video Tracking · Multi-Granular Semantic Hierarchy · Spatiotemporal-Causal Reasoning
📃 Paper 📑 PDF 🪧 Poster 📹 Slides 🌐 Platform 🔧 Toolkit 💾 Dataset
ATCTrack: Aligning Target-Context Cues with Dynamic Target States for Robust Vision-Language Tracking
X. Feng*, Shiyu Hu*, X. Li, D. Zhang, M. Wu, J. Zhang, X. Chen, K. Huang (*Equal Contributions)
International Conference on Computer Vision (CCF-A Conference, Highlight)
Vision-Language Tracking · Dynamic Target States · Target-Context Cue Alignment
📃 Paper 📑 PDF
MATrack: Efficient Multiscale Adaptive Tracker for Real-Time Nighttime UAV Operations
X. Li*, X. Li*, Shiyu Hu✉️
International Conference on Robotics and Automation (CCF-B Conference)
Nighttime UAV Tracking · Multiscale Adaptation · Real-Time Deployment
📃 Paper 📑 PDF
DARTer: Dynamic Adaptive Representation Tracker for Nighttime UAV Tracking
X. Li*, X. Li*, Shiyu Hu✉️
International Conference on Multimedia Retrieval (CCF-B Conference)
Nighttime UAV Tracking · Dynamic Representation Adaptation · Efficient Deployment
📃 Paper 📑 PDF
Visual Intelligence Evaluation Techniques for Single Object Tracking: A Survey (单目标跟踪中的视觉智能评估技术综述)
Shiyu Hu, X. Zhao, K. Huang
Journal of Image and Graphics (《中国图象图形学报》, CCF-T2 Chinese Journal)
Visual Intelligence Evaluation · Task–Environment–Executor–Mechanism · Human-Machine Comparison
📃 Paper 📑 PDF
Improved SAR Aircraft Detection Algorithm Based on Visual State Space Models
Y. Wang, J. Zhang, Y. Wang, Shiyu Hu✉️, B. Shen, Z. Hou, W. Zhou
IET Computer Vision (CCF-C Journal, JCR Q4, IF 1.4)
Remote Sensing · SAR Aircraft Detection · Vision State-Space Models
📃 Paper
Selected lead or corresponding-author publications are shown for concise browsing.
Collaborative Work
META-PAP: Meta-learning for Prompt-aware Preference Pairing in LLM Alignment
P. Zhao, Shiyu Hu, J. Zhang, M. Li
Conference on Neural Information Processing Systems (CCF-A Conference)
LLM Alignment · Prompt-Aware Preference Pairing · Bilevel Meta-Reweighting
📃 Paper
NarrLV: Towards a Comprehensive Narrative-Centric Evaluation for Long Video Generation Models
X. Feng, H. Yu, M. Wu, Shiyu Hu, J. Chen, C. Zhu, J. Wu, X. Chu, K. Huang
International Conference on Learning Representations (CCF-A Conference)
Generative Video Models · Narrative Coherence · Narrative-Centric Evaluation
📃 Paper 📑 PDF
Tracking by Detection and Query: An Efficient End-to-End Framework for Multi-Object Tracking
S. Jia, Shiyu Hu, Y. Cao, F. Yang, X. Lu, X. Lu
Pattern Recognition (CCF-B Journal, JCR Q1, IF 9.1)
Multi-Object Tracking · Query-Based Association · Efficient End-to-End Learning
📃 Paper 📑 PDF
Talk with Your Fingers: A Depth-Aware Benchmark for Air-Writing Recognition
M. Wu, Y. Zhao, X. Li, Shiyu Hu, Y. Cai, J. Wu, W. Wang, K. Huang
IEEE Transactions on Circuits and Systems for Video Technology (CCF-B Journal, JCR Q1, IF 10.8)
Human-Centered Vision · Depth-Aware Air-Writing · Multimodal Benchmark
📃 Paper
COAL: Counterfactual and Observation-Enhanced Alignment Learning for Discriminative Referring Multi-Object Tracking
S. Jia, Shiyu Hu, Y. Wang, X. Cheng, Y. Cao, X. Lu
International Joint Conference on Artificial Intelligence (CCF-B Conference)
Referring Multi-Object Tracking · Robust Language Grounding · Counterfactual Alignment Learning
📃 Paper 📑 PDF
DASTrack: Rethinking Temporal Modeling in Visual Object Tracking via Decoupled Auxiliary Supervision
D. Zhang, Shiyu Hu, H. Fu, X. Feng, Y. Wang, KH Cheong, K. Huang
European Conference on Computer Vision (CCF-B Conference)
Visual Object Tracking · Temporal Representation Learning · Decoupled Auxiliary Supervision
📃 Paper
Fake News Court: A Multi-Agent Adversarial Framework for Robust Detection of LLM-Generated Fake News
M. Peng, H. Gu, Y. Ma, Shiyu Hu, X. Huang
Conference on Empirical Methods in Natural Language Processing (CCF-B Conference)
LLM-Generated Fake News · Multi-Agent Adversarial Reasoning · Robust Detection
Beyond Accuracy: Evaluating Grounded Visual Evidence in Thinking with Images
X. Li*, X. Li*, R. Pi, Shiyu Hu, J. Zhao, J. Gao
Conference on Empirical Methods in Natural Language Processing (EMNLP Findings, CCF-B Conference)
Agentic Visual Reasoning · Grounded Visual Evidence · Dual-Axis Diagnosis
📃 Paper 📑 PDF
Global-Local Semi-Supervised Modeling for Retinal Layer Boundary Estimation in OCT
K. Li, B. Parikh, H. Yue, Shiyu Hu, S. W. Tan, W. Y. Low, X. Su, KH Cheong
Annual International Conference of the IEEE Engineering in Medicine and Biology Society (CAAI-B Conference)
Medical Imaging · Retinal Boundary Estimation · Semi-Supervised Structure Learning
Beyond Hyper-Rationality: Persona-Driven Heterogeneity in Networked Agent-Based Systems
S. Hirashima, Shiyu Hu, KH Cheong
IEEE Transactions on Network Science and Engineering (JCR Q1, IF 7.3)
LLM Agents · Persona-Driven Heterogeneity · Agent-Based Social Simulation
📃 Paper
From Observation to Generation: A Complex-Systems Perspective on Generative Human-AI Social Systems
Y. Ma*, M. Peng*, Shiyu Hu, B. Zhu, Y. Wang, Y. Chen, S. Liu, KH Cheong
Chaos, Solitons & Fractals (JCR Q1, IF 5.7)
Generative Human-AI Social Systems · Complex Systems · Collective Dynamics
CalcTutor: Multi-Agent LLM Grading of Handwritten Mathematics with RAG-Grounded Feedback for Adaptive Learning Support
L. Tan, B. Zhu, Shiyu Hu, A. Mishra, Darren J. Yeo, KH Cheong
Mathematics (JCR Q1, IF 2.3)
AI for Education · Multi-Agent Assessment · RAG-Grounded Feedback
📃 Paper
A benchmark of expert-level academic questions to assess AI capabilities
Center for AI Safety, Scale AI, and HLE Contributors Consortium
Nature, 649, 1139–1146 (2026) (JCR Q1, IF 56.1)
Contribution: Shiyu Hu submitted expert-level questions related to AI to the HLE benchmark as a member of the HLE Contributors Consortium.
Frontier AI Evaluation · Expert-Level Benchmarking · Consortium Contribution
📃 Paper 🌐 Benchmark
CSTrack: Enhancing RGB-X Tracking via Compact Spatiotemporal Features
X. Feng, D. Zhang, Shiyu Hu, X. Li, M. Wu, J. Zhang, X. Chen, K. Huang
International Conference on Machine Learning (CCF-A Conference)
Multimodal Tracking · Spatiotemporal Representation Learning · Efficient Sensor Fusion
📃 Paper 📑 PDF
Enhancing Vision-Language Tracking by Effectively Converting Textual Cues into Visual Cues
X. Feng, D. Zhang, Shiyu Hu, X. Li, M. Wu, J. Zhang, X. Chen, K. Huang
IEEE International Conference on Acoustics, Speech, and Signal Processing (CCF-B Conference)
Vision-Language Tracking · Visual Grounding · Foundation Model Transfer
📃 Paper 📑 PDF
Artificial Intelligence-Enabled Adaptive Learning Platforms: A Review
L. Tan, Shiyu Hu, Darren J. Yeo, KH Cheong
Computers & Education: Artificial Intelligence (JCR Q1, IF 23.4)
AI for Education · Personalized Learning · Adaptive Learning Systems
📃 Paper 📑 PDF
A Comprehensive Review on Automated Grading Systems in STEM Using AI Techniques
L. Tan, Shiyu Hu, Darren J. Yeo, KH Cheong
Mathematics (JCR Q1, IF 2.3)
AI for Education · Automated STEM Assessment · Learning Analytics
📃 Paper
Trustworthy AI in education: Framework, cases, and governance strategies
Y. Ma, X. Li, Shiyu Hu, S. Liu, KH Cheong
Innovation and Emerging Technologies (JCR Q2, IF 2.3)
Trustworthy AI · AI for Education · Governance and Fairness
📃 Paper
A Review of Intelligent Psychological Assessment Based on Interactive Environment (基于交互环境的智能化心理测评)
K. Huang, Y. Kang, C. Yan, Shiyu Hu, L. Wang, T. Tao, W. Gao
Chinese Mental Health Journal (《中国心理卫生杂志》, CSCD Core, PKU Core)
Game-Based Psychological Assessment · Interactive Environments · Validity and Ethics
Robust Single-particle Cryo-EM Image Denoising and Restoration
J. Zhang, T. Zhao, Shiyu Hu, X. Zhao
IEEE International Conference on Acoustics, Speech, and Signal Processing (CCF-B Conference)
Scientific Imaging · Cryo-EM Restoration · Diffusion Models
📃 Paper 📑 PDF
Finger in Camera Speaks Everything: Unconstrained Air-Writing for Real-World
M. Wu, K. Huang, Y. Cai, Shiyu Hu, Y. Zhao, W. Wang
IEEE Transactions on Circuits and Systems for Video Technology (CCF-B Journal, JCR Q1, IF 10.8)
Real-World Air-Writing · Unconstrained Recognition · Large-Scale Benchmark
📃 Paper 📑 PDF 🔧 Toolkit
VS-LLM: Visual-Semantic Depression Assessment based on LLM for Drawing Projection Test
M. Wu, Y. Kang, X. Li, Shiyu Hu, X. Chen, Y. kang, W. Wang, K. Huang
Chinese Conference on Pattern Recognition and Computer Vision (CCF-C Conference)
Drawing-Based Depression Assessment · Multimodal Mental Health · LLM-Assisted Interpretation
📃 Paper 📑 PDF
A Hierarchical Theme Recognition Model for Sandplay Therapy
X. Feng, Shiyu Hu, X. Chen, K. Huang
Chinese Conference on Pattern Recognition and Computer Vision (CCF-C Conference)
Sandplay Therapy · Hierarchical Theme Recognition · Knowledge-Guided Recognition
📃 Paper 📑 PDF 📎 Supplementary 🪧 Poster
Workshop
AAAIW 2026Learning to Be Taught: A Structured SOEI Framework for Modeling and Evaluating Personality-Aligned Virtual Student Agents, Y. Ma*, Shiyu Hu*, X. Li, Y. Wang, Y. Chen, S. Liu, KH Cheong (*Equal Contributions), the AI for Education Workshop in the 40th Annual AAAI Conference on Artificial Intelligence (Workshop in CCF-A Conference, Oral), 📹 SlidesAAAIW 2026Redefining Educational Simulation: EduVerse as a User-Defined and Developmental Multi-Agent Simulation Space, Y. Ma*, Shiyu Hu*, B. Zhu, Y. Wang, Y. Kang, S. Liu, KH Cheong (*Equal Contributions), the AI for Education Workshop in the 40th Annual AAAI Conference on Artificial Intelligence (Workshop in CCF-A Conference, Oral), 📹 SlidesAAAIW 2026From Objective to Subjective: A Benchmark for Virtual Student Abilities, B. Zhu*, Shiyu Hu*, Y. Ma, Y. Zhang, KH Cheong (*Equal Contributions), the AI for Education Workshop in the 40th Annual AAAI Conference on Artificial Intelligence (Workshop in CCF-A Conference, Oral), 📹 SlidesCVPRW 2024Diverse Text Generation for Visual Language Tracking Based on LLM, X. Li, X. Feng, Shiyu Hu, M. Wu, D. Zhang, J. Zhang, K. Huang, the 3rd Workshop on Vision Datasets Understanding and DataCV Challenge in CVPR 2024 (Workshop in CCF-A Conference, Oral, Best Paper Honorable Mention), 📃 Paper 📑 PDF 🪧 Poster 📹 Slides 🌐 Platform 🔧 Toolkit 💾 Dataset 🏆 Award
Selected collaborative publications are shown for concise browsing.
Preprints
When LLMs Learn to be Students: The SOEI Framework for Modeling and Evaluating Virtual Student Agents in Educational Interaction
Y. Ma*, Shiyu Hu*, X. Li, Y. Wang, Y. Chen, S. Liu, KH Cheong (*Equal Contributions)
Virtual Student Agents · Personality-Grounded Learner Modeling · Interaction-Centered Evaluation
📃 Paper 📑 PDF
SOI is the Root of All Evil: Quantifying and Breaking Similar Object Interference in Single Object Tracking
Y. Wang*, Shiyu Hu*, S. Jia, P. Xu, H. Ma, Y. Ma, J. Zhang, X. Lu, X. Zhao (*Equal Contributions)
Similar-Object Interference · Quantified Failure Analysis · VLM-Guided Correction
📃 Paper 📑 PDF
Selected preprints are shown for concise browsing.
Projects
The evaluation platforms below are the infrastructure behind the across-conditions and over-time lines: they make task spaces, long videos, and physical disturbance testable at scale. Funded projects, academic challenges, and earlier research software follow; associated research outputs are listed under Publications.
Research Platforms
Data snapshot Platform usage figures below are reported as of July 2026.
VideoCube / MGIT
Evaluation infrastructure for the Global Instance Tracking and Multi-modal Global Instance Tracking studies published at TPAMI 2023 and NeurIPS 2023. The platform supports dataset access, tracker registration, standardized submission, and reproducible evaluation.
SOTVerse / VLTVerse
A task-space evaluation platform for the SOTVerse and VLTVerse research program, including the study published in IJCV 2024. It provides structured evaluation across tracking scenarios, target categories, and linguistic specifications.
BioDrone
Benchmark and evaluation infrastructure for real-world drone tracking research, including the BioDrone study published in IJCV 2024. It supports dataset access, standardized benchmarking, and comparative evaluation.
GOT-10k
Long-term maintenance of the large-scale evaluation platform supporting the GOT-10k research published in TPAMI 2021. It provides benchmark access, tracker registration, result submission, and evaluation services.
Funded Research Projects
MOE2024-TRF-004
MOE2022-TRF-029
2023 Intelligent Education PhD Research Fund
Challenges
Hislopvision Challenge
Organization of the Hislopvision track for the 3rd High-speed and Low-power Visual Understanding Challenge at PRCV 2023, with participating teams from Tsinghua University, Beijing Institute of Technology, and Jilin University.
Cell Tracking Challenge
Our method ranked second on Fluo-C2FL-MSC+ and third on Fluo-C2FL-Huh7, based on the challenge rankings recorded in October 2023.
Research Software
Darknet-Cross
Lightweight Deep Learning Framework for Heterogeneous Computing
A cross-platform acceleration framework developed for Android and Ubuntu across mobile and desktop GPUs. This work formed the engineering component of my master's thesis at HKU.
Background
Professional Experience
Research Fellow
Nanyang Technological University (NTU)
School of Physical and Mathematical Sciences (SPMS)
Research: Computer Vision, Multimodal AI, and AI for Education
PI: Assoc. Prof. Kang Hao Cheong (IEEE Senior Member)
Research Assistant
Direction: High Performance Computing and Heterogeneous Computing
PI: Prof. Choli Wang
Research Intern
Institute of Electronics, Chinese Academy of Sciences (CASIE)
Education
Ph.D.
Institute of Automation, Chinese Academy of Sciences (CASIA)
Field: Computer Application Technology
Supervisor: Prof. Kaiqi Huang (IAPR Fellow, IEEE Senior Member)
Co-supervisor: Prof. Xin Zhao (IEEE Senior Member)
Thesis and degree details
Thesis: Research of Intelligence Evaluation Techniques for Single Object Tracking
Supervisor: Prof. Kaiqi Huang (IAPR Fellow, IEEE Senior Member, 10,000 Talents Program – Leading Talents)
Co-supervisor: Prof. Xin Zhao (IEEE Senior Member, Beijing Science Fund for Distinguished Young Scholars)
Thesis Committee: Prof. Jianbin Jiao; Prof. Yuxin Peng (National Science Fund for Distinguished Young Scholars); Prof. Yao Zhao (IEEE Fellow, IET Fellow, National Science Fund for Distinguished Young Scholars); Prof. Yunhong Wang (IEEE Fellow, IAPR Fellow, CCF Fellow); and Prof. Ming Tang
Thesis Defense Grade: Excellent
M.Sc. in Computer Science
Supervisor: Prof. Choli Wang
Thesis and degree details
Thesis: NightRunner: Deep Learning for Autonomous Driving Cars after Dark [Project]
Thesis Defense Grade: A+
B.E. in the Elite Class
Beijing Institute of Technology (BIT)
School of Information and Electronics
Field: Information Engineering
Thesis and degree details
Undergraduate Thesis Supervisor: Prof. Senlin Luo
Thesis: Text Sentiment Analysis Based on Deep Neural Network
Thesis Defense Grade: Excellent
Honors and Awards
- Award 2026 Reviewer Award, the 43rd International Conference on Machine Learning (ICML 2026)
- Award 2025 IEEE SMCS TEAM Program Award by the IEEE Systems, Man, and Cybernetics Society
- Award 2024 Best Paper Honorable Mention in the 3rd Workshop on Vision Datasets Understanding and DataCV Challenge in CVPR 2024 (CVPRW最佳论文提名)
- Honor 2024 Beijing Outstanding Graduates (北京市优秀毕业生, top 5%)
- Award 2023 China National Scholarship (国家奖学金, top 1%, only 8 Ph.D. students in main campus of University of Chinese Academy of Sciences win this scholarship)
- Award 2023 First Prize of Climbing Scholarship in Institute of Automation, Chinese Academy of Sciences (攀登一等奖学金, only 6 students in Institute of Automation, Chinese Academy of Sciences win this scholarship)
Honor 2022 Merit Student of University of Chinese Academy of Sciences (中国科学院大学三好学生)
- Honor 2017 Excellent Innovative Student of Beijing Institute of Technology (北京理工大学优秀创新学生)
- Award 2016 College Scholarship of Chinese Academy of Sciences (中国科学院大学生奖学金)
- Honor 2016 Excellent League Member on Youth Day Competition of Beijing Institute of Technology (北京理工大学优秀团员)
- Award 2015 National First Prize in Contemporary Undergraduate Mathematical Contest in Modeling (CUMCM) (全国大学生数学建模竞赛国家一等奖, top 1%, only 1 team in Beijing Institute of Technology win this prize) [📑PDF] [Selected and Reviewed Outstanding Papers in CUMCM (2011-2015) (Chapter 9)]
- Award 2015 First Prize of Mathematics Modeling Competition within Beijing Institute of Technology (北京理工大学数学建模校内选拔赛第一名)
- Honor 2015 Outstanding Individual on Summer Social Practice of Beijing Institute of Technology (北京理工大学暑期社会实践优秀个人)
- Award 2015 Second Prize on Summer Social Practice of Beijing Institute of Technology (北京理工大学暑期社会实践二等奖, team leader)
- Honor 2015 Outstanding Student Cadre of Beijing Institute of Technology (北京理工大学优秀学生干部)
- Honor 2015 Outstanding League Cadre on Youth Day Competition of Beijing Institute of Technology (北京理工大学优秀团干部)
- Honor 2015 Outstanding Youth League Branch of Beijing Institute of Technology (北京理工大学优秀团支部, team leader)
- Honor 2015 Top-10 Activities on Youth Day Competition of Beijing Institute of Technology (北京理工大学十佳团日活动, team leader)
- Honor 2014 Outstanding Student of Beijing Institute of Technology (北京理工大学优秀学生)
- Award 2014, 2015, 2016, 2017 Academic Scholarship of Beijing Institute of Technology (北京理工大学学业奖学金)
Activities and Service
Tutorial
34th International Joint Conference on Artificial Intelligence (IJCAI)
- Title: Human-Centric and Multimodal Evaluation for Explainable AI: Moving Beyond Benchmarks
- Date & Location: 14:00-15:30, 18th August, 2025, Montreal, Canada
28th European Conference on Artificial Intelligence (ECAI)
- Title: From Benchmarking to Trustworthy AI: Rethinking Evaluation Methods Across Vision and Complex Systems
- Date & Location: 26th October, 2025, Bologna, Italy
Webpage
2025 IEEE International Conference on Systems, Man, and Cybernetics (SMC)
- Title: The Synergy of Large Language Models and Evolutionary Optimization on Complex Networks
- Date & Location: 5th October, 2025, Vienna, Austria
31st IEEE International Conference on Image Processing (ICIP)
- Title: An Evaluation Perspective in Visual Object Tracking: from Task Design to Benchmark Construction and Algorithm Analysis
- Date & Location: 9:00-12:30, 27th October, 2024, Abu Dhabi, United Arab Emirates
- Duration: Half-day
Slides Webpage
27th International Conference on Pattern Recognition (ICPR)
- Title: Visual Turing Test in Visual Object Tracking: A New Vision Intelligence Evaluation Technique based on Human-Machine Comparison
- Date & Location: 14:30-18:00, 1st December, 2024, Kolkata, India
- Duration: Half-day
Slides
17th Asian Conference on Computer Vision (ACCV)
- Title: From Machine-Machine Comparison to Human-Machine Comparison: Adapting Visual Turing Test in Visual Object Tracking
- Date & Location: 9:00-12:00, 9th December, 2024, Hanoi, Vietnam
- Duration: Half-day
Slides Webpage
Mini-Symposium
The Fifth International Nonlinear Dynamics Conference (NODYCON 2026)
- Title: Complex Network Systems and Large Language Models
- Date & Location: 20th-23rd September, 2026, Sapienza University of Rome, Italy
Webpage
Talk
College of Liberal Arts, Wenzhou-Kean University
- Series: CLA Invited Academic Talks
- Title: Studying the Human Mind with AI: From Behavioral Analysis to Role Simulation
- Date & Location: November, 2026, Wenzhou, China
14th International Conference on Image and Graphics (ICIG 2026) (CCF-C Conference)
- Forum: Visual Intelligence in Transition: From Physical Perception to Psychological Cognition
- Title: From Object Tracking to Process Understanding: State Modeling in Dynamic Vision
- Date & Location: 4th October, 2026, Nanyang Technological University, Singapore
Chinese Congress on Image and Graphics (CCIG 2026)
- Forum: Intelligent Evolution of Video and Image Security: Perception, Reasoning, and Adversarial Challenges
- Title: Visual Understanding Reliability in Open Environments: from Robust Perception to Semantic Consistency
- Date & Location: 29th May, 2026, Guangzhou, China
- Role: Alumni speaker, research experience sharing
- Date & Location: 13th December, 2025, Beijing, China
Visual Intelligence International Postdoctoral (Doctoral) Academic Frontier Seminar
- Title: Human-Centric Multimodal Visual Intelligence: From Benchmarks to Human-Like Modeling
- Date & Location: 30th August, 2025, Online
Publicity Chair
2026 CSIG Annual Conference on Video and Image Security
- Date & Location: 21st November, 2026, Xiong’an, China
TPC Member
- Pacific Graphics 2026 (CCF-B Conference)
Area Chair
Virtual Collection Host
- Journal: The Innovation Informatics (Virtual Collection: Control and Decision Intelligence for Unmanned Systems; submissions open until 30 June 2027)
Lead Guest Editor
- Journal: Electronics (Special Issues: Techniques and Applications of Multimodal Data Fusion — 1st Edition and 2nd Edition)
Associate Editor
- Journal: Innovation and Emerging Technologies
Reviewer
- Conferences: NeurIPS, ICML, ICLR, CVPR, ECCV, ICCV, ACL, AAAI, IJCAI, ACM MM, ICRA, and AISTATS.
- Journals: ACM Computing Surveys, IEEE Transactions on Image Processing, SCIENCE CHINA Information Sciences, Pattern Recognition, Transactions on Machine Learning Research, IEEE Transactions on Network Science and Engineering, IEEE Transactions on Vehicular Technology, Information Fusion, Visual Intelligence, Engineering Applications of Artificial Intelligence, Expert Systems with Applications, Neurocomputing, and Knowledge-Based Systems.
Member
Contact
- shiyu.hu@ntu.edu.sg (Main)
- hushiyu199510@gmail.com (Personal)
hushiyu2019@ia.ac.cn(Valid from 2019.06 - 2024.07)