Close

Haoying(Jack) Zhou

Research & Development, Surgical Robotics, AI

Download Resume Download CV

About Me

Jack Zhou

As a fifth-year PhD candidate in Robotics Engineering at Worcester Polytechnic Institute, I am passionate about developing innovative solutions for exoskeleton and surgical robot control, perception, simulation, optimization and automation. I have been working with the da Vinci Research Kit (dVRK), a first-generation da Vinci surgical system, since 2021. I have contributed multiple hardware solutions to dVRK community, including but not limit to robot arm encoder replacement, robot arm brake replacement, viewer console linear actuator custom control enablement and surgical instrument lubrication & reactivation. I also take care of both hardware and software infrastructure development and maintenance for dVRK at WPI.
Currently, I am a visiting graduate scholar at LCSR JHU Robotics, where I conduct research on suturing tasks automation based on skills learned from demonstrations in simulation. Meanwhile, I also conduct research on accelerating surgery automation using sim-to-real method via creating more realistic simulation environments. In addition, I contribute to the software infrastructure construction of the AccelNet Surgical Robotics Challenge, which is an online challenge in simulation with following physical implementation aims to advance the state-of-the-art in suturing automation using dVRK. Additionally, I design frameworks for customized controller teleoperation, robot motion recording and replaying using ROS. Moreover, I also build realistic models for 6D pose estimation using deep learning and Pytorch.
Previously, I was a robotic intern at Philips, where I designed a synthetic motion simulator with GUI in Python using 3D DICOM data as the only input, and improved the image refreshing rate by over 30 times.
I have a master's degree in Mechanical Engineering from Boston University, and a bachelor's degree in Mechanical Engineering from Beijing Institute of Technology with senior-year exchange experience to University of California, Berkeley. I have also completed multiple online courses and certifications in machine learning and IRB trainings. I am proficient in Python, C++, MATLAB and familiar with both Linux and Windows development environments. I also have experience with ROS, Blender, VTK, ITK, VMTK, and Magic Leap 1& Xbox controller integration with robots. I am fluent in English and Chinese, and have strong communication and collaboration skills.
I am actively seeking for 2025 full-time position or internships in robotics, simulation, or AI-related fields.

Experience

Laboratory for Computational Sensing and Robotics, Johns Hopkins University

Visiting Graduate Scholar

- Core Developer of the AccelNet Surgical Robotics Challenge .
- Conduct research on simulation environment construction and the applications of various AI/ML algorithms for the dVRK
- Conduct research on Augmented Reality integration with the dVRK, such as recovery from teleoperation loss and AR-based measurement tool
- Develop infrastructures to be shared with the dVRK community.
- Led the dVRK workshop demonstration at 2024 ISMR.
- Integrate the dVRK with multiple platform, including Magic Leap 1 and NVIDIA Clara AGX.

Philips Research North America

Image-Guided Therapy Robotics Intern

- Design a synthetic motion simulator with GUI in Python using 3D DICOM data as the only input.
- Implement phantom feature extraction, volume rendering and 3D volume visualization with VTK, ITK and VMTK.
- Construct data auto-generator based on the synthetic motion simulator with flexible configuration inputs.
- Integrate the simulator with Xbox controller as motion control input.
- Improve the refreshing rate of the simulator from 0.15 fps to 5 fps.
- Implement analytical analysis on the generated data.

Automation and Interventional Medicine Robotics Research Laboratory, Worcester Polytechnic Institute

Research Assistant

- Manage and lead all da Vinci Research Kit(dVRK) related projects, including suturing automation, dynamic identification, customized controller teleoperation, kinematic & dynamic controller design and customized tool integration.
- Reactivate a full da Vinci surgical system with dVRK software framework; actively repair, maintenance and improve both hardware and software infrastructures.
- Integrate probe for photoacoustic scanning on dVRK PSM (patient side manipulator) with auto-scanning enabled.
- Lead and manage user study for collecting human motion patterns on the physical dVRK.
- Implement suturing subtask automation with learning from demonstrations in simulation.

Worcester Polytechnic Institute

Teaching Assistant

- TA for Control Engineering, Introduction to Dynamic Systems, Design of Machine Elements
- Design and Construct lab documents and GitHub repository for Control Engineering course
- Lead conference lectures for undergraduate courses
- Hold TA session to answer students' questions about homework assignments, labs and lectures

Education

Worcester Polytechnic Institute

Sep 2020 - Present

Doctor of Philosophy in Robotics Engineering

GPA: 3.95/4.00

Boston University

Sep 2018 - May 2020

Master of Science in Mechanical Engineering

GPA: 3.78/4.00

University of California, Berkeley

Sep 2017 - May 2018

Bachelor of Science in Mechanical Engineering

GPA: 3.95/4.00

Beijing Institute of Technology

Sep 2014 - May 2018

Bachelor of Science in Mechanical Engineering

Ranked: 10/33

Senior year exchanged to University of California, Berkeley

Research Projects

Suturing Automation based on Imitation Learning

WPI AIM Lab & JHU LCSR

- Construct user study for collecting human demonstration in both simulation and physical dVRK
- Implement learning from demonstrations with dynamic movement primitives(DMP) for suturing automation
- Achieve high generality for suturing automation, on the order of 91.5%, from experienced human subjects' demonstrations

Suture Needle Pose Estimation in Simulation

WPI AIM Lab & JHU LCSR

- Implement suturing needle 6D pose estimation algorithm using keypoint-based method and deep learning(fast RCNN)
- Achieve estimated position errors less than 1mm and orientation errors around 2 degrees
- Win second prize on 2021 AccelNet Surgical Robotics Challenge
- Generate a synthetic segmented 6D pose estimation dataset in simulation and train deep learning models on the dataset
- Related publication: Publication 3

Point Cloud Completion

WPI VISLab

- Purpose a novel chamfer distance loss function for point cloud completion task using Landau distribution
- Achieve new state-of-the-art results on some benchmark datasets

dVRK Customized Photoacoustic Scanning Instrument Integration and Automation

WPI AIM Lab & WPI FUSION Lab

- Integrate a customized surgical instrument using ultrasound probe with dVRK for photoacoustic scanning
- Utilize April Tag and computer vision technique to find the transformation between the probe and the endoscope
- Construct kinematic model for the customized instrument
- Build a ROS network to enable scanning automation
- Related publication: Publication 1 & 2

dVRK PSM Teleoperation using sEMG Proportional Control

WPI AIM Lab

- Utilize sEMG bio-signal differences on muscles to control the surgical instrument grippers open or close
- Leverage the trackers of a motion capture system to implement dVRK PSM teleoperation for human subjects
- Related publication: Publication 4

Lower Limb Exoskeleton Automation based on Learning from Demonstration

WPI AIM Lab

- Construct and optimize an algorithm on imitation learning with Task-Parameterized Gaussian Mixture Model (TPGMM) applied to human walking strategies for lower-limb exoskeleton
- Collect data using motion capture technique with real-life human motion
- Leverage AMBF for simulating the exoskeleton and human lower limb movements
- Design and implement iLQR controller to above algorithm and managed to find the optimal weight matrix
- Related publication: Publication 5

Reach-to-Grasp Imitation Learning

BU Robotics Lab

-Implement generalized research-to-grasp automation using imitation learning with dynamic movement primitives(DMP) in Python
-Leverage VREP for simulating robot arm motions
-Collect human demonstration data using joystick controller
-Accomplish 6D pose imitation learning for the end-effector of Baxter Robot
-Write and defend my Master’s Thesis based the research project

View Project

Technical Projects

Physical dVRK Reactivation

Worcester Polytechnic Institute

-Deploy both hardware and software infrastructures of dVRK
-Resolve multiple mechanical failures for components of the da Vinci Surgical System, including broken joint encoders and joint brakes
-Upgrade the stereo viewer of the da Vinci Surgical System
-Implement full dVRK system teleoperation using MTM and dVRK control boxes
-Enable Geomagic Touch device and Razer Hydra controller as alternative teleoperation options for dVRK
-Implement dynamic identification on dVRK PSM

Surgical Instrument Lubrication

Worcester Polytechnic Institute

-Develop a practical method to lubricate the surgical instruments of the first generation da Vinci Surgical system
-Tighten the internal cables of the instruments to enable solid grasping solutions
-Decrease the mechanical noise due to the instruments by two or three orders of magnitude

Selected Course Projects

Adaptive Robustness Control Design for UAV with ROS Gazebo

Worcester Polytechnic Institute

-Construct dynamic model for Crazyflie 2.0 drone and simulate the drone in ROS Gazebo
-Implement robustness control algorithm for UAV hovering
-Design adaptive robustness controller for UAV following specific trajectories

Autonomous Race Car Modelling and Control

University of California, Berkeley

-Construct the vehicle model and its tire model under different road conditions for predicting the vehicle movements
-Utilize MATLAB, Simulink, and ROS Turtlesim to simulate BARC movements under different conditions
-Remotely connect to the Linux-based vehicle operating system via VNC Viewer and SSH
-Implement lane keeping, drift parking, and adaptive cruise control on BARC with linear controllers such as PID and LQR
-Analyze data in MATLAB, including camera calibration and result data analysis utilizing methods such as FFT interpolation and least square method to obtain and discern relationships between variables

Tri-Bot Design and Manufacturing

University of California, Berkeley

-Utilize MyRio as the microprocessor and program in LabVIEW for target detection and tracking algorithm in the upper camera system
-Leverage Arduino board as the processor and code in Arduino for lane keeping algorithm in the lower moving system
-Champion full design and manufacturing of product from conception to delivery

View Project

Other Selected Projects

Academic Publications

  1. Gao, Shang, Yang Wang, Xihan Ma, Haoying Zhou, Yiwei Jiang, Kehan Yang, Liang Lu et al. "Intraoperative laparoscopic photoacoustic image guidance system in the da Vinci surgical system." Biomedical optics express 14, no. 9 (2023): 4914-4928.
  2. Gao, Shang, Yang Wang, Haoying Zhou, Kehan Yang, Yiwei Jiang, Liang Lu, Shiyue Wang et al. "Laparoscopic photoacoustic imaging system integrated with the da Vinci surgical system." In Medical Imaging 2023: Image-Guided Procedures, Robotic Interventions, and Modeling, vol. 12466, pp. 62-70. SPIE, 2023.
  3. Jiang, Yiwei, Haoying Zhou, and Gregory S. Fischer. "Markerless Suture Needle Tracking From A Robotic Endoscope Based On Deep Learning." In 2023 International Symposium on Medical Robotics (ISMR), pp. 1-7. IEEE, 2023.
  4. Yang, Kehan, Tess B. Meier, Haoying Zhou, Gregory S. Fischer, and Christopher J. Nycz. "A sEMG Proportional Control for the Gripper of Patient Side Manipulator in da Vinci Surgical System." In 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pp. 4843-4848. IEEE, 2022.
  5. Goldfarb, Nathaniel, Haoying Zhou, Charles Bales, and Gregory S. Fischer. "Control of a lower limb exoskeleton using Learning from Demonstration and an iterative Linear Quadratic Regulator Controller: A simulation study." In 2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pp. 4687-4693. IEEE, 2021.

** Some publications submitted in 2023 are still under review. The list of publications will be updated when the decisions are made.

Skills

Contact Information

hzhou6@wpi.edu / zhyjack0718@gmail.com
(774)-823-0984
Worcester, Massachusetts, USA