Intelligent Vehicles and Systems Group

Investigators: Jariullah Safi, Mohamed Wahba, Dr. Sean Brennan

The goal of the ROAR project was to investigate the potential for partially automating garbage collection with robots. A demo for the project in late 2015 (embedded below) showed how a garbage truck, a quadcopter, and a four wheeled robot could collaborate to bring garbage bins from a house back to the truck and empty it.

This prototype was built through a collaboration between research groups at Chalmers University in Gothenburg, M�lardalens H�gskola in V�ster�s, and the IVSG at The Pennsylvania State University. The IVSG's role in the project was as follows:

Map Generation: We had access to realistic colorized point-clouds of three demo sites in Sweden. While comprehensive, this information was too dense and could not be used efficiently for the purposes of the project. We converted this data into low order representations of the world that the could be used for both simulation based design and visualization for monitoring and user control.

User Interface for the Driver: The driver of the truck needs a way to monitor the status of all robots without having to stand outside the truck. Using the point-cloud data and map based localization of the robots, we created an intuitive user interface that can show the driver where the robots and trash bins are and where they are going. In addition, the interface would show the current status of the robots.

Robot Operating System Support: Among the participating universities, the IVSG at Penn State has the most experience using the Robot Operating System (ROS). We have provided ROS training to participating students from both Chalmers and Malardalans and continually assisted them with ROS as the need arised.

Research: Volvo gifted a Clearpath Husky robot to the IVSG for this project, which we then used to do some research:

Estimation of Instantaneous Center of Rotation (ICR) based Model for Skid Steer Robots:
We built upon lab alum Jesse Pentzer’s thesis work of estimating ICR models for skid steer robots using GPS by replacing the GPS with indoor map based localization using LiDAR.

This solution converged significantly faster than a GPS based version since LiDARs operate an order of magnitude faster. We also discovered a relationship between ICR locations and robot weight distribution; this has implications for a trash carrying robots since they would have to deal with cans of different weights and non-symmetrical weight distriutions. Some of these results were presented at the ISTVS 2016 conference.
Citation: ON-LINE ESTIMATION OF SKID-STEER INSTANTANEOUS CENTERS OF ROTATION IN GPS-DENIED ENVIRONMENTS Jesse Pentzer, Jariulla Safi, Sean Brennan and Karl Reichard

Virtual and Augmented Reality Driving of Ground Robots: Getting a good feeling for a Robot’s environment is a challenge for remote operators of robots. Video feeds often lack depth and only cover a small section of the scene at any given time. We experimented with a potential solution to these woes by creating a 3D map of a robot’s environment and then localizing the robot inside it. We then put the robot operator inside this virtual environment by using a VR headset. When teleoperating or monitoring a robot without camera sensors, such a system can help operators feel more comfortable by surrounding them with pre-collected visual information they can digest while simultaneously projecting information from other sensors (e.g. LiDAR).

Any available video feeds can also be projected in the environment in a similar fashion creating an augmented reality representation for the operator.

PIs: Sean Brennan and Jason Moore
Graduate Students: Kelilah Wolkowicz (Ph.D. Candidate)
Undergraduate Students: Aditya Agarwal and Taylor Baum
Collaborators: Sun Han and Bruce Gluckman

Sponsors: This work is supported by the National Science Foundation under Grant No. DGE1255832.

Persons with disabilities are increasingly reliant on technology to provide freedom of mobility. However, current wheelchair technology requires extensive direct user joystick input, which limits functionality and creates error between the desired and actual system output. This research project aims to create a semi-autonomous navigation and control system for an electric wheelchair, which is more sensitive to the progression of neuromuscular diseases, resulting in improved freedom of mobility for those with Alzheimer’s, Parkinson’s, multiple sclerosis (MS), amyotrophic later sclerosis (ALS), and Huntington’s disease. Additionally, the wheelchair as a smart platform enables health monitoring of the user’s disease progression.

The project has developed a robotic wheelchair platform with sensors, computers, and a power management system to conduct robotic wheelchair experiments. A key focus of our ongoing work is to enhance the safety of wheelchair operation, investigate modality of user inputs, coordinate wheelchair motion events with user decision-making measurements both on the joystick and on EEG/EMG, develop GPS-free indoor localization technologies, and develop power management and monitoring strategies supporting users. The ultimate goal is to integrate the autonomous system with a computer-brain interface (BCI) to allow those with severe mobility impairments to regain navigational control in their daily-lives.

Figure 1. The robotic wheelchair has been outfitted with a variety of sensors to obtain position, orientation, and environmental information and to enhance safety features for operation.

Wheelchair Safety
Positive and Negative Obstacle Detection: Safe user assistance and automation requires the measurement of positive obstacles such as collision hazards, and negative obstacles such as curb drop-offs and stairwells. These measurements may require a variety of sensors, including ultrasonic sensors, LiDAR Lite laser rangefinders, and a LiDAR laser rangefinder; these sensors are mounted and are being tested for obstacle detection.

ICR Tire Slip Detection: Wheelchair tire slip is a result of icy or low-friction surfaces, often representative of dangerous conditions. This slip can be detected by estimating the instantaneous center of rotation (ICR) locations of wheelchair wheels relative to the ground surface using an Extended Kalman Filter (EKF). Any departure of the estimated ICR positions from the wheel contact point indicates slippage is occurring. We are developing kinematic models of wheelchairs that allow automated and rapid estimation of slip conditions.

Figure 2. ICR EKF experimental results for estimating wheelchair tire slip. While the wheelchair is driven under normal operating conditions on the non-slip surface, the ICR locations do not vary significantly. However, as the wheelchair tires experience slip (shown in grey), the estimated ICR location errors were up to 42 times larger than the maximum error seen without a slippery surface.

Modality of User Inputs
Electroencephalogram (EEG)- and Electromyography (EMG)-based BCI: We are investigating new methods of blending multiple signal inputs for safe wheelchair operation. We are focusing on blending signal inputs from EEG and EMG during wheelchair operation. Portability of the data acquisition system is important when the user is driving a wheelchair. Consequently, a new BCI DAQ board is being used that is portable (0.025m2) with a moderate channel count (16 ports) and a sampling rate of 256 Hz per channel.

Virtual Environment Training: A key goal for the robotic wheelchair-based research is to ease the transition for ALS patients from a joystick-controlled wheelchair to a BCI-controlled wheelchair. The objectives of this work are to determine: 1) where users perform decision point inputs based on their location and 2) if inputs other than those of a joystick (e.g. EEG and EMG) can be used for navigation within a virtual environment. A simulated environment has been created for virtual wheelchair testing using inputs from two joysticks in conjunction with EEG and EMG signals.

Correlation Between Left/Right Joystick Motion and Left/Right Brain Signals: The objective of this work is to establish a correlation between left/right joystick motion and left/right brain signals. Two independent joysticks are used to control a virtual wheelchair; the left joystick is used for left turns and the right joystick is used for right turns. When only one joystick is used, it is difficult to differentiate right/left motion from the EEG signals. But when both a left and right joystick are used for wheelchair navigation, EEG signals are readily detected from the right and left brain hemispheres. When only one joystick is used, these same EEG signals can only be detected easily in one hemisphere.

Figure 3. Investigator driving robotic wheelchair using right and left joysticks while measuring EEG brain signals.

Imagined-Motion Training: Typically, the use of BCI requires training activities to occur that are mediated by trained professionals. This approach is costly and time consuming, yet provides limited exposure to patients’ data within the disease progression. We are investigating new training methodologies based on sensorimotor rhythm modulation resulting from the imagination of motor action, produced by imagining movement by the right and left hand. The advantage of this approach is that training could be performed by collecting BCI data while the user is navigating through a simulated office environment and controlling a joystick-driven virtual wheelchair. Such training seeks to ease the transition of wheelchair control from joystick to BCI, and may extend to training during actual use of the wheelchair rather than simulated environments.

Indoor Localization Technologies
Localization and Mapping using Ambient Magnetic Fields: Ambient magnetic fields in any indoor location in an urban setting have a non-uniform distribution due to the presence of metallic and/or magnetic materials used in construction, in furniture, and in household electronic equipment. The measurements of these local non-uniformities in the magnetic field are being investigated for use as features for indoor localization of the wheelchair.

Figure 4. Likelihood of wheelchair’s position within the map as a function of location and ambient magnetic field strength using a density estimation approach.

Investigators: Jesse Pentzer, Dr. Sean Brennan, Dr. Karl Reichard

Sponsors: This work is supported by the Exploratory and Foundational Research Program, administered by the Applied Research Laboratory, The Pennsylvania State University. No official endorsement should be inferred.

This project focuses on developing kinematic models of skid-steer robot movement that are easily adapted to changing terrains and robot designs. In this work, skid-steer movement is modeled using kinematic equations incorporating the Instantaneous Centers of Rotation (ICRs) between the tracks or wheels of the vehicle and the ground. The surface contact points of the tracks or wheels are in pure rotation around the ICR. The locations of the track or wheel ICRs relative to the geometric center of the robot provide information on the amount of slippage occurring during movement and are useful in predicting vehicle motion when the track or wheels speeds are known. In general, ICRs for moving rigid bodies are constantly changing, but prior work with skid-steer vehicles has found that the ICRs of the tracks or wheels are relatively constant, regardless of vehicle maneuver type, for low speeds on flat terrain. This property enables us to use a nonlinear estimator, an extended Kalman filter (EKF) in this case, to learn the ICR locations during operation through the measurement of vehicle position and heading. This algorithm has been implemented on both tracked and skid-steer platforms during operation at low speeds on flat terrain. The resulting estimations of ICR locations provide good prediction of vehicle motion when applied open-loop with only knowledge of input track of wheel speeds. Current and future work is focused on adapting the algorithm for use on non-flat terrains and field testing to determine the operational envelope for which the algorithm provides reliable ICR location estimates.

Tracked and wheeled robots developed for skid-steer vehicle field testing. Each vehicle utilizes a differential GPS system, magnetic heading sensor, and wheel or track speed sensors.

Comparison of GPS measured position (black) with open-loop odometry estimates using ICR kinematics (red) and a no-slip assumption (blue). The ICR kinematic model utilized ICR locations learned during operation on the same terrain, and provides a significant improvement over a model assuming no slippage occurs.

Investigators: Adam Crimboli, Dr. Sean Brennan, Dr. Karl Reichard Sponsors: This research is supported in part by funding from the National Institute for Standards and Technology, and Exploratory and Foundational Research Program, administered by the Penn State Applied Research Laboratory.

This project focuses on developing improved methods for measuring and characterizing robot and operator performance. The methods apply to a NIST ground robot testing validation method as well as its employment in operator and terrain variability studies for bomb disposal robot performance. The testing method developed improves on a novel robot tracking system developed previously and utilizes a standardized NIST testing arena. Overhead cameras and an onboard power logger capture data during a test, and computer algorithms are employed for further processing. Fiducial tracking and background subtraction algorithms calculate a robots position, speed, and lap progress over time during a test. Further processing is employed to calculate a robots most common path and subsequent deviation from this path, as a metric of consistency. Power usage and energy consumption over a test are also considered. Two primary applications of this testing method are demonstrated. Changing the terrain inside the testing arena between tests allows the robot performance metrics previously outlined to be studied as a function of terrain. Second, choosing one terrain of interest but utilizing different robot drivers for otherwise similar tests allows robot performance as a function of operator ability to be studied.

The Talon bomb-disposal robot being driven through a figure-8 course. Overhead cameras track the green fiducial on the robot. An onboard logger tracks power consumption.

A 3D histogram showing how the robot has traveled over the course of a 50 lap test.

Investigators: Jesse Pentzer, Dr. Sean Brennan, Dr. Tim Simpson, Dr. Karl Reichard, Dr. Chris Rogan

Sponsors: NAVEOD, Navy

This project focuses on the development of physics-based, empirical, and allometric models to determine ground robot capabilities and power requirements as functions of robot size and design. Physics-based models are utilized to determine the capability of a robot when performing tasks such as climbing stairs and traversing ditches. The models utilize robot design parameters, such as track length and center of gravity location, and powertrain properties, such as available torque, to determine robot capabilities. Experimental measurements were collected from a fleet of operational EOD robots including robot power usage while operating at varying speeds over different terrain types. These measurements have been used to model the power requirements for robot motion as a function of robot mass, velocity, and terrain type. Allometric (e.g. size-dependent) models were also developed for individual powertrain components as a function of required power from manufacturer data. These powertrain scaling models are combined with the physics-based capability and experimental power usage models to quickly determine robot design and mission feasibility. Current work in this area includes determining the effects of continuously variable transmissions on powertrain efficiency, development of hybrid robot powertrain models and control strategies, and evaluating mission feasibility by comparing robot capabilities and endurance with mission requirements.

PERFORMANCE CAPABILITIES: A variety of robot platforms utilizing different locomotion techniques are used to examine and validate predictive modeling of performance and power usage.

Investigators: Drew Logan, Dr. Sean Brennan
Sponsors: Department of Defense, NAVEOD Division

Hybrid powertrain systems can potentially provide many benefits for unmanned ground vehicles including extended mission duration (from several hours to tens of hours, possibly even days), the use of common fuels or batteries for both manned and unmanned systems, and integrated system health monitoring capabilities. This project, led by the Penn State Applied Research Lab, seeks to develop and demonstrate an extensible hybrid power and energy system for unmanned ground vehicles. The effort runs concurrently with the Technology Demonstration and System Development phases of the Advanced EOD Robotic Systems program at Penn State, and thus will be applicable to the entire AEODRS Product Family.

Two key hardware challenges are being addressed as part of this research: Identification and evaluation of scalable power generation and energy storage technologies suitable for EOD robots; and development of hardware which allows component-level power control and load sharing, e.g. “smart” bi-directional DC/DC converters. The former is being investigated by this research group, while ARL is focusing on the latter.

A primary goal of the study is to analyze hybrid system architecture and controllers in the context of scalability, in order to determine the power requirements for robots as a function of robot size. The following task items are being looked into: development of a scalable and generalized model for a robot hybrid powertrain, definition of representative mission profiles, development of hybrid powertrain control algorithms, optimal profile management using Dynamic Programming, and comparison of the resulting performances across different power train architectures to determine how sizing of hybrid components is best implemented between robot platforms and missions.

COMPARISON OF HYBRID TECHNOLOGIES: The RONS power is measured while it drives around a test track. Power profiles are used to compare the relative performance of different hybrid electric technologies under equivalent conditions.

Investigators: Adam Dean, Dr. Sean Brennan
Sponsors: Department of Defense (NAVEOD)

Ground robots continue to increase in capability, including higher speeds to keep up with running or mounted humans, greater traction to negotiate ever harsher terrain, and greater lifting and payload capacity. As a result, stability analysis and consideration in robot chassis control is becoming particularly critical to prevent rollover and sideslip instabilities.

This project utilized a rolling-roadway simulator to test robot chassis stability and control using a testing concept similar to an aircraft in a wind-tunnel: a pseudo-stationary vehicle was driven on a moving treadmill surface. An advantage of this system is that it allows testing at all speeds and configurations, enables advanced chassis controller tuning, and safely permits fault-testing. This system can operate as a stand-alone robot chassis tester, or as a hardware-in-the-loop system wherein subcomponents of the robot are tested individually while operating within a moving robot. Additionally, the surface can be wetted, tilted right/left at beyond 20 degrees, and/or tilted fore/aft at up to 6 degrees. The lateral motion in particular is useful to examine the robot behavior at the limits of adhesion or under sharp turning motions.

TANKBOT ON THE PURRS: The mobile robot "TankBot" under dynamic analysis on the Pennsylvania State University Rolling Roadway Simulator (PURRS)