Lars Ruthotto | Emory REU Computational Mathematics for Data Science

Comparing Reinforcement Learning to Optimal Control Methods on the Continuous Mountain Car Problem

Fri, 05 Aug 2022 00:00:00 +0000

What is the best way to get a car out of the bottom of a hill?

Introduction

This blog post was written by Jacob Mantooth,Dewan Chowdhury, Arjun Sethi-Olowin and published with minor edits.The team was advised by Dr.Lars Ruthotto.In addition to this post, the team has also given a midterm presentation, filmed a poster blitz video, created a poster, published code, and written a paper.

The word around town is that reinforcement learning is the top dog and has the answers to all our problems. We wanted to see if that really was the case, so this summer we took a trip to Emory University where we looked at the continuous mountain car problem to see if reinforcement learning really, was the best. The continuous mountain car problem is an example of an optimal control problem. In the image below you can see an example of what the continuous mountain car problem looks like.

You may be asking yourself what is an optimal control problem? An optimal control problem is problem that consists of controlling a dynamical system to minimize (or maximize) a given objective function. In our case the continuous mountain can be modeled as a ODE where $u$ is some controllable function. In the continuous mountain car problem our control,$u$, is whether the car accelerates left or right. In an optimal control problem, we seek to optimize some objective function, in our case we will minimize the objective function. Our two objective functions are the running cost, which penalizes the car for acceleration. While our other objective function is the terminal cost which penalizes the car for not reaching the goal, the top of the hill, in time.

Why This Problem?

You may be wondering why choose the Continuous Mountain Car Problem? Here are a couple of reasons why we picked this example,

Established benchmark for RL models
2-D state-space allows for good plots and visualizations
Both RL and optimal control problem
Finite horizon (time)
Continuous state and motion

The whole reason we are doing this is because we want to compare three different ways of solving the continuous mountain car problem and see which one really is the best. Our three approaches are

Local solution using numerical ODE solvers and nonlinear optimization (baseline)
Reinforcement learning with actor-critic algorithm (data-based approach)
Optimal control using both model and data

Our Three Approaches

A Local Method

Our first method that we looked at during this REU was the local method. We tried to find the optimal control $u_h$ by formulating an optimization problem.

We first discretize the control, state and the Lagrangian.

Setting $z_h^{(0)}=z_t$ and $\ell_h^{(0)}=0$, allows us to use a forward Euler scheme for some control $u$.

We then approximate our objective function which yields the optimization problem

To solve our optimization problem, we used gradient descent. By taking an initial guess for $u_h$ and repeatedly updating $u_h$ using the gradient of the objective function and step size $\alpha$

$(u_h)_0 = \vec0$

$\vdots$

$(u_h)_6 = (u_h)_5 - \alpha( \nabla J((u_h)_5))$

$\vdots$

$(u_h)_* = (u_h)_19 - \alpha( \nabla J((u_h)_19))$

Below is a nice visual example of what all this math means. When the tail reaches the dotted line, it means our car has reached the top of the hill.

The graph shows us the position vs velocity of the car. In the graph the black dot represents t and the tail of the plot, when x-position is .45 is time T. In the graph we see the color change from red-blue, in our plot the blue color is when the car is accelerating, and control is positive but red otherwise.

Our next goal was how do we create a nice visualization of what the actual solution looks like.

We see in this video what our optimal local solution looks like. A couple of things that should be noted is, if we move the car to a new position then this local solution may no longer work. The same can be said if we slowed down/speed up a bit then this solution may not even let the car get to the top of the mountain. Another downside of the local is that it is a non-linear and non-convex problem which makes this method slow. This local solution will serve as a baseline so we can compare other methods to something to see which one is really the best.

Global Methods

Now that we have established a baseline, we will discuss our other two methods. Our other two methods that we will be looking at are global methods, the first being reinforcement learning method and the other being optimal control method. You may be asking yourself what is the difference? Reinforcement learning is more of a data driven approach while the optimal control method is a hybrid approach, using both a model and data.

Reinforcement Learning

Our first stop in exploring global methods is reinforcement learning. We will be using reinforcement learning with actor-critic algorithm. This approach is completely data-based approach. In reinforcement learning it has no knowledge of the model, it only considers the objective function. In reinforcement learning we would like to maximize a reward, so in our case we will maximize negative cost. Reinforcement learning is stochastic in two ways with initial position and action space which allows for exploration. In Reinforcement learning we are trying to estimate an optimal control policy. One of the big things that we have yet to discuss is, what is actor-critic algorithm ? The actor-critic (AC) architecture for RL is well-suited for a continuous action-space as in the continuous mountain car problem 1 . In the actor-critic algorithm the critic must learn about and critique whatever policy is currently being followed by the actor. We worked in the OpenAI gym mountain car environment, so we were able to find preexisting code for our project. We also were able to adapted the TD advantage actor-critic algorithm adapted from here. The thing is our preexisting code was not the same as our problem, so we had to modify it some. After some modification to the code, the following video is the results that we were able to get after many training cycles.

In this video we see that RL gave us a sub optimal solution compared to the local solution. You may also notice that in the reinforcement learning method our car takes an extra swing backwards to get to the top of the hill. In our reinforcement learning method it took 1000’s episodes just to get the car to our goal. We saw that reinforcement learning is very fragile, a couple of changes saw our success rate go from 70% to barely making it all. The picture below is position vs velocity of the car.

as you can see compared to the local solution, we see that the RL solution is very sub optimal solution.

Optimal control method

Our last two methods were vastly different with reinforcement learning using a data driven approach and the local method using a model-based approach. We will now be looking at the optimal control method which combines both model and data driven approaches. In this approach, we aim to adhere to the method discussed here. We will estimate the corresponding value function with neural network approximators utilizing feedback from the Hamilton-Jacobi-Bellman equation and Hamiltonian.

Using OC method we were able to produce the following

Once again, we created a position vs velocity of the car graph. As you can see this graph is a sub optimal solution compared to the local method. Compared to the RL method, we see how much better OC was for our problem. We see through testing of the RL method that there are some draw backs to forgetting the model and just being purely driven by data.

Our Experiences

Week 1

In week one we decided to make a game plan for the following weeks. We would work on the local method for just a week since it was basically finished. For the other two methods we would spend two weeks on each method. Lastly, we would save the last week to wrap up all three methods and anything else that is left over. During the first week we wanted to look at the local method and explore it some.

Week 2 & 3

We decided to spend two weeks to look at reinforcement learning, during these two week we were able to produce a PowerPoint in beamer for our mid-week presentation. In week two and three we looked at our first global method. Dr.Ruthotto handed us some pre written code to play around with. A thing that should be noted is that this prewritten code only worked maybe 50% of the time. Looking deeper into the code we realized that we would have to mess around with the code to get it to match our problem. After we made these couple of changes in our code, we saw how fragile reinforcement learning is, instead of working 50% of the time our code barely worked at all. During this week we were also able to produce a rendering of both of our local and RL methods, adding a nice touch to our presentation that we gave.

Week 4 & 5

During week 4 and 5 we looked at optimal control method. During week 4 we were able to produce a rough draft of the website while also taking a deeper look into optimal control method. Week five we created a rough and final draft of our poster. While working on our poster we were able to produce a graph for OC so we can compare it to our other methods.

Week 6

We wrapped up any unfinished work including our paper, OC method and this website. After struggling with code for method three we finally found that our OC method had better results than RL. During week six we also gave a poster talk to the Emory staff and students. Our group ended up winning best poster and we each won some amazon gift cards. We will continue working on method three to try and get it to work for 150 steps, we also look to fine tune our code for method three and two.

More About Our Team

Reference

U. M. Ascher and C. Greif. “Chapter 9 (Optimization).” A First Course on Numerical Methods. SIAM. SIAM,2011

U. M. Ascher and C. Greif. “Chapter 14 (Numerical time integrators).” A First Course on Numerical Methods. SIAM. SIAM,2011

Undergraduate students and teacher develop segmentation methods for diagnosing Chiari Malformation

Tue, 14 Dec 2021 00:00:00 +0000

This post was written by Justin Smith, Elle Buser, Emma Hart, and Ben Hueneman and published with minor edits. The team was advised by Dr. Lars Ruthotto. In addition to this post, the team has also created slides for a midterm presentation, a poster blitz video, code, and a paper.

Collaboration Never Sleeps.

During our summer research at Emory University 2021 REU/RET program, our group focused on the algorithmic diagnosis of Chiari malformation from DENSE MRIs. We created an algorithm that can accurately and efficiently segment the cerebellum and brain stem from a magnitude image and use displacement data to classify whether or not a patient has the Chiari malformation. In doing so, we investigated two approaches; one that segments the given image by aligning and comparing the image to a known atlas and another that segments through deep learning.

Did Somebody Say Chiari Malformation?

Chiari malformation is a condition in which brain tissue extends into the spinal canal. While it can be difficult to diagnose Chari from anatomical images, a promising new direction for diagnosis is by looking at brain movement . Using an MRI technique called DENSE (shown below) that records how the brain moves, Dr. Oshinsky’s group (at Emory’s Dept. of Radiology) collected data about how Chiari patients have more brain movement in the cerebellum and brainstem than controls.

This method may be more accurate in diagnosing Chiari, however, the large number of manual processing steps may limit its use as a wide-spread screening tool. This project aims at exploring the use of machine learning algorithms to automize parts of the image processing pipeline, most critically the segmentation of the image into different brain regions. The teams worked with image data that has been collected and labeled by Dr. Oshinski’s group in a previous research study. The project is accessible to the team members since we can build upon recent progress and software made in image processing and computer vision and the image data is two-dimensional and of limited resolution, which enables fast experimentation. Despite this simplicity, the project allows us to investigate ML in a realistic setting and investigate the generalization properties and robustness of the approach.

Leave the SEGMENTATION to US!

We develop this project to solve the problem of identifying where the brain stem and cerebellum are in a given MRI. By finding or, in the language of the field, by segmenting the brain stem and cerebellum, we find the most relevant regions to look at brain movement. Using the DENSE MRI data, we can then average the movement over those regions to produce a biomarker that can help predict whether or not a patient has the Chiari Malformation. By producing these segmentations (examples above) automatically with the machine learning or atlas-based approaches, the diagnosis process could become much cheaper and more efficient.

Atlas Based Image Registration vs Machine Learning.

We first looked into atlas-based image registration as a way to produce automatic segmentations of the brain stem and cerebellum. Using the FAIR toolbox in MATLAB, the idea behind this method was to have a bank of MRI images with manually drawn segmentations that we could compare a new MR image to. Once we find a transformation (example below) between the known and new images, we can use the same transformation to produce a new segmentation from the know one.

We also looked into a machine learning approach. The goal here was to find a relationship between the DENSE images and their corresponding manual segmentations by training a model using convolutional neural networks (CNN). The network "learns" to identify images features, and, if successful, DENSE images can be used as inputs and the model will automatically segment the brain stem and cerebellum.

Our project implemented a CNN called U-Net, you can find out more about this network and the code we used here: U-Net: A PyTorch Implementation in 60 lines of Code

Overall, we found that the machine learning method produces better results, both in terms of segmentations of the brain stem and cerebellum, and in terms of how accurate the biomarkers calculated from those segmentations are. Results were very close to the manually produced target results, and we have ideas for further work that could make them even closer!

Additional information: to learn about atlas-based image registration and machine learning, check out these links! What is Image Registration? What is Machine Learning?

Time Management is Everything!

Here’s an outline of our process, as it evolved with time.

Week 1: During our first week, we created a working atlas-based image registration example, using FAIR: a MATLAB image registration toolset. We also began to look at the at a machine learning method, called U-Net, that we began setting up using PyTorch. We also set up GitHub, that we used throughout the project to collaborate on and publish codes.

Week 2: One of the first things we noticed when we began working with our data set the first week was the variability in the brightness and contrast of our images. In week two, we explored some different methods to help enhance the images. We decided to use a tool in MATLAB’s Image Processing Toolbox to normalize the images in a process called histogram normalization, which made our images much more consistent.

With this process complete, we began working on other atlas-based examples, and setting up the neural network with default parameters.

Week 3: We spent most of the third week preparing for our midterm presentation. It was helpful to practice presentation skills and get familiar using Beamer - a math presentation tool we weren’t yet familiar with that is the gold standard for mathematic presentations - and also to reflect on the progress we made, and talk through next steps with others after we presented. After our presentation, we started refining the machine learning method to use a new optimizer that automatically chooses the algorithm’s learning rate using a line search. This produced much better results!

Week 4: In the fourth week, we began working on our paper manuscript. Although there was still much of the research left to be done, coding in MATLAB and PyTorch, starting on this process of outlining our problem and explaining our methods helped us clarify next steps. At the end of the week, we turned back to our codes. On the atlas-based side, we explored ways to choose the images to compare to each other, because it can make a big difference in how good the outputted segmentation is. We also began devising a way to compare a new image we want to segment to more than one images we already have segmentations before, hoping that by averaging those results together, we could produce even better segmentations. On the machine learning side, we looked at ways to figure out the ideal number of iterations to use, so that we were neither overfitting nor underfitting our model to training data.

Week 5: Week five progressed very similarly to week four, as we continued working on our paper and developing better methods. We also began working on a poster to present our final results more broadly - and on this website!

Week 6: In the last week of the program, we had a lot left to do, finishing up our poster, website, and paper manuscript. We finalized every parameter in each method before finally looking at the testing set for the first time and analyzing our final results.

Poster Blitz Video

More About the Team

Elle Buser is a rising senior at the University of Wyoming majoring in mathematics and minoring in physics. At UWyo, she has worked as a tutor/supplemental instructor for Calc II and as a research assistant in the physics and astronomy department. Outside of school, she spends her time reading, doing arts & crafts, hiking, running, or just enjoying the great outdoors.
Emma Hart is a rising senior at Colgate University, double majoring in applied mathematics and educational studies. She enjoys combining these interests in math and education by volunteering as a math tutor and also by working for Colgate as a Writing Center peer consultant and computational mathematics grader. Outside of school, she enjoys crafting and spending time outside with friends and family.
Ben Huenemann is a rising junior at the University of Utah majoring in mathematics and computer science. He loves all things math, but is mainly interested in pure mathematics and plans to go on studying algebra or something similar in graduate school. Outside of school, he enjoys playing piano, evening bike rides, and board games with friends/family.
Justin Smith is an Early Childhood educator in Atlanta, Georgia, where he is in his 10th year of teaching. He is certified in Early Childhood Education and holds a Gifted Endorsement in the state of Georgia. He currently holds the Teacher of the Year award at his school. He is very active in his community and school, where started an organization called Obama Gentlemen with the primary duty to improve academics and address behavioral concerns of young men by instilling morals and values that will help transition young adolescent males into young adulthood. During his leisure time, he enjoys traveling, shopping, listening to music, and spending time with his family.

Call for Summer 2025 Applications

Thu, 19 Dec 2024 00:00:00 +0000

We are pleased to announce that our applications website for REU applicants is now live. As we did last year, we are using NSF’s new ETAP system for the first time, which should make it easy for applicants to apply. We will start reviewing applications on February 1 and all applications received by March 1 will receive full consideration.

Our theme will be Models meet Data. For more information about the application process, see this page and for a list of projects see the Summer 2025 tab.

Lars Ruthotto

Thu, 19 Dec 2024 00:00:00 +0000

Lars Ruthotto is the Principal Investigator of this REU/RET site. He is an Winship Distinguished Research Associate Professor in the Department of Mathematics and the Department of Computer Science at Emory University and a member of the Scientific Computing Group. Prior to joining Emory, he was a postdoc at the University of British Columbia and held PhD positions at the University of Lübeck and the University of Münster.

Highlights from Our 2024 Summer REU Program

Thu, 07 Nov 2024 00:00:00 +0000

This summer, we welcomed 15 talented students from across the country on Emory’s beatiful campus. Under the umbrella theme Learning from Images, the students worked in five teams, each diving into projects aimed at advancing our understanding and ability to learn from imaging data. This program, held over eight weeks on campus, provided the young researchers with an opportunity to engage in cutting-edge computational mathematics and machine learning.

The Learning from Images theme tackled a timely challenge: how to make sense of the overwhelming amount of imaging data generated daily. Computational algorithms can unlock insights from this data, improving our ability to detect new patterns and enhance image quality, especially in areas critical to healthcare. Each project addressed a unique aspect of this challenge—from improving image distribution models and optimizing measurement designs to refining algorithms for reconstructing image sequences. You can read more about the projects here.

The students worked with their mentors to explore and develop advanced imaging techniques that leverage machine learning, numerical linear algebra, optimization, and differential equations. Weekly project meetings, a seminar series, and various social events fostered an engaging learning environment. Students had the chance to present their progress in midterm presentations, culminating in a highly anticipated final poster session.

The poster session marked the program’s grand finale, drawing a large crowd of faculty, students, and postdocs. Each team showcased their findings and shared insights from their experiments, highlighting the strengths and potential areas of improvement for their algorithms. By a narrow margin, this year’s poster award went to the team of Malia Walewski, Rishi Leburu, Claire Gan, and Callihan Bertley for their project on Improving VAEs with Normalizing Flows. The picture below shows Malia and Rishi presenting their work.

All teams created a poster blitz video and wrote blog posts about their work, which are now live on the 2024 tab. These additional resources capture the excitement and innovation of the program and give a closer look into the students’ experiences.

We’re proud of our 2024 REU cohort and look forward to following their careers in the years to come!

Call for Summer 2024 Applications

Thu, 21 Dec 2023 00:00:00 +0000

We are pleased to announce that our applications website for REU applicants is now live. This year, we are using NSF’s new ETAP system for the first time, which should make it easier for applicants to apply. We will start reviewing applications on February 1 and all applications received by March 1 will receive full consideration.

For more information about the application process, see this page and stay tuned for a list of projects to be posted a the Summer 2024 tab.

Learning from Images

The amount of imaging data generated every day exceeds human imagination. With their ability to statistically analyze such large datasets, computational algorithms can enhance our ability to discover new patterns and improve imaging data quality for critical healthcare applications and beyond.

This theme’s projects provide new mathematical insights and algorithms enabling learning from image data. The goals of the individual projects include improving algorithms for learning the distribution of image data, optimizing the measurement design to improve image quality, developing efficient algorithms for reconstructing image sequences, and generalizing machine learning techniques to learn transformations between images.

The projects build upon and advance state-of-the-art techniques from machine learning, numerical linear algebra, and differential equations. Students will learn about these techniques and be trained to combine them in new ways to build effective algorithms. Their analysis and experiments will provide new insights into the strengths and weaknesses of their approaches.

Emory Welcomes 2023 REU/RET Participants on Campus

Tue, 13 Jun 2023 00:00:00 +0000

This is an exciting week for us as we finally entered the core phase of our 2023 REU/RET. We are delighted to have a diverse and talented group of participants joining us from different states and institutions. They will spend the next six weeks on Emory’s campus and will have access to state-of-the-art facilities, resources, and mentorship. We look forward to getting to know them and supporting them throughout this journey.

The theme of this year’s program is data for social justice and our five teams will explore various topics related to this theme from a mathematical perspective. Our projects span topics motivated by health disparities, environmental justice, and fairness in deep learning. The participants will study and develop mathematical algorithms and computational tools to analyze and solve open challenges in these areas. They will have plenty of opportunities to meet with their mentors and peers who will provide feedback and guidance.

In the next six weeks, our participants will not only work on their research projects, but also engage in a variety of activities that will enhance their skills and knowledge in data science. Our weekly seminar and ad-hoc lectures will train participants in most aspects of mathematical research in this area and give advice on career development.

We are excited to kick off this program and look forward to a productive and fun summer with our participants!

Participants present their work at the REU/RET Poster Session

Sun, 26 Jun 2022 00:00:00 +0000

In their last week on campus, our REU/RET teams presented their work in our annual poster session. The event was held in our beautiful atrium and was a magnet for many students, postdocs, and faculty from the Departments of Mathematics and Computer Science.

In preparation for the event and to advertise their work, the teams made two-minute Poster Blitz videos. Since the videos were full of creativity, we played them during the event using our smart screen. The videos are now hosted at:

During the event, more than a dozen judges had the difficult task of scoring the quality of the posters, how well the teams presented and explained their work, and the results of the research itself, to determine the winning team. All teams scored very well on and with the smallest of margins, the poster on reinforcement learning vs. optimal control won the award. The team consists of Arjun Sethi-Olowin (a rising senior at Rice University), Dewan Chowdhury (a rising junior at Rutgers University), and Jacob Mantooth (a rising senior at East Central University). They each received a $50 gift card from a major online retailer.

But no one left empty-handed! Our official REU/RET t-shirts, designed by our outreach committee, arrived just in time. Also, we held a raffle with Emory and SIAM merchandise. Here, Jim Nagy presents one of the main prizes, a pair of SIAM socks, going to Logan Knudsen:

We thank the Department of Mathematics for making the poster session a huge hit and providing the prizes.

Emory Welcomes 2022 REU/RET Participants on Campus

Mon, 13 Jun 2022 00:00:00 +0000

Today marks the first day of our core REU/RET phase 2022, and we are delighted to welcome our participants from across the country to our beautiful campus. This year on our site, 22 undergraduate researchers and four teachers will work on eight projects under the theme Models meet Data. All projects will combine tools from computational mathematics (modeling, differential equations, optimization, linear algebra) with statistical and data science techniques (deep learning, reinforcement learning, data assimilation) to tackle challenging research questions.

Over the next six weeks, the groups will have several milestones which give opportunities to practice most modes of scientific communication. In their research contracts, the participants will organize their collaboration and develop a timeline for the project. Their midterm presentations give an excellent opportunity to expose synergies and ideas across the teams. The participants will also create a blog post about their work, and there will be a poster session toward the end of the program. Throughout the six weeks, the teams will work on a manuscript that describes their project.

In addition to working on this packed research schedule, we will have many events to foster learning (e.g., ad-hoc lectures, a weekly seminar), networking (e.g., excursions and other social events organized by the participants), and professional development (e.g., scientific writing workshops, career panels).

Besides the NSF support, we are also grateful for the support from Emory’s Department of Mathematics, which provides the top-notch facilities to run this program smoothly and funds to host events such as this morning’s breakfast. In addition to one co-working area, a computer pool, and two seminar rooms dedicated to our site, we’ll also try out our department’s computing facilities and new digital whiteboard.

REU Participants Receive National Fellowships

Fri, 20 May 2022 00:00:00 +0000

We are excited to congratulate Manuel Santana and Katie Keegan, who participated in our program in the summer of 2021, on winning nationally-competitive scholarships that will accelerate enhance their graduate school career.

Manuel Santana, a 2021 Goldwater Scholar, who graduated with a BA in Computational Mathematics from Utah State University was awarded a prestigious scholarship through the NSF’s Graduate Research Fellowship Program (GRFP). Besides his research on computational methods that accelerate the reconstruction of computed tomography on our site, he also worked on combinatorics, and shared many tutorials for solving algebra problems. He will join Cal Tech for his PhD.

Katie Keegan was offered not only an NSF GRFP but also a [Computational Science Graduate Fellowship by the US Department of Energy]0(https://www.krellinst.org/csgf/), which she accepted. Katie completed her BS in Applied Mathematics at Mary Baldwin College this spring and participated in two REUs. In the summer of 2020, she was part of ICERM’s REU and published a paper on a modified watermarking scheme based on the SVD, which was later also featured in SIAM News. On our site she developed a tensor SVD-based classification algorithm for fMRI data. We are extremely delighted that Katie will join our Computational Mathematics PhD program starting this fall.

2022 Projects and Mentors Announced

Thu, 24 Feb 2022 00:00:00 +0000

With the application deadline for the summer 2022 program approaching, we’re delighted to announce the project topics and mentors. This year, we will have six projects centered on the theme models meet data. Applications include storm surge modeling and neuroscience and mathematical techniques cover a variety of topics ranging from data assimilation to deep neural networks and reinforcement learning. The teams will be mentored by faculty from Emory’s department of Mathematics and Computer Science and two former faculty and students who will join us as guest mentors. A full list of topics can be found on the Summer 2022 page. Applications for the REU and RET are due on March 1.