What does it take to get some of Johns Hopkins Medicine’s greatest minds—from many different specialties—to sync their schedules for an hour together every week? The cancer with the lowest survival rate in America.

“This is our Manhattan Project,” says radiologist Elliot Fishman. “We’ve declared war on pancreatic cancer.”

With the fancifully monikered Felix Project (so named for the magical potion in the Harry Potter series that brings success to its drinkers), Fishman and this team of medicine’s heaviest hitters are harnessing the latest technology to catch pancreatic tumors earlier, when they’re still operable.

They face a formidable foe. Every year, more than 50,000 Americans are diagnosed with pancreatic cancer. Some 93 percent of patients will die within five years—mostly because their disease wasn’t caught in time. Though still relatively rare, pancreatic cancer is increasing in the United States as the population gets older and more obese, both risk factors for the disease.

Several years ago, the Lustgarten Foundation, whose sole focus is defeating pancreatic cancer, turned to early detection pioneer Bert Vogelstein for project ideas. They wanted something futuristic, with real promise. Vogelstein, who has been at Johns Hopkins since he began his M.D. here in 1970, was the first to discover the genetic basis of a common cancer (colorectal). His work has shaped our modern understanding of cancer pathogenesis and has led to breakthroughs such as the sequencing of “cancer genomes” and the development of blood tests for early cancer detection.

Vogelstein didn’t need to think long before proposing an idea: Use artificial intelligence (AI) to train computers to spot tumors in CT scans when the tumors are still small and localized. The project would require expertise in radiology and computer science. But he knew he could rely on some of the world’s greatest minds in these areas without even leaving Johns Hopkins.

The Felix Project

Fishman diagnoses hundreds of cases of pancreatic cancer each year as director of diagnostic imaging and body CT at Johns Hopkins, one of the largest pancreatic cancer centers in the world.

Another Johns Hopkins lifer, Fishman has been transforming the field of radiology for nearly 40 years, especially through his early adoption of new technologies, like the internet and 3-D imaging.

Pancreatic cancer is hard to catch for a number of reasons, he explains. The pancreas, whose job is to produce insulin and digestive enzymes, is a tadpole-shaped organ located deep in the abdomen, surrounded by other organs. Its size, shape and position vary among individuals, and initial symptoms of a tumor can be rather vague, like back pain or gastrointestinal issues. There’s also pancreatic cancer’s relative rarity, which means most radiologists don’t see many cases each year.

In February 2016, as part of a speaker series he hosts annually to introduce out-of-the-box ideas, Fishman brought in Jensen Huang, CEO of NVIDIA. That’s the company that invented the graphics processing unit in 1999, which sparked the growth of the PC gaming market and redefined modern computer graphics. Today, NVIDIA is at the forefront of AI.

After the talk, Huang and Fishman got chatting with attendee Alan Yuille, a mathematician and computer scientist who had just joined the faculty at The Johns Hopkins University as a Bloomberg Distinguished Professor. Yuille is an expert on deep learning, a form of AI that’s well adapted for processing images, like CT scans. As Fishman remembers it, Huang declared, “Alan, you need to work with Elliot; Elliot, you need to work with Alan.”

And that’s how the Felix Project was born. Credit for the name goes to Karen Horton, who enjoys reading the Harry Potter books with her daughter. Horton has been at Johns Hopkins since she began medical school here in 1988 and is known for her work in CT imaging and the processing of 3-D data. She was named director of the Russell H. Morgan Department of Radiology and Radiological Science in 2017.

Of course, the project’s success won’t be due to magic or wizardry—instead, it will rely on the laser focus and close collaboration of two teams of scientists: radiologists and computer engineers.

Color-Coding the Abdomen

Deep learning, a type of machine learning, is the dominant method of AI used for interpreting images. At its core, it’s a computing technique that mimics some functions of the human brain. The data is parsed by artificial “neural networks,” layers of interconnected computer algorithms that gradually reveal useful information, as output from one layer becomes input for the next.

As with all machine learning, deep learning algorithms are designed to improve over time by adapting their models to make better predictions. The more data sets they process, the better they get. Although, Fishman warns, “Garbage in, garbage out.”

Yuille explains that human annotators on the Felix Project team specify the “ground truth” for the algorithms. In this case, that means accurately labeling every voxel, or three-dimensional pixel, of a CT image of the abdomen as healthy organ or tumor.

Using a digital “paintbrush,” retired radiation oncologist Eva Zinreich and five other teammates color-code 28 organs in every scan, including the pancreas, gallbladder, spleen, kidneys, intestines and blood vessels. Each scan takes two to three hours to label, with three-dimensional organ boundaries being checked from the axial, sagittal and coronal points of view.

To begin annotating a new scan, Zinreich checks a box next to the organ she will label: the stomach. The cursor’s arrow becomes a solid green circle. She moves it to the edge of the stomach and holds down the mouse button as she follows the contour of the organ. (It’s not quite as hard as signing your name with a mouse because the high-tech paintbrush senses the edges of things.)

So far, the team has annotated 1,140 normal abdominal scans and 1,000 scans from patients with pancreatic ductal adenocarcinoma (PDAC), the predominant kind of pancreatic cancer. Now they are working on scans of pancreatic endocrine tumors, which are more challenging to catch because they are generally smaller than PDACs and more vascular, but just as crucial.

Now 72, Zinreich volunteers her time “for fun.” She was invited to join the project by Fishman, who was a resident under her mentorship at Sinai Hospital many years ago.

Back in the 1970s, she started doing similar work manually, using millimeter graph paper to draw out plans for cancer radiation treatments. “I didn’t grow up with computers,” she says. “It's very impressive to see and be involved in the evolution of deep learning.”

Weekly Meeting of the Minds

Equipped with the annotated scans, Yuille and his team of five computer science graduate students and postdoctoral fellows train and test the deep network algorithms.

“At present, humans outperform AI on almost everything. But AI is good for tasks in medical image analysis because the human visual system evolved to deal with real-world, 2-D images, not with 3-D CT images,” Yuille says.

The computer’s current success rate in identifying pancreatic ductal adenocarcinoma is nearly 100 percent when working with annotated images from The Johns Hopkins Hospital. That’s a lot better than the 10 to 30 percent “miss rate” in pancreatic cancer identification that occurs with human radiologists. But Yuille says the real test will be whether their system works on data from other hospitals, whose machines and methods vary.

Of course, the team’s “catch” rate wasn’t always so high, and there are still some instances when a human-annotated scan differs from the computer’s prediction. These are the cases the AI team presents to the radiologists at their weekly meetings.

At a meeting in mid-February, Vogelstein, Fishman, Horton and Yuille sit around a conference room table in the outpatient center. (Pathologist Ralph Hruban and molecular geneticist Ken Kinzler were away that week.) As many as 15 other members of the team, including radiologist Linda Chu, pull up chairs nearby.

On a screen on the wall, graduate student Fengze Liu presents his progress on an algorithm to detect dilation of the pancreatic duct. This happens when a tumor obstructs the duct, causing the enzymatic juices to collect upstream. Fishman says duct dilation is the number one sign missed by radiologists. To account for natural variation in the size and shape of the duct, Liu applied a “scale-space distance transform,” which sets up a coordinate system based on the “skeleton” of any given pancreas. To predict a tumor, the algorithm then effectively calculates the radius of the duct at each point and looks for truncations where the radius is inflated and then abruptly narrower downstream. Next, Liu shows several discrepancy cases. One gets thrown out because it’s clear to the radiologists the patient had a postoperative drain in place. Another, labeled a false positive, detected a real tumor, but it was in the kidney, not the pancreas. (Fishman’s suspicion is that the computer found a rare case of Von Hippel-Lindau disease.)

Of most concern are the false negatives. Yuille says these tumors are usually missed by the computers because of their small size or unusual shape, position, or density. “It is precisely by correcting these errors that the algorithms get better,” he says.

A Fruitful Marriage

What’s striking is that the AI team members can’t refine their algorithms without the guidance of the radiologists, and without the computer scientists, the radiologists are left with a bunch of color-coded images.

Hruban revels in the unique learning opportunity that the Felix Project has been for everyone. Director of the Sol Goldman Pancreatic Cancer Research Center at Johns Hopkins, he received his medical degree from Johns Hopkins in 1985 and joined the faculty here five years later. He began his pathology career assessing heart and lung transplants but was drawn to pancreatic cancer when John Cameron, then head of surgery, started doing the complex Whipple procedure to remove cancerous pancreatic tissue. “He asked me to verify what he was removing, and I fell in love,” he says.

“At the beginning of the Felix Project, the AI team didn’t know what a pancreas looked like, so I brought in a model and showed them,” says Hruban. “They had wonderful, insightful questions. Then we, the doctors, had to learn lots about computer algorithms and AI.”

Vogelstein, who’s neither a radiologist nor a computer scientist, considers his role to be that of “marriage broker.”

“In truth, we humans cannot create from nothing,” he says. “What passes for human creativity is connecting two things that have never been connected before. What Alan and Elliot are creating is something neither of them could have created alone.”

Their final product will be a software program that analyzes CT images and flags suspicious areas. It won’t replace the radiologist but will be like always having an expert consultant on hand. Fishman believes they are about two years away from starting a new company to develop and market their software or partnering with a well-established company. Either way, a more effective way to spot pancreatic cancer in its earliest stages is going to happen, he says.

“The Lustgarten Foundation is rightly demanding something that saves lives,” says Fishman. “Nearly one-third of the cases I see could have been detected four to 12 months earlier by a discerning radiologist. That’s our window of opportunity.”