In the not-so-distant future, our mirrors may recognize us. First thing in the morning, seeing our faces, they might turn on the music we like, let us know the weather, or give us reminders about the day ahead. With technology being developed at Cornell Tech, our mirrors might also screen for disease—by scanning our skin for changes that could indicate cancer.
“We wanted to find something that would benefit your life, that you could do in three seconds every morning,” says master’s degree candidate Evan Kesten ’18, Connective Media at the Jacobs Technion-Cornell Institute at Cornell Tech. “An easy target was skin cancer. It affects one in five Americans over the course of their lifetime, and you can tell a lot about a malignant mole or anomaly just by visual inspection. We thought we could put computer vision and machine learning technology to use to tackle this disease.”
How Does Reflective Health Work?
With advising from Deborah Estrin, Computer Science at Cornell Tech and Healthcare Policy and Research at Weill Cornell Medicine, and Arnaud Sahuguet, director of The Foundry at Cornell Tech, Kesten and his team are now developing and honing their product, called Reflective Health. The current setup consists of a two-way mirror with a display monitor behind it, with the reflective ratio of the mirror allowing the user to see themselves as well as what’s on-screen behind the glass. The setup is also equipped with a Raspberry Pi, a computer the size of a credit card.
The idea is that Reflective Health will take photos of a user’s face and upper body every day, and algorithms will cull and crop the photos to create a profile for each mole. “So for each mole, there’ll be a historical record that shows what it looks like, the estimated size changes over time, discolorations, changes at the edges,” Kesten says.
Eventually, Reflective Health could provide the doctor with a kind of dashboard that displays significant changes. “It might say that out of 15 moles on patient A’s body, two of them look like they have higher than that patient’s average mole growth in the last six months,” Kesten says. This would give doctors more information and context than they get now with only annual or biennial visits.
Kesten and his team are well aware that having a patient’s bedroom or bathroom mirror take photos of them may seem like a dangerous invasion of privacy. To combat the dangers, the system will be designed to protect the identity of the user. Reflective Health will work without an internet connection, to prevent remote access or theft, and the system wouldn’t save original images, keeping only those cropped images that are relevant. That way, only an image of a mole would be shared with the user and doctor and not a view of the user standing at the bathroom mirror.
“It might say that out of 15 moles on patient A’s body, two of them look like they have higher than that patient’s average mole growth in the last six months.”
Refining the Technology
Kesten and his collaborators have built their second hardware prototype and are now spending much of their time honing the machine learning algorithms.
“We use a simple computer vision technique called a blob detector, which, as it sounds, detects blobs, sections of discoloration,” Kesten says. “But there’s a lot of noise—hair, stuff on your shirt, etcetera. So we take all of those blobs and essentially take little crops and run them through a machine learning model that’s meant to filter the images to just moles. That’s the part we’re putting a lot of work into now.”
One of the biggest challenges has been getting enough of the right kind of data to train their models to identify moles accurately. Recently, the team attained data from another study run by Oregon Health and Science University, which consisted of smartphone photos of over 3,000 moles. “Now we’re trying to make that data work for us to train our model,” Kesten says.
A Model for Future Projects
As Kesten’s team approaches graduation, they are hoping that future Cornell Tech students use Reflective Health as a foundation for their own research projects. The effort is one of a few projects that bridges two programs at the Jacobs Institute—Connective Media and Health Tech. Kesten’s partners, Jillian Sue ’18 and Yating Zhan ’18, are both completing Technion-Cornell dual master’s degrees in Health Tech this spring. “We think it’ll be a great way to keep combining the best parts of these two programs and have teams of students build out both sides, the user experience and the medical component,” Kesten says. “I think it has a lot of potential to grow as an internal project within Cornell Tech, until it reaches a point where it’s ready to be explored as a viable product.”
Connective Media—Thinking about the Humans Who’ll Use a Technology
Kesten is in his second year of the Jacobs Institute’s Connective Media program at Cornell Tech, which guides its students to develop technologies with an awareness of how they are used and integrated into people’s lives. “Most of the work I’d done before coming to Cornell Tech was simply about developing technology that was going to create value to a company. Connective Media trains software engineers, project managers, and entrepreneurs to think about the humans using this technology,” Kesten says.
The deep integration of technology in our lives is happening fast. Twenty years ago, for example, we might sit down to a desktop computer, logging on and off of the internet, a finite session. Now, many of us carry the internet around with us on smartphones. We are rarely logged off. To Kesten, it’s this integration of technology and human experience that provides vast and exciting opportunities.
“There’s a lot of potential that hasn’t been explored yet, potential to improve people’s lives,” he says. “I think everyone wants to do something that changes the world for the better in some small way, and technology seems to be a very clear way to do that in today’s world.”
From Cornell Tech to Startup
For Kesten, the next phase in the Cornell Tech curriculum, and possibly for him professionally, is developing a startup. Kesten has already had a taste of the startup culture. As an undergraduate at Binghamton University, he developed technology for a startup his friend founded. After graduation, however, he worked in the corporate sector.
“I’ve gone a short way down a lot of different paths,” he says. “But I think a startup is a place where, if you have a mission or a dream, you can put everything into it and make it happen.”
Kesten is remaining open-minded about the technology around which he might build a startup. He’s passionate about reducing waste—time, money, or other resources—but he isn’t sure where that passion will lead. He is confident that Cornell Tech is the best place to build an idea from the ground up.
“A lot of what we do at Cornell Tech lends itself very nicely to startup culture—product development, building something and getting people to use it, making sure you build something that has the maximum amount of value for the people you’re interested in helping,” he says. “What really drew me to Cornell Tech was this mission. They have a really unique perspective on what it means to get a graduate education.”
Kesten values Cornell Tech’s position in the New York City technology sector, its faculty, and its new campus. What is most striking to him is the integrated community. He remembers his first experience at Cornell Tech, seeing one big area where everyone was working together. “It was a really strong community where everyone, no matter what their major was, what they were interested in—whether business or software, operations research, law—everyone was there because of a passion for the industry of technology,” he says. “Those fields had been divided in my mind. Cornell Tech really brings them together, which is important because the truth is, in the real world, they are together. Everyone works together for the same goal.”