Carola Schönlieb is an applied mathematician at the University of Cambridge.

She’s also a Turing Fellow at the Alan Turing Institute and the head of the Image Analysis group at Cambridge’s Department of Applied Mathematics and Theoretical Physics.

---

---

Subscribe

iTunes
Breaker
Google Play
Stitcher
SoundCloud
RSS

---

Transcript

Craig Cannon [00:00] – Hey, how’s it going? This is Craig Cannon and you’re listening to Y Combinator’s podcast. Today’s episode is with Carola Schönlieb. Carola is an applied mathematician at the University of Cambridge. She’s also a Turing fellow at the Alan Turing Institute, and the head of the Image Analysis Group at Cambridge’s department of Applied Mathematics and Theoretical Physics. In this episode we cover mathematical approaches to image processing. Alright, here we go. We ought to start with a little bit of your background. What did you start researching and what are you researching now?

Carola Schönlieb [00:34] – I started out my research in mathematics in Austria, Vienna, where I actually didn’t look at image processing or imaging at all. I started out with so-called partial differential equations, which are equations of a function and its derivatives. You’d take an express change over time or space and they are models for various natural phenomena in physics and biology. Lots of things are explained why it is differential equations and my first paper, again, had nothing to do with image process, it was actually on Cahn-Hilliard equation which is an equation that describes phase separation and coarsening in alloys, in metallic alloys, for instance. When you cool them down to a certain temperature, you have a mixture of two and if you cool them down to a certain temperature, they are starting to separate from each other and coursing out and build these larger areas.

Craig Cannon [01:29] – Yeah.

Carola Schönlieb [01:32] – There is an equation that models this kind of phenomenon which is the Cahn-Hilliard equation.

Craig Cannon [01:37] – Okay.

Carola Schönlieb [01:37] – My first paper was on the stability analysis of a certain type of solutions to this Cahn-Hilliard equation. Stability analysis meaning that, if you perturbed your initial condition a little bit, how much is your stationary solution, that is when you let time evolve infinitely–

Craig Cannon [01:56] – Okay.

Carola Schönlieb [01:57] – How, when a stationary state is a state of where the system is in no change. How much do these stationary states differ from each other when you just perturb the initial condition a little bit.

Craig Cannon [02:11] – This is in the context of creating alloys or building alloys for structures or was there any particular purpose?

Carola Schönlieb [02:17] – Well, the purposes, a lot with these differential equations is to simulate certain phenomena. If you understand how stable these stationary states are, so if you’re at a stationary state and then, you perturb the stationary state a little bit, is it going back to the same stationary state? Or is it going somewhere completely different, so you understand how these systems react to perturbations that are naturally occurring because we are in real life and things happen.

Craig Cannon [02:47] – Gotcha.

Carola Schönlieb [02:48] – It’s more an understand of the the physical processes involved in mixture of alloys, for instance, or things like that.

Craig Cannon [02:57] – Were you at a technical university where you would be focusing on alloys or this was a personal interest–

Carola Schönlieb [03:02] – Not at all, it was just, actually, you know, a lot of applied mathematics on the continent which is everything else in the U.K., basically, here in Europe, is applied mathematics very much means that what you’re doing is inspired by applications, but eventually you end up with a mathematical problem. It was really the driving factor was, while we were interesting in analyzing this equation and there were techniques coming up that were kind of cool, yeah, so it was just kind of an intellectual interest in this equation. It was the driving factor for this particular paper. But then, during writing this paper, the research at UCLA, researchers at UCLA, in particular, the group of Andrea Bertozzi used this same equation to do image restoration. And image restoration meaning you have a digital image and there are parts of this image which are damaged for some reason or which are… Where you have objects which are occluding some other object of interest that you want to get rid of the occlusion or something like that, so you have one part on the image–

Carola Schönlieb [04:20] – That you somehow want to replace by something that is suggested by the surrounding area of this region.

Craig Cannon [04:29] – Is this similar to content aware fill in Photoshop?

Carola Schönlieb [04:33] – Exactly.

Craig Cannon [04:33] – Okay.

Carola Schönlieb [04:34] – Yeah, exactly.

Craig Cannon [04:35] – But this predates the Photoshop development, I assume.

Carola Schönlieb [04:38] – It actually does and, I mean, also the content aware fill is actually very much based on some of the things that have been initiated by people like Andrea Bertozzi.

Craig Cannon [04:47] – Okay.

Carola Schönlieb [04:48] – And so, the technique is different in what Photoshop is using but it’s still based on research in mathematics, in fact. It’s a differential equation, maybe, if you wish, that is more, it’s not the Cahn-Hilliard equation, it is a different type of differential equation that is not local, it is taking patches in images and kind of copy and pasting them into the region that you want to replace. But anyway, she used the Cahn-Hilliard equation to do that and that was the kind of eye opening moment and I moved into image processing still sticking to differential equations at the time. Actually looking at image restoration, so at this Photoshop content aware fill–

Carola Schönlieb [05:36] – Type problem. That was basically my PhD. My PhD was about image restoration. During my post-doc, I moved more and more into what is called inverse imaging problems where what you are observing or what you’re measuring in the first place is not an image, like when you take a photo, the digital image is an image. But there are certain applications like in biomedical imaging, where what you’re observing is not an image directly but is some transform of this image. Like an image tomography, for instance.

Carola Schönlieb [06:18] – Think about CT, for instance, computer tomography, what the CT, what to tomograph is measuring are projections of your three dimensional object which is whatever you have in your body. And from that, you want to reconstruct the object. Projections, meaning in the CT sense, a particular sense, which that you send x-rays through the body and what you’re measuring, so what you’re sending them through, what you’re measuring at the other end is the attenuation that they feel when the travel through the body, depending on which type of tissues they hit. That’s what you’re measuring on the other end and you can model that by saying what you’re measuring is a line, is an integral along the line that the x-ray takes through your body where you’re integrating over the attenuation that it feels.

Carola Schönlieb [07:14] – From that, and that is a very old problem, it goes back to Radon transform, what you’re measuring is not an image but it’s the Radon transform of your image. Which are line integrals or an image density that you want to reconstruct. The object consists and the density is different in different parts of your body and then, you can see organs in your body and stuff like that.

Craig Cannon [07:35] – Right, and so, the likelihood of there to be some amount of it missing that you need to fill or recreate or de-noise is much higher–

Carola Schönlieb [07:42] – It’s very likely.

Craig Cannon [07:43] – Than an image, and it’s obvious.

Carola Schönlieb [07:45] – That’s quite obvious because, first of all, we are in a finite dimensional world, so you don’t have all possible, infinitely many line integrals of your body measured. And then, it’s not even… That would be still okay if you’re measuring as many line integrals as you’re corresponding to the resolution of the image that you, then, want to compute from these line integrals but, then, very often it’s not like that because you don’t want to, you want a very high resolution image because you want to look at all the details in the body. But, you don’t want to measure so many line integrals because you don’t want to radiate the patient so much, you don’t want to send tons of x-rays through the patient. You have a lack of data, you don’t have as much data as you want for a high resolution image to reconstruct and then there is noise because these are measurements.

Carola Schönlieb [08:48] – And there is always noise in measurements.

Craig Cannon [08:51] – Were you doing de-noising work as well, at the same time?

Carola Schönlieb [08:54] – It’s integrated in the reconstruction approach, so in the mathematical algorithm that reconstructs the image or the three dimensional inside of your body from these line measurements, there is the denoising is integrated into this reconstruction step coming from these line integrals reconstructing–

Craig Cannon [09:23] – Gotcha. What I know about denoising mostly through audio Fourier transform and that kind of thing. How are you doing it with an image, how are you de-noising in the algorithm?

Carola Schönlieb [09:34] – So with images, it depends of what you think is important in an image, that will determine how you’re going to denoise it. A very successful assumption that has been made for designing image denoising approaches is and has been and still is that the most important information that visually guides you of what this image is showing you but also that helps you if you, later, want to quantify something in the image, are the edges in the image, this is the most important thing where are boundaries between different objects. When you think about it, what really makes an impression on you of what this image shows are colors, and the boundary between these colors. Where are the colors changing and these are the edges in the image.

Craig Cannon [10:32] – Interesting.

Carola Schönlieb [10:33] – And to preserve those and not make them blurry, blurred out, is something that a lot of research in image denoising has gone into. Image denoising methods which can preserve edges in an image, and so, the Fourier, Fourier type techniques are good, they can smooth out the noise–

Craig Cannon [11:02] – Yeah, it’s dangerous.

Carola Schönlieb [11:03] – By taking away the high frequencies but they will take away the high frequencies everywhere.

Craig Cannon [11:07] – Right.

Carola Schönlieb [11:07] – Which means they will also take away the high frequencies that corresponds to edges where the image function is changing rapidly.

Craig Cannon [11:12] – Yeah, so you’re looking for the delta–

Carola Schönlieb [11:15] – This is a very high frequency component of your image but this is a component you would like to keep. You want to differentiate between the high frequency components in your image which are just noise and the high frequency components which correspond to these very characteristic features that you want to keep. There are various techniques but one very successful one is total variation regularization, for instance, which is a technique that has been used a lot by people in image denoising that models this assumption that you have sharpness continuities. Median filtering is a, maybe, simpler thing to understand or that people might have heard about which is not exactly total variation denoising but it’s related.

Carola Schönlieb [12:03] – So median filtering instead of Gaussian filtering. Maybe, where Gaussian filtering corresponds to your Fourier, taking away the high frequency–

Craig Cannon [12:11] – Oh okay, gotcha. It’s so funny, when I was doing Photoshop at the Onion, we were always actually interested in blurring edges because on of the most obvious things to spot a Photoshop is a sharp edge and a soft edge in the same photo.

Carola Schönlieb [12:23] – Okay.

Craig Cannon [12:25] – For instance, if I were to cut you out and then put you in front of the white house, if the photo has a slight blur, like, the depth of field in the photo is, say, a 1.4 aperture which creates a very, very, shallow depth of field, so there’s a lot of blur, but if you’re crispy, someone can immediately spot that you were dropped into the photo. It was all about blurring the edges–

Carola Schönlieb [12:46] – Oh, interesting.

Craig Cannon [12:47] – To trick someone into thinking that it was in the same photo.

Carola Schönlieb [12:48] – Yeah, okay.

Craig Cannon [12:51] – In your context, these algorithms that will handle the edge sharpness, are they hand coded or are you using machine learning to create them, how does that work?

Carola Schönlieb [13:02] – They are classically hand coded. This is maybe something that is now, more and more being replaced by other things where image denoising, nowadays the best image denoising approaches are actually coming from deep neural networks. These hand crafted methods get more and more beaten in terms of performance by some of these neural network approaches. They get beaten in certain scenarios though.

Carola Schönlieb [13:38] – They get beaten on the type of examples they have seen already or similar type of images that they have seen already, right. If you present them with something completely different, if you only train them one photographs of animals or whatever– And then, you present them with a CT image of the CT scan, they will not be able to handle that. That is one of the things where still hand crafted models have a certain justification of existence, in a sense. Because there is not, there is still, you know, we can do GPU programming and everything, there is still not enough computation power to train a machine to know everything. To learn everything about the world.

Carola Schönlieb [14:28] – While in certain scenarios, if you know what you want to apply your image denoising–

Craig Cannon [14:35] – Well, it’s like the ImageNet thing from almost 10 years ago.

Carola Schönlieb [14:37] – Exactly. If you know that, then, it’s fine. And that’s good.

Carola Schönlieb [14:42] – Think about, for instance, one big thing in CT, let’s say, or in different types of biomedical imaging, let’s say, MRI, magnetic resonance tomography, the type of image that you get, the resolution, the contrast and everything very much depends on how you do the acquisition. How many, let’s say in the CT case, how many x-rays you have been shooting through the patient, but also, and that is actually connected to what I just said also, the type of scanner you’re using. Are you using GE or Siemens or Toshiba or whatever? They have different settings. And they have different ways of going from the measurements to an image. If you train an algorithm, for instance, a neural network on one of these scanners, it doesn’t mean that it works on images of another scanner.

Craig Cannon [15:47] – Really, so they’re producing entirely different data? I thought they were just basically the same tools inside with a different logo.

Carola Schönlieb [15:54] – Well, so, this is the other interesting thing, it’s not entirely different, right, you might not spot also visually what the difference is but this is one of the things that also people start, more and more, hopefully, start to do some research in understanding this that even small perturbances that are consistent. Small differences that are consistent between the scanners might contribute to your algorithm then failing. I don’t know if you have seen these adversarial errors where you do a little perturbation and then, all of a sudden, it classifies image into something completely different. The really very exciting and, for mathematicians and particularly exciting opportunity that neural networks are now offering in contrast to these hand crafted models–

Carola Schönlieb [16:56] – Are that they can go beyond just saying, I want an algorithm that preserves edges. Which is a very simplistic view of the world. But on the other hand, there are a lot of unknowns in this algorithms, on the one hand, that mathematicians should be exploring and try to bring some of the analysis and some of the methodologies that help us to understand why these hand crafted models work because we can prove properties about the denoising abilities of these methods of how stable they are, for instance, to perturbations in the images, we know how that works, so we can prove things about that, we have error estimates and things like this and to bring over to neural networks, is very exciting but for that, bringing some structure into these neural networks is also important. On the other hand, when you think about these neural networks having these 100, millions of parameters that are adapting themselves to the data, maybe, in some case, it would be better to not have a million parameters but have an intelligent, structural

Carola Schönlieb [18:17] – way of reducing the search space and, as such, bring some structure into the problem which helps you make statements about stability and things like that.

Craig Cannon [18:30] – Then, and also, statements about what the algorithm is actually doing and what it’s thinking.

Carola Schönlieb [18:34] – Statements about what the algorithm is doing. That is another thing, right? When you look at these hand crafted models, you have started with a hypothesis. You have started with a hypothesis of edges are important in images. And then, you come up with a mathematics algorithm that is exactly doing what you want it to do, right. Or, then, you have to make sure that it’s actually doing what you want it to do.

Craig Cannon [19:03] – If it doesn’t, then that code is bad, then it’s not–

Carola Schönlieb [19:05] – The code is bad, or your model is bad, maybe you have to change your model in a certain way. Okay. But you understand why things are happening. If you have millions of parameters and then, you train this algorithm to do something and then, you get a parametrization that has one million different parameters, how are you ever going to interpret that? There are ways where machine learning people are trying to interpret classification results, for instance, you have these salient features that you can detect an image what was important for the classification to do this or this. But it’s still limited. There are lots of very cool opportunities.

Craig Cannon [19:53] – Are you guys working on hand stitching to two together, at this point, what’s the status of the current research?

Carola Schönlieb [19:59] – Different people are trying to do different things. I can first tell you what I’ve been doing over the last couple of years. The last couple of years, what I’ve been doing is trying to, starting with these more hand crafted models, nothing to do yet with neural networks. I started with the hand crafted models and then, for certain parts in these models where I wasn’t quite sure about our edges, really, the only thing I’m looking for, for instance, I’ve tried to parametrize them in a certain way. But not with a million parameters, but maybe with 10 parameters or something like this, and then, learn these parameters from actual examples that I would like my hand crafted model to spit out. And this is what we call bi-level optimization or parameter estimation. People have been doing this for a long time but, now, I think the motivation comes more from, you know, there’s a certain interpretation in terms of machine learning that is kind of exciting where people are…

Carola Schönlieb [21:10] – More interested in. This is one way and levels of parametrization vary in this context but the good thing is, you have a hand crafted model in the end that you still understand. And, that you can still prove things about, you still have guarantees on your solution. You have guarantees that if you, you don’t have these adversarial errors that if you perturb a little bit, you get a completely different result. This is really something you don’t want. The other thing is, and this is more blue sky, and this actually goes a little bit against what I said before.

Carola Schönlieb [21:52] – Which is, we have been starting to use deep neural networks for problems in computer tomography, for instance. At that moment, we cannot prove a lot of things, but we can see some ways of how to combine these more hand crafted models with neural networks in the sense of what you feed them with, for instance. The prior information you feed them with, the data, maybe not just the measurements but, maybe also, the information that the measurements are actually line integrals of the 3D object that you want to construct. And doing this in a kind of iterated fashion where you always go back to the fact that, ah, actually remember neural network, these are line measurements that I’m feeding you with, remember this and then you do another sweep through a neural network–

Craig Cannon [22:46] – But then, how does that work in the context of building a model around, say… I don’t even know and MRI how many images are created or lines are monitored, but say you have 10,000 images, but you want to create a combination of a hand coded algorithm and a machine learning system, how do you go about tagging all that stuff?

Carola Schönlieb [23:10] – What do you mean exactly, how are you going to–

Craig Cannon [23:12] – What I understand that you’re saying is you’re giving it more data than just the original source material.

Carola Schönlieb [23:18] – Yes.

Craig Cannon [23:20] – How do you do that at larger scale?

Carola Schönlieb [23:25] – Ah, computational, you mean?

Craig Cannon [23:27] – Yeah.

Carola Schönlieb [23:28] – Computationally, we’re doing this in a sequential manner.

Craig Cannon [23:32] – You can do it in different ways but in a sequential manner means that you’re not feeding it 10,000 images at the same time, but you’re doing it bit by bit and you’re adapting your objective towards this.

Carola Schönlieb [23:47] – Another thing about computational performance is also, of course, that the optimization that is underlying but this is not just the problem that we have, it is a problem that neural networks have in general is that you do not necessarily need to solve your optimization problem, your training exactly, and maybe sometimes or most of the time, you actually don’t want it, want to save it exactly because you only have a finite amount of training examples. And so, when you think about what these neural networks are doing, they’re trying to minimize a loss over the training examples that you have. But this loss is only an approximation of many, many, many more images that you want your neural network to work for. And so, very often, you do not want to solve that exactly, you don’t want to minimize your loss exactly for this training set.

Carola Schönlieb [24:41] – There are different types of optimization methods that people are using but the main thing in machine learning is to cost optimization so you don’t minimize exactly for all the variables you have but you randomly pick a certain amount in every sweep through the network that you’re optimizing for and then you randomly change which ones you’re optimizing the next sweep and so on.

Craig Cannon [25:03] – Just so I understand, minimizing loss, why don’t you want to do that?

Carola Schönlieb [25:10] – What you’re minimizing… The loss, let’s say, could be the least squares error. Let’s go back to denoising, let’s say you want to train you neural network to optimally denoise images by saying, for this training set where I have both noisy and clean images– I want that, if I sum over the difference between the denoised image, so you feed your neural network with a noisy image, it gives you a denoised image, you want that this denoised image is closest and the least square sense to the clean image that you know in this case, because you have a training set, you have a label, you have a true label for this noisy image which the label, in this case, is your ground truth image.

Carola Schönlieb [26:08] – And you want your denoising method which is this neural network to produce denoised images such that all of them are, in these least square sense, closest to the original label, to the ground label which is the clean image and you want that to work over all the images in a training set.

Craig Cannon [26:29] – Gotcha, okay.

Carola Schönlieb [26:32] – Let’s say you have 10,000 of these images that you both know the clean and the noisy image, if you would perfectly fit to this training set, if you would perfectly minimize this loss function, you could think and, again, people are not really understanding this and I also don’t really understand this but, conceptually, the idea is what you actually want to minimize is not the loss just over the training set but it’s the loss over an infinite amount of images which you then want to denoise. But you don’t have all these infinite amount of images. Why would you want to very accurately minimize the loss of this finite amount of images? Maybe you don’t. Maybe you only approximately want such that you still have freedom– Such that it could be optimal, also, for more images that you don’t have–

Craig Cannon [27:28] – In other words, you can train it on the wrong thing and it could only work for denoising photos of apple trees.

Carola Schönlieb [27:34] – Exactly.

Craig Cannon [27:35] – Then you’re in the same place that you were in the beginning.

Carola Schönlieb [27:37] – Exactly. The idea is, if you only do it approximately, you might be able to generalize it more. There are some attempts to understand this, but I’m hand-waving here because I can’t really say anything mathematically– About that.

Craig Cannon [27:55] – But have you pushed your research into practical applications, at this point. Are you working with companies or student groups or anyone else?

Carola Schönlieb [28:05] – My main collaborations are actually people in academia but from other disciplines. We have been collaborating a lot, in recent years, with people in the hospital, in the university hospital in Cambridge with clinicians and medical physicists. Different types of applications, one of the things that I said before is that I got more and more interested in these problems where you don’t measure an image directly but only indirectly via these x-rays, for instance. Developing algorithms which can get the most out of a very limited amount of data, the most out of in terms of very high resolution images, is something we have been collaborating a lot with people in magnetic resonance tomography, in particular in the Addenbrooke’s Hospital which is the local Cambridge hospital here but also with people in chemical engineering where one of the driving factors for people in chemical engineering is, for instance… There is a group here which is the Magnetic Resonance Research Center where they look, in particular, at processes which are dynamic, so they have these tubes filled with water and then, they pump certain things through and they want to understand what the dynamics of this process are. Now, if you think about not just having a static 3D object but having something that changes over time as well. Thinking back about how many x-rays not in magnetic resonance tomography, these are not x-rays but just going back to an example we had before.

Carola Schönlieb [29:45] – Not sending through so many x-rays means you don’t have a lot data to reconstruct. Which, now, if you wanted to track something dynamically, also means you’re not measuring a lot per time stamp. If you wat to have a very high resolution over time, it means per time stamp, you can’t acquire as much data as if you would have, if you just have one second for reconstructing your organ inside the body, at this particular time stamp, then the organ is moving again and you need to go the next time stamp and so on. You have less data for reconstructing each time stamp, as if you would have a static object and you would have 10 seconds to acquire this instead of one second, you can measure much more.

Carola Schönlieb [30:35] – Then you reconstruct just one image but now we have maybe you want to reconstruct not just one image in 10 seconds but 10 images because we want to see something evolving over time. Here are also the challenges of getting high resolution out of limited data. Another thing which is not connected to indirect measurements so much than these applications in magnetic resonance tomography is that we have collaborations with people in plant sciences for instance. They are interested in monitoring forest health or forest constituencies, let’s say, from airborne imaging data, so they fly mostly, in my collaboration, they fly, so not so much satellite but they fly over forest regions. And then, they acquire different types of imaging data. They acquire just photographs, aerial photographs. Hypo spectral imaging data or multi spectral imaging data which means you do not only have RGB but you have a broader range, you cover a broader range over the light spectrums. Also, the invisible light, so you don’t have just three channels but you have 200 channels or something in your image. And hyper spectral imaging is interesting, so the spectral component that you get from these measurements gives you an idea of what the material properties are of these trees. It tells you something about what–

Craig Cannon [32:07] – Really?

Carola Schönlieb [32:09] – The spectral component tells you something about the material that you’re looking at. The different materials have a different signature in the light spectrum of how they reflect light back, they have a different signature in the light spectrum.

Craig Cannon [32:26] – The intent would be to figure out, say for instance, an invasive tree that was taking over an area, they could figure that out just by flying right over it?

Carola Schönlieb [32:36] – Mm-hm. And then, the other thing, so this is one or two aerial photographs in hyper spectral imaging, and then, the third thing that they’re often acquiring are LiDAR measurements. Where you do not just get kind of a planar picture of the trees but you actually get a 3D model of the trees.

Carola Schönlieb [32:58] – This is also nice.

Craig Cannon [32:59] – I was just watching a documentary about that, about searching for Mayan ruins with LiDAR. Flying over the Yucatan peninsula or something. Essentially saying, hey, we can take 20 years for an archeologist to dig around in the dirt or we can just fly over it and look for the hard stuff and see what happens.

Carola Schönlieb [33:16] – Yeah, yeah.

Craig Cannon [33:18] – Very interesting. And are people also looking to this in the context of, for instance, denoising camera footage from anything, like security, on one hand?

Carola Schönlieb [33:30] – I haven’t done so much work in that myself but there are, of course, the CCTV cameras are everywhere.

Craig Cannon [33:36] – It’s kind of the terrifying output of figuring out this research, like being tracked everywhere like, in the U.K., in particular. I imagine people are looking to do this, right?

Carola Schönlieb [33:48] – You know, it’s quite funny because when you think about these crime TV shows, CSI Miami, whatever, there are always spy things–

Craig Cannon [34:00] – Enhance.

Carola Schönlieb [34:03] – You have a very pixelated image and you press a magic button and you can zoom in and all of a sudden, you can see everything. When you think of it, this is ridiculous. Of course, you can’t do that but you can do it now, maybe, if you have all these machine learning methods which have learned to look at just pixels and then know what is a very probable match in terms of high resolution, maybe, at some point you can do it, but then, you don’t know if you’re right or wrong, right?

Craig Cannon [34:43] – Just by chance, I was reading a New Yorker article from, I think, 2010 about this guy in Montreal allegedly finding 500 year old fingerprints using different kinds of spectral photography.

Carola Schönlieb [34:57] – Okay, cool. I haven’t heard about that– Tell me more about it.

Craig Cannon [35:03] – I don’t want to give away the whole thing but, then, there was an ensuing lawsuit, actually, from him to the New Yorker saying it was libel. But basically what happened is, he was accused of faking these fingerprints that may or may not have–

Carola Schönlieb [35:20] – Oh man, okay.

Craig Cannon [35:21] – And copying them from a real one, duplicating them on the back using proprietary methods to find them out. But you are interested in doing whether or whether not it’s legit, you want to work with–

Carola Schönlieb [35:36] – I hope so, I mean I’m going to tell people that it’s fake. Yeah, that’s the whole idea.

Craig Cannon [35:43] – Yeah, what direction are you going with art?

Carola Schönlieb [35:46] – When I, again, during my PhD, in Vienna, there was a collaboration we had with conservators who were looking at particular wall frescoes, at frescoes in and old apartment in the city center of Vienna which are called The Night… Frescoes, I’m not going more into detail but they were in the process of restoring these frescoes. That was my first-hand experience there and the idea was that, you know, it takes them a long time to physically restore these wall paintings. And once you have restored it, there is no way back, you have to decide what to do. Because then it sticks. Our idea was to help them by creating a virtual template of how the restoration could look if they do this or this or this.

Craig Cannon [36:56] – Yeah, because the important part is the frescoes is actually part of the wall, chemically, it’s not paint.

Carola Schönlieb [37:00] – Yes, exactly. Exactly, but even with paintings, if you do something, if you manually really, physically restore them… You’ve done it. I mean, you can still maybe try to go back but– You’re changing a historical piece… Of the world, right.

Carola Schönlieb [37:30] – Coming here to Cambridge, I got to know people in the Fitzwilliam Museum which is a museum here in Cambridge, and they’re interested in illuminated manuscripts. I met a very good colleague of mine who was the keeper of manuscripts in the Fitzwilliam Museum. Got interested in this idea of virtual restoration because illuminated manuscripts are so fragile that the culture is, you never physically restore them. You never physically restore them. If they get damaged, or altered over time, you leave it.

Craig Cannon [38:13] – Wow, okay.

Carola Schönlieb [38:15] – You leave them like this. There, the idea was, couldn’t we create a virtual restoration and exhibit the original manuscript and the virtual restoration next to each other. Last year there was an exhibition in the Fitzwilliam Museum which was called Color. In this exhibition, we had one piece which was a page of an illuminated manuscript which had been altered over time, actually, manually over painted. What we did was that we exhibited the manuscript and next to it, the virtual restoration where we took off the over paint. That has lead to other things, but this is kind of the idea that you don’t physically change something but you virtually do it. Which is, you know, nothing damaged, you just virtually create a digital copy of this manuscript and you play around with it.

Craig Cannon [39:14] – You’re not only going back in time to see, maybe, restoring it to its original vitality, it’s original color, but you’re actually going deeper into the layers–

Carola Schönlieb [39:25] – In this case, yes.

Craig Cannon [39:26] – This has been painted over. You can go farther in with imaging and then, you kind of apply you might already–

Carola Schönlieb [39:31] – Wow, that’s pretty cool.

Craig Cannon [39:34] – If someone’s really excited about this kind of research, if they want to get into it, what would you point them to, where should they get started?

Carola Schönlieb [39:43] – Depends what their background is.

Craig Cannon [39:46] – Yhey have like a CS degree, they’re interested in imaging, so they’re technical but they haven’t done anything in particular, in this field.

Carola Schönlieb [39:58] – What I would advise is to look… In particular, when you think about the U.S., I think some of the cool things that came out of image processing in the last couple of years were from UCLA. If you look at some of the applied math faculty there and some of the online lecture material or YouTube videos of some of their talks, I think that would be a good source to look at. Very classic names, Stan Osher, Andrea Bertozzi, I mentioned, Malik Parvona, Stefano Soatto. There are lots of people, there’s a… Not a name, in this case, I can tell you a few more things afterwards but just look for mathematical approaches to image processing, I think would be the first thing I would do. There are very good introductory books to look at, that explain a bit of the basics.

Craig Cannon [41:10] – Great.

Carola Schönlieb [41:11] – I would first start reading a little bit in these more general foundation books and then just starting from that, you immediately go to the more modern years recent years research. That would be a good way to start.

Craig Cannon [41:30] – They can catch up to you, maybe. Or apply here. Awesome, well thank you so much, thanks for making time.

Carola Schönlieb [41:36] – Thanks.

Craig Cannon [41:39] – Alright, thanks for listening. As always, you can find the transcript and the video at blog.ycombinator.com. If you have a second, it would be awesome to give us a rating and review wherever you find your podcasts. See you next time.