Project CETI: How to Decode the Language of Other Species - EMF 2026

[gasps] Thank you. All right. So, I figured this would be kind of a fitting in the- moment thing because EMF this year is themed on space and uh this is not a talk about space, but this is a talk about in some sense aliens. Uh so, project SETI is the citation translation initiative. Uh we are a group of scientists interdisciplinary from all over the world from universities from uh the west coast of the US to the UK. Uh we have folks at Berkeley, MIT, Oxford, um Hifa, everywhere, all over and all disciplines. Maybe not every single discipline, but we have uh folks who are um roboticists. We have marine biologists, of course. Uh we have machine learning specialists because you have to have AI. Uh we have um ethicists and legal scholars and I'll tell you a little bit more about that later in the talk, but it's an extremely diverse group of folks. And what we're really trying to do uh and citation translation initiative is a little bit of a backronym because we wanted to name ourselves after SETI the uh um extraterrestrial search for intelligent life. Uh but what we're really do is not so much translating as trying to understand what is it that whales and specifically sperm whales are saying to each other. Uh, and just a note about me, uh, I am a program manager on the group. So, I know a little bit about a lot of things. Um, but I'm not going to go super in-depth on any one thing because we have in-depth researchers who tell you that. So, this is going to be just an overview of what we're doing. I will show you lots of pictures and videos of sperm whales and tell you about some of the cool things that I discovered working on this project uh, and learned about by getting to work with all of these incredible people everywhere. Right. So, how did we end up with the citation translation initiative? Where where did you get a bunch of scientists to sign up to this kind of crazy wild sounding project? Uh in early 2020 before the pandemic hit, uh there were some folks who were on a fellowship at Harvard. Unfortunately, I wasn't one of them. Um but my now good friends were there together. And there was a um marine biologist named David Gruber, who is now leading the project. And uh this biologist named Shane working out in Dominica recording whales and doing lots of conservation work, had sent him these recordings of sperm whales, not really knowing what to do with them, not really sort of any agenda. They were just two biologists sharing information, um marine biologist working in the sea. and David was playing them in his office and someone walks by who happens to be a cryptographer. Uh some of you might be familiar with zero knowledge proofs. Uh a woman named Shawi Goldwer was one of the co-authors of that paper and she was the one who happened to be walking by and she went that sounds like code. That sounds like something that you could unencrypt. It sounds like Morse code. Uh, and so she popped her head and she said, "I think we should do something with this. Hi, I'm Shafi. I think we should do something with this." And then that's how this project started. Uh, and through those connections, both David and Shawi have people that they've obviously worked with over the years. They pulled together this incredible group of people. Um a and since then we've done a huge amount of trying to figure out can we actually look at what whales are doing and understand what they're saying. So okay why sperm whales exactly? [sighs] So sperm whales have the biggest brains of any animals on the planet. They are huge. They also communicate using sound over very very long distances. Um they have these complex societies. Uh and they interestingly we already knew before we started this project through Shane the biologist that was working in Dominica that they have really interesting cultural dialects. So, we knew that there was something going on with their language before we even started looking at it in detail because Shane, who spent 30 odd years listening to these whales and knows every whale in Dominica by name just by like its hump on the horizon, uh he's incredible. He had gone to different areas in the world like Sri Lanka, listen to different whales and gone, "Oh, they are speaking differently. I can recognize the patterns of the clicks well enough to know that whales in other parts of the world versus the whales that he knew and had these relationships with and Dominica uh were actually using different dialects. So we had an inkling already that sperm whale language was complex and that it would really be fruitful to research how sperm whales were communing to each other uh in as a way of trying to understand get a foothold into the animal communication world. Okay, how do they talk? So many of you might be familiar with this idea of whaling going to kill sperm whales um you know the Moby Dicks of the world for the spermicetti that lives inside their huge organ. Well that is actually what they use to talk or part of what they use to talk. So they have uh they force air through their big nostrils and then it actually reverberates back through the spermicetti organ and then forces out and makes clicks that are so loud that if you were next to a sperm whale when it clicked, it would actually uh probably damage your hearing pretty severely. So don't do that or or try not to do that. Um they also have these things called kodas. So these are distinct from the like clicks that they put out into the water to echolocate to try and find each other. A kota is actually like a distinct pattern of short and long clicks. So it's so that you can actually start to identify patterns. This thing that kind of sounds like Morse code. Um so they're making huge sounds out into the water. uh you can put a microphone, we call it a hydrophone, on the end of a stick off a boat and actually hear them from a very long ways away. So, let's see if we can show you a little bit of sperm whale speaking. Let's run that again. Oh. Okay, you can just about hear that there. They're just like making these clicks at each other as they go through the water. Let's just play it again. Okay, so that was a little bit jumpy, but it sounds a bit like uh and they're they're quite distinct. Let's just Sorry folks. All right, we'll just keep going through the slides. So the way that we work as a global institution is we are located in Dominica because sperm whales throughout the rest of the world they are migratory. They uh do long migratory distances but in Dominica it happens to be really special. Uh this is where a lot of sperm whales go to have babies. So they actually stay there year round and there's this one corridor in Dominica that you can actually find them. So we forged a pretty strong relationship with the local government in Dominica to build a research lab there. Uh and we also built an educational program. We have uh fellows who are from the local community college. They don't have a university in Dominica but they have a 2-year community institution. So we have a fellowship program for students from there to join our team and learn how to do field research. Uh and then we have this global network of researchers that we bring in. So we have um a linguistics team that's part of UC Berkeley and I'll go through a little bit of our work later. Uh we have a robotics team at Harvard who is figuring out the pretty difficult challenge of how do you actually record uh a underwater a massive massive animal that just dives really really deep really really fast to go hunt uh and then might hang out in the surface for a little bit and then dive and hang out the surface. Um how are you going to actually get this sound? We picked the possibly hardest animal to do data collection for this one. Uh and then what we are really trying to do is build an interdisciplinary group. Um so David who founded the project is marine biologist. uh there is a big sort of marine biology center to the thing but we have also all of these people who are building the tags figuring out how to deploy the tags figuring out how to handle the data in the data pipeline. Let me tell you it's not really obvious but there are a bunch of logistical challenges that come up when you are doing data collection in a country that doesn't have reliable internet and also is beset by tropical storms. Um there is also uh you I mentioned Shai earlier she's doing a lot of theoretical work to figure out how can we actually know when we think we've made discoveries about the language how can we validate that theoretically uh in the least invasive way possible right so I mentioned that collecting data in a marine environment in general general is pretty hard. It is especially hard when you are trying to collect data on big animals who dive as deep as they can to go uh search for their food source, which is squid. Uh we had some really really rosy ideas at the beginning of this project that we were going to stick a bunch of microphones in the water in different parts of Dominica and they would just kind of ambiently record all of the sperm whales all of the time and that would be great. we would have just a ton of data, the biggest collection of data ever on sperm whale vocalizations. Uh I will tell you in a minute why we had to pivot on that a little bit. Uh but we did end up investing quite a lot of time and this is what the Harvard team works on in figuring out how to uh attach tags to sperm whales in a way that is non-invasive. People have done sperm whale tagging before and other kinds of whales tagging, but what they tend to do is actually build a tag that hooks into the animal's skin. And we felt like we wanted to be gentler than that in our approach. So, we spent a lot of time figuring out how to make tags that would stay on with suction cups. And we also more recently focused on how to have an automated glider that would follow these sperm whales around the suction tags. And this is a kind of hard to see photo, but that's actually a drone dropping the tags on the whales. When we first started this project, the default way of putting a tag on a whale was to go out in a little dinghy, sit there, take a big long pole, and try and like stick it on the whale's back. Uh, as you might imagine, this worked some of the time. Uh, not very much of the time. [laughter] Uh which is why when we first started working on the data collection that already existed that was collected by people tagging in this way it was really really small um small enough that people were sending it to each other via email. [laughter] So we were we were not working with big data sets at first. Uh so what we have done is one really really work on the tags themselves. So, one thing that's important is how can you suction effectively onto the whale long enough for it to stay on and stay on under like quite a lot of force because you these whales they stay maybe up at the top of the water for like an hour or so chit chatting and particularly in Dominica they stay up at the water more often because um the babies don't actually dive. Uh so when they've got young there will be uh at least a young whale up at the top and then the the parents will dive, eat, come back up. Um but the other thing is we wanted quite a lot of data collection capabilities because one of the interesting things that allows us to start picking up patterns etc in what the whales are doing and identifying that with what their clicks are saying is being able to tie an accelerometer to that particular data set. For example, um we had ambitions to do video. We're not quite there yet, but we've done, I think, threeish generations of tags to try and get the suction better um to try and get the uh tag to be able to have more data recording collection mechanisms on it and also be a little bit smaller so that those suckers don't have to work as hard to stay on. And uh a big big difference was this drone tagging. Um this was a bit of a gamecher because we were able to have someone usually this was some we have a lot of um kind of undergrads who are really good at playing video games come and do the drone tagging for us [laughter] um as also you know they're they're trained roboticists and they they are trained to be in the field but uh I think their main skill uh above the kind of older people that work in sort of Harvard for a long time is this hey I know how to work a controller really Well, h and they basically go out and they do it with this little drone and they get it above the whale and then they drop it uh and it suctions onto the back of the whale h and stays there for roughly 24 hours and and then we are able to um retrieve it when it kind of falls off and bobs up out of the water and get the data off of it. Um so this was probably our biggest data collection challenge. I said that we were maybe a little bit rosy eyed about this idea of uh we can just put hydrophones everywhere in the water on buoys and we're going to listen to everything that's happening. Turns out that when you try to attach something to the seafloor in a country that has a lot of tropical storms, there is a lot of force on the cable that is attaching the buoy to the seabed floor. And if you have a buoy that has a lot of tech on it, that's going to be an interesting problem. So, we got this amazing buoy together. You can see it in this photo. [sighs] We put it out into the sea. Everybody was so excited. And I think about 5 days after we got it in, the first tropical storm of the season came, twisted the thing until the cable broke loose and it went floing off the way and was found by some fishermen uh who told us where it was. So, we did actually get it back out of the water. And I think we tried two or three more times with uh different types of cables and doing things like um uh type different types of attachments to the seafloor. Uh we haven't quite got a buoy to stay in place yet. Um but that's okay because we have this cool thing uh with a autonomous glider that actually follows the whales around. So, we partnered with a French company that does um marine sensor technology. And this glider uh is an autonomous underwater drone basically. Uh and it can follow the whales, record everything that they're saying. And um it has the nice property of not needing to be attached to a seafloor and uh corresponding to stone to storms. Cool. So, now that you know all about our uh fun problems trying to get data, which I will say that this is actually the hardest part of the project, anybody that's worked on a machine learning or AI project probably knows that it is the data that is key. So, the data collection of this project is in my opinion the really interesting part. But we have also managed to collect enough data uh throw enough models at it to learn some interesting things. So I mentioned we have a team of linguists out in Berkeley. um they have been starting to identify parts of whale speech and and I think they have reasonable claim to saying that they have identified an analogous part of speech to vowels and dip thongs in whale kodas. Um, so for those of you who are unfamiliar, dip thong is like when you kind of have two vowels in one. Um, when you're saying a word like out in English has a um, so whales also do this thing where they kind of stretch out a kod in a form that's analogous to default. So I'll play this. Um, it sounds I'm sorry a little bit like a fart. It's not a fart. It's a whale making a little very quick series of clicks. So, let's just play it. Can you hear that? Okay. Yeah. Do you hear that? Very quick. Uh, and from that they built on this idea of being able to identify the phenology of sperm oil. Uh so really looking building on that okay we have vowels and dip thongs but what are the what is the grammar of sperm oil speech and for those of you who are interested uh I will be around for questions I believe there is a question tent somewhere that the Harold kind of direct us to um and I can share with you all of the papers links for all of this stuff so that you can read all about the nitty-gritty details if you are a linguist nerd for example meaning we're going to skip the video for [laughter] I'm not sure the that it's quite working there. Um we are also because we are in the age of generative AI we are uh generating sperm whale vocalizations. Um, now I want to emphasize something. We are not generating vocalizations that we intend to play back to whales. Um, and this is a bit of an interesting one because how do you naively how would you test whether you are saying something in a language and and someone else understands it? Well, you speak to them and you see if you get the response that you're expecting. This has a couple of problems in an animal environment. One is well, if somebody is speaking to me and I don't know who they are, if they look weird, like, you know, they look like an underwater autonomous drone or if they look like a boat that I'm familiar with in another context and they're saying something suddenly in a language that I don't expect, that's weird. And then when you bring generative AI into the mix, I'm pretty sure that we are all familiar with AI gobbledygook. I think it would be a little bit weird and maybe slightly unethical if we were to then point that gobbledygook at a bunch of whales and say, "Hi, we're just going to like see how you behave and if it upsets you." Even in playing back recordings of whales themselves. So there was an incident with a humpback whale for example in Alaska uh where the scientists recorded the whale played back its own voice to itself and it started attacking their research boat. So we know that actually animals get upset when we do this in the past. There's also a slightly heartbreaking incident of somebody who works with elephants who played back a recording of a dead matriarch to the herd and all of the elephants in the herd also started showing signs of distress essentially. Um, so I think it's pretty clear that we don't want to just create a bunch of AI gobbledygook. We don't know what it says and just blast it out into the water. that's not the aim of a project that has at its heart really actually protecting these species um and bringing understanding to their life for the purpose of protecting them. So what we're doing instead is trying to use generative models that are based on language uh generation models that work in other contexts. So if we know that we can um generate language speech in the that like theoretically works with languages that we both know that are pretty far apart. Like for example, there are some tribes that speak in whistles. Um and that's a pretty different way of speaking than for example English uh or even most other languages. If we can get models that actually can correctly do generation for those, then we can start to uh reasonably think that we can generate a and create large amounts of synthetic whale data that is actually accurate. Uh and I think one of the most exciting things that our team managed to do uh and this is really a matter of being in the right place at the right time because we had set up this significant operation in Dominica and because we know that this is where sperm whales go to cal is we did manage to record the first ever sperm whale birth. Um, I will play a video, but given that the videos have been a bit janky, I will also point you towards the SEI website where you can watch quite a few of these videos if you're interested. But this was a really interesting moment. We had um our hydrophones in the water, but we also had drones with video cameras uh actually circling above the whales. So we what we got was this really cool data set uh of the sperm whale vocalizations uh along with the video of the birth and what was happening in real time. And a couple of interesting things to highlight from that one is a sperm whale birth is incredibly communal. This is a societal event. So, the mother comes up to the top of the surface water. Um, and all of these different whales start to circle her. Uh, and as the birth started actually kind of picking up and happening, um, and part of like how we knew that this was happening in real time when the the folks in the field were there was the vocalization started picking up. So, they started talking to each other a ton like they were they were just communicating like we hadn't seen before. Uh and then they were helping actually to hold up the mother and calf at the surface because of course sperm whales are mammals so they do also breathe air. Um and there was this you know group collaboration. All of this is mentioned in our we have a review of the paper, but one thing that we didn't emphasize in the paper because we kind of thought people would think maybe we were a little bit kooky and weird if we uh leaned into this too hard. And also, we are still figuring out what this meant is it wasn't just sperm whales. A whole bunch of pilot whales actually also showed up for this birth. Now, pilot whales are traditionally considered to be sperm whale enemies. Um, they harass them a little bit. Uh, and you they they are the marine biologists who are very attached to sperm whales of whom I've met quite a few recently are very like anti-pilot whale. It's like a little little taco war. But uh these pilot whales weren't engaging in harassing behavior for this birth. We actually don't know what they were doing. Um and we are trying to figure that out. Um but to me it's really fascinating and I will say on a personal note uh right about the time we were working on this paper I also gave birth. So, I had like a lot of thoughts and feelings [laughter] about this kind of like birth experience that this whale had gone through and that we were, you know, watching like a little bit closely. Anyway, um it was really interesting to see the social dynamics happening within one species and maybe we still don't know also across species for this whale birth. Uh, let's see if I can show you a little bit about what that looked like. Yeah. So, it's a short clip. Um, let me just play it again for you. So, that's the group of whales um that you can see there. And we've highlighted them for you. And you can see that they're kind of all circling around one spot. Um, and this actually took about an hour. So that we have a bunch of different clips of this. Um, but they were just like really circling and kind of lifting up this mother as she was being born. So a really like social group uh very communicative moment um in this moment's life. So I I mentioned that we had some you know ethical reserves uh reservations about how we interact and communicate with sperm whales. So as part of that um one of the people that we've part partnered with really closely is a group called More than Human Rights. They're out of NYU. I recommend checking them out if you're interested in these questions. um but they helped us put together uh an ethical framework for working with whales um and other species in general. So it's called how do we talk to our animal kin and I think just in summary this is a really challenging space there's a lot of interesting stuff going on and one thing that we are very very conscious of is kind of not becoming like this person who I have put on here he is a cautionary tale this is John C Liy who did a lot of work with dolphins in the 1970s he is known for giving LSD to dolphins as an experimental technique uh I think we are not going to be giving LSD to whales anytime soon to see how they react. Um, all right. So, I will end with if you want to know more and if you want to actually participate a little bit, you can go to the SETI website. And in particular, I would point out this listen to the whales campaign which allows you to interact with a bunch of the SETI recording, SETI data, and also um think about how you would want to communicate with the whale community if that were an option. All right. [applause]