Learning the Language of Privacy: Language Learning Apps and Privacy

So, hello everyone. I'm Matthew Plasencia, and in today's presentation, I will be talking about the project that got me my master's degree. And this master's degree project was titled privacy invasion in iOS language learning applications. And the deal that I did with that is I used some open source apps for digital forensics and just overall cybersecurity in order to help me find out privacy vulnerabilities in common language learning apps. >> [snorts] >> So, before I get into the actual meat and potatoes of this project, I didn't Let me introduce myself. So, I am Matthew Plasencia, and as you know, as I already said, I have a master's degree in computer science from Cal Poly Pomona. And I got that this past May. So, I'm very happy about that. And I was lucky to get a job in my field of digital forensics very quickly after that. And I'm working there now. And one of my many hobbies on the side is working with computers and very and a lot of Linux and this computer is running Debian at the moment. And I also like to try and run other operating systems like Red Hat and the BSDs and some other Linuxes. And now this presentation, as I said, is going to outline my master's degree project and all of the stuff that I did inside it. And how I used these digital forensics tools in order to help me find out digital forensic insights of these language learning apps that I was interested in. And I will do this by first laying out the background of all of the things that I did, then I'll go into the apps that I used, then the open-source tools, and after that, the fun parts, which are the methodology and the results that I got. And the end of this will be after I do that. And so, for my background, that's going to be first, so this project started really at the end of the end of maybe three semesters of floundering around through various other projects. And actually, this project started out of out of the idea that was the genesis of the CTF that was presented here last year for the scale CTF 2025. So, you have the idea that went into this to thank for that if you played it. And so, while I was doing [snorts] the idea before this, which was to generate CTF questions and generate my own sort of CTF CTF material for that, my advisor was telling me no that that was not in-depth enough. And so, one of the apps that I was using, which was Duolingo for questions, I looked into that a little bit more and then I kept going deeper and deeper into that and until I looked into how it worked with the advertising systems. and after looking into advertising systems, I got I got something that my advisor said yes on that I can move forward and delve deeper. So, that is how it happened. And so, I ended up diving deeper into how ads are generated and how ads send data across from the phone to the network and to the places where they send all this data. So, that's pretty cool and you'll hear more about that later on. But, towards the end of the research, I came up with this question to make the report complete. And this question was, is there any user identifying data that is packaged up and sent between between the phone and the servers? That's common between Okay, wait. Is any user identifying data packaged and sent in a different way between free and paid applications mobile language learning applications? And I cut out a lot of stuff from this presentation about advertising systems because I didn't think that was relevant. But, some of the main things that you may want to know about advertising systems are some definitions such as advertising terminal or such as the demand-side clients and supply-side clients, which are Demand-side clients are things like this computer that I'm presenting on or the phone that some of you guys may use and those are the devices that receive the ads. And supply side clients are the clients >> [snorts] >> like servers up in the cloud somewhere that deliver the ads over to you. And real-time bidding is the process by which these ads are selected that you see. And so real-time bidding processes are some very a split-second and very precise processes that allow the highest paid bidders on a on a certain ad to show their ad over to you. And one that you'll be hearing twice or more during this presentation is going to be the word interstitial, which means just coming between two things. So in this case, interstitial would be referring to an ad such as an ad that you would see on your game when you're when you press a button, say you're on your home screen and you say I want 25 gems and I'm going to watch an ad for that. And that interstitial ad would play between you pressing the button and getting the ad. And content distribution networks are just the servers that supply us with all this cool content that we see and all the annoying ads. And the apps that I used were Duolingo, Busuu, and Memrise. So we'll see a little bit more about these in this section. So Duolingo is one that you guys all may know and I'm very familiar with. And the reason you guys know Duolingo is probably because of this green little owl guy which tends to be a little needy for your attention. Sending you notifications like I'm going to steal your parents or something along those lines, something funny that gets your attention. But anyways, this works in attracting people to it and getting and even getting them to pay. Because it does have a lot of ads in its free version and its paid version is the answer to that problem of wanting to get rid of the ads. And so for this whatever free version and paid version you get certain features that are the same mostly. Which are exercise types that you'll see as well as some AI features. And these AI features, some of them are locked to the highest paid tiers, but we don't need to talk about those today. And one of the things that I liked very much about this is the amount of languages that it had. So it had languages [snorts] that you like like Greek, French, German, and all those others. And other interesting ones like Arabic, Turkish, and even some fictional ones like High Valyrian and Klingon. Which I didn't play. And for those character-based languages like Korean, Greek, Russian, and Arabic, they had very cool character exercises that I played through for ones like Arabic and Greek, which I liked very much. And they taught me the scripts. And so the exercise types that you get are things like what you'll see on first of all, to the to my right, your left, to the left of your screen you you see things that start with a work being exercise, a size with fill in the blank, and a size with a microphone, which you would say the words bir saniye lutfen in Turkish, and you would get that right. And you also have a path of a bunch of lessons that will allow you to progress through the app on Duolingo. And the one of those types of lessons that you'll see is also a story, and another one is a podcast type. And those two are the same. And the story, like this one, which is l'ascenseur, which is the elevator in the French, will give you a story about an elevator, and you have to understand it and answer questions throughout about your comprehension of whatever was being said through that. And you do a similar thing in your podcast exercises exercises, but in the podcast exercises, the only difference that it has is that it's much shorter than the story. And it has only two sort of two of these exercises within it. And my favorite feature of Duolingo, which you'll see here that I was doing for a long time, which is the streak in the app. So, I played this consistently every day from August 19th, 2020, to mid-June of last year, which was totaling over 2,000 days. And I learned bits and pieces of a lot of languages throughout all that time. And I know people spend a lot of days streaking on one language, but I couldn't do more than probably 500 days on a language. And they have two paid tiers, which I show one here I show one of which here, which is the one that I paid for, which is Duolingo Super Duolingo. It was going to say Super Duolingo Super. But yeah, that's you can see the paid features up there, which are not very relevant to what we have here. But now, unlike Duolingo, we have Busuu and this Busuu app is very different than Duolingo, but also very similar in that it's different because it's run by the people at Chegg, which is very interesting. And it is also free with a paid tier. If not, I wouldn't have used it for this. And the interesting difference that it has between this and Duolingo is that it has history parts of lessons, and I enjoy that for knowing the context of how certain things in languages came about and how different languages interact with each other being in similar regions as one another. >> [snorts] >> And having some interplay there. And so, you'll see here on your left that Busuu also has the path type thing, but in a different kind of way than other ones. And it also has character-based exercises for things like Arabic, which allow you to learn characters in a similar way to Duolingo, but without really tracing them out and doing them by hand as you would write them with your finger as if you're doing it on it on a tablet with a pen, that would be even better. And like Duolingo, it has a profile page and has some friends list features to do things with your friends and learn in a community. So now I have Memrise, which is a similar but also very different. And it's mostly very different because >> [snorts] >> although it's a free and a paid tier, it has a very different system in that this system has a space repetition system only with levels in which you click the button to start a lesson and you keep on leveling up by each lesson that you do and each one that you get right is counted less than the ones that you get wrong. And it has a very interesting feature, too, that allows you to listen to native speakers speak the language for reals rather than the other two with their AI-generated voice lines, which is pretty fun. And for this one, I don't have any pictures since the report that I did for this didn't include any pictures. And as a result of that, I will be going into my open-source tools. And there's a few very good tools here if you've been in the digital front digital forensics community. >> [snorts] >> And one of them is my one of my favorite ones for extracting information from iPhones, which is the device that I use for this. And that is the Universal Forensic Apple Device Extractor or UFED by one of my favorite guys in the digital forensics community, who is Christian Peter. And this one is an all-in-one suite for extracting things from iPhones. So so long as you have a jailbroken iPhone, you're going to be able to take information from the iPhone in a full file system format, which gets you all the files and all of the data on the iPhone including your iTunes information and any any any any app information from the iPhone. As opposed to your things like regular iTunes backups and logical plus backups, which just capture iTunes data and other data related to that. And in addition to these, they also have different log generation techniques, which are your Apple sysdiagnose log, which gives you diagnostics data, and your Apple unified logs, which gives you unified log data from the sysdiagnose and other things that are processes running on the iPhone. >> [snorts] >> And you can see this on his GitHub page and you're going to like what you see if you want to gather data from your iPhone. And so with UFED, these next this slide and the one after it shows pictures of how it's used. So how I did it was first of all, I I actually first of all, there there are binaries for this, but since I was using this on Mac to make it easier to work with the iPhone and do everything all in one platform, I went ahead and said, "Okay, I need to gather every single dependency." So after I gathered every single dependency and then I was able to see the bar on the left side of the window. I entered my path in that enter path thing. And then after I clicked acquisition options and it clicking that took me to the acquisition options from which I picked full file system and I just clicked okay since this is a default and the documentation said that this was okay. And this phone that I used on here was not the phone that I actually used for the research, but it's just as an example, so the one that I actually used took longer than this one since it was a 64 GB phone. And on that phone, I'll get into this more later, but it didn't have the data that I wanted. And the data that I wanted dealt more so with network logs and for those of you who are into cybersecurity and know a thing about network traffic, you'll know that Wireshark is one tool that we can use to gather that. And for you for those of you guys who don't know about Wireshark, it's just an all-in-one tool that's used by us cybersecurity people to gather every single activity of a network. When I opened this with root privileges, which is basically admin if you're in Windows land, so when I opened it with root privileges, I was able to get something like the screen that you see on the screen over there and this showed something along the lines of all of these interfaces and picking something like the one that gives me the most data, which is your e n zero, which is your network, gives us a capture like this. And all that you need to know about this capture is that most of the data in this capture is going to be encrypted. And with encrypted captures, we're not able to see much. So, we need a tool that deploys something to get in between those thing between the Mac or whatever device and the network and decrypt the packets so that we could see what's inside and be able to see what information is being sent between the phone and the servers. And the first thing I found was a cleverly named app called Wireshark Helper. And the Wireshark Helper is obligatory between the Mac and the iOS system. And in this in during my research, it was free and open-source for both Mac and iOS. But today when I checked it, the Mac version was locked behind a paywall of $10. So, you need to shell out a little bit of money in order to do this if you want to use the Wireshark Helper. And the way that did my man in the man in the middle proxy, which is how this whole operation works, is through an interface on the Wireshark called the remote virtual interface or RVI. And in this case, it was my first slot for RVI since I have nothing and that's called RVI0. And so, in order to set this up, I just had to run the command as you see in this frame from the video. And it gave me three things. First of all, an NSS key log file which I put into a certain directory in Wireshark. And after that, it gave me a root CA certificate, which I have to put into on my phone. I have to go to the link that it shows. It would show on my screen and download that onto the phone, trust it, and or install it and trust it, I should say. And then after that, I can use the Wireshark helper app on the phone as you see on the right of the screen and type in the network address on my local host network. And connect to it and once I start gathering traffic from the phone, it will look something like this with everything decrypted. And that is very cool. But one thing that I found is that this was a little bit inefficient for me to gather my traffic from. So, I ended up looking for things that are on the iPhone only. And the one that I ended up finding pretty quickly was Storm Sniffer which is an all-in-one platform for gathering network information and analyzing that information. But unfortunately, it's neither fully free neither open source. So, you're going to have to pay a little bit of money. And that money is only $3. But $3 will get you this man-in-the-middle or MITM function on the far right of your slide. I mean, on the far left. And [snorts] once you get that function by doing the same thing that I described before by gathering the CA certificate and installing and trusting it, you can press the start button and capture all your traffic and stop it by pressing the green stop button that would pop up. And then in the second picture, you're going to see you're going to see two full captures that happened. And those two full captures that happened are stored in HR format, which I inspected on Wireshark just to do everything in my Mac with a tool called But that ended up being a little bit too too strenuous for me, so I ended up just fully using the app at a certain point. And what how the app gave our information is very cool. It had you can scroll through scroll through left to right on the top bar to see all of the major sections of what you would find in a packet. And scroll up and down for all the minor parts of that section of the packet. And one thing that we'll go into is the packet information of certain requests later on. And this is the experimentation side, which lists my trials and tribulations of doing this experiment. And so after gathering all of my applications and gathering all of the requirements to run them on my Mac and my iPhone, and having it jailbroken, I first had UFED and I ended up skipping UFED in the very beginning stages of this because all that the iPhone had on it was basically data that was more along the lines of app executables and files related to things that the apps made and not necessarily network information that was gathered. And the golden ticket to my network information was actually capturing the packets that went out to the network from the phone. And so when I went to went into network sniffing, I had to do the same basic thing, which was go into my VPN profiles and trust each one I mean install each of them and then go into the privacy and security settings and trust each of the certificates. And once I did something like that, the dark theme image is what I was able to get and that meant that it went all the way through. And so, both of these apps did a pretty good job of getting all of the things that I needed, but Storm Sniffer had some problems and those problems mainly being a problem of me not being able to get through lessons early on when I was going all the way through the lesson gathering information. With the mic, caused me to just go all the way to Storm Sniffer for everything. And throughout the first parts of my research, I was heavily doing ad ad-based traffic in order to see if it led me to any malicious ads. And towards the end of it, my advisor told me to also add add traffic with ads inside it in order to make sure that these apps, when you pay for them and pay for not getting ads or not getting anything that's showing your device information or anything about you to advertisers because that's probably a violation of privacy. And we'll see more about that later on, but for now, I talked about HTTP Cat, but I did use that in order to look at some of the information in the stuff on my Mac by air dropping in the HAR over to my Mac from the iPhone through the early stages of the process. And the flowchart was basically this where I went ahead and started the lesson and for most of the process I got maybe one or two less one or two exercises exercises before the end and when I got to that point I went in and started the Storm Sniffer or whatever other one and said and said, "Oh, yeah, I'm going to I'm going I'm going to see what ad what ads or what information is being sent over and I I played everything and then after playing everything I stopped it and analyzed whatever happened and whatever traffic went through and I recorded it as a data point. And so many of my data points for the ad traffic went to these sites which are anything from the Unity Ads Network anything from osdk.mediation.unity3d.com anything from anything from the AppLovin network anything and anything from the Google Ads network. And when it came to the osdk.mediation that was one of the big providers of the data that I put in my findings and we'll talk about more about that in a few moments. But one of the fun things that I found is that the Unity 3D ad network was used to supply all the ads for these apps. And the Google Ads network was what all these ads came through and through my Google Ad Center, I was able to meticulously go through all of the ad types that I had and see what ads that I actually wanted to see more of and which I wanted to see less of. So, you'll see here that on ad types such as the games, I clicked less of them with a minus button. And with the plus button, I could get more of home furniture or food or anything like that. And so, the data that actually goes to the network is pretty interesting here. So, first of all, what you'll see here, if you can actually see the small text on the screen, is that you're going to see each link to mediation has three selected types of identifiers to it. And the two main ones that I like to focus on are the ad format and the compression type. And the ad format here is three, which is interstitial, and that is expected since I got this ad probably in a Duolingo lesson after finishing the entire thing and clicking the next button before I went home. And the compression type refers to the file that you see here in the body of this request. And if you guys don't know anything about web requests, this is JSON data, as you can see by the leading curly brace, and you can see it's encrypted in a cer- or encoded in a certain type of format by the first few letters, which are 4 H 4 S I and three A's after that. And if you [snorts] want to get a head go ahead and see the slides, I will send them to you later on. But I have a link in this slide to where I found this command. But since I was on a Unix-like system, I was able to use a cool command to actually decode all this all this JSON information in here. And that was the base64 command. And if I put all the data that's in this body into a file, say something arbitrary like the name input file, and run base64 minus capital D on the Mac or lowercase D on other Unix functions, I on this and pipe the results of that into zcat, I'm able to see it. And zcat gives me some sort of thing that looks like this. And so you'll see here, if you can actually see all of that on the screen, you'll see that it has things like device data, the date and time that it was going on, maybe locate data about the location and time zone, and maybe even timestamps, if you're lucky. So that was what I found in the ad. And we're going to see some a little bit more of that in some of the ad-free versions of it. But before we see that, we're going to go into seeing other stuff, which is a call to app app-analytics. ATT.com, which is an Apple proprietary thing, which just syncs with the server telling you what kind of ad preferences and tracking preferences you have on that device. And that was a post request of a GIF with HTTP error code HTTP success code of 204. And to get into the results that I added to this, which were specifically for the ad-free versions, which are Duolingo paid, Busuu paid, and Memrise paid, you're going to see some information here that may not be necessarily the same as what you would get these days if you were to do the same experiment. Because during the experimentation process, I found that some of these things varied at certain times. So, just take some of these results with a grain of salt. But, what I found on the mediation contents for Duolingo paid is I found similar data to to actually what's on a couple slides before, similar data to that. Sent to mediation on our on our Duolingo paid version, and according to their privacy policies which I read, they're not sharing any sharing or collecting any of your data when you pay for them. So, that's a pretty interesting difference than their privacy policy. And in the same way, actually before I move on to that, I should say that this basically by the time of around March of 2025, this stopped happening for me. So, that was pretty interesting. But, one thing that did not stop happening for me is getting mediation contents being sent over to Busuu in the Busuu paid version. And in the same way, there was similar information between Busuu and on the paid and free versions. And the interesting thing also is it showed metadata of the transaction which are things like probably your account number, your account ID, your device type, and other information about the phone. And whereas the [snorts] other two sent things to mediation, I didn't really see much, maybe two or three pings to mediation on the free version of Memrise, but mostly Memrise didn't send much to mediation, but across all of my research and testing and across all times, it's an interesting things to both the ATT and ep2.facebook.com, which I found is part of the Facebook Ads Network, which is something that I didn't expect it to do, but nevertheless, that was interesting to find. And in finding that, and finding all of that stuff, I leveled all of this impact up as these mobile language learning apps are cool, but they may be breaking their privacy policies by gathering this data with the APIs that they use to gather it. Because even though they say that they're not showing you ads, they still have the APIs for these advertiser functions within their apps, and >> [snorts] >> that is not a very good idea for our privacy. And I recommend that you guys stay away from this if you guys don't want to have this happen to you, and you don't want your data being sent off to third parties that you don't agree with. And in my report, I summarized future work as to look at the Android versions and to see how they're similar and differ to this. And to do further research on the ad formats in the Unity 3D ad mediation because I saw during this research that there was certain things there was also a code two, but I didn't see much about a code two, I think. And also to see how to figure out what's being pinged over to that ATT servers in that GIF. And and [clears throat] some more stuff, but that other stuff wasn't important. And with work and life going the way it is right now, I don't think I'll ever get to this, but it may be something for you guys to do if I interested you guys in doing some research on these language learning apps in Android. And now I'm getting to the end of this. So, the summary that I prepared for this is that language learning apps are new and interesting ways of learning languages that allow you to interact with your phone or a computer in order to learn these languages and they also may include online advertising, which can be a bit of a hard sell for some people who are privacy conscious. >> [snorts] >> And if you want to gather data from these apps on a phone such as an iPhone, you need a man-in-the-middle wireless sniffer app in order to decode all of the traffic. And the ones who do that, you can find that apps like Duolingo and Busuu share data about your device with the Unity 3D ads mediation, and that is not a good idea. >> [snorts] >> And that'll be the end of the That's the end of the presentation. So, thank you guys for reading and watching. And you can contact me at email address at plasenciamat31@gmail.com if you want to have a copy of these slides. And you can also check out my Substack at substack.com/@matthewplasencia @matthewplasencia in order to see what I write about about cybersecurity, digital forensics, technology, philosophy, and some other stuff. And I also have a YouTube channel that covers topics that go in with that. So, now I'm at about 40 minutes of presentation time, so we have probably about 10 minutes to do some question and answer.