SA-TIED Seminar : Are South Africa’s employment numbers telling the full story?

Hi everyone and um thank you for for for attending. I hope this is useful and we uh yeah can have a useful discussion afterwards as well as hopefully um me telling you something interesting for the first 40 minutes. Um sorry [clears throat] I just want to see if I can minimize the video screen because seeing people's videos working great. Okay. Uh so this work is um by myself Amy Thornton um who's a researcher at data first and Paris Wiro who's a PhD student um school of economics and works for data first all at the University of Cape Town and this paper was uh funded by the SA tide program of you and your wider so thank you very much to to SA Tide for um for facilitating this research. Okay, so as probably everyone here knows uh high unemployment and low employment are key features of South Africa and in this paper we have three main contributions. Uh we document very large differences in employment between two very similar sets of Stat South Africa surveys. The general household surveys and the courts labor force surveys. uh these differences have got bigger um postco and um they're substantial enough that we thought it's worth uh documenting them and trying to interrogate them and and also just highlighting them um particularly because of point two we have tried to explain the differences between them um and we haven't had much success in that and um yeah you can you can tell us if you've got lots of other explanations at at the end Um and then just in the process of trying to explain the differences providing technical detail about the surveys that are not uh in the public domain or sometimes are in the public domain but not well uh well known. Okay. So as I said um we we have a working paper wider working paper that is um the basis of most of this presentation and that paper's actually been accepted for publication in South African journal of economics. In that paper we start at 2023 um I have done a bit of work just checking 2025 comparing the GHS which came out a few months ago and the curees from from 2025. Um, and I'll I'll I'll mention that briefly. Um, but most of the figures, all of the figures that you see are going to go up to 20 2023. Um, I'm not talking much about the CEO of Capitex claim that the unemployment rate is uh 10% in any detail, but I'm just going to briefly discuss uh and maybe these two bullet points kind of disagree. I'm not sure exactly what I was trying to say, but briefly discuss these claims and then move on to the substance of the paper. And as I say over here, please note my pronouns I and we. Um, most of this is we, Andrew, Amy, and Paris. And some of it uh in the beginning is I, Andrew. Okay. So, we submitted this uh proposal to WER in 2024. And in 2025, the Capek CEO came out and said that the unemployment rate is really 10%. and that the QFS does not measure self-employment which is just plainly incorrect. Um and many people waded into that debate and um my initial response was over here a letter to the editor of business day that published that um interview and with Fared and I said here there are legitimate concerns about the QFS which I and others have written about including the poor data quality uh and earnings and employment numbers being higher in the general household survey than the quarterly labor force. survey but that stat South Africa does not count the self-employed as employed is not one of the concerns. So hiding in that letter was a statement that yes there are some differences between the GHS and the QFS um but those differences are not going to take South Africa's unemployment rate to 10%. And you know that generated a huge amount of heat noise and I think very little useful um other than yeah very very little useful in in that whole debate. Um and I just want to remind each other everyone on the call that that's that's a debate that's been happening for a long time. One of them was in the 2010s, early 2010s at Corp came and said that Stat South Africa's employment numbers were junk and that they had the best numbers and again the unemployment rate was extremely low. Uh and myself and Martin Vittenberg wrote two criticisms of uh of the ADC Corp employment index and a year after that the index uh kind of disappeared. So that was I think one part. It also this debates also sounds familiar in the 1990s there was a discussion again that just immediately post apartate was unemployment extremely high and the conclusion from that debate there were people saying it was much lower than the official numbers from stat South Africa uh in the 1990s were saying and that debate I think was resolved pretty firmly. Stephan classman and then Ingred Willard wrote a paper saying basically yes very high unemployment is is the case. And then if you go further back in time John Knight um in the South African J of economics 1982 um responding to um an argument that unemployment is voluntary and kind of therefore really something to worry about. um trying to argue uh Knight is trying to argue that actually unemployment was very high um and uh and potentially rising. So just to say you know I think we get into the treadmill of you know this quarter's results say this and that and the the next thing or the capy tech CEO said ABC but just in the sort of long long run there've been these kind of discussions going along and certainly my view is that you know there's no question that unemployment is is extremely high and every now and then someone pops up to say it isn't and academics or others come in and say actually probably it um not probably it is um and yeah so so I just want to to give a bit of historical context to to that discussion okay so as I said I'm not going to spend very long on on that discussion debate etc um but just to say I think it's good to debate uh I think it is good to interrogate uh both status surveys and statistics that we do in this paper um and also But also I think people trust the private sector numbers too much. I think there was there's sort of too much suspicion of stat South Africa and too little of um of numbers that come out of the private sector. So that's just my my own view. Um and you know our contribution in this paper is to to enter that debate. uh we don't mention 10% unemployment or anything like that but because we want to position this as a as a academic research exercise um not not trying to point score. Okay. So just just to mention briefly some previous academic literature on uh on measuring employment. Um there was a debate in the 1990s about jobless growth. um firm surveys suggested that employment was shrinking or at least employment rates were shrinking. Uh household surveys said something more optimistic. Um and that in the end I think was resolved on the side of there wasn't actually jobless growth. Um the firm surveys that were being used to say that uh were probably not as as good quality as they as they could have been. There's been other important contributions. Cassell NL 2004 talking about the importance of changing definitions of what exactly is employment how survey enumerators were prompted to ask respondents are you employed you know do you do including car guarding or at least you know even if it's only one hour a week um you know trying to trying to push quite hard actually on um on measuring even you know smallcale employment that might be missed if someone just had in their and wage labor for which they got paid you know salary or earnings at the end of the week or end of the month. Um that also included subsistence agriculture which was eventually cut out of the definition of employment when the cure happened in 2008. Um also important are how we calibrate the surveys to some model of how many people actually out there in the population. That's also um an important part part of the story. Um myself and Martin contributed thinking about by thinking about and worrying about the sampling of small households. Small households were under sampled in the 1990s and small households have important or higher rates of employment. Um people living alone mostly are working. Um, and so the fact that small households were better sampled in the 2000s increased employment somewhat artificially because some of it was just capturing employment that was always there but not measured because of small households being under uh under sampled in October household surveys. And then there's also a small literature worrying about fabrication of data. And I'm certainly not suggesting that in the rest of my talk, but just to say it's not an impossibility. NIDs was, you know, is ran for five five waves between 2008 2017 was and is a highly regarded survey. And in the process of wave two, uh, Arden and Viml, the authors and and the survey team discovered some cheating, fieldworker cheating. That also happened in the 1993 PSLSD survey that was only discovered five years later when Aquazun Hotel income dynamics study was done and the people trying to conduct a follow-up of those 1993 households found well didn't find anyone in two particular areas and it turned out was pretty small but it turned out that that that data from 1993 had been fabricated. So, you know, there's a whole range of possible explanations for why employment may or may not be well well measured or is the measurement of employment has changed and how that does or doesn't affect our ability to um to discuss changes over time. Uh the general household survey is a stud South Africa survey that's been run annually since 2002. It's very it's more general as its name suggests than the labor force surveys or or cure affairs and um it asks very similar employment and earnings questions and I sort of stumbled upon this in my previous research on earnings um realizing that it was a good source of earnings because I was figuring out and later arguing that the public QFS data earnings data is not very reliable and um I used the GHS data is a check on um on that QFS earnings data. So um it hasn't been used for employment um and that's what we've done um in this in this paper over here. Okay. So how many people actually are working in South Africa? We came to this topic because myself and Amy um were both working on versions of what became PACES post aarted socioeconomic series. um that's a micro data set that harmonizes the GHS's up until 2024 actually I think um not 2023 and in the process of doing that we realized uh that actually employment in the LF in the QFS's and in the GHS's was quite different and paces itself was based on another data uh set that myself Martin Vittenberg and David Lamb are the authors on and that data set both of them are freely available through through data first at the University of Cape Town paces was also funded by essay tide um and the updates palms version 4 update was also funded by by tide okay so this paper came about as part of uh trying to uh trying to figure out how these ghs's how they can be linked together over time do they tell a cons consistent story and does that story uh differ at all from what the cure was saying? Um so just when I use palms uh when we use palms in the figures um that's just referring to a combination of the labor force surveys that ran between 2007 and the QFS and most of our um discussion is going to be focused on the QFS but we call it palms because the stats we derive are we just use palms to to derive them but underlying palms palms is a set of harmonized surveys that come from stata October household surveys, labor force surveys, QF QFS's plus the 1993 PSLSD that was run by Seldrrew at UC. Okay, we give a few figures from NIDS where one to five. Um but after the first I think two figures, we just end up focusing on the GHS's and the LFS's and QFS's. Nits was designed primarily as a panel. the sample size is much smaller um and so it's not ideal for measuring um employments in the same way that the GHS and the LFS and QFS are. And we shown 95% confidence intervals around our point estimates uh in shaded areas. You'll see that uh in a bit. Okay, so here's our main result. If you want to leave early and you can just leave after I explain this figure. We have three different surveys. So here's the um palms is black dots ghs blue dots ns red dots. I'm not actually going to say anything about ns. And what you can see is that in the LFS period over here between 2002 20078 the surveys track each other very well. there's um quite a difference opens up at the time of the financial crisis 2008 things are go back to being kind of similar in perhaps 2010 2011 after that there's uh ghs grows quite substantially so these are millions of people on the y- axis until there's about a gap of um roughly a million people say between in 2018 and um then co hits uh I I won't say too much about CO, but the 2020 and 2021 surveys conducted telephonically, they were uh had much lower response rates. They're probably not as good quality as the rest of the surveys that before or after. Um but what ends up happening is that there's a very large gap uh opens up between the CUFSes and the the GHS's uh in the post-aparted period, sorry, postco period. Um yeah, so I think that's our that's our main finding. It's kind of strange um that two surveys and a way that I'll explain are quite similar and yet end up with quite large differences um in in employment. Just a note that GHS is conducted from 2013 onwards I think across the year in all four quarters but because the sample sizes in each quarter are quite small in the way that the survey is designed it's supposed to be representative of the entire year. So GHS 2023 uh it's supposed to give us sort of overall number of employed during 2023 some perhaps kind of average over the four quarters. So when we doing some comparisons between the GHS and the QFS, we're doing something like in 2023 comparing quarter 1 QFS with the 2023 GHS number, 20 uh quarter 2 QFS number with the 2023 GHS number, quarter 3 QFS with the GHS 2023 number and the sample quarter form. Um but yeah, that's why there are many fewer blue dots than black dots. NS uh very briefly I think the last time I mentioned ns but just to say um the standard errors are much higher those are the bars over here um and that's because as I said the survey is much smaller the there were only 400 clusters or enumeration areas in the survey compared to 3,000 or 3,300 in the in the stats survey so much more uncertainty about what the true number is and it's had a different kind of uh goal in mind um and different funer etc. So um yeah I'm not not going to say much about uh more about that and focus mainly on these quite large differences and then very large differences between the ghs and the QFS in the postcoid period. Okay, so that's on total employment. Just having a look at the employment rate, the fraction of adults 18 to 59 uh who are in work and you can see that the surveys tell a similar story in the early 2000s roughly the same. Uh ghs opens up quite a big gap of several percentage points um in the 2010s and that number is much larger again uh in 2023. Okay. And if we zoom on in on the differences um in this now we have the difference between the QFS uh and the GHS or the LFS and the GHS in this period. And what you can see in terms of both numbers and percentages there's kind of very little difference over here. It's just mechanically the result from comparing the previous two uh two slides. And then again the difference opens up of um you know up to a million people. That's the black dots over here in the 2010s. Um and then even more than that 1.5 million um or even 2 million in some of these uh particular surveys. So and then the percentage differences uh you know in this later period it's maybe 10% somewhere between five and and uh 10% uh in this period over here and very little difference in the NFS's. Okay, so I mentioned we started in 2023. I just checked the 2025 uh data a few days ago. And in the GHS 2025, employment is still 2 million people higher in in the GHS compared to either the Q1 or the Q2 CFS 2025. I hadn't easily arranged the later two KFSes for 2025, but my guess is that they're going to tell a similar story. I didn't check though. Um and just and also the employment rate 45% in those QFS's the Q1 Q2 and 49% uh in the GHS. So again this is not something that's somehow corrected itself. We had changes immediately after CO that have now kind of gone back to normal. This appears to be something that's that's continued all the way until 2025. Okay. So we've been focusing on the employment rate because that's where we started. Um, but it turns out that there are important differences. You know, if you're not if you're if GHS is finding a whole lot more people employed mechanically, those people have to be in the QFS where they're not employed, they have to be something else. Um, and what's interesting about uh these graphs over here is that in the 2010s, the difference was that the um that the GHS found many fewer note economically active people and um the QFS found more and the numbers of unemployed were roughly the same. Okay. So the the key difference seems to be palm seems to be uh sorry the QFS seem to be finding fewer employed people and more note economically active people. But in the postcoavid period, you can see that the differences in employment and note economically active are now quite big and so gh is finding quite a lot more uh employed from the previous graphs, more unemployed and many fewer note economically active. This difference is almost 3 million people um by the end of 2023. So yeah, it's not just employment. uh that's the focus of our paper. Um but there are definitely large differences both in in uh the number of unemployed only in the postcoid period and in NEA larger differences uh not economically active larger differences postco but also smaller ones um in the 2010s. Okay. and and because of the different statuses and how the definitions of employment unemployment NEA work, there's not a huge difference in the um in the unemployment rates um over the entire period. Maybe a little bit uh over here um palms the QFS is slightly higher um but but not not very not very different. So it affects certainly the numbers of people but because of the way the three definitions work together not huge changes um in in in the unemployment rate or differences in the unemployment rate excuse me. Okay. So why is this odd? I mean surely you're kind of surprised about this but why is it why is it surprising? Why is it odd? And for us, you know, the same organization is conducting the surveys. It's not like Stat South Africa and another organization. The sample design of the two surveys is very similar. um up until 2018, they actually share some many or all of the same clusters, the same small pieces of ground that are chosen uh where 10 dwelling units, 10 households are chosen uh in each of those pieces of ground. So they're going up until 2018 at least the QFS and GHS, you know, they both run by Stat South Africa. The sample design is the same and up until 2018 they're both going to the same places or at least overlapping places uh to conduct the surveys. In addition, the questions on employment are virtually identical and very similar questions identify the narrow unemployed. Okay, so this is strange and you know if you if you want a preview well I think I already gave you one. We don't have a good answer. Um, we're hoping Stat South Africa will come back to us and everyone else with with a better answer. I think they're better placed than us to try and answer. But the rest of the uh seminar is going to try and show you why the things that we've ruled out even if we um we weren't able to find the true uh explanation or an explanation. So, three possible explanations. Something about the weights, maybe the way the weights are constructed. um the demographic model uh that's used in each of the surveys. Perhaps that explains uh the difference. A second option is the sample. Maybe uh one of the surveys just always choosing a different sample in a weird way that means for example the ghs finds many more people that the ghs samples somehow targeting people who have a higher uh propensity to be employed. Um and then the third category very broad some kind of survey effects. Maybe the enumerators are somehow different in the ghs and the cure the fest. Maybe the respondents behave differently based on the way the enumerators act respond etc. Uh maybe there's other fieldwork protocols that differ between the two surveys. Maybe there are differences in the questionnaires. we rule out one and two um and one small part of three um so then we're left with a puzzle. We we don't have a good answer as to why these numbers are are so different and as I say ideally stud South Africa will be able to give us some um some further explanations. So we can immediately rule out weights as the primary um explanation and we can do that because the raw samples if you just don't use weights at all the raw samples basically tell you the same story. In the top we have the total number of people the actual number of participants in the survey. So this is like 25,000 people are employed in and in the sample of the surveys. Uh that's what that number over there means. And what you can see is over the period um GHS consistently finds more people um employed than uh the QFS over the period. And that's also true in this post uh co period over here. And the same thing is true for the employment rate. If we don't use weights at all to make these surveys look like the supposed population, if we just check all the people in the survey and ask what fraction of adults 18 to 59 um are are employed, you get a much higher number in the ghs. A higher fraction are employed uh in the ghs than you do in the ceiles. Okay. As I said, we tried to put some uh information about the surveys into the public domain. We rule out uh weights as a as an example uh as an explanation, excuse me. But we did find that the QFS and GHS use quite different demographic models. Both of these are fairly outdated and the QFS is more outdated than the GHS because of these raw numbers. We don't think that's the main explanation, but it it might have some uh might be some part of the explanation, some small part. Are these two surveys drawing very different samples? The short answer is no. Uh when we look at the fraction of female male in the samples, people of different age groups, so for example, young people much less likely to be to be employed than older people. um we don't find many important differences in the in terms of the fractions of people that are of different ages that are different races. We do find a small difference in the average household sizes. So ghs is finding a little bit larger households and a smaller fraction of phhes which is single person households. Um, but it doesn't look like somehow the ghs is finding groups of people who are much more likely to be to be employed compared to the the QFS or or LFS. Okay. And something that's makes this even more puzzling is that um between 2012 and 2018, this shows you the number of EAS. That's the small pieces of ground where the survey is actually done. Stats, they say it chooses 3,300 or 3,000 pieces of ground where they actually do the survey. They don't go all over the country. They choose naturally representative sample of these uh pieces of ground enumeration areas or clusters and they're as I said about 3,000 and between 2008 and 2018 there was huge overlap sometimes almost complete overlap like in 2015 3,232 of the EAS the small pieces of ground were covered both in the QFS and in the GHS they went to the field workers that's to say field workers went in the same places. GHS had three EAS that QFS doesn't visit and QFS had 68 pieces of ground that the GHS didn't visit. But the overlap is almost completely identical. And yet that was a year when there was actually quite a large difference in total employment, probably about a million people from my recollection and a few percentage point difference in in the employment rate. So despite going to the same pieces of ground um asking very similar questions about employment, the GHS found substantially more perhaps up to a million more people in 20 2015. That overlap stopped in 2019. Uh GHS went to a new set of EAS from my memory. Uh QFS stayed with the the old ones. Um and GHS because of CO carried on using the same ones. palms QFS carried on using them at least for a few years their own ones and so there was the separation so after 2080 there isn't an overlap anymore statsa numerators and these two servers are not going to the same pieces of ground u from what we can see in the data and from what we know from from talking to stats okay um I'm going to skip this because I just don't want to um don't want to run out of time so we run a very simple regression employed is a one, un not employed uh is a zero. Uh it's called a linear probability model regression. And we put in a bunch of characteristics. So controlling for gender, age, household size, population, group, years, years of schooling, and what province you live in. And we tried a huge bunch of different things. again finding that the GHS after controlling for all of this stuff uh the GHS is still finding for the most part employment rates that are about 3 percentage points higher than the QFS. So this says uh this column over here column 4 says EA fixed effects E A Fe and what that means is we're taking between 2013 and 2018 we're taking we're controlling for somebody being surveyed in the GHS and in the QFS and those two people being in the same area and so for someone who's of the same gender age household size population group years of schooling in the same province and in the same EA, the GHS is still finding that that person has um a higher 3 percentage point higher chance of being employed than the identical well the person with those exact same characteristics who was enumerated in in the QFS happening maybe you know down the road in the same area. So for us that's yeah kind of even stronger evidence that there's something here that we don't understand for why the ghs is finding more more employment. Okay as I said survey effects can mean a huge diff lot of things and you know we're not South Africa so we don't know about most of these issues. Um one thing we've sort of toyed about you know permanent contracts um maybe that differs but over the period in which the differences open up so from 2011 onwards basically all the field workers in the QFS and the GHS had permanent contracts. Um yeah one possible explanation is differences in the questionnaire pathway uh length. If you're employed in the QFS you get lots of questions. If you are not economically active or unemployed, you get many fewer questions. Whereas in the GHS, whether you're employed, not economically active or unemployed, you get very similar numbers of questions. So, one thought we had is, well, maybe either enumerators or respondents have strong incentives in the QFS to go to um to answer in ways that reduce the number of questions that they have to answer, whereas in the GHS, they don't. The problem for that explanation is that that's true both in the periods where um things were very different um the the employment numbers are very different and in periods where they're bit more similar. So yeah we we thought of we toyed about this idea but in the end kind of yeah didn't didn't think that that was very very useful. Um so one thing we can check is that both the QFS and the GHS GHS not well known QFS more well known is that these are both actually panels of dwellings. Stats to say it goes back to the same p dwellings several times in a row and the QFS four times for across four consecutive quarters and the GHS um over a number of years only survey each household once a year but over several years. Okay. So maybe if you had a really long interview last time on all your employment, you're more likely to just say actually I'm not working in the QFS whereas the ghs it doesn't really matter uh which pathway you take in the questionnaire whether you choose employed unemployed uh you know maybe the ghs people uh won't adjust very much but the QFS people will all say actually we don't want all these long questions about employment they'll just say I'm not working okay and that isn't really true So what we do is take the group of households in the GHS and the QFS's that were visited the first time first time their household ID shows up in the survey um and we go and look at those households and basically we see no difference the very first time a household is interviewed so they haven't had any prior experience of knowing whether the ghs or I mean whether in the cure fest if you say I'm employed you get so many questions and if you say I'm not economically active you get many fewer um that doesn't seem to make any difference the GHS is still getting higher employment rates uh than than the QFS. Okay. So one other possible explanation is or route of investigation is to just check the quarterly employment statistics firm survey numbers. Now that's very different uh to the GHS and QFS as it only uh this QES only interviews VAT registered firms. Um but it's a partial check. Um and the QES and the QFS employment growth is quite similar between 2010 and 2019 and the GHS is a bit higher and between 2019 and 2023 uh where the differences in the GHS and the QFS opened up quite a lot. Employment growth was 3% in the QES, 1% in the QFS, 5% in QFS formal and 8% in GHS. So GHS growth does seem to be quite a lot higher um than well is definitely higher than both the QFS and the QS is sitting maybe in the middle but a little bit more towards towards the QFS. So that's a little bit of evidence that perhaps the QFS change over these last five six years uh is is more accurate. So but you know these are just three surveys. It's it's hard to know. I don't think that resolves our um our puzzle and I don't think it comes down firmly on the side of of the QFS being correct and the GHS being incorrect, but it is an extra piece of information. Okay, so that that's where um where I'm going to leave you just to say there are very large differences um in employment particularly co uh postco between the GHS and the QFS. These are very similar surveys and it's kind of a puzzle why they would have such large large differences. It doesn't make much difference to the narrow unemployment rate even though the numbers of employed, unemployed and not economically active do differ quite a lot between the surveys. Um and the most recent year 2025 there's two million more people employed in the GHS than than the QFS. Um, >> thanks Andrew. >> Uh, yeah, sorry Abina. I'm going to give me one more two more minutes, please. My watch says 339, >> please go ahead. >> Thank you. Um, we've ruled out weights as a primary explanation. We've ruled out differences in the samples. The samples look very similar. We also ruled out respondent learning, but there's lots of other potential explanations that start to say quick try and answer uh or investigate. and and we haven't been able to just because we don't know much about what goes on inside stats to say and then just a very diff very definitely this is an I pronoun slide um I think we should all ignore what the quarterly unemployment frenzy we can call it qu to just have nice acronyms that sound like QES QFS etc um there's huge amounts of noise in the media experts political party statements about um the quartertoquarter changes and the QFS the QES etc. Um my suggestion Andrew not our research team my suggestion is that we should not pay much attention to the noise. I think those quarterto- quarter changes don't give us much information. They're generally not statistically significant. The changes are small. Um they're probably not real. And and just to give a a long-term perspective again in 2023 the employment rate was higher in both LFS uh than higher in the QFS than it was in the 2002 LFS and it was much higher in the 2023 GHS than the 20 2002 GHS. So you know whichever one you believe compared to 2002 a much higher or a higher fraction of people are working in South Africa than was the case 20 years ago. And the employment rate as best it can be measured in 1993 was roughly the same as 2002. So yes, it's true the unemployment rate has grown substantially. There are a lot more people saying I want to work. But at the same time uh the fraction of people who uh are employed has actually increased uh since 1993. So you know when people talk about unemployment and job losses we should try and hold those two things kind of in tension. actually a higher fraction of adults is working now than was working in 1993. Whichever survey you choose to to believe either GHS or or QFS. Um and at the same time for sure the unemployment rate has increased um quite substantially since then. And I will stop there. Thank you Andrew. Um I think for me what's quite striking is that two surveys can ask almost the same question but give us a very different picture of the labor market and even after looking at your your slide on the weights and the samples I think there's no simple explanation for that gap. So there's quite a lot for us to unpack during the Q&A and discussion. I will now open it up to the audience. If you have a question please raise your virtual hand or type your question in the chat. I see you already have a question in the chat from Dan Stian Gump, but I will hand over first to Tando. You have your ra your hand raised. Please go ahead and unmute yourself and ask Andrew a question. >> Thanks so much, Andrew. That was very informative. Um, I've got a question around I think you you put it under number three. So, broad survey effects. I mean, and and and and and the the the panelist just mentioned it. I think there's a strong reliance on these questions being asked exactly the same way because the assumption is that uh when these people get to my house they speak English, right? Because the survey questionnaires are in English. I can tell you for free I hardly ever take a call from a call center in English, right? Because I can hear that the other person understands my language and I will automatically switch. So I think I think there's value in maybe questioning status around that and the reason I mention it is that when you do switch to let's say is closer right the way that you ask the question is leading whereas if you are reading it in English it may not be so so the way that you ask me um like do I work it's very different to the opposite which is like you can almost tell I don't want to say the length of the conversation but if someone knocks at your door and is asking that way it may tell me that if I say yes dear then it leads to a wider can I sit down whereas if I just say then it's a shut the door thank you Jehovah's witnesses um I don't know if I'm allowed to say that but um I think I think there's something worth with with worth worth exploring there and then I also have a question around you know h how status a takes the the survey. I mean, is it telephonic? Is it physical? I'm asking because and I'll post it in the chat here. I was reading something from um a company called Ask Yazi and they were talking about how ineffective surveys are. Uh what they did is they there was a Hierrox event, right? It's like a an exercise event for people, right? Uh there was a hierox Cape Town and previously they used to send out the survey about a week later via email to participants right this year they sent it via WhatsApp they literally sent it on the day probably straight after the event and they also sent it to uh spectators and the results are just wildly different right but the main thing that I took out from that was that they saw a better response because people can respond when they are home or when they have downtime to do this. So, the message can come in at 11:00 a.m., but if I've got time at 3:00, then that's when I respond to it. Whereas, if you're calling me at 15:45, I'm in a call, I'm chatting to Andrew, I'm going to hear that you're asking about this and I'm just going to drop the phone or I'm never going to get back to you. And the reason that I mentioned this is that I was at a consumer conference about I want to say three weeks ago and someone pointed out that everyone is collecting data from consumers but no one has really figured out how to how to incentivize consumers to first respond and to to respond truthfully which is something that you touched on here um which which came up where people are fabricating actual results. So, if Vodiccom, if I take out a contract with Vodiccom, unless I have an especially bad uh service rep uh experience, I don't have an incentive to go and rank them after the fact or or rate them after the fact. Same thing with the the hierox. But if it's on a WhatsApp chat and it's sitting there, I may or may not respond. But I can tell you if you're calling me during work hours and I'm not getting anything out of it, I may respond in such a way that I can tell is this going to lead to a question too or is this person going to drop the phone hopefully. Um, so just something worth worth exploring. Thanks. >> Thanks. >> Okay. Um, there's a lot of stuff there to unpack. Um, I'll just respond two things. The language of the interview is yes, very important. Not something we investigated. Um, and we didn't ask Stat to say about that. My understanding is kind of similar to yours that the question is only in English and the field workers are responsible for for translating that. I guess the question is why you know that's a big you know the bigger question how accurate is it if I you know have received a questionnaire in English but I'm a field worker and you know suzu is my home language or whatever. Um, but the question then is why would the GHS enumerators be speaking uh, you know, or or saying doing their own translations on the fly and getting massively more employment than the QFS ones presumably between the GHS and the QFS. There's not, you know, only a certain language group in the GHS and a different one in the QFS. I don't know, something like that. So, yeah, I mean, I I think that's very important. you're raising a lot of big issues about surveys but just to say I don't think that can explain these strange patterns that that we see even if for example it might be uh you know an explanation for overall employment being too high or or too low yes status does do face-toface interviews um they're very expensive um you know they have a huge fieldwork team they spend a lot of money trying to do a high quality survey and you know WhatsApp and you know calling people and whatever the response rates are generally much lower um and potentially therefore more unreliable. So, you know, a gold standard is doing a face-to-face uh interview and status say, you know, the their response rates are pretty pretty good. I would say way higher than than any other kind of survey. Telephone survey is 20%, 30% maybe. Uh status has got 85% uh or higher. Yeah, >> thanks Andrew. I'll go to um Dan Skin Gump and then we'll follow through with Elvis. Dan says that um there are also important differences between the QES and the QLFS as well as SARS tax panel. Could you comment on which sources you think are most representative of development in median earnings and employment changes? >> Okay. Uh that's a complicated question involving multiple data sets and multiple uh labor market outcomes. Um I'll start with the easiest one. Uh the currents the QFS [clears throat] earnings data has undergone some changes since 2022. Um and so it's I would say pretty good um in terms of getting us changes in in earnings say between last year um in terms of median earnings. So, that's a nice one because that's someone sitting in the middle of the distribution. Um, you know, the the tax data is going to tell us how the people at the very top are doing and do that excellently. Um, but it's not getting anyone who's um who's in the informal sector. So, I I think QFS pretty good for now. Um, one of the complaints I have about the QFS is that they changed the methodology for how they do the imputations 2022 going forward. If you want to compare to previous uh pre2020 let's say before co you can't do that at the moment legitimately you can do it the data is there but the data that's available on his bal web is um using the old methodology so you're basically comparing apples and bananas u and yeah so so that's not that's a slight kind of nuance but if you want to compare 2025 in 2024. Great. I mean, the problem at the moment is that 2024 was only released, I think, a few months back in July. So, the QFS earnings data uh comes out with a huge delay. Um, and that's also a problem. So, the QFS employment data comes out very quickly. Um, and the QFS is comes out with quite a quite a long delay. The QES is a whole different story. Um, and I don't know enough about exactly the Ernie's questions to to be able to say much about that other than SARS and QES are kind of very different from from the QFS. If we care about the median person in South Africa, you should use the QFS, but you shouldn't be comparing pre202 and post 2022. >> Thanks. I see an applause from Dan. So, he's covered. Elvis, please unmute yourself. We would just ask that you please keep your question brief and to the point. We have couple of questions in the chat but please unmute yourself. >> Thank you. Thank you so much. Uh thanks to Andra for for the presentation. I have two quick questions. The first one revolves around um some differences in what the two surveys do cover the GHS and the QFS. and particularly that relates to the question structures, the sequencing of those questions, the reference periods, um the waiting and calibration as well as the age groups covered by those two. Um I think it's not worth for me to mention the differences between those variables. They are quite clear. So my issue is now um within this analysis how is then yes that been addressed. For example, I'll mention one when stats is conducting the QFS surveys, it it makes reference to one week period prior to the interview. Whereas the GHS is just an annual I don't know how that was balanced. So in relation to all those other variables that I I've mentioned before, if you may please shed some light. Uh we have tried to create a balance um uh along those lines. Then the last question particularly relates to the GHS variable in terms of measurement of employment um and un employment by by official uh definition. Um you would recall that in the QFS um the standard three conditions for one then to be defined as being either employed or in or um unemployed with reference to those um um variables I I I mentioned before. It is that one is without employment and is available for work and is also actively seeking for work or making some active steps to seek employment. The ghs seems silent about that. Those are some of the aspects as well which it doesn't cover. So in making these kind of analysis and and some comparisons between the numbers of the employed between the two surveys, how is that been addressed within your paper or if not um what could be your views in that regard? Thank you. >> Cool. Um yeah, so so the details are are there in the paper, Elvis. I'd suggest you go and have a look at it. Um and kind of you know, it sounds sounds like you know the QFS well. I'm not sure if you you want to identify yourself as working at stat South Africa if indeed you do but uh um uh the ghs also has exactly the same question on employment. That's that's why I said it in the presentation. It's about the pri pre prior week and we have in the pen next to our paper we have comparisons of 2017 cure fest just as an example and the 2017 ghs. So in terms of employment they're identical. There are a few small differences between them on on um on some of the other things like NEA versus unemployed but but in the end we checked that and and the differences are really pretty minor in terms of how many people answer the QFS which has this extra question I think I can't remember the details but basically we did we did compare them very carefully both the questionnaires and the responses people gave to different questions and how they're then classified into the different uh statuses and and our view was actually they're pretty similar, very similar in in employment pretty much identical. Um and in the other questions pretty minor differences as far as we could see unable to explain any of these differences. >> Thanks Elvis. I hope you're covered. Um Andrew Neil Coleman had a question but I think Amy has addressed it. His follow-up question is has stats looked at your paper and what is their response? Um we haven't had an official response from Stat South Africa. We um did ask people uh questions quite a lot of questions um probably too many questions from their perspective um about a whole bunch of different things relating to these two surveys and so we had input from them which was extremely helpful but we haven't had an official response on you know why are these large differences and and you know can stats as they explain them. So Neil, no we haven't had an official response. >> Thanks. And then I have something from Gabok who asks until the discrepancy is resolved, how should policy makers use the statistics in practice? Should the QLFS remain the primary measurement or measure or should government triangulate it systematically with the GHS and administrative data or should official reporting present a range? Um yeah, I don't think that our paper suggests that there's a need to deviate from the cure when it comes to measuring unemployment to employment. Um perhaps that sounds strange considering how large the differences are, but [snorts] until we better understand the problem or the differences and where they're coming from, it doesn't make sense to swap to something else. As I said, you know, if you made me say which is more correct, I would say because of the QES results, I'd probably say that the QFS is more likely to be correct. I don't know that. I'm not confident about that. But if you made me choose one, I'd say the QFS. So, can to answer your question, I don't think we should be changing. I think this is not, you know, this is one paper by three people. um stats say I think should should do some more work and hopefully other researchers can also do more work to to try and think about these differences and until then I don't think we should make any wholesale changes to to the way that numbers are reported by government or or whoever else but I think you know nothing to stop anyone you know the QFS I think is the official labor market statistics and I don't think our paper provides enough evidence that we should you know there's a very strong case to be made that that's say or anyone else should move away from you know from that any in any case the unemployment rates don't differ that much and that's you know I think the thing that most people seem to be worried about the employment rate is very different and and there I don't think that's not a statistic that gets spoken about a lot anyway so probably you know most people not too bothered about it I mean I think we should yeah Hey, >> Neil Coleman, I can see your thing in the comment in the chat. Under employment, we have uh if you mean like who's not working very many hours, the ghs doesn't ask any questions on hours worked. So, we can't say anything about um anything about undermployment in that sense. >> Thanks. And then in the last two minutes, Andrew, there is an additional question from Gavo and he asks he or she asks, "Can you tell whether the additional employment captured by the ghs is concentrated in particular kinds of work such as travel, intermittent, informal um without knowing what kinds of jobs account for the gap, how should policy makers interpret the employment recovery?" >> That's a great question. Thank you, Kavo. Um the short answer is we can't tell. Uh the ghs is by its name general. It asks a lot of questions about all sorts of things. And so the space in the ghs for um for asking lots of questions about employment, the type of work exactly as you say, short hours, casual etc. that's uh very very limited. And so the short answer is we don't know much about the work other than uh in terms of earnings. um in the ghs and there we run into the problem that the QFS earnings are uh I would my own view is not good um and so I would be nervous in this postco period about comparing the QFS earnings and and the ghs earnings but that definitely could be u uh one option so you know what is your monthly earnings as you say if it's casual if it's intermittent or form informal if these you know extra 2 million jobs in the G GHS are kind of real. Maybe we should and and where we picking up where the difference is coming from. We might find, for example, that the GHS has very, you know, lower levels of earnings. Um, yeah, a longer bottom tail, etc. But that's that's going to be a bit complicated. >> Nice. And that brings us to the end of the seminar. Huge round of applause for Andrew. Thank you.