Ethics Pathways for Evaluators

Good day and welcome. It's good to have you here at this AES New South Wales Network event where we're going to be talking about ethics pathways for evaluators. So good uh to be thinking about this and we have three people who are going to be sharing with us about pathways, maybe some tips and tricks as well. Um, our first speaker is Kieran Gregory. Kieran is currently a principal adviser with the Office of the Public Service Commissioner. He's been leading the integrity and ethics function. The team's recent focus has been on developing and embedding a new code of ethics and conduct for New South Wales public sector employees, as well as strategies to build pro-tegrity cultures in New South Wales. Kieran has previously worked in whistleblowing prevention in Queensland and New South Wales and is a sessional academic with Qout Faculty of Business and Law. Our second speaker today is Kelly Jones. Kelly is a proud Gomeroy descendant through her maternal grandfather with family connections to Boommy in New South Wales. She was born and raised on Gatagle country and now serves as the ethics coordinator at the Aboriginal Health and Medical Research Council. She is the ethics coordinator and has held that position for five years. Kelly has shared her knowledge and experience at a number of national forums these last few years, including the Licha Institute International Conference, the AH andMRC ethics forum, as well as the policy health summit and the reconciliation Australia conference. She's clearly passionate about supporting researchers and institutions to understand and uphold Aboriginal governance and research, advocating for culturally culturally respectful approaches that lead to meaningful outcomes for communities. And our third speaker is Gerard Atkinson. He is the managing director and founder of Iris Ethics. This is Australia's first human research ethics committee dedicated to serving the evaluation, social research and market research sectors. He began his career in market research and evaluation in 2001 and started Iris Ethics after being an executive in the ARTD consultants and uh Australian government agencies. at ARTD. He led internal training on ethics and HRE processes as well as delivering low-risk reviews of projects for the firm. So before I um hand the conversation to the team, I'm going to take a moment to acknowledge that we are all coming together on various Aboriginal lands and pay respect to the elders of those nations. um acknowledging the wealth of knowledge and sharing of knowledge that has come before and the privilege that we have to come together and continue to share knowledge and learning. It's a privilege. I'm so glad to have you all here and I'm going to stop talking now and we're going to hear from our excellent panelists. Um Kieran, >> take it away. >> Great. Thank you so much um Emily. Um, and I'd also like to acknowledge um that I'm joining uh this meeting from Gatagle country today. Um, now I had a deck, but it's currently corrupted and not able to be opened. So, um, you'll just need to listen to the dosit tones of my voice for the next 10 minutes or so. I promise there'll be sufficient particulars in here in relation to this topic. Um, and look, this is a topic that is really close to my heart. It's something that I'm really really passionate about and I was so excited to get the invitation to come and talk to you about uh what to do when we identify there might be a little bit of an issue while we're undertaking an evaluation. I think it's worth acknowledging at the outset that evaluation is such a critical part of the policy process and indeed of project management. Um and we know that uh a lot of the times uh it's not happening as consistently uh as it perhaps should be. Um we also know that it's such a great opportunity to look at a project and then reflect or a policy or research and take a step back and say okay well what worked what didn't um and what can we do differently moving forward. It's also a really great opportunity, and I'm preaching to the converted here, I know, to identify those systemic improvements and then really reflect on, okay, what can we do differently down the track? For those reasons, it is possible, I'm not going to say likely, but it's possible that while you're undertaking an evaluation, you will come across something that uh you do not necessarily agree with or is egregiously wrong. um you might even come across serious from joining. And so what I wanted to do today was just have a bit of a discussion in relation to the formal pathways that exist in the New South Wales government sector um as they relate to raising issues of concern um or allegations of wrongdoing. But then I also wanted to dedicate a bit of time just to reflecting on um being really strategic in terms of how you prosecute different issues uh and reflecting on well what's the best way to actually get this resolved. The other point I'll make at the outset before I go into the particulars um is that I think it's important for us to take a step back and reflect on uh if we are identifying an issue is it uh an an issue that was open to the team or the decision maker to make. So, it was a correct decision or a preferable decision, but it wasn't a decision that you necessarily agree with or has there been a breach of policy? Um, or you know, has there been a breach of legislation? Um, and these are huge risk points. The reason that I think it's important to differentiate between the different categories is it's relevant to the process that you'll pursue, but it's also relevant in relation to the strategy that you'll pursue as well. Now, for those of us who work in the New South Wales government sector, you hopefully know that um as of the 1st of November last year, uh the public service commissioner adopted a new code of ethics and conduct for the government sector. So, this um code was uh had a really lengthy consultation process. It hadn't been reviewed for a decade or so. Uh and so what we wanted to do when we were developing the new code was make sure that it captured the the breadth of issues that we're likely to come across um in our work as public servants. We also had to balance that by make with um the fact that we've got such a diverse sector. We've got agencies and departments that um do vastly different work. And so it had to be a document that applied to everyone. The impetus for the change uh or one of the impetuses for the change was um a shift in the requirement um in the enabling legislation that the code is issued under previous and it's important I promise I don't want to get too in the weeds but um so previously the commissioner issued a direction to agency heads that said you need to take steps for your people to comply with the code that shifted to people in the government sector need to comply with the code. So or government sector employees rather need to comply with the code. So that shift in emphasis combined with the breadth um of additional issues that the new code covers means that it's really really important that everybody's got a great understanding of what's in it and then also what their obligations are in terms of reporting wrongdoing. So through an evaluation you may come across uh you know uh breaches of policy for example you might identify um issues from um employees where uh you think hey look they haven't actually complied with with the code you know they've breached their obligations here. So the minimum standard is that where you become aware of wrongdoing um you know it should be reported um and uh different agencies uh have shifted some of that language to must be reported. So what I would encourage everyone to do is just make sure you've got a really good understanding of that code as a starting point. um and then reflect on okay well as I'm going through the evaluation are there potential issues here that I need to identify and deal with or identify and raise rather in terms of um so so that's that's the code um now different agencies have added different things to it as well so um one of the tricky things about the public sector is we've got so many different pathways that people can pursue so the best advice and different obligations. So, what I would encourage everyone to do is just make sure you've got a really great understanding of the code as it applies in your agency um because that will give you an understanding of the types of issues to look out for. You would also hopefully be aware that um that uh in a couple of years ago we got new whistleblowing legislation in New South Wales. So, this was great. This strengthened the whistleblowing protections in New South Wales. Uh and uh there are now a number of different categories of wrongdoing related to things like corrupt conduct and maladministration and misuse of public resources. So we're talking about the serious stuff here for the that kind of serious stuff. Um you're not expected to assess it yourselves. You're not expected to go off and get all the evidence or anything like that. But if you do identify something where you're like, "This doesn't pass the sniff test and I think it's quite serious," you should have a look at your AY's PID policy um which will detail what you need to do uh in order to um make one authorities. One of the benefits of the new legislation is that you can make them verbally, you can make them in writing, you can make them to your manager, you can make them to a range of different people listed in your policy. So this is useful. It means that you don't have to find the right door in order to ventilate your concerns. Now, um it's not your job to investigate. So whether or not you identify a breach of the of the code potentially or if you think, hey, look, this might fall in that pit category, it's not your job to go off and do the investigation. And please don't do that. Um particularly if you're then starting to access information that you shouldn't be. I wouldn't want to see you guys get in trouble um as a result of you know accessing records and things like that um because you were trying to pull together evidence to support your complaint. I'm mindful of the time so I'll just quickly talk to uh complaints um and you know and how to practically raise um either a breach of the code um or potentially something more serious. One of the uh things that we um talk to people about in terms of complaint making is just making sure that you've got sufficient particulars there uh in order for um the whoever's assessing it to really get a grip of what the issue is. So identifying clearly here is the issue. Um if you think that there's been a breach of policy or a breach of legislation then articulating that and then giving some information in terms of why you formed that view. Um so again we're not going off and getting evidence but we want to give the decision maker or the assessor like enough information so they can say okay on the face of it this warrants a further look. Now, there will be times where um you just don't agree with the decision. Um so, it might conflict with your personal values as opposed to um as opposed to being a breach of um a policy or legislation and the like. I think that they're still worth raising. But the way and the reason that I say that is that talking about values and values based decision- making and how our personal values can come into conflict with others is actually really useful in terms of getting consistency of practice moving forward. And it's also a great opportunity for whoever's made the decision or has, you know, done something that you don't agree with to reflect and say, am I going to back myself and say, yes, I think that, you know, that was right or um or they might say, actually, no, now that you've raised it, I could have done that differently, but here are the pressures that led to me doing that. So maybe we need to have a look at those. for those kind of issues. Um it's for my personal take it's worth uh having conversations with the relevant team. There's no formal pathway for those. Um but I think those internal discussions are really valuable. Um and I really hope that you're working in agencies where um you are able to have those conversations productively. If you can't, then it's really important to build the business case for why the stuff you do is so critical because it is. Um, I think that's my time. >> Yeah. Thank you, Karen. Those are great tips and traps and things to be thinking about. Um, as we uh now we'll turn to Whoops, I've lost you. Where'd you go? We'll now hear from Kelly. Kelly, would you uh please? >> Sure. Thank you so much, Emily, and thank you again for inviting me to speak about something that I feel very passionate about. And this is coming from a secretariat side of things as well. So, um I am my name is Kelly. I'm a Gmory descendant and yes, I am working as the secretariat and ethics coordinator here at the AHNMRC. So I've just got a presentation on basically a little bit about our committee for those who don't know about it and essentially the the pathways we use in order to review our applications. Um and I'll also discuss about the quality improvement projects um as well and how we determine whether or not when it needs to come to us and how we sort of review that. Um and um yes, so I'll just share uh the screen with you now and um get that under way. Okay, can everybody see this? >> Yes. Uh additional criteria >> is it is it all there? Can is it the ahmi? Perfect. Okay. So um first of all what I really always like to do is just uh provide just a little bit of insight and I've got a small video which has my incredible elders who established our committee in 1996 and um also our co-chairs um who just speak uh a little bit about why we do what we do and why and just a brief history. So I'll just play that for you all now. At the core of what the ethics committee is and the role of it is actually just to make sure that research is conducted in a culturally safe way and to the benefit of uh New South Wales aberiginal and tourist islander people. >> I think it's really to try and uh have a central place for which researchers had to put an application in where it can be reviewed against a particular standard and seen as appropriate or not appropriate for for our community. The AHNMRC established the Hatre uh was it 26 years ago? >> 1996. >> 1996 because the local atresses in New South Wales were conducting the research reviews themselves. >> The elders decided we need to step in. We need an ATRI. >> The elders really built the ethics committee and the legacy that we have today is amazing because of them. We could not do the work that we do without having the elders come through and break down a lot of the barriers. >> We first started, we come on the committee because we was um board members of the AH and NAC at the time. We had a broad range of grassroots representation on the ethics and we still sit there today. when we was able to turn things around with the research that was done in the far west, it just wasn't appropriate research that was conducted and how it went. It had an impact not only that we were the first nations people here but also too it had an impact on our dream that went on for some time that research before we was able to get it squashed and I think that was a good outcome for us to having something turn around that would have been very detrimental to to our culture and and to our people. when people are conducting research, we're often directly affected in one way or another by the research subject. And I think it's it's incredibly significant that we have an aiginal ethics committee that then reviews those applications with that kind of a perspective. It's the mechanism to make researchers have to have Aboriginal people involved with research that impacts us. The key thing that we look for is how are ainal tourist people involved in the research you're doing? Without the ethics committee, I think we'd have a lot of cowboys that are out there doing things their own way without taking into consideration the needs of our communities. If you do want to submit an ethics application, we have all of our materials online that outline how to submit an ethics application with the AH and MRC. Also on the website there are details of when deadlines are and when meetings are on as well. So all that information can be found there very easily. >> This is something that people are passionate about and are willing to work towards and when we give people feedback we are honoring the commitment that the elders have placed on us to continue their legacy. So that's just a little understanding and a brief understanding of our history and what we do. I always like to um show that video because it really promotes the importance of Aboriginal ethics and also our incredible elders. Um now um just a brief rundown. This is our ethics committee at the moment. Um so when you are putting in your evaluations or research projects this is who will be reviewing your applications um and we've got obviously the beautiful Aboriginal elders who establish that and then um our co-chairs um researchers legal reps um and all Aboriginal um people uh except for David Pierre's who's a researcher and GP. So sorry my thing has just there we go. So they've spoken about the history you know just you already know it was established in 1996 and uh the reason for it was because the aberiginal medical services themselves were conducting um were reviewing the research that was being done within their communities and then they decided to all get together and and establish this. So um essentially that's how it became. So in terms of uh when we look at an evaluation or a research project, we need to make sure and this is also on our website for you guys to to have a look at in terms of pathways. when the experience of Aboriginal people is the explicit focus of all or part of the research. When data collection is directed at Aboriginal people, when Aboriginal people are to be examined in the results. Um when the information has an impact on Aboriginal communities, when Aboriginal health funds are a source of funding. Um and then obviously we also look at uh the Aboriginal experience or medical condition being studied or known or likely to be different from the overall population. are the Aboriginal people uh who use the service. So you might be doing it on the Aboriginal medical service, an ATO or PHN um and Aboriginal people are known or significantly uh represented within the group being studied. Now in terms of doing quality improvement projects um in evaluations, I know that there's a lot of that we determine whether or not a quality improvement project needs to come to us is if it is going to be reported on um in a public domain. Um, so if it's only going to be internal, then that's fine, but in terms of the processes and pathways, that's what we look for. So, always follow our criteria. It's on our website. Um, and yeah, and then our application uh processes and assessment. So, basically, we assess um just like most HRES as well. Um but the only difference is is that we need to have Aboriginal governance um on our um applications such as support letters or an Aboriginal reference group. Um we do have flowcharts on our website as well and all the resources there. Um and uh that essentially is um a main component in comparison to the other hatres um in terms of uh when you're submitting evaluations etc. Um and then obviously if you're doing a national project you will need to go to the other ainal hat tracks within New South um within Australia. Uh currently we also know that Lwicher is looking at a national aberiginal hre as well. So in terms of pathways as well that um is what we expect from and then obviously like most we do the national statement. So when we're reviewing as well our pathways of merit integrity justice benefici beneficients respect and vulnerable participants and then we obviously has been recently updated the ethical conduct in research as well. Um oh sorry that was a bit fast. So that's spirit and integrity cultural continuity um equity reciprocity respect and responsibility that we also look for. Um and these are all on our website as well. And then we have our resources such as the AH and MRC guidelines and these are our key principles that we also must go by in terms of reviewing our application. So we want to make sure that there's net benefits for the Aboriginal people and communities within the evaluation and research. um Aboriginal community control of the research that you've embedded the cultural sensitivity that there's reimbursement of cost for their time and obviously enhancing a originalinal skills and knowledge if if there's any um if you're able to within that evaluation at all. Um but in terms of that that's it from me. Um, I'm happy at the end to to speak to you guys in with any questions, but um, as I said, all of our resources are on our website there for you all. Um, in terms of, um, yeah, and yeah, that's it from me. That was quite fast. >> It was. Thank you, Kelly. >> Pleasure. >> Now, we're we're going to bring Gerard up to share about Iris ethics. >> Good day, everyone. And before I put up my own slide deck, I'd firstly like to acknowledge that I'm coming to you from beautiful Gundich Mara country down in southwest Victoria and pay my respect to Aboriginal elders past, present and emerging and extend that to all Aboriginal, toouristrit islander, Pacifica, Maui and first nations people joining us today. I also want to acknowledge that um you know it's a real honor to be on a panel with Kelly from the AH and MRC because when we looked at starting up Iris Ethics as a company, we looked at you know what are models of delivery of service that we want to emulate and strive to be like and you know AHMRC was one of those. So um we hope that we can you know do them proud in terms of emulating you know the quality of service that they provide. So I'll bring up my little slide deck for you all because anybody who knows me is probably familiar with the fact that I'm a former consultant and I would not be a good consultant without a slide deck. So you get a slide deck. And so we're Australia's newest human research ethics committee, at least as far as I know. We opened for business on the 1st of July this year. And we're the first open access human research ethics committee that's working in the evaluation, social research, and market research sectors. That's where we specialize and that's where we work. And our goal is to provide ethical oversight and advice quickly and at a very high quality. And we do that through having committee expertise that includes people who work within the evaluation, social research, market research sectors. Um, we've got people who understand the different policy areas, understand the nuances and the challenges of working in those areas. And we also bring in to that committee a diversity of lived experience perspectives, be that Aboriginal toouristr islander, culturally, linguistically diverse, people with disability, um, LGBTQIA perspectives. So that when we get an application, we match that application to members of the committee that understand the kind of work that is being done and are able to provide good insights relative to the national statement. So what do we actually provide as a company? So first off, we do the full ethical review spectrum. everything from those full committee reviews, but we also do lower risk review pathways. So things that don't necessarily meet that threshold for a full committee review, we can still provide that oversight and advice around how to conduct those projects ethically and consistently with the national statement. We also do statements of exemption for those cases where there is a need to have a formal review that says that no, you don't need a formal review. Um, little bit of a paradox there, but that's something we provide. But alongside those reviews, we founded this company with the goal of elevating the conversation around ethics in these sectors and trying to build a consistent practice, a transparent practice so that people knew how to work ethically and it becomes the, you know, the default in a in a way because there's information out there that can support good practice. And so we're doing that through providing resources for practitioners. So pretty much every week a new resource goes onto our website and that's freely accessible and it provides rundowns of particular issues or elements of the national statement and what that means in practice for evaluation. And we also include with that you know explainers, guides, templates and checklists that we're developing to also support good ethical practice. The other thing that we do and as far as we know this is first in market which is determinations. So if you're planning a project and you've got a question around whether something is ethical or not in relation to the principles of the national statement, you can actually apply to us and say look I'm thinking hypothetically of doing this hypothetically how would a committee um land on that and we can work with you to unpack that situation and identify how the national statement applies in that particular scenario and then give you at least the confidence to know how that can then be applied in your project. The third thing that we do is training. So providing you know custommade tailored advice and and training to agencies and firms around ethical practice and evaluation and how to make that work and that builds upon you know our resource base and the experience of our committee. So I mentioned the quality stuff now the fast stuff. We come from the evaluation sector and particularly for me I come from consulting and a rule of consulting is that your clients always want everything done yesterday and I've also been in government so I understand why everything needs to be done yesterday. There's time pressures and you know we wanted to build a service that could meet those demands. So we are an ondemand service rather than having scheduled meeting dates which means that each application is treated in isolation. We select a member of a set of members of the committee to look at it and they have 10 business days to do that review. And for other services we shrink that down to five business days because we know getting that feedback fast is really important. And to do that, we also provide ongoing engagement with applicants. So if our committee are discussing things and they've got a virtual platform where they can do that and we start to see there's particular issues that could be clarified, we can reach out to the applicant and say, "Hey, what did you mean at 12, page 12 of this application? You know, can you just clarify for us so that we're making decisions based on the best information possible?" Um, we also engage applicants. Should we have a sort of formal virtual meeting like we're having now, we invite applicants to that so that they can be there and talk with the committee through that decision process. So again, the committee is able to make the best possible decision based on the evidence. I've got a couple of flowcharts. Don't worry, these are actually on our website if you do want to to see them. So we've got a review process that sets out the the pathway from applying for review what we do on our side where there are decision points and things where you might have to provide more information or um be involved in the process and we try to make that as straightforward as possible. The fact that I can get it down to a page, I think is pretty good. And I'm hoping we can streamline that further as we build. Before I get into the other flowchart that I want to show you, I want to just bust a couple of myths um get on my soap box and anticipate a few questions. And the three myths that I've got are the first one that evaluation and QA are exempt from ethical review. That's not the case. Um I know that there has been statements put out there particularly in light of the NHMRC's guidelines on ethical conduct of evaluation and quality assurance activities that the national statement and the review pathways there don't necessarily apply. It's actually quite the opposite. um evaluation and QA are held to a higher standard than other research because of that extra document. So um there are different thresholds but the national statement still applies first and foremost. The second myth is kind of the flip side of all evaluation and QA needs to go to a committee especially if vulnerable people are involved. Um, this was more so the case with previous editions of the statement. The 2025 national statement has completely rewritten section 4, which deals with activities that are taking place with certain groups of people and it takes a a much more holistic view of risk and vulnerability. And so now it's the case that you treat each project on a case-byase basis and understand the different stakeholders, different activities and different risks. But it does mean that not all evaluation and QA needs to go to a full committee. It in my experience most evaluation and QA is probably best served by a lower risk review process. um but that there are definitely evaluations and QA activities that should be going to a full HRE. The last one there is saying evaluation and QA are less risky than clinical research. I don't believe that's the case. Just because we're not taking blood samples or um testing medicines does not mean that there are significant risks of harm to people involved and communities involved in that work. And when you look at the national statement, it provides a a large list of the types of harms that can occur including you know social, psychological, economic and legal harms that can exist. And I would say you know in evaluation particularly those risks are perhaps amplified because of the nature of the work we do and the contexts that we work in. So with all that myth busting there, just want to quickly flick to the national statement navigator that we've developed. And this is a flowchart that's on our website. And it's a really good way to look at your projects and say, okay, how do I identify which ethical pathway is most appropriate in my circumstances? And you can see at the start there, the first question is about harm. Is there a non-negligible risk of harm? If yes, straight to full HR. If not, then we go through the flowchart and you can see there is this breakout for quality assurance and evaluation as well. And it builds in the criteria from the national statement and the advice on evaluation and quality assurance. So, it's a good guide um and it can really help when you're planning your projects to think about these things. Of course, it's not universal and as we say, if there are questions, you can come to us. We want to be there as a service for the evaluation, market research, and social research communities. So, with that, I'll finish the slides. We're actually at a couple of conferences coming up, including the Australian Evaluation Society conference. So, if you do have questions that don't get answered today or you just want to have a chat to us about your projects, we'll be there in person at these conferences. Thank you so much. Thank you, Gerard. Thank you to everyone. I'm going to bring our speakers back to the uh foreground here. Scary thing being in the foreground. Um, what do I bring you back here? There we are. There we go. Welcome back everyone and thank you everyone. Um, please do join me. I saw some hands going up with the applause. Please do join me in thanking our panelists. What an excellent um breadth of um perspective and experience and opportunity and thinking spaces. Um it's fabulous. I'm so so glad to have you here. We've had a couple questions come in the chat which um about the Aboriginal Health Ethics Committee and Kelly, you've been fabulous putting in some responses. But for those who haven't seen the chat, there's been a question about data sovereignty principles. So please go and scan that. Um and Kelly, I might ask and uh well to the panel um positions about data security, data sovereignty. There's something we need to be thinking about any tips and tricks from our uh from our experts on the screen in that matter. Who's going to be brave? No one. >> Uh I I guess I I sort of answered it on on behalf of the AH and MRC in in that question box, but yeah, we just make sure that um in our ethics applications that there is like Aboriginal community control and authority throughout every stage where it's being stored. Um and um yeah, just making sure that um they provide us with all the updates, any changes throughout the entire process of their research project, but um it's specifically with data sovereignty, we're very very hard on that too. So we want to make sure that Aboriginal people, especially within the community, have that control over it. >> What sort of evidence would come into play for that sort of thing, Kelly? What do you look for? >> We look for where it's stored. We look for how long for, what's going to happen um with the results like with the data that they've collected. Um how long are you storing it for? We need you know the evidence. Um that's essentially what we ask them in terms of that. Um and every year they have to um in their annual report provide us with more information on that data um sovereignty and data storage. We've had one come in which sort of sits in line with that. How what's an example of a community retaining ownership? >> That could be um our Aboriginal medical services. So let's just say it was a health um a health research project in Mori to deal with cancer. uh the Aboriginal medical service would want to hold that data sovereignty and hold that data for themselves you know um and um make sure that it's culturally safe um that the data being collection collected is culturally safe and um and making sure that if anyone does you know want that data how you know how how will they be um uh reporting on that. So um and that's essentially where we come in. We want to make sure that anything that's being reported in a public domain is done in a culturally safe way. So that's that's sort of what we mean by community retaining ownership of data. >> Gerard, I saw you come off mute. Did you have something to add? >> Yeah, I just wanted to sort of add as from our perspective, you know, we would say that when it comes to data security, um, your practice is not in isolation of other guidelines. And so things like the Australian privacy principles are just as applicable in these scenarios. Um when it comes to data sovereignty as our HRE does undertake reviews of applications that do involve Aboriginal tourist rate islander communities. We've really recently updated our position statement and practice around that. Um which I direct people to the website to take a look at. Um but you know what we are looking for and from the committee's perspective is evidence of that ownership of data by the community um and respect for the fact that the community has volunteered that information um as something given to the the researchers, the evaluators, but it is still owned by that community and has to be respected as such. >> Okay, we've had a a question come in. How do you distinguish if at all between the ethical risks from the program being evaluated and the risks from the evaluation itself? This seems to be a particular challenge for evaluation unlike research. Thinking of the points mentioned around range of harms like economic, legal harms when evaluating compliance, regulatory or criminal programs, justice programs, excuse me, and those risks arise because of the program rather than the evaluation hearing you're nodding. What are your thoughts on that? >> I mean, it's a really live issue, isn't it? Um and I think from a practical perspective inherently there's likely to be some crossover. Um because uh a good evaluation may um like canvas some of those risks and uh identify and then like kind of respond to those risks and reflect on well were they managed effectively. Um, so depending on how you're structuring the methodology of the evaluation, like it's potentially covering some of the similar ground. Um, so I just think it's a great question. I don't have an easy answer there in terms of how to differentiate it. Um, apart from really thinking through, um, really I think talking about it internally. So this is what I always come back to. So talking about it internally and having those conversations to help kind of tease out um where the risks are differentiated um even if they're broadly the same there might be kind of minor differences between them but I don't have an easy answer. I'm sorry. >> Yeah Jerro. >> Yeah I'll I'll throw in my two cents on this. You know, whilst it's easy from a theoretical standpoint to distinguish the evaluation and the evaluand, um we know as practitioners that's, you know, they're intertwined. And so in doing the evaluation, we're trying to consider the harms in the context of what we as evaluators may encounter in the process of doing the evaluation. And so if if the evaluand carries with it certain risks and a great example is one where you are working as Robert you've said in the criminal justice system where you could be identifying activities that are illegal activities that may require some form of mandatory reporting framework that has to be taken into account in the design of the evaluation. And as a committee, my committee would be looking at what are the mitigation strategies that you have in place to address those potential scenarios as as and when they arise. So we would be looking as part of the evaluation how that interacts with how it reflects the nature of the program and the inherent risks within that program itself. We've had a question come in that says, "How do we and they're pointing at a yes, but I I feel like it's all of our responsibility to prevent improper practice in terms of following ethical ethics guidelines um with priv by private consultants. So clients from the public sector or NGO services might not be experienced and experienced enough to recognize when the services they procure are in breach of ethical guidelines. Any suggestions about how to navigate that? I guess for us it's more education. So we provide lots of education sessions um to institutes hatres um holding things like this. I I think for us in particular we just try and educate as many people as possible um and try to refer them to our guidelines um or or send them out as well in hard copies. I know that's a bit old school but sometimes that seems to work as well. >> That's what we do. >> Yep. Um, and there's been a question come through. So this is the New South Wales network of the Australian Evaluation Society. If we're working on a project that is carried out in multiple states and territories of Australia, do we need approval in every state and territory in which we're working or are there um places where you can get mutual recognition and approval? Kieran, you've come off mute. >> Um, that was a rookie mistake on my >> um, look, requirements may differ depending on the jurisdiction. So, I'm not aware of kind a whole of um, Australia or jurisdiction approach in relation to that. Um, it's likely that um, approval will be required across the board. >> Um, I probably need to jump in there and say that's not the case. Um, we are a HRE that operates nationally. >> Oh, brilliant. >> We're not the only one. Um >> what it does mean for us operationally is understanding the jurisdictions in which an evaluation or an activity is taking place and then ensuring um that not only is the program in compliance with that but that our responsibilities as a HRE are also in compliance with the relevant jurisdictions. I mentioned earlier mandatory reporting is an absolutely great example of this because it is very different the frameworks around mandatory reporting in each state and territory and so as a committee we have to be across that. Now, um I will say that an exception around this, and look, I could get into the weeds around NHMRC's push for mutual recognition um of uh approvals across committees, but there is a major exception and that is around um work with Aboriginal and toouristr islander communities where you know I I think I should throw to Kelly there as the expert on on that particular area. Yeah, thank you. Uh, yes. So, the answer is yes, you do need to go to each state. So, you've got us, uh, then you've got Wahek in Western Australia, you've got AXA in South Australia. Um, and then you've got Mensy's uh, which is in the Northern Territory. Vacho in Victoria is about to open up their own one. I think Quake is also in Queensland is looking and then we've also in the process of Lich doing a national Aboriginal hat track as well. So yes, just because in terms of countries um in uh everyone has different jurisdictions and laws lo and so we want to make sure that in terms of being culturally appropriate everyone um within different regions obviously works in a different way. Um so yes and we do understand that it is um consuming and and it and it can take a long time but usually once you've gone through the process once it's actually quite easy to go through the rest cuz we ask similar questions. Yes you might have to you know obviously um provide different feedback but this the standard thing is still there. So yes the answer is yes. I feel like that planning piece that all of you have mentioned from the start, if you're working from the start and really thinking through all of those aspects of who, the why, the how, the where, every bit and the whatifs, you're going to have a really clear um application. and to be able to go through that process and pathway um with a bit more ease and um comfort maybe. Um any other questions? Uh please feel free to um unmute and and ask a question directly if you'd like. >> Just just another thing, sorry I didn't actually mention in my presentation now that I think about it. In terms of the changes as well to the national statement in terms of risk um uh we do not consider any uh research to do with Aboriginal or touristra islander people uh to be of low risk. So any research we suggest you please put it forward to us. Uh we have uh just uh created a statement on that that is now on our website and we will be handing that out as well in hard copy um because we find that sometimes that works too. So, just so you know, um, in in terms of that risk when we were talking about it before. >> Great. Thank you very much. I'm going to be really cheeky and, uh, share my screen for just a second and encourage everyone to, uh, scan the QR code and provide us a little bit of feedback on what you've thought regarding this uh, session today. It'd be really great to hear from you um, and any ideas you might have going forward. But please continue to provide uh questions in the chat or unmute and ask a question as we have we still have a few minutes and we have our experts um with us. Um it's really I'm so excited that everyone came along. What a great u great crew and great speakers. Please uh join me once more in acknowledging our panelists. Um you've really made this a fabulous session. Thank you so much. I'm going