Evaluation Library – making evaluations available for learning and transparency

um really looking forward to hearing about the creation of this evaluation library and really seeing how we might all be able to use and interact with it as such a great resource. Um you would hopefully have all seen the links that were in the registration um when you when you first signed up. If not, I'm sure we'll be able to drop those into the chat at a future point in time. But I'd really like to hand over to Douglas or and Bridget um Van Rue to uh take us through to this afternoon session. Thank you. >> Thanks Caroline. Um thanks so much to the AES for inviting us uh today. Um we're going to be talking about the ACE evaluation library which um we're very proud of. Um and uh it's been a um uh a project that we've been working on for the last almost a year I think. Um we've been working on this in partnership with the um APO. Uh so um I don't know whether people can see my slides but um are you having a we've got a little bit of an issue which is that um Treasury we can't access Zoom. So, I'm I'm uh I'm accessing um Zoom through my um my iPad [laughter] and Caroline has very kindly agreed to um run the presentation for us uh to steer the PowerPoint. Um yeah, so uh my name is Doug Lassau. I I'm um an assistant director at the ACE. I'm based out of Melbourne, but currently I'm uh in um Western Australia, so I'm a couple of hours behind you. And uh joining me uh is Bridget Van Monroy, who is the director of the APO. Um if we could go to the next slide. Um and possibly we don't need to go to the next slide after that given that you've welcomed people. Um an outline of the presentation. So um we'll talk about the ACE. Then we'll also talk a little bit about the APO. We'll explain the context in which all of this is happening. We'll refer to the Commonwealth evaluation policy and we'll make a few references to the state of evaluation 2025 um report that we released in February. Uh we'll talk about the evaluation library um why what why we've done this and and how we've done it and we'll have a bit of a deep dive um into the library and Bridget will will do that for us. So we'll have a kind of live demo around how you can use the library uh to best effect and there's lots and lots of ways in which you can do that and then we'll talk about some of the next steps um now that we've got this up and running. So uh about the Australian Center for Evaluation um and next slide. Um so the Australian Center for Evaluation was um established to uh improve the uh use the quality and the use of evaluation evidence and ensure that um it supports better policy and programs to improve the lives of Australians. And uh our mission is to put evaluation evidence at the heart of policies policy design and decision-m. Uh and there you can clearly see uh why um uh an evaluation library of this sort would be you know um part of our agenda in in um enabling easy access to evaluation evidence. Um next uh slide please. So we have uh four key areas of focus at the ACE. Um one is uh evaluation, leadership and promotion and we do that through a number of um initiatives but these include the commonwealth evaluation policy that I'll talk a little bit about and the commonwealth evaluation toolkit um uh both of which you can find online through our website. Uh we are also uh championing um high quality impact evaluations with an emphasis on randomized control trials. Uh and we have a range of partnerships uh with um Commonwealth government agencies uh in order to to uh design and deliver uh randomized control trials. We also have a range of initiatives that are aimed to um build evaluation capability um which involves um the evaluation profession for example uh it involves a range of training programs that we're developing uh and also you know one of the projects that we're working on at the moment is the evaluation maturity model the development of an evaluation uh maturity model for the commonwealth um that we hope will be a tool to um uh guide uh um capability, organizational capability in particular and then uh a strong emphasis on the use of evaluation um in budget processes um and strengthening evaluation planning and use uh generally. Thanks Caz. Uh so we sit within uh the department of uh treasury and um there are two parts to the Australian center for evaluation. One is the impact evaluation unit which is led by Harry Greenwill who I think is on the call um in the seminar somewhere. Harry jump in at any point if you want to add something. And then the evaluation leadership policy and capability unit uh which is led by my director Sesan Butler. Uh the impact evaluation unit focuses on the delivery design uh and delivery and capability building around impact evaluations and uh ELQ uh bit of a clumsy acronym I guess um focuses on all those other things leadership policy and capability with a kind of emphasis on systemwide um uplift. Um, next slide Bridget, over to you. >> Oh, hello. Um, thanks Caz. We can go straight to the next slide. So, hi, I'm Bridget. I'm the director of APO. Um, APO was established in 2002 at Swinburn University as Australian policy online. Um in 2018 we changed our name to the analysis of Aussie Observatory. Um and this was to reflect the fact that we don't just collect Australian policy and research. We curate from New Zealand, the UK, the US and any other country where we think they're doing policy that could be influential and useful for Australia to know about. Um in 2024 we left the burn university and we moved to McKinnon. McKinnon is a philanthropic organization that is working towards better government for a stronger Australia. And so that brings us to today and on Monday you've got a sneak peek at our new rant and our new name, our new old name. So, we're returning back to Australian policy online. And the reason we're doing that is because we the fact that it says what's on the tin and it rea restates our focus that we are collecting policy and research for Australian um policy makers and practitioners and researchers and anyone else who works in the policy um ecosystem. Um so, what is APO? We're a repository of open access policy and research. We focus on collecting publications from organizations such as government, universities, research institutes, think tanks and advocacy organizations. We support EV evidence in composition and policym by making policy and research accessible and usable. And the main way we do that is by providing our free open access website repository and delivering free newsletters each week updating people on the latest policy and research. So this free service is supported by our commissioned work. Um we curate collections of research and policy um which is supported by our state-of-the-art information architecture and we also build engagement with um that policy and research and those curated collections. So you can see from below that we have um uh 44,000 policy and research resources. We curate on a daily basis. So that's always growing. Um we have 400,000 website users from Australia and New Zealand. Um this um turns into 2,00 and sorry 250,000 downloads per year. Um and that's not just the views, that's the download. And we have more than 15,000 newslet subscribers. um around half of those are working in government but another around a fifth around a quarter are working in non forprofits in the policy ecosystem and we've got another 15% who are working at universities >> thanks Bridget I'm just going to um frame uh the um discussion on the library by talking a little bit about evaluation in the APS um more broadly the training in public service. Um, next slide please, Caris. So, I guess one of the, you know, the kind of key um framers for the for the work that we do and for the the library um is the Commonwealth evaluation policy. uh and this is a principlesbased policy um and uh to support um fitfor-purpose evaluative approaches across the public sector and in line with the PGPA uh act 2013 the public governance performance and accountability act. Um the policy has uh kind of three emphases. One is to embed a cultural evaluation and learning. Um the other kind of emphasis is on measuring assessing and measuring impact and uh the other is uh improving um the quality uh of um uh monitoring evaluation. Um the the uh the comalth evaluation policy has a number of principles in it and you can see them here. fit for purpose. Um evaluations should be um uh commensurate with the the scale of the evaluation. Um the resource should be commenurate with the nature of the program. Um the evaluation should provide useful um it should be useful should provide useful uh um analysis and um results. It needs to be robust, ethical and culturally appropriate, credible and I guess uh this you know the ACE evaluation library really supports this final principle um uh transparent uh and where appropriate um uh and that that's where um you know there aren't where there where there are good reasons not to publish um evaluation u results for example for issues of um confidentiality or uh sensitivity. Um the next slide uh talks about the um the state of evaluation in the Australian government. Um you may have seen and if you haven't I encourage you to take a look at our website um the state of valuation uh in the Australian government report 2025 uh which is the first time that um uh stocktake uh and analysis of evaluation uh across the Australian government um has been undertaken and um that uh report looks at a number of um dimensions um the level of evaluation activity across the Australian public sector. It looks at um evaluation design and planning, the use of evaluation evidence and evaluation capability. Uh so in relation to the use of um evaluation evidence, uh we can see pretty um uh good results I guess um in relation to the use of evaluation in business planning processes. 90% of the um entities that participated in the survey, that was 92 entities um use evaluation across business planning processes. 68% use evaluations to um inform policy and program design and decision making in relation to uh investing resources in um in in in programs and policy. And then another 68% use outcome or impact evaluations to to get a sense of what um policies and programs are achieving. Uh the last kind of statistic I I want to um call out um and it's that 31% of entities routinely and publicly disseminate evaluation findings or reports. Uh and this is something that um we're seeking to change and and increase that percentage and the ACE evaluation library um is an initiative that will support that we hope. Okay, next um slide please. So yeah, let's uh focus down on the ACE evaluation library and purpose of of the library is to support evidence-based policy across the Australian government by providing easy access to relevant evaluation documents um that have been published or cleared for publication by Commonwealth entities. So it is very much focused on Commonwealth departments and agencies at this point in time. Although uh this may the scope of this may be expanded in the future. Um uh next slide please. So how have we done this? Um we've worked with the APO to develop uh an open access digital library. Um, and it is the first time that we have a kind of one-stop shop, a central point that uh you can go um as a um program designer, as someone who's wanting to make decisions about um um uh policies or programs based on evidence, you can go here and you can find uh evaluations from across the Commonwealth. And um at the moment we have 111 resources um uh but that is uh you know increasing daily. I'm not so sure daily now but um it it uh we are adding to it as we go. And one of the really interesting things about this um the development of the of the ACE library was the work that we did with Bridget um and across the ACE team. Uh and I kind of emphasize that this is very much a collaborative effort by the um by everyone in the team because we um went through a process of developing a taxonomy for evaluation which was added to the APO's uh public policy taxonomy. Uh and so um when we uh collected and classified evaluations, we tagged these evaluations in line with that taxonomy. Uh and so when Bridget starts uh to give you the tour through the kind of search functionality of the site, um that process of developing the taxonomy and and classifying and tagging these um documents that kind of becomes apparent in ensuring that you can find um the evidence that you need uh for the kinds of programs that you you're interested in. Um yeah, next slide please. Okay. Yeah, so without further ado, let's um take a look. Let's deep dive into the library and show you the power of it. Thanks, Bridget. >> Thanks very much, Doug. So, I' I've just got this slide here just to show you what we're I'm going to take you through today. um how to access the library um the different elements of the library including topics if I guess how you can browse the library and also how you can search for evaluations on APO and as well as in the library um and then I'll get talk to you more about how you can keep um updated about new eval evaluations being added to the library and to APO um so yes now I Will um cash share my screen. So the way you can easily access the library is go to apo.org.auu AU um and you'll come to our page um and then if you head to collections you'll see that we've got a number of um collections that we've been commissioned to um maintain at the moment and ACE evaluation library is the first one you'll see there. So, welcome to the library. As Doug said, we do have 111 resources. Um, at the end, we'll talk to you about if you um uh do have know of an Australian governor evaluation that can be added, how you can let us know about that one. Um, so we'll start from the bottom here and you can see here we've got a range of featured resources that are in um the library. We try to keep these up to date. So as new evaluations are added to the library, we'll add them here. So you can see there. Now the um we particularly as um collections grow, it's always useful to find new ways to enter into the collection and to focus on things that you might be interested in. And so ACE thought it would be good um that users may be interested in exploring the library by different types of evaluation. Um, and as the library grows, we'll be adding new topics in focus to address the diversity of evaluations that will be in the library. Um, so you can explore economic evaluations, indigenous evaluations. Um, and I guess one of the main focus of the library will be um, demonstrating different types of randomized control trials. So you can see here these are the randomized trials that we have in library at the moment. Um we have um 15 in total. Um so the library does accept um well the library includes any evaluations that have been offered um published or commissioned by the Australian government and their agencies. Um and that includes journal articles as well. Um so APO does include journal articles. Um we don't prioritize journal articles for our broader collection because um they they do have a way of being found and being made accessible. But we do um where we see fit or where it meets the criteria of our collections, we do add open access journal articles to APO. So here you can see um you can then explore um the randomized trials in the library by different subjects um the high level subjects or more focused subjects and I'll get to that more you can export by publisher by authors and geographic coverage but I will let's go back to the so you can see here you can go back to the library or you can go to another topic in focus. So let's go back to the library um and let's now uh browse the entire collection which is alone um evaluation. So let's talk about more about how you can find what you're looking for. So um I think one of the most helpful ways to look at the different libraries is that you may want to look at particular valuations in particular policy area. So with health I would recommend um rather than trying to look at the um the department um this is a particular problem if you look at um which department all of them because the department of health has many different names over the years. So the best thing to do if you want to look at health evaluations is look at on the top subject level and you can see that we have up there we can um we have 20 evaluations I think 20 evaluations in the area of health and then you can continue to um explore that more. Um so the subjects as um that was the aim we did add a lot of new subjects um to um APS public policy taxonomy. Um normally our subjects are about the topic but in this case it was really important that people could explore the library by different types of evaluation methods. So that we added those um subjects to our taxonomy and you can now look at different types of evaluation as well as different types of methods that have been used for that evaluation. Um so if we go back and look at all the evaluations in the library um you can explore it by um the publisher and a lot of the cases it will be the government of Australia but sometimes when um departments have commissioned uh a consulting company um to do an evaluation that'll be the publisher um and you can also look at it by orders as well as um organizations who have orders. So that's where you will find your um consultant companies who have been commissioned by government to do an evaluation and again uh of course these are in Australian government valuations. So most of them will be about Australia more broadly but sometimes there is valuations that focus on particular area and particularly the international relations evaluation will cover different countries as well. And again, you can see we have a reporting strategy there. Um, which probably shouldn't be there. So, I'll go check that out later. Okay. So, that's um browsing the collection. Um, now you might want to be you might be searching for a particular type of evaluation or a particular area. Um, so that's where you would use a search function. Now, it's not possible to search just the collection. Um, I probably think that's a good thing because we may have relevant valuations that you're looking for that might might be in APO and might not be collection. So, let's um look at um evaluations around maternity. So search for maternity evaluation and the reason I chose those search terms was to show how these subject top levels can really come in handy because um your maternity valuations might be related to health and maternity services but it al might be related to um economics and employment in terms of maternity leave. Um so that's where these um subject top levels come in handy to start refining what you're looking for. So you may decide that you're really looking for maternity evaluations around maternity services. Um now when um you may if you just want to look at um be searching for um evaluations um in the collection you can click on um the library and that way you only get or take off help when this way you're only getting the um the evaluations are in the ACE um ACE library, but you might decide to broaden out your search and look at um uh anything on APA to do with maternity evaluation. Again, you may want to be only looking at um valuations. So, we've got reports, journal articles, all that kind of thing, but you might be want to be focusing on evaluations and then you can just focus your search there. So once you've done some exploring and really you um been able to find what you're looking for um as I've said um well as we both said the library will continue to grow in years and also APO curates on a daily basis. So there's always something new being added to APO. So, how can you keep up to date with what's new on APO, particularly been added to APO and to the library? Well, um the first thing that's available to everyone is our free newsletters that I mentioned. So, you anyone can um subscribe to our free newsletters. Um the two main options we've got here is the policy pulse and that comes out on a twice weekly basis on Mondays and Thursdays and that will contain anything new that's been added to APO that's got all the latest and research um if you are not in on the um twice weekly newsletter we also have the policy weekly and that's more curated selection really f focusing on the most popular needs we So that will be of interest to our readers particularly the most recent stuff. Um so that's what you can do. So um then also there's an option of um two news two services that will also got features where you don't have to be um scrolling through general newsletters. Um, so we've got a service called um, sorry, see the top of my screen. We've got our AP service now to anyone with Australian government email address.gov.au email address. Um and it's got a number of features subject based newsletters and um save search alerts which I think will be of interest um for people who want to keep up to date valuations being added to APO. Um, but if you don't have a government email address, um, not to worry. We also have our my EPO class membership service and it has the same features of APOV except it's also got the metrics button um, which gives you metrics on how many times every result on APO has been viewed and downloaded. So, um, anyone can also add a publish resource to APO. um that don't want to tell you about um added getting things added to the library, but anyone can add a resource APO and then you join my AP plus you can see how many times it's been viewed downloaded. Um so the two features that we think that are open to my plus members and NP members is subject based newsletters. Um and the government newsletter comes out um once a fortnight and it includes anything to do with government um business which of course includes evaluation, evidence based policy making, regulation and just general things to do with public service. Um you can also um subscribe to more topic focused um newsletters as well. Um this is a really good alternative to receiving those generalist newsletters. And then slap searches. Um this is a great um feature where you can um search for something. So you could do a search um filter on the ACE evaluation library and save that search and that means you will then get an email every time a new evaluation has been added to the library. um you can tailor that search to what could be interested in evaluations on particular topic so that you will get um uh yeah an email let you know when that's been added to um I think that's it for the things no questions come up >> thanks Bridget >> um I think one of the um real strengths of the collection uh and obviously it'll become stronger as we add more documents um to to to the library um is that you can search for examp you know if you're if you're an evaluator and you're interested to see how um you know developmental evaluation for example has been applied within a government context text, you can search um search to find documents. Um if you're interested um as Bridget has pointed out, there's um dedicated collection for randomized trials. uh and that is a really useful um source of reference for for how how randomized trials have been um conducted within a government context. Um, and the the search functionality enables you to kind of drill down and and just kind of identify if you're particularly interested in specific methods, then then you can drill down into the library to find those those documents. Um, at the moment there might not be a huge amount of documents related to, for example, development evaluation, developmental evaluation, but in the future as we add that will grow. Um I think that's probably the main things to say about the library. Ah one one key thing I think we should probably add is that um the process of collecting the documents uh that are now in the site part of that came out of um departments um contributions to the stock take that we conducted uh as part of the state of evaluation. um uh report um development uh and so you know I think we had perhaps um possibly 200 evaluations that were identified through that process. Then it was a matter of finding those evaluations and then um tagging. So we're still I guess in the process of doing that. Um but it's important to flag that we have not um made any kind of assessment as to the quality of evaluations. We've um classified on the basis of the information that we found within the evaluation reports themselves and tag them accordingly. Um now there are some questions that have come up in the chat. Perhaps we should address them now while we have a a kind of natural break between now and the next section. So, >> so the third question, Doug, was how are the methodological tags identified and attached to each publication? Um, so maybe if we talk a bit more about how um how we came up with the subject terms. Um so um between um ACE and APO well really ACE really focused on the kinds of methodologies that they wanted um to be identified among the evaluations and then APO went through a process of looking at those terms and then adding them to our public policy taxonomy. And when an evaluation is added to the library, we look at the methodology section of the evaluation and then um choose the appropriate tags um to meet that um section. Yeah, it's a it's a it's been a really interesting process because in some cases it's been difficult to ascertain um the the kind of level of detail that we need for the the methodological um tags. And so, you know, the taxonomy that was developed kind of drills down one of the interesting debates that we had was what's an outcome evaluation? How does it deliver to an impact evaluation? when is something tagged an outcome evaluation. So um that there has been you know process of I guess um in some cases hasn't been explicit on the face of the evaluation document. So we've had to read around the document uh and and make some calls around what we believe the evaluation to be. In some cases um an evaluation has been identified as as one thing and we've looked at it and thought actually no this isn't that. Uh so there is there has been a process of um uh kind of judgment I guess around um you know around the classification and the tagging process. Um the other questions relate to um what kinds of documents can be added to the library or rather what um uh publishers can be added to the library. So at the moment we are only focused on uh Commonwealth government um published evaluations. Um so that's uh government documents that already out there in the public realm or that have been cleared for publication u by commonwealth entities only. Um we are thinking about expanding in the future to um uh uh states and territories but that's something that's uh a bit down the track. Um, yeah, but just to note, Doug, and I'll put uh some details in the chat, is that if you have another evaluation that you want to add to APO and um it's not doesn't meet the criteria in the library, you can still add it to APO. There's just a number of requirements to do this. Um it's a free service, so we do ask that the evaluation should be no more than 6 months old. Um, and it needs to be a published evaluation. Um, and by adding it to APO, you're giving um, APO permission to hold that evaluation. And I put a link to more details about how you can add your evaluations to APO chat, but please know it needs to be no more than um, 6 months of old. And this is because um our we like to put things that have been at APO into our newsletters and our newsletter subscribers really value currency. Um but if you do have an evaluation that's coming up and about to be published um add it to APO and we can um comply with any embargo dates that >> I don't know if you saw that Bridget but there was a follow-up question as to whether they needed to be an author to submit a publication to APO. >> No, you don't need to be an author. If you see a really great evaluation out there that you think everyone needs to see, you can add it to APO. The only requirement has to be published and publicly available and then we comply with any copyright or license restrictions as well. Oh, it obviously has to be open access. Yeah. >> Thank you. Are there any other questions from people? Anyone would like to raise a hand and actually speak? Uh, always an option. I think um I'd be really interested um Doug and and Bridget too since you've been part of this journey and you may or may not be able to answer. Have you seen that there's been a particular I guess type of evaluation that's been coming through more frequently and I I acknowledge that you've been kind of going back and forth a little bit and trying to decide what type of evaluation some of these reports are. Um but have you seen a particular type that seems to have been used more more frequently or more um across the ones that you've been looking at? >> I think pos possibly the most common are outcome evaluations and frequently um these are tagged as mixed or sorry not tagged but identified within the documents themselves as mixed methods. um which frequently means the use of um multiple methods of data collection and analysis not necessarily kind of mixed evaluation approaches. So it's rarer to see an evaluation that comp you that consists of elements of an impact evaluation with a randomized control trial plus um a more qualitative um methodological approach. Uh so I haven't seen that many evaluations within there that kind of use mixed methods in that way. It's usually more talking about mixed methods in terms of the types of data collection um approaches being used and the analytical methods. >> Um yeah um if you don't mind I can just share and um we can answer that question um >> with the stats >> with the actual um collection. Oh, don't know what's happened there. I'll share screen. Okay. Um yeah, so the most popular one and it's probably not a surprise to the evaluation out there is um process evaluation. Um so uh uh yeah, they make up more than half of the collection. And then as Doug said um another popular one is outcomes. And so just to also be aware that um we're we're going with the methodology section of the evaluation and sometimes that's very brief, not very detailed and sometimes it has amazing level of detail and that will determine how much we how many tags how detailed that hanging can be. Um we also go with a bit of a selfidentified except with the impact evaluations. We've been very careful to ensure that um that they are actually impact evaluations. There's a few evaluations out there that claim to be impact but they're more on the outcome side. And so we're very careful to try and be sure that they are evaluations. And so you can see and the thing around the mixed methods is to um yes there's a lot of mixed methods research but we have another term here um mixed evaluation approaches. Um so as you can see the next is research. Um we probably got more than that um um in the library. Um most most evaluations are using mixed methods but um we tagged it as mixed methods when they really call that out as a very um deliberate strategy to the evaluation. Um, but as Doug says, there's very few evaluations that have deliberately done different types of evaluation approaches. Thank you. Um, it's very interesting and I probably should should have delved into that before as well to see that, but it's been incredibly useful particularly from my perspective seeing how you actually go about navigating that and the different taxonomy that sits there and and what that looks like. It's really really handy to see that in practice. Thank you. Uh, do we have any other questions, Morris? >> Oh, sorry. >> Yeah. Hi, Doug. I just you've talked about including states and territory evaluations maybe a bit further down the track. I'm just curious to know if you've got any ideas on a kind of timeline for that. That's the space I work in and in particularly in health. There are a lot of things you know that we've or a few projects that we've evaluated in Queensland that have gone on to be rolled out on a national level or then gone on to be trial in other states or territories as well. So it seems like it could be a really good resource for either people looking at things that have been tried in other states or territories or looking at evaluations for things you know before they stepped up to a national level. >> Yeah, absolutely. Um so at the moment we have uh funding for this project for the next three years and um I think you know with after the first year so this started on July the 1st [laughter] um towards the end of the first year so you know kind of May June um next year we'll be kind of looking at at how the um the the library is going how it's been used who's been using it um and then I think at that point we'll will have further discussions around expanding that and then that will involve a conversation with states and and territories. Um so I wouldn't expect that you know we'd see something for the next you know 18 months at least. Um but I don't know Harry if you were on the line you might want to chip into this. [laughter] >> Hi Doug. Yeah. Um it's a it's a great question and um what Doug is delicately trying to do is um strike a balance between what we can manage as a team and the um excitement and enthusiasm that we're getting from uh many people about uh the the library and its potential. I guess um Morris maybe one thing that I can suggest to you and and to to others is that um the uh the challenge for us is that there there is a little bit of time um with that process of prepare like working through the taxonomy uh as as Doug and Bridget have described um like we think the taxonomy is going to be useful to people but It takes a little bit of time to make sure that's reasonably accurate as well as then preparing the the the summaries that we think are also important for for people to get a quick sense of of the evaluations. I think as long as all of that work lies on us then and then with Bridget's team to to review and and do quality assurance is going to be a slower process. So a question for us is whether there's some way in which we can um work with your areas where sort of you do the first pass and then um uh with based on some guidance that we prepare. So I think um I won't promise more than what Doug has suggested at this stage but I just say that um we'll give some thought to whether there's some way of um uh speeding up the process but that would probably be on the basis of um looking to other teams that were enthusiastic to see their work uploaded to provide you with the guidance and and then sort of pass some of the the work across to you to to support that. But um we're still working through how that would work. Um so um uh we we'll take you as another vote in favor of trying to uh get this moving faster if we can and we'll um we'll see what we can come up with. And by the way, sorry if people don't know who on earth I am or why I've jumped in. My name is Harry Grimwell. I'm acting assistant secretary in the um Australian Center for Evaluation. I'm filling in for Eleanor Williams who um uh will uh return to this seat next week after um acting for our first assistant secretary. Thanks. Sorry. I'll pass back to the chair and to you Doug. >> Thanks Harry. And if anyone is excited or keen and has the resources um to establish their own library or collection where we put it on APO and can start to build that um database, then obviously you're more than welcome to reach out to APO as well. >> I like this. It feels like uh we're not going to promise too much but live in hope. uh kind of conversation there. And I think Maurice, your vote was counted and it's been recorded so we can pass it on as needed uh [laughter] to support any future conversations. Um before we uh close for the session, are there any last questions? And if not, I'm not sure if I've done next steps or not. Doug, you're gonna have to >> Yeah, the next the next steps are very simple. Um, we will be continuing to add to this library on a monthly basis. As Harry kind of flagged, it's a um it's a kind of effort that's happening across our team and and um uh and it'll be a gradual um process of of adding. So, we'll be doing that monthly. Um and uh the way that you can contribute your um evaluations is by sending us an email. We have um an inbox. It's evaluation. I think it's on the last [laughter] It's on the last slide. Um evalu What is it? I should know this, shouldn't I? [gasps] Um yeah, it's just our standard evaluation at treasury.gov.auu. Uh and if you want to find out more about work about our team at the ACE, uh that's our website. Um we are in the process of uh refreshing the website and refreshing our toolkit. So uh yeah, there's plenty going on at the ACE for you to um to be interested in and get involved in some way or another. >> Thank you. It's very exciting. I I look forward to seeing it. Sounds like you still got, you know, another 90 odd uh coming. >> Yes. [laughter] >> Picked that up, didn't you? >> Yeah. So, I look forward and I might need to subscribe, Bridgette, to to see them all as they um go live on the actual website itself. So, thank you. It's been really helpful just having you both kind of talk us through both the construction and the and how you've landed on some of those taxonomy and the understanding how it's been built. I find really helpful in terms of how you then use it and interpret it um as an individual as well. And I'll be sure I'll be sharing this resources with quite a few of the organizations that I work and engage with because it's always easier when you've got a spot to head to to start off with rather than, you know, asking our good friend Google or uh whichever chatbot you're talking to at any particular time. Um, oh, and we've just had one last question pop in and since we do have five minutes, I was going to I was going to wrap up, but um, >> yeah, that's a fantastic suggestion. And yeah, we do have um we do have one of those performers that we did use with a bunch of departments to make the task easier for them and for us in terms of um uh in particular, I think Harry mentioned this, but there is a summary, we should have highlighted this, there is a summary that's provided of each evaluation um and so that focuses on um key findings and recommendations. So that does involve a bit of work and I think for some agencies they prefer to write that summary themselves so they can kind of most clearly communicate their um intent. Uh and so yeah we're we're happy to um we're happy to send that performance to those not quite sure how we do it but um yeah let us know send an email to our evaluation inbox and we'll we'll give you that. Um, and we'll also think about how perhaps we might be able to put that up on our site, too. Thank you for the suggestion. It's great. >> Sounds like they could be volunteering to be part of the piloting or triing of that uh option for you all. Um, thank you v both very much. Like I said, it's been really just great to hear that background and to understand how to um to work through the actual site um itself. So, I really appreciate you both spending the time this afternoon talking through talking us through