About this talk
In this talk, Tiffany Ferris, CEO of Palunteer, discusses the integration of measurable success and joyful collaboration within the context of a digital transformation project for the Wisconsin Department of Health Services. She emphasizes the importance of a people-centered approach, which focuses on team culture, psychological safety, and meaningful metrics rather than traditional performance metrics that could lead to burnout and dissatisfaction. Tiffany outlines the challenges faced during the project, including the need for data integration and platform modernization, while successfully adhering to budget and timeline constraints by adopting agile methodologies. She presents a case study of how her team fostered a collaborative environment that included clients in the development process, ensuring that all stakeholders felt heard and valued. This session concludes by highlighting the relevance of psychological safety and effective communication in driving project success.
Full transcript
Hello everyone and good afternoon. Uh my name is Tiffany Ferris. I'm the CEO of palunteer.net and you were in the room for uh serious results surprisingly fun a playbook for KPIdriven digital transformation. So I'm really happy that you're joining me today. I know there are some fabulous talks at the same time. So, um, this session is going to be a little bit case study, a little bit
process study, um, and also, um, an added huge dash of presentation karaoke because this wasn't originally my presentation. So, um, it was originally written by the agile coach on the project. I was the executive stakeholder. So, you're going to get my patina on this session as well. So what you can expect from uh this talk today, it's very much about how measurable success and meaningful joyful collaboration
don't have to be mutually exclusive. And in fact, I would argue that they're inherently linked. So I mean I hope you walk away knowing a little bit more about why joy is not anathema to work about how our particular project focused on both the people and the numbers in kind of equal measure how to strengthen your own team's culture and psychological safety and how to thread success
outcomes from the project from start to finish. So I'm just going to lay a little bit of uh background in the first section about the project about what we were doing. In the second section I'm going to talk a little bit more about the team culture aspects and then in the last I'll talk specifically about the way we decomposed metrics. So, uh, the vision for the project,
the state of Wisconsin, um, Department of Health Services, um, had been our client for quite a while, and we had done a modernization project with them for several years, and then they, um, a new project kind of opportunity came up where we could extend that platform and add on um, what they were calling at the time virtual ADRC. ADRC's, if you're not familiar, are aging and disability
resource centers, and they are dedicated to supporting residents who are aging and/or living with disabilities or those who care for them. So, it's an incredible group um within DHS, uh Wisconsin's Department of Health Services um called Bader. And so, the Bureau of Aging and Disability um and so their job is to to support and to staff all of these ADRC's. There's um you know 59 of them
that serve both the tribes as well as the 74 counties um all over the state of Wisconsin. So their resource specialists connect people to resources. It's a very hightouch government service if you've never needed it before. they really do take a lot of care and a lot of process around, you know, connecting families or individuals who are in need with exactly the right services that um depending
on what their level of need is, whether they need it to be inh home care, assisted technology, social support, everything. So, their goal is very much to provide equal access and quality of resources to everyone. And if you did a little quick bath in your head, there are 59 resource centers and 74 counties. So they already had, you know, big, you know, service areas that they had
to provide for. And prior to COVID, they um they did everything kind of in person in an ADRC center. Um and each one of them was responsible for maintaining their own list of resources. So they were incredibly distributed and each of the folks in those ADRC's did a lot of um did a lot of extra work. But when COVID shut down the ability for them to work
in person became very clear to them that they really needed to one do some data integration, data um consolidation and that they needed to have some way for people to be able to interact with them um asynchronously and online. So, you know, Bader's goal is really to provide that kind of equal access. Um and um and so part of that they were able to secure ARPA funds
to be able to modernize that entire infrastructure. Some of them were on Excel spreadsheets, some of them were on FileMaker Pro, some of them were on I mean you name it. It just is you know anything you can imagine. Um word documents. They weren't standardized in terms of their data formatting or anything like that. So this was a very much like a once- in a generation opportunity
for um for Bader to use that ARPA funding to embark on a project that would allow them to centralize and digitize um these 9,000 resources into one platform um so that Bader could ensure that there was kind of an an accurate information but a consistent experience for anybody regardless of where you were in the state of Wisconsin. So uh we've partnered with them uh starting about three
years ago, two and a half, three years ago on this particular building out this particular experience um and and integrating it into that um DHS public Drupal site which we had already um you know spent years modernizing for them. So the way it kind of works is that Drupal fetches the data via an API from the CRM that they um that they purchased as part of this
project called it's a system called Pure Place if anyone's familiar with it. Um, and one of the things that we do is when we go out via the API to get the information from Pure Place, we actually do some pretty amazing quality checks on the back end to make sure that the not not just the there's no duplication, but that it's formatted in a nice way and
it comes across. So, um, we can then leverage Drupal to do some pretty robust um, searching and filtering and saving and printing on the front end. But we do all of this in a way that is entirely HIPPA compliant. So there is no no login, right? So we created a print basket so you can organize and just save in your in your own session, but that we're
not tracking you and you don't have to release any information back over to us. So um this experience went live as I think I might have said at on the DHS website about a year ago and so that in itself was very much a monumental achievement. So virtual ADI ADRC this project um is classified as a large federally funded project which inherently made it high risk. So
it's estimated if you've ever done any kind of federal work that 30% only 30% of comparable projects are actually completed on budget um at this size. And when they are they are on average completed 15 months late and typically with fewer features than was originally scoped. So th that's when when you say a project is high risk, that's what it means. It means like that you're not
going to hit the budget, you're not going to hit the timeline, and you're not going to hit what you originally set out to do. Um I'm happy to announce we hit all of our things. We were on budget, we were on time, and we were fully featured. So I I certainly agree with those who are advocating very strongly that government projects need to become more agile, and
that's certainly a part of how Palanteer approaches it. Um and I I'd love to see more agile in procurement. And I'd love to see more agile in practice. But even with agile, large scale projects are still really hard. They're still incredibly difficult. And your odds of success are improved 6x if you use agile. Um, but it still requires kind of massive cultural shifts. So I want to
talk about redefining success, right? On federal projects again, it's, you know, if it's on time, on budget, and in scope, like that's a success. But if um if you've worked in government projects for uh yourself or know others who have um often that comes at a at a at a cost a human cost to the people who are involved in those projects, right? You get driven pretty
hard to do that. And I certainly agree with taking an outcome focused approach. Um but projects are not built by automatons. They're built by people. Maybe not yet. There's a probably a different session opposite me right now that'll teach you how to do it with a ton of tons. But this project was built with people with humans. And I I think that it is um it's a
shame that rarely is there consideration given to the people who are building it and the team conditions during the project. When you're talking about something that's lasting years, you it's it's not crunch time. You can't be in crunch mode for two years or two and a half years. So I would argue that perhaps the omission of process objectives and metrics specifically about team conditions um is something
that either certainly contributes to and perhaps even drives some of that high project failure rates on really large multi-year projects. So our hypothesis, some of the thing that we've been working on for many years at Halunteer has been that project success has happens when you have the kind of this equal mix of focus on your people experience of the of the folks who are actually working on
the project and the business objectives. It needs to have both. And I think too often we view those um as kind of uh competing, right? as as that you can have one or the other and that you're going to experience trade-offs like it's some sort of zero- sum game. And I don't agree that it is. So, it was over 10 years ago, I think almost 12 years
ago now, I wrote a blog post that was called work can't suck. And ever since then, you know, Palunteer's really been on this mission to live up to that, right? It's it's not enough for your team to be wildly loyal and happy, right? But maybe doesn't have enough work. And it's also not enough on the other side to have happy clients and you know these fully successful
projects, but to do so at the at the expense of your team, right? To just absolutely ring people dry. And I've known too many projects that that do that that they're really important big good projects, but the the people who worked on them are um are harmed by the way that the project is is run. So on the our ADRC project, these are actual quotes from our
team and our clients. This is what it felt like to be on this project. I didn't dread it. Okay, that's kind of a low bar, but I'll take it. I felt supported. We're getting better. We were excited to share what we did. Everyone was curious and invested. So this quote's from someone on our team. This is a quote from the client. So for me, it was enjoyable.
Um it was very person- centered. It was nice to be listened to and for them to move with us. Here's another quote from one of the our client team members. So lucky to be connected with a team that truly seems to value each other and encourages free flowing thoughts and ideas. So part of the way that Palanteer approaches these projects and is that we embed um our
clients inside our process like inside our project team. So, we're meeting with them constantly, not as a share out, but in an actual day-to-day collaborative sort of sense, like they're there when we're doing they're invited to backlog. Um, group backlog uh >> grooming. >> No, not that word. >> Refinement. >> Thank you. Refinement. I'm not gonna say that. Um, you can tell how long I've been working
in scrum. Uh, so anyway, uh, part of that has to do with psychological safety. So if you develop a mindset of us versus them, no matter how good the psychological safety is within your own team, you're going to start to create these artificial barriers between you and the client. So part of the hypothesis we've been working under is that we need to work very directly, very closely
with our our client teams as we're building the psychological safety of the team coming together. So um some of you may already be familiar with project Aristotle um which was that research project undertaken by Google that where they wanted to really understand um you know what increases performance and you know as they went into that project they had some assumptions about what that would be they thought
it would be some sort of like you know the the mythical high performers your 10x people some sort of experienced manager of people type things and unlimited resources and the reality is they were wrong. the most important thing to project success was this idea of psychological safety which um had really been coined in the 90s by um Harvard professor Amy Edmonson. So I like to think of
psychological safety as as the ability for for me to show up in a space and feel like I can share what what I know and that others that what I do is I create conditions that where others want to share the same things. That's kind of it, right? team members need to feel comfortable sharing their ideas, asking their questions, making mistakes without fear of this kind of
judgment or um or punishment. So, it's it's not about happiness really. It's just about feeling like you are able to participate participate fully and that those around you feel the same thing, right? No one's really holding back either what they can do or what they're concerned about. And that's the kind of at its core one of the things that we really try to um we try to
cultivate in any project and certainly on our longest lived projects because without it we're never going to be we're never going to get to the finish line. So this is an actual picture from one of our one of our Google Meet calls. But so our integrated ADRC team sought to really create a team culture and and to work in a certain way. So um a lot of
the folks who were on this project actually had worked on the the earlier DHS modernization project as well which was I think our largest experiment today at that time in creating the sort of embedded team model right where you the client is has full transparency into everything and and and everyone and everyone's interacting right so at Palunteer there's no role that doesn't talk to clients it's not
no such thing as like client facing and not client facing we're all just doing the work together. Um, so I think folks came to to the ADRC project with with a lot of ideas about, you know, what they would like to iterate on this one. And, you know, one of the quotes from the team member in the retro was like, this wasn't actually about chasing shiny features.
It was really about building core alignment and then finding a vibe. That's what they wanted to do. That's how they wanted to set out about doing it. So in order to do that they the team was really relentless in the way that they wanted to think about um their kind of team values right so clarity was one of the most important things on the project and especially
when you have an embedded team that includes your clients you can't use the shorthand of those of us who do this work every day so if we talked about something like a data model or API integrations or regressions those words mean very specific things to me to probably many of you in this room, but our client had never heard them, right? And it and when you invite
someone into your home, you need them to feel comfortable. You need to feel you wanted them to feel like they could belong. So, we wanted to make sure that anytime we used any any jargon or anything like that that we helped them understand what it meant, like not just definitionally, but like what it meant to the project, why it was important, what it so that they could
engage with us with their expertise, right? Good ideas come from anywhere. And I firmly believe that. It's one of the reasons we have these kind of integrated crossf functional teams. So part of what clarity meant to us is is the ability to kind of ask for what you needed. Um you know, I need to be able to communicate this out to other people. You know, can you
say this in a different way? I'm not sure I'm there with you. Could you maybe draw it for me? Could I have a visual around that or maybe a metaphor? Um I I need we needed to make sure that everyone not just in the project team but anybody else that they had to talk to outside the project team could really understand exactly what we were talking about.
Um and that that we understood what each other needed in terms of um you know meeting that that you know the sprint objectives for that week or you know what I needed to be able to be successful and my commitments. The next one was connection. So I think research is super clear on this one that people do better work when they are allowed to be people, right?
So we kind of make this into all of our interactions. Um so I think that um we're an entirely remote first um environment. So you may be coming directly from one Google Meet to another Google Meet and you need a minute like to kind of orient to okay this is the meeting I'm in. We use check-ins to really help us do that. Um, so the check-in question
helps us not just to be engaged or show up present at the meeting, but yeah, also kind of learn a little bit about each other. So, you know, um, check-in questions could be anything from what are you most looking forward to this week? What was the name of your first pet? Um, you know, or as in the case here, draw your best pumpkin, right? So, it just
could be anything. Their time box, they're just little at the beginning and they should never take more than than five minutes. But another way that this project team kind of really humanized each other was in the sprint reviews. Um so there would be uh there are a lot of holidays in the you know United States. So um the product lead would would find out whatever obscure holiday
that day was and then do a tiny little 30-cond intro on exactly what holiday we were celebrating today. So you know did you know that January 9th was Marzipan day? Well, now you do, right? Great. We're going to check in on Miles Japan Day. So, it's just it's that random weird quirky this is the personality of of the team that kind of cuts the tension, breaks the
ice, right? Sprint reviews could be tense, especially if you you're reporting, oh, hey, we didn't quite get as far as we wanted to or we ran into this roadblock, right? You start off on something that just diffuses that tension right away, that anxiety. It was super helpful. So, you know, we were only able to bring this to the party though because of the client. And I'm I'm
I want to I want to stress this, right? We were able to, you know, show up in a kind of fun, quirky sort of way um because they were willing to meet us there. Not all of our clients are like that, right? We had a federal client who absolutely forbade us from ever doing check-ins, even on their TED box, because they felt that it was a waste
of government money. And that's fine. So this this isn't the only way to do it. Um we had to to kind of take a different route, right? You don't give up on establishing that connection. You just find other ways to do it. So it is what it is. It's going to be based on the personality of the client that you're working with and you got to meet
them where they're at. But the point is the same, right? Connection is what leads to better teamwork. So, alpaca day. Um, was alpaca September 26, 2024 was alpaca day and apparently it's spring radio. Um, the last one is kind of co-creation. So, as I've talked about a little bit, um, we really focused on this idea of not stakeholder management. There was no us and them. There was
just a project team, right? So, this kind of embedded collaboration means that we have to use different tools. Um, we're not Palunteer is deeply collaborative anyway. um but we are just one team so we're all developing it together. We all need to be able to engage with these topics at the same time. So a couple of things that we did um to kind of help us with
that is we used um a lot of behavior driven development um and in particular we liked using example mapping. So example mapping really level sets and allows anybody at any level of expertise to be able to talk about what this thing needs to do and that was the exact right place to form those conversations. We're not here to second guess the expertise of the architect who's going
to who's going to build it. What we do need is we need for everyone to be able to understand what the objectives are and to be able to do it in a pretty fast way. So example mapping uh would bring the clients and the non-devs um kind of onto a level playing field with those of us who had kind of technical expertise and it asks us to
define what the tool needs to do from the user's perspective. So not from the perspective of how it's going to be coded and um you know there's an entire separate kind of webinar I guess about behavior driven development but but what it did it allowed us to actually do that kind of co-creation in real time. So it was never about throwing things over the wall. We were
literally making our artifacts together in real time. So what that meant is that these non-technical stakeholders at any point if their execs or the project sponsors asked them what was going on, they knew. They always And then reflection was a big part of it. Um when you're dealing with a project that goes on for two two and a half years, you need to get better over the
course of the project. If you don't reflect and you don't improve, you are getting worse. So that's not acceptable. You can't keep getting worse over time. I'd rather see us uh get better, figure out um what's going on. And also the process of making our reflection increments pretty small means that you'd never have a chance for that that kind of bubbling pot or that bubbling issue or
festering little, you know, inflammation, right, to become an outright full-on, you know, boilover situation, right? You want to be able to deal with it early and often. And the more you do that, the more likely it is that someone's going to speak up and speak up before it really starts to be contagious to others or starts to infect your your project team. So, uh, we made a
lot of adjustments. We did a lot of things wrong. Uh, we made a lot of mistakes and we would learn from those and we would try something new and sometimes it would work and sometimes it would and then you try something else if it didn't work. And that just became a normal part of the project team and we did it in front of the client with the
client embedded there the whole time. Right. And so that builds a lot of um a lot of psychological safety and a lot of connection between all of them. So obviously this is an example of a hot air balloon retro very standard one. Um but you know we try to make sure that the tools and the techniques we use ensure that everyone has an equal voice and um
that you're not penalized for being direct or honest and that there is no us in them. There is no veil. go ahead and just say it because then it allows people to take bolder bolder risks and encourages them to do that. And so um this is how we created this kind of culture of of learning and feedback and honoring what was going on Um the these are
some examples um from the final retro after two and a half years on the project. Um and we didn't ask a question like how was your psychological safety? Um the question was how did it go? What did you remember? What stood out for you? And these are some of the things that the actual team members said, you know, that um you know, agile can work in government.
Woohoo. Um having difficult conversations is getting easier for me. Um our care for each other and our work, our support for our boundaries went well. Having a large team come together and work so well with each other. So I should mention, you know, that the ideal agile team is considered to be about five to seven people. The ADRC team was consistently 20 to 25 people. So it
was a very very large team. Um and that's just on the Palunteer side. um on the um when you integrate the client, it could be another three or four people depending on what it was in the course So that's kind of the short version of how we intentionally established our culture um that that fostered psychological safety. But that's one side of the coin. The flip side of
the coin has to be what are we doing and why and how can we make this how can we define it in a way that that isn't technically prescriptive but so that it allows us to kind of flex and flow with what we find out over time but also doesn't allow the goalpost to be moved right um so this is the KPI section of the talk um
or whatever you want to call them how whatever framework you like to use but um we all know that you're if you're not meeting your project and business objectives. If you can't prove the value of your work or then you know all of your efforts to build a really strong team aren't really going to going to help you as much. They're not going to matter as much.
So the practice of of kind of defining and meeting business outcomes I think is very well established um in business culture. There's a lot of different frameworks you could choose, best practices, models, things like that. Um, and and I often wonder like since there's so many ways that people love to tell you how to do this, um, how come they still all fail, right? I think it
does kind of come down to to what we've been talking about. It's some sort of secret sauce of like is the team actually engaged in this and and do you does that framework actually allow you to define the outcomes in a way that everybody understands and can believe in. Right? So I've been working well palent this is Palanter's 30th anniversary. So I've been doing this work for
30 years and I certainly have been part my um been part of a lot of share of failed projects right my my fair share. So I think that there's um I think that the psychological safety piece is is crucial when combined with some of these frameworks. So what we do is um you know we like to start with kind of a northstar document and we make sure
that the metrics that we establish um are objectively measurable that everybody on the team understands them internal and external that they repeat them. It's not like it's not like a checkbox, right? Setting out to do your northstar document isn't like a oo and now we're done, right? If you aren't coming back to what your core metrics, your core success metrics are in every sprint in every day,
then they're not they're not going to work for you, right? Because they don't establish that shared language and that shared expectation of what you're all kind of driving at. Um, and then we need it to be unassalable, like really really clear. Did we do it? Did we not do it? Right? It can't be something that is super subjective because otherwise someone's going to question the success, So,
I'm I'm not going to go deep into this isn't this isn't a conversation about how we do our Northstar kind of planning document, but what we do in that in that um in that document at the very beginning of the project is that we're looking to kind of articulate the business goals, the user needs and and this kind of baseline orienting story about what we've been asked
collectively to do on the project. So, we try to keep it simple, we try to keep it readable. unarguable. Uh we try to keep it memorable. Um so I think when the project started three years ago um you know we had an upfront discovery and planning process we created these are the success metrics that we wanted to validate against for the entire project. So one of the
most effective practices that we used was that we were incrementally um checking back in on them. So there were and again all of our checks were very transparent. So the bi-weekly sprint reviews were, you know, it was very clear. Did we hit it? Did we not hit it? Do we think we need to do something else? So, um, I think that helped us to do it. Um,
and I think also the act of being clear about when we didn't hit something and how the team pivoted quickly, that it wasn't an embarrassment. It wasn't this a secret. We weren't trying to like squirrel it away or hide it under the rug or anything like that. made it safer for the client to tell us things that they thought might not be working well earlier in the
process. So, um I think that the most important thing here is to make sure that you don't skip the validation. We all know how important and how valuable user research is. Um but one of the things I really appreciated is that we did a lot of scrappy UX validation. We would take just tiny groups and just throughout the project over time making sure that we were demonstrating
that we were going down the right direction whether it was just with um an advisory council that we had access to or with the internal ADRC staff or with the Bader staff. We're constantly making sure that we're aligned with um that with the data that we're that's emergent is um aligning with where we wanted to go. So part of that is a a very strong initial baseline
set of metrics so that you can demonstrate improvement um in the work that we do. Anytime you're doing something that that has an aesthetic quality to it, um it can feel subjective. And so part of what our core metrics help us to do is to change those conversations and move them in the direction of something that is much more objective. Can people more easily find what they
were looking for? are they, you know, getting to the resources faster, things like that. So, it's about setting up a really lightweight framework so that you can move um move pretty quickly and adjust and adapt if you um if it isn't isn't quite working. So, that's a lot of kind of handwavy stuff. Let's dive in a little bit. I'll show you kind of um the examples of
the frameworks and how we use them. And I think one of the things that underlies the success of these frameworks as we use them was the ability to decompose the big project into components that weren't technically specific. So a lot of times when your project gets decomposed, it is done in a way that you know your user stories actually tell you exactly how you're meant to implement
it or your your you know um acceptance criteria are um again presume the way that it's going to be done. And so part of the way that we we moved away from that was essentially having a hierarchy and the accountability framework. So the core success metrics those are set at the project level up front in the beginning of the project you know three years ago. Then we
had an agile implementation roadmap and you move down from that into our sprint goals. So every sprint goal needed to add up to where we were on the road map and then we would have key sprint outcomes indicators and the project and then our our progress toward that indicator. Not all indicators could um ended up being finished every um every sprint. So when they weren't, you would
reflect, you'd report, you'd repeat again. So here's what that looks like. So I've showed you these before, right? These are the core success metrics at the project level. You know, can visitors get the help they need and feel empowered? Um users can can easily find it um because it was embedded on a larger platform and it wasn't didn't have its own URL. This is part of the
the one website strategy for Services. They don't want a ton of little fragmented composable sites. They want a they want a monolith and that's how they um feel that they can best use their resources. So that means that if somebody's coming for an ADRC, they're not going to a specific URL. Um can they find it within the the broader DHS website? So we have all of these
different metrics and then we had our quarterly goals at the road map level. And again, we use this kind of now, next, and later framework. So, we have a tremendous amount of resolution and clarity and certainty around what's happening now. That's good. Hopefully, everyone has certainty around what's happening now. Um, we had um slightly less um certainty around what would happen next. We knew generally in broad
strokes, we had a a high degree of confidence about what would be happening in the next quarter in any given quarter. And then what would happen later? Well, that could change, right? And so again, this is one of the ways that we socialized our client to be able to to handle the uncertainty cone um with while still feeling that they had control of the project. Um I
think that's one of the the most difficult pieces um when non-technical users are tasked with managing agile projects, they feel like it gets away from them and that's that's I think one of the drivers for for projects being 15 months delayed or thinking things like that. So then you break it down into a sprint goal. So this would be an example. So we go into the sprint
uh review meeting and um our team would have taken a crack at what we think the next sprint goal would be and we would have some key sprint outcomes that we would have identified and suggested upfront and then when we would get into the sprint review meeting uh where the the client team members were part of it uh we'd workshop it together. So we would go into
breakouts or we would do different things to make sure and jij it and and see what kind of um they needed to have happen next. Then we would break it down um and then we report back on on it. So if we had our key outcomes and our indicators, we would actually say and so this is is from an actual sprint um sprint review um we would
say okay well is it still in progress? Is it blocked? What needs to happen? Do we think it's not you know not moving forward sufficiently? Whatever. That's how the the client was able to understand at any given point exactly what the state of the project was and and to understand how things are adding up. So, you know, whether you think of those indicators as um as your
acceptance criteria, I think they're sort of proto acceptance criteria. They do get played out a little bit deeper when you actually get to the um user story and the ticket level. But then you'd come in as well with okay, what is the next sprint? And here's what we're thinking we're going to do for our key outcomes. and um what our next sprint goal needs to be. So
then you just keep repeating um so this is this is our sprint 41 goal, right? So we're sort of halfway through at that point and then you would break it down. So the team would come out once we had an alignment around it, we would say, okay, how do we actually break down those key outcomes into potential activities? So again, we're not taking away the autonomy of
the team members. part of what was important about the way that um you know Palanteer's been um investing very deeply in in becoming more of a self-organizing team. So we're not hierarchical at any uh kind of level, certainly not at the team level. And so it's important there's not one person who's sitting there saying, "Okay, this is what you're going to be working on this week." They
would come together based around those sprint outcomes and and each person would say, "This is what I can commit to. This is what I would like to take on this week. Um and this is how I think we're going to do this." So um I think that's a a really important part of of it. But when you have that kind of autonomy and that agency to decide
define what you're working on, it becomes even more important that you have this kind of core alignment not just amongst yourselves but al along with the stakeholder to make sure that that that deep position is going to add up to exactly where they need to be when they need to be there. So in terms of the metrics framework that also needs decomposed because again you know if
you set up your core um success metrics at the beginning of the project and you can't evaluate them again till the end of the project. That's how you end up with you know being massively off course right you want to be able to check in on it at a smaller level throughout the project. So we'd have these measurable indicators, we'd have the benchmark, and then we would
test against them as soon and as often as we could, even in little like non um statistically significant sorts of sets just to get that heristic. So you know, even little five person sets of of user research um any given sprint was super valuable to make sure are we on are we on target and we can include that data. So um you know when we have the
kind of high level metrics both of we had both quantitative and qualitative for how we would break down um each of those individual metrics. So you know each each one of those metrics had many different facets. So it wouldn't just be like one facet. We weren't using just one set of numbers. It was actually quite um quite a huge spreadsheet. But you can see we would establish
a baseline in October before we did the project based on their current um experiences that they were delivering. And then we would uh test it in February. we tested in March. And so at the end of the project, they had monthly metrics that they could monitor internally. They didn't even need us anymore to be able to continue to monitor the the advancement of it both on the
qu quantitative level and at the qualitative level with surveys and feedback. So they had a survey that they could set up that we had already validated against um on the project side that they could run every quarter just to make sure things were staying on target because as I mentioned at the beginning um this is a once in a generation kind of funding for the ADRC's like
they aren't going to be able to maintain the connection with us to be able to deeply run user research all the time and so what we needed to do was to hand them something so that they could continue to make sure that the service was delivering on the promise and the vision that they had for it with their internal resources. So this is another part of the
again this is just one of the metrics that goes into metric one. How do we know we're successful? You can see kind of the improvement over time um on engagement. Um again this is another report out toward the end of the project last year. We wanted to make sure um that we were being inclusive and accessible and and these are this entire um framework is something that
they've been able to continue even without you know Palunteers's kind of ongoing guidance in that way. So um this project again as I said we hit those kind of standard um success metrics we were on time we were on budget we did deliver it fully featured but none of that really matters if we didn't actually improve it. So what we saw that is that by one month
post launch we had made you know little tweaks to see a 664% increase in resource engagement. We had 2000 whatever percent increase in the printed resource lists. We had longer average engagement time and we had lots of different category browses. Um which is really helpful for people to be able to access these resources that that they didn't used to have access to at all unless they walked
into an ADRC office. So, one of the user tests was like, "Oh my god, can you please just let me have access to it?" That's that's one of the dangers of of letting users um have access to something that's in development before that they need um like people who actually need the service. Um letting them have access to it before it's available publicly as they actually really
want it. So, it was So, I think this is something that is repeatable not just at Palanteer, you can do it, too. So, this is kind of what we talked about, right? there's the the metric side and the psych safety side and it's going to be different for each of your teams how you implement and go about these things but I do think that being committed to
you know just a handful of you know key success metrics that really map to both business and user outcomes as opposed to putting those two apart but knits them together I think that's really important benchmark them upfront so that you can show out of the gate how much improvement you've had and then coach the team consistently throughout the entire your project about the metrics. Everybody should know
them all the time. Repeat them, stick with them, and do your incremental success um check-ins because then you can celebrate little wins as you're making these changes. Like, it can be a lot. No matter how great your team is, no matter how awesome and and worthy and meaningful the project you're on is, being on one project intensely for a couple years, um you know, it just can
it can get heavy at times, right? So those small wins, knowing that you're doing you're going in the right direction, you're doing the right thing, um is really helpful. And then on the the team side, whatever human connection looks like in your team, right? My team happens to be really quirky and weird and we like our weirdness. So that's how we're going to show up. You show
up in the way that's authentic to you and to your team. But however you do it, make sure that you're all talking in the same language and that everybody is um on a level playing field when you're making decisions. So, you know, it it really doesn't help you to provide a lot of of information in context if they don't know what to do with it or they
don't understand it or don't know how to apply it. So, um we in order to kind of address that last piece, we did a lot of co-creation um real time co-creation as much as possible. Um and then a lot of reflection, a lot of retros and again our retros included the the client teams as well. They would have a retro portion as well. Um, and you know
you whatever metrics you want to use or however you want to measure the psychological safety, just pay attention to it over the course of the project. If people start to get quiet, if people aren't talking in the meeting, if you notice someone who is kind of not turning their camera on as much as they used to, you know, just monitor those little things. It doesn't have to
be a um, you know, it doesn't have to be like a a those surveys of like how's your satisfaction over time? Like we don't we don't do any of that stuff, right? But we do monitor the little stuff, the little signs and I think it's almost more effective in that So I I think that it's important for us um you know as we think about these projects
particularly the bigger larger projects that take years of of our lives and many of our lives that it's not just about it's not just about the end it's about the journey right that success I think the successes you have on the journey are the ones that are going to determine kind of the boundaries of the success afterward on on the outcome side so I would argue that
that this is a case study and a process study hits both those marks, right? You know, we were able to stay connected as amazing humans and we were able to do work that was really meaningful and made a big difference for the people of the state of Wisconsin. So, with that, if there um here are some links to different things I talked about, if you want to
learn more about either behavior driven development, which I didn't go into very much, project on Aristotle, which I kind of referred to, um want to see the ADRC's go for that. And then um psych safety u the slides will be up but then there's also a QR code for any feedback and uh with that if anybody has any questions I'm happy to answer them. Yeah. Is that
sort of a group own responsibility or is there one person or group of people that are like more to that keeping it on track? >> So the question is um when you're thinking about monitoring psychological safety is that is there like one person who's tasked with that or or a small group who's tasked with that? um psychological safety is everybody's responsibility. Um I will say that we
have some folks who are like the whole team is trained like in that but we have a couple different roles where the activities of that role lend them to to discover certain things about it. So whether it's the agile coach on the project um you know who's running the retros and kind of helping prep for the retros or doing individual one-on- ones with people like it can
come up there. Um I also think that the team lead role which is our version of the project manager also can notice things because they're facilitating um and designing many of the sessions. I think the pro project lead or product need um is another role where you might you might see someone's um throughput dropping and you might want to know about that and and so I think
that within each of the disciplines and each of the skill sets that an agile team kind of brings to bear. There are facets that you can monitor for psychological safety like hey you know is everything going okay or how about we co-work right? So, a lot of times if you just have like that spidey sense about your teammate that like something's off, right? You just ask for,
"Hey, do you want to do you want to have a co-work today or you want to show up?" I can't even tell you that the teams continue to evolve how they think about things and they have these things called domes. There's like a candy dome and whatever. They just give it crazy names so it's easy to show up and that's where you might ask a question or
that's where you might, you know, um, capture all the things that we don't know or the things that we're uncertain about or whatever it is. But I think it's about keeping it pretty right. Even though the work is meaningful and the projects are huge, like you're breaking down and decomposing, you know, a six or seven figure project over the course of however many people over a number
of years, right? That's a big daunting thing. It can be, right? So you kind of have to have each other's backs on it. >> Others questions. >> Yeah. Yeah. So the question is um was there ever a situation where uh the KPI itself uh you know the thing that that people were driving toward and that the intention of the product um kind of fell by the wayside.
Um you know it's it's the it's the modern work equivalent of teaching to the test, right? you know, you know, you know, you're going to be tested on this one thing and do you try and do that? Um, on on our team in this project, that isn't something that really came up. And the reason that I think it didn't is because the client was there all the
time, right? Because we had the embedded client experience and the trust between the subject matter experts and the the technical and design teams um was so close. I think that everybody was kind of operating in that space. And I tend to come from um a position of assuming positive intent, meaning that I don't think that that happens because someone is trying to make a shortcut. I think
that it's more of a symbol of a lack of understanding. And so if you don't have a sense of what the big the whole big picture is, I think it becomes much easier to flatten it. Right? So these metrics are intended to be three-dimensional and round and um but no metric ever can be. That's why we felt so strongly that the the team needed to have that
full understanding of the project of what we were trying to go for and the experience of um you know whether it was the end users or our clients or those um who serve people in the ADRC's. So um you know we would we would it wasn't just our UXers who would interact with the end users like we might have there might be a front-end developer who would
sit in on um and be the notetaker for you know user interviews or help help code um any of the user research we would have and I think that's the advantage of of approaching it as a cross functional team that you aren't just one thing you are expected every person on the team is expected to do whatever is most important for the success of the project in
this particular increment. >> Any other questions? Yes. >> Inevitably there's confidence and then a non- hierarchical team of sorts. What are your insights that gets resolved? So the question is that um when there are people there is conflict and when you have a non- hierarchical team and you can't just be like okay I made the decision we're moving on like how do you actually resolve that? Um
I think the first part of it is actually to surface it before it becomes the explosion right to to take the to take the lid off that bubbling pot so that it's not going to boil over. So the retros are a key component in that. And I think that it's the difference between um a performative retro where you're going through the motions like what are we going
to keep doing? What are we going to stop doing? What are we going to start doing? Right? You can you can do a retro without actually getting any meaning out of a retro. Right? You need somebody who's going to facilitate the retro and ask the hard question and like push on that little spot that hurts a little bit because you're not oblivious. Right? the the the value
of the retro isn't that it happened. The value of the retro is in is in what it surfaced and how you collectively recognized and looked at something and said this is what how we want to do something a little bit different. And so I think a lot of the most important conflict resolution in a project team especially a longived project team is in getting things stated um
and because otherwise they're just going to kind of bubble over. I think the team would kind of swallow things that they didn't necessarily want to bring up that might be uncomfortable. You know, oh hey, I your tickets keep rolling, right? I mean, it's really hard in a non-herical team of the of people who become friends to sometimes hold each other accountable for the impact, especially in a
really long lived project, because it doesn't feel like you're you're all going to miss the deadline because of this thing. So, do I really say anything? Um again that's where that role of the agile coach and saying oh hey I'm noticing this and I think that it's not uncommon for people to notice things like that right but I think the question then becomes what do you do
about it and how do you talk about it so there's this idea of the problem can be in between the two people like person A notices something and talks to person B about it and is it is that tension in between them or is it a challenge forward right oh hey I'm noticing this thing how can I help you to do that, right? So, it's really about
flipping a lot of those conflicts because they're not like a the conflicts aren't really in between people necessarily, but but I think we feel sometimes that they might be if you feel like embarrassed or shame or whatever, like you didn't ask for help or you don't know that thing, right? Then you kind of hold on to and it feels like the person who called you out on
it like you have a conflict with them and the reality is you don't. You don't have a conflict with them. very rarely do I actually see conflicts in the team between people. Um I can see things framed that way. I can see things like talked about that way. Um but I think there's ways to flip it and that's where like your agile coach or um can be
really helpful to help reframe. Okay, what is the problem we're trying to solve and how do we need to solve that? Who and what what do you need to be >> Yeah, other questions. So um how what's the suggestion you have if uh I know you have like a small agent agent company but it's enterprise department team and with all those KPIs coming from up level management
team and they keep changing and how do you like identify or changing dep on that to satisfy your team Right. Very valid question. Very valid question. So it's um if you So in the context of Palunteer, we do project- based work. And so that's a little bit different than an in-house team in a larger enterprise where the KPIs are set at a level above and outside the
um the team that's responsible for implementing them. And how do you how do you achieve satisfaction when you are literally standing on nothing concrete? Is that fair? Okay. Yeah, that's the question. Yeah, that's hard. That's really hard. And I think the way that I would approach that is again if you're in an enterprise context, you probably have uh much more defined roles and responsibilities. And so the
question is um you know, who is the product lead? Who's accountable for that? And how are they communicating? These are the impacts, right? So I think to a certain extent if you have a low low organizational maturity which is how I would describe if your execs keep changing your KPIs you have low organizational maturity and that's bad for your enterprise anyway. Um but if that keeps happening
then what you need to do is you need to change your increment to be thinner and smaller so that you can actually get to that vision of success and then start to implement small practices to show to to try to build out that um that roadmap of the now next later and be able to say you know if we do this now this is the consequence and
this particular objective is going to take x amount longer right so it's I I like to have the the flexibility in those contexts of being able to say it's options and information. Awesome. Oh, this need this metric needs to change. Great. The impact on it is this on time and on budget or on resources that we need and that's okay. And is that the most important thing
or if we change that now, these other things are becoming deprioritized and they're going to fall back to next quarter or they're going to drop off entirely. So, it's about someone who's willing to hold a bounding box around it. Um, I don't think that those I think those make it a little bit more manageable for people to feel like um they're not being taken advantage of, but
I don't think that that gets you all the way to feeling satisfied and invested in the end product when it when it keeps changing. So, um there's a I don't know if you've come across his blog, but John Cutler talks a lot about these kinds of things and how you might approach them. And it depends a lot on the the culture of your organization and you know
are you a um you are there cap you know essentially do you have OKRs throughout the entire organization and can you tie them to them and can you speak the language of the exec that keeps changing those right now can you make it in their interest to align right so that's that idea conceptually of moving a challenge forward saying oh okay this thing keeps happening we both
I need your help to solve this thing because we're seeing that we're not our throughput isn't what we needed to be or we could, you know, whatever they're evaluated on, whatever that executive is evaluated on, find a way to tie the behaviors that you're looking for to their outcomes that they're evaluated on. Like that's the the short completely nonspecific answer, but that's how I would approach it.
Other questions? I think we're we're finished. Thank you so much for coming and I appreciate your time and attention.
