From Search Bars to Chatbots: Decoding User Expectations for AI Chat

This talk focuses on understanding user expectations and experiences with AI chat tools. The speakers, Emily and Alma from Sandstorm, present research findings based on moderated usability tests conducted on various websites featuring AI chat implementations. They explore how users instinctively seek support, their trust levels in different AI chat situations, and the critical role of search functionality over chat options. The research reveals that users typically prefer to initiate with searches rather than chats, view Google as a trusted resource due to its citation practices, and harbor skepticism towards AI responses, particularly for sensitive topics like medical inquiries. The speakers conclude with actionable recommendations for improving user interactions with chat tools, emphasizing the importance of transparency, appropriate placement, and enhancing search capabilities.

Welcome everyone from search bars to chat bots. Uh decoding user expectations for AI chat. If you're in the wrong spot, now's the time to go. If not, this is going to be super fun. Um we hope you enjoy and stay. Uh we are Sandstorm. There's many more of us here, too. We have our own little fan club over here in the front helping us out, cheering us

on. Um but this is Emily and I'm Alma. Um, Sandstorm, we design and build accessible digital experiences for organizations that are trying to make the world a little bit better. So, just good people and good work. Um, I'm Elma. I'm female presenting. I'm kind of short. I have dark brown hair and I wear glasses. My pronouns are she, her, and I'm wearing a very cute blue dress

with a bunch of icons on it. And Emily and I are matching twinsies today. We sometimes wear matching dresses. It's a thing. I'm Emily Codner. I am um a uh 40something white woman um with a general mom appearance. >> Thank you. >> All right. So, our clients keep asking us about AI chat. So, we thought we should do some research on that and talk about it with

a community of people that are probably implementing those tools. Uh so, that's what we're going to do today. We did a little bit of research and we're going to talk about that. Take it away. >> Yeah. So just going through what our methods were, what we studied and how um we wanted to understand where do users inst instinctively look for support, what kind of experiences foster trust,

where do users expect to find AI chat specifically, what features improve or in increase or decrease trust in that experience? And we did the study um as moderated usability tests. We did them via Zoom. Um so we uh had users do a series of um realistic tasks on a variety of websites and at use the think aloud protocol. So they talked through what they liked, what they

were thinking as they were making decisions. Um we did this on desktop and we also did a survey at the end. And here's a little overview of our sample. So we did nine. We generally for a taskbased usability study you want at least five to six. So, we did a little little extra credit because we wanted a good age representation. Our youngest participant was 11 and our

oldest was 73. All right. So, during this study, we wanted to look at a variety of AI and chat experiences. And so, we looked at several different types of those implementations. So we looked at search with AI elements, generative AI, transactional chats, hybrid where there's start with a little bot interaction and then hand over to a human. Um and then domain specific, so one website, the AI

just knows about that one site and gives you answers based on their content, not everything else out out in the web. Um and we chose tasks that were for a general audience. Um so that our finding findings could be broadly applied. Um, we know that all of you work in a variety of industries and on a variety of clients. So, we didn't want to narrow our focus

into just one thing that was interesting to us. Um, so hopefully some of you and all of you can take something back to the type of sites that you're building. Um, and we'll walk through screenshots of all the different tasks that we did in the study so you can get a sense of kind of how we structured things and what we were looking for. So, first task

was Google. Uh, we selected Google because it's Google. Um it has a very prominent AI search overview at the top of it. So people see that right away. Um and for our first task, we asked users to find symptoms and treatments for type 2 diabetes. So something very common they might be looking for. Maybe a friend family member or friend was recently diagnosed or is starting to

experience some of those symptoms. Um so we picked something very generic. And then we had them um look at this, give us their information, what they think about it, if they thought AI was involved. And then we asked them to click on this dive deeper button and explore the AI mode, which some of them noticed, some of them didn't. Um, and for this one, we wanted to

look at it a little bit deeper. Um, this tool has some generative AI elements to it. So very similar to Chat GPT, which leads us to our very next task where we did look at chat GPT. Um, we had them do the exact same task where we're asking them to find symptoms and treatments for type 2 diabetes but using Chad GPT instead. And what we were trying

to find here is we were looking for were there any differences between between the ways that they looked at and used a generative a AI tool like chat GBT versus using a search engine like Google. Um, and we'll share some interesting things we learned along the way there. Um, we also asked them how trustworthy those tools were. um and how different the experiences were. Next, we looked

at Chipotle. So, love a good burrito from Chipotle. Um we I do not love this chat personally, but we'll tell you more about what our users thought about this later on. We were asking them to use the chat to find upcoming promotions. Um we specifically wanted them to use the chat and not try and find the content anywhere else on the site. So, this was true for

all of our tasks. We wanted them to interact with those chat features or search features on the site. So this content may be available somewhere else but we are very intentional about looking at those specific experiences. Um and this one in particular the chat was located in kind of a traditional where we think of as designers a chat is supposed to go. So the bottom right hand

corner of the page it had kind of a little bit of a bubble feel to it. Um but we'll talk about some of the things that we learned there. Uh next we looked at the frigid air site. So, uh, I have an apartment. My dishwasher sadly died and the dishwasher did not have a serial number on it because the sticker had worn off. Uh, so I actually

experienced this out in the real world on my own and I'm like, great, I have one that will do a chat to a real human person. So, we had all of our participants do that exact same task where they were trying to get service on this dishwasher. Um, so this hybrid chat starts out as Pixie, their automated assistant, and then transitions to a human customer service representative

after you answer several questions. Next, we looked at Open Table. Um, so again, apparently we were feeling a little snacky as we were writing this. Um, and we wanted people to see if there were any vegan options without looking at the menu. So sometimes people are in a hurry. They don't have time to read through every single thing on the site. um they have this very interesting

chat feature called concierge. Um and we're doing this type of interaction because it's domain specific. It's looking just at the information on Open Table and not going anywhere else out on the web. Um and then our last task was another example of a domain specific experience. Here we asked users to see if they were required to file a tax return using the interactive tax assistant um on

the IRS site. So there we go. Okay. So, what did we learn? Um, we're going to go through five what we're calling universal behaviors, things that we observed as findings in each participant. So, number one, users look for search first. Um, even before they were looking for navigation, before they were looking for any chat, before reading the page, when the task was help oriented, they were looking

for search. Even the 11-year-old specifically looked for the magnifying glass. Where is that magnifying glass at? Um, so when search wasn't available or or failed, participants felt lost or frustrated. Oh, I'm back. All right. Um, number two, Google still wins the trust game. So, everybody knows the name Google. Um, but participants trusted Google's AI implementation more than some of the others that we experienced during this study.

um they appreciated that Google had citations and source links and a familiar interface. They've been working with this experience that you saw earlier for a very long time. So it garnered a great deal of trust. And number three, nobody takes AI at face value. So all participants said they would verify a AI responses for high stake decisions. um medical information in particular, we had those two tasks

that were focused on on medical issues. Those consistently would require verification with authoritative s sources. They would double check or verify elsewhere. Um those those words appeared in every session and applied across all age groups. All right. So for our next finding before we get into that um we want to show you just a little um wireframe that we created to help us as part of our

study. So while it's great to look at things that are out in the wild in the real world, we wanted to kind of strip all of that out and just focus on the search element. So not a brand that they know, not anything that they're familiar with, but just you know raw content on the page asking where they're looking for help. So we created this little prototype

here. We've got four different versions of a membership page. Um, we do a lot of work in the association space, so this one was important for us to learn about, but can be used across a broad range of sites. Um, and we asked users to tell us how they would find help with each of the four options and which ones they thought would be the most helpful.

So, option A is a search bar right at the top. Option B is that traditional chat bubble um, at the bottom right hand corner. Option C is chat with context links. And then the last option is some FAQs exposed along with the chat. And what we learned was that participants chose option D as the most comprehensive and trustworthy. And some of the reasons that they said this,

they felt like it showed more effort on the behalf of the company that they actually thought about what is a user going to come to this page looking for? What are the questions that they might have? It provided multiple ways for someone to answer the question on their own and they really appreciated those FAQs telling you this is exactly the type of things that we know how

to answer and we know you're asking for right away. Okay. And then perhaps our most important finding number five is that people have chat baggage. They bring chat trauma with them. They have all we've we've all had bad experiences. So when we come to the website and we see chat, we're like, "Oh, >> this is not going to end well. >> I hope I don't have to

use that." >> Yeah. Yeah. When when they mention the previous chat experiences, they categorically describe them negatively. Um, so this led to active avoidance behavior, strong aversion, um, and it really shaped their experiences with future chat interactions. So this is this is the reason we're going to have to keep testing this. like we tested uh this the the study that we did right now is just a

couple months old. We're going to keep testing. We're testing right now and some other experiences and we're going to see if this trend changes as the quality of chat improves. Only participants without prior chat experiences are the ones that had or with the less experience they had, the more open they were to um to interacting. >> All right, time for a picture. You need social posts or

things to share with your team. Here's our five universal findings. >> Take a beat here. I see the cameras out. Appreciate you all. >> Oh, I still see more cameras. All right. Sorry. We're good. We're good. We're good. All right. So, now on to some surprising things. Uh, so nobody was looking for chat. Sorry to say. Um, many companies are rushing to implement AI chats. Maybe they're

getting pressure from, you know, boards or directors or something saying, "Hey, we've got to catch up. We need to be a part of this AI thing." But, you know, they come with the baggage and they had pretty low expectations about what those chats could do. And negative impressions of them right out of the gate without even interacting with them. Um many of them tried to use search

first or navigation even when we told them the task was interact with the chat. Um so they are seeing chat as like a last resort. I if I have to absolutely use it then I will but I really don't want to. Um and so again like I said during the frigid air task the way that we read that to them was please use the chat to schedule

service. Um, and they were like, "Well, is there a different way I could do it?" They're like hunting around in the nav and clicking on other things and we're like, "I'm so sorry. Could you please go back to that other page and interact with the chat?" So, even when the task is specifically do this one thing, there was a lot of hesitation to do that. >> And

I think this one was really important because we were one of our big questions was where where do we put chat so people will find it? But the problem is they're not looking for it. So, you can't win at that. Um and so number two, age expectations were reversed. So usually you have more experience with something you trust it more and that was the opposite here. Um

so uh one of the more interesting things was actually around like medical findings. Um the seven the 73year-old had had uh recent exposure to like using Google for medical issues and like oh it's it's pretty it's pretty right. I'm I'm I'm good with this. Yeah, it's been it's been what I expected. had a lot of good information and he was ready to roll with that information. And

the 11-year-old and the 15-year-old with the same task were like, I would never stop here. Like I and they, you know, were immediately going to the sites to look for the authoritative source, the cited information. Or the 11-year-old's like, I would ask a doctor. >> What are you talking about? Yeah. >> All right. Um, so this is a really interesting finding. So the IRS site beat chat

GBT. Now we had them do different tasks on it, but at the end we asked them to rate the different experiences. Um, and the IRS site rated higher than ChatGBT in every session that we conducted. Now this may not be true. If you go out and do more research, maybe more people will like chat GBT. Depends. Um, but the IRS site ranked consistently higher and trusted um

as being effective. Um, and it sometimes even beat Google even though government sites sometimes can get a bad rep about having poor UX. Um, again, this comes back to them being the authoritative source. So, the government should have all of the answers about the things that the government wants you to do. So people felt like that was very trustworthy and that they would continue to work and

try to find things even if the tools did not work initially or gave them no response. So in our sessions we saw people interacting with that little search bar that you saw. They were typing in questions and they were getting a page that said no response found or zero results in the search. But they were like I would keep going because I know that the stuff is

here somewhere even though I haven't found it yet. So, it must be a me problem, which that's a whole different topic and we should try to solve that. But it's interesting finding that the name and the source of truth really does help your credibility. Um, this perhaps not surprising, but um, users type differently depending on the solution. I mean, depending on the uh, the interaction. So, for

Google, um, we had one user uh, who who called it um, caveman speak and just like as few words as possible. was going to put like the three four words that I think will will show up in the query. Um, but when they went over to chat GPT, they formed a nice sentence with punctuation and capitalization even. Um, so just seeing that different behavior and as we're

looking at site search experiences, thinking about are we looking for a two, three to word phrase, are we also going to handle well that sentence that they put in and how do we uh set them up to put the right thing in. >> Yep. And then our last surprising finding here is that uh our stat standard chat patterns don't really work. So we kind of teased this

a little earlier that you know that bottom right hand corner is where we see a lot of organizations putting chat. Um sometimes they have a little bubble or a little tail on it. Um but that didn't always work. Many of our participants could not locate the chat. Again, even when we were asking them to find chat, they weren't looking in that area for it. Um, especially Chipotle's

pepper chat. Um, because it was just called pepper, they had no idea what that meant. So, it's just a red floating button in the bottom corner that says pepper. So, that's not telling you, hey, this might be a powerful tool that will answer your questions. So the real question here isn't how do we design better chat. The question is when do people actually want to engage with

All right, picture time again. >> Be sure to get Ma in it. She's >> all right. I don't see any more cameras. Okay, we're good. So with all of the findings, it really came down to how do we overcome that chat package? How do we overcome that skepticism? So here's five tips, five recommendations for how to do that. So number one, lead with search and FAQs. Position

chat as the backup. So instead of um leading with it, it's more of a do you still need help? Did we not get there yet? Here, I'm here to help you. you've got to take them down where they want to go first and then introduce it. Um that pattern um as we looked at the four different examples, that pattern on the right was consistently most successful because

it it gave you the FAQs, it gave you all of the things that you wanted for your answer finding, the search interface, um and then it let you get get into the chat. Um, and we're actually running another study right now on an association website and found that AI enhanced search was even more successful than the chat framing. So framing it maybe even um directly within the

search experience is likely to be the most successful right now. >> Yeah. And again on this one because we were asking folks, you know, why they felt like the search and the FAQs were more successful. again they felt like the company that they were interacting with that organization was putting more effort into meeting them where they were. Uh it felt like okay I can trust them because

they thought about what questions I might need to answer and are providing those for me right away making it less guesswork right we want to make them do a lot less work and get to the answers and have a great experience every time. Uh okay. So next we do not want to hide the AI. Um we want to explain it. So users are not upset that AI

is happening on sites. They are not shy about using it. Um and they want to build that trust with organizations. And the way that we do that is by being transparent about what the tools do and how we're using those technologies. Um, and what was really interesting is during this study e every age group had a pretty sophisticated way of evaluating how tools were working even if

they're not in technology. So most of the folks that we talk to are not technology people. They don't do computers for their day-to-day life. Um, obviously we have children in our user group. Um, and we have those who are retired. So maybe they're using computers for their personal life, but they're not developers or, you know, people who work in the technology industry. Um, but they kind of

have a sense of how some of these things work in their own world. And especially when it comes to AI, they have their own personal opinions and strategies for how to figure out what things are doing what. Um, but they definitely appreciated it when those tools were giving them sources, citations, and confidence levels. So, for any of these tools that you are implementing, we definitely recommend that

you're clearly labeling when users are interacting with AI. Um, this could be really simple things like providing source links for the factual information, um, and including confidence indicators like here's where we got this or here's how this information came about. Um, you could even consider statements like this AI response is based on X, Y, and Z. Um, rather than hiding it or not saying it upfront. Um,

and a bonus tip here, um, the names mattered more than expected. Um, so like we talked about with Chipotle's pepper, which is apparently my arch nemesis now. Um, people were really confused. Again, I know that a lot of brands try to create really cute names for things like, "Oh, this has a personality and it's a person." It's not a person. People know that it's not a person.

Um, I know that we want to give things a little bit more friendly language and things like that, but calling it Pepper actually reduced the discoverability of that tool because again, if you don't know what Pepper means, then you're not going to be looking for it. And it actually reduced trust, right? because they're like, I know this isn't a person, but you're telling me through all of

the language and, you know, layout that it is a person, but I know that it's not. So, now we're breaking that trust. Um, another example of this was Pixie. Um, so on the Frigid Air site, it started out with, you know, a beautiful picture of this person whose name is Pixie and it's very cheery and how can I help you today and give me all your information

so that I can get you to the right place. um and it's interacting with you and giving you feedback and responses, but then when it transitioned to a real person, it said, "Hold on, I'm going to hand you off." Then it goes silent for a super long time just waiting for this person to come online and then gray bubble, white letter in it. Hi, I'm Dave. And

it's like, what what happened to Pixie and like the cute conversation we were having like and that started breaking down trust too, right? because we've built up this expectation of here's how this conversation is going to flow, here's the way that it's going to happen, and now we've transitioned to something that's totally different. We actually had some users who were like, is this another robot? Like, what

happened to the picture and you know the person's name? Like, this doesn't really match up. So, we wanted to show you an example of where transparency actually helped. So, on Open Table, the AI is clearly labeled here. Um, and it's not using a cute name. It's just pretty straightforward. It's called concierge. Maybe they will change that name in the future. Who knows? Um, but at the time

of our study, concierge is pretty simple, right? Like it tells you what it is. It's going to be helpful. Um, and they liked that the AI was called out in multiple locations. So, it's kind of hard to see in the gray background. This does a little popover window. Um, but on the main screen, it says AI in the little box behind it. But then here in the

concierge window up at the top next to the word concierge, it says AI. And then at the very bottom, sadly, this screenshot is a little cut off here on our screen. Um, but at the bottom, it also says the results here were generated by AI. It also does a couple of other things that are pretty common for AI generative tools. So as it's thinking, it has like

a little bubble saying that it's thinking and then it generates the text in lines. So again, the users are understanding what's happening. They see that it's thinking behind the scenes. They can see that it's AI in multiple places and then they get a response and can make a judgment on whether it's answered their question or not. And on that front, the the old pixie to Dave handoff,

it's really important to be intentional about this from day one. So, we want uh one of the the things about the fridge dare experience, it was really nice that they did get handed over to a human. And a lot of people when they were entering that experience didn't expect that. They thought they were just going to have to fight with the chat. So framing that upfront that

there's a human there that like it's you know when you're calling and you're getting the phone tree and you can say representative you know it's it's sort of that same experience you need to indicate that there is going to be a human if needed um you want to maintain that chat history as we talked to users about what they wanted to happen in that they wanted like

I don't want to have to repeat myself please can they read what I already wrote um you want to set some expectations about wait times because as you are waiting for a real human it might not be instant And that that's part of the price of the human, right? Um and you want to train those human agents to acknowledge um the conversation that just happened. So you've

kind of killed that that fear that I'm going to have to start over and provide clear indications of when it's hand off like I am, you know, you don't want to say I am human, but something that that indicates that that that we've >> real person on the other end of that. Um, and then the uh association members that we've talked to, they expected, they were really

generous. They're like, if I get an answer within 20 minutes, that's okay. So, just being transparent about that that wait time um was was the important thing for that handoff. >> Oh, and this pix. >> Yeah, here's here's the picture of I guess it was Mark, not Dave, but Pixie and Dave. Yeah. So, it's just so weird. Like there's a human photo and then gray bubble that

>> and it looks canned. I mean that's a canned response right there. Are you still logged in? Like >> Yeah. Well, and that was that was one of the things that that reduced trust as they were they're like, "Am I getting a human?" That was a lot of sentences real quick. And but then they're like, "Well, if I were that human, I would probably try to speed

up my work and and have a a canned intro also." So that's a that's a harder one to solve. >> Yeah. All right. Uh so next thing that we need to do is we need to tell people where to start. Uh so instead of an empty chat box which is very intimidating if you've never used a tool before or you're in a hurry or you're stressed out

about something. Um we should try to provide pre-populated questions. Um again responds uh participants responded really well to things like open tables concierge. So on that first page, which I should have put a screenshot of that in here now that I'm thinking about it, add it on the fly. Um, so on the first page, the concierge had an open box so you could type a question, but

it also had three prepopulated questions to give you an idea of what kind of things the tool was going to be good at answering. So those type of things are really helpful. They help users judge the level of effort based on the type of content that's presented in there. So, we would absolutely recommend leading with three to five common questions, you know, depending on how much room

you have, where the chat thing is in the interface, um, rather than just having an open text field where they can type in whatever they want. This reduces cognitive load. It improves their confidence and it helps them understand what the tool can actually do. um you can allow the open text as kind of a secondary option, but giving them that helping hand helps reduce stress and gets

them into using the tool right >> And I think it's very important right now when we're in this world where is it search, is it chat, like how should I phrase this, what can it do? Like setting that example is going to be more and more important right now. And number five, test with your actual users. as we, you know, we went through a lot of different

kinds of experiences and like we mentioned, we're going to have to keep testing because these expectations and uh what users are bringing to the the the background that users are bringing to their experiences, it's going to be changing a lot over time. So, it's really important to look at your context and the kind of questions that your website is seeking to answer. Um, so that we can

get to the right place because we can't just rely on conventions. We found that like, you know, if we're like, "Okay, bottom right. Okay, not there. Can I do top right?" No. like it's it's it's it's going to be more specific and more nuanced depending on the nature of your >> All right, last one. This is a really good one. So, this is the one that you

should take take back share with everybody closer to grade. Come on. >> Love it. >> It's going on socials. Yeah. And then is it just oh so just to give you a little flavor for some of the actual quotes that that happened during the study um you know just the difference of opinions we shared a lot of things that were the themes but some of the individual

perspectives so that AI skepticism starts early and like we said the some of the younger participants had used AI more and they were more skeptical. So the 15year-old sports fan it's AI you can't 100% trust it. Um but then >> said with a lot of attitude of course >> and then the personal experience shapes trust. So the 73year-old has been impressed with AI accuracy when using Google

for medical research. He called himself a self-described technology dinosaur. Um and then some reactions are emotional not logical. So, uh, using chat GBT for medical symptoms just felt icky or, uh, there was one, um, user, um, a guitar teacher who, uh, had their own little trust hierarchy. So, okay.gov websites were better than Wikipedia, which were better than AI tools, which were better than chats, right? >> So,

we just go down the list. Which one are you? >> Yeah. Not all AI is equal. The 11-year-old says that Gemini kind of sucks. All right. Uh, so thank you so much for joining us today. Uh, please connect with us on socials on drupal.org. Uh, got a little QR code here to provide some feedback and we would love to answer some questions. And if you ask a

really awesome question, which is any question, we have some really cool stickers for you. >> Okay. Yes. I'm very curious if you come across chat bots or icon or whatever in the top right because I'm thinking about you always put like things together, right? If they're searching, if they're FAQing, if they're chat botting. >> So the frigid air site go question. >> Sorry, I'm ready just to

answer the questions. I'm like got a question. >> Uh so >> because it makes sense to keep that search. I mean, that's where search is. It's been there for a lot longer than a chatbot's been at the bottom. >> Yeah. The the question was, did we test or have we looked at placing the chat in the top right corner in the more traditional search location? >> Yes.

>> Yes. So, one example of that that we saw during our study was on the Frigid Air site. So, the Frigid Air site actually had chat in the bottom right hand corner. Um, the pages that we were looking at were specifically for scheduling maintenance. So, they were pretty slim down. There wasn't any call outs or extra content on the page. It was just a form. There was

a single chat bubble in the bottom right hand corner. And then there was a little icon in the top right hand corner that was a person's face with a little headset. And we did see that a lot of our participants were trying to click on that instead of clicking on the chat So, I would definitely test it because again, it depends on what kind of help are

they looking for. what is the task they're trying to accomplish? And again, giving them multiple ways to solve those problems is the best answer. >> I could see something like in the top >> having a drop like clicking on the >> and having a drop down where you have your options. Exactly. Search your FAQ, your chat. >> Yeah. And I I think some that one challenge that

some of the cases that we looked at might have with that is that it wasn't sitewide, but the chat was only available in certain contexts. So that's probably why the solution that we're familiar with exists, but something something to address. >> Next question. Yeah. >> So I I was like on Chipotle and I couldn't find their chat, but you know it's not like there. >> Um >>

yeah, >> you have to go to contact us. >> Yep, >> you do. Okay, interesting. But um so somebody like posted to me. I don't know if you ran into this at all, but um like a screenshot of somebody like they were saying they were really hungry, but first they had to organize their files and if it could write them a script and it gave them like

a really awesome Python script. >> Cool. >> Um you asked the chatbots to do fun things like that. No, but to repeat the question >> have you asked chat bots on these like consumer websites to do your coding for >> We the question is have we asked the chat bots on the consumer websites to do our coding for us? >> not yet >> because apparently Chipotle is

>> is really good at it. >> Apparently Chipotle will do some Python. That's a that's a good tip. So Pepper is a developer, right? That's what we're saying there. Okay. >> Yeah, you could live test it now. That'd be great. >> Um, can you talk a little bit more about um any recommendations or experiences you have about placement of uh the the chat interface, not in terms

of the the page, but in terms of information architecture. Um, my bias is that it needs to be very specific rather than very general unless you're, you know, really integrating the the the IIA into your search, but I've been told that I'm wrong before often by my children. So, um, is there any is there any do you have any thoughts about that? >> Yeah. So, the let

me try to paraphrase the question. So, the question is, is >> you have to go to contact us? Should it be on a down page or should it be more ubiquitous? I I don't I don't know. >> Yeah. So, the question was where should chat and search be in a site? That's a very simplified form of the question that you asked. Um and I think there's several

things to know here. So, one um we need to think about context. So, what questions are people trying to ask? Um so, search is always a good way. Again, in our research that we did here, we found that people often started with search. They're very familiar with that. experience that's been on the web for a very long time. So, we don't want to hide that. You know,

we've got some standards that guide people to those things like an open box where they can type in keywords. Magnifying glass or the word search is very helpful. We want to make sure that those things are accessible. So, if someone's using, you know, screen readers or technology like that, they can access those things. Um, FAQs, I used to be of the mind that FAQs kind of burn

in a fire because we would want people to actually go to that content. However, I'm having to reframe the context of my opinion. Um we've done several um talks on AI and how to make sure that you are showing up in the AI search engines. So just like SEO, you need to have the right keywords and good content on your site. Um the AI engines are looking

for your content to be structured in a specific way. And the way that those search engines are looking for things is by looking for things like FAQs where you're asking questions. Um, so previously a lot of organizations would put those on one giant page called FAQs with a thousand accordion tabs. Um, recommendations that we've been making to a lot of organizations recently are trying to put those

more in context. So you're building a page that talks about a specific topic or area of topics. Um, and then that page has an FAQs within it so that the search engine and AI engine can read those answers in place of the additional content that you have. So, and it really depends. I know that's a answer that all UX people give is it depends. But >> I

I really like the idea of like I'm already thinking okay well then maybe in an FAQ context then you would have Do you have a different question? We can >> Right. Exactly. Well, and I think it came down to meeting the users where they are. So, if they are going to go to FAQ first or search first and then and then be open to the the other

solution like offer it there. So like this the D1 100% of the time and that's in context. I'm not I'm not navigating to it. It's not like even in the search. It's okay, let's let's get into this and offer that perhaps wherever users might want to dig further or be able to. >> Okay. Thank you. >> Yeah. uh was the um chat baggage with younger uh participants

consistent and the reason I ask is like you know going back a few years there's some research that conversational UIs can outperform traditional websites but of course context matters as well. >> Yeah. Um the question was was there a difference in the chat baggage with younger participants? um perhaps nuances in the diff in in their um in their experiences, but it was 100% negative. Nobody nobody was

excited about it and nobody thought they'd get good answers >> sadly. All the way in the back that we'll come over here. Uh so clients I work with their target audience doesn't range that far. So it's entirey >> Is there any other range of audience? Is there any other patterns that might have emerged with >> I think the question was that for your website, you have a

more narrow demographic audience on the younger side. Were there any findings to lean into that were the trends among the younger? I think with our sample size we don't like it's it would be hard to say you know all young people are are this direction but we have done uh usability studies on higher education sites with chat and again the user base mattered. We were doing a

a study on a site um for a social work school and the participants were all again adverse to the chat experience or or more likely to be adverse to it. Um preferring to find the content or to talk to the human. >> Yeah. I I will also say I think it I hate saying the word it depends but I say it so many times. So uh it

I think the content of a higher education is also part of that conversation. So we worked with an organization where um their the education that they were providing was all around social justice and connecting humans with other humans. they found AI to be much more upsetting to them than other um organizations where maybe they're more techfocused or their degrees are more generalized. Um and so again, those

people are very empathetic. They want to talk to humans and connect to humans. That's why they're going to school. They want to learn those skills and then put those out in the world. So again, I mean I the last slide where it's test with your people. You really have to start talking to your audience. collect feedback. If you've got a contact form or can do feedback forms

on your site, um pilot program with a small group of people, I would highly recommend doing that because every audience is going to be a little bit different. >> Oh, sorry. Um I just I just want to just share that um I was surprised too. I have a uh 21 and 24 year old and you know we talk about like chachi tea or whatever in the house

and you know they're they're college graduates and they have an aversion to it and they're they try to tell me like mom you know you can't believe do not trust cat they're like they're always trying to tell me like just you know don't trust that and I never rely on that and and uh I think too knowing it could be at least from my experience it because

I think it's really interesting too because I thought that they would be using it for everything and like loving it but they uh just from my own experience uh my two boys are very skeptical of it and they're always like telling me like you know and um it also could be too that uh it's been drilled into them since like high school from their teachers and their

educa you know their instructors that like do not use chat you know do not use AI uh to write your or do your research or like you know so it's probably been drilled into them as like a very young age that >> untrustworthy as a resource because that's what they've been told forever from their teachers and you know and their professors and things too. So be part

of it maybe just to >> parse to it because they're like I'm gonna get in trouble, >> you know, because kids get in trouble all the time for using that for >> things. Yeah. >> Well, just to paraphrase for the recording, um your experience with your 21 and 24 year old is that they're more adverse or perhaps have a negative association because they've been told over and

over to not use it for things like homework and they are encouraging you to be more skeptical of it in your use of it. Um yeah, I think one of the a lot of different nuances in how they think about it and one of the younger participants mentioned um that it was good when you had a big volume of knowledge that you needed to sort through like

if there was a something that had lots of stuff written on it and you wanted to find a theme or you wanted to find the answer within that big body of knowledge. So that was again like surprisingly nuanced um perspective from a teenager. >> Yeah, for sure. Uh, I would also say, I mean, again, your experience is similar to what we saw. So, you can tell your

kids, hey, I saw a thing and they totally are saying exactly the same thing that you're telling me. Um, I also think that again for some of these tools, you know, they've been told, don't use this, don't use this. But if you've never interacted with a tool, then you may not know what the tool even does. So, that's why I do think it's really important to start

showing what some of these tools can do. If we just put a chat box on a page and it says chat with us, do we know how to chat with somebody and what possibly it could answer and if our very first interaction out of the gate with that chatbot is I don't know or sorry we don't have an answer for this and then it closes the window

then that's going to be a negative experience and build that chat baggage or chat trauma that we were talking about. So if we can do things to set it up so that they have successful experiences engaging with this type of technology, then they understand what the limitations are, what kind of questions it can answer, and how they're supposed to interact with it. >> And I think you

had a question too. >> Uh the comparative research on the various language models or AI providers on which one is the most effective on something like an ingrained uh search? The question was, yeah, have we done any research on um different LLMs which are more effective in in this sort of chat? Um we we have not >> would love to. >> any other questions? Yeah, >> I

guess like kind of tied to that like have you looked into like curated search and you're like you know things like best bets and actually like putting like when people are actually using traditional search like pointing them to the right things and is that I guess more trusted or less trusted than you know chat bots. >> Yeah. So the question is um search where basically where you're

giving them the list of pages with their answers rather than the answer, right? Is that more effective? >> And I that's what several of the participants were looking for. They're like, I want to go look at that page that's got the answer and get the whole context. Um even in Google um the reason that experience was so trusted they have even made more prominent the sort the

cited sources like you can see pictures now next to the results and they were immediately clicking into those answers to verify it. So I think I think being sure to if you're going to give an answer also give the pages that have the answers on it and then you're going to have that full transparency and I can go check the facts. >> Yes. I'm curious whether you

learned anything about how the affordances accompanying a chat box affect trustworthiness. Like the little thumb up, thumb down is there. The little reload thing is there. Does that help at all or >> Yeah. So the question is, do different affordances within the chat experience impact trust? So thumbs up, thumbs down, the the spinner. Um I think that some of those patterns cue them that it was AI.

So help them understand that it was going to be AI. Um I don't know if you have any >> Yeah. Yeah. Um so I do think that there was some helpfulness in the way that information loaded. So for example um in Pixie, you know, it's right out of the gate saying that it's an automated assistant. It's going to help you get to the right location and it's

very snappy. Um, and then what would have been really great is if they were doing the handoff, there could have been some kind of visual indication that it's thinking or doing something to make that connection. For several of our participants, they kind of sat there and waited for me to tell them like, "Hey, we're going to go on to the next task, but we were actually waiting

for the person to connect online." Um, and when they finally did, it then popped up another thing. So, it would have been more helpful um if it had done something in between that time to take up space or even better plus that up by saying, "Hey, it's going to be about x number of minutes for you to wait for this person to come online." So, I do

agree the more we can kind of put in there that shows them how it's working, how it's interacting, again, back to the open table example, they loved that it kind of typed in the content. So, kind of like chat GPT where you tell it a thing to do and then it does its little thinking bubbles and then it starts filling in the answers. Um, that's a pattern

that's becoming very common. Um, the open table one was using that and they were like, "Oh, I like this. It's like thinking about it and then giving me the answer instead of just boom, it's all on the page. It looks like it's interacting with you." Yeah. >> Um, so yes, I do agree that having those little things in place do help. The thumbs up, thumbs down. Again,

we didn't experience any of those while we were testing. Again, our sample size was pretty small, and we were trying to do for very general audiences. And it can be surprising to try and find sites that have reliable chats that tell you that they're using AI, which was the whole purpose of our discussion here. Um, so we we didn't see any of those in uh in the

wild. Oh, this is just something happening on your thing. Um, but I do agree that those help build trust because again it's another example of that organization saying, "Hey, we want your feedback. We want to improve this. We're thinking about you as we build and implement this." Whether they actually do something with that information, who knows? But it it does help build trust >> and and as

we asked for anything that would help them uh find or notice or understand the chat better, that was one of the questions we asked. like a 100% of the time they want the little dialogue little pointer thing. Pepper. We're gonna keep picking on pepper. I >> Um but it's a >> I'll still eat Chipotle. >> Just a a little oval button, but they're like if there had

been a little tail on it, I might have known. Yeah. >> You know, so the tail and the the twinkle um were were both things that were mentioned. >> Anyone else? Oh, yeah. >> Um curious, in what ways have you been able to use this research to bring back to clients to then like make decisions or direct are going to use these tools. >> Sure. Uh the

question is in what ways have we been able to use this research to bring back to clients to help them with their implementation. So we've got a handful of clients that you know were just early adopters out of the gate. They went and got chat and they have it on their site. Um and they've had some time uh now to get uh you know experience and see

how much it's interacted with and a lot of them have been kind of frustrated with the rate of adoption that there hasn't been a lot of um use with it. So, we're excited to have this very fresh research like I mean literally a few weeks ago um available so that we can help think about what's next and about how can we use that powerful technology you've done

something that's actually pretty great that can answer questions about the content on your site in a thoughtful way and we've tested and refined it um but how can we bring it into the experience where they want to find it. So that's I think I think we're going to be looking at a lot of updates to search pages and uh and and how we uh involve it there.

>> Yeah. And again like I mentioned earlier, you know, my stance on FAQs has absolutely changed, right? The world is changing every day with these AI engines. So before we would want to incorporate all that content just kind of on the page, make sure it's in place where users are looking for it. But we have different things that are looking at the site. Now we've got these,

you know, bots and things that are checking out our content. We need them to have answers because people are not always going to Google. sometimes they're using these chat tools and getting a link from there and coming to our site. So, we want to make sure that we're showing up in those um and that, you know, there's the right link back to our site. >> Yeah. Yeah.

And I think the other um the other element that we're going to be looking at is really the focus on search because that's you know the path of least resistance. that's where we're going where we where do we bring in AI and then later or provide the option for the chat the more you know sort of conversational approach as we expect users to want that more and

to be more comfortable with it also please keep testing with real people humans use computers not just other computers >> anyone else I thought I saw a hand back >> I think we're right at time so yeah thank you so Thank you.