You Know How to Test. Now Learn How to Prompt - by Anton Angelov

This talk explores the fundamentals of prompt engineering, emphasizing its significance in generating higher-quality responses from AI models. The speaker elaborates on various patterns used in crafting effective prompts, such as the persona pattern, template, and examples, to enhance the interactions with AI systems. He provides practical advice, including the use of delimiters to organize input data, establishing boundaries for AI responses, and employing cognitive verifiers to ensure quality output. Additionally, the speaker discusses advanced concepts such as context engineering and spec-driven development, highlighting how these practices can improve the effectiveness of AI assistants in test automation and other applications.

Challenge accepted. [music] Challenge accepted. Challenge accepted. P2 [music] is always challenge accepted. Rena just opened Pandora AI box for us. >> And just like in the myth, hope was still inside. >> In our case, that hope is the knowledge of how to use AI wisely. >> You know how to test. >> Now it's time to learn how to prompt. >> And who better to show us than

someone who's practically a legend at accepted >> and ask me the question. Yeah, >> not only a legend but also an author of eight, I repeat, eight bestselling books on on test automation and you can get some some of them uh at the reception making by donation. Please welcome the inventor of Bellatrix test framework, the multiple winner of QA of the year award and a proud winner

of QA of the decade, >> the co-founder of automate the planet and one of the strongest voices putting Bulgaria on the global QA map, ANTON [screaming] ANGELO. HI, thank you for having me. Uh this lecture will be about prompt engineering and you know many people question the value of prompts and I'm not going to teach you about prompts only for generative AI like using it in chats

but also we uh need to use it in assistance in AI agents when we use uh AI powered um ids like uh cloud code or uh cursor And uh this is why this topic is really important. And before I begin, I need to say a few words. U in my opinion there are three type of lectures. The first ones are these funny ones with memes full of

personal stories, right? Uh then you have this inspirational talks with big images talking about the future, right? And then the third type in my opinion are the technical ones that are straight to the point without fleshy images. And a couple of weeks ago I delivered a speech at Diguel in one. It was uh from this second type inspirational talk with lots of images. And um some people

that uh I really respect told me that maybe this was one of my best features back then. And other people told me that they like the talk. But um they expected from me something more practical and this is why today my talk will be fully technical without flash images just straight to the point. So uh I will begin just shortly introducing what is prompt engineering then we

will look into um maybe the best uh prompt patterns in my opinion. you'll receive 10 pro tips from me and then we are going to talk really briefly at the end uh what's beyond prompt engineering because there are some practices like uh context engineering specdriven development etc. So let's begin. First, what is the prompt engineering? This is the practice where uh you need to use these formulas

uh that are discovered by scientists to formulate better your input so that the AOM answers in in a little bit more uh strict way and you get higher quality answer. Um and uh this is the prompt engineering. It's similar to design patterns that we use uh when we code. Uh so yeah, let's begin. The first pattern that I want to share with you is the persona pattern.

Uh I'm sure that most of you use it. We usually use it at first. And um one of the tips that I haven't written later, but it's really important uh when you write an assistant prompt and when you stack multiple patterns, you need to begin first with the most important information uh that uh you want to instruct the aoom. Uh this is how it works. So first

we start with this persona. Uh you instruct the AOM uh to be in a specific role. Imagine that if your prompt is much more general, you will receive a much more general answer that probably it its quality uh won't be good, right? And um like we have different roles in QA like uh senior QA, we have test automation, we have security, you have manual. Uh you need

to put the role but it's not just about um like uh people roles, right? You can uh for example when you write an AI agent or assistant in your test automation framework you can define a role like uh you're a very good assistant for finding the proper locators or your uh professional XML parser or something like that and uh this totally changes uh the final output. Then

the second very important pattern is the template. This is setting the structure of how the AOM will return your answer. Without it, you will receive probably much more general and every time different answer. However, imagine that you have an assistant for creating test cases. You can define the different um like uh titles, properties that you want to receive at the end. And also, it's very important to

see that right now we don't have any specific information. We'll put that later in another pattern. But here we have this placeholders like uh using brackets and imagine that uh if you have a book report or test case you have priority severities which means that probably you have different options that they will be different for each specific project. Um and uh usually we place them inside the

placeholders and you say for this particular project I have these five different priorities. Later on you can describe what they mean like a severity matrix describing what they mean for your particular project so that the system knows how to um how to create them better uh in the output. And then of course the next that combines very well uh with the template is the fshot. Uh fshot

sounds weird weird uh but you can think of it as examples. Now we use our template but with exact patterns and data. Uh usually we provide two to three examples so that the AI can learn from this patterns. Uh for example if you are talking about book reports or test cases it's uh much better to uh again uh create examples with varying data like different priorities, different

steps etc. Otherwise probably they won't help much. And you will see also other terms uh that you will meet zero shot one shot this means uh usually we don't provide any examples just straight to the point um and if you're lucky like this is like if you're lucky with the Google this is zero shot then you have one shot just one example but as I said usually

the more um var examples you provide better then another thing through my practice that uh I um found is the so-called delimiters. These are symbols that they separate the different sections of the data that you provide. Imagine that you want to generate uh test cases based on stories, acceptance criteria or a piece of code whatever and uh if you just provide the plain text uh you will

find out that uh you'll find out that uh probably the output sometimes might be not of a good quality but if you separate them with different symbols like hashtags or dollar signs etc and you explain the AI between which sections what you have. Uh you will quickly find out that the output is much better. And then we need to put some boundaries. Uh and uh this is

one of my favorite ones especially when you are creating a specialized AI agents in test automation frameworks. Do not explain or expand. Sometimes this is helpful because uh usually these explanations they help later on during the conversation for the AI to answer better. However, when we write specialized AI agents or assistants in a framework, imagine that in our framework, we have a special feature for uh finding

a locator expert by prompt. And then I don't want explanations. I don't want quotes. I don't want anything. And this is uh one of the boundaries, one of the limitations of the answer. Then we have the preip flip interaction pattern. This is quite useful when you use assistance uh with multiple uh patterns together. Uh imagine that you don't uh sometimes your inputs are a little bit vague.

Um and uh you can uh you can put this instruction in the prompt so that it asks you a few questions beforehand to understand fully your context, your requirements. Uh and even um in some of my assistants, I usually directly instruct the AI to ask me five to 10 questions up front. If you have used the deep research of Chad GBT, Germany or whatever, usually they start

with asking such questions. They won't start working 15 minutes uh using so much electricity before asking you those questions to clear all the requirements. By the way, yesterday I was on another conference. There was a pretty interesting statistic that for generation of one image uh the electricity that uh is used by the AOM is equal to uh working of your refrigerator for an hour. And you can

imagine how much electricity will be uh used by using a deep research without uh this particular context. Right? Then we have the cognitive verifier. Uh this is very important. uh we put it at the end of our agent prompts or assistance is basically teaching the um to QA itself. Imagine again um I found it very useful in using it in uh this agentic feature in our framework

without it sometimes like for example one of 10 times uh it was returning experts with quotes or not well formatted but when you put uh this cognitive verifier for checking its output again whether it follows all the expert rules or whether it uh it didn't mistakenly put some quotes it work uh much better and you can use it for test cases for book reports whatever whether they

really follow your templates and patterns then if we use assistance um I found this uh very useful uh to make them uh interactable imagine that uh these old school applications come online for big assistants that they can do multiple things you can insert such menu it's called a menu pattern because uh it lists all of the options that the assistant can do for you and uh when

you pick an option it will do the particular uh action for example create a book report or create a test case for you um and this works very well with another pattern that I haven't included it's called tail generation the tail generation is you uh usually used by some of the more advanced models by default it's like summarizing what they uh what was the conversation and this

is uh because of the way how a work they forget things they forget them because uh they forget things because uh of the context window the AOM is not uh like uh it's not having its own memory right um and then this tail generation is is generating this summary of the conversation and how this fits Basically we can return the menu at the tail generation every time.

So you can pick the next option or repeat um the whole process etc. And uh the last pattern that I wanted to mention is called alternative approaches. This is very helpful for architects or designers that you need to come up with a brainstorming uh an idea. Imagine that uh you are building uh new automated tests and um like today one of the topics will be test data

preparation and uh this is uh very important for high quality automation right so you need to pick whether to use real data um mock data or uh prepare it yourself with the API or directly in the database uh and even if you don't know all of the ways you can ask the AI to brainstorm ideas with you and it can even create pros and cons table and

basically brainstorm together with you. This is very useful and we can uh integrate it in into uh some of the systems and then directly to the pro tips uh as I said technical talk straight to the point. So, um I am sure that most of you used very big documents sometimes uh with the AI and I'm sure that it's very hard to determine whether the AI is

reading them or not. And I can tell you that for sure it's not reading them in full. Like for example, it cannot read more than 10 pages at once. Even if you upload a bigger document, it will probably answer you uh like it knows something but uh quickly you will find out especially if it's not something that it was trained on before that uh it doesn't know

it. Um and this is because actually behind the scenes uh this is from chat GBT other um genai charts have it as well. It's basically you can see that it's generating Python scripts behind the scenes to read all the documents the uh the word files the PDFs the OCR for the images etc. Um and when you want to work with bigger documents like 100 pages 300 pages

you need another approach and uh the approach uh I found out that it's very useful is to create an index. What is the index? The index it's like if you remember reading a book at the end of the book you have the index with the terms every term have a specific page. So basically you describe all the different topics from which page they are for example 1

to 10 then 20 to 30 etc. And with a few sentences you describe uh what's inside this particular section. When you want to use the document, you upload both the index and the document and you instruct the OM first to check the index to see where the relevant topic is between which pages and then to read these exact pages. Uh I made an assistant that helped me

to craft UML diagrams. The reference guide for this specific programming languages designing the UML diagrams was 300 pages. You can imagine that without it without the reference guide totally all of the diagrams were I I won't use the word uh but uh when we use these both approaches it worked like a charm then uh when you don't know uh when you are learning about prompts especially now

you're watching a lecture when you are starting up front uh probably you won't write the best prompt patterns at first and that's okay. Uh but you can open the new chat and maybe put your instructions there and ask um the chat GBT or the other uh generative AIS to put all the best practices inside this prompt. It's working. So, so I mean when you describe the particular

patterns, it works better. It's better than nothing. But basically, you can turn this generative AI charts into your personal trainers like uh you can ask it even to put some comments and explain why it used this particular patterns and little by little I'm sure that you learn some of the patterns. Then uh you can go one step further. It's like writing high quality code. When we repeat

ourself and writing the same logic again and again, we can put it inside an assistant. The assistant basically is something to reuse our prompts every time. When you type that, it will use the prompt again. So uh in my case when I began to learn prompt engineering, I created for myself uh custom GPT for uh prompt crafting. Basically I instructed it to know about all these 30

to 40 patterns and every time to uh use them and format my prompts right and to improve them and uh this is a general tip for any um assistant that you're writing for any prompt. Again this is like coding when uh you have repeating logic in your tests you are moving it first to a method then you are moving it to a class and is the same

here. If you're repeating yourself and writing the same prompt again and again probably you need an assistant in chubby this is called project in Germany this is gems and um depending on what you use uh you can define this prompt in main prompt is called the root prompt uh usually it consists of multiple patterns stuck together at the end there is a slide with a formula that

will help you then um the next obvious step is to go multimodel if you haven't heard the term multimodel means that we don't use just plain text to instruct the assistants and the agents but instead we use it together with files like um you can provide a PDF a word CSV XML images it doesn't matter it works both ways it can generate files CSVs uh or other

files then uh even uh many people make the rookie mistake first when they try to talk with CHGBT or other generative AI chats that they try to have this conversation and they think that it's working because they're reframing the output is not well the first time then they clarify what they want then they clarify again and they're repeating this process 10 times but uh the way again

this a work they start to forget really quickly because of the context window and because it doesn't have a memory right and this is why When you know about these formulas, these patterns, it's best to spend 30 seconds or one minute to really define properly what you want and then uh to do it. Um and and even if you uh don't get the right answer, even after

you put some prompt patterns, um this doesn't mean that the AI doesn't know how to how to do the job, right? Uh instead you can open the new chat start fresh when you have the proper prompt or you can ask in the new chat what this was my input this was my uh this was the answer how to improve it please help me how you are going

to rephrase that uh to to have a better answer and then we have uh I already put this pattern like boundaries but uh I will show you one of my favorites to explain the AI not to hallucinate. Some people will uh will argue with me that their research that the AOM are not very good in following commands like do not do something. In my own experience, I

usually put some constraints like that and they work for me. For example, I always say if you don't know how to do something, please put this into the Excel sheet or this particular row or something and it's working fine. Especially now with the capability of searching using this tool, you can ask it first to double check if it doesn't know and then try again. Then uh I

already told you about uh the usage of delimiters like symbols but but you can go beyond that. Many often you will see many of the systems and um AI agents uh like prompts that they are using actually markdown and this is because the markdown format um within it's using a lot of symbols for example the headings they are using hashtags uh for the bolt we have asterisk

right and this doesn't mean that you have to um type it yourself or using um any editor because this is hard nobody does that but you can build an assistant for translating your prompt into markdown. This is what I'm doing. However, we come now to pro tip 10. I saw some of my colleagues try directly to um ask the assistant to save the markdown directly into a

file because you will find out really quickly that you cannot copy the markdown from the chat but you need to save it and instruct the the chat to save it. However, you'll find out because the chat uh the genai is very lazy sometimes because of the Python scripts uh instead of having 100 lines uh in the markdown file, you'll end up with 10. However, if you use

the stepbystep plan, uh you will see that um it works uh much better when you basically uh ask it first to generate the markdown and then save the file. And this works obviously for many other types of information. It's not just the And uh this is uh like the formula for creating a basic assistant. First you start with persona who you are. Um then add a template

at examples with patterns uh the AI can learn the patterns. Add some boundaries constraints. do not explain or expand and then use these delimters to separate if you provide additional information into the and then what's beyond right now if you have heard the term vip coding vip coding means that you put some uh vague instructions to um like ids or coding agents like cursor or cloud code

or wable and then they create this full-blown application right uh In my opinion, uh this works best if you know already how to code, how to create architecture. Otherwise, u you will end up really quickly with just a few pages that are working and after that nothing works. And this is why new type of engineering practices emerge. This is the context engineering. Now we are not asking

just what and um defining these prompts. Uh but now the question is what to feed inside this uh 100k tokens uh context. This is basically what we sent every time to the a uh what what type of documents to feed to know like your coding standards your uh full um architecture of your current application with all the the files etc. uh this is actually what uh the

different coding agents are doing basically in this uh cursor uh markdown files instructions etc in in parts uh as I said the context engineering is much beyond that because you put uh much more information into this context and then uh we have the specdriven development what's that uh like a few weeks ago maybe two weeks ago GitHub released a new library it's called spec kit and they

found out that just prompt engineering and vibe coding it's not as I said um it's not working very well this is why we need to define the context or the requirements up front the spec kit is generating with AI all the requirements in girking format for you uh and uh you use it basically together with the context engineering and all of the other practices these three practices

together to craft uh very strong prompts. This is how this And uh yeah, we discussed the prompt engineering, the 10 prompt patterns, prompt tips. Uh you can find some resources here. I really like the two books here mentioned. Uh for the second one, I was a technical editor. It's really strong book. The first one from Microsoft uh is really deep diving into a last chapter even discussing

serious math. Maybe you can miss it. It's it's fine. Um and then there are a couple of courses about prompt engineering in Corsera. Um I really recommend them. Uh they're really great. And uh there are a couple of frameworks actually uh and online sites that can help you directly to learn prompts and uh craft your uh like uh prompting guide AI and there are a couple of

others. And you can contact me on LinkedIn every time if you have questions. And yeah, thank you. >> Thank you. Thank you. Thank you, Anton. [applause] Uh, do do you want to answer a question or two now or in the speaker corner? >> Maybe in the speaker's corner. >> Yeah. Okay, you heard him. You can find him in the speaker cornering during the next break. Thank you

very much. Let's hang him off the stage with big round of applause. Anton, ANGELO