Fundamentals

Deep-dive: Optimizing prompts for AI Actions

Practical examples to generate desired responses

This guide demonstrates the process of optimizing a prompt for AI Actions feature. By following this real-world example, you'll learn how to craft effective prompts to extract valuable insights from your curated articles.

What you'll learn

Before diving into the details, here are the key insights you'll gain from this guide:

• Setting a goal and working backward from the expected result leads to effective prompts.
• Iterative refinement is crucial for improving prompt performance.
• Being specific in our requests avoids confusing the AI due to ambiguity.
• Allowing the AI some flexibility can lead to richer, more insightful responses.
• Clear instructions are essential, especially when asking the AI to make inferences or assessments.
• Testing the prompt on various articles helps ensure its robustness and effectiveness across different contexts.

Note: For purposes of illustration, we focused these examples on a market intelligence use case. The general guidance applies to other use cases as well.

Before you start

Clarifying objectives

To illustrate the process, let's use a real user request as an example:

From this request, we can distill the following objectives:

• Identify and build a list of use cases
• Label the technology and business domain for each use case
• Track specific examples of each use case

Selecting appropriate test data

For an effective iteration process, we'll run prompts on articles with familiar content. This allows us to accurately assess the output quality and quickly identify areas for prompt improvement.

Outlining the desired output

Working backward from the desired output is an efficient technique to build an effective prompt.

From the requirement, we understand this is a scenario of a recurring action on each article. We want to extract the same information from each article consistently.

We need a list of "Use Cases" with the following details for each:

• Tech label
• Business domain
• Examples

A table format can be useful for organizing this information:

The optimization process

Iteration 1: Initial prompt

While crafting the first prompt, we decided to add more columns to generate a more comprehensive output.

Step 1: Crafting the Initial Prompt

Prompt 1

Step 2: Analyzing the Results

After running this prompt, we received the following table:

Result 1

We analyze the results to look for:

• Missing information
• Misinterpretations
• Opportunities for more detailed or relevant output

Step 3: Identifying Areas for Improvement

• Article Reference: Having the article's title readily available would help in quickly assessing the results.
  • Action: Add a column for the article title and URL for quick reference.
• Business Domain vs. Business Unit: The user expects "Business Domain" to be more specific (e.g., operations, sales) rather than generic (e.g., aviation, hospitality). We should add a separate "Business Unit" column.
  • Action: Introduce a "Business Unit" column with examples to guide the AI.
• Concerned Parties: The AI returned specific names, but we're looking for more generic values (e.g., "airlines" instead of airline names).
  • Action: Replace "Concerned Parties" with "Key Stakeholders" for clarity, and create separate columns for types and names.

ℹ️ Note: Similar to the business domain/unit nuance, the interpretation of "Concerned Parties" can vary. To avoid ambiguity, it's crucial to either adapt to the AI's interpretation or provide a precise definition in the prompt.

• Multiple Use Cases: Some articles mention multiple use cases, but the AI only provided one or aggregated the details.
  • Action: Add a note to explicitly request all use cases from each article.

ℹ️ Note: The AI will typically optimize for a concise response. Among other reasons, generating text is one of the most resource-intensive tasks for an AI, and usually, a short, concise, and aggregated response is sufficient for users. If we require a granular and very detailed response, it's best practice to explicitly state this in the prompt.

Iteration 2: Refined prompt

We updated our prompt based on the previous observations. To test our refinements effectively, we'll focus on article [3], which contains multiple use cases. This targeted approach helps us evaluate specific improvements and refine our prompt more precisely.

Step 1: Implementing improvements

Prompt 2

Prompt 2: Initial refinement - Structuring the output and guiding the AI

Step 2: Analyzing the New Results

After running the updated prompt, we received more detailed results:

Result 2

Step 3: Further Optimization Opportunities

• Use Case Name: The AI provided very specific internal code names or project names. For better interpretability, we want more explicit names.
  • Action: Update the prompt to request "a short sentence describing the use case" for the Use Case Name.
• Flexibility in Column Values: While the AI correctly filled in the columns based on our examples, we want to give it more freedom to come up with appropriate values without being constrained by our examples.
  • Action: Remove specific examples from the prompt to allow for more diverse and potentially insightful responses.

ℹ️ Note: The AI often possesses broader knowledge than humans in general. Giving them the freedom to generate results without constraining them with examples of expected output might lead to valuable insights we hadn't considered. This approach leverages the AI's extensive knowledge base and can result in more comprehensive and nuanced responses.

Iteration 3: Second round of refinement

Step 1: Updating the prompt

Based on these observations, we made another refinement to the prompt:

Prompt 3

Prompt 3: Intermediate refinement - Enhancing clarity and flexibility

Step 2: Analyzing the results

Result 3

Step 3: More Observations and Improvements

• Key Stakeholders: Without the example constraints, the AI's definition of stakeholders varied. In the "Stakeholders name" column, we got the name of the project instead of the name of the overall entity implementing the project.
  • Action: Change the column "Stakeholders name" to "The party implementing the use case". This update also inspired us to request information about the solution provider.

ℹ️ Note: The variation in stakeholder definitions demonstrates the AI's adaptability rather than inaccuracy. This flexibility can lead to valuable insights, but it also highlights the importance of precise prompting when specific information is required.

• TRL Evaluation: The AI didn't provide an evaluation of the TRL (Technology Readiness Level), stating that there wasn't enough information in the article. This is because our AI is instructed not to give responses if the information is not available in the articles.
  • Action: To encourage the AI to provide its own evaluation or recommendation based on interpretation, we need to explicitly mention this in the prompt. We'll clarify the TRL column: "Evaluated TRL (Give your assessment if the information is not explicit in the article)"

ℹ️ Note: In the previous iterations, the AI gave results for TRL. The inconsistency in TRL evaluations reflects the ambiguity in our prompt rather than a limitation of the AI. By using the term "Evaluated TRL," we left room for interpretation, whether the AI should make its own evaluation or retrieve an evaluation from the articles. This highlights the importance of clear instructions, particularly when asking the AI to make inferences or assessments beyond the explicit content of the articles.

Iteration 4: Final prompt

Step 1: Final refinements

Based on these observations, we made our final refinements to the prompt:

Prompt 4

Prompt 4: Final refinement - Specifying implementers and encouraging AI evaluation

Step 2: Validating the final results

Result 4

Step 3: Assessing the Final Prompt

To confirm the effectiveness of our optimized prompt, we tested it on two new different articles.

Final result

The results met all our expectations, providing detailed and accurate information for each use case.

Conclusion

Optimizing prompts is an iterative process that requires careful analysis and continuous refinement. By following these steps and adapting them to your specific needs, you can create more effective prompts for AI Actions in Feedly. Remember to:

• Start with a clear understanding of your requirements.
• Craft an initial prompt based on your desired output.
• Analyze the results thoroughly, identifying both strengths and areas for improvement.
• Refine your prompt based on your observations, being specific about what you want.
• Test your refined prompt and analyze the new results.
• Continue iterating until you achieve the desired output quality.
• Allow flexibility in your prompts to leverage the LLM's broad knowledge base, which may lead to unexpected insights.

This guide provides a comprehensive learning experience for users looking to optimize their prompts in AI Actions by sharing the detailed observations and logic behind each refinement. Remember that prompt optimization is as much about understanding AI's capabilities as it is about clearly defining your requirements.

Try AI Actions

Synthesize, translate, extract data, create reports, summarize, and more.START FREE TRIAL