Intermediate
AI Prompting
#hallucinations
#verification
#critical-thinking
Evaluating LLM Outputs
3 exercises — spot hallucinations, verify technical claims, and write targeted correction prompts.
0 / 43 completed
1 / 43
An LLM confidently states: "The React
useLayoutEffect hook was introduced in React 18." How do you identify and flag this as a potential hallucination?Option C is correct.
Signs that a response may be hallucinated:
• Specific version numbers, dates, or statistics without a cited source
• API names that "feel right" but don't match documentation
• High-confidence language on verifiable facts ("X was introduced in Y")
• Names of papers, authors, or RFC numbers that might be fabricated
Verification workflow for code-related LLM output:
1. Run the code locally — does it actually work?
2. Check official docs for any API, version, or behaviour claims
3. Search for the specific function signature — does it match?
4. Be especially sceptical of "new in version X" or "deprecated in version Y" claims
Asking the same model again is not verification — it will often give the same incorrect answer, or vary between answers, which tells you it's uncertain.
useLayoutEffect was introduced in React 16.8 (February 2019, the same release as all hooks). The model's statement that it was "introduced in React 18" is a hallucination with false precision.Signs that a response may be hallucinated:
• Specific version numbers, dates, or statistics without a cited source
• API names that "feel right" but don't match documentation
• High-confidence language on verifiable facts ("X was introduced in Y")
• Names of papers, authors, or RFC numbers that might be fabricated
Verification workflow for code-related LLM output:
1. Run the code locally — does it actually work?
2. Check official docs for any API, version, or behaviour claims
3. Search for the specific function signature — does it match?
4. Be especially sceptical of "new in version X" or "deprecated in version Y" claims
Asking the same model again is not verification — it will often give the same incorrect answer, or vary between answers, which tells you it's uncertain.
2 / 43
The LLM explains: "The
Array.prototype.flatMap() method is supported in all modern browsers and Node.js. It's equivalent to calling .map() followed by .flat(1)." How do you evaluate this output?Option B is the correct professional approach. The statement is actually accurate —
1. Separate claims — "browser support" and "functional equivalence" are two distinct factual claims
2. Use authoritative sources — MDN for browser compatibility and spec behaviour, caniuse.com for compatibility tables
3. Test the equivalence — especially for code explanations, a quick test in a REPL confirms behaviour
The lesson: don't reject LLM output wholesale, and don't accept it wholesale. Evaluate each factual claim independently.
The danger of Option A ("trust it") is not that this specific claim is wrong — it isn't — but that the habit of unconditional trust will fail you when the claim is subtly wrong in a production-critical context.
flatMap() is equivalent to .map().flat(1) and is widely supported. But Option B illustrates the proper verification workflow even when you suspect the answer is correct:1. Separate claims — "browser support" and "functional equivalence" are two distinct factual claims
2. Use authoritative sources — MDN for browser compatibility and spec behaviour, caniuse.com for compatibility tables
3. Test the equivalence — especially for code explanations, a quick test in a REPL confirms behaviour
The lesson: don't reject LLM output wholesale, and don't accept it wholesale. Evaluate each factual claim independently.
The danger of Option A ("trust it") is not that this specific claim is wrong — it isn't — but that the habit of unconditional trust will fail you when the claim is subtly wrong in a production-critical context.
3 / 43
You get an LLM response that partially solves your problem but misunderstands your database schema. Which follow-up prompt is most effective for iterative correction?
Option C is the professional iterative-correction prompt. It works because it:
• Identifies exactly what was wrong — "you misunderstood the schema" and what specifically was misunderstood
• States the correct information — the exact schema structure with column names
• Gives a concrete instruction — "rewrite using a JOIN on user_roles instead of filtering on users.role"
The iterative prompting loop for LLM output evaluation:
1. Identify the error category: (a) factual hallucination; (b) misunderstood context; (c) correct approach, wrong output format; (d) missing edge case
2. Provide the correction: the specific fact, schema, or constraint that was wrong
3. Specify what to redo: "rewrite X", "fix only the query", "keep the explanation, replace the code"
Options A and B ("wrong, try again" / "not what I asked") give the model no information about what to fix and will often produce a different but equally wrong answer.
• Identifies exactly what was wrong — "you misunderstood the schema" and what specifically was misunderstood
• States the correct information — the exact schema structure with column names
• Gives a concrete instruction — "rewrite using a JOIN on user_roles instead of filtering on users.role"
The iterative prompting loop for LLM output evaluation:
1. Identify the error category: (a) factual hallucination; (b) misunderstood context; (c) correct approach, wrong output format; (d) missing edge case
2. Provide the correction: the specific fact, schema, or constraint that was wrong
3. Specify what to redo: "rewrite X", "fix only the query", "keep the explanation, replace the code"
Options A and B ("wrong, try again" / "not what I asked") give the model no information about what to fix and will often produce a different but equally wrong answer.
4 / 43
Sarah: "Hey team, I've used the LLM to generate some SQL queries for our user data analysis. It returned this query: `SELECT * FROM users WHERE last_login > DATE('now', '-7 days') AND city = 'New York';`. Seems straightforward enough."The question presents a realistic scenario where a developer shares an LLM-generated query. While the basic structure looks correct – filtering by recent logins and location – it's crucial to evaluate whether the response fully addresses the requirements and potential issues. Simply stating 'seems straightforward' isn't enough; a good code review would identify areas for improvement such as performance optimization or security considerations. The correct answer acknowledges that the query is well-formed in its basic intent, demonstrating an understanding of what constitutes a minimally acceptable output from an LLM.
5 / 43
Reviewer: 'This PR introduces a complex lambda function within the `calculateTaxRate` method. The LLM suggested this implementation to handle edge cases related to sales tax calculations. While technically correct, it's significantly more verbose than a simple if/else statement and could impact performance. I'm concerned about maintainability and potential future scaling issues. You (as the tech lead) need to communicate this back to the developer. Which of the following is the MOST effective way to phrase your feedback in a Slack message to the developer?The key here is to frame the feedback constructively. Option A dismisses the LLM's suggestion without understanding its rationale, potentially frustrating the developer. Option B is overly demanding and doesn't foster collaboration. Option D jumps straight to criticism without initial engagement. Option 3 – 'Let's start by acknowledging the LLM's output and then gently raise concerns about performance and maintainability…' - demonstrates a collaborative approach. This allows you to understand *why* the LLM generated that response (perhaps it was optimized for specific edge cases) before offering alternative solutions, which is crucial when working with AI-generated code.
6 / 43
During a code review of a PR that uses an LLM to generate unit tests, the LLM produces this test case: `assert(calculateDiscount(100, 'VIP') === 90);`. The reviewer suspects the LLM might be oversimplifying the discount logic. Which approach is MOST effective for quickly verifying the test's thoroughness and potential gaps in coverage without immediately challenging the LLM's output?
Running the test in a CI environment with diverse data sets provides empirical evidence of the test's effectiveness and reveals hidden issues. Directly prompting the LLM for edge case tests can be helpful but doesn't directly validate the existing test. Flagging it immediately without investigation risks prematurely dismissing potentially valuable output. Crafting a similar, boundary-focused test offers a targeted comparison, allowing for a more nuanced assessment of coverage than simply relying on the original LLM-generated test.
7 / 43
The key here is recognizing that an LLM-generated summary, even if seemingly positive, often lacks crucial supporting evidence. Simply stating 'customers loved' isn't sufficient; you need data to back it up. Insufficient correctly identifies this gap – the response needs metrics (e.g., satisfaction scores) or direct quotes to validate the claims. Options B and C are overly optimistic without verification, and option D highlights a critical omission: failing to acknowledge potential issues.
8 / 43
You're reviewing a pull request where an LLM generated the following code snippet for handling user authentication:
```javascript
const isValidUser = (username, password) => {
// Placeholder for actual authentication logic - replace with your database query
if (username === 'admin' && password === 'password') {
return true;
} else {
return false;
}
};
```
During the review, you notice this is a very basic implementation and lacks any security measures like salting or hashing passwords. A colleague suggests, "Let's prompt the LLM to add password hashing using bcrypt."
Which of the following actions represents the MOST effective way to refine the LLM's response and ensure a secure authentication process?
The best approach is option 1 – prompting the LLM to incorporate `bcrypt`. This directly addresses the identified weakness (lack of security measures) and guides the LLM towards a more secure solution. Simply requesting a complete flow might lead to an overly complex response, while ignoring the LLM's output or creating a JSON schema doesn't solve the fundamental issue of insecure password storage. Focusing on a specific implementation detail like `bcrypt` is the most targeted and effective way to steer the AI towards a better outcome in this scenario.
9 / 43
Sarah: "Hey team, I've used the LLM to generate some SQL queries for our user data analysis. It returned this query: `SELECT * FROM users WHERE last_login > DATE('now', '-7 days') AND city = 'New York';`. Seems straightforward enough."The question presents a realistic scenario where a developer shares an LLM-generated query. While the basic structure looks correct – filtering by recent logins and location – it's crucial to evaluate whether the response fully addresses the requirements and potential issues. Simply stating 'seems straightforward' isn't enough; a good code review would identify areas for improvement such as performance optimization or security considerations. The correct answer acknowledges that the query is well-formed in its basic intent, demonstrating an understanding of what constitutes a minimally acceptable output from an LLM.
10 / 43
Reviewer: 'This PR introduces a complex lambda function within the `calculateTaxRate` method. The LLM suggested this implementation to handle edge cases related to sales tax calculations. While technically correct, it's significantly more verbose than a simple if/else statement and could impact performance. I'm concerned about maintainability and potential future scaling issues. You (as the tech lead) need to communicate this back to the developer. Which of the following is the MOST effective way to phrase your feedback in a Slack message to the developer?The key here is to frame the feedback constructively. Option A dismisses the LLM's suggestion without understanding its rationale, potentially frustrating the developer. Option B is overly demanding and doesn't foster collaboration. Option D jumps straight to criticism without initial engagement. Option 3 – 'Let's start by acknowledging the LLM's output and then gently raise concerns about performance and maintainability…' - demonstrates a collaborative approach. This allows you to understand *why* the LLM generated that response (perhaps it was optimized for specific edge cases) before offering alternative solutions, which is crucial when working with AI-generated code.
11 / 43
During a code review of a PR that uses an LLM to generate unit tests, the LLM produces this test case: `assert(calculateDiscount(100, 'VIP') === 90);`. The reviewer suspects the LLM might be oversimplifying the discount logic. Which approach is MOST effective for quickly verifying the test's thoroughness and potential gaps in coverage without immediately challenging the LLM's output?
Running the test in a CI environment with diverse data sets provides empirical evidence of the test's effectiveness and reveals hidden issues. Directly prompting the LLM for edge case tests can be helpful but doesn't directly validate the existing test. Flagging it immediately without investigation risks prematurely dismissing potentially valuable output. Crafting a similar, boundary-focused test offers a targeted comparison, allowing for a more nuanced assessment of coverage than simply relying on the original LLM-generated test.
12 / 43
The key here is recognizing that an LLM-generated summary, even if seemingly positive, often lacks crucial supporting evidence. Simply stating 'customers loved' isn't sufficient; you need data to back it up. Insufficient correctly identifies this gap – the response needs metrics (e.g., satisfaction scores) or direct quotes to validate the claims. Options B and C are overly optimistic without verification, and option D highlights a critical omission: failing to acknowledge potential issues.
13 / 43
You're reviewing a pull request where an LLM generated the following code snippet for handling user authentication:
```javascript
const isValidUser = (username, password) => {
// Placeholder for actual authentication logic - replace with your database query
if (username === 'admin' && password === 'password') {
return true;
} else {
return false;
}
};
```
During the review, you notice this is a very basic implementation and lacks any security measures like salting or hashing passwords. A colleague suggests, "Let's prompt the LLM to add password hashing using bcrypt."
Which of the following actions represents the MOST effective way to refine the LLM's response and ensure a secure authentication process?
The best approach is option 1 – prompting the LLM to incorporate `bcrypt`. This directly addresses the identified weakness (lack of security measures) and guides the LLM towards a more secure solution. Simply requesting a complete flow might lead to an overly complex response, while ignoring the LLM's output or creating a JSON schema doesn't solve the fundamental issue of insecure password storage. Focusing on a specific implementation detail like `bcrypt` is the most targeted and effective way to steer the AI towards a better outcome in this scenario.
14 / 43
Sarah: "Hey team, I've used the LLM to generate some SQL queries for our user data analysis. It returned this query: `SELECT * FROM users WHERE last_login > DATE('now', '-7 days') AND city = 'New York';`. Seems straightforward enough."The question presents a realistic scenario where a developer shares an LLM-generated query. While the basic structure looks correct – filtering by recent logins and location – it's crucial to evaluate whether the response fully addresses the requirements and potential issues. Simply stating 'seems straightforward' isn't enough; a good code review would identify areas for improvement such as performance optimization or security considerations. The correct answer acknowledges that the query is well-formed in its basic intent, demonstrating an understanding of what constitutes a minimally acceptable output from an LLM.
15 / 43
Reviewer: 'This PR introduces a complex lambda function within the `calculateTaxRate` method. The LLM suggested this implementation to handle edge cases related to sales tax calculations. While technically correct, it's significantly more verbose than a simple if/else statement and could impact performance. I'm concerned about maintainability and potential future scaling issues. You (as the tech lead) need to communicate this back to the developer. Which of the following is the MOST effective way to phrase your feedback in a Slack message to the developer?The key here is to frame the feedback constructively. Option A dismisses the LLM's suggestion without understanding its rationale, potentially frustrating the developer. Option B is overly demanding and doesn't foster collaboration. Option D jumps straight to criticism without initial engagement. Option 3 – 'Let's start by acknowledging the LLM's output and then gently raise concerns about performance and maintainability…' - demonstrates a collaborative approach. This allows you to understand *why* the LLM generated that response (perhaps it was optimized for specific edge cases) before offering alternative solutions, which is crucial when working with AI-generated code.
16 / 43
During a code review of a PR that uses an LLM to generate unit tests, the LLM produces this test case: `assert(calculateDiscount(100, 'VIP') === 90);`. The reviewer suspects the LLM might be oversimplifying the discount logic. Which approach is MOST effective for quickly verifying the test's thoroughness and potential gaps in coverage without immediately challenging the LLM's output?
Running the test in a CI environment with diverse data sets provides empirical evidence of the test's effectiveness and reveals hidden issues. Directly prompting the LLM for edge case tests can be helpful but doesn't directly validate the existing test. Flagging it immediately without investigation risks prematurely dismissing potentially valuable output. Crafting a similar, boundary-focused test offers a targeted comparison, allowing for a more nuanced assessment of coverage than simply relying on the original LLM-generated test.
17 / 43
The key here is recognizing that an LLM-generated summary, even if seemingly positive, often lacks crucial supporting evidence. Simply stating 'customers loved' isn't sufficient; you need data to back it up. Insufficient correctly identifies this gap – the response needs metrics (e.g., satisfaction scores) or direct quotes to validate the claims. Options B and C are overly optimistic without verification, and option D highlights a critical omission: failing to acknowledge potential issues.
18 / 43
You're reviewing a pull request where an LLM generated the following code snippet for handling user authentication:
```javascript
const isValidUser = (username, password) => {
// Placeholder for actual authentication logic - replace with your database query
if (username === 'admin' && password === 'password') {
return true;
} else {
return false;
}
};
```
During the review, you notice this is a very basic implementation and lacks any security measures like salting or hashing passwords. A colleague suggests, "Let's prompt the LLM to add password hashing using bcrypt."
Which of the following actions represents the MOST effective way to refine the LLM's response and ensure a secure authentication process?
The best approach is option 1 – prompting the LLM to incorporate `bcrypt`. This directly addresses the identified weakness (lack of security measures) and guides the LLM towards a more secure solution. Simply requesting a complete flow might lead to an overly complex response, while ignoring the LLM's output or creating a JSON schema doesn't solve the fundamental issue of insecure password storage. Focusing on a specific implementation detail like `bcrypt` is the most targeted and effective way to steer the AI towards a better outcome in this scenario.
19 / 43
Sarah: "Hey team, I've used the LLM to generate some SQL queries for our user data analysis. It returned this query: `SELECT * FROM users WHERE last_login > DATE('now', '-7 days') AND city = 'New York';`. Seems straightforward enough."The question presents a realistic scenario where a developer shares an LLM-generated query. While the basic structure looks correct – filtering by recent logins and location – it's crucial to evaluate whether the response fully addresses the requirements and potential issues. Simply stating 'seems straightforward' isn't enough; a good code review would identify areas for improvement such as performance optimization or security considerations. The correct answer acknowledges that the query is well-formed in its basic intent, demonstrating an understanding of what constitutes a minimally acceptable output from an LLM.
20 / 43
Reviewer: 'This PR introduces a complex lambda function within the `calculateTaxRate` method. The LLM suggested this implementation to handle edge cases related to sales tax calculations. While technically correct, it's significantly more verbose than a simple if/else statement and could impact performance. I'm concerned about maintainability and potential future scaling issues. You (as the tech lead) need to communicate this back to the developer. Which of the following is the MOST effective way to phrase your feedback in a Slack message to the developer?The key here is to frame the feedback constructively. Option A dismisses the LLM's suggestion without understanding its rationale, potentially frustrating the developer. Option B is overly demanding and doesn't foster collaboration. Option D jumps straight to criticism without initial engagement. Option 3 – 'Let's start by acknowledging the LLM's output and then gently raise concerns about performance and maintainability…' - demonstrates a collaborative approach. This allows you to understand *why* the LLM generated that response (perhaps it was optimized for specific edge cases) before offering alternative solutions, which is crucial when working with AI-generated code.
21 / 43
During a code review of a PR that uses an LLM to generate unit tests, the LLM produces this test case: `assert(calculateDiscount(100, 'VIP') === 90);`. The reviewer suspects the LLM might be oversimplifying the discount logic. Which approach is MOST effective for quickly verifying the test's thoroughness and potential gaps in coverage without immediately challenging the LLM's output?
Running the test in a CI environment with diverse data sets provides empirical evidence of the test's effectiveness and reveals hidden issues. Directly prompting the LLM for edge case tests can be helpful but doesn't directly validate the existing test. Flagging it immediately without investigation risks prematurely dismissing potentially valuable output. Crafting a similar, boundary-focused test offers a targeted comparison, allowing for a more nuanced assessment of coverage than simply relying on the original LLM-generated test.
22 / 43
The key here is recognizing that an LLM-generated summary, even if seemingly positive, often lacks crucial supporting evidence. Simply stating 'customers loved' isn't sufficient; you need data to back it up. Insufficient correctly identifies this gap – the response needs metrics (e.g., satisfaction scores) or direct quotes to validate the claims. Options B and C are overly optimistic without verification, and option D highlights a critical omission: failing to acknowledge potential issues.
23 / 43
You're reviewing a pull request where an LLM generated the following code snippet for handling user authentication:
```javascript
const isValidUser = (username, password) => {
// Placeholder for actual authentication logic - replace with your database query
if (username === 'admin' && password === 'password') {
return true;
} else {
return false;
}
};
```
During the review, you notice this is a very basic implementation and lacks any security measures like salting or hashing passwords. A colleague suggests, "Let's prompt the LLM to add password hashing using bcrypt."
Which of the following actions represents the MOST effective way to refine the LLM's response and ensure a secure authentication process?
The best approach is option 1 – prompting the LLM to incorporate `bcrypt`. This directly addresses the identified weakness (lack of security measures) and guides the LLM towards a more secure solution. Simply requesting a complete flow might lead to an overly complex response, while ignoring the LLM's output or creating a JSON schema doesn't solve the fundamental issue of insecure password storage. Focusing on a specific implementation detail like `bcrypt` is the most targeted and effective way to steer the AI towards a better outcome in this scenario.
24 / 43
Sarah: "Hey team, I've used the LLM to generate some SQL queries for our user data analysis. It returned this query: `SELECT * FROM users WHERE last_login > DATE('now', '-7 days') AND city = 'New York';`. Seems straightforward enough."The question presents a realistic scenario where a developer shares an LLM-generated query. While the basic structure looks correct – filtering by recent logins and location – it's crucial to evaluate whether the response fully addresses the requirements and potential issues. Simply stating 'seems straightforward' isn't enough; a good code review would identify areas for improvement such as performance optimization or security considerations. The correct answer acknowledges that the query is well-formed in its basic intent, demonstrating an understanding of what constitutes a minimally acceptable output from an LLM.
25 / 43
Reviewer: 'This PR introduces a complex lambda function within the `calculateTaxRate` method. The LLM suggested this implementation to handle edge cases related to sales tax calculations. While technically correct, it's significantly more verbose than a simple if/else statement and could impact performance. I'm concerned about maintainability and potential future scaling issues. You (as the tech lead) need to communicate this back to the developer. Which of the following is the MOST effective way to phrase your feedback in a Slack message to the developer?The key here is to frame the feedback constructively. Option A dismisses the LLM's suggestion without understanding its rationale, potentially frustrating the developer. Option B is overly demanding and doesn't foster collaboration. Option D jumps straight to criticism without initial engagement. Option 3 – 'Let's start by acknowledging the LLM's output and then gently raise concerns about performance and maintainability…' - demonstrates a collaborative approach. This allows you to understand *why* the LLM generated that response (perhaps it was optimized for specific edge cases) before offering alternative solutions, which is crucial when working with AI-generated code.
26 / 43
During a code review of a PR that uses an LLM to generate unit tests, the LLM produces this test case: `assert(calculateDiscount(100, 'VIP') === 90);`. The reviewer suspects the LLM might be oversimplifying the discount logic. Which approach is MOST effective for quickly verifying the test's thoroughness and potential gaps in coverage without immediately challenging the LLM's output?
Running the test in a CI environment with diverse data sets provides empirical evidence of the test's effectiveness and reveals hidden issues. Directly prompting the LLM for edge case tests can be helpful but doesn't directly validate the existing test. Flagging it immediately without investigation risks prematurely dismissing potentially valuable output. Crafting a similar, boundary-focused test offers a targeted comparison, allowing for a more nuanced assessment of coverage than simply relying on the original LLM-generated test.
27 / 43
The key here is recognizing that an LLM-generated summary, even if seemingly positive, often lacks crucial supporting evidence. Simply stating 'customers loved' isn't sufficient; you need data to back it up. Insufficient correctly identifies this gap – the response needs metrics (e.g., satisfaction scores) or direct quotes to validate the claims. Options B and C are overly optimistic without verification, and option D highlights a critical omission: failing to acknowledge potential issues.
28 / 43
You're reviewing a pull request where an LLM generated the following code snippet for handling user authentication:
```javascript
const isValidUser = (username, password) => {
// Placeholder for actual authentication logic - replace with your database query
if (username === 'admin' && password === 'password') {
return true;
} else {
return false;
}
};
```
During the review, you notice this is a very basic implementation and lacks any security measures like salting or hashing passwords. A colleague suggests, "Let's prompt the LLM to add password hashing using bcrypt."
Which of the following actions represents the MOST effective way to refine the LLM's response and ensure a secure authentication process?
The best approach is option 1 – prompting the LLM to incorporate `bcrypt`. This directly addresses the identified weakness (lack of security measures) and guides the LLM towards a more secure solution. Simply requesting a complete flow might lead to an overly complex response, while ignoring the LLM's output or creating a JSON schema doesn't solve the fundamental issue of insecure password storage. Focusing on a specific implementation detail like `bcrypt` is the most targeted and effective way to steer the AI towards a better outcome in this scenario.
29 / 43
Sarah: "Hey team, I've used the LLM to generate some SQL queries for our user data analysis. It returned this query: `SELECT * FROM users WHERE last_login > DATE('now', '-7 days') AND city = 'New York';`. Seems straightforward enough."The question presents a realistic scenario where a developer shares an LLM-generated query. While the basic structure looks correct – filtering by recent logins and location – it's crucial to evaluate whether the response fully addresses the requirements and potential issues. Simply stating 'seems straightforward' isn't enough; a good code review would identify areas for improvement such as performance optimization or security considerations. The correct answer acknowledges that the query is well-formed in its basic intent, demonstrating an understanding of what constitutes a minimally acceptable output from an LLM.
30 / 43
Reviewer: 'This PR introduces a complex lambda function within the `calculateTaxRate` method. The LLM suggested this implementation to handle edge cases related to sales tax calculations. While technically correct, it's significantly more verbose than a simple if/else statement and could impact performance. I'm concerned about maintainability and potential future scaling issues. You (as the tech lead) need to communicate this back to the developer. Which of the following is the MOST effective way to phrase your feedback in a Slack message to the developer?The key here is to frame the feedback constructively. Option A dismisses the LLM's suggestion without understanding its rationale, potentially frustrating the developer. Option B is overly demanding and doesn't foster collaboration. Option D jumps straight to criticism without initial engagement. Option 3 – 'Let's start by acknowledging the LLM's output and then gently raise concerns about performance and maintainability…' - demonstrates a collaborative approach. This allows you to understand *why* the LLM generated that response (perhaps it was optimized for specific edge cases) before offering alternative solutions, which is crucial when working with AI-generated code.
31 / 43
During a code review of a PR that uses an LLM to generate unit tests, the LLM produces this test case: `assert(calculateDiscount(100, 'VIP') === 90);`. The reviewer suspects the LLM might be oversimplifying the discount logic. Which approach is MOST effective for quickly verifying the test's thoroughness and potential gaps in coverage without immediately challenging the LLM's output?
Running the test in a CI environment with diverse data sets provides empirical evidence of the test's effectiveness and reveals hidden issues. Directly prompting the LLM for edge case tests can be helpful but doesn't directly validate the existing test. Flagging it immediately without investigation risks prematurely dismissing potentially valuable output. Crafting a similar, boundary-focused test offers a targeted comparison, allowing for a more nuanced assessment of coverage than simply relying on the original LLM-generated test.
32 / 43
The key here is recognizing that an LLM-generated summary, even if seemingly positive, often lacks crucial supporting evidence. Simply stating 'customers loved' isn't sufficient; you need data to back it up. Insufficient correctly identifies this gap – the response needs metrics (e.g., satisfaction scores) or direct quotes to validate the claims. Options B and C are overly optimistic without verification, and option D highlights a critical omission: failing to acknowledge potential issues.
33 / 43
You're reviewing a pull request where an LLM generated the following code snippet for handling user authentication:
```javascript
const isValidUser = (username, password) => {
// Placeholder for actual authentication logic - replace with your database query
if (username === 'admin' && password === 'password') {
return true;
} else {
return false;
}
};
```
During the review, you notice this is a very basic implementation and lacks any security measures like salting or hashing passwords. A colleague suggests, "Let's prompt the LLM to add password hashing using bcrypt."
Which of the following actions represents the MOST effective way to refine the LLM's response and ensure a secure authentication process?
The best approach is option 1 – prompting the LLM to incorporate `bcrypt`. This directly addresses the identified weakness (lack of security measures) and guides the LLM towards a more secure solution. Simply requesting a complete flow might lead to an overly complex response, while ignoring the LLM's output or creating a JSON schema doesn't solve the fundamental issue of insecure password storage. Focusing on a specific implementation detail like `bcrypt` is the most targeted and effective way to steer the AI towards a better outcome in this scenario.
34 / 43
Sarah: "Hey team, I've used the LLM to generate some SQL queries for our user data analysis. It returned this query: `SELECT * FROM users WHERE last_login > DATE('now', '-7 days') AND city = 'New York';`. Seems straightforward enough."The question presents a realistic scenario where a developer shares an LLM-generated query. While the basic structure looks correct – filtering by recent logins and location – it's crucial to evaluate whether the response fully addresses the requirements and potential issues. Simply stating 'seems straightforward' isn't enough; a good code review would identify areas for improvement such as performance optimization or security considerations. The correct answer acknowledges that the query is well-formed in its basic intent, demonstrating an understanding of what constitutes a minimally acceptable output from an LLM.
35 / 43
Reviewer: 'This PR introduces a complex lambda function within the `calculateTaxRate` method. The LLM suggested this implementation to handle edge cases related to sales tax calculations. While technically correct, it's significantly more verbose than a simple if/else statement and could impact performance. I'm concerned about maintainability and potential future scaling issues. You (as the tech lead) need to communicate this back to the developer. Which of the following is the MOST effective way to phrase your feedback in a Slack message to the developer?The key here is to frame the feedback constructively. Option A dismisses the LLM's suggestion without understanding its rationale, potentially frustrating the developer. Option B is overly demanding and doesn't foster collaboration. Option D jumps straight to criticism without initial engagement. Option 3 – 'Let's start by acknowledging the LLM's output and then gently raise concerns about performance and maintainability…' - demonstrates a collaborative approach. This allows you to understand *why* the LLM generated that response (perhaps it was optimized for specific edge cases) before offering alternative solutions, which is crucial when working with AI-generated code.
36 / 43
During a code review of a PR that uses an LLM to generate unit tests, the LLM produces this test case: `assert(calculateDiscount(100, 'VIP') === 90);`. The reviewer suspects the LLM might be oversimplifying the discount logic. Which approach is MOST effective for quickly verifying the test's thoroughness and potential gaps in coverage without immediately challenging the LLM's output?
Running the test in a CI environment with diverse data sets provides empirical evidence of the test's effectiveness and reveals hidden issues. Directly prompting the LLM for edge case tests can be helpful but doesn't directly validate the existing test. Flagging it immediately without investigation risks prematurely dismissing potentially valuable output. Crafting a similar, boundary-focused test offers a targeted comparison, allowing for a more nuanced assessment of coverage than simply relying on the original LLM-generated test.
37 / 43
The key here is recognizing that an LLM-generated summary, even if seemingly positive, often lacks crucial supporting evidence. Simply stating 'customers loved' isn't sufficient; you need data to back it up. Insufficient correctly identifies this gap – the response needs metrics (e.g., satisfaction scores) or direct quotes to validate the claims. Options B and C are overly optimistic without verification, and option D highlights a critical omission: failing to acknowledge potential issues.
38 / 43
You're reviewing a pull request where an LLM generated the following code snippet for handling user authentication:
```javascript
const isValidUser = (username, password) => {
// Placeholder for actual authentication logic - replace with your database query
if (username === 'admin' && password === 'password') {
return true;
} else {
return false;
}
};
```
During the review, you notice this is a very basic implementation and lacks any security measures like salting or hashing passwords. A colleague suggests, "Let's prompt the LLM to add password hashing using bcrypt."
Which of the following actions represents the MOST effective way to refine the LLM's response and ensure a secure authentication process?
The best approach is option 1 – prompting the LLM to incorporate `bcrypt`. This directly addresses the identified weakness (lack of security measures) and guides the LLM towards a more secure solution. Simply requesting a complete flow might lead to an overly complex response, while ignoring the LLM's output or creating a JSON schema doesn't solve the fundamental issue of insecure password storage. Focusing on a specific implementation detail like `bcrypt` is the most targeted and effective way to steer the AI towards a better outcome in this scenario.
39 / 43
Sarah: "Hey team, I've used the LLM to generate some SQL queries for our user data analysis. It returned this query: `SELECT * FROM users WHERE last_login > DATE('now', '-7 days') AND city = 'New York';`. Seems straightforward enough."The question presents a realistic scenario where a developer shares an LLM-generated query. While the basic structure looks correct – filtering by recent logins and location – it's crucial to evaluate whether the response fully addresses the requirements and potential issues. Simply stating 'seems straightforward' isn't enough; a good code review would identify areas for improvement such as performance optimization or security considerations. The correct answer acknowledges that the query is well-formed in its basic intent, demonstrating an understanding of what constitutes a minimally acceptable output from an LLM.
40 / 43
Reviewer: 'This PR introduces a complex lambda function within the `calculateTaxRate` method. The LLM suggested this implementation to handle edge cases related to sales tax calculations. While technically correct, it's significantly more verbose than a simple if/else statement and could impact performance. I'm concerned about maintainability and potential future scaling issues. You (as the tech lead) need to communicate this back to the developer. Which of the following is the MOST effective way to phrase your feedback in a Slack message to the developer?The key here is to frame the feedback constructively. Option A dismisses the LLM's suggestion without understanding its rationale, potentially frustrating the developer. Option B is overly demanding and doesn't foster collaboration. Option D jumps straight to criticism without initial engagement. Option 3 – 'Let's start by acknowledging the LLM's output and then gently raise concerns about performance and maintainability…' - demonstrates a collaborative approach. This allows you to understand *why* the LLM generated that response (perhaps it was optimized for specific edge cases) before offering alternative solutions, which is crucial when working with AI-generated code.
41 / 43
During a code review of a PR that uses an LLM to generate unit tests, the LLM produces this test case: `assert(calculateDiscount(100, 'VIP') === 90);`. The reviewer suspects the LLM might be oversimplifying the discount logic. Which approach is MOST effective for quickly verifying the test's thoroughness and potential gaps in coverage without immediately challenging the LLM's output?
Running the test in a CI environment with diverse data sets provides empirical evidence of the test's effectiveness and reveals hidden issues. Directly prompting the LLM for edge case tests can be helpful but doesn't directly validate the existing test. Flagging it immediately without investigation risks prematurely dismissing potentially valuable output. Crafting a similar, boundary-focused test offers a targeted comparison, allowing for a more nuanced assessment of coverage than simply relying on the original LLM-generated test.
42 / 43
The key here is recognizing that an LLM-generated summary, even if seemingly positive, often lacks crucial supporting evidence. Simply stating 'customers loved' isn't sufficient; you need data to back it up. Insufficient correctly identifies this gap – the response needs metrics (e.g., satisfaction scores) or direct quotes to validate the claims. Options B and C are overly optimistic without verification, and option D highlights a critical omission: failing to acknowledge potential issues.
43 / 43
You're reviewing a pull request where an LLM generated the following code snippet for handling user authentication:
```javascript
const isValidUser = (username, password) => {
// Placeholder for actual authentication logic - replace with your database query
if (username === 'admin' && password === 'password') {
return true;
} else {
return false;
}
};
```
During the review, you notice this is a very basic implementation and lacks any security measures like salting or hashing passwords. A colleague suggests, "Let's prompt the LLM to add password hashing using bcrypt."
Which of the following actions represents the MOST effective way to refine the LLM's response and ensure a secure authentication process?
The best approach is option 1 – prompting the LLM to incorporate `bcrypt`. This directly addresses the identified weakness (lack of security measures) and guides the LLM towards a more secure solution. Simply requesting a complete flow might lead to an overly complex response, while ignoring the LLM's output or creating a JSON schema doesn't solve the fundamental issue of insecure password storage. Focusing on a specific implementation detail like `bcrypt` is the most targeted and effective way to steer the AI towards a better outcome in this scenario.
Next up: AI Vocabulary →
Frequently Asked Questions
What will I practice in "Evaluating LLM Outputs — AI Prompting English Exercise"?
This is an AI Prompting exercise set. It walks through 43 scenario-based multiple-choice questions built around real usage of AI Prompting terminology that IT professionals encounter on the job.
Is this exercise free to use?
Yes. Every exercise on CoderSlingo, including this one, is free to complete with no account, sign-up, or paywall.
How many questions are in this exercise?
This set contains 43 questions. Each one shows immediate feedback and a detailed explanation after you answer, so you learn the correct usage right away rather than waiting for a final score.
Do I need prior experience to complete this exercise?
No prior experience is required. Each question includes a full explanation covering the reasoning behind the correct answer, so the exercise itself teaches the AI Prompting vocabulary as you go.
Can I retry the exercise if I get questions wrong?
Yes — use the "Try again" button on the results screen to reset your answers and go through all the questions again. There is no limit on attempts.
Is my progress saved?
Your answers and score for the current session are tracked in the browser as you go. No account or login is needed, and there is nothing to install.
What if I don't understand a term used in a question?
Read the explanation shown after you answer each question — it breaks down the correct term in plain English with a real-world example. You can also check the site Glossary for quick definitions.
How is this different from reading a blog article on the topic?
Exercises like this one are interactive drills that test and reinforce specific vocabulary through multiple-choice questions, while blog articles explain concepts in prose. Practising here after reading builds active recall, not just passive recognition.
Where can I find more AI Prompting exercises?
See the AI Prompting exercises hub for the full set of related pages, or browse all exercise categories from the main Exercises index.
Can I use this exercise to prepare for a technical interview?
Yes — AI Prompting vocabulary comes up often in technical discussions and interviews. Pair this exercise with our dedicated Interview Preparation section for role-specific practice.