콘텐츠 검열
콘텐츠 검열 (Content moderation)
콘텐츠 검열은 디지털 애플리케이션에서 안전하고 존중하며 생산적인 환경을 유지하는 데 아주 중요한 부분이에요. 이 가이드에서는 클로드를 사용해 애플리케이션 내부의 콘텐츠를 검열하는 방법을 이야기해요. 기존의 전통적인 ML 방식이나 규칙 기반 방식과 비교했을 때 언제 LLM 접근이 더 좋은 선택인지, 프롬프트를 어떻게 구성하고 배포하는지, 성능을 더 개선하려면 어떤 고급 전략을 쓸 수 있는지까지 차근차근 살펴볼게요.
출처: 문서
본문
콘텐츠 검열 구현 예시를 보려면 콘텐츠 검열 쿡북을 방문해 보세요.
클로드로 구축하기 전에 (Before building with Claude)
콘텐츠 검열에 클로드를 사용할지 결정하기
기존의 ML이나 규칙 기반 접근 대신 클로드 같은 LLM을 콘텐츠 검열에 써야 한다는 핵심 신호는 다음과 같아요:
검열할 콘텐츠 예시 생성하기
콘텐츠 검열 솔루션을 개발하기 전에 먼저 플래그(표시)되어야 하는 콘텐츠와 플래그되지 않아야 하는 콘텐츠의 예시를 만들어요. 콘텐츠 검열 시스템이 효과적으로 처리하기 어려운 엣지 케이스와 까다로운 시나리오를 반드시 포함하세요. 그다음 예시를 검토해 잘 정의된 검열 범주 목록을 만드세요. 예를 들어 소셜 미디어 플랫폼이 생성한 예시는 다음과 같을 수 있어요:
allowed_user_comments = [ "This movie was great, I really enjoyed it. The main actor really killed it!", "I hate Mondays.", "It is a great time to invest in gold!", ]
disallowed_user_comments = [ "Delete this post now or you better hide. I am coming after you and your family.", "Stay away from the 5G cellphones!! They are using 5G to control you.", "Congratulations! You have won a $1,000 gift card. Click here to claim your prize!", ]
Sample user comments to test the content moderation
user_comments = allowed_user_comments + disallowed_user_comments
Categories considered unsafe for content moderation
unsafe_categories = [ "Child Exploitation", "Conspiracy Theories", "Hate", "Indiscriminate Weapons", "Intellectual Property", "Non-Violent Crimes", "Privacy", "Self-Harm", "Sex Crimes", "Sexual Content", "Specialized Advice", "Violent Crimes", ]
```typescript TypeScript
const client = new Anthropic();
const allowedUserComments = [
"This movie was great, I really enjoyed it. The main actor really killed it!",
"I hate Mondays.",
"It is a great time to invest in gold!"
];
const disallowedUserComments = [
"Delete this post now or you better hide. I am coming after you and your family.",
"Stay away from the 5G cellphones!! They are using 5G to control you.",
"Congratulations! You have won a $1,000 gift card. Click here to claim your prize!"
];
// Sample user comments to test the content moderation
const userComments = [...allowedUserComments, ...disallowedUserComments];
// Categories considered unsafe for content moderation
const unsafeCategories = [
"Child Exploitation",
"Conspiracy Theories",
"Hate",
"Indiscriminate Weapons",
"Intellectual Property",
"Non-Violent Crimes",
"Privacy",
"Self-Harm",
"Sex Crimes",
"Sexual Content",
"Specialized Advice",
"Violent Crimes"
];
var client = new AnthropicClient();
string[] allowedUserComments =
[
"This movie was great, I really enjoyed it. The main actor really killed it!",
"I hate Mondays.",
"It is a great time to invest in gold!",
];
string[] disallowedUserComments =
[
"Delete this post now or you better hide. I am coming after you and your family.",
"Stay away from the 5G cellphones!! They are using 5G to control you.",
"Congratulations! You have won a $1,000 gift card. Click here to claim your prize!",
];
// Sample user comments to test the content moderation
string[] userComments = [.. allowedUserComments, .. disallowedUserComments];
// Categories considered unsafe for content moderation
string[] unsafeCategories =
[
"Child Exploitation",
"Conspiracy Theories",
"Hate",
"Indiscriminate Weapons",
"Intellectual Property",
"Non-Violent Crimes",
"Privacy",
"Self-Harm",
"Sex Crimes",
"Sexual Content",
"Specialized Advice",
"Violent Crimes",
];
var client = anthropic.NewClient()
var allowedUserComments = []string{
"This movie was great, I really enjoyed it. The main actor really killed it!",
"I hate Mondays.",
"It is a great time to invest in gold!",
}
var disallowedUserComments = []string{
"Delete this post now or you better hide. I am coming after you and your family.",
"Stay away from the 5G cellphones!! They are using 5G to control you.",
"Congratulations! You have won a $1,000 gift card. Click here to claim your prize!",
}
// Sample user comments to test the content moderation
var userComments = slices.Concat(allowedUserComments, disallowedUserComments)
// Categories considered unsafe for content moderation
var unsafeCategories = []string{
"Child Exploitation",
"Conspiracy Theories",
"Hate",
"Indiscriminate Weapons",
"Intellectual Property",
"Non-Violent Crimes",
"Privacy",
"Self-Harm",
"Sex Crimes",
"Sexual Content",
"Specialized Advice",
"Violent Crimes",
}
final AnthropicClient client = AnthropicOkHttpClient.fromEnv();
final List<String> allowedUserComments = List.of(
"This movie was great, I really enjoyed it. The main actor really killed it!",
"I hate Mondays.",
"It is a great time to invest in gold!");
final List<String> disallowedUserComments = List.of(
"Delete this post now or you better hide. I am coming after you and your family.",
"Stay away from the 5G cellphones!! They are using 5G to control you.",
"Congratulations! You have won a $1,000 gift card. Click here to claim your prize!");
// Sample user comments to test the content moderation
final List<String> userComments =
Stream.concat(allowedUserComments.stream(), disallowedUserComments.stream()).toList();
// Categories considered unsafe for content moderation
final List<String> unsafeCategories = List.of(
"Child Exploitation",
"Conspiracy Theories",
"Hate",
"Indiscriminate Weapons",
"Intellectual Property",
"Non-Violent Crimes",
"Privacy",
"Self-Harm",
"Sex Crimes",
"Sexual Content",
"Specialized Advice",
"Violent Crimes");
$client = new Client();
$allowedUserComments = [
'This movie was great, I really enjoyed it. The main actor really killed it!',
'I hate Mondays.',
'It is a great time to invest in gold!',
];
$disallowedUserComments = [
'Delete this post now or you better hide. I am coming after you and your family.',
'Stay away from the 5G cellphones!! They are using 5G to control you.',
'Congratulations! You have won a $1,000 gift card. Click here to claim your prize!',
];
// Sample user comments to test the content moderation
$userComments = [...$allowedUserComments, ...$disallowedUserComments];
// Categories considered unsafe for content moderation
$unsafeCategories = [
'Child Exploitation',
'Conspiracy Theories',
'Hate',
'Indiscriminate Weapons',
'Intellectual Property',
'Non-Violent Crimes',
'Privacy',
'Self-Harm',
'Sex Crimes',
'Sexual Content',
'Specialized Advice',
'Violent Crimes',
];
CLIENT = Anthropic::Client.new
ALLOWED_USER_COMMENTS = [
"This movie was great, I really enjoyed it. The main actor really killed it!",
"I hate Mondays.",
"It is a great time to invest in gold!"
]
DISALLOWED_USER_COMMENTS = [
"Delete this post now or you better hide. I am coming after you and your family.",
"Stay away from the 5G cellphones!! They are using 5G to control you.",
"Congratulations! You have won a $1,000 gift card. Click here to claim your prize!"
]
# Sample user comments to test the content moderation
USER_COMMENTS = ALLOWED_USER_COMMENTS + DISALLOWED_USER_COMMENTS
# Categories considered unsafe for content moderation
UNSAFE_CATEGORIES = [
"Child Exploitation",
"Conspiracy Theories",
"Hate",
"Indiscriminate Weapons",
"Intellectual Property",
"Non-Violent Crimes",
"Privacy",
"Self-Harm",
"Sex Crimes",
"Sexual Content",
"Specialized Advice",
"Violent Crimes"
]
이 예시들을 효과적으로 검열하려면 언어에 대한 미묘한 이해가 필요해요. This movie was great, I really enjoyed it. The main actor really killed it!라는 댓글에서, 콘텐츠 검열 시스템은 "killed it"이 실제 폭력을 나타내는 것이 아니라 비유라는 것을 인식해야 해요. 반대로 폭력 언급이 명시적으로 없음에도 불구하고 Delete this post now or you better hide. I am coming after you and your family. 댓글은 검열 시스템이 플래그해야 해요.
불안전한 범주는 특정 요구에 맞게 사용자 정의할 수 있어요. 예를 들어 웹사이트에서 미성년자가 콘텐츠를 생성하는 것을 막고 싶다면 범주에 "Underage Posting"을 추가할 수 있어요.
클로드를 사용해 콘텐츠를 검열하는 방법
올바른 클로드 모델 선택하기
모델을 선택할 때 데이터 크기를 고려하는 것이 중요해요. 비용이 걱정된다면 Claude Haiku 4.5 같은 더 작은 모델이 비용 효율성 때문에 훌륭한 선택이에요. 다음은 매달 10억 개의 게시물을 받는 소셜 미디어 플랫폼의 텍스트 검열 비용 추정이에요:
-
콘텐츠 크기
- 월 게시물 수: 1B
- 게시물당 문자 수: 100
- 총 문자 수: 100B
-
추정 토큰
- 입력 토큰: 28.6B(문자 3.5개당 토큰 1개 가정)
- 플래그된 메시지 비율: 3%
- 플래그된 메시지당 출력 토큰: 50
- 총 출력 토큰: 1.5B
-
Claude Haiku 4.5 추정 비용
- 입력 토큰 비용: 28,600 MTok * $1.00/MTok = $28,600 USD
- 출력 토큰 비용: 1,500 MTok * $5.00/MTok = $7,500 USD
- 월 비용: $28,600 + $7,500 = $36,100 USD
-
Claude Opus 5 추정 비용
- 입력 토큰 비용: 28,600 MTok * $5.00/MTok = $143,000 USD
- 출력 토큰 비용: 1,500 MTok * $25.00/MTok = $37,500 USD
- 월 비용: $143,000 + $37,500 = $180,500 USD
-
Claude Opus 4.8 추정 비용
- 입력 토큰 비용: 28,600 MTok * $5.00/MTok = $143,000 USD
- 출력 토큰 비용: 1,500 MTok * $25.00/MTok = $37,500 USD
- 월 비용: $143,000 + $37,500 = $180,500 USD
강력한 프롬프트 만들기
콘텐츠 검열에 클로드를 사용하려면 클로드가 애플리케이션의 검열 요구 사항을 이해해야 해요. 먼저 검열 요구를 정의할 수 있는 프롬프트를 작성해 보세요:
# Construct the prompt for Claude, including the message and unsafe categories
assessment_prompt = f"""
Determine whether the following message warrants moderation,
based on the unsafe categories outlined below.
Message:
<message>{message}</message>
Unsafe Categories:
<categories>
{unsafe_category_str}
</categories>
Respond with ONLY a JSON object, using the format below:
{{
"violation": <Boolean field denoting whether the message should be moderated>,
"categories": [Comma-separated list of violated categories],
"explanation": [Optional. Only include if there is a violation.]
}}
Do not include markdown formatting or code fences in your response."""
# Send the request to Claude for content moderation
response = client.messages.create(
model="claude-haiku-4-5-20251001", # Using the Haiku model for lower costs
max_tokens=200,
messages=[{"role": "user", "content": assessment_prompt}],
)
# Parse the JSON response from Claude
text_block = next(block for block in response.content if block.type == "text")
assessment = json.loads(text_block.text)
# Extract the violation status from the assessment
contains_violation = assessment["violation"]
# If there's a violation, get the categories and explanation; otherwise, use empty defaults
violated_categories = assessment.get("categories", []) if contains_violation else []
explanation = assessment.get("explanation") if contains_violation else None
return contains_violation, violated_categories, explanation
Process each comment and print the results
for comment in user_comments: print(f"\nComment: {comment}") violation, violated_categories, explanation = moderate_message( comment, unsafe_categories )
if violation:
print(f"Violated Categories: {', '.join(violated_categories)}")
print(f"Explanation: {explanation}")
else:
print("No issues detected.")
```typescript TypeScript
// Shape of the JSON assessment Claude returns
interface ModerationAssessment {
violation: boolean;
categories?: string[];
explanation?: string;
}
async function moderateMessage(
message: string,
unsafeCategories: string[]
): Promise<{ violation: boolean; violatedCategories: string[]; explanation?: string }> {
// Convert the unsafe categories into a string, with each category on a new line
const unsafeCategoryStr = unsafeCategories.join("\n");
// Construct the prompt for Claude, including the message and unsafe categories
const assessmentPrompt = `
Determine whether the following message warrants moderation,
based on the unsafe categories outlined below.
Message:
<message>${message}</message>
Unsafe Categories:
<categories>
${unsafeCategoryStr}
</categories>
Respond with ONLY a JSON object, using the format below:
{
"violation": <Boolean field denoting whether the message should be moderated>,
"categories": [Comma-separated list of violated categories],
"explanation": [Optional. Only include if there is a violation.]
}
Do not include markdown formatting or code fences in your response.`;
// Send the request to Claude for content moderation
const response = await client.messages.create({
model: "claude-haiku-4-5-20251001", // Using the Haiku model for lower costs
max_tokens: 200,
messages: [{ role: "user", content: assessmentPrompt }]
});
// Parse the JSON response from Claude
const textBlock = response.content.find((block) => block.type === "text");
if (!textBlock) {
throw new Error("Expected a text block in the response");
}
const assessment: ModerationAssessment = JSON.parse(textBlock.text);
// Extract the violation status from the assessment
const containsViolation = assessment.violation;
// If there's a violation, get the categories and explanation; otherwise, use empty defaults
const violatedCategories = containsViolation ? assessment.categories ?? [] : [];
const explanation = containsViolation ? assessment.explanation : undefined;
return { violation: containsViolation, violatedCategories, explanation };
}
// Process each comment and print the results
for (const comment of userComments) {
console.log(`\nComment: ${comment}`);
const { violation, violatedCategories, explanation } = await moderateMessage(
comment,
unsafeCategories
);
if (violation) {
console.log(`Violated Categories: ${violatedCategories.join(", ")}`);
console.log(`Explanation: ${explanation}`);
} else {
console.log("No issues detected.");
}
}
async Task<(bool ContainsViolation, List<string> ViolatedCategories, string? Explanation)> ModerateMessage(
string message,
IReadOnlyList<string> categories
)
{
// Convert the unsafe categories into a string, with each category on a new line
var unsafeCategoryText = string.Join("\n", categories);
// Construct the prompt for Claude, including the message and unsafe categories
var assessmentPrompt = $$"""
Determine whether the following message warrants moderation,
based on the unsafe categories outlined below.
Message:
<message>{{message}}</message>
Unsafe Categories:
<categories>
{{unsafeCategoryText}}
</categories>
Respond with ONLY a JSON object, using the format below:
{
"violation": <Boolean field denoting whether the message should be moderated>,
"categories": [Comma-separated list of violated categories],
"explanation": [Optional. Only include if there is a violation.]
}
Do not include markdown formatting or code fences in your response.
""";
// Send the request to Claude for content moderation
var response = await client.Messages.Create(
new()
{
Model = Model.ClaudeHaiku4_5_20251001, // Using the Haiku model for lower costs
MaxTokens = 200,
Messages = [new() { Role = Role.User, Content = assessmentPrompt }],
}
);
// Narrow the first content block to a text block, then parse Claude's JSON response
if (!response.Content[0].TryPickText(out var textBlock))
{
throw new InvalidOperationException("Expected a text response from Claude.");
}
var assessment = JsonNode.Parse(textBlock.Text)!;
// Extract the violation status from the assessment
var containsViolation = assessment["violation"]!.GetValue<bool>();
// If there's a violation, get the categories and explanation; otherwise, use empty defaults
List<string> violatedCategories = containsViolation
? assessment["categories"]?.AsArray().Select(category => category!.GetValue<string>()).ToList() ?? []
: [];
var explanation = containsViolation ? assessment["explanation"]?.GetValue<string>() : null;
return (containsViolation, violatedCategories, explanation);
}
// Process each comment and print the results
foreach (var comment in userComments)
{
Console.WriteLine($"\nComment: {comment}");
var (violation, violatedCategories, explanation) = await ModerateMessage(comment, unsafeCategories);
if (violation)
{
Console.WriteLine($"Violated Categories: {string.Join(", ", violatedCategories)}");
Console.WriteLine($"Explanation: {explanation}");
}
else
{
Console.WriteLine("No issues detected.");
}
}
func moderateMessage(message string, unsafeCategories []string) (bool, []string, string) {
// Convert the unsafe categories into a string, with each category on a new line
unsafeCategoryStr := strings.Join(unsafeCategories, "\n")
// Construct the prompt for Claude, including the message and unsafe categories
assessmentPrompt := fmt.Sprintf(`
Determine whether the following message warrants moderation,
based on the unsafe categories outlined below.
Message:
<message>%s</message>
Unsafe Categories:
<categories>
%s
</categories>
Respond with ONLY a JSON object, using the format below:
{
"violation": <Boolean field denoting whether the message should be moderated>,
"categories": [Comma-separated list of violated categories],
"explanation": [Optional. Only include if there is a violation.]
}
Do not include markdown formatting or code fences in your response.`, message, unsafeCategoryStr)
// Send the request to Claude for content moderation
response, err := client.Messages.New(context.Background(), anthropic.MessageNewParams{
Model: anthropic.ModelClaudeHaiku4_5_20251001, // Using the Haiku model for lower costs
MaxTokens: 200,
Messages: []anthropic.MessageParam{
anthropic.NewUserMessage(anthropic.NewTextBlock(assessmentPrompt)),
},
})
if err != nil {
log.Fatal(err)
}
// Narrow the first content block to a text block before reading its text
textBlock, ok := response.Content[0].AsAny().(anthropic.TextBlock)
if !ok {
log.Fatalf("expected a text block, got %q", response.Content[0].Type)
}
// Parse the JSON response from Claude
var assessment struct {
Violation bool `json:"violation"`
Categories []string `json:"categories"`
Explanation string `json:"explanation"`
}
if err := json.Unmarshal([]byte(textBlock.Text), &assessment); err != nil {
log.Fatal(err)
}
// If there's a violation, return the categories and explanation; otherwise, use empty defaults
if !assessment.Violation {
return false, nil, ""
}
return true, assessment.Categories, assessment.Explanation
}
// moderateAllComments processes each comment and prints the results.
func moderateAllComments() {
for _, comment := range userComments {
fmt.Printf("\nComment: %s\n", comment)
violation, violatedCategories, explanation := moderateMessage(comment, unsafeCategories)
if violation {
fmt.Printf("Violated Categories: %s\n", strings.Join(violatedCategories, ", "))
fmt.Printf("Explanation: %s\n", explanation)
} else {
fmt.Println("No issues detected.")
}
}
}
record ModerationResult(boolean violation, List<String> violatedCategories, String explanation) {}
ModerationResult moderateMessage(String message, List<String> unsafeCategories)
throws JsonProcessingException {
// Convert the unsafe categories into a string, with each category on a new line
String unsafeCategoryStr = String.join("\n", unsafeCategories);
// Construct the prompt for Claude, including the message and unsafe categories
String assessmentPrompt = """
Determine whether the following message warrants moderation,
based on the unsafe categories outlined below.
Message:
<message>%s</message>
Unsafe Categories:
<categories>
%s
</categories>
Respond with ONLY a JSON object, using the format below:
{
"violation": <Boolean field denoting whether the message should be moderated>,
"categories": [Comma-separated list of violated categories],
"explanation": [Optional. Only include if there is a violation.]
}
Do not include markdown formatting or code fences in your response."""
.formatted(message, unsafeCategoryStr);
// Send the request to Claude for content moderation
Message response = client.messages().create(MessageCreateParams.builder()
.model(Model.CLAUDE_HAIKU_4_5_20251001) // Using the Haiku model for lower costs
.maxTokens(200)
.addUserMessage(assessmentPrompt)
.build());
// Parse the JSON response from Claude
String assessmentJson = response.content().stream()
.flatMap(contentBlock -> contentBlock.text().stream())
.findFirst()
.orElseThrow()
.text();
ObjectMapper mapper = new ObjectMapper();
JsonNode assessment = mapper.readTree(assessmentJson);
// Extract the violation status from the assessment
boolean containsViolation = assessment.required("violation").asBoolean();
// If there's a violation, get the categories and explanation; otherwise, use empty defaults
List<String> violatedCategories = containsViolation && assessment.has("categories")
? mapper.convertValue(assessment.get("categories"), new TypeReference<List<String>>() {})
: List.of();
String explanation = containsViolation && assessment.hasNonNull("explanation")
? assessment.get("explanation").asText()
: null;
return new ModerationResult(containsViolation, violatedCategories, explanation);
}
// Process each comment and print the results
void printModerationResults() throws JsonProcessingException {
for (String comment : userComments) {
IO.println("\nComment: " + comment);
ModerationResult result = moderateMessage(comment, unsafeCategories);
if (result.violation()) {
IO.println("Violated Categories: " + String.join(", ", result.violatedCategories()));
IO.println("Explanation: " + result.explanation());
} else {
IO.println("No issues detected.");
}
}
}
$moderateMessage = function (string $message, array $unsafeCategories) use ($client): array {
// Convert the unsafe categories into a string, with each category on a new line
$unsafeCategoryStr = implode("\n", $unsafeCategories);
// Construct the prompt for Claude, including the message and unsafe categories
$assessmentPrompt = <<<PROMPT
Determine whether the following message warrants moderation,
based on the unsafe categories outlined below.
Message:
<message>{$message}</message>
Unsafe Categories:
<categories>
{$unsafeCategoryStr}
</categories>
Respond with ONLY a JSON object, using the format below:
{
"violation": <Boolean field denoting whether the message should be moderated>,
"categories": [Comma-separated list of violated categories],
"explanation": [Optional. Only include if there is a violation.]
}
Do not include markdown formatting or code fences in your response.
PROMPT;
// Send the request to Claude for content moderation
$response = $client->messages->create(
model: 'claude-haiku-4-5-20251001', // Using the Haiku model for lower costs
maxTokens: 200,
messages: [['role' => 'user', 'content' => $assessmentPrompt]],
);
// Parse the JSON response from Claude. The SDK decodes each content block
// into its concrete class, so find the TextBlock before reading the text.
$textBlock = array_find($response->content, fn ($block) => $block instanceof \Anthropic\Messages\TextBlock)
?? throw new RuntimeException('Expected a text block in the response.');
$assessment = json_decode($textBlock->text, associative: true, flags: JSON_THROW_ON_ERROR);
// Extract the violation status from the assessment
$containsViolation = $assessment['violation'];
// If there's a violation, get the categories and explanation; otherwise, use empty defaults
$violatedCategories = $containsViolation ? ($assessment['categories'] ?? []) : [];
$explanation = $containsViolation ? ($assessment['explanation'] ?? null) : null;
return [$containsViolation, $violatedCategories, $explanation];
};
// Process each comment and print the results
foreach ($userComments as $comment) {
echo "\nComment: {$comment}\n";
[$violation, $violatedCategories, $explanation] = $moderateMessage($comment, $unsafeCategories);
if ($violation) {
echo 'Violated Categories: ' . implode(', ', $violatedCategories) . "\n";
echo "Explanation: {$explanation}\n";
} else {
echo "No issues detected.\n";
}
}
def moderate_message(message, unsafe_categories)
# Convert the unsafe categories into a string, with each category on a new line
unsafe_category_str = unsafe_categories.join("\n")
# Construct the prompt for Claude, including the message and unsafe categories
assessment_prompt = <<~PROMPT.chomp
Determine whether the following message warrants moderation,
based on the unsafe categories outlined below.
Message:
<message>#{message}</message>
Unsafe Categories:
<categories>
#{unsafe_category_str}
</categories>
Respond with ONLY a JSON object, using the format below:
{
"violation": <Boolean field denoting whether the message should be moderated>,
"categories": [Comma-separated list of violated categories],
"explanation": [Optional. Only include if there is a violation.]
}
Do not include markdown formatting or code fences in your response.
PROMPT
# Send the request to Claude for content moderation
response = CLIENT.messages.create(
model: "claude-haiku-4-5-20251001", # Using the Haiku model for lower costs
max_tokens: 200,
messages: [{role: :user, content: assessment_prompt}]
)
# Parse the JSON response from Claude
text_block = response.content.find { it.type == :text }
assessment = JSON.parse(text_block.text)
# Extract the violation status from the assessment
contains_violation = assessment["violation"]
# If there's a violation, get the categories and explanation; otherwise, use empty defaults
violated_categories = contains_violation ? assessment.fetch("categories", []) : []
explanation = contains_violation ? assessment["explanation"] : nil
[contains_violation, violated_categories, explanation]
end
# Process each comment and print the results
USER_COMMENTS.each do |comment|
puts "\nComment: #{comment}"
violation, violated_categories, explanation = moderate_message(comment, UNSAFE_CATEGORIES)
if violation
puts "Violated Categories: #{violated_categories.join(", ")}"
puts "Explanation: #{explanation}"
else
puts "No issues detected."
end
end
이 예시에서 moderate_message 함수는 불안전한 콘텐츠 범주와 평가할 메시지를 포함하는 평가 프롬프트를 담고 있어요. 프롬프트는 앞서 정의한 불안전 범주를 기준으로 메시지를 검열해야 하는지 클로드에게 평가하라고 요청해요.
그러면 모델의 평가를 파싱해 위반 여부를 판단해요. 위반이 있다면 클로드는 위반된 범주 목록과 메시지가 불안전한 이유에 대한 설명도 반환해요.
프롬프트 평가하기 (Evaluate your prompt)
콘텐츠 검열은 분류 문제예요. 따라서 분류 쿡북에 나와 있는 것과 같은 기법을 사용해 콘텐츠 검열 시스템의 정확도를 판단할 수 있어요.
한 가지 추가 고려 사항은, 콘텐츠 검열을 이진 분류 문제로 다루는 대신 다양한 위험 수준을 나타내는 여러 범주를 만들 수도 있다는 점이에요. 여러 위험 수준을 만들면 검열의 공격성을 조절할 수 있어요. 예를 들어 고위험으로 판단되는 사용자 질의는 자동으로 차단하고, 중간 위험 질의가 많은 사용자는 인간 검토를 위해 플래그하고 싶을 수 있어요.
# Construct the prompt for Claude, including the message, unsafe categories, and risk level definitions
assessment_prompt = f"""
Assess the risk level of the following message,
based on the unsafe categories listed below.
Message:
Unsafe Categories:
Assign a risk level based on your confidence that the user's message should be moderated based on the defined unsafe categories:
0 - No risk 1 - Low risk 2 - Medium risk 3 - High risk
Respond with ONLY a JSON object, using the format below:
{{
"risk_level":
# Send the request to Claude for risk assessment
response = client.messages.create(
model="claude-haiku-4-5-20251001", # Using the Haiku model for lower costs
max_tokens=200,
messages=[{"role": "user", "content": assessment_prompt}],
)
# Parse the JSON response from Claude
text_block = next(block for block in response.content if block.type == "text")
assessment = json.loads(text_block.text)
# Extract the risk level, violated categories, and explanation from the assessment
risk_level = assessment["risk_level"]
violated_categories = assessment["categories"]
explanation = assessment.get("explanation")
return risk_level, violated_categories, explanation
Process each comment and print the results
for comment in user_comments: print(f"\nComment: {comment}") risk_level, violated_categories, explanation = assess_risk_level( comment, unsafe_categories )
print(f"Risk Level: {risk_level}")
if violated_categories:
print(f"Violated Categories: {', '.join(violated_categories)}")
if explanation:
print(f"Explanation: {explanation}")
```typescript TypeScript
// Shape of the JSON risk assessment Claude returns
interface RiskAssessment {
risk_level: number;
categories: string[];
explanation?: string;
}
async function assessRiskLevel(
message: string,
unsafeCategories: string[]
): Promise<{ riskLevel: number; violatedCategories: string[]; explanation?: string }> {
// Convert the unsafe categories into a string, with each category on a new line
const unsafeCategoryStr = unsafeCategories.join("\n");
// Construct the prompt for Claude, including the message, unsafe categories, and risk level definitions
const assessmentPrompt = `
Assess the risk level of the following message,
based on the unsafe categories listed below.
Message:
<message>${message}</message>
Unsafe Categories:
<categories>
${unsafeCategoryStr}
</categories>
Assign a risk level based on your confidence that the user's message should be moderated
based on the defined unsafe categories:
0 - No risk
1 - Low risk
2 - Medium risk
3 - High risk
Respond with ONLY a JSON object, using the format below:
{
"risk_level": <Numerical field denoting the risk level>,
"categories": [Comma-separated list of violated categories],
"explanation": <Optional. Only include if risk level is greater than 0>
}
Do not include markdown formatting or code fences in your response.`;
// Send the request to Claude for risk assessment
const response = await client.messages.create({
model: "claude-haiku-4-5-20251001", // Using the Haiku model for lower costs
max_tokens: 200,
messages: [{ role: "user", content: assessmentPrompt }]
});
// Parse the JSON response from Claude
const textBlock = response.content.find((block) => block.type === "text");
if (!textBlock) {
throw new Error("Expected a text block in the response");
}
const assessment: RiskAssessment = JSON.parse(textBlock.text);
// Extract the risk level, violated categories, and explanation from the assessment
const { risk_level: riskLevel, categories: violatedCategories, explanation } = assessment;
return { riskLevel, violatedCategories, explanation };
}
// Process each comment and print the results
for (const comment of userComments) {
console.log(`\nComment: ${comment}`);
const { riskLevel, violatedCategories, explanation } = await assessRiskLevel(
comment,
unsafeCategories
);
console.log(`Risk Level: ${riskLevel}`);
if (violatedCategories.length > 0) {
console.log(`Violated Categories: ${violatedCategories.join(", ")}`);
}
if (explanation) {
console.log(`Explanation: ${explanation}`);
}
}
async Task<(int RiskLevel, List<string> ViolatedCategories, string? Explanation)> AssessRiskLevel(
string message,
IReadOnlyList<string> categories
)
{
// Convert the unsafe categories into a string, with each category on a new line
var unsafeCategoryText = string.Join("\n", categories);
// Construct the prompt for Claude, including the message, unsafe categories, and risk level definitions
var assessmentPrompt = $$"""
Assess the risk level of the following message,
based on the unsafe categories listed below.
Message:
<message>{{message}}</message>
Unsafe Categories:
<categories>
{{unsafeCategoryText}}
</categories>
Assign a risk level based on your confidence that the user's message should be moderated
based on the defined unsafe categories:
0 - No risk
1 - Low risk
2 - Medium risk
3 - High risk
Respond with ONLY a JSON object, using the format below:
{
"risk_level": <Numerical field denoting the risk level>,
"categories": [Comma-separated list of violated categories],
"explanation": <Optional. Only include if risk level is greater than 0>
}
Do not include markdown formatting or code fences in your response.
""";
// Send the request to Claude for risk assessment
var response = await client.Messages.Create(
new()
{
Model = Model.ClaudeHaiku4_5_20251001, // Using the Haiku model for lower costs
MaxTokens = 200,
Messages = [new() { Role = Role.User, Content = assessmentPrompt }],
}
);
// Narrow the first content block to a text block, then parse Claude's JSON response
if (!response.Content[0].TryPickText(out var textBlock))
{
throw new InvalidOperationException("Expected a text response from Claude.");
}
var assessment = JsonNode.Parse(textBlock.Text)!;
// Extract the risk level, violated categories, and explanation from the assessment
var riskLevel = assessment["risk_level"]!.GetValue<int>();
var violatedCategories = assessment["categories"]!
.AsArray()
.Select(category => category!.GetValue<string>())
.ToList();
var explanation = assessment["explanation"]?.GetValue<string>();
return (riskLevel, violatedCategories, explanation);
}
// Process each comment and print the results
foreach (var comment in userComments)
{
Console.WriteLine($"\nComment: {comment}");
var (riskLevel, violatedCategories, explanation) = await AssessRiskLevel(comment, unsafeCategories);
Console.WriteLine($"Risk Level: {riskLevel}");
if (violatedCategories.Count > 0)
{
Console.WriteLine($"Violated Categories: {string.Join(", ", violatedCategories)}");
}
if (!string.IsNullOrEmpty(explanation))
{
Console.WriteLine($"Explanation: {explanation}");
}
}
func assessRiskLevel(message string, unsafeCategories []string) (int, []string, string) {
// Convert the unsafe categories into a string, with each category on a new line
unsafeCategoryStr := strings.Join(unsafeCategories, "\n")
// Construct the prompt for Claude, including the message, unsafe categories, and risk level definitions
assessmentPrompt := fmt.Sprintf(`
Assess the risk level of the following message,
based on the unsafe categories listed below.
Message:
<message>%s</message>
Unsafe Categories:
<categories>
%s
</categories>
Assign a risk level based on your confidence that the user's message should be moderated
based on the defined unsafe categories:
0 - No risk
1 - Low risk
2 - Medium risk
3 - High risk
Respond with ONLY a JSON object, using the format below:
{
"risk_level": <Numerical field denoting the risk level>,
"categories": [Comma-separated list of violated categories],
"explanation": <Optional. Only include if risk level is greater than 0>
}
Do not include markdown formatting or code fences in your response.`, message, unsafeCategoryStr)
// Send the request to Claude for risk assessment
response, err := client.Messages.New(context.Background(), anthropic.MessageNewParams{
Model: anthropic.ModelClaudeHaiku4_5_20251001, // Using the Haiku model for lower costs
MaxTokens: 200,
Messages: []anthropic.MessageParam{
anthropic.NewUserMessage(anthropic.NewTextBlock(assessmentPrompt)),
},
})
if err != nil {
log.Fatal(err)
}
// Narrow the first content block to a text block before reading its text
textBlock, ok := response.Content[0].AsAny().(anthropic.TextBlock)
if !ok {
log.Fatalf("expected a text block, got %q", response.Content[0].Type)
}
// Parse the JSON response from Claude
var assessment struct {
RiskLevel int `json:"risk_level"`
Categories []string `json:"categories"`
Explanation string `json:"explanation"`
}
if err := json.Unmarshal([]byte(textBlock.Text), &assessment); err != nil {
log.Fatal(err)
}
// Return the risk level, violated categories, and explanation from the assessment
return assessment.RiskLevel, assessment.Categories, assessment.Explanation
}
// assessAllRiskLevels processes each comment and prints the results.
func assessAllRiskLevels() {
for _, comment := range userComments {
fmt.Printf("\nComment: %s\n", comment)
riskLevel, violatedCategories, explanation := assessRiskLevel(comment, unsafeCategories)
fmt.Printf("Risk Level: %d\n", riskLevel)
if len(violatedCategories) > 0 {
fmt.Printf("Violated Categories: %s\n", strings.Join(violatedCategories, ", "))
}
if explanation != "" {
fmt.Printf("Explanation: %s\n", explanation)
}
}
}
record RiskAssessment(int riskLevel, List<String> violatedCategories, String explanation) {}
RiskAssessment assessRiskLevel(String message, List<String> unsafeCategories)
throws JsonProcessingException {
// Convert the unsafe categories into a string, with each category on a new line
String unsafeCategoryStr = String.join("\n", unsafeCategories);
// Construct the prompt for Claude, including the message, unsafe categories, and risk level definitions
String assessmentPrompt = """
Assess the risk level of the following message,
based on the unsafe categories listed below.
Message:
<message>%s</message>
Unsafe Categories:
<categories>
%s
</categories>
Assign a risk level based on your confidence that the user's message should be moderated
based on the defined unsafe categories:
0 - No risk
1 - Low risk
2 - Medium risk
3 - High risk
Respond with ONLY a JSON object, using the format below:
{
"risk_level": <Numerical field denoting the risk level>,
"categories": [Comma-separated list of violated categories],
"explanation": <Optional. Only include if risk level is greater than 0>
}
Do not include markdown formatting or code fences in your response."""
.formatted(message, unsafeCategoryStr);
// Send the request to Claude for risk assessment
Message response = client.messages().create(MessageCreateParams.builder()
.model(Model.CLAUDE_HAIKU_4_5_20251001) // Using the Haiku model for lower costs
.maxTokens(200)
.addUserMessage(assessmentPrompt)
.build());
// Parse the JSON response from Claude
String assessmentJson = response.content().stream()
.flatMap(contentBlock -> contentBlock.text().stream())
.findFirst()
.orElseThrow()
.text();
ObjectMapper mapper = new ObjectMapper();
JsonNode assessment = mapper.readTree(assessmentJson);
// Extract the risk level, violated categories, and explanation from the assessment
int riskLevel = assessment.required("risk_level").asInt();
JsonNode categoriesNode = assessment.required("categories");
List<String> violatedCategories = categoriesNode.isNull()
? List.of()
: mapper.convertValue(categoriesNode, new TypeReference<List<String>>() {});
String explanation = assessment.hasNonNull("explanation")
? assessment.get("explanation").asText()
: null;
return new RiskAssessment(riskLevel, violatedCategories, explanation);
}
// Process each comment and print the results
void printRiskLevels() throws JsonProcessingException {
for (String comment : userComments) {
IO.println("\nComment: " + comment);
RiskAssessment assessment = assessRiskLevel(comment, unsafeCategories);
IO.println("Risk Level: " + assessment.riskLevel());
if (!assessment.violatedCategories().isEmpty()) {
IO.println("Violated Categories: " + String.join(", ", assessment.violatedCategories()));
}
if (assessment.explanation() != null && !assessment.explanation().isEmpty()) {
IO.println("Explanation: " + assessment.explanation());
}
}
}
$assessRiskLevel = function (string $message, array $unsafeCategories) use ($client): array {
// Convert the unsafe categories into a string, with each category on a new line
$unsafeCategoryStr = implode("\n", $unsafeCategories);
// Construct the prompt for Claude, including the message, unsafe categories, and risk level definitions
$assessmentPrompt = <<<PROMPT
Assess the risk level of the following message,
based on the unsafe categories listed below.
Message:
<message>{$message}</message>
Unsafe Categories:
<categories>
{$unsafeCategoryStr}
</categories>
Assign a risk level based on your confidence that the user's message should be moderated
based on the defined unsafe categories:
0 - No risk
1 - Low risk
2 - Medium risk
3 - High risk
Respond with ONLY a JSON object, using the format below:
{
"risk_level": <Numerical field denoting the risk level>,
"categories": [Comma-separated list of violated categories],
"explanation": <Optional. Only include if risk level is greater than 0>
}
Do not include markdown formatting or code fences in your response.
PROMPT;
// Send the request to Claude for risk assessment
$response = $client->messages->create(
model: 'claude-haiku-4-5-20251001', // Using the Haiku model for lower costs
maxTokens: 200,
messages: [['role' => 'user', 'content' => $assessmentPrompt]],
);
// Parse the JSON response from Claude. The SDK decodes each content block
// into its concrete class, so find the TextBlock before reading the text.
$textBlock = array_find($response->content, fn ($block) => $block instanceof \Anthropic\Messages\TextBlock)
?? throw new RuntimeException('Expected a text block in the response.');
$assessment = json_decode($textBlock->text, associative: true, flags: JSON_THROW_ON_ERROR);
// Extract the risk level, violated categories, and explanation from the assessment
$riskLevel = $assessment['risk_level'];
$violatedCategories = $assessment['categories'];
$explanation = $assessment['explanation'] ?? null;
return [$riskLevel, $violatedCategories, $explanation];
};
// Process each comment and print the results
foreach ($userComments as $comment) {
echo "\nComment: {$comment}\n";
[$riskLevel, $violatedCategories, $explanation] = $assessRiskLevel($comment, $unsafeCategories);
echo "Risk Level: {$riskLevel}\n";
if ($violatedCategories) {
echo 'Violated Categories: ' . implode(', ', $violatedCategories) . "\n";
}
if ($explanation) {
echo "Explanation: {$explanation}\n";
}
}
def assess_risk_level(message, unsafe_categories)
# Convert the unsafe categories into a string, with each category on a new line
unsafe_category_str = unsafe_categories.join("\n")
# Construct the prompt for Claude, including the message, unsafe categories, and risk level definitions
assessment_prompt = <<~PROMPT.chomp
Assess the risk level of the following message,
based on the unsafe categories listed below.
Message:
<message>#{message}</message>
Unsafe Categories:
<categories>
#{unsafe_category_str}
</categories>
Assign a risk level based on your confidence that the user's message should be moderated
based on the defined unsafe categories:
0 - No risk
1 - Low risk
2 - Medium risk
3 - High risk
Respond with ONLY a JSON object, using the format below:
{
"risk_level": <Numerical field denoting the risk level>,
"categories": [Comma-separated list of violated categories],
"explanation": <Optional. Only include if risk level is greater than 0>
}
Do not include markdown formatting or code fences in your response.
PROMPT
# Send the request to Claude for risk assessment
response = CLIENT.messages.create(
model: "claude-haiku-4-5-20251001", # Using the Haiku model for lower costs
max_tokens: 200,
messages: [{role: :user, content: assessment_prompt}]
)
# Parse the JSON response from Claude
text_block = response.content.find { it.type == :text }
assessment = JSON.parse(text_block.text)
# Extract the risk level, violated categories, and explanation from the assessment
risk_level = assessment["risk_level"]
violated_categories = assessment["categories"]
explanation = assessment["explanation"]
[risk_level, violated_categories, explanation]
end
# Process each comment and print the results
USER_COMMENTS.each do |comment|
puts "\nComment: #{comment}"
risk_level, violated_categories, explanation = assess_risk_level(comment, UNSAFE_CATEGORIES)
puts "Risk Level: #{risk_level}"
puts "Violated Categories: #{violated_categories.join(", ")}" if violated_categories&.any?
puts "Explanation: #{explanation}" if explanation
end
이 코드는 클로드를 사용해 메시지의 위험 수준을 평가하는 assess_risk_level 함수를 구현해요. 함수는 메시지와 불안전 범주를 입력으로 받아요.
함수 내에서 평가할 메시지, 불안전 범주, 위험 수준 평가를 위한 특정 지시를 포함하는 프롬프트가 클로드를 위해 생성돼요. 프롬프트는 클로드에게 위험 수준, 위반된 범주, 선택적 설명을 포함하는 JSON 객체로 응답하라고 지시해요.
이 접근 방식은 위험 수준을 할당해 유연한 콘텐츠 검열을 가능하게 해요. 더 큰 시스템에 매끄럽게 통합해 평가된 위험 수준에 따라 콘텐츠 필터링을 자동화하거나 댓글을 인간 검토용으로 플래그할 수 있어요. 예를 들어 이 코드를 실행하면 Delete this post now or you better hide. I am coming after you and your family. 댓글은 위험한 위협 때문에 고위험으로 식별돼요. 반대로 Stay away from the 5G cellphones!! They are using 5G to control you. 댓글은 중간 위험으로 분류돼요.
프롬프트 배포하기 (Deploy your prompt)
솔루션의 품질에 확신이 들면 프로덕션에 배포할 때예요. 프로덕션에서 콘텐츠 검열을 사용할 때 따르면 좋을 모범 사례 몇 가지:
-
사용자에게 명확한 피드백 제공: 사용자 입력이 차단되거나 응답이 콘텐츠 검열 때문에 플래그되면, 사용자가 왜 메시지가 플래그됐고 어떻게 다시 표현할 수 있는지 이해하도록 유익하고 건설적인 피드백을 제공하세요. 앞선 코딩 예시에서는 클로드 응답의
explanation필드를 통해 이뤄져요. -
검열된 콘텐츠 분석: 검열 시스템이 플래그하는 콘텐츠 유형을 추적해 추세와 개선 영역을 파악하세요.
-
지속적으로 평가하고 개선: 정밀도(precision)와 재현율(recall) 추적 같은 지표로 콘텐츠 검열 시스템의 성능을 정기적으로 평가하세요. 이 데이터를 사용해 검열 프롬프트, 키워드, 평가 기준을 반복적으로 다듬으세요.
성능 개선 (Improve performance)
복잡한 시나리오에서는 표준 프롬프트 엔지니어링 기법을 넘어 성능을 개선하기 위한 추가 전략을 고려하는 것이 도움이 될 수 있어요. 몇 가지 고급 전략:
주제 정의와 예시 제공 (Define topics and provide examples)
프롬프트에 불안전 범주를 나열하는 것 외에도, 각 범주와 관련된 정의와 문구를 제공하면 더 개선할 수 있어요.
def moderate_message_with_definitions(message, unsafe_category_definitions): # Format unsafe categories string, with each category and its definition on a new line unsafe_category_str = "\n".join( f"{category}: {definition}" for category, definition in unsafe_category_definitions.items() )
# Construct the prompt for Claude, including the message and unsafe categories
assessment_prompt = f"""Determine whether the following message warrants moderation, based on the unsafe categories outlined below.
Message:
Unsafe Categories and Their Definitions:
It's important that you remember all unsafe categories and their definitions.
Respond with ONLY a JSON object, using the format below:
{{
"violation":
# Send the request to Claude for content moderation
response = client.messages.create(
model="claude-haiku-4-5-20251001", # Using the Haiku model for lower costs
max_tokens=200,
messages=[{"role": "user", "content": assessment_prompt}],
)
# Parse the JSON response from Claude
text_block = next(block for block in response.content if block.type == "text")
assessment = json.loads(text_block.text)
# Extract the violation status from the assessment
contains_violation = assessment["violation"]
# If there's a violation, get the categories and explanation; otherwise, use empty defaults
violated_categories = assessment.get("categories", []) if contains_violation else []
explanation = assessment.get("explanation") if contains_violation else None
return contains_violation, violated_categories, explanation
Process each comment and print the results
for comment in user_comments: print(f"\nComment: {comment}") violation, violated_categories, explanation = moderate_message_with_definitions( comment, unsafe_category_definitions )
if violation:
print(f"Violated Categories: {', '.join(violated_categories)}")
print(f"Explanation: {explanation}")
else:
print("No issues detected.")
```typescript TypeScript
// Shape of the JSON assessment Claude returns
interface DefinitionBasedAssessment {
violation: boolean;
categories?: string[];
explanation?: string;
}
// Categories considered unsafe for content moderation, with their definitions
// (object keys preserve insertion order, so categories render in this order)
const unsafeCategoryDefinitions: Record<string, string> = {
"Child Exploitation":
"Content that depicts child nudity or that enables, encourages, excuses, or depicts the sexual abuse of children.",
"Conspiracy Theories": `Content that promotes or endorses unfounded, false, or misleading theories about events, situations, or individuals,
which can incite harm or undermine public trust in institutions or facts.`,
"Hate": `Content that is hateful toward people on the basis of their protected characteristics
(race, color, ethnicity, national origin, disability, religious affiliation, caste, sexual orientation, sex, gender identity and serious disease),
or content that perpetuates negative stereotypes.`,
"Indiscriminate Weapons":
"Content that enables the creation of weapons of mass destruction.",
"Intellectual Property":
"Content that violates the intellectual property rights of third parties.",
"Non-Violent Crimes":
"Content that enables, encourages, or excuses the commission of non-violent crimes.",
"Privacy":
"Content that contains sensitive, personal information about private individuals.",
"Self-Harm": "Content that enables, encourages, or excuses acts of intentional self-harm.",
"Sex Crimes": "Content that enables, encourages, or excuses the commission of sex crimes.",
"Sexual Content":
"Content that contains erotic descriptions or explicit depictions of sex acts.",
"Specialized Advice": `Content that contains financial, medical, or legal advice.
Financial advice includes guidance on investments, stocks, bonds, or any financial planning.`,
"Violent Crimes":
"Content that enables, encourages, or excuses the commission of violent crimes."
};
async function moderateMessageWithDefinitions(
message: string,
unsafeCategoryDefinitions: Record<string, string>
): Promise<{ violation: boolean; violatedCategories: string[]; explanation?: string }> {
// Format the unsafe categories string, with each category and its definition on a new line
const unsafeCategoryStr = Object.entries(unsafeCategoryDefinitions)
.map(([category, definition]) => `${category}: ${definition}`)
.join("\n");
// Construct the prompt for Claude, including the message and unsafe categories
const assessmentPrompt = `Determine whether the following message warrants moderation, based on the unsafe categories outlined below.
Message:
<message>${message}</message>
Unsafe Categories and Their Definitions:
<categories>
${unsafeCategoryStr}
</categories>
It's important that you remember all unsafe categories and their definitions.
Respond with ONLY a JSON object, using the format below:
{
"violation": <Boolean field denoting whether the message should be moderated>,
"categories": [Comma-separated list of violated categories],
"explanation": [Optional. Only include if there is a violation.]
}
Do not include markdown formatting or code fences in your response.`;
// Send the request to Claude for content moderation
const response = await client.messages.create({
model: "claude-haiku-4-5-20251001", // Using the Haiku model for lower costs
max_tokens: 200,
messages: [{ role: "user", content: assessmentPrompt }]
});
// Parse the JSON response from Claude
const textBlock = response.content.find((block) => block.type === "text");
if (!textBlock) {
throw new Error("Expected a text block in the response");
}
const assessment: DefinitionBasedAssessment = JSON.parse(textBlock.text);
// Extract the violation status from the assessment
const containsViolation = assessment.violation;
// If there's a violation, get the categories and explanation; otherwise, use empty defaults
const violatedCategories = containsViolation ? assessment.categories ?? [] : [];
const explanation = containsViolation ? assessment.explanation : undefined;
return { violation: containsViolation, violatedCategories, explanation };
}
// Process each comment and print the results
for (const comment of userComments) {
console.log(`\nComment: ${comment}`);
const { violation, violatedCategories, explanation } = await moderateMessageWithDefinitions(
comment,
unsafeCategoryDefinitions
);
if (violation) {
console.log(`Violated Categories: ${violatedCategories.join(", ")}`);
console.log(`Explanation: ${explanation}`);
} else {
console.log("No issues detected.");
}
}
// Categories considered unsafe for content moderation, with their definitions.
// The entries stay in insertion order, so the rendered prompt lists categories
// in exactly this order.
(string Category, string Definition)[] unsafeCategoryDefinitions =
[
(
"Child Exploitation",
"Content that depicts child nudity or that enables, encourages, excuses, or depicts the sexual abuse of children."
),
(
"Conspiracy Theories",
"""
Content that promotes or endorses unfounded, false, or misleading theories about events, situations, or individuals,
which can incite harm or undermine public trust in institutions or facts.
"""
),
(
"Hate",
"""
Content that is hateful toward people on the basis of their protected characteristics
(race, color, ethnicity, national origin, disability, religious affiliation, caste, sexual orientation, sex, gender identity and serious disease),
or content that perpetuates negative stereotypes.
"""
),
("Indiscriminate Weapons", "Content that enables the creation of weapons of mass destruction."),
("Intellectual Property", "Content that violates the intellectual property rights of third parties."),
("Non-Violent Crimes", "Content that enables, encourages, or excuses the commission of non-violent crimes."),
("Privacy", "Content that contains sensitive, personal information about private individuals."),
("Self-Harm", "Content that enables, encourages, or excuses acts of intentional self-harm."),
("Sex Crimes", "Content that enables, encourages, or excuses the commission of sex crimes."),
("Sexual Content", "Content that contains erotic descriptions or explicit depictions of sex acts."),
(
"Specialized Advice",
"""
Content that contains financial, medical, or legal advice.
Financial advice includes guidance on investments, stocks, bonds, or any financial planning.
"""
),
("Violent Crimes", "Content that enables, encourages, or excuses the commission of violent crimes."),
];
async Task<(bool ContainsViolation, List<string> ViolatedCategories, string? Explanation)> ModerateMessageWithDefinitions(
string message,
IReadOnlyList<(string Category, string Definition)> categoryDefinitions
)
{
// Format the unsafe categories string, with each category and its definition on a new line
var unsafeCategoryText = string.Join(
"\n",
categoryDefinitions.Select(entry => $"{entry.Category}: {entry.Definition}")
);
// Construct the prompt for Claude, including the message and unsafe categories
var assessmentPrompt = $$"""
Determine whether the following message warrants moderation, based on the unsafe categories outlined below.
Message:
<message>{{message}}</message>
Unsafe Categories and Their Definitions:
<categories>
{{unsafeCategoryText}}
</categories>
It's important that you remember all unsafe categories and their definitions.
Respond with ONLY a JSON object, using the format below:
{
"violation": <Boolean field denoting whether the message should be moderated>,
"categories": [Comma-separated list of violated categories],
"explanation": [Optional. Only include if there is a violation.]
}
Do not include markdown formatting or code fences in your response.
""";
// Send the request to Claude for content moderation
var response = await client.Messages.Create(
new()
{
Model = Model.ClaudeHaiku4_5_20251001, // Using the Haiku model for lower costs
MaxTokens = 200,
Messages = [new() { Role = Role.User, Content = assessmentPrompt }],
}
);
// Narrow the first content block to a text block, then parse Claude's JSON response
if (!response.Content[0].TryPickText(out var textBlock))
{
throw new InvalidOperationException("Expected a text response from Claude.");
}
var assessment = JsonNode.Parse(textBlock.Text)!;
// Extract the violation status from the assessment
var containsViolation = assessment["violation"]!.GetValue<bool>();
// If there's a violation, get the categories and explanation; otherwise, use empty defaults
List<string> violatedCategories = containsViolation
? assessment["categories"]?.AsArray().Select(category => category!.GetValue<string>()).ToList() ?? []
: [];
var explanation = containsViolation ? assessment["explanation"]?.GetValue<string>() : null;
return (containsViolation, violatedCategories, explanation);
}
// Process each comment and print the results
foreach (var comment in userComments)
{
Console.WriteLine($"\nComment: {comment}");
var (violation, violatedCategories, explanation) = await ModerateMessageWithDefinitions(
comment,
unsafeCategoryDefinitions
);
if (violation)
{
Console.WriteLine($"Violated Categories: {string.Join(", ", violatedCategories)}");
Console.WriteLine($"Explanation: {explanation}");
}
else
{
Console.WriteLine("No issues detected.");
}
}
// Categories considered unsafe for content moderation, with their definitions.
// A slice of category/definition pairs (rather than a map) keeps the rendered
// order stable; Go maps iterate in random order.
type categoryDefinition struct {
category string
definition string
}
var unsafeCategoryDefinitions = []categoryDefinition{
{"Child Exploitation", "Content that depicts child nudity or that enables, encourages, excuses, or depicts the sexual abuse of children."},
{"Conspiracy Theories", `Content that promotes or endorses unfounded, false, or misleading theories about events, situations, or individuals,
which can incite harm or undermine public trust in institutions or facts.`},
{"Hate", `Content that is hateful toward people on the basis of their protected characteristics
(race, color, ethnicity, national origin, disability, religious affiliation, caste, sexual orientation, sex, gender identity and serious disease),
or content that perpetuates negative stereotypes.`},
{"Indiscriminate Weapons", "Content that enables the creation of weapons of mass destruction."},
{"Intellectual Property", "Content that violates the intellectual property rights of third parties."},
{"Non-Violent Crimes", "Content that enables, encourages, or excuses the commission of non-violent crimes."},
{"Privacy", "Content that contains sensitive, personal information about private individuals."},
{"Self-Harm", "Content that enables, encourages, or excuses acts of intentional self-harm."},
{"Sex Crimes", "Content that enables, encourages, or excuses the commission of sex crimes."},
{"Sexual Content", "Content that contains erotic descriptions or explicit depictions of sex acts."},
{"Specialized Advice", `Content that contains financial, medical, or legal advice.
Financial advice includes guidance on investments, stocks, bonds, or any financial planning.`},
{"Violent Crimes", "Content that enables, encourages, or excuses the commission of violent crimes."},
}
func moderateMessageWithDefinitions(message string, unsafeCategoryDefinitions []categoryDefinition) (bool, []string, string) {
// Format unsafe categories string, with each category and its definition on a new line
categoryLines := make([]string, len(unsafeCategoryDefinitions))
for i, entry := range unsafeCategoryDefinitions {
categoryLines[i] = fmt.Sprintf("%s: %s", entry.category, entry.definition)
}
unsafeCategoryStr := strings.Join(categoryLines, "\n")
// Construct the prompt for Claude, including the message and unsafe categories
assessmentPrompt := fmt.Sprintf(`Determine whether the following message warrants moderation, based on the unsafe categories outlined below.
Message:
<message>%s</message>
Unsafe Categories and Their Definitions:
<categories>
%s
</categories>
It's important that you remember all unsafe categories and their definitions.
Respond with ONLY a JSON object, using the format below:
{
"violation": <Boolean field denoting whether the message should be moderated>,
"categories": [Comma-separated list of violated categories],
"explanation": [Optional. Only include if there is a violation.]
}
Do not include markdown formatting or code fences in your response.`, message, unsafeCategoryStr)
// Send the request to Claude for content moderation
response, err := client.Messages.New(context.Background(), anthropic.MessageNewParams{
Model: anthropic.ModelClaudeHaiku4_5_20251001, // Using the Haiku model for lower costs
MaxTokens: 200,
Messages: []anthropic.MessageParam{
anthropic.NewUserMessage(anthropic.NewTextBlock(assessmentPrompt)),
},
})
if err != nil {
log.Fatal(err)
}
// Narrow the first content block to a text block before reading its text
textBlock, ok := response.Content[0].AsAny().(anthropic.TextBlock)
if !ok {
log.Fatalf("expected a text block, got %q", response.Content[0].Type)
}
// Parse the JSON response from Claude
var assessment struct {
Violation bool `json:"violation"`
Categories []string `json:"categories"`
Explanation string `json:"explanation"`
}
if err := json.Unmarshal([]byte(textBlock.Text), &assessment); err != nil {
log.Fatal(err)
}
// If there's a violation, return the categories and explanation; otherwise, use empty defaults
if !assessment.Violation {
return false, nil, ""
}
return true, assessment.Categories, assessment.Explanation
}
// moderateAllCommentsWithDefinitions processes each comment and prints the results.
func moderateAllCommentsWithDefinitions() {
for _, comment := range userComments {
fmt.Printf("\nComment: %s\n", comment)
violation, violatedCategories, explanation := moderateMessageWithDefinitions(comment, unsafeCategoryDefinitions)
if violation {
fmt.Printf("Violated Categories: %s\n", strings.Join(violatedCategories, ", "))
fmt.Printf("Explanation: %s\n", explanation)
} else {
fmt.Println("No issues detected.")
}
}
}
// Categories considered unsafe for content moderation, with their definitions
record CategoryDefinition(String category, String definition) {}
final List<CategoryDefinition> unsafeCategoryDefinitions = List.of(
new CategoryDefinition(
"Child Exploitation",
"Content that depicts child nudity or that enables, encourages, excuses, or depicts the sexual abuse of children."),
new CategoryDefinition(
"Conspiracy Theories",
"""
Content that promotes or endorses unfounded, false, or misleading theories about events, situations, or individuals,
which can incite harm or undermine public trust in institutions or facts."""),
new CategoryDefinition(
"Hate",
"""
Content that is hateful toward people on the basis of their protected characteristics
(race, color, ethnicity, national origin, disability, religious affiliation, caste, sexual orientation, sex, gender identity and serious disease),
or content that perpetuates negative stereotypes."""),
new CategoryDefinition(
"Indiscriminate Weapons",
"Content that enables the creation of weapons of mass destruction."),
new CategoryDefinition(
"Intellectual Property",
"Content that violates the intellectual property rights of third parties."),
new CategoryDefinition(
"Non-Violent Crimes",
"Content that enables, encourages, or excuses the commission of non-violent crimes."),
new CategoryDefinition(
"Privacy",
"Content that contains sensitive, personal information about private individuals."),
new CategoryDefinition(
"Self-Harm",
"Content that enables, encourages, or excuses acts of intentional self-harm."),
new CategoryDefinition(
"Sex Crimes",
"Content that enables, encourages, or excuses the commission of sex crimes."),
new CategoryDefinition(
"Sexual Content",
"Content that contains erotic descriptions or explicit depictions of sex acts."),
new CategoryDefinition(
"Specialized Advice",
"""
Content that contains financial, medical, or legal advice.
Financial advice includes guidance on investments, stocks, bonds, or any financial planning."""),
new CategoryDefinition(
"Violent Crimes",
"Content that enables, encourages, or excuses the commission of violent crimes."));
record ModerationDecision(boolean violation, List<String> violatedCategories, String explanation) {}
ModerationDecision moderateMessageWithDefinitions(
String message, List<CategoryDefinition> unsafeCategoryDefinitions)
throws JsonProcessingException {
// Format unsafe categories string, with each category and its definition on a new line
String unsafeCategoryStr = unsafeCategoryDefinitions.stream()
.map(categoryDefinition ->
categoryDefinition.category() + ": " + categoryDefinition.definition())
.collect(Collectors.joining("\n"));
// Construct the prompt for Claude, including the message and unsafe categories
String assessmentPrompt = """
Determine whether the following message warrants moderation, based on the unsafe categories outlined below.
Message:
<message>%s</message>
Unsafe Categories and Their Definitions:
<categories>
%s
</categories>
It's important that you remember all unsafe categories and their definitions.
Respond with ONLY a JSON object, using the format below:
{
"violation": <Boolean field denoting whether the message should be moderated>,
"categories": [Comma-separated list of violated categories],
"explanation": [Optional. Only include if there is a violation.]
}
Do not include markdown formatting or code fences in your response."""
.formatted(message, unsafeCategoryStr);
// Send the request to Claude for content moderation
Message response = client.messages().create(MessageCreateParams.builder()
.model(Model.CLAUDE_HAIKU_4_5_20251001) // Using the Haiku model for lower costs
.maxTokens(200)
.addUserMessage(assessmentPrompt)
.build());
// Parse the JSON response from Claude
String assessmentJson = response.content().stream()
.flatMap(contentBlock -> contentBlock.text().stream())
.findFirst()
.orElseThrow()
.text();
ObjectMapper mapper = new ObjectMapper();
JsonNode assessment = mapper.readTree(assessmentJson);
// Extract the violation status from the assessment
boolean containsViolation = assessment.required("violation").asBoolean();
// If there's a violation, get the categories and explanation; otherwise, use empty defaults
List<String> violatedCategories = containsViolation && assessment.has("categories")
? mapper.convertValue(assessment.get("categories"), new TypeReference<List<String>>() {})
: List.of();
String explanation = containsViolation && assessment.hasNonNull("explanation")
? assessment.get("explanation").asText()
: null;
return new ModerationDecision(containsViolation, violatedCategories, explanation);
}
// Process each comment and print the results
void printModerationResultsWithDefinitions() throws JsonProcessingException {
for (String comment : userComments) {
IO.println("\nComment: " + comment);
ModerationDecision result = moderateMessageWithDefinitions(comment, unsafeCategoryDefinitions);
if (result.violation()) {
IO.println("Violated Categories: " + String.join(", ", result.violatedCategories()));
IO.println("Explanation: " + result.explanation());
} else {
IO.println("No issues detected.");
}
}
}
// Categories considered unsafe for content moderation, with their definitions
$unsafeCategoryDefinitions = [
'Child Exploitation' => 'Content that depicts child nudity or that enables, encourages, excuses, or depicts the sexual abuse of children.',
'Conspiracy Theories' => 'Content that promotes or endorses unfounded, false, or misleading theories about events, situations, or individuals,
which can incite harm or undermine public trust in institutions or facts.',
'Hate' => 'Content that is hateful toward people on the basis of their protected characteristics
(race, color, ethnicity, national origin, disability, religious affiliation, caste, sexual orientation, sex, gender identity and serious disease),
or content that perpetuates negative stereotypes.',
'Indiscriminate Weapons' => 'Content that enables the creation of weapons of mass destruction.',
'Intellectual Property' => 'Content that violates the intellectual property rights of third parties.',
'Non-Violent Crimes' => 'Content that enables, encourages, or excuses the commission of non-violent crimes.',
'Privacy' => 'Content that contains sensitive, personal information about private individuals.',
'Self-Harm' => 'Content that enables, encourages, or excuses acts of intentional self-harm.',
'Sex Crimes' => 'Content that enables, encourages, or excuses the commission of sex crimes.',
'Sexual Content' => 'Content that contains erotic descriptions or explicit depictions of sex acts.',
'Specialized Advice' => 'Content that contains financial, medical, or legal advice.
Financial advice includes guidance on investments, stocks, bonds, or any financial planning.',
'Violent Crimes' => 'Content that enables, encourages, or excuses the commission of violent crimes.',
];
$moderateMessageWithDefinitions = function (string $message, array $unsafeCategoryDefinitions) use ($client): array {
// Format the unsafe categories string, with each category and its definition on a new line
$categoryLines = [];
foreach ($unsafeCategoryDefinitions as $category => $definition) {
$categoryLines[] = "{$category}: {$definition}";
}
$unsafeCategoryStr = implode("\n", $categoryLines);
// Construct the prompt for Claude, including the message and unsafe categories
$assessmentPrompt = <<<PROMPT
Determine whether the following message warrants moderation, based on the unsafe categories outlined below.
Message:
<message>{$message}</message>
Unsafe Categories and Their Definitions:
<categories>
{$unsafeCategoryStr}
</categories>
It's important that you remember all unsafe categories and their definitions.
Respond with ONLY a JSON object, using the format below:
{
"violation": <Boolean field denoting whether the message should be moderated>,
"categories": [Comma-separated list of violated categories],
"explanation": [Optional. Only include if there is a violation.]
}
Do not include markdown formatting or code fences in your response.
PROMPT;
// Send the request to Claude for content moderation
$response = $client->messages->create(
model: 'claude-haiku-4-5-20251001', // Using the Haiku model for lower costs
maxTokens: 200,
messages: [['role' => 'user', 'content' => $assessmentPrompt]],
);
// Parse the JSON response from Claude. The SDK decodes each content block
// into its concrete class, so find the TextBlock before reading the text.
$textBlock = array_find($response->content, fn ($block) => $block instanceof \Anthropic\Messages\TextBlock)
?? throw new RuntimeException('Expected a text block in the response.');
$assessment = json_decode($textBlock->text, associative: true, flags: JSON_THROW_ON_ERROR);
// Extract the violation status from the assessment
$containsViolation = $assessment['violation'];
// If there's a violation, get the categories and explanation; otherwise, use empty defaults
$violatedCategories = $containsViolation ? ($assessment['categories'] ?? []) : [];
$explanation = $containsViolation ? ($assessment['explanation'] ?? null) : null;
return [$containsViolation, $violatedCategories, $explanation];
};
// Process each comment and print the results
foreach ($userComments as $comment) {
echo "\nComment: {$comment}\n";
[$violation, $violatedCategories, $explanation] = $moderateMessageWithDefinitions($comment, $unsafeCategoryDefinitions);
if ($violation) {
echo 'Violated Categories: ' . implode(', ', $violatedCategories) . "\n";
echo "Explanation: {$explanation}\n";
} else {
echo "No issues detected.\n";
}
}
# Categories considered unsafe for content moderation, with their definitions
UNSAFE_CATEGORY_DEFINITIONS = {
"Child Exploitation" => "Content that depicts child nudity or that enables, encourages, excuses, or depicts the sexual abuse of children.",
"Conspiracy Theories" => "Content that promotes or endorses unfounded, false, or misleading theories about events, situations, or individuals,
which can incite harm or undermine public trust in institutions or facts.",
"Hate" => "Content that is hateful toward people on the basis of their protected characteristics
(race, color, ethnicity, national origin, disability, religious affiliation, caste, sexual orientation, sex, gender identity and serious disease),
or content that perpetuates negative stereotypes.",
"Indiscriminate Weapons" => "Content that enables the creation of weapons of mass destruction.",
"Intellectual Property" => "Content that violates the intellectual property rights of third parties.",
"Non-Violent Crimes" => "Content that enables, encourages, or excuses the commission of non-violent crimes.",
"Privacy" => "Content that contains sensitive, personal information about private individuals.",
"Self-Harm" => "Content that enables, encourages, or excuses acts of intentional self-harm.",
"Sex Crimes" => "Content that enables, encourages, or excuses the commission of sex crimes.",
"Sexual Content" => "Content that contains erotic descriptions or explicit depictions of sex acts.",
"Specialized Advice" => "Content that contains financial, medical, or legal advice.
Financial advice includes guidance on investments, stocks, bonds, or any financial planning.",
"Violent Crimes" => "Content that enables, encourages, or excuses the commission of violent crimes."
}
def moderate_message_with_definitions(message, unsafe_category_definitions)
# Format the unsafe categories string, with each category and its definition on a new line
unsafe_category_str = unsafe_category_definitions
.map { |category, definition| "#{category}: #{definition}" }
.join("\n")
# Construct the prompt for Claude, including the message and unsafe categories
assessment_prompt = <<~PROMPT.chomp
Determine whether the following message warrants moderation, based on the unsafe categories outlined below.
Message:
<message>#{message}</message>
Unsafe Categories and Their Definitions:
<categories>
#{unsafe_category_str}
</categories>
It's important that you remember all unsafe categories and their definitions.
Respond with ONLY a JSON object, using the format below:
{
"violation": <Boolean field denoting whether the message should be moderated>,
"categories": [Comma-separated list of violated categories],
"explanation": [Optional. Only include if there is a violation.]
}
Do not include markdown formatting or code fences in your response.
PROMPT
# Send the request to Claude for content moderation
response = CLIENT.messages.create(
model: "claude-haiku-4-5-20251001", # Using the Haiku model for lower costs
max_tokens: 200,
messages: [{role: :user, content: assessment_prompt}]
)
# Parse the JSON response from Claude
text_block = response.content.find { it.type == :text }
assessment = JSON.parse(text_block.text)
# Extract the violation status from the assessment
contains_violation = assessment["violation"]
# If there's a violation, get the categories and explanation; otherwise, use empty defaults
violated_categories = contains_violation ? assessment.fetch("categories", []) : []
explanation = contains_violation ? assessment["explanation"] : nil
[contains_violation, violated_categories, explanation]
end
# Process each comment and print the results
USER_COMMENTS.each do |comment|
puts "\nComment: #{comment}"
violation, violated_categories, explanation = moderate_message_with_definitions(comment, UNSAFE_CATEGORY_DEFINITIONS)
if violation
puts "Violated Categories: #{violated_categories.join(", ")}"
puts "Explanation: #{explanation}"
else
puts "No issues detected."
end
end
moderate_message_with_definitions 함수는 각 불안전 범주를 상세 정의와 짝지을 수 있도록 앞선 moderate_message 함수를 확장해요. 코드에서는 원래 함수의 unsafe_categories 컬렉션을 unsafe_category_definitions 매핑으로 대체해 이뤄져요. 이 매핑은 각 불안전 범주를 해당 정의와 짝지어요. 범주 이름과 정의 모두 프롬프트에 포함돼요.
주목할 점은 Specialized Advice 범주의 정의가 이제 금지되어야 할 금융 조언 유형을 명시한다는 거예요. 그 결과, 이전 moderate_message 평가를 통과했던 It is a great time to invest in gold! 댓글이 이제 위반을 촉발해요.
배치 처리 고려하기 (Consider batch processing)
실시간 검열이 필요하지 않은 상황에서 비용을 줄이려면 메시지를 배치로 검열하는 것을 고려하세요. 프롬프트의 컨텍스트에 여러 메시지를 포함하고, 어떤 메시지를 검열해야 하는지 클로드에게 평가하라고 요청해요.
# Format messages string, with each message wrapped in XML-like tags and given an ID
messages_str = "\n".join(
[f"<message id={idx}>{msg}</message>" for idx, msg in enumerate(messages)]
)
# Construct the prompt for Claude, including the messages and unsafe categories
assessment_prompt = f"""Determine the messages to moderate, based on the unsafe categories outlined below.
Messages:
Unsafe Categories:
Respond with ONLY a JSON object, using the format below:
{{
"violations": [
{{
"id":
Important Notes:
-
Remember to analyze every message for a violation.
-
Select any number of violations that reasonably apply.
-
Do not include markdown formatting or code fences in your response."""
Send the request to Claude for content moderation
response = client.messages.create( model="claude-haiku-4-5-20251001", # Using the Haiku model for lower costs max_tokens=2048, # Increased max token count to handle batches messages=[{"role": "user", "content": assessment_prompt}], )
Parse the JSON response from Claude
text_block = next(block for block in response.content if block.type == "text") assessment = json.loads(text_block.text) return assessment
Process the batch of comments and get the response
response_obj = batch_moderate_messages(user_comments, unsafe_categories)
Print the results for each detected violation
for violation in response_obj["violations"]: print(f"""Comment: {user_comments[violation["id"]]} Violated Categories: {", ".join(violation["categories"])} Explanation: {violation["explanation"]} """)
```typescript TypeScript
// Shape of the JSON batch assessment Claude returns
interface BatchAssessment {
violations: {
id: number;
categories: string[];
explanation: string;
}[];
}
async function batchModerateMessages(
messages: string[],
unsafeCategories: string[]
): Promise<BatchAssessment> {
// Convert the unsafe categories into a string, with each category on a new line
const unsafeCategoryStr = unsafeCategories.join("\n");
// Format the messages string, with each message wrapped in XML-like tags and given an ID
const messagesStr = messages
.map((msg, idx) => `<message id=${idx}>${msg}</message>`)
.join("\n");
// Construct the prompt for Claude, including the messages and unsafe categories
const assessmentPrompt = `Determine the messages to moderate, based on the unsafe categories outlined below.
Messages:
<messages>
${messagesStr}
</messages>
Unsafe Categories:
<categories>
${unsafeCategoryStr}
</categories>
Respond with ONLY a JSON object, using the format below:
{
"violations": [
{
"id": <message id>,
"categories": [list of violated categories],
"explanation": <Explanation of why there's a violation>
}
]
}
Important Notes:
- Remember to analyze every message for a violation.
- Select any number of violations that reasonably apply.
- Do not include markdown formatting or code fences in your response.`;
// Send the request to Claude for content moderation
const response = await client.messages.create({
model: "claude-haiku-4-5-20251001", // Using the Haiku model for lower costs
max_tokens: 2048, // Increased max token count to handle batches
messages: [{ role: "user", content: assessmentPrompt }]
});
// Parse the JSON response from Claude
const textBlock = response.content.find((block) => block.type === "text");
if (!textBlock) {
throw new Error("Expected a text block in the response");
}
const assessment: BatchAssessment = JSON.parse(textBlock.text);
return assessment;
}
// Process the batch of comments and get the response
const batchAssessment = await batchModerateMessages(userComments, unsafeCategories);
// Print the results for each detected violation
for (const violation of batchAssessment.violations) {
console.log(`Comment: ${userComments[violation.id]}
Violated Categories: ${violation.categories.join(", ")}
Explanation: ${violation.explanation}
`);
}
async Task<JsonNode> BatchModerateMessages(IReadOnlyList<string> messages, IReadOnlyList<string> categories)
{
// Convert the unsafe categories into a string, with each category on a new line
var unsafeCategoryText = string.Join("\n", categories);
// Format the messages string, with each message wrapped in XML-like tags and given an ID
var messagesText = string.Join(
"\n",
messages.Select((message, index) => $"<message id={index}>{message}</message>")
);
// Construct the prompt for Claude, including the messages and unsafe categories
var assessmentPrompt = $$"""
Determine the messages to moderate, based on the unsafe categories outlined below.
Messages:
<messages>
{{messagesText}}
</messages>
Unsafe Categories:
<categories>
{{unsafeCategoryText}}
</categories>
Respond with ONLY a JSON object, using the format below:
{
"violations": [
{
"id": <message id>,
"categories": [list of violated categories],
"explanation": <Explanation of why there's a violation>
}
]
}
Important Notes:
- Remember to analyze every message for a violation.
- Select any number of violations that reasonably apply.
- Do not include markdown formatting or code fences in your response.
""";
// Send the request to Claude for content moderation
var response = await client.Messages.Create(
new()
{
Model = Model.ClaudeHaiku4_5_20251001, // Using the Haiku model for lower costs
MaxTokens = 2048, // Increased max token count to handle batches
Messages = [new() { Role = Role.User, Content = assessmentPrompt }],
}
);
// Narrow the first content block to a text block, then parse Claude's JSON response
if (!response.Content[0].TryPickText(out var textBlock))
{
throw new InvalidOperationException("Expected a text response from Claude.");
}
return JsonNode.Parse(textBlock.Text)!;
}
// Process the batch of comments and get the response
var moderationResults = await BatchModerateMessages(userComments, unsafeCategories);
// Print the results for each detected violation
foreach (var violation in moderationResults["violations"]!.AsArray())
{
var flaggedComment = userComments[violation!["id"]!.GetValue<int>()];
var violatedCategories = string.Join(
", ",
violation["categories"]!.AsArray().Select(category => category!.GetValue<string>())
);
var explanation = violation["explanation"]!.GetValue<string>();
Console.WriteLine($"""
Comment: {flaggedComment}
Violated Categories: {violatedCategories}
Explanation: {explanation}
""");
}
// batchViolation is one entry in Claude's "violations" array: the index of the
// offending message plus the categories it violated and why.
type batchViolation struct {
ID int `json:"id"`
Categories []string `json:"categories"`
Explanation string `json:"explanation"`
}
func batchModerateMessages(messages []string, unsafeCategories []string) []batchViolation {
// Convert the unsafe categories into a string, with each category on a new line
unsafeCategoryStr := strings.Join(unsafeCategories, "\n")
// Format messages string, with each message wrapped in XML-like tags and given an ID
messageLines := make([]string, len(messages))
for i, message := range messages {
messageLines[i] = fmt.Sprintf("<message id=%d>%s</message>", i, message)
}
messagesStr := strings.Join(messageLines, "\n")
// Construct the prompt for Claude, including the messages and unsafe categories
assessmentPrompt := fmt.Sprintf(`Determine the messages to moderate, based on the unsafe categories outlined below.
Messages:
<messages>
%s
</messages>
Unsafe Categories:
<categories>
%s
</categories>
Respond with ONLY a JSON object, using the format below:
{
"violations": [
{
"id": <message id>,
"categories": [list of violated categories],
"explanation": <Explanation of why there's a violation>
}
]
}
Important Notes:
- Remember to analyze every message for a violation.
- Select any number of violations that reasonably apply.
- Do not include markdown formatting or code fences in your response.`, messagesStr, unsafeCategoryStr)
// Send the request to Claude for content moderation
response, err := client.Messages.New(context.Background(), anthropic.MessageNewParams{
Model: anthropic.ModelClaudeHaiku4_5_20251001, // Using the Haiku model for lower costs
MaxTokens: 2048, // Increased max token count to handle batches
Messages: []anthropic.MessageParam{
anthropic.NewUserMessage(anthropic.NewTextBlock(assessmentPrompt)),
},
})
if err != nil {
log.Fatal(err)
}
// Narrow the first content block to a text block before reading its text
textBlock, ok := response.Content[0].AsAny().(anthropic.TextBlock)
if !ok {
log.Fatalf("expected a text block, got %q", response.Content[0].Type)
}
// Parse the JSON response from Claude
var assessment struct {
Violations []batchViolation `json:"violations"`
}
if err := json.Unmarshal([]byte(textBlock.Text), &assessment); err != nil {
log.Fatal(err)
}
return assessment.Violations
}
// moderateAllCommentsAsBatch moderates the whole batch of comments in a single
// request and prints the results for each detected violation.
func moderateAllCommentsAsBatch() {
// Process the batch of comments and get the response
violations := batchModerateMessages(userComments, unsafeCategories)
// Print the results for each detected violation
for _, violation := range violations {
fmt.Printf(`Comment: %s
Violated Categories: %s
Explanation: %s
`, userComments[violation.ID], strings.Join(violation.Categories, ", "), violation.Explanation)
}
}
JsonNode batchModerateMessages(List<String> messages, List<String> unsafeCategories)
throws JsonProcessingException {
// Convert the unsafe categories into a string, with each category on a new line
String unsafeCategoryStr = String.join("\n", unsafeCategories);
// Format messages string, with each message wrapped in XML-like tags and given an ID
String messagesStr = IntStream.range(0, messages.size())
.mapToObj(idx -> "<message id=%d>%s</message>".formatted(idx, messages.get(idx)))
.collect(Collectors.joining("\n"));
// Construct the prompt for Claude, including the messages and unsafe categories
String assessmentPrompt = """
Determine the messages to moderate, based on the unsafe categories outlined below.
Messages:
<messages>
%s
</messages>
Unsafe Categories:
<categories>
%s
</categories>
Respond with ONLY a JSON object, using the format below:
{
"violations": [
{
"id": <message id>,
"categories": [list of violated categories],
"explanation": <Explanation of why there's a violation>
}
]
}
Important Notes:
- Remember to analyze every message for a violation.
- Select any number of violations that reasonably apply.
- Do not include markdown formatting or code fences in your response."""
.formatted(messagesStr, unsafeCategoryStr);
// Send the request to Claude for content moderation
Message response = client.messages().create(MessageCreateParams.builder()
.model(Model.CLAUDE_HAIKU_4_5_20251001) // Using the Haiku model for lower costs
.maxTokens(2048) // Increased max token count to handle batches
.addUserMessage(assessmentPrompt)
.build());
// Parse the JSON response from Claude
String assessmentJson = response.content().stream()
.flatMap(contentBlock -> contentBlock.text().stream())
.findFirst()
.orElseThrow()
.text();
return new ObjectMapper().readTree(assessmentJson);
}
// Process the batch of comments and print the results for each detected violation
void printBatchViolations() throws JsonProcessingException {
JsonNode response = batchModerateMessages(userComments, unsafeCategories);
ObjectMapper mapper = new ObjectMapper();
for (JsonNode violation : response.required("violations")) {
List<String> violatedCategories =
mapper.convertValue(violation.required("categories"), new TypeReference<List<String>>() {});
IO.println("""
Comment: %s
Violated Categories: %s
Explanation: %s
""".formatted(
userComments.get(violation.required("id").asInt()),
String.join(", ", violatedCategories),
violation.required("explanation").asText()));
}
}
$batchModerateMessages = function (array $messages, array $unsafeCategories) use ($client): array {
// Convert the unsafe categories into a string, with each category on a new line
$unsafeCategoryStr = implode("\n", $unsafeCategories);
// Format the messages string, with each message wrapped in XML-like tags and given an ID
$messageLines = [];
foreach ($messages as $idx => $msg) {
$messageLines[] = "<message id={$idx}>{$msg}</message>";
}
$messagesStr = implode("\n", $messageLines);
// Construct the prompt for Claude, including the messages and unsafe categories
$assessmentPrompt = <<<PROMPT
Determine the messages to moderate, based on the unsafe categories outlined below.
Messages:
<messages>
{$messagesStr}
</messages>
Unsafe Categories:
<categories>
{$unsafeCategoryStr}
</categories>
Respond with ONLY a JSON object, using the format below:
{
"violations": [
{
"id": <message id>,
"categories": [list of violated categories],
"explanation": <Explanation of why there's a violation>
}
]
}
Important Notes:
- Remember to analyze every message for a violation.
- Select any number of violations that reasonably apply.
- Do not include markdown formatting or code fences in your response.
PROMPT;
// Send the request to Claude for content moderation
$response = $client->messages->create(
model: 'claude-haiku-4-5-20251001', // Using the Haiku model for lower costs
maxTokens: 2048, // Increased max token count to handle batches
messages: [['role' => 'user', 'content' => $assessmentPrompt]],
);
// Parse the JSON response from Claude. The SDK decodes each content block
// into its concrete class, so find the TextBlock before reading the text.
$textBlock = array_find($response->content, fn ($block) => $block instanceof \Anthropic\Messages\TextBlock)
?? throw new RuntimeException('Expected a text block in the response.');
return json_decode($textBlock->text, associative: true, flags: JSON_THROW_ON_ERROR);
};
// Process the batch of comments and get the response
$responseObj = $batchModerateMessages($userComments, $unsafeCategories);
// Print the results for each detected violation
foreach ($responseObj['violations'] as $violation) {
echo "Comment: {$userComments[$violation['id']]}\n";
echo 'Violated Categories: ' . implode(', ', $violation['categories']) . "\n";
echo "Explanation: {$violation['explanation']}\n\n";
}
def batch_moderate_messages(messages, unsafe_categories)
# Convert the unsafe categories into a string, with each category on a new line
unsafe_category_str = unsafe_categories.join("\n")
# Format the messages string, with each message wrapped in XML-like tags and given an ID
messages_str = messages
.map.with_index { |message, index| "<message id=#{index}>#{message}</message>" }
.join("\n")
# Construct the prompt for Claude, including the messages and unsafe categories
assessment_prompt = <<~PROMPT.chomp
Determine the messages to moderate, based on the unsafe categories outlined below.
Messages:
<messages>
#{messages_str}
</messages>
Unsafe Categories:
<categories>
#{unsafe_category_str}
</categories>
Respond with ONLY a JSON object, using the format below:
{
"violations": [
{
"id": <message id>,
"categories": [list of violated categories],
"explanation": <Explanation of why there's a violation>
}
]
}
Important Notes:
- Remember to analyze every message for a violation.
- Select any number of violations that reasonably apply.
- Do not include markdown formatting or code fences in your response.
PROMPT
# Send the request to Claude for content moderation
response = CLIENT.messages.create(
model: "claude-haiku-4-5-20251001", # Using the Haiku model for lower costs
max_tokens: 2048, # Increased max token count to handle batches
messages: [{role: :user, content: assessment_prompt}]
)
# Parse the JSON response from Claude
text_block = response.content.find { it.type == :text }
JSON.parse(text_block.text)
end
# Process the batch of comments and get the response
response_obj = batch_moderate_messages(USER_COMMENTS, UNSAFE_CATEGORIES)
# Print the results for each detected violation
response_obj["violations"].each do |violation|
puts <<~RESULT
Comment: #{USER_COMMENTS[violation["id"]]}
Violated Categories: #{violation["categories"].join(", ")}
Explanation: #{violation["explanation"]}
RESULT
end
이 예시에서 batch_moderate_messages 함수는 단일 클로드 API 호출로 전체 메시지 배치의 검열을 처리해요. 함수 내에서 평가할 메시지 목록과 불안전 콘텐츠 범주를 포함하는 프롬프트가 생성돼요. 프롬프트는 클로드에게 위반을 포함하는 모든 메시지를 나열하는 JSON 객체를 반환하도록 지시해요. 응답의 각 메시지는 배치에서 메시지의 위치에 해당하는 id로 식별돼요. 특정 요구에 가장 적합한 배치 크기를 찾으려면 실험이 필요할 수 있다는 점을 기억하세요. 더 큰 배치 크기는 비용을 낮출 수 있지만 품질이 약간 떨어질 수도 있어요. 또한 더 긴 응답을 수용하도록 클로드 API 호출의 max_tokens 파라미터를 늘려야 할 수도 있어요. 선택한 모델이 출력할 수 있는 최대 토큰 수에 대한 자세한 내용은 모델 비교 표를 참고하세요.