데이터셋 관리 자동화

데이터셋 관리 자동화 (Automate Dataset Management)

Confident API를 통해 골든스를 데이터셋에 프로그래밍 방식으로 푸시하는 방법을 배워볼게요. 데이터셋의 골든스를 코드로 관리하면서, 싱글턴·멀티턴 골든스를 푸시하고, 확정(finalized) 상태를 정하며, 커스텀 열 값과 개별 골든스의 수정·삭제, 데이터셋 삭제까지 다룰 수 있어요.

출처: 문서

본문

개요

이 섹션은 Confident API를 사용해 데이터셋의 골든스를 프로그래밍 방식으로 관리하는 방법을 다뤄요:

싱글턴·멀티턴 골든스를 데이터셋에 푸시
finalized=True로 평가에 사용할 수 있게 하거나, finalized=False로 검토 대기열에 넣기
골든스 푸시 시 커스텀 열 값 포함
id로 개별 골든스 수정 또는 삭제
프로그래밍 방식으로 데이터셋 삭제

확정된 골든스만 평가에 가져와져요.

골든스 푸시 (Push Goldens)

골든스를 데이터셋에 푸시해요. 데이터셋이 이미 존재하지 않으면 Confident AI가 자동으로 만들어줘요.

Python

싱글턴 데이터셋의 경우:

from deepeval.dataset import EvaluationDataset, Golden

goldens = [Golden(input="How tall is Mt. Everest?")]
dataset = EvaluationDataset(goldens=goldens)

# Push as finalized (ready for evaluation)
dataset.push(alias="YOUR-DATASET-ALIAS", finalized=True)

# Or push as unfinalized (queued for review)
dataset.push(alias="YOUR-DATASET-ALIAS", finalized=False)

멀티턴 데이터셋의 경우:

With Turns

from deepeval.dataset import EvaluationDataset, ConversationalGolden
from deepeval.test_case import Turn

goldens = [
ConversationalGolden(
scenario="Angry user asking for a refund.",
turns=[Turn(role="user", content="Give me my money!")]
)
]
dataset = EvaluationDataset(goldens=goldens)

dataset.push(alias="YOUR-DATASET-ALIAS", finalized=True)

Without Turns

from deepeval.dataset import EvaluationDataset, ConversationalGolden

goldens = [ConversationalGolden(scenario="Angry user asking for a refund.")]
dataset = EvaluationDataset(goldens=goldens)

dataset.push(alias="YOUR-DATASET-ALIAS", finalized=True)

TypeScript

싱글턴 데이터셋의 경우:

import { EvaluationDataset, Golden } from "deepeval";

const goldens = [new Golden({ input: "How tall is Mt. Everest?" })];
const dataset = new EvaluationDataset({ goldens: goldens });

// Push as finalized (ready for evaluation)
dataset.push({ alias: "YOUR-DATASET-ALIAS", finalized: true });

// Or push as unfinalized (queued for review)
dataset.push({ alias: "YOUR-DATASET-ALIAS", finalized: false });

멀티턴 데이터셋의 경우:

With Turns

import { EvaluationDataset, ConversationalGolden, Turn } from "deepeval";

const firstTurn = new Turn({
  role: "user",
  content: "Where's my money!?",
});

const goldens = [
  new ConversationalGolden({
    scenario: "Angry user asking for a refund.",
    turns: [firstTurn],
  }),
];
const dataset = new EvaluationDataset({ goldens: goldens });

dataset.push({ alias: "YOUR-DATASET-ALIAS", finalized: true });

Without Turns

import { EvaluationDataset, ConversationalGolden } from "deepeval";

const conversationalGolden = new ConversationalGolden({
  scenario: "Angry user asking for a refund.",
});

const goldens = [conversationalGolden];
const dataset = new EvaluationDataset({ goldens: goldens });

dataset.push({ alias: "YOUR-DATASET-ALIAS", finalized: true });

curl

평가 준비가 된 골든스에는 finalized를 true로, 검토 대기열에는 false로 설정해요.

Request (POST /v1/datasets/{alias}) — API reference

curl -X POST "https://api.confident-ai.com/v1/datasets/{alias}" \
  -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>" \
  -H "Content-Type: application/json" \
  -d '{
  "finalized": true,
  "goldens": [
    {
      "input": "How is the weather like in NYC?",
      "expectedOutput": "No idea"
    }
  ]
}'

import requests

response = requests.post(
    "https://api.confident-ai.com/v1/datasets/{alias}",
    headers={
        "CONFIDENT_API_KEY": "<PROJECT-API-KEY>",
    },
    json={
        "finalized": True,
        "goldens": [
            {
                "input": "How is the weather like in NYC?",
                "expectedOutput": "No idea"
            }
        ]
    },
)

print(response.json())

const response = await fetch("https://api.confident-ai.com/v1/datasets/{alias}", {
  method: "POST",
  headers: {
    "CONFIDENT_API_KEY": "<PROJECT-API-KEY>",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    "finalized": true,
    "goldens": [
      {
        "input": "How is the weather like in NYC?",
        "expectedOutput": "No idea"
      }
    ]
  }),
});

const data = await response.json();
console.log(data);

package main

import (
	"fmt"
	"io"
	"net/http"
	"strings"
)

func main() {
	body := `{
  "finalized": true,
  "goldens": [
    {
      "input": "How is the weather like in NYC?",
      "expectedOutput": "No idea"
    }
  ]
}`

	req, err := http.NewRequest("POST", "https://api.confident-ai.com/v1/datasets/{alias}", strings.NewReader(body))
	if err != nil {
		panic(err)
	}
	req.Header.Set("CONFIDENT_API_KEY", "<PROJECT-API-KEY>")
	req.Header.Set("Content-Type", "application/json")

	res, err := http.DefaultClient.Do(req)
	if err != nil {
		panic(err)
	}
	defer res.Body.Close()

	out, err := io.ReadAll(res.Body)
	if err != nil {
		panic(err)
	}

	fmt.Println(string(out))
}

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;

public class Example {
    public static void main(String[] args) throws Exception {
        String body = """
            {
              "finalized": true,
              "goldens": [
                {
                  "input": "How is the weather like in NYC?",
                  "expectedOutput": "No idea"
                }
              ]
            }""";

        HttpRequest request = HttpRequest.newBuilder()
            .uri(URI.create("https://api.confident-ai.com/v1/datasets/{alias}"))
            .header("CONFIDENT_API_KEY", "<PROJECT-API-KEY>")
            .header("Content-Type", "application/json")
            .POST(HttpRequest.BodyPublishers.ofString(body))
            .build();

        HttpResponse<String> response = HttpClient.newHttpClient()
            .send(request, HttpResponse.BodyHandlers.ofString());

        System.out.println(response.body());
    }
}

use serde_json::json;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let response = reqwest::Client::new()
        .post("https://api.confident-ai.com/v1/datasets/{alias}")
        .header("CONFIDENT_API_KEY", "<PROJECT-API-KEY>")
        .json(&json!({
          "finalized": true,
          "goldens": [
            {
              "input": "How is the weather like in NYC?",
              "expectedOutput": "No idea"
            }
          ]
        }))
        .send()
        .await?;

    println!("{}", response.text().await?);

    Ok(())
}

멀티턴 데이터셋의 경우:

Request (POST /v1/datasets/{alias}) — API reference

curl -X POST "https://api.confident-ai.com/v1/datasets/{alias}" \
  -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>" \
  -H "Content-Type: application/json" \
  -d '{
  "finalized": true,
  "conversationalGoldens": [
    {
      "scenario": "Booking a hotel",
      "expectedOutcome": "Successfully booked",
      "userDescription": "Finds hotels in NYC"
    }
  ]
}'

import requests

response = requests.post(
    "https://api.confident-ai.com/v1/datasets/{alias}",
    headers={
        "CONFIDENT_API_KEY": "<PROJECT-API-KEY>",
    },
    json={
        "finalized": True,
        "conversationalGoldens": [
            {
                "scenario": "Booking a hotel",
                "expectedOutcome": "Successfully booked",
                "userDescription": "Finds hotels in NYC"
            }
        ]
    },
)

print(response.json())

const response = await fetch("https://api.confident-ai.com/v1/datasets/{alias}", {
  method: "POST",
  headers: {
    "CONFIDENT_API_KEY": "<PROJECT-API-KEY>",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    "finalized": true,
    "conversationalGoldens": [
      {
        "scenario": "Booking a hotel",
        "expectedOutcome": "Successfully booked",
        "userDescription": "Finds hotels in NYC"
      }
    ]
  }),
});

const data = await response.json();
console.log(data);

package main

import (
	"fmt"
	"io"
	"net/http"
	"strings"
)

func main() {
	body := `{
  "finalized": true,
  "conversationalGoldens": [
    {
      "scenario": "Booking a hotel",
      "expectedOutcome": "Successfully booked",
      "userDescription": "Finds hotels in NYC"
    }
  ]
}`

	req, err := http.NewRequest("POST", "https://api.confident-ai.com/v1/datasets/{alias}", strings.NewReader(body))
	if err != nil {
		panic(err)
	}
	req.Header.Set("CONFIDENT_API_KEY", "<PROJECT-API-KEY>")
	req.Header.Set("Content-Type", "application/json")

	res, err := http.DefaultClient.Do(req)
	if err != nil {
		panic(err)
	}
	defer res.Body.Close()

	out, err := io.ReadAll(res.Body)
	if err != nil {
		panic(err)
	}

	fmt.Println(string(out))
}

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;

public class Example {
    public static void main(String[] args) throws Exception {
        String body = """
            {
              "finalized": true,
              "conversationalGoldens": [
                {
                  "scenario": "Booking a hotel",
                  "expectedOutcome": "Successfully booked",
                  "userDescription": "Finds hotels in NYC"
                }
              ]
            }""";

        HttpRequest request = HttpRequest.newBuilder()
            .uri(URI.create("https://api.confident-ai.com/v1/datasets/{alias}"))
            .header("CONFIDENT_API_KEY", "<PROJECT-API-KEY>")
            .header("Content-Type", "application/json")
            .POST(HttpRequest.BodyPublishers.ofString(body))
            .build();

        HttpResponse<String> response = HttpClient.newHttpClient()
            .send(request, HttpResponse.BodyHandlers.ofString());

        System.out.println(response.body());
    }
}

use serde_json::json;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let response = reqwest::Client::new()
        .post("https://api.confident-ai.com/v1/datasets/{alias}")
        .header("CONFIDENT_API_KEY", "<PROJECT-API-KEY>")
        .json(&json!({
          "finalized": true,
          "conversationalGoldens": [
            {
              "scenario": "Booking a hotel",
              "expectedOutcome": "Successfully booked",
              "userDescription": "Finds hotels in NYC"
            }
          ]
        }))
        .send()
        .await?;

    println!("{}", response.text().await?);

    Ok(())
}

커스텀 열 추가 (Add Custom Columns)

골든스를 푸시할 때 커스텀 열 값을 포함할 수 있어요. 커스텀 열은 데이터셋에 이미 존재해야 하며, 없으면 Confident AI가 만들어줘요.

Python

from deepeval.dataset import Golden, ConversationalGolden

golden = Golden(
    input="How tall is Mt. Everest?",
    custom_column_key_values={"difficulty": "easy", "category": "geography"}
)

multiturn_golden = ConversationalGolden(
    scenario="User asking for a refund.",
    custom_column_key_values={"sentiment": "angry", "priority": "high"}
)

TypeScript

import { Golden, ConversationalGolden } from "deepeval";

const golden = new Golden({
  input: "How tall is Mt. Everest?",
  customColumnKeyValues: { difficulty: "easy", category: "geography" },
});

const multiturnGolden = new ConversationalGolden({
  scenario: "User asking for a refund.",
  customColumnKeyValues: { sentiment: "angry", priority: "high" },
});

curl

Request (POST /v1/datasets/{alias}) — API reference

curl -X POST "https://api.confident-ai.com/v1/datasets/{alias}" \
  -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>" \
  -H "Content-Type: application/json" \
  -d '{
  "finalized": true,
  "goldens": [
    {
      "input": "What is 2 + 2?",
      "customColumnKeyValues": {
        "key": "value"
      }
    }
  ]
}'

import requests

response = requests.post(
    "https://api.confident-ai.com/v1/datasets/{alias}",
    headers={
        "CONFIDENT_API_KEY": "<PROJECT-API-KEY>",
    },
    json={
        "finalized": True,
        "goldens": [
            {
                "input": "What is 2 + 2?",
                "customColumnKeyValues": {
                    "key": "value"
                }
            }
        ]
    },
)

print(response.json())

const response = await fetch("https://api.confident-ai.com/v1/datasets/{alias}", {
  method: "POST",
  headers: {
    "CONFIDENT_API_KEY": "<PROJECT-API-KEY>",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    "finalized": true,
    "goldens": [
      {
        "input": "What is 2 + 2?",
        "customColumnKeyValues": {
          "key": "value"
        }
      }
    ]
  }),
});

const data = await response.json();
console.log(data);

package main

import (
	"fmt"
	"io"
	"net/http"
	"strings"
)

func main() {
	body := `{
  "finalized": true,
  "goldens": [
    {
      "input": "What is 2 + 2?",
      "customColumnKeyValues": {
        "key": "value"
      }
    }
  ]
}`

	req, err := http.NewRequest("POST", "https://api.confident-ai.com/v1/datasets/{alias}", strings.NewReader(body))
	if err != nil {
		panic(err)
	}
	req.Header.Set("CONFIDENT_API_KEY", "<PROJECT-API-KEY>")
	req.Header.Set("Content-Type", "application/json")

	res, err := http.DefaultClient.Do(req)
	if err != nil {
		panic(err)
	}
	defer res.Body.Close()

	out, err := io.ReadAll(res.Body)
	if err != nil {
		panic(err)
	}

	fmt.Println(string(out))
}

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;

public class Example {
    public static void main(String[] args) throws Exception {
        String body = """
            {
              "finalized": true,
              "goldens": [
                {
                  "input": "What is 2 + 2?",
                  "customColumnKeyValues": {
                    "key": "value"
                  }
                }
              ]
            }""";

        HttpRequest request = HttpRequest.newBuilder()
            .uri(URI.create("https://api.confident-ai.com/v1/datasets/{alias}"))
            .header("CONFIDENT_API_KEY", "<PROJECT-API-KEY>")
            .header("Content-Type", "application/json")
            .POST(HttpRequest.BodyPublishers.ofString(body))
            .build();

        HttpResponse<String> response = HttpClient.newHttpClient()
            .send(request, HttpResponse.BodyHandlers.ofString());

        System.out.println(response.body());
    }
}

use serde_json::json;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let response = reqwest::Client::new()
        .post("https://api.confident-ai.com/v1/datasets/{alias}")
        .header("CONFIDENT_API_KEY", "<PROJECT-API-KEY>")
        .json(&json!({
          "finalized": true,
          "goldens": [
            {
              "input": "What is 2 + 2?",
              "customColumnKeyValues": {
                "key": "value"
              }
            }
          ]
        }))
        .send()
        .await?;

    println!("{}", response.text().await?);

    Ok(())
}

데이터셋 버전 관리 (Versioning Datasets)

데이터셋은 불변(immutable)이고 이름 있는 버전을 지원해, 평가 실행을 골든스의 특정 스냅샷에 고정할 수 있어요.

버전 만들기: 데이터셋의 현재 상태를 스냅샷으로 만드세요.
version 없이 푸시: 최신 버전(또는 데이터셋에 버전이 없으면 비버전)에 골든스를 추가해요.
version=...으로 푸시: 특정 버전에 골든스를 추가해요.
version 없이 가져오기: 최신 버전을 읽어요. version=...으로 가져오기: 특정 버전을 읽어요.
버전 가져오기: 모든 스냅샷을 새것부터 나열해요.

버전 만들기 (Create a version)

Python

from deepeval.dataset import EvaluationDataset

dataset = EvaluationDataset()
version = dataset.create_version(alias="YOUR-DATASET-ALIAS")
# version -> "00.00.01"

TypeScript

import { EvaluationDataset } from "deepeval";

const dataset = new EvaluationDataset();
const { version } = await dataset.createVersion({ alias: "YOUR-DATASET-ALIAS" });
// version -> "00.00.01"

curl

Request (POST /v1/datasets/{alias}/versions) — API reference

curl -X POST "https://api.confident-ai.com/v1/datasets/{alias}/versions" \
  -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>"

import requests

response = requests.post(
    "https://api.confident-ai.com/v1/datasets/{alias}/versions",
    headers={
        "CONFIDENT_API_KEY": "<PROJECT-API-KEY>",
    },
)

print(response.json())

const response = await fetch("https://api.confident-ai.com/v1/datasets/{alias}/versions", {
  method: "POST",
  headers: {
    "CONFIDENT_API_KEY": "<PROJECT-API-KEY>",
  },
});

const data = await response.json();
console.log(data);

package main

import (
	"fmt"
	"io"
	"net/http"
)

func main() {
	req, err := http.NewRequest("POST", "https://api.confident-ai.com/v1/datasets/{alias}/versions", nil)
	if err != nil {
		panic(err)
	}
	req.Header.Set("CONFIDENT_API_KEY", "<PROJECT-API-KEY>")

	res, err := http.DefaultClient.Do(req)
	if err != nil {
		panic(err)
	}
	defer res.Body.Close()

	out, err := io.ReadAll(res.Body)
	if err != nil {
		panic(err)
	}

	fmt.Println(string(out))
}

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;

public class Example {
    public static void main(String[] args) throws Exception {
        HttpRequest request = HttpRequest.newBuilder()
            .uri(URI.create("https://api.confident-ai.com/v1/datasets/{alias}/versions"))
            .header("CONFIDENT_API_KEY", "<PROJECT-API-KEY>")
            .POST(HttpRequest.BodyPublishers.noBody())
            .build();

        HttpResponse<String> response = HttpClient.newHttpClient()
            .send(request, HttpResponse.BodyHandlers.ofString());

        System.out.println(response.body());
    }
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let response = reqwest::Client::new()
        .post("https://api.confident-ai.com/v1/datasets/{alias}/versions")
        .header("CONFIDENT_API_KEY", "<PROJECT-API-KEY>")
        .send()
        .await?;

    println!("{}", response.text().await?);

    Ok(())
}

create_version의 첫 호출은 기존의 모든 비버전 골든스를 새 버전으로 백필(backfill)해요. 이후 호출은 이전 버전의 모든 골든스(새 ID로)를 스냅샷하고 버전 번호를 자동 증가시켜요.

버전 나열 (List versions)

Python

from deepeval.dataset import EvaluationDataset

dataset = EvaluationDataset()
versions = dataset.get_versions(alias="YOUR-DATASET-ALIAS")
for v in versions:
    print(v.version, v.id)

TypeScript

import { EvaluationDataset } from "deepeval";

const dataset = new EvaluationDataset();
const versions = await dataset.getVersions({ alias: "YOUR-DATASET-ALIAS" });
for (const v of versions) {
  console.log(v.version, v.id);
}

curl

Request (GET /v1/datasets/{alias}/versions) — API reference

curl -X GET "https://api.confident-ai.com/v1/datasets/{alias}/versions" \
  -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>"

import requests

response = requests.get(
    "https://api.confident-ai.com/v1/datasets/{alias}/versions",
    headers={
        "CONFIDENT_API_KEY": "<PROJECT-API-KEY>",
    },
)

print(response.json())

const response = await fetch("https://api.confident-ai.com/v1/datasets/{alias}/versions", {
  method: "GET",
  headers: {
    "CONFIDENT_API_KEY": "<PROJECT-API-KEY>",
  },
});

const data = await response.json();
console.log(data);

package main

import (
	"fmt"
	"io"
	"net/http"
)

func main() {
	req, err := http.NewRequest("GET", "https://api.confident-ai.com/v1/datasets/{alias}/versions", nil)
	if err != nil {
		panic(err)
	}
	req.Header.Set("CONFIDENT_API_KEY", "<PROJECT-API-KEY>")

	res, err := http.DefaultClient.Do(req)
	if err != nil {
		panic(err)
	}
	defer res.Body.Close()

	out, err := io.ReadAll(res.Body)
	if err != nil {
		panic(err)
	}

	fmt.Println(string(out))
}

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;

public class Example {
    public static void main(String[] args) throws Exception {
        HttpRequest request = HttpRequest.newBuilder()
            .uri(URI.create("https://api.confident-ai.com/v1/datasets/{alias}/versions"))
            .header("CONFIDENT_API_KEY", "<PROJECT-API-KEY>")
            .GET()
            .build();

        HttpResponse<String> response = HttpClient.newHttpClient()
            .send(request, HttpResponse.BodyHandlers.ofString());

        System.out.println(response.body());
    }
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let response = reqwest::Client::new()
        .get("https://api.confident-ai.com/v1/datasets/{alias}/versions")
        .header("CONFIDENT_API_KEY", "<PROJECT-API-KEY>")
        .send()
        .await?;

    println!("{}", response.text().await?);

    Ok(())
}

특정 버전 푸시 및 가져오기 (Push and pull a specific version)

Python

from deepeval.dataset import EvaluationDataset, Golden

dataset = EvaluationDataset(goldens=[Golden(input="...", expected_output="...")])

# Push goldens onto version 00.00.01
dataset.push(alias="YOUR-DATASET-ALIAS", version="00.00.01")

# Pull a specific version
dataset.pull(alias="YOUR-DATASET-ALIAS", version="00.00.01")
print(dataset._version)  # -> "00.00.01"

TypeScript

import { EvaluationDataset, Golden } from "deepeval";

const dataset = new EvaluationDataset();
dataset.addGolden(new Golden({ input: "...", expectedOutput: "..." }));

// Push goldens onto version 00.00.01
await dataset.push({ alias: "YOUR-DATASET-ALIAS", version: "00.00.01" });

// Pull a specific version
await dataset.pull({ alias: "YOUR-DATASET-ALIAS", version: "00.00.01" });

version이 생략되면 푸시와 가져오기는 최신 버전을 대상으로 해요. 데이터셋에 아직 버전이 없으면 푸시는 골든스를 비버전으로 남기고, 가져오기는 version: null과 함께 그 비버전 골든스를 반환해요.

골든스 수정 (Update a Golden)

id로 단일 골든스를 제자리에서 수정해요. 먼저 데이터셋을 가져와 각 골든스가 안정적인 id를 가지게 한 뒤, 원하는 필드를 편집하고 업데이트를 보내요.

골든스의 필드는 보낸 값으로 교체돼요. tags와 커스텀 열은 포함할 때만 변경돼요.

Python

from deepeval.dataset import EvaluationDataset

dataset = EvaluationDataset()
dataset.pull(alias="YOUR-DATASET-ALIAS")

# Each pulled golden carries a stable id
golden = dataset.goldens[0]
golden.input = "How tall is Mt. Everest, in meters?"

dataset.update_golden(golden=golden)

TypeScript

import { EvaluationDataset } from "deepeval";

const dataset = new EvaluationDataset();
await dataset.pull({ alias: "YOUR-DATASET-ALIAS" });

// Each pulled golden carries a stable id
const golden = dataset.goldens[0];
golden.input = "How tall is Mt. Everest, in meters?";

await dataset.updateGolden({ golden });

curl

싱글턴 데이터셋의 경우:

Request (PUT /v1/datasets/{alias}/goldens/{goldenId}) — API reference

curl -X PUT "https://api.confident-ai.com/v1/datasets/{alias}/goldens/{goldenId}" \
  -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>" \
  -H "Content-Type: application/json" \
  -d '{
  "input": "How is the weather like in NYC?",
  "expectedOutput": "Sunny with a chance of rain.",
  "finalized": true
}'

import requests

response = requests.put(
    "https://api.confident-ai.com/v1/datasets/{alias}/goldens/{goldenId}",
    headers={
        "CONFIDENT_API_KEY": "<PROJECT-API-KEY>",
    },
    json={
        "input": "How is the weather like in NYC?",
        "expectedOutput": "Sunny with a chance of rain.",
        "finalized": True
    },
)

print(response.json())

const response = await fetch("https://api.confident-ai.com/v1/datasets/{alias}/goldens/{goldenId}", {
  method: "PUT",
  headers: {
    "CONFIDENT_API_KEY": "<PROJECT-API-KEY>",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    "input": "How is the weather like in NYC?",
    "expectedOutput": "Sunny with a chance of rain.",
    "finalized": true
  }),
});

const data = await response.json();
console.log(data);

package main

import (
	"fmt"
	"io"
	"net/http"
	"strings"
)

func main() {
	body := `{
  "input": "How is the weather like in NYC?",
  "expectedOutput": "Sunny with a chance of rain.",
  "finalized": true
}`

	req, err := http.NewRequest("PUT", "https://api.confident-ai.com/v1/datasets/{alias}/goldens/{goldenId}", strings.NewReader(body))
	if err != nil {
		panic(err)
	}
	req.Header.Set("CONFIDENT_API_KEY", "<PROJECT-API-KEY>")
	req.Header.Set("Content-Type", "application/json")

	res, err := http.DefaultClient.Do(req)
	if err != nil {
		panic(err)
	}
	defer res.Body.Close()

	out, err := io.ReadAll(res.Body)
	if err != nil {
		panic(err)
	}

	fmt.Println(string(out))
}

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;

public class Example {
    public static void main(String[] args) throws Exception {
        String body = """
            {
              "input": "How is the weather like in NYC?",
              "expectedOutput": "Sunny with a chance of rain.",
              "finalized": true
            }""";

        HttpRequest request = HttpRequest.newBuilder()
            .uri(URI.create("https://api.confident-ai.com/v1/datasets/{alias}/goldens/{goldenId}"))
            .header("CONFIDENT_API_KEY", "<PROJECT-API-KEY>")
            .header("Content-Type", "application/json")
            .PUT(HttpRequest.BodyPublishers.ofString(body))
            .build();

        HttpResponse<String> response = HttpClient.newHttpClient()
            .send(request, HttpResponse.BodyHandlers.ofString());

        System.out.println(response.body());
    }
}

use serde_json::json;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let response = reqwest::Client::new()
        .put("https://api.confident-ai.com/v1/datasets/{alias}/goldens/{goldenId}")
        .header("CONFIDENT_API_KEY", "<PROJECT-API-KEY>")
        .json(&json!({
          "input": "How is the weather like in NYC?",
          "expectedOutput": "Sunny with a chance of rain.",
          "finalized": true
        }))
        .send()
        .await?;

    println!("{}", response.text().await?);

    Ok(())
}

멀티턴 데이터셋의 경우:

Request (PUT /v1/datasets/{alias}/goldens/{goldenId}) — API reference

curl -X PUT "https://api.confident-ai.com/v1/datasets/{alias}/goldens/{goldenId}" \
  -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>" \
  -H "Content-Type: application/json" \
  -d '{
  "scenario": "Booking a hotel",
  "expectedOutcome": "Successfully booked",
  "userDescription": "Finds hotels in NYC",
  "finalized": true
}'

import requests

response = requests.put(
    "https://api.confident-ai.com/v1/datasets/{alias}/goldens/{goldenId}",
    headers={
        "CONFIDENT_API_KEY": "<PROJECT-API-KEY>",
    },
    json={
        "scenario": "Booking a hotel",
        "expectedOutcome": "Successfully booked",
        "userDescription": "Finds hotels in NYC",
        "finalized": True
    },
)

print(response.json())

const response = await fetch("https://api.confident-ai.com/v1/datasets/{alias}/goldens/{goldenId}", {
  method: "PUT",
  headers: {
    "CONFIDENT_API_KEY": "<PROJECT-API-KEY>",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    "scenario": "Booking a hotel",
    "expectedOutcome": "Successfully booked",
    "userDescription": "Finds hotels in NYC",
    "finalized": true
  }),
});

const data = await response.json();
console.log(data);

package main

import (
	"fmt"
	"io"
	"net/http"
	"strings"
)

func main() {
	body := `{
  "scenario": "Booking a hotel",
  "expectedOutcome": "Successfully booked",
  "userDescription": "Finds hotels in NYC",
  "finalized": true
}`

	req, err := http.NewRequest("PUT", "https://api.confident-ai.com/v1/datasets/{alias}/goldens/{goldenId}", strings.NewReader(body))
	if err != nil {
		panic(err)
	}
	req.Header.Set("CONFIDENT_API_KEY", "<PROJECT-API-KEY>")
	req.Header.Set("Content-Type", "application/json")

	res, err := http.DefaultClient.Do(req)
	if err != nil {
		panic(err)
	}
	defer res.Body.Close()

	out, err := io.ReadAll(res.Body)
	if err != nil {
		panic(err)
	}

	fmt.Println(string(out))
}

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;

public class Example {
    public static void main(String[] args) throws Exception {
        String body = """
            {
              "scenario": "Booking a hotel",
              "expectedOutcome": "Successfully booked",
              "userDescription": "Finds hotels in NYC",
              "finalized": true
            }""";

        HttpRequest request = HttpRequest.newBuilder()
            .uri(URI.create("https://api.confident-ai.com/v1/datasets/{alias}/goldens/{goldenId}"))
            .header("CONFIDENT_API_KEY", "<PROJECT-API-KEY>")
            .header("Content-Type", "application/json")
            .PUT(HttpRequest.BodyPublishers.ofString(body))
            .build();

        HttpResponse<String> response = HttpClient.newHttpClient()
            .send(request, HttpResponse.BodyHandlers.ofString());

        System.out.println(response.body());
    }
}

use serde_json::json;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let response = reqwest::Client::new()
        .put("https://api.confident-ai.com/v1/datasets/{alias}/goldens/{goldenId}")
        .header("CONFIDENT_API_KEY", "<PROJECT-API-KEY>")
        .json(&json!({
          "scenario": "Booking a hotel",
          "expectedOutcome": "Successfully booked",
          "userDescription": "Finds hotels in NYC",
          "finalized": true
        }))
        .send()
        .await?;

    println!("{}", response.text().await?);

    Ok(())
}

골든스 삭제 (Delete a Golden)

id로 데이터셋에서 단일 골든스를 제거해요. 그 골든스만 제거되며 데이터셋의 나머지는 그대로예요.

Python

from deepeval.dataset import EvaluationDataset

dataset = EvaluationDataset()
dataset.pull(alias="YOUR-DATASET-ALIAS")

golden = dataset.goldens[0]
dataset.delete_golden(golden=golden)

# Or delete by id directly
dataset.delete_golden(golden="GOLDEN-ID")

TypeScript

import { EvaluationDataset } from "deepeval";

const dataset = new EvaluationDataset();
await dataset.pull({ alias: "YOUR-DATASET-ALIAS" });

const golden = dataset.goldens[0];
await dataset.deleteGolden({ golden });

// Or delete by id directly
await dataset.deleteGolden({ golden: "GOLDEN-ID" });

curl

Request (DELETE /v1/datasets/{alias}/goldens/{goldenId}) — API reference

curl -X DELETE "https://api.confident-ai.com/v1/datasets/{alias}/goldens/{goldenId}" \
  -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>"

import requests

response = requests.delete(
    "https://api.confident-ai.com/v1/datasets/{alias}/goldens/{goldenId}",
    headers={
        "CONFIDENT_API_KEY": "<PROJECT-API-KEY>",
    },
)

print(response.json())

const response = await fetch("https://api.confident-ai.com/v1/datasets/{alias}/goldens/{goldenId}", {
  method: "DELETE",
  headers: {
    "CONFIDENT_API_KEY": "<PROJECT-API-KEY>",
  },
});

const data = await response.json();
console.log(data);

package main

import (
	"fmt"
	"io"
	"net/http"
)

func main() {
	req, err := http.NewRequest("DELETE", "https://api.confident-ai.com/v1/datasets/{alias}/goldens/{goldenId}", nil)
	if err != nil {
		panic(err)
	}
	req.Header.Set("CONFIDENT_API_KEY", "<PROJECT-API-KEY>")

	res, err := http.DefaultClient.Do(req)
	if err != nil {
		panic(err)
	}
	defer res.Body.Close()

	out, err := io.ReadAll(res.Body)
	if err != nil {
		panic(err)
	}

	fmt.Println(string(out))
}

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;

public class Example {
    public static void main(String[] args) throws Exception {
        HttpRequest request = HttpRequest.newBuilder()
            .uri(URI.create("https://api.confident-ai.com/v1/datasets/{alias}/goldens/{goldenId}"))
            .header("CONFIDENT_API_KEY", "<PROJECT-API-KEY>")
            .DELETE()
            .build();

        HttpResponse<String> response = HttpClient.newHttpClient()
            .send(request, HttpResponse.BodyHandlers.ofString());

        System.out.println(response.body());
    }
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let response = reqwest::Client::new()
        .delete("https://api.confident-ai.com/v1/datasets/{alias}/goldens/{goldenId}")
        .header("CONFIDENT_API_KEY", "<PROJECT-API-KEY>")
        .send()
        .await?;

    println!("{}", response.text().await?);

    Ok(())
}

이 작업은 되돌릴 수 없어요. 골든스는 데이터셋에서 영구 제거돼요.

데이터셋 삭제 (Delete Dataset)

Confident API를 통해 프로그래밍 방식으로 데이터셋을 삭제해요.

이 작업은 되돌릴 수 없어요. 데이터셋의 모든 골든스 또는 대화형 골든스가 영구 삭제돼요.

Python

from deepeval.dataset import EvaluationDataset

dataset = EvaluationDataset()
dataset.delete(alias="YOUR-DATASET-ALIAS")

TypeScript

import { EvaluationDataset } from "deepeval";

const dataset = new EvaluationDataset();
dataset.delete({ alias: "YOUR-DATASET-ALIAS" });

curl

Request (DELETE /v1/datasets/{alias}) — API reference

curl -X DELETE "https://api.confident-ai.com/v1/datasets/{alias}" \
  -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>"

import requests

response = requests.delete(
    "https://api.confident-ai.com/v1/datasets/{alias}",
    headers={
        "CONFIDENT_API_KEY": "<PROJECT-API-KEY>",
    },
)

print(response.json())

const response = await fetch("https://api.confident-ai.com/v1/datasets/{alias}", {
  method: "DELETE",
  headers: {
    "CONFIDENT_API_KEY": "<PROJECT-API-KEY>",
  },
});

const data = await response.json();
console.log(data);

package main

import (
	"fmt"
	"io"
	"net/http"
)

func main() {
	req, err := http.NewRequest("DELETE", "https://api.confident-ai.com/v1/datasets/{alias}", nil)
	if err != nil {
		panic(err)
	}
	req.Header.Set("CONFIDENT_API_KEY", "<PROJECT-API-KEY>")

	res, err := http.DefaultClient.Do(req)
	if err != nil {
		panic(err)
	}
	defer res.Body.Close()

	out, err := io.ReadAll(res.Body)
	if err != nil {
		panic(err)
	}

	fmt.Println(string(out))
}

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;

public class Example {
    public static void main(String[] args) throws Exception {
        HttpRequest request = HttpRequest.newBuilder()
            .uri(URI.create("https://api.confident-ai.com/v1/datasets/{alias}"))
            .header("CONFIDENT_API_KEY", "<PROJECT-API-KEY>")
            .DELETE()
            .build();

        HttpResponse<String> response = HttpClient.newHttpClient()
            .send(request, HttpResponse.BodyHandlers.ofString());

        System.out.println(response.body());
    }
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let response = reqwest::Client::new()
        .delete("https://api.confident-ai.com/v1/datasets/{alias}")
        .header("CONFIDENT_API_KEY", "<PROJECT-API-KEY>")
        .send()
        .await?;

    println!("{}", response.text().await?);

    Ok(())
}

프로젝트 전환 (Switching Projects)

CONFIDENT_API_KEY를 구성해 어떤 프로젝트에서든 데이터셋을 푸시하거나 관리할 수 있어요.

기본 사용 시 CONFIDENT_API_KEY를 환경 변수로 설정하세요.
특정 프로젝트를 대상으로 하려면 EvaluationDataset을 만들 때 confident_api_key를 직접 전달하세요.

Python

from deepeval.dataset import EvaluationDataset

dataset = EvaluationDataset(confident_api_key="confident_us...")

TypeScript

import { EvaluationDataset } from "deepeval";

const dataset = new EvaluationDataset({ confidenApiKey: "confident_us..." });

둘 다 제공되면 EvaluationDataset에 전달한 confident_api_key가 환경 변수보다 항상 우선해요.

다음 단계

골든스를 푸시하는 방법을 알았으니, 평가를 위해 가져오는 방법을 배워볼게요.

데이터셋 가져오기 (Pull Datasets)

데이터셋을 로컬로 가져와 코드 기반 평가에 사용해요.

데이터셋 관리 자동화