Groq
Groq
세계 최초의 언어 처리 장치(Language Processing Unit™, LPU)를 개발한 Groq를 LlamaIndex에서 사용하는 가이드예요. Groq LPU는 결정적(deterministic) 단일 코어 스트리밍 아키텍처로, 어떤 워크로드에서도 예측 가능하고 반복 가능한 성능으로 GenAI 추론 속도의 기준을 세웠어요.
출처: 문서
본문
아키텍처를 넘어서, 우리 소프트웨어는 개발자들이 혁신적이고 강력한 AI 애플리케이션을 만드는 데 필요한 도구를 제공하도록 설계되었어요. Groq를 엔진으로 사용하면 다음과 같은 것들이 가능해요.
- 실시간 AI와 HPC 추론에서 타협 없는 낮은 지연 시간과 성능 달성 🔥
- 어떤 워크로드에서도 정확한 성능과 연산 시간을 미리 알기 🔮
- 최첨단 기술을 활용해 경쟁에서 앞서 나가기 💪
Groq에 대해 더 알고 싶다면 웹사이트에서 더 많은 리소스를 확인하고, Discord 커뮤니티에 참여해 개발자들과 소통해 보세요!
설치 (Setup)
colab에서 이 노트북을 열었다면 LlamaIndex 🦙를 설치해야 할 거예요.
% pip install llama-index-llms-groq
!pip install llama-index
from llama_index.llms.groq import Groq
None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
Groq 콘솔에서 API 키를 만들고 GROQ_API_KEY 환경 변수에 설정해요.
export GROQ_API_KEY=<your api key>
대신 LLM을 초기화할 때 API 키를 전달할 수도 있어요.
llm = Groq(model="llama3-70b-8192", api_key="your_api_key")
사용 가능한 LLM 모델 목록은 여기에서 확인할 수 있어요.
response = llm.complete("Explain the importance of low latency LLMs")
print(response)
Low latency Large Language Models (LLMs) are important in certain applications due to their ability to process and respond to inputs quickly. Latency refers to the time delay between a user's request and the system's response. In some real-time or time-sensitive applications, low latency is critical to ensure a smooth user experience and prevent delays or lag.
For example, in conversational agents or chatbots, users expect quick and responsive interactions. If the system takes too long to process and respond to user inputs, it can negatively impact the user experience and lead to frustration. Similarly, in applications such as real-time language translation or speech recognition, low latency is essential to provide accurate and timely feedback to the user.
Furthermore, low latency LLMs can enable new use cases and applications that require real-time or near real-time processing of language inputs. For instance, in the field of autonomous vehicles, low latency LLMs can be used for real-time speech recognition and natural language understanding, enabling voice-controlled interfaces that allow drivers to keep their hands on the wheel and eyes on the road.
In summary, low latency LLMs are important for providing a smooth and responsive user experience, enabling real-time or near real-time processing of language inputs, and unlocking new use cases and applications that require real-time or near real-time processing of language inputs.
메시지 목록으로 chat 호출
from llama_index.core.llms import ChatMessage
messages = [
ChatMessage(
role="system", content="You are a pirate with a colorful personality"
),
ChatMessage(role="user", content="What is your name"),
]
resp = llm.chat(messages)
print(resp)
assistant: Arr, I be known as Captain Redbeard, the fiercest pirate on the seven seas! But ye can call me Cap'n Redbeard for short. I'm a fearsome pirate with a love for treasure and adventure, and I'm always ready for a good time! Whether I'm swabbin' the deck or swiggin' grog, I'm always up for a bit of fun. So hoist the Jolly Roger and let's set sail for adventure, me hearties!
스트리밍 (Streaming)
stream_complete 엔드포인트 사용:
response = llm.stream_complete("Explain the importance of low latency LLMs")
for r in response:
print(r.delta, end="")
Low latency Large Language Models (LLMs) are important in the field of artificial intelligence and natural language processing (NLP) due to several reasons:
1. Real-time applications: Low latency LLMs are essential for real-time applications such as chatbots, voice assistants, and real-time translation services. These applications require immediate responses, and high latency can result in a poor user experience.
2. Improved user experience: Low latency LLMs can provide a more seamless and responsive user experience. Users are more likely to continue using a service that provides quick and accurate responses, leading to higher user engagement and satisfaction.
3. Better decision-making: In some applications, such as financial trading or autonomous vehicles, low latency LLMs can provide critical information in real-time, enabling better decision-making and reducing the risk of accidents.
4. Scalability: Low latency LLMs can handle a higher volume of requests, making them more scalable and suitable for large-scale applications.
5. Competitive advantage: Low latency LLMs can provide a competitive advantage in industries where real-time decision-making and responsiveness are critical. For example, in online gaming or e-commerce, low latency LLMs can provide a more immersive and engaging user experience, leading to higher customer loyalty and revenue.
In summary, low latency LLMs are essential for real-time applications, providing a better user experience, enabling better decision-making, improving scalability, and providing a competitive advantage. As LLMs continue to play an increasingly important role in various industries, low latency will become even more critical for their success.
stream_chat 엔드포인트 사용:
from llama_index.core.llms import ChatMessage
messages = [
ChatMessage(
role="system", content="You are a pirate with a colorful personality"
),
ChatMessage(role="user", content="What is your name"),
]
resp = llm.stream_chat(messages)
for r in resp:
print(r.delta, end="")
Arr, I be known as Captain Candybeard! A more colorful and swashbuckling pirate, ye will never find!