[WIP] v1.0.0 Updates

[orangetin]

To-Do:

• Async client across all classes
• Abstracted engine
• Persistant session
• Client class
• Structured typing
• Chat Completions support
• Update CLI
• Update price estimate function
• Update readme and examples
• Add unit tests
• Add integration tests
• Add pre-commit linter & update github workflow
• Replace requests/aiohttp with httpx
• Allow strict pydantic typing
• add overload for all api classes
• Add timeout to header
• fix CLI comments and help hints

Endpoints to support:

• Completions
• Chat Completions
• Embeddings
• Finetune
• Files
• Images
• Models

Example usage:

import os
import json

api_key = os.getenv("TOGETHER_API_KEY")

import together

client = together.Together(
  api_key=api_key
)

# Chat Completions
response = client.chat.completions.create(model="togethercomputer/llama-2-7b-chat", max_tokens=10, messages=[{"role": "user", "content": "hello there"}])
print(response.choices[0].message.content)

# Completions
response = client.completions.create(model="togethercomputer/llama-2-7b", max_tokens=10, prompt="hello there")
print(response.choices[0].text)

# Embeddings
response = client.embeddings.create(model="bert-base-uncased", input=["test"])
print(response.data[0].embedding)

# Fine Tuning
response = client.fine_tuning.create(training_file="file-6e432514-18e8-407d-b36e-ba904e4d4856", model="togethercomputer/llama-2-7b")
print(json.dumps(response.model_dump(), indent=4))

# Files
response = client.files.upload("unified_joke_explanations.jsonl")
print(json.dumps(response.model_dump(), indent=4))

# Images
response = client.images.generate(prompt="space robots", model="stabilityai/stable-diffusion-xl-base-1.0", steps=10, n=4)
print(response.data[0].b64_json)

# Models
response = client.models.list()
print(response[0].id)

Updated contribution style

Setting up pre-commit for dev:

poetry install --with quality,tests
pre-commit install

[linear]

ENG-900 Support messages (chat endpoint) in together python library

Multiple customers are getting confused that using prompt through the together python library uses the raw prompt since the preferred way to do it through the REST API and OpenAI package is using messages which adds prompt formatting.

Therefore, we want to support messages so that all 3 ways of using our inference API are consistent.

More context here: https://www.notion.so/together-docs/Prompt-template-discrepancy-proposal-a557d4fb7f5d49d59a9b79480e0926b9

[clam004]

here is my async demo

import os
import time
from together import Together

TOGETHER_API_KEY = os.getenv('TOGETHER_API_KEY')

def sync_chat_completion(messages, max_tokens):
    client = Together(api_key=TOGETHER_API_KEY)
    
    start_time = time.time()
    
    for message in messages:
        response = client.chat.completions.create(
            model="togethercomputer/llama-2-7b-chat", 
            max_tokens=max_tokens, 
            messages=[{"role": "user", "content": message}]
        )
        print(response.choices[0].message.content)
    
    end_time = time.time()
    print("Synchronous total execution time:", end_time - start_time, "seconds")

async def async_chat_completion(messages, max_tokens):
    async_client = AsyncTogether(api_key=TOGETHER_API_KEY)
    
    start_time = time.time()
    
    tasks = [async_client.chat.completions.create(
                model="togethercomputer/llama-2-7b-chat", 
                max_tokens=max_tokens, 
                messages=[{"role": "user", "content": message}]
             ) for message in messages]
             
    responses = await asyncio.gather(*tasks)
    
    for response in responses:
        print(response.choices[0].message.content)
    
    end_time = time.time()
    print("Asynchronous total execution time:", end_time - start_time, "seconds")

in jupyter notebook

messages = ["hi there what is the meaning of life?", "What country is Paris in?"]
sync_chat_completion(messages, 32)
await async_chat_completion(messages, 32)

otherwise

messages = ["hi there what is the meaning of life?", "What country is Paris in?"]
sync_chat_completion(messages, 32)
asyncio.run(async_chat_completion(messages, 32))

expected output

  The meaning of life is a question that has puzzled philosophers, theologians, and scientists for centuries. There are many different perspectives
  Paris is located in France. It is the capital and largest city of France, situated in the northern central part of the country.
Synchronous total execution time: 0.7738921642303467 seconds
  The meaning of life is a question that has puzzled philosophers, theologians, and scientists for centuries. There are many different perspectives
  Paris is located in France. It is the capital and largest city of France, situated in the northern central part of the country.
Asynchronous total execution time: 0.4429478645324707 seconds

[Nutlope]

Looks great Abhy, amazing work! Feel free to merge, then I can make a new PR to update the README (and update our actual docs), then we can open source this repo + announce.