Improving the function search

[alex-aipolabs]

We are trying to find better ways of implementing function search. Currently openai embeddings are created by json dumping the fields of the FunctionEmbeddingFields class and passing that to the OPENAI_EMBEDDING_MODE with a OPENAI_EMBEDDING_DIMENSION. class FunctionEmbeddingFields(BaseModel): name: str description: str parameters: dict

The semantic similarity would then be between this embedding and the embedded intent. I wanted to make this a discussion and talk about different solutions here. What would be important is to evaluate the performance of the model by creating a synthetic dataset of intents mapping to the correct functions. I wanted to discuss also if providing many options as a response even makes sense and for which use cases.

[irmadong]

I think another way is to define some taxonomy with the definitions and let llm to do intent classification. We can do unsupervised learning first if we have some available utterances to do mapping and user intent exploration.

[billyzhaoyh]

do you mean if we have a unified layer of intents which are more verbose or LLM friendly versions of the different apps then we can recast this to an intent classification problem potentially?

[billyzhaoyh]

Just looked through the codebase and found the evaluation pipeline currently in place for the search functions. I am wondering if it is possible to populate a hugging face dataset for running evaluation rather then generate synthetic dataset each time? This allows better curation of datasets and can dump different datasets generated by different prompts (also can incorporate annoynimised usage data in there later)

[alex-aipolabs]

Hey @billyzhaoyh, we are actually uploading the dataset to weights and biases which handles versioning. With weights and biases we have a nice history of experiments with metrics and an associated dataset version, this is the link to previous experiments wandb. We originally had the dataset in huggingface but hf doesn't have a built in versioning system and it was difficult to implement well since the dataset will change quite a lot and potentially the implementation of the search aswell so I thought we would use an experimental approach. We could potentially include a webhook to upload the dataset from wandb to huggingface but not sure how valuable that may be currently. What's important right now is to include more prompts for generating synthetic intents of different difficulty