> ## Documentation Index > Fetch the complete documentation index at: https://docs.coreweave.com/llms.txt > Use this file to discover all available pages before exploring further. # Évaluer des applications RAG > Créez et évaluez des applications RAG avec Weave à l’aide de juges LLM export const GitHubLink = ({url, compact = false}) => {compact ? "View source" : <> GitHub source } ; export const ColabLink = ({url}) => Try in Colab ;
La génération augmentée par récupération (RAG, Retrieval Augmented Generation) est une approche courante pour créer des applications d’IA générative capables d’accéder à des bases de connaissances personnalisées. Ce tutoriel vous accompagne dans la création d’une application RAG et vous montre comment utiliser Weave pour suivre ses étapes de récupération et évaluer ses réponses avec un juge LLM, afin de mesurer et d’améliorer la qualité des réponses renvoyées par votre application. Ce guide s’adresse aux développeurs qui créent des applications RAG et souhaitent doter leurs pipelines d’observabilité et d’une évaluation systématique. Evals hero

Ce que vous allez apprendre

Ce guide vous montre comment : * Construire une base de connaissances. * Créer une application RAG dotée d’une étape de récupération qui trouve les documents pertinents. * Suivre les étapes de récupération avec Weave. * Évaluer des applications RAG à l’aide d’un juge LLM pour mesurer la précision du contexte. * Définir des fonctions de score personnalisées.

Prérequis

* Un [compte CoreWeave Forge](https://id.coreweave.com/signup) * Python 3.10+ ou Node.js 18+ * Les paquets requis installés : * **Python** : `pip install weave openai` * **TypeScript** : `npm install weave openai` * Une [clé API OpenAI](https://platform.openai.com/api-keys) définie dans une variable d’environnement.

Créer une base de connaissances

La base de connaissances est le corpus dans lequel votre application RAG effectue ses recherches au moment de la requête. Dans cette section, vous calculez des embeddings vectoriels pour un petit ensemble d’articles afin de pouvoir ensuite récupérer l’article le plus pertinent pour une question donnée. Commencez par calculer les embeddings des articles. En général, cette opération n’est effectuée qu’une seule fois sur vos articles, puis les embeddings et les métadonnées sont stockés dans une base de données. Ici, par souci de simplicité, elle est répétée à chaque exécution du script. ```python lines theme={"system"} from openai import OpenAI import weave from weave import Model import numpy as np import json import asyncio articles = [ "Novo Nordisk and Eli Lilly rival soars 32 percent after promising weight loss drug results Shares of Denmarks Zealand Pharma shot 32 percent higher in morning trade, after results showed success in its liver disease treatment survodutide, which is also on trial as a drug to treat obesity. The trial “tells us that the 6mg dose is safe, which is the top dose used in the ongoing [Phase 3] obesity trial too,” one analyst said in a note. The results come amid feverish investor interest in drugs that can be used for weight loss.", "Berkshire shares jump after big profit gain as Buffetts conglomerate nears $1 trillion valuation Berkshire Hathaway shares rose on Monday after Warren Buffetts conglomerate posted strong earnings for the fourth quarter over the weekend. Berkshires Class A and B shares jumped more than 1.5%, each. Class A shares are higher by more than 17% this year, while Class B has gained more than 18%. Berkshire was last valued at $930.1 billion, up from $905.5 billion where it closed on Friday, according to FactSet. Berkshire on Saturday posted fourth-quarter operating earnings of $8.481 billion, about 28 percent higher than the $6.625 billion from the year-ago period, driven by big gains in its insurance business. Operating earnings refers to profits from businesses across insurance, railroads and utilities. Meanwhile, Berkshires cash levels also swelled to record levels. The conglomerate held $167.6 billion in cash in the fourth quarter, surpassing the $157.2 billion record the conglomerate held in the prior quarter.", "Highmark Health says its combining tech from Google and Epic to give doctors easier access to information Highmark Health announced it is integrating technology from Google Cloud and the health-care software company Epic Systems. The integration aims to make it easier for both payers and providers to access key information they need, even if its stored across multiple points and formats, the company said. Highmark is the parent company of a health plan with 7 million members, a provider network of 14 hospitals and other entities", "Rivian and Lucid shares plunge after weak EV earnings reports Shares of electric vehicle makers Rivian and Lucid fell Thursday after the companies reported stagnant production in their fourth-quarter earnings after the bell Wednesday. Rivian shares sank about 25 percent, and Lucids stock dropped around 17 percent. Rivian forecast it will make 57,000 vehicles in 2024, slightly less than the 57,232 vehicles it produced in 2023. Lucid said it expects to make 9,000 vehicles in 2024, more than the 8,428 vehicles it made in 2023.", "Mauritius blocks Norwegian cruise ship over fears of a potential cholera outbreak Local authorities on Sunday denied permission for the Norwegian Dawn ship, which has 2,184 passengers and 1,026 crew on board, to access the Mauritius capital of Port Louis, citing “potential health risks.” The Mauritius Ports Authority said Sunday that samples were taken from at least 15 passengers on board the cruise ship. A spokesperson for the U.S.-headquartered Norwegian Cruise Line Holdings said Sunday that 'a small number of guests experienced mild symptoms of a stomach-related illness' during Norwegian Dawns South Africa voyage.", "Intuitive Machines lands on the moon in historic first for a U.S. company Intuitive Machines Nova-C cargo lander, named Odysseus after the mythological Greek hero, is the first U.S. spacecraft to soft land on the lunar surface since 1972. Intuitive Machines is the first company to pull off a moon landing — government agencies have carried out all previously successful missions. The company's stock surged in extended trading Thursday, after falling 11 percent in regular trading.", "Lunar landing photos: Intuitive Machines Odysseus sends back first images from the moon Intuitive Machines cargo moon lander Odysseus returned its first images from the surface. Company executives believe the lander caught its landing gear sideways on the moon's surface while touching down and tipped over. Despite resting on its side, the company's historic IM-1 mission is still operating on the moon.", ] def docs_to_embeddings(docs: list) -> list: openai = OpenAI() document_embeddings = [] for doc in docs: response = ( openai.embeddings.create(input=doc, model="text-embedding-3-small") .data[0] .embedding ) document_embeddings.append(response) return document_embeddings article_embeddings = docs_to_embeddings(articles) # Remarque : en général, cette opération n’est effectuée qu’une seule fois sur vos articles, et les embeddings ainsi que les métadonnées sont ensuite stockés dans une base de données ``` ```typescript twoslash lines theme={"system"} // @noErrors require('dotenv').config(); import { OpenAI } from 'openai'; import * as weave from 'weave'; interface Article { text: string; embedding?: number[]; } const articles: Article[] = [ { text: `Novo Nordisk and Eli Lilly rival soars 32 percent after promising weight loss drug results Shares of Denmarks Zealand Pharma shot 32 percent higher in morning trade, after results showed success in its liver disease treatment survodutide, which is also on trial as a drug to treat obesity. The trial tells us that the 6mg dose is safe, which is the top dose used in the ongoing [Phase 3] obesity trial too, one analyst said in a note. The results come amid feverish investor interest in drugs that can be used for weight loss.` }, { text: `Berkshire shares jump after big profit gain as Buffetts conglomerate nears $1 trillion valuation Berkshire Hathaway shares rose on Monday after Warren Buffetts conglomerate posted strong earnings for the fourth quarter over the weekend. Berkshires Class A and B shares jumped more than 1.5%, each. Class A shares are higher by more than 17% this year, while Class B has gained more than 18%. Berkshire was last valued at $930.1 billion, up from $905.5 billion where it closed on Friday, according to FactSet. Berkshire on Saturday posted fourth-quarter operating earnings of $8.481 billion, about 28 percent higher than the $6.625 billion from the year-ago period, driven by big gains in its insurance business. Operating earnings refers to profits from businesses across insurance, railroads and utilities. Meanwhile, Berkshires cash levels also swelled to record levels. The conglomerate held $167.6 billion in cash in the fourth quarter, surpassing the $157.2 billion record the conglomerate held in the prior quarter.` }, { text: `Highmark Health says its combining tech from Google and Epic to give doctors easier access to information Highmark Health announced it is integrating technology from Google Cloud and the health-care software company Epic Systems. The integration aims to make it easier for both payers and providers to access key information they need, even if its stored across multiple points and formats, the company said. Highmark is the parent company of a health plan with 7 million members, a provider network of 14 hospitals and other entities` } ]; function cosineSimilarity(a: number[], b: number[]): number { const dotProduct = a.reduce((sum, val, i) => sum + val * b[i], 0); const magnitudeA = Math.sqrt(a.reduce((sum, val) => sum + val * val, 0)); const magnitudeB = Math.sqrt(b.reduce((sum, val) => sum + val * val, 0)); return dotProduct / (magnitudeA * magnitudeB); } const docsToEmbeddings = weave.op(async function(docs: Article[]): Promise { const openai = new OpenAI(); const enrichedDocs = await Promise.all(docs.map(async (doc) => { const response = await openai.embeddings.create({ input: doc.text, model: "text-embedding-3-small" }); return { ...doc, embedding: response.data[0].embedding }; })); return enrichedDocs; }); ```

Créer une application RAG

Une fois la base de connaissances en place, vous pouvez créer l’application RAG proprement dite. Cette section associe l’étape de récupération à un appel LLM, puis encapsule l’ensemble avec Weave afin que chaque entrée et chaque sortie soient suivies automatiquement. Ensuite, encapsulez la fonction de récupération `get_most_relevant_document` avec un décorateur `weave.op()` et créez une classe `Model`. En encapsulant la fonction de récupération avec `weave.op()`, vous permettez à Weave de capturer ses entrées et ses sorties à chaque appel, ce qui vous permettra ensuite d’inspecter l’étape de récupération. Appelez `weave.init('/rag-quickstart')` pour commencer à suivre toutes les entrées et sorties de vos fonctions afin de les inspecter ultérieurement. Si vous ne spécifiez pas de nom d’équipe, Weave enregistre la sortie dans votre [équipe ou entity W\&B par défaut](/fr/products/wandb/platform/app/settings-page/user-settings#default-team). ```python lines theme={"system"} from openai import OpenAI import weave from weave import Model import numpy as np import asyncio @weave.op() def get_most_relevant_document(query): openai = OpenAI() query_embedding = ( openai.embeddings.create(input=query, model="text-embedding-3-small") .data[0] .embedding ) similarities = [ np.dot(query_embedding, doc_emb) / (np.linalg.norm(query_embedding) * np.linalg.norm(doc_emb)) for doc_emb in article_embeddings ] # Obtenir l’index du document le plus similaire most_relevant_doc_index = np.argmax(similarities) return articles[most_relevant_doc_index] class RAGModel(Model): system_message: str model_name: str = "gpt-3.5-turbo-1106" @weave.op() def predict(self, question: str) -> dict: # remarque : `question` servira plus tard à sélectionner les données dans les lignes d’évaluation from openai import OpenAI context = get_most_relevant_document(question) client = OpenAI() query = f"""Use the following information to answer the subsequent question. If the answer cannot be found, write "I don't know." Context: \"\"\" {context} \"\"\" Question: {question}""" response = client.chat.completions.create( model=self.model_name, messages=[ {"role": "system", "content": self.system_message}, {"role": "user", "content": query}, ], temperature=0.0, response_format={"type": "text"}, ) answer = response.choices[0].message.content return {'answer': answer, 'context': context} # Définir les noms de votre équipe et de votre projet weave.init('/rag-quickstart') model = RAGModel( system_message="You are an expert in finance and answer questions related to finance, financial services, and financial markets. When responding based on provided information, be sure to cite the source." ) model.predict("What significant result was reported about Zealand Pharma's obesity trial?") ``` ```typescript twoslash lines theme={"system"} // @noErrors class RAGModel { private openai: OpenAI; private systemMessage: string; private modelName: string; private articleEmbeddings: Article[]; constructor(config: { systemMessage: string; modelName?: string; articleEmbeddings: Article[]; }) { this.openai = new OpenAI(); this.systemMessage = config.systemMessage; this.modelName = config.modelName || "gpt-3.5-turbo-1106"; this.articleEmbeddings = config.articleEmbeddings; this.predict = weave.op(this, this.predict); } async predict(question: string): Promise<{ answer: string; context: string; }> { const context = await this.getMostRelevantDocument(question); const response = await this.openai.chat.completions.create({ model: this.modelName, messages: [ { role: "system", content: this.systemMessage }, { role: "user", content: `Use the following information to answer the subsequent question. If the answer cannot be found, write "I don't know." Context: """ ${context} """ Question: ${question}` } ], temperature: 0 }); return { answer: response.choices[0].message.content || "", context }; } } ```

Évaluer avec un juge LLM

Maintenant que votre application RAG est en cours d’exécution et suivie par Weave, l’étape suivante consiste à évaluer la qualité de ses réponses. Cette section explique comment utiliser un LLM comme juge automatique afin de noter les réponses de votre application sans avoir à les étiqueter manuellement. Lorsqu’il n’existe pas de moyen simple d’évaluer votre application, vous pouvez confier l’évaluation de certains de ses aspects à un LLM. Voici un exemple dans lequel un juge LLM mesure la précision du contexte : un prompt lui demande de vérifier si le contexte a été utile pour aboutir à la réponse donnée. Ce prompt est une version enrichie de celui du célèbre [framework RAGAS](https://docs.ragas.io/en/stable/).

Définir une fonction de score

Comme dans le [tutoriel Créer un pipeline d’évaluation](/fr/products/wandb/weave/tutorial-eval), définissez un ensemble de lignes d’exemple sur lesquelles tester votre application, ainsi qu’une fonction de score. La fonction de score reçoit une ligne et l’évalue. Les arguments d’entrée doivent correspondre aux clés de votre ligne : ici, `question` est donc extrait du dictionnaire de la ligne. `output` correspond à la sortie du modèle. L’entrée du modèle est extraite de l’exemple d’après son argument d’entrée, c’est-à-dire ici aussi `question`. Cet exemple utilise des fonctions `async` afin qu’elles s’exécutent en parallèle. Pour une brève introduction à la programmation asynchrone, consultez la [documentation Python sur `asyncio`](https://docs.python.org/3/library/asyncio.html). ```python lines theme={"system"} from openai import OpenAI import weave import asyncio @weave.op() async def context_precision_score(question, output): context_precision_prompt = """Given question, answer and context verify if the context was useful in arriving at the given answer. Give verdict as "1" if useful and "0" if not with json output. Output in only valid JSON format. question: {question} context: {context} answer: {answer} verdict: """ client = OpenAI() prompt = context_precision_prompt.format( question=question, context=output['context'], answer=output['answer'], ) response = client.chat.completions.create( model="gpt-4-turbo-preview", messages=[{"role": "user", "content": prompt}], response_format={ "type": "json_object" } ) response_message = response.choices[0].message response = json.loads(response_message.content) return { "verdict": int(response["verdict"]) == 1, } questions = [ {"question": "What significant result was reported about Zealand Pharma's obesity trial?"}, {"question": "How much did Berkshire Hathaway's cash levels increase in the fourth quarter?"}, {"question": "What is the goal of Highmark Health's integration of Google Cloud and Epic Systems technology?"}, {"question": "What were Rivian and Lucid's vehicle production forecasts for 2024?"}, {"question": "Why was the Norwegian Dawn cruise ship denied access to Mauritius?"}, {"question": "Which company achieved the first U.S. moon landing since 1972?"}, {"question": "What issue did Intuitive Machines' lunar lander encounter upon landing on the moon?"} ] evaluation = weave.Evaluation(dataset=questions, scorers=[context_precision_score]) asyncio.run(evaluation.evaluate(model)) # remarque : vous devez définir un modèle à évaluer ``` ```typescript twoslash lines theme={"system"} // @noErrors const contextPrecisionScore = weave.op(async function(args: { datasetRow: QuestionRow; modelOutput: { answer: string; context: string; } }): Promise { const openai = new OpenAI(); const prompt = `Given question, answer and context verify if the context was useful...`; const response = await openai.chat.completions.create({ model: "gpt-4-turbo-preview", messages: [{ role: "user", content: prompt }], response_format: { type: "json_object" } }); const result = JSON.parse(response.choices[0].message.content || "{}"); return { verdict: parseInt(result.verdict) === 1 }; }); const evaluation = new weave.Evaluation({ dataset: createQuestionDataset(), scorers: [contextPrecisionScore] }); await evaluation.evaluate({ model: weave.op((args: { datasetRow: QuestionRow }) => model.predict(args.datasetRow.question) ) }); ```

Facultatif : définir une classe `Scorer`

La fonction de score de la section précédente convient aux cas simples, mais une classe `Scorer` s’avère utile lorsque vous souhaitez réutiliser le même juge dans plusieurs évaluations ou personnaliser la manière dont les scores sont agrégés. Les étapes suivantes indiquent quand et comment en définir une. Dans certaines applications, vous pouvez avoir besoin de créer des classes d’évaluation personnalisées (par exemple, une classe `LLMJudge` standardisée avec des paramètres tels que le modèle de chat et le prompt), de personnaliser le scoring de chaque ligne ainsi que le calcul d’un score agrégé. Weave propose une liste de classes `Scorer` prêtes à l’emploi et facilite également la création d’un `Scorer` personnalisé. L’exemple suivant montre comment créer une `class CorrectnessLLMJudge(Scorer)` personnalisée. Dans les grandes lignes, voici les étapes pour créer un Scorer personnalisé : 1. Définissez une classe personnalisée qui hérite de `weave.flow.scorer.Scorer`. 2. Redéfinissez la fonction `score` et ajoutez un `@weave.op()` si vous souhaitez suivre chaque appel de la fonction. * Cette fonction doit définir un argument `output` qui reçoit la prédiction du modèle. Déclarez-le avec le type `Optional[dict]` au cas où le modèle renverrait « None ». * Les autres arguments peuvent être de type général `Any` ou `dict`, ou bien sélectionner des colonnes spécifiques du dataset utilisé pour évaluer le modèle avec la classe `weave.Evaluate`. Leurs noms doivent correspondre exactement aux noms de colonnes ou aux clés d’une ligne après son passage dans `preprocess_model_input`, si cette fonction est utilisée. 3. Facultatif : redéfinissez la fonction `summarize` pour personnaliser le calcul du score agrégé. Si vous ne définissez pas de fonction personnalisée, Weave utilise par défaut la fonction `weave.flow.scorer.auto_summarize`. * Cette fonction doit comporter un décorateur `@weave.op()`. ```python lines theme={"system"} from weave import Scorer class CorrectnessLLMJudge(Scorer): prompt: str model_name: str device: str @weave.op() async def score(self, output: Optional[dict], query: str, answer: str) -> Any: """Évalue l’exactitude des prédictions en comparant pred, query et target. Arguments : - output : le dict fourni par le modèle évalué - query : la question posée, telle que définie dans le dataset - answer : la réponse cible, telle que définie dans le dataset Renvoie : - un seul dict {nom de la métrique : valeur d’évaluation unique}""" # get_model est défini comme un getter de modèle général basé sur les params fournis (OpenAI, HF...) eval_model = get_model( model_name = self.model_name, prompt = self.prompt device = self.device, ) # évaluation asynchrone pour accélérer le traitement ; l’asynchrone n’est pas obligatoire grade = await eval_model.async_predict( { "query": query, "answer": answer, "result": output.get("result"), } ) # parsing de la sortie ; pourrait être rendu plus robuste avec pydantic evaluation = "incorrect" not in grade["text"].strip().lower() # le nom de colonne affiché dans Weave return {"correct": evaluation} @weave.op() def summarize(self, score_rows: list) -> Optional[dict]: """Agrège tous les scores calculés pour chaque ligne par la fonction de score. Arguments : - score_rows : une liste de dicts. Chaque dict contient des métriques et des scores Renvoie : - un dict imbriqué ayant la même structure que l’entrée""" # si rien n’est fourni, la fonction weave.flow.scorer.auto_summarize est utilisée # return auto_summarize(score_rows) valid_data = [x.get("correct") for x in score_rows if x.get("correct") is not None] count_true = list(valid_data).count(True) int_data = [int(x) for x in valid_data] sample_mean = np.mean(int_data) if int_data else 0 sample_variance = np.var(int_data) if int_data else 0 sample_error = np.sqrt(sample_variance / len(int_data)) if int_data else 0 # le niveau supplémentaire "correct" n’est pas nécessaire, mais structure l’affichage dans l’interface utilisateur return { "correct": { "true_count": count_true, "true_fraction": sample_mean, "stderr": sample_error, } } ``` ```plaintext theme={"system"} Cette fonctionnalité n’est pas encore disponible en TypeScript. ``` Pour utiliser cette classe comme évaluateur, initialisez-la et transmettez-la à l’argument `scorers` de votre `Evaluation`, comme suit : ```python lines theme={"system"} evaluation = weave.Evaluation(dataset=questions, scorers=[CorrectnessLLMJudge()]) ``` ```plaintext theme={"system"} Cette fonctionnalité n’est pas encore disponible en TypeScript. ```

Assembler le tout

Cette section regroupe tout ce qui a été vu dans les étapes précédentes en un seul exemple de bout en bout, afin que vous puissiez voir comment les différents éléments s’articulent et les adapter à votre propre application RAG. Pour obtenir le même résultat avec vos applications RAG : * Encapsulez les appels LLM et les fonctions de l’étape de récupération avec `weave.op()`. * Facultatif : créez une sous-classe de `Model` avec une fonction `predict` et les détails de l’application. * Rassemblez des exemples à évaluer. * Créez des fonctions de score qui évaluent un exemple à la fois. * Utilisez la classe `Evaluation` pour exécuter des évaluations sur vos exemples. **REMARQUE :** l’exécution asynchrone des évaluations peut parfois déclencher une limite de débit sur les modèles d’OpenAI, d’Anthropic et d’autres fournisseurs. Pour l’éviter, vous pouvez définir une variable d’environnement qui limite le nombre de workers parallèles, par exemple `WEAVE_PARALLELISM=3`. Voici le code complet. ```python lines {34,52-77} theme={"system"} from openai import OpenAI import weave from weave import Model import numpy as np import json import asyncio # Exemples à utiliser pour les évaluations articles = [ "Novo Nordisk and Eli Lilly rival soars 32 percent after promising weight loss drug results Shares of Denmarks Zealand Pharma shot 32 percent higher in morning trade, after results showed success in its liver disease treatment survodutide, which is also on trial as a drug to treat obesity. The trial “tells us that the 6mg dose is safe, which is the top dose used in the ongoing [Phase 3] obesity trial too,” one analyst said in a note. The results come amid feverish investor interest in drugs that can be used for weight loss.", "Berkshire shares jump after big profit gain as Buffetts conglomerate nears $1 trillion valuation Berkshire Hathaway shares rose on Monday after Warren Buffetts conglomerate posted strong earnings for the fourth quarter over the weekend. Berkshires Class A and B shares jumped more than 1.5%, each. Class A shares are higher by more than 17% this year, while Class B has gained more than 18%. Berkshire was last valued at $930.1 billion, up from $905.5 billion where it closed on Friday, according to FactSet. Berkshire on Saturday posted fourth-quarter operating earnings of $8.481 billion, about 28 percent higher than the $6.625 billion from the year-ago period, driven by big gains in its insurance business. Operating earnings refers to profits from businesses across insurance, railroads and utilities. Meanwhile, Berkshires cash levels also swelled to record levels. The conglomerate held $167.6 billion in cash in the fourth quarter, surpassing the $157.2 billion record the conglomerate held in the prior quarter.", "Highmark Health says its combining tech from Google and Epic to give doctors easier access to information Highmark Health announced it is integrating technology from Google Cloud and the health-care software company Epic Systems. The integration aims to make it easier for both payers and providers to access key information they need, even if it's stored across multiple points and formats, the company said. Highmark is the parent company of a health plan with 7 million members, a provider network of 14 hospitals and other entities", "Rivian and Lucid shares plunge after weak EV earnings reports Shares of electric vehicle makers Rivian and Lucid fell Thursday after the companies reported stagnant production in their fourth-quarter earnings after the bell Wednesday. Rivian shares sank about 25 percent, and Lucids stock dropped around 17 percent. Rivian forecast it will make 57,000 vehicles in 2024, slightly less than the 57,232 vehicles it produced in 2023. Lucid said it expects to make 9,000 vehicles in 2024, more than the 8,428 vehicles it made in 2023.", "Mauritius blocks Norwegian cruise ship over fears of a potential cholera outbreak Local authorities on Sunday denied permission for the Norwegian Dawn ship, which has 2,184 passengers and 1,026 crew on board, to access the Mauritius capital of Port Louis, citing “potential health risks.” The Mauritius Ports Authority said Sunday that samples were taken from at least 15 passengers on board the cruise ship. A spokesperson for the U.S.-headquartered Norwegian Cruise Line Holdings said Sunday that 'a small number of guests experienced mild symptoms of a stomach-related illness' during Norwegian Dawns South Africa voyage.", "Intuitive Machines lands on the moon in historic first for a U.S. company Intuitive Machines Nova-C cargo lander, named Odysseus after the mythological Greek hero, is the first U.S. spacecraft to soft land on the lunar surface since 1972. Intuitive Machines is the first company to pull off a moon landing — government agencies have carried out all previously successful missions. The company's stock surged in extended trading Thursday, after falling 11 percent in regular trading.", "Lunar landing photos: Intuitive Machines Odysseus sends back first images from the moon Intuitive Machines cargo moon lander Odysseus returned its first images from the surface. Company executives believe the lander caught its landing gear sideways on the surface of the moon while touching down and tipped over. Despite resting on its side, the company's historic IM-1 mission is still operating on the moon.", ] def docs_to_embeddings(docs: list) -> list: openai = OpenAI() document_embeddings = [] for doc in docs: response = ( openai.embeddings.create(input=doc, model="text-embedding-3-small") .data[0] .embedding ) document_embeddings.append(response) return document_embeddings article_embeddings = docs_to_embeddings(articles) # Remarque : en général, cette opération s'effectue une seule fois sur vos articles, puis les embeddings et les métadonnées sont stockés dans une base de données # Ajouter un décorateur à l'étape de récupération @weave.op() def get_most_relevant_document(query): openai = OpenAI() query_embedding = ( openai.embeddings.create(input=query, model="text-embedding-3-small") .data[0] .embedding ) similarities = [ np.dot(query_embedding, doc_emb) / (np.linalg.norm(query_embedding) * np.linalg.norm(doc_emb)) for doc_emb in article_embeddings ] # Obtenir l'index du document le plus similaire most_relevant_doc_index = np.argmax(similarities) return articles[most_relevant_doc_index] # Créer une sous-classe de Model décrivant l'application, avec une fonction predict qui génère une réponse class RAGModel(Model): system_message: str model_name: str = "gpt-3.5-turbo-1106" @weave.op() def predict(self, question: str) -> dict: # remarque : `question` servira plus tard à sélectionner les données dans nos lignes d'évaluation from openai import OpenAI context = get_most_relevant_document(question) client = OpenAI() query = f"""Use the following information to answer the subsequent question. If the answer cannot be found, write "I don't know." Context: \"\"\" {context} \"\"\" Question: {question}""" response = client.chat.completions.create( model=self.model_name, messages=[ {"role": "system", "content": self.system_message}, {"role": "user", "content": query}, ], temperature=0.0, response_format={"type": "text"}, ) answer = response.choices[0].message.content return {'answer': answer, 'context': context} # Définir les noms de votre équipe et de votre projet weave.init('/rag-quickstart') model = RAGModel( system_message="You are an expert in finance and answer questions related to finance, financial services, and financial markets. When responding based on provided information, be sure to cite the source." ) # Voici notre fonction de score, qui utilise la question et la sortie pour produire un score @weave.op() async def context_precision_score(question, output): context_precision_prompt = """Given question, answer and context verify if the context was useful in arriving at the given answer. Give verdict as "1" if useful and "0" if not with json output. Output in only valid JSON format. question: {question} context: {context} answer: {answer} verdict: """ client = OpenAI() prompt = context_precision_prompt.format( question=question, context=output['context'], answer=output['answer'], ) response = client.chat.completions.create( model="gpt-4-turbo-preview", messages=[{"role": "user", "content": prompt}], response_format={ "type": "json_object" } ) response_message = response.choices[0].message response = json.loads(response_message.content) return { "verdict": int(response["verdict"]) == 1, } questions = [ {"question": "What significant result was reported about Zealand Pharma's obesity trial?"}, {"question": "How much did Berkshire Hathaway's cash levels increase in the fourth quarter?"}, {"question": "What is the goal of Highmark Health's integration of Google Cloud and Epic Systems technology?"}, {"question": "What were Rivian and Lucid's vehicle production forecasts for 2024?"}, {"question": "Why was the Norwegian Dawn cruise ship denied access to Mauritius?"}, {"question": "Which company achieved the first U.S. moon landing since 1972?"}, {"question": "What issue did Intuitive Machines' lunar lander encounter upon landing on the moon?"} ] # Définir un objet Evaluation et lui transmettre les exemples de questions ainsi que les fonctions de score evaluation = weave.Evaluation(dataset=questions, scorers=[context_precision_score]) asyncio.run(evaluation.evaluate(model)) ``` ```typescript twoslash lines theme={"system"} // @noErrors require('dotenv').config(); import { OpenAI } from 'openai'; import * as weave from 'weave'; interface Article { text: string; embedding?: number[]; } const articles: Article[] = [ { text: `Novo Nordisk and Eli Lilly rival soars 32 percent after promising weight loss drug results Shares of Denmarks Zealand Pharma shot 32 percent higher in morning trade, after results showed success in its liver disease treatment survodutide, which is also on trial as a drug to treat obesity. The trial tells us that the 6mg dose is safe, which is the top dose used in the ongoing [Phase 3] obesity trial too, one analyst said in a note. The results come amid feverish investor interest in drugs that can be used for weight loss.` }, { text: `Berkshire shares jump after big profit gain as Buffetts conglomerate nears $1 trillion valuation Berkshire Hathaway shares rose on Monday after Warren Buffetts conglomerate posted strong earnings for the fourth quarter over the weekend. Berkshires Class A and B shares jumped more than 1.5%, each. Class A shares are higher by more than 17% this year, while Class B has gained more than 18%. Berkshire was last valued at $930.1 billion, up from $905.5 billion where it closed on Friday, according to FactSet. Berkshire on Saturday posted fourth-quarter operating earnings of $8.481 billion, about 28 percent higher than the $6.625 billion from the year-ago period, driven by big gains in its insurance business. Operating earnings refers to profits from businesses across insurance, railroads and utilities. Meanwhile, Berkshires cash levels also swelled to record levels. The conglomerate held $167.6 billion in cash in the fourth quarter, surpassing the $157.2 billion record the conglomerate held in the prior quarter.` }, { text: `Highmark Health says its combining tech from Google and Epic to give doctors easier access to information Highmark Health announced it is integrating technology from Google Cloud and the health-care software company Epic Systems. The integration aims to make it easier for both payers and providers to access key information they need, even if its stored across multiple points and formats, the company said. Highmark is the parent company of a health plan with 7 million members, a provider network of 14 hospitals and other entities` } ]; function cosineSimilarity(a: number[], b: number[]): number { const dotProduct = a.reduce((sum, val, i) => sum + val * b[i], 0); const magnitudeA = Math.sqrt(a.reduce((sum, val) => sum + val * val, 0)); const magnitudeB = Math.sqrt(b.reduce((sum, val) => sum + val * val, 0)); return dotProduct / (magnitudeA * magnitudeB); } const docsToEmbeddings = weave.op(async function(docs: Article[]): Promise { const openai = new OpenAI(); const enrichedDocs = await Promise.all(docs.map(async (doc) => { const response = await openai.embeddings.create({ input: doc.text, model: "text-embedding-3-small" }); return { ...doc, embedding: response.data[0].embedding }; })); return enrichedDocs; }); class RAGModel { private openai: OpenAI; private systemMessage: string; private modelName: string; private articleEmbeddings: Article[]; constructor(config: { systemMessage: string; modelName?: string; articleEmbeddings: Article[]; }) { this.openai = new OpenAI(); this.systemMessage = config.systemMessage; this.modelName = config.modelName || "gpt-3.5-turbo-1106"; this.articleEmbeddings = config.articleEmbeddings; this.predict = weave.op(this, this.predict); } private async getMostRelevantDocument(query: string): Promise { const queryEmbedding = await this.openai.embeddings.create({ input: query, model: "text-embedding-3-small" }); const similarities = this.articleEmbeddings.map(doc => { if (!doc.embedding) return 0; return cosineSimilarity(queryEmbedding.data[0].embedding, doc.embedding); }); const mostRelevantIndex = similarities.indexOf(Math.max(...similarities)); return this.articleEmbeddings[mostRelevantIndex].text; } async predict(question: string): Promise<{ answer: string; context: string; }> { const context = await this.getMostRelevantDocument(question); const response = await this.openai.chat.completions.create({ model: this.modelName, messages: [ { role: "system", content: this.systemMessage }, { role: "user", content: `Use the following information to answer the subsequent question. If the answer cannot be found, write "I don't know." Context: """ ${context} """ Question: ${question}` } ], temperature: 0 }); return { answer: response.choices[0].message.content || "", context }; } } interface ScorerResult { verdict: boolean; } interface QuestionRow { question: string; } function createQuestionDataset(): weave.Dataset { return new weave.Dataset({ id: 'rag-questions', rows: [ { question: "What significant result was reported about Zealand Pharma's obesity trial?" }, { question: "How much did Berkshire Hathaway's cash levels increase in the fourth quarter?" }, { question: "What is the goal of Highmark Health's integration of Google Cloud and Epic Systems technology?" } ] }); } const contextPrecisionScore = weave.op(async function(args: { datasetRow: QuestionRow; modelOutput: { answer: string; context: string; } }): Promise { const openai = new OpenAI(); const prompt = `Given question, answer and context verify if the context was useful in arriving at the given answer. Give verdict as "1" if useful and "0" if not with json output. Output in only valid JSON format. question: ${args.datasetRow.question} context: ${args.modelOutput.context} answer: ${args.modelOutput.answer} verdict: `; const response = await openai.chat.completions.create({ model: "gpt-4-turbo-preview", messages: [{ role: "user", content: prompt }], response_format: { type: "json_object" } }); const result = JSON.parse(response.choices[0].message.content || "{}"); return { verdict: parseInt(result.verdict) === 1 }; }); async function main() { # Indiquez le nom de votre équipe et de votre projet await weave.init('/rag-quickstart'); const articleEmbeddings = await docsToEmbeddings(articles); const model = new RAGModel({ systemMessage: "You are an expert in finance and answer questions related to finance, financial services, and financial markets. When responding based on provided information, be sure to cite the source.", articleEmbeddings }); const evaluation = new weave.Evaluation({ dataset: createQuestionDataset(), scorers: [contextPrecisionScore] }); const results = await evaluation.evaluate({ model: weave.op((args: { datasetRow: QuestionRow }) => model.predict(args.datasetRow.question) ) }); console.log('Evaluation results:', results); } if (require.main === module) { main().catch(console.error); } ```

Conclusion

Ce tutoriel vous a montré comment intégrer l’observabilité aux différentes étapes de vos applications, comme l’étape de récupération dans cet exemple. Vous avez également appris à concevoir des fonctions de score plus complexes, comme un juge LLM, pour évaluer automatiquement les réponses de vos applications. ## Related topics - [Démarrage rapide : instrumenter et tracer des fonctions](/fr/products/wandb/weave/quickstart.md)