Skip to content

Commit 3c0bee4

Browse files
authored
Merge pull request #30 from future-agi/dev
dev to main
2 parents 4d167df + 39fcaa4 commit 3c0bee4

229 files changed

Lines changed: 8058 additions & 2126 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎cookbook/cookbook1/AI-Evaluation-for-Meeting-Summarization.mdx‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -136,6 +136,9 @@ for column in summary_columns:
136136
evaluate_bertscore(dataset, column)
137137
print()
138138
```
139+
140+
**Output:**
141+
139142
```plaintext
140143
Evaluating Summary Quality for summary-gpt-4o...
141144
Evaluating BERTScore for summary-gpt-4o...
@@ -167,6 +170,8 @@ print(tabulate(
167170
))
168171
```
169172

173+
**Output:**
174+
170175
<div style={{ border: '1px solid #000', borderCollapse: 'collapse', width: '100%' }}>
171176
<table style={{ border: '1px solid #000', width: '100%', textAlign: 'center' }}>
172177
<thead>

‎cookbook/cookbook2/AI-Evaluation-for-AI-SDR.mdx‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@ pd.set_option('display.max_colwidth', None)
1414
```
1515

1616
Below the sample of dataset used in this cookbook:
17+
1718
```plaintext
1819
value_proposition:
1920
Get location information of your social media following to place better ads and sponsorships
@@ -157,6 +158,8 @@ print("\nEvaluation on Prompt 2")
157158
print(tabulate(complete_result_prompt2, headers='keys', tablefmt='fancy_grid',showindex=False))
158159
```
159160

161+
**Output:**
162+
160163
### Evaluation on Prompt 1
161164

162165
<div style={{ border: '1px solid #000', borderCollapse: 'collapse', width: '100%' }}>
@@ -284,6 +287,8 @@ else:
284287
print(f"\nWinner Prompt: {winner}")
285288
```
286289

290+
**Output:**
291+
287292
<div style={{ border: '1px solid #000', borderCollapse: 'collapse', width: '100%' }}>
288293
<table style={{ border: '1px solid #000', width: '100%', textAlign: 'center' }}>
289294
<thead>

‎cookbook/cookbook3/Mastering-Evaluation-of-AI-Agents.mdx‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,3 @@
1-
# Cookbook: Mastering Evaluation for AI Agents
2-
31
## 1. Installing Future AGI
42

53
```bash
Lines changed: 237 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,237 @@
1+
2+
### In this cookbook we demonstrate how FutureAGI SDK can be used to evaluate images:
3+
4+
- **Alignment with Descriptions**: Ensuring that generated images accurately represent the textual descriptions provided.
5+
- **Meeting Specific Requirements**: Evaluating images against predefined artistic or thematic standards.
6+
- **Evaluating Image Replacements**: Assessing the quality and fit of new images replacing previous versions.
7+
8+
# Installing FutureAGI
9+
10+
```python
11+
!pip install futureagi==0.3.3
12+
!pip install pillow
13+
```
14+
15+
### Making Necessary Imports
16+
17+
```python
18+
import json
19+
import requests
20+
21+
from IPython.display import Image, display
22+
23+
from fi.evals import EvalClient
24+
from fi.evals import Deterministic, ImageInputOutput, ImageInstruction
25+
from fi.testcases import MLLMTestCase
26+
```
27+
28+
29+
### Loading and Visualising Data
30+
31+
```python
32+
path = '/content/data.json'
33+
# Open and load the JSON file
34+
with open(path, 'r') as file:
35+
datapoints = json.load(file)
36+
```
37+
38+
```python
39+
# Sample Datapoint
40+
41+
datapoint = {
42+
'id': 'masked_id',
43+
'image_url': './images/output_8_0.png',
44+
'output_image_url': './images/output_26_0.png',
45+
'prompt': 'an asian man, closeup, on new york city street',
46+
'type': 'T2I',
47+
'category': 'Ethnicity',
48+
'question': 'Does the image follow the Ethnicity mentioned in the prompt?'
49+
}
50+
```
51+
52+
```python
53+
# Sample Image
54+
55+
response = requests.get(datapoint['image_url'])
56+
57+
# Display the image in the notebook
58+
if response.status_code == 200:
59+
display(Image(response.content))
60+
else:
61+
print("Failed to fetch the image.")
62+
```
63+
64+
**Output:**
65+
66+
![Sample Image](./images/output_8_0.png)
67+
68+
### Initializing the FutureAGI Evaluator Class and Deterministic Eval
69+
70+
```python
71+
from getpass import getpass
72+
from fi.evals import EvalClient
73+
74+
fi_api_key = getpass("Enter your FI API Key: ")
75+
fi_secret_key = getpass("Enter your FI Secret Key: ")
76+
77+
evaluator = EvalClient(
78+
fi_api_key=fi_api_key,
79+
fi_secret_key=fi_secret_key,
80+
fi_base_url="https://dev.api.futureagi.com"
81+
)
82+
83+
print("Evaluator client initialized successfully!")
84+
```
85+
86+
#### Evaluating Alignment with Descriptions
87+
88+
```python
89+
image_eval = ImageInstruction(
90+
config={
91+
"criteria": """
92+
Evaluate the image based on:
93+
1. Accuracy of object representation
94+
2. Setting accuracy
95+
3. Image quality and realism
96+
"""
97+
}
98+
)
99+
```
100+
101+
```python
102+
class ImageEvalTestCase(MLLMTestCase):
103+
input: str
104+
image_url: str
105+
```
106+
107+
```python
108+
test_case_img_eval = ImageEvalTestCase(
109+
input=datapoint['prompt'],
110+
image_url=datapoint['image_url']
111+
)
112+
```
113+
114+
```python
115+
import textwrap
116+
batch_result = evaluator.evaluate([image_eval], [test_case_img_eval])
117+
wrapped_text = textwrap.fill(batch_result.eval_results[0].reason, width=80)
118+
119+
print(wrapped_text)
120+
```
121+
122+
**Output:**
123+
124+
```plaintext
125+
The image accurately represents an Asian man and a New York City street, but the anime style affects realism and image quality.
126+
```
127+
128+
#### Evaluating Subjective Requirements
129+
130+
```python
131+
deterministic_eval = Deterministic(config={
132+
"multi_choice": False,
133+
"choices": ["Yes", "No"],
134+
"rule_prompt": "Prompt : {{input_key2}}, Image : {{input_key3}}. Given the prompt and the corresponding image, answer the Question : {{input_key1}}. Focus only on the {{input_key4}}",
135+
"input": {
136+
"input_key1": "question",
137+
"input_key2": "prompt",
138+
"input_key3": "image_url",
139+
"input_key4": "category"
140+
}
141+
})
142+
```
143+
144+
```python
145+
class DeterministicTestCase(MLLMTestCase):
146+
question: str
147+
prompt: str
148+
image_url: str
149+
category: str
150+
```
151+
152+
```python
153+
test_case = DeterministicTestCase(
154+
question=datapoint['question'],
155+
prompt=datapoint['prompt'],
156+
image_url=datapoint['image_url'],
157+
category=datapoint['category']
158+
)
159+
```
160+
161+
```python
162+
batch_result = evaluator.evaluate([deterministic_eval], [test_case])
163+
```
164+
165+
```python
166+
batch_result.eval_results[0].data
167+
```
168+
169+
**Output:**
170+
171+
```plaintext
172+
['Yes']
173+
```
174+
175+
```python
176+
print(textwrap.fill(batch_result.eval_results[0].reason, width=80))
177+
```
178+
179+
**Output:**
180+
181+
```plaintext
182+
The image depicts an animated character with traits commonly associated with Asian ethnicity.
183+
```
184+
185+
#### Evaluating Changes Based on Text Instructions
186+
187+
```python
188+
image_input_output_eval = ImageInputOutput(config={
189+
"criteria": """
190+
Evaluate the output image based on:
191+
1. Adherence to input instruction
192+
2. Preservation of key elements from input image
193+
3. Quality of color modification
194+
4. Image quality and realism
195+
"""
196+
})
197+
```
198+
199+
```python
200+
class ImageInputOutputTestCase(MLLMTestCase):
201+
input: str
202+
input_image_url: str
203+
output_image_url: str
204+
```
205+
206+
```python
207+
response = requests.get(datapoint['output_image_url'])
208+
209+
# Display the image in the notebook
210+
if response.status_code == 200:
211+
display(Image(response.content))
212+
else:
213+
print("Failed to fetch the image.")
214+
```
215+
216+
**Output:**
217+
218+
![Modified Image](./images/output_26_0.png)
219+
220+
```python
221+
test_case_image_input_output = ImageInputOutputTestCase(
222+
input='Replace the man with a man of African ethnicity',
223+
input_image_url=datapoint['image_url'],
224+
output_image_url=datapoint['output_image_url']
225+
)
226+
```
227+
228+
```python
229+
batch_result = evaluator.evaluate([image_input_output_eval], [test_case_image_input_output])
230+
print(textwrap.fill(batch_result.eval_results[0].reason, width=80))
231+
```
232+
233+
**Output:**
234+
235+
```plaintext
236+
The output image accurately replaces the man with one of African ethnicity while preserving all key elements, maintaining high image quality and realism.
237+
```
278 KB
Loading
264 KB
Loading

‎future-agi/admin-settings.mdx‎

Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,107 @@
1+
---
2+
title: "Admin & Settings"
3+
---
4+
This guide provides step-by-step instructions for navigating the **Administration Panel**, managing **API Keys**, configuring **Custom Models**, tracking **Usage Summary**, handling **Billing & Plans**, and updating **User Profile settings**.
5+
6+
---
7+
8+
## **ADMINISTRATION**
9+
10+
### **1. Usage Summary**
11+
12+
This section provides a detailed view of token consumption, evaluation usage, and system performance. Users can monitor their usage trends through graphical representations and filter data by selecting a specific date range. Additionally, reports can be exported for billing and resource tracking.
13+
14+
15+
16+
- Go to **Settings > Usage Summary**.
17+
- Select a **date range** to filter data.
18+
- Review the **graphical representation** of usage trends.
19+
20+
### **2. User Management**
21+
22+
This section allows administrators to manage team members, assign roles, and control access permissions. Users can view a list of registered members, invite new users, modify existing roles, and approve or reject pending invitations. Removing team members from the system can also be done from this section.
23+
24+
25+
26+
- Go to **Settings > User Management**.
27+
- View the list of registered users.
28+
- Click **Add User** to invite a new team member.
29+
1. Enter the **user’s email address**.
30+
2. Select an **access role** from the dropdown.
31+
3. Click **Submit** to send the invitation.
32+
33+
### **3. API Keys**
34+
35+
This section enables users to configure API access for third-party integrations such as OpenAI, Azure, Anthropic, and Hugging Face.
36+
37+
38+
39+
- Go to **Settings > API Keys**.
40+
- View the list of existing API keys.
41+
1. Select the **provider** (e.g., OpenAI, Azure, Anthropic).
42+
2. Enter the **API Key** in the designated field.
43+
3. Click **Save**.
44+
45+
### **4. Custom Models**
46+
47+
This section allows users to integrate third-party AI models from providers such as Vertex AI, AWS SageMaker, and AWS Bedrock. Users can register and manage their models by configuring authentication credentials and defining input and output token costs.
48+
49+
50+
51+
- Go to **Settings > Custom Models**.
52+
- View the list of available custom models.
53+
- Click **Add Model** to configure a new model.
54+
- Select a **Model Provider** (e.g., Vertex AI, AWS SageMaker, AWS Bedrock).
55+
- Enter the required details:
56+
- **Model Name**
57+
- **Input Token Cost**
58+
- **Output Token Cost**
59+
- **Provider-Specific Credentials** (e.g., API keys, JSON credentials, region settings).
60+
- Click **Save**.
61+
62+
---
63+
64+
## **PLAN SUMMARY**
65+
66+
### **1. Plans & Pricing**
67+
68+
This section enables users to select and manage their subscription plans. Users can compare different pricing tiers, toggle between monthly and yearly billing cycles, and upgrade or downgrade their plans based on their needs.
69+
70+
71+
72+
- Go to **Settings > Plans & Pricing**.
73+
- View available subscription plans.
74+
- Select the **subscription tier** that best suits your requirements.
75+
76+
(Connect with us to best understand your needs)`[hyperlink to “contact sales”]`
77+
78+
### **2. Billing**
79+
80+
This section provides users with an overview of their credit balance, billing history, and payment details. You can modify payment methods, add funds, and download invoices for record-keeping.
81+
82+
83+
84+
- Go to **Settings > Billing**.
85+
- View **billing history** and **credit balance**.
86+
- Edit **payment methods** if necessary.
87+
88+
`[hyperlink to “billing”]`
89+
90+
---
91+
92+
## **USER**
93+
94+
### **Profile**
95+
96+
This section allows users to manage their personal details and security settings. Users can update full name or change their password.
97+
98+
99+
100+
**To Reset Your Password:**
101+
102+
- Navigate to **Settings > Profile**.
103+
- Click on the “external link” shown in “Password”.
104+
- An email containing “password reset link” will be sent yo your email.
105+
- Follow the instructions in email
106+
107+
---

‎future-agi/home.mdx‎

100644100755
File mode changed.

‎future-agi/integrations/google-bigquery/getting-started.mdx‎

100644100755
File mode changed.

‎future-agi/integrations/google-bigquery/google-bigquery-faq.mdx‎

100644100755
File mode changed.

0 commit comments

Comments
 (0)