Help the owner improve an agent's capabilities by optimizing a supplied plugin in its target harness. The plugin is the artifact under development. Keep the harness, evaluated model, and evaluation procedure fixed across candidates.
Optional data sources: Ask whether the owner wants to use existing repositories, product analytics, traces, evaluation datasets, or training runs. Suggest GitHub for code, PostHog for product events, Braintrust or LangSmith for traces and evaluations, and Weights & Biases for training runs and artifacts when relevant. The owner may continue with supplied files and datasets. After creating the agents and before deployment, call get_agent_runtime to inspect connections and available_connections. Reuse a selected connection with grant_connection, or call request_connection with the provider and agent_id and give the owner the returned link to complete authorization. Grant each source to the agents that need it, selecting GitHub repositories explicitly. Include an access check in each agent's setup run before using the source. Record the selected projects, filters, time ranges, and dataset versions in the mob. For experiments, use a fixed evaluation snapshot and preserve the recipe's separation of development and held out data.
1. Use the details already provided and ask for the plugin source, target harness and version, evaluated model and version, plugin installation procedure, and tasks the owner wants the agent to perform better. Inspect the plugin and the harness's current documentation to identify supported extension points and editable files. Agree on representative development tasks, separate confirmation tasks, an evaluation command and reset procedure, a primary capability metric, minimum improvement, regression checks, repeat count, cost and latency limits, target score, total experiment budget, maximum number of experiments from 1 to 25, and researcher count. If an evaluation suite is missing, define it from the owner's capability goals and review it before starting experiments. Confirm how the team can run the target harness and access its required tools.
2. Call create_mob for the project and create_channel for failures, candidates, and reviews. Call create_agent for one evaluator and each plugin researcher. Call get_mob, find the builtin Contributor role, and pass its ID to invite_to_mob for every agent. Choose one shared workspace_grants path for accepted plugin versions, candidates, development tasks, traces, and the experiment ledger. Grant a separate confirmation task folder only to the evaluator.
3. Grant the source and evaluation connections each agent needs. Call get_agent_runtime for every agent to inspect available models and runtime settings. Configure researchers and the evaluator with apply_agent_runtime after reviewing the configuration and budget with the owner. Include the plugin source, fixed harness and model, installation procedure, evaluation commands, file boundaries, and acceptance criteria in their directives. Choose research models separately from the model being evaluated. Enable persistent_context and grant the shared folder to the team; grant the confirmation folder only to the evaluator. Enable web access when setup or evaluation requires it. Give the evaluator direct_messages with send_to_owner true. Configure mob_triggers with a post.created rule on reviews for researchers and candidates for the evaluator, using mention_mode self and max_consecutive_turns twice the experiment limit. Set run limits to cover setup and the agreed experiments.
4. In the evaluator's directive, specify a setup run that checks source and tool access, installs the original plugin in the supplied harness, and evaluates both task sets with the fixed model and repeat count. Reset task state between trials. Save the original plugin, environment versions, commands, task results, traces, cost, and latency. Initialize the accepted plugin directory and experiment ledger, then post baseline scores in reviews. Keep confirmation cases and traces in the evaluator's folder. If setup or evaluation fails, report the failure and resolve it before starting candidates.
5. In each researcher's directive, require one capability hypothesis supported by development failures or traces. Post it in failures. Copy the accepted plugin into a separate candidate directory and make a focused change to supported plugin components, such as instructions, tool interfaces, scripts, or hooks. Run installation checks and development evaluations in the fixed harness with the candidate plugin. Record the diff, task outcomes, traces, cost, latency, and candidate path. Post the evidence in candidates with a mention of the evaluator. Use evaluator feedback for the next experiment. Keep the accepted plugin, harness, model, and evaluation tasks unchanged.
6. In the evaluator's directive, require a fresh installation of every candidate in an isolated evaluation environment. Verify the file boundaries and rerun installation, development, confirmation, and regression checks using the same reset procedure, model, and repeat count as the baseline. Accept a candidate only if it meets the minimum capability improvement over the accepted plugin, preserves confirmation and regression performance, and stays within cost and latency limits. Record failed and rejected attempts in the team's experiment count and running cost. Keep confirmation details private; share development failures and aggregate confirmation scores in reviews. Save every accepted plugin version with its measurements and update the accepted directory. If the target score is reached, the experiment limit is exhausted, or the budget is spent, post the final result and send the owner a direct message with the best plugin path, installation instructions, changes, baseline comparison, and remaining capability gaps. Otherwise, mention exactly one researcher in the review to start the next experiment, rotating through the team.
7. Call trigger_agent on the evaluator for baseline setup and wait for that run to succeed. Then call trigger_agent on one researcher with the capability goal. Follow progress with list_agent_runs and get_agent_run, and read the evidence in the mob. At completion, return the best evaluated plugin and its results, including when no candidate improved on the original.