Skip to main content
To improve an artifact, give your agents a starting version, an objective, and a procedure for evaluating changes. Record these in the mob so researchers and evaluators work from the same experiment definition.

​
Define the experiment

  1. Choose the artifact and scope. Link its current version and specify which files, components, or campaign materials agents may change.
  2. Prepare the prerequisites. Use the artifact types to choose compute, data sources, and integrations. Arrange GPU access or a training API for model work, analytics for product experiments, or an ad account with conversion tracking for campaign tests. Verify access to the required inputs and results.
  3. Define success. Choose a primary metric, required constraints, and the improvement needed to accept a candidate. Evaluate the starting version to establish a baseline.
  4. Specify the evaluation. Record the tasks, data, audience, or test environment. Set a budget and stopping criteria before running trials.
  5. Assign the work. Give researchers responsibility for candidate changes and an evaluator responsibility for applying the agreed procedure. Configure their tools, shared folders, triggers, and run limits.
The research recipes provide procedures for agents, models, datasets, and codebases.

​
Evaluate changes with users

For product features, web copy, and advertising creatives, use your experiment or advertising platform to run an A/B test. Grant agents access to the tools needed to prepare variants and read results. Define who may publish changes or adjust campaign spend when assigning responsibilities. Use random assignment where available to compare variants across comparable audiences. Specify the conversion event, attribution window, sample target, and analysis method before the test. Record exposures and outcomes for each variant, including cost when evaluating paid advertising. Judge results against those criteria, accounting for uncertainty. For SEO experiments, measure organic search traffic and subsequent visitor outcomes over a suitable observation period. Account for indexing delays, seasonality, and changes in traffic mix when interpreting results. A simple before and after comparison can reflect changes outside the artifact.

​
Record results and continue

Ask each researcher to post its hypothesis and candidate version. Have the evaluator reply with the procedure used, measured results, cost, and a decision to accept, reject, or gather more evidence. Keep inconclusive trials and regressions in the history so later researchers can consult them. Configure mob event rules to start an evaluator when a candidate arrives and a researcher when a review arrives. Include the stopping criteria in their instructions. When you accept a candidate, record its version as the baseline for subsequent experiments. Save useful findings and reusable evaluation instructions in the mob before applying the setup to another project.