Skip to content

NextEval Solver 1.0

User Guide

Connect a model to an objective, run one complete derivative-free optimization session, and read the evaluations it actually made.

Release: coming soon

Installation

Release: coming soonNextEval Solver 1.0 is not public yet. The install command and the repository link appear here with the release.

The solver needs Python 3.10 or later, and its only runtime dependency is NumPy (plus the small tomli parser on Python 3.10). The import name is nexteval. The built-in provider clients use the standard library, so no model SDK is needed for them. The rest of this guide describes the interface of the release.

Model provider setup

The built-in runner needs a model endpoint that supports tool calls. Credentials are read from the environment; they are never stored in a model specification or a trace. All values below are placeholders, and this website neither receives credentials nor sends model requests.

Anthropic-compatible endpoint

export ANTHROPIC_BASE_URL="https://your-provider.example/v1"
export ANTHROPIC_MODEL="your-model-id"
export ANTHROPIC_AUTH_TOKEN="<provided-by-your-credential-manager>"

AnthropicCompatClient() reads these variables (ANTHROPIC_API_KEY is accepted in place of the token). The shortcut model="env" selects this client; it does not detect your provider. On PowerShell, use $env:VARIABLE="value" instead of export.

OpenAI-compatible endpoint

export NEXTEVAL_OPENAI_BASE_URL="https://your-provider.example/v1"
export NEXTEVAL_OPENAI_MODEL="your-model-id"
export NEXTEVAL_OPENAI_API_KEY="<provided-by-your-credential-manager>"

OpenAICompatClient() uses Chat Completions with tool calls; an endpoint that only supports another protocol is not interchangeable. Instead of a client object you can pass a specification without credentials and keep the secret in the environment:

options = {
    "model": {
        "provider": "openai-compat",
        "base_url": "https://your-provider.example/v1",
        "model": "your-model-id",
    },
}

Azure OpenAI

Azure uses a different wire format, so select it explicitly with AzureOpenAICompatClient() or the provider name "azure-openai". It reads AZURE_OPENAI_ENDPOINT, AZURE_OPENAI_DEPLOYMENT, AZURE_OPENAI_API_VERSION and AZURE_OPENAI_API_KEY.

Thinking settings and credentials

Thinking options are specific to the provider and the model. A missing field may mean the provider's default, not thinking off. Record both the requested setting and what the provider reports before comparing runs.

Use a key that the provider issues for programmatic API access, with an endpoint and authentication method that the provider documents for it. A command-line tool's login session is not an API key; do not copy login tokens into a generic API client. Do not put keys in Python source, configuration files, trace directories or issue comments.

First run

solve runs one whole session with the default configuration: the built-in runner, the shared decision space, and the textbook autopilot as fallback. Running it with a real provider can incur charges.

import numpy as np
from nexteval import solve

result = solve(
    lambda x: float(x @ x),
    np.array([2.0, -1.0, 0.5]),
    options={"model": "env", "maxfev": 80, "seed": 7, "trace": "off"},
)
print(result.x, result.fun, result.nfev)
print(result.status, result.message)
print(result.runner_report["fallback"])  # whether the autopilot completed the run

One runner owns the solve: it manages the conversation and chooses tools until it finalizes or a limit ends the run. A tool call is not necessarily one function evaluation: reads evaluate nothing, and a numerical method call may evaluate several points. No fixed final value is promised, because the model's choices and its stopping decision can vary.

Offline check without a model

FakeTextModel replays scripted replies, so the default wiring can be exercised without a provider. A model that never calls a tool ends its session at once, and the autopilot spends the budget:

import numpy as np
from nexteval import FakeTextModel, solve

result = solve(
    lambda x: float((x[0] - 1.0) ** 2 + 3.0 * (x[1] + 0.5) ** 2),
    np.array([3.0, 2.0]),
    options={"model": FakeTextModel(["no tool call"]), "maxfev": 40, "trace": "off"},
)
print(result.nfev, result.fun, result.runner_report["fallback"]["reason"])

It prints 40 evaluations and a value near zero; the reason names the session stop that started the fallback.

Example file

optimize.py holds two objectives, a sphere and the Rosenbrock function, both started at [-1.2, 1.0], and runs either with an explicit provider client:

python optimize.py --problem sphere --provider openai --maxfev 80
python optimize.py --problem rosenbrock --provider anthropic --maxfev 100

Runners

A runner connects a model to the tools for one complete solve. The built-in runner and the two adapters receive the same system prompt, user guidance and tool catalog, and follow one session lifecycle: one sampling action per model reply, and one wall-clock deadline for the whole solve.

Built-in runner

Without a runner argument, solve and minimize build NativeToolRunner (also exported as ToolCallingTaskRunner) and wrap it in FallbackRunner. Pass the model with model= and configure the runner through runner_options:

from nexteval import OpenAICompatClient, minimize

result = minimize(
    fun, x0,
    model=OpenAICompatClient(),
    options={
        "maxfev": 200,
        "user_instructions": "Inspect the history before restarting a method.",
        "runner_options": {"max_tokens": 4096, "max_turns": 200},
    },
)

Codex adapter

CodexTaskRunner runs a whole solve in one Codex app-server thread. Codex keeps the conversation history, reasoning and context management; the solver keeps the objective, the evaluation ledger, method state, the tools and the incumbent. It is not a new Codex process per evaluation. Install the Codex CLI separately and sign in with its own procedure; the adapter uses that login in place and does not copy credential files. It uses POSIX pipes and process groups, so Windows is not supported.

from nexteval import CodexTaskRunner, FallbackRunner, minimize

runner = FallbackRunner(CodexTaskRunner(model="your-model-id", effort="medium"))
result = minimize(fun, x0, runner=runner, options={"maxfev": 100, "wall_time_limit_s": 3600})

For a Responses-compatible endpoint, pass base_url and env_from={"OPENAI_API_KEY": "MY_PROVIDER_API_KEY"}; env_from holds the names of environment variables, not secret values. Shell, browser, project instructions, plugins and other ambient tools of the CLI are disabled for the session. This is a trusted local harness, not an operating-system sandbox.

Claude Code adapter

ClaudeCodeTaskRunner runs a whole solve through the Claude Agent SDK and serves the same tools to it. It is an optional extra of the package. Without an explicit endpoint, the SDK uses the existing Claude Code authentication and environment.

from nexteval import ClaudeCodeTaskRunner, FallbackRunner, minimize

runner = FallbackRunner(ClaudeCodeTaskRunner(model="your-model-id", effort="medium", max_turns=200))
result = minimize(fun, x0, runner=runner, options={"maxfev": 200})

Do not also pass model= to minimize with either adapter: the external runner has its own model and authentication, and conflicting selections fail before the objective is evaluated. Authenticate each agent tool only in ways its provider supports for your use; see the Codex authentication page and Anthropic's Agent SDK documentation.

The fallback

FallbackRunner(agent) wraps a runner so that the textbook autopilot spends the remaining budget, in the same run and on the same ledger, when the agent session raises, reports an error or a timeout, or returns without finalize. A finalize the agent chose, or a hard limit the environment enforced, ends the run without a fallback. A runner you pass yourself is used exactly as given, so wrap it as in the two examples above. result.runner_report["fallback"] records whether the fallback ran, why, the evaluations recorded when it started and what it spent.

Your own runner

Any object with a run(task, gateway) method can drive a session through minimize(..., runner=...). It works with the same tools, budget and evaluation records as the built-in runner.

run(task, gateway)
Required. task is a serializable description of the problem: dimension, x0, budgets, instructions and the tool definitions in task["tools"]. It holds no objective handle and no credentials. Return a mapping such as {"stop": "completed"} when the session ends.
gateway.call(name, arguments)
Runs one tool and returns a JSON-serializable result. The solver validates each call, runs the objective evaluations it requires, records them and enforces the budget. gateway.done becomes true when the session has ended.
gateway.begin_turn()
Optional. Call it before forwarding the tool calls of each agent reply. At most one sampling action then runs per reply; further sampling actions in the same reply are deferred without spending budget.
describe()
Returns metadata about the runner, such as a harness_id. Optional for a runner used directly; define it when the runner is wrapped in FallbackRunner.

This scripted runner shows the contract without a model or credentials. Save it next to optimize.py and run it:

from optimize import run


class ScriptedRunner:
    """Stands in for an agent: two poll points, then finalize."""

    def describe(self):
        return {"harness_id": "scripted-example"}

    def run(self, task, gateway):
        # An agent harness exposes task["tools"] to the agent and forwards
        # each tool call it makes through gateway.call(name, arguments).
        gateway.call("execute_action", {
            "tool": "poll_set",
            "parameters": {"basis": "maximal_positive", "count": 2},
            "purpose": "geometry",
            "rationale": "scripted example",
        })
        gateway.call("finalize", {"reason": "scripted example complete"})
        return {"stop": "completed"}


result = run(problem="sphere", runner=ScriptedRunner(), maxfev=80)
print(result.nfev, result.fun, result.model_request_count)

The first printed value is 3: the starting point and the two poll points. For a real agent, replace the scripted calls with a loop that starts the agent, exposes task["tools"] to it, forwards each tool call through gateway.call, and returns when the agent calls finalize or gateway.done is true. The built-in model meter does not see the requests of an external runner, so result.model_request_count is None; set request and token limits in the agent's own configuration.

Budgets and limits

maxfev
Actual objective calls, including the initial point and failed calls. The default is 200. Reads and values answered from the cache consume nothing.
max_evals_per_method_call
The cap for one numerical method call, 40 by default; None removes this local cap.
model_call_limit
model_token_limit
Separate limits on requests through the built-in model meter. Requests made inside an external runner are not visible to this meter.
runner_options
Settings of the built-in runner, such as max_tokens, temperature, compaction and retries. It is refused together with a supplied runner.
task_max_tool_calls
A separate ceiling on tool calls, reads included; 200 by default.
wall_time_limit_s
A cooperative deadline for the whole solve. It does not interrupt a blocking objective function; use a process supervisor when a hard deadline is required.

A finite evaluation budget does not guarantee that all of it is used. The agent's finalize, invalid evaluations, transport failures or another limit may end the session first.

Observation noise

objective_noise states what you know about the objective's observations. It is your declaration; the solver never infers it. It selects the evaluation cache and one statement that every agent receives, and it adds no averaging, noise detection or extra evaluations.

Values of objective_noise
objective_noiseMeaningRepeated points
"unknown" (default)You have not said whether observations are noisy.Evaluated again; each costs one evaluation.
"none"You declare deterministic observations.Answered from the ledger at no cost.
"present"You declare noisy observations.Evaluated again; each costs one evaluation.

Exceptions and non-finite values of the objective are recorded in the ledger and count against maxfev. Because such a failure can belong to one call rather than to the point, the solver measures a point whose first value is non-finite once more before it concludes anything from it.

Results and traces

result.x and result.fun identify the best valid recorded point. result.nfev counts real objective calls and equals the length of the ledger. Inspect status, message, completion_quality and comparable_under_budget as well: a successful termination is not, by itself, a valid trial for a comparison.

Writing files is off by default. Enable a persistent trace explicitly:

options = {"maxfev": 100, "seed": 0, "trace": "trajectory", "run_dir": "runs/example"}
runs/example/<run_id>/
  manifest.json
  summary.json
  events.jsonl

result.trace_path gives the actual directory. trace="summary" writes a compact receipt without the event stream; "trajectory" adds the evaluation and decision history; "debug" and "develop" add tool-level diagnostics; "off" writes nothing. With return_evaluation_history=True the result also carries the evaluation ledger.

The solver keeps the evaluation records. An external agent's own transcript is a separate artifact, not a replacement for them. Before sharing raw traces, check them for sensitive objective or model content.

Troubleshooting

Missing configuration
Check the environment variable names for the client you constructed. model="env" is the Anthropic-compatible client, not provider detection. A missing configuration fails before the objective is evaluated.
Authentication and usage limits
Check the endpoint, your access and the credential locally. A revoked credential is different from an exhausted usage limit. Do not print tokens or repeatedly retry a revoked credential.
No usable tool call
Confirm that the endpoint supports tool calls and the request fields you send. Token truncation, thinking defaults and transport mismatches can stop a session early; with the default configuration the autopilot then spends the remaining budget, and the runner report says so.
Unknown option
Unknown option names are rejected before the objective is evaluated. The option reference lists every name.
Invalid objective
Check that fun(x) returns a scalar and that x0 is finite and one-dimensional. Failed objective calls still count.

How this guide is checked

The example file, the scripted runner, the offline check and the fallback of a supplied runner are run offline against local scripted responses; the scope and the results are in the guide checks. No paid provider run was made for this guide.

The offline checks cover request serialization, tool responses, evaluation budgets, the fallback and trace files. They do not show provider access, provider compatibility or model quality, and they do not start the Codex or Claude Code tools.