Skip to main content

Configuration

Control and Egress accept -config PATH with a strict, versioned JSON file. Unknown fields are rejected so typos do not silently change behavior. Input is bounded to 4 MiB before decoding. Without -config, both binaries use local defaults.

Control​

{
  "config_version": "v1",
  "control": {
    "server": {"host": "0.0.0.0", "api_port": 8080, "metrics_port": 9090},
    "request": {
      "max_inline_request_body_bytes": 1048576,
      "max_inline_response_body_bytes": 1048576,
      "max_timeout_ms": 120000
    },
    "transport": {"max_frame_data_bytes": 1048576},
    "nats": {"servers": ["nats://127.0.0.1:4222"]},
    "runtime_admin": {"enabled": false},
    "object_storage": {"enabled": false}
  }
}

Set STRAW_AUTH_TOKEN to require Authorization: Bearer <token> on REST requests and Proxy-Authorization: Bearer <token> on forward-proxy requests. An unset token permits requests and is appropriate only for a loopback or otherwise trusted development network.

Forward-proxy routing hints use the reserved X-Straw-* request-header namespace; they do not add configuration fields. The supported headers, limits, normalization, and stripping behavior are documented in HTTP and HTTPS proxy ingress. The configured token is checked before those values are parsed. Routing rules, executor-pool capability filters, sticky TTLs, and allow_sticky_fallback apply equally to REST, absolute-form proxy, and CONNECT ingress modes.

Optional runtime administration​

runtime_admin.enabled opts into durable runtime configuration. Its defaults are token_env: "STRAW_ADMIN_TOKEN", bucket: "STRAW_RUNTIME_CONFIG", and history_limit: 64. The named token environment variable must be non-empty, and NATS must have JetStream file storage enabled. See Runtime administration.

Runtime snapshot validation​

The Admin API accepts a complete snapshot, not a partial patch. Before compare-and-swap persistence or activation, Control validates route match fields, pool tags and capability lists, sticky settings, destination rules, injection policies, fingerprint records, and worker settings. A rejected snapshot returns 422 and leaves the durable record, active cache, history, and worker publication unchanged.

Destination policy rules use the raw representation below. The normalized fields are server-derived response fields and must not be used as client input. host takes a hostname; host_suffix and cname_suffix take a hostname or a leading *./. suffix; cidr takes a CIDR; ip takes one IP; private_range takes a private CIDR; and metadata_ip takes one of Straw's recognized metadata addresses. Control canonicalizes these values before returning the stored record and clears derived fields from a previous pattern:

{
  "destination_policy": [
    {"id": "deny-api", "rule_type": "host_suffix", "action": "deny", "enabled": true, "raw_pattern": "*.Example.COM"},
    {"id": "deny-net", "rule_type": "cidr", "action": "deny", "enabled": true, "raw_pattern": "192.168.1.42/24"}
  ]
}

Use action: "deny" for a normal block or action: "allow_override" only for an explicitly reviewed exception; the latter is checked against resolved addresses and built-in safety ranges as described in the architecture guide.

The response contains normalized_host: "example.com" and normalized_cidr: "192.168.1.0/24". Supplying malformed patterns, unsupported actions, invalid header operations or values, duplicate policy records, or fingerprint metadata that the official Egress worker cannot execute is rejected at PUT time. Existing snapshots that predate raw patterns remain readable when their normalized destination field is valid.

Complete runtime snapshot example​

The Admin API replaces the complete runtime snapshot. A request that contains only one field is not a partial update; it is an invalid or incomplete policy unless the omitted collections are intentionally empty. This is the smallest useful snapshot for a deployment with one official Egress pool and a default route:

{
  "config_version": 1,
  "default_timeout_ms": 60000,
  "max_timeout_ms": 300000,
  "routing_rules": [
    {
      "id": "default",
      "priority": 100,
      "enabled": true,
      "target_pool_id": "default",
      "match": {}
    }
  ],
  "executor_pools": [
    {"id": "default", "executor_type": "egress", "enabled": true}
  ],
  "destination_policy": [],
  "injection_policies": [],
  "fingerprint_profiles": [
    {
      "name": "default",
      "enabled": true,
      "supported_by_worker": true,
      "executor_type": "egress",
      "profile_ref": "default"
    }
  ],
  "worker_settings": []
}

config_version is numeric inside a runtime snapshot and is advanced by Control. The static file envelope uses the separate string literal "config_version": "v1"; do not copy the file envelope into the Admin API. Snapshot timeout values are milliseconds. An omitted request timeout uses default_timeout_ms and is capped at max_timeout_ms.

Runtime header injection​

Injection policies are enabled groups of ordered set, append, and remove operations. Enabled policies are applied in policy-ID order; operations within a policy keep their declared order. Values are standard base64 and are decoded before the upstream request is built. This example sets a trace source, appends a second value, and removes a debug header:

{
  "injection_policies": [
    {
      "id": "trace",
      "enabled": true,
      "operations": [
        {"op": "set", "header_name": "X-Trace-Source", "value_base64": "c3RyYXc="},
        {"op": "append", "header_name": "X-Trace-Source", "value_base64": "c2VydmljZQ=="}
      ]
    },
    {
      "id": "remove-debug",
      "enabled": true,
      "operations": [
        {"op": "remove", "header_name": "X-Debug", "value_base64": ""}
      ]
    }
  ]
}

Header names must be valid HTTP field names and are limited to 256 bytes at snapshot validation. A duplicate enabled set for the same header is rejected. Host, Content-Length, Transfer-Encoding, Connection, Proxy-Authorization, and every X-Straw-* header are reserved and cannot be injected. The aggregate injected name and decoded-value bytes must fit control.transport.max_frame_data_bytes; CR/LF and non-standard base64 are rejected.

Executor pools and routing rules are part of the runtime snapshot. A pool is deployment-scoped: its enabled flag, exact executor_type, required tags, degraded-worker policy, and allowed country/region/IP-type lists are enforced when Control selects a worker. For example, a deployment can route residential traffic to a dedicated pool:

{
  "config_version": 1,
  "executor_pools": [
    {
      "id": "default",
      "executor_type": "egress",
      "enabled": true
    },
    {
      "id": "residential-au",
      "executor_type": "egress",
      "tags": ["residential"],
      "enabled": true,
      "allow_degraded_workers": false,
      "allowed_countries": ["AU"],
      "allowed_regions": ["ap-southeast-2"],
      "allowed_ip_types": ["residential"]
    }
  ],
  "routing_rules": [
    {
      "id": "residential-au",
      "priority": 10,
      "enabled": true,
      "target_pool_id": "residential-au",
      "match": {"country": "AU", "ip_type": "residential"}
    }
  ]
}

enabled: false stops new assignments to that pool while existing requests continue. A worker is eligible only when it claims membership in the target pool, has the pool's exact executor type, advertises every required pool tag, and keeps all claimed countries, regions, and IP types within the pool's allowed lists. A non-empty request constraint must also be advertised by the worker. allow_degraded_workers controls whether a live degraded worker may be used. Unknown pool references and duplicate pool memberships are rejected before registration or activation.

An optional upstream_proxy object changes a pool from direct-local execution to trusted upstream-proxy remote resolution. It contains only the profile identity and the explicit trust acknowledgement:

{
  "executor_pools": [
    {
      "id": "brightdata-residential-au-v1",
      "executor_type": "egress",
      "enabled": true,
      "tags": ["residential"],
      "allowed_countries": ["AU"],
      "allowed_ip_types": ["residential"],
      "upstream_proxy": {
        "id": "brightdata-resi-v1",
        "trusted_remote_resolution": true
      }
    }
  ],
  "routing_rules": [
    {
      "id": "retailer-products-v1",
      "priority": 10,
      "enabled": true,
      "match": {
        "target_host": "*.retailer.example",
        "country": "AU",
        "ip_type": "residential"
      },
      "target_pool_id": "brightdata-residential-au-v1",
      "sticky_session_ttl_seconds": 900,
      "allow_sticky_fallback": true
    }
  ]
}

An absent upstream_proxy keeps direct-local behavior. When present, id is required and trusted_remote_resolution must be exactly true; false is a snapshot validation error, not a request-time direct fallback. Use a fresh pool ID that was never claimed by a protocol-minor-0 or minor-1 worker. Control admits a worker to the pool only when capabilities.allowed_pools[].upstream_proxy_id exactly matches this ID and the negotiated protocol minor is at least 2. Multiple pools may intentionally share a profile ID, but one pool maps to at most one profile.

Existing exact and *.suffix target_host matching selects the proxy pool; callers cannot name arbitrary profiles. Control stores no proxy endpoint, username, password, or authentication template. Those values remain worker-local.

Optional shared runtime state​

runtime_state.backend defaults to memory. Set it to redis only when multiple Control instances must be interchangeable. Redis credentials belong in the URL named by redis_url_env (default STRAW_REDIS_URL); both redis:// and TLS-protected rediss:// URLs are accepted.

{
  "runtime_state": {
    "backend": "redis",
    "redis_url_env": "STRAW_REDIS_URL",
    "key_prefix": "straw",
    "instance_id_env": "STRAW_CONTROL_INSTANCE_ID",
    "worker_ttl_ms": 30000,
    "request_ttl_ms": 130000,
    "instance_ttl_ms": 15000,
    "operation_timeout_ms": 1000
  }
}

request_ttl_ms must exceed request.max_timeout_ms. Instance IDs must be unique NATS subject tokens; if the named environment variable is empty, Control generates a process-unique ID. See Highly available Control for TTL and outage behavior.

Optional object storage​

object_storage.enabled defaults to false. The local backend defaults to .straw/objects; the s3 backend requires endpoint and bucket. Common defaults are 1 GiB maximum objects, 16 MiB maximum parts, 24-hour retention, five-minute assignment URLs, and hourly cleanup. download_base_url must be reachable by Egress.

Secrets are read from signing_key_env (STRAW_RECEIPT_SIGNING_KEY), access_key_env, secret_key_env, and session_token_env; never place their values in JSON. The signing key must contain at least 32 bytes. S3 server-side encryption accepts AES256 or aws:kms; the latter also requires kms_key_id. See Object storage and receipts for the full lifecycle and examples.

Egress​

{
  "config_version": "v1",
  "egress": {
    "worker_id": "egress-1",
    "heartbeat_interval_ms": 5000,
    "health_port": 8090,
    "capabilities": {
      "max_concurrency": 4,
      "allowed_pools": [
        {"pool_id": "default"},
        {"pool_id": "residential-au"}
      ]
    },
    "nats": {"servers": ["nats://127.0.0.1:4222"]},
    "upstream_connection_pool": {"enabled": false},
    "http2": {"enabled": false, "fallback_cache_ttl_ms": 300000}
  }
}

Worker IDs must be unique within a deployment. max_concurrency defaults to 4. When allowed_pools is omitted, the official worker claims default/default, preserving the original behavior. Each entry may omit deployment_id, which defaults to default; entries must be unique and refer to the deployment's configured pools. The official worker does not provide tenant or cross-deployment authorization. The optional connection pool can reuse direct-local upstream connections; when enabled its defaults are 8 idle connections per deployment/host, 30 seconds idle timeout, and 5 minutes maximum lifetime.

Upstream proxy profiles​

Proxy-backed workers bind each claimed pool to an exact worker-local profile and keep provider credentials in named environment variables:

{
  "config_version": "v1",
  "egress": {
    "worker_id": "egress-proxy-1",
    "capabilities": {
      "allowed_pools": [
        {
          "pool_id": "brightdata-residential-au-v1",
          "upstream_proxy_id": "brightdata-resi-v1"
        }
      ],
      "countries": ["AU"],
      "ip_types": ["residential"],
      "max_concurrency": 64
    },
    "upstream_proxies": [
      {
        "id": "brightdata-resi-v1",
        "endpoint": "http://brd.superproxy.io:22225",
        "auth": {
          "type": "basic",
          "username_env": "BRIGHTDATA_USERNAME",
          "password_env": "BRIGHTDATA_PASSWORD",
          "username_template": "{{.Username}}{{if .Country}}-country-{{lower .Country}}{{end}}{{if .Session}}-session-{{.Session}}{{end}}"
        },
        "defaults": {
          "country": "AU",
          "ip_type": "residential"
        }
      }
    ]
  }
}

Profile constraints are:

  • id uses the bounded snapshot-ID syntax and must be unique; every profile must be referenced by at least one allowed pool, and every non-empty pool upstream_proxy_id must reference a configured profile;
  • endpoint must be an http or https URL with an explicit hostname and port. User information, non-root paths, query strings, and fragments are rejected;
  • auth.type is none or basic. none permits no credential fields. basic requires a named, non-empty username_env; password_env is optional, but when named it must exist and may resolve to an empty password;
  • secret values are read once from the environment at startup. JSON contains environment-variable names, never values;
  • a Basic username_template defaults to {{.Username}}. Only lower and upper functions and the fields Username, Session, Country, Region, and IPType are available;
  • effective Country, Region, and IPType values use the per-request Control instruction first, then the profile's defaults. Username comes from username_env, and Session is Control's pseudonymous provider session value;
  • defaults must be normalized and cannot contradict the worker's advertised capabilities. Use conditionals to avoid provider delimiters when optional values are empty.

Straw never consults process HTTP_PROXY, HTTPS_PROXY, NO_PROXY, or equivalent variables. It always opens an HTTP CONNECT tunnel for decoded HTTP, decoded HTTPS, named TLS fingerprints, HTTP/2 targets, and raw CONNECT ingress. A TLS proxy endpoint adds a separate verified outer TLS connection. Application-level proxy connection pooling is disabled: every proxied request opens a new CONNECT tunnel even when upstream_connection_pool.enabled is true; direct-local pooling is unchanged.

The redacted multi-provider example illustrates Bright Data, Oxylabs, Apify Proxy, and Scrape.do proxy mode. Gateway and username contracts can change; verify them against the provider account before rollout. The file has no credentials and is not selected by the production Compose stack.

The official Egress worker advertises the complete built-in fingerprint catalogue. The default Control snapshot enables those exact profiles with contract revision tls-client-v1.15.1-http1-http2; use their names in fingerprint_profile. Runtime-admin snapshots may disable profiles but cannot make an unknown profile executable. See Compatibility for the normative list and PSK behavior.

NATS authentication​

Both services support:

  • user_credentials_file: path to a NATS credentials file; or
  • username_env and password_env: names of environment variables containing credentials.

The production example uses STRAW_NATS_USER and STRAW_NATS_PASSWORD. Never put secret values directly in the JSON files. Connection tuning fields are reconnect_attempts, reconnect_wait_ms, ping_interval_ms, max_ping_failures, and max_payload_bytes.

The selected deploy/local and deploy/production service JSON files are canonical working examples. Files ending in .example.json are redacted configuration templates and are not selected by Compose.

Static field reference​

Static fields take effect only after the affected process restarts. Environment values are read at startup; JSON stores environment-variable names, never secret values.

ObjectFieldDefault / constraint
fileconfig_versionrequired literal v1; exactly one Control or Egress object
Control serverhost, api_port, metrics_port0.0.0.0, 8080, 9090; ports 1–65535
Control requestmax_inline_request_body_bytes, max_inline_response_body_bytes, max_timeout_ms1 MiB, 1 MiB, 120000 ms; positive bounded request behavior
Control transportmax_frame_data_bytes1 MiB and compatible with NATS max_payload_bytes
runtime adminenabled, token_env, bucket, history_limitfalse, STRAW_ADMIN_TOKEN, STRAW_RUNTIME_CONFIG, 64; history 1–64 when enabled
runtime statebackend, redis_url_env, key_prefix, instance_id_envmemory, STRAW_REDIS_URL, straw, STRAW_CONTROL_INSTANCE_ID; backend memory or redis
runtime state TTLworker_ttl_ms, request_ttl_ms, instance_ttl_ms, operation_timeout_ms30000, 130000, 15000, 1000; all positive and request TTL greater than max request timeout
object storage identityenabled, backend, local_directory, endpoint, bucket, regionfalse, local, .straw/objects, empty, empty, us-east-1; S3 requires absolute endpoint and bucket
object storage secret namesaccess_key_env, secret_key_env, session_token_env, signing_key_envSTRAW_S3_ACCESS_KEY, STRAW_S3_SECRET_KEY, STRAW_S3_SESSION_TOKEN, STRAW_RECEIPT_SIGNING_KEY
object storage limitsdownload_base_url, max_object_bytes, max_part_bytes, retention_seconds, assignment_ttl_seconds, cleanup_interval_secondshttp://control:8080, 1 GiB, 16 MiB, 86400, 300, 3600; positive; part ≤ object
object encryptionserver_side_encryption, kms_key_idempty; AES256 or aws:kms; KMS mode requires key ID
Egress identityworker_id, heartbeat_interval_ms, health_portegress-1, 5000, 8090; non-empty/positive valid port
Egress capabilitiesallowed_pools, tags, countries, regions, ip_types, supported_ingress_modes, max_concurrencyallowed_pools defaults to [{"deployment_id":"default","pool_id":"default"}]; other lists are empty except ingress rest, http_proxy, connect; official workers advertise the built-in fingerprint catalogue; concurrency defaults to 4 at worker composition
Egress upstream proxiesupstream_proxies, id, endpoint, auth, defaultsempty; each profile is worker-local, uniquely identified, referenced by an allowed pool, and uses an explicit HTTP/HTTPS host and port
Egress proxy authtype, username_env, password_env, username_templatenone or basic; Basic username is required from the environment, password is optional, template defaults to {{.Username}}
connection poolenabled, max_idle_conns_per_host, idle_timeout_ms, max_lifetime_msfalse; when enabled 8, 30000, 300000
HTTP/2enabled, fallback_cache_ttl_msfalse, 300000
NATSservers, user_credentials_file, username_env, password_envnats://127.0.0.1:4222, empty; credential file or named user/password environment variables
NATS livenessreconnect_attempts, reconnect_wait_ms, ping_interval_ms, max_ping_failures, max_payload_bytes10, 2000, 30000, 3, server-discovered; payload must fit configured frames/inline limits

Complete field index​

The following names are the normative index for runtime snapshot and optional-profile fields. Defaults are produced by the validated configuration loader; omitted optional values retain those defaults. Every change requires restart unless it is part of a runtime snapshot activated through the Admin API.

ObjectFields and constraints
routing rulerouting_rules, id, priority, enabled, match, target_pool_id, sticky_session_ttl_seconds, allow_sticky_fallback; priorities define ordering, referenced pools must exist, and sticky fallback requires a positive TTL
matchtags, country, region, ip_type, ingress_type, target_host; omitted members do not restrict the match
executor poolexecutor_pools, id, enabled, executor_type, tags, allow_degraded_workers, allowed_ip_types, allowed_countries, allowed_regions, upstream_proxy, trusted_remote_resolution; pool IDs are unique; disabled pools receive no new assignments; executor type, exact proxy identity, and tags are hard worker-eligibility constraints; non-empty allowed lists bound the worker's advertised capabilities
destination ruledestination_policy, rule_type, action, reason, raw_pattern, normalized_host, normalized_cidr, normalized_ip, normalized_name; raw patterns are authoritative, normalized fields are server output, and malformed rules are rejected before activation
injection policyinjection_policies, id, enabled, operations, op, header_name, value_base64; policies are applied by ID, operations preserve declared order, reserved headers and duplicate enabled set operations are rejected
fingerprint profilefingerprint_profiles, name, scope_type, supported_by_worker, executor_type, profile_ref, contract_revision; activation requires worker support
worker settingworker_settings, worker_id, enabled, draining; lifecycle changes are deployment-scoped
snapshotconfig_version, default_timeout_ms, max_timeout_ms; versions increase and timeout bounds must be positive and ordered
Egress capabilitiesallowed_pools, upstream_proxy_id, countries, regions, ip_types, supported_ingress_modes; values describe pool membership and admission capabilities rather than network authorization; omitted pool membership is default/default; duplicate, unknown, mismatched, or non-default deployment references are rejected
Egress upstream profileupstream_proxies, id, endpoint, auth, type, username_env, password_env, username_template, defaults, country, region, ip_type; profiles are immutable startup configuration, credentials come from named environment variables, and unused profiles are rejected
upstream connection poolmax_idle_conns_per_host, idle_timeout_ms, max_lifetime_ms; zero/negative invalid values are rejected by config validation
object storagelocal_directory, max_part_bytes, assignment_ttl_seconds, cleanup_interval_seconds, server_side_encryption; local storage is development-only and production encryption/retention are operator responsibilities