End-to-End Smoke Test Guide

Overview

The smoke test validates the full Daedalus pipeline end-to-end:

NATS queue (A2A SendMessageRequest) -> Proxy (queue consumer) -> ACP Client (TCP JSON-RPC 2.0) -> Copilot CLI (GitHub Copilot in ACP mode) -> Result back through NATS

This exercises every component in the stack with a real GitHub Copilot CLI, not a mock. It verifies that prompts flow through the entire pipeline and produce valid A2A Task results.

Prerequisites

• Docker and Docker Compose (v2+ with docker compose syntax)
• Go 1.25+ (for the Go smoke test)
• GITHUB_TOKEN with GitHub Copilot access (fine-grained PAT with "Copilot Requests: read" permission). Store in smoke.env at the repo root (gitignored) - see Credentials Setup below.
• Available ports:
• 4222 - NATS client connections
• 8222 - NATS monitoring HTTP
• 3000 - Copilot CLI ACP listener

Credentials Setup

The smoke test requires a GitHub fine-grained PAT with Copilot Requests: read permission.

1. Generate a token at https://github.com/settings/tokens?type=beta
2. Select Copilot > Requests > Read-only
3. Add your token to smoke.env at the repo root (this file is gitignored):

GITHUB_TOKEN=github_pat_...your-token-here...

1. The make test-smoke target auto-sources smoke.env if GITHUB_TOKEN is not already set in the environment. You can also export it manually:

bash export GITHUB_TOKEN=github_pat_... make test-smoke

Security: smoke.env is listed in .gitignore and must never be committed. Do not add it to .env files that are tracked by git.

Quick Start

```bash

Option A: use smoke.env (recommended)

Copy smoke.env.example to smoke.env and fill in GITHUB_TOKEN

cp smoke.env.example smoke.env

Edit smoke.env and set GITHUB_TOKEN=

make test-smoke

Option B: set env var directly

GITHUB_TOKEN= make test-smoke ```

Using the validation script (bash)

The bash script is a standalone validator with timing instrumentation:

```bash

Set GITHUB_TOKEN in smoke.env or environment, then:

./test/scripts/validate-copilot-cli.sh ```

This builds images, starts the stack, creates NATS streams, publishes a test task, waits for the result, and prints a latency summary table.

Using the Go smoke test

The Go test provides structured assertions, parallel-safe task IDs, and detailed latency measurement:

bash make test-smoke

Or directly with go test (requires GITHUB_TOKEN in env):

bash go test ./test/integration/... -tags=smoke -v -count=1 -timeout=600s

Using the Go integration test (mock ACP, no token needed)

For faster iteration without a real Copilot CLI:

bash make test-integration

This uses the mock ACP server and the standard docker-compose.yml stack.

What Gets Tested

The smoke test validates each stage of the pipeline:

1. Proxy connects to Copilot CLI via ACP (TCP JSON-RPC 2.0 on port 3000)
2. ACP capability negotiation (initialize handshake)
3. Session creation (session/new)
4. Prompt delivery and streaming response (session/prompt)
5. Result published to NATS with correct A2A Task format
6. Status transitions: working -> completed
7. Task ID propagation through the full pipeline (publish -> status -> result)

Go Test Cases

Test	What It Validates
TestSmoke_EndToEnd	Full round-trip: publish task, receive result, verify completed state and non-empty artifacts. Logs latency at each step.
TestSmoke_StatusTransitions	Correct state machine ordering: working precedes completed. Captures error messages on failure.
TestSmoke_TaskIDPropagation	Task ID flows correctly through NATS subjects, status updates, and result payload.

Expected Latencies

Measured against @github/copilot@1.0.36 with the auto model (Claude Sonnet 4.6), running in Docker on an Azure VM, 2025-05-01:

Phase	Measured	Notes
Publish -> First Status	~1ms	Proxy picks up task almost instantly
First Status -> Complete	~53s	Includes ACP round-trips and model inference
Total Round-Trip	~53s	publish to final result on NATS

The Go test logs per-step latency:

=== Smoke Test Latency === Publish -> First Status: 1ms First Status -> Complete: 53148ms Total Round-Trip: 53150ms Status Transitions: [working completed]

Latency varies with model response time and whether the agent uses tools (file creation prompts trigger one or more session/request_permission round-trips before the model responds, adding 5-15s per tool call).

Interpreting Results

Success

• All tests pass
• Status transitions show: working -> completed
• Artifacts contain non-empty response text
• Latency summary shows reasonable times (see table above)

Common Failures

Symptom	Likely Cause	Fix
"GITHUB_TOKEN must be set"	Missing or invalid token	Export a valid PAT with Copilot scope
Timeout waiting for health	Copilot CLI failed to start ACP listener	Check docker compose logs copilot-cli
Task state: failed	ACP protocol error or CLI crash	Check proxy logs: docker compose logs proxy
Connection refused on 4222	NATS not started	Check docker compose logs nats
"no status updates received"	Proxy not publishing status	Verify proxy connects: docker compose logs proxy
Task ID mismatch	Proxy not propagating ID	Check proxy task handling logic

Architecture

+-------+ +-------------+ +---------+ +-------------+ | NATS | --> | Proxy | --> | ACP | --> | Copilot CLI | | Queue | | (consumer) | | Client | | (ACP mode) | +-------+ +-------------+ +---------+ +-------------+ ^ | | JSON-RPC 2.0 over TCP | +----------------------------------------------------+ | Results + Status published to NATS |

NATS JetStream Subjects: - agent.tasks.> (AGENT_TASKS stream) - incoming task requests - agent.results.> (AGENT_RESULTS stream) - completed task results - agent.status.> (AGENT_STATUS stream) - status updates (working, completed, failed)

Docker Compose Services: - nats - NATS server with JetStream enabled - nats-box - Utility container for NATS CLI operations - copilot-cli - GitHub Copilot CLI running in ACP mode - proxy - Daedalus proxy bridging NATS to ACP

Debugging

View all container logs

bash docker compose -f deploy/docker/docker-compose.copilot-cli.yml logs

View specific service logs

bash docker compose -f deploy/docker/docker-compose.copilot-cli.yml logs proxy --tail=50 docker compose -f deploy/docker/docker-compose.copilot-cli.yml logs copilot-cli --tail=50 docker compose -f deploy/docker/docker-compose.copilot-cli.yml logs nats --tail=50

NATS monitoring

Open http://localhost:8222 in a browser for the NATS monitoring dashboard.

Useful endpoints: - http://localhost:8222/jsz - JetStream information - http://localhost:8222/connz - Connection details - http://localhost:8222/subsz - Subscription details

Check NATS streams

bash docker compose -f deploy/docker/docker-compose.copilot-cli.yml exec nats-box \ nats --server nats://nats:4222 stream ls

Check stream contents

bash docker compose -f deploy/docker/docker-compose.copilot-cli.yml exec nats-box \ nats --server nats://nats:4222 stream view AGENT_RESULTS

Manual stack management

```bash

Start stack

docker compose -f deploy/docker/docker-compose.copilot-cli.yml up -d

Check service status

docker compose -f deploy/docker/docker-compose.copilot-cli.yml ps

Tear down (including volumes)

docker compose -f deploy/docker/docker-compose.copilot-cli.yml down -v ```