From 07d9c15be9d764d32c3b40f8c2708b9a23f8cc31 Mon Sep 17 00:00:00 2001 From: lolwierd Date: Fri, 8 May 2026 17:45:37 +0530 Subject: [PATCH] feat: buckets --- README.md | 5 ++- skills/excloud-cli/SKILL.md | 76 +++++++++++++++++++++++++++++++++++-- 2 files changed, 76 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 2209736..a542105 100644 --- a/README.md +++ b/README.md @@ -44,9 +44,9 @@ By default skills land under the agent's standard directory (`~/.claude/skills/` ### `excloud-cli` -Safe end-to-end control of Excloud resources through the `exc` CLI. Covers compute (create / inspect / resize / restart / terminate, delete protection, exec / scp / console), networking (subnets, public IPv4, security groups and rules/bindings), volumes and snapshots, SSH keys, Kubernetes (clusters, workers, kubeconfig fetch / merge), IAM (accounts, service accounts, API keys, policies), billing, quota, serial console logs, and metrics. +Safe end-to-end control of Excloud resources through the `exc` CLI. Covers compute (create / inspect / resize / restart / terminate, delete protection, exec / scp / console), networking (subnets, public IPv4, security groups and rules/bindings), volumes and snapshots, SSH keys, Kubernetes (clusters, workers, kubeconfig fetch / merge), object storage buckets (bucket lifecycle, object copy/sync/delete/share, access keys, multipart uploads), IAM (accounts, service accounts, API keys, policies), billing, quota, serial console logs, and metrics. -**Use when:** the user asks to plan or run `exc` commands, provision / introspect / tear down VMs, attach a public IP, adjust a security group, pull a kubeconfig, debug a stuck boot via serial logs, or exec / scp against a VM. +**Use when:** the user asks to plan or run `exc` commands, provision / introspect / tear down VMs, attach a public IP, adjust a security group, pull a kubeconfig, manage buckets or S3 access keys, debug a stuck boot via serial logs, or exec / scp against a VM. **Key guidance the skill encodes:** @@ -54,6 +54,7 @@ Safe end-to-end control of Excloud resources through the `exc` CLI. Covers compu - Discovery first: lookup tables (`exc compute instancetype list`, `image list`, `subnet list`, `securitygroup list`, ...) as the source of truth — no hard-coded IDs. - Safety guardrails around `terminate`, `publicip release`, `rule delete`, `cluster delete`, `apikey delete`, and destructive shell commands over `exec`. - Interactive access patterns: when to use `exec` (one-shot, bash-interpreted), when `scp` (upload/download, symlinks rejected), when `console` (interactive TTY, SSH ↔ WS fallback). +- Bucket workflows: safe recursive deletes, command-specific remote refs (`bucket/key` and `s3://bucket/key`, with `buckets cp` requiring `s3://` for remote operands), auto multipart uploads, `sync --dry-run`, presigned URLs, and AWS profile configuration for `buckets.excloud.dev`. - Output-format buckets so agents pipe to `awk`/`jq` correctly. - A cheat sheet of error messages the CLI actually emits, paired with what each means and how to fix it. diff --git a/skills/excloud-cli/SKILL.md b/skills/excloud-cli/SKILL.md index 59305a1..2341941 100644 --- a/skills/excloud-cli/SKILL.md +++ b/skills/excloud-cli/SKILL.md @@ -1,6 +1,6 @@ --- name: excloud-cli -description: Drive Excloud resources (compute, networking, security groups, volumes, snapshots, public IPs, IAM, billing, Kubernetes) through the `exc` CLI. Use when a user asks to plan or execute `exc` commands - creating / inspecting / updating / deleting VMs, running commands on them via `exec` / `scp` / `console`, managing security groups and public IPs, or pulling Kubernetes kubeconfigs - with safety guardrails and auth checks. +description: Drive Excloud resources (compute, networking, security groups, volumes, snapshots, public IPs, IAM, billing, Kubernetes, object storage buckets) through the `exc` CLI. Use when a user asks to plan or execute `exc` commands - creating / inspecting / updating / deleting VMs, running commands on them via `exec` / `scp` / `console`, managing security groups and public IPs, pulling Kubernetes kubeconfigs, or managing buckets/objects/access keys - with safety guardrails and auth checks. --- # Excloud CLI @@ -47,6 +47,8 @@ Require explicit user confirmation before running any of these: - `exc securitygroup delete`, `exc securitygroup rule delete`, `exc securitygroup binding delete`. - `exc k8s cluster delete`, `exc k8s cluster worker delete`. - `exc account revoke`, `exc serviceaccount delete`, `exc apikey delete`, `exc policy delete`, `exc policy binding delete`. +- `exc buckets delete`, `exc buckets rm`, `exc buckets objects delete`, recursive bucket/object deletes, and `exc buckets multipart abort`. +- `exc buckets keys delete`; also treat `exc buckets keys create` output as sensitive because the secret is shown once. For shell commands delivered through `exc compute exec` or an `exec` script file, refuse or confirm explicitly before running anything like `shutdown`, `reboot`, `rm -rf`, `mkfs`, `dd`, `wipefs`, rewrites of `/etc/fstab`, bootloader edits, or `systemctl stop ssh*` (the last one will make the VM unreachable over SSH — see Interactive access). @@ -60,6 +62,9 @@ The skill does _not_ hard-code IDs, instance type names, image IDs, subnet IDs, - `exc compute subnet list` + `exc compute subnet get --id ` — check `DISABLE_IPV4_PUBLIC_IP`: subnets with this set cannot take `--allocate_public_ipv4=true` at create time. - `exc securitygroup list` + `exc securitygroup rule list --security_group_id ` + `exc securitygroup binding list --security_group_id ` (or `--interface_id `) — confirm what a SG allows and where it's bound before relying on it. - `exc compute publicip list` / `exc compute key list` / `exc compute volume list` / `exc compute snapshot list` — authoritative inventories for each resource type. +- `exc buckets list` / `exc buckets get ` / `exc buckets usage` — authoritative object-storage inventory and quota checks. +- `exc buckets objects list [--prefix ] [--all]` — discover keys before copy/delete/share operations. +- `exc buckets keys list` and `exc buckets multipart list ` — discover S3 access keys and abandoned multipart uploads before managing them. If `--help` on the installed CLI shows commands or flags not documented here, prefer `--help`. @@ -275,6 +280,69 @@ exc compute volume list # the root volume should disappear / move to - `exc k8s cluster kubeconfig merge --cluster_id [--kubeconfig ] [--backup=true|false]` — merges into `~/.kube/config` (or `--kubeconfig`) using `kubectl config view --merge --flatten --raw`. Requires `kubectl` on PATH. `--backup` defaults to `true` and writes `.bak`, `.bak1`, ... before overwriting. - `exc k8s bootstrap controlplane get --vm_id --x-exc-imds-token ` — operator bootstrap path; the IMDS token must come from inside the VM's IMDS agent, not be invented. +## Object storage buckets + +The buckets API endpoint is `https://buckets.excloud.dev`. The `exc buckets` commands are for Excloud object storage management; S3-compatible clients use bucket access keys configured through `exc buckets keys configure`. + +Always check the installed CLI first: + +```bash +exc buckets --help +exc buckets objects --help +exc buckets keys --help +exc buckets multipart --help +``` + +- `exc buckets list [--json]` — table of bucket name, public flag, object count, size, and creation time. +- `exc buckets create [--public]` — bucket names are lowercase letters, digits, and hyphens, 3-63 chars, no leading/trailing hyphen. +- `exc buckets get ` — JSON bucket detail, including public URL when public. +- `exc buckets update --public=true|false` +- `exc buckets usage` — JSON org-wide bucket/object/byte counters and quotas. +- `exc buckets delete [--force] [--yes] [--json]` — destructive. Without `--force`, expect the API to refuse non-empty buckets; with `--force`, the CLI empties the bucket via object listing and bulk delete before deleting. Confirm unless the user already authorised it. +- `exc buckets ls` — lightweight bucket table with bucket name, size, and creation time. +- `exc buckets ls [--all] [--json]` — object listing shortcut. + +Remote object references are command-specific. Commands that take a combined remote ref generally accept either `bucket/key` or `s3://bucket/key`; commands with explicit ` ` forms also accept those explicit args. Two important exceptions: `exc buckets cp` treats operands as remote only when they use `s3://bucket/key`, and `exc buckets objects list` only splits a combined single arg when it is `s3://bucket/prefix`; use `exc buckets objects list [prefix]` or `--prefix` for the plain form. + +- `exc buckets objects list [prefix] [--prefix ] [--cursor ] [--all] [--json]` +- `exc buckets objects upload [--key ] [--content-type ] [--recursive]` + - Files over the single-upload cap are automatically uploaded using multipart. + - Directories require `--recursive`. + - If `--key` ends in `/`, the local basename is appended. +- `exc buckets objects download [--output ]` — also accepts a single `bucket/key` or `s3://bucket/key` arg. +- `exc buckets objects copy ` — server-side copy; no local data transfer. +- `exc buckets objects delete [--recursive] [--yes] [--json]` — destructive. `--recursive` expands the first key as a prefix and deletes matching objects in chunks. +- `exc buckets objects presign [--ttl ]` — also accepts a single `bucket/key` or `s3://bucket/key` arg. Scriptable: prints only the URL. +- `exc buckets objects share [--ttl ]` — also accepts a single `bucket/key` or `s3://bucket/key` arg. Prints the URL and expiry. +- `exc buckets objects mkdir ` — also accepts a single `bucket/prefix` or `s3://bucket/prefix` arg. Creates a virtual folder placeholder; appends `/` if missing. +- `exc buckets objects sync [--delete] [--dry-run] [--exclude ] [--include ]` + - Use `--dry-run` before a large sync, especially with `--delete`. + - The CLI compares size and ETag when possible. Multipart-style ETags may not be reproducible locally, so changed-vs-skipped decisions should be treated conservatively. +- `exc buckets cp [--recursive] [--content-type ]` + - Remote operands must use `s3://bucket/key`; plain `bucket/key` is local for this command. + - local file → remote uploads one file. + - local directory → remote requires `--recursive`. + - remote → local downloads one object, or a prefix with `--recursive`. + - remote → remote uses server-side copy. + - local → local is a usage error; use normal shell tools. +- `exc buckets rm [--recursive] [--yes] [--json]` — destructive shortcut for object delete. + +- `exc buckets keys list [--json]` +- `exc buckets keys create ` — prints the `secret_access_key` once. Capture it immediately; the CLI cannot fetch it again later. +- `exc buckets keys delete ` — destructive, immediate revocation. +- `exc buckets keys configure [--profile ] [--secret ] [--set-default]` + - Writes `~/.aws/credentials` and `~/.aws/config` for S3-compatible access to `https://buckets.excloud.dev`. + - Prefer the interactive secret prompt or `EXCLOUD_BUCKETS_SECRET`; `--secret` is convenient but can land in shell history. + - Default AWS profile name is typically `excloud`. Test with `aws s3 ls --profile excloud`. + +Normal users should prefer `objects upload`; it automatically uses multipart for large files. Use `exc buckets multipart` for manual part control or abandoned-upload cleanup: + +- `exc buckets multipart list [--prefix ] [--limit ] [--all] [--json]` +- `exc buckets multipart create [--content-type ]` — also accepts a single `bucket/key` or `s3://bucket/key` arg. +- `exc buckets multipart uploadpart ` — part numbers are 1-10000. +- `exc buckets multipart complete --parts ''` — `--parts` can be inline JSON or a path to a JSON file. +- `exc buckets multipart abort ` — destructive cleanup of an unfinished upload. + ## IAM, billing, quota - `exc org list` @@ -296,8 +364,10 @@ exc compute volume list # the root volume should disappear / move to Every command either prints a column table (or TSV) or prints JSON — no command should print raw Go-struct dumps anymore. Both shapes are machine-parseable; pick your tool accordingly. -- **Column tables / TSV** (awk / `cut` / `awk -F\t` friendly): `compute list`, `compute get`, `compute create`, `compute terminate` (TSV `vm_id\tstate`), `compute instancetype list` / `capacity`, `compute image list`, `compute subnet list`, `compute volume list`, `compute volume get`, `compute snapshot list`, `compute publicip list`, `compute key list`, `securitygroup list` / `rule list` / `binding list`, `org list`, `account list`, `apikey list`, `policy list`, `config list`, `compute seriallogs`. -- **JSON** (pipe through `jq`): `me`, `quota`, `billing get`, `compute health` (`{"raw":"OK"}`), `k8s health`, `compute subnet get`, `compute publicip get`, `compute key get`, `securitygroup get`, `compute metrics`, `compute connect`, `serviceaccount list`, `compute protect`, `compute unprotect`, `compute rename`, `k8s cluster kubeconfig get` (raw kubeconfig YAML, not JSON-wrapped), and the inline `kubeconfig` field inside the JSON response from `k8s cluster create` when `-o` is not set. +- **Column tables / TSV** (awk / `cut` / `awk -F\t` friendly): `compute list`, `compute get`, `compute create`, `compute terminate` (TSV `vm_id\tstate`), `compute instancetype list` / `capacity`, `compute image list`, `compute subnet list`, `compute volume list`, `compute volume get`, `compute snapshot list`, `compute publicip list`, `compute key list`, `securitygroup list` / `rule list` / `binding list`, `org list`, `account list`, `apikey list`, `policy list`, `config list`, `compute seriallogs`, `buckets list`, `buckets ls`, `buckets objects list`, `buckets keys list`, `buckets multipart list`. +- **JSON** (pipe through `jq`): `me`, `quota`, `billing get`, `compute health` (`{"raw":"OK"}`), `k8s health`, `compute subnet get`, `compute publicip get`, `compute key get`, `securitygroup get`, `compute metrics`, `compute connect`, `serviceaccount list`, `compute protect`, `compute unprotect`, `compute rename`, `k8s cluster kubeconfig get` (raw kubeconfig YAML, not JSON-wrapped), `buckets get`, `buckets usage`, `buckets keys create`, multipart create/uploadpart/complete/abort, and the inline `kubeconfig` field inside the JSON response from `k8s cluster create` when `-o` is not set. + +Many bucket list/delete commands have explicit `--json`; use it for scripts instead of scraping tables. `buckets objects presign` intentionally prints only the URL so it can be piped to tools like `pbcopy`. Before scripting heavy logic against a command, run it once and check the shape. The split between "table" and "JSON" is not always guessable — lists tend to be tables, getters tend to be JSON, but verify.