Allow curl requests to any HTTPS host; remove experimental domain allowlist

README.md

@@ -34,9 +34,9 @@ jq '{id, title, completed}' todo.json
 
 Or ask Pi: **Fetch https://jsonplaceholder.typicode.com/todos/1, save it as todo.json, and tell me its title and whether it is completed.**
 
-Allowed HTTPS hosts: `api.github.com`, `raw.githubusercontent.com`, `huggingface.co`, `jsonplaceholder.typicode.com`, and `httpbin.org`. GET, HEAD, and POST are enabled, with an 8-second deadline and 1 MiB response limit. Use `set -o pipefail` before pipelines to retain download failures.
+Any HTTPS URL is allowed, including GitHub Gists; there is no host allowlist. GET, HEAD, and POST are enabled, with an 8-second deadline and 1 MiB response limit. Use `set -o pipefail` before pipelines to retain download failures.
 
-Browser CORS applies. Redirects require the final URL directly: browsers hide redirect destinations, so this copy reports an error instead of silently returning empty output or following unchecked hosts. Cookies are omitted. Explicit request URLs, headers, and bodies are sent to the destination; inference and workspace storage remain local. Node.js, Python, npm, and native programs remain unavailable.
+Browser CORS applies. Redirects require the final URL directly: browsers hide redirect destinations, so this copy reports an error instead of silently returning empty output or following unchecked redirect schemes. Cookies are omitted. Explicit request URLs, headers, and bodies are sent to the destination; inference and workspace storage remain local. Node.js, Python, npm, and native programs remain unavailable.
 
 Source is under `app/`. Rebuild with `cd app && npm ci && npm test && npm run build`, then `python3 scripts/stage.py`. The HTTP implementation is in `app/src/browser-http.mjs`; it uses just-bash's existing curl commands and custom transport hook. Tests and results are in `validation/`.
 
@@ -44,4 +44,4 @@ The small 4-bit model can make mistakes. Shell transport checks and actual-model
 
 ## Deployed results
 
-The [evaluation report](validation/REPORT.md) separates 15 passing shell checks from seven real-model trials (five fully passed; two partial results). Basic API fetching works well. Duplicate POSTs and excessive retries on blocked requests remain model limitations.
+The initial, host-restricted [evaluation report](validation/REPORT.md) separates 15 passing shell checks from seven real-model trials (five fully passed; two partial results). Basic API fetching works well. Duplicate POSTs and excessive retries on blocked requests remain model limitations.

app/README.md

@@ -52,7 +52,7 @@ Message padding follows Pi 0.85.1 defaults: one column of output padding, one in
 
 The full Node CLI, host filesystem, OAuth providers, extensions, package installation, and RPC server are not included. xterm.js is the terminal emulator; it does not add Node.js or a native shell. Model inference and tool execution remain in the browser worker. Terminal control sequences from model/file output are stripped before rendering, and Markdown images do not trigger downloads.
 
-just-bash is a simulated shell, not a Linux VM. Shell scripts, pipes, redirects, `awk`, `jq`, `sed`, `grep`, `find`, and other supported text commands execute in JavaScript. Node.js, Python, npm, native executables, and SQLite are unavailable. This HTTP experiment enables curl for selected HTTPS APIs through a browser fetch adapter; see public/http-help.html for hosts, examples, and limits. The installed browser bundle also excludes working gzip/zlib commands. Shell cwd and variables reset between commands; filesystem changes persist. Commands have a 10-second deadline, 3,000-command cap, 16 MiB filesystem budget, and 256 KiB output budget. Tool output passed to the model is capped at 6,000 characters.
+just-bash is a simulated shell, not a Linux VM. Shell scripts, pipes, redirects, `awk`, `jq`, `sed`, `grep`, `find`, and other supported text commands execute in JavaScript. Node.js, Python, npm, native executables, and SQLite are unavailable. This HTTP experiment enables curl for any HTTPS URL through a browser fetch adapter; see public/http-help.html for examples and limits. The installed browser bundle also excludes working gzip/zlib commands. Shell cwd and variables reset between commands; filesystem changes persist. Commands have a 10-second deadline, 3,000-command cap, 16 MiB filesystem budget, and 256 KiB output budget. Tool output passed to the model is capped at 6,000 characters.
 
 Local model and filesystem tool calls can run with the browser network disabled after loading. HTTP commands require a connection and CORS permission from the destination. Reopening the app itself still requires its static HTML/JS/WASM assets; there is no offline service worker. Browser storage belongs to this site's origin and may be evicted by the browser. Export important work.
 

app/index.html

@@ -65,7 +65,7 @@
         </form>
         <div id="shell-hint"><span>Enter to run · Shift+Enter for newline</span><span>↑↓ history</span></div>
       </details>
-      <p class="workspace-note">HTTP experiment: curl can reach selected APIs. Browser CORS applies. <a href="./http-help.html" target="_blank" rel="noopener">Hosts &amp; examples ↗</a></p>
+      <p class="workspace-note">HTTP: curl can request any HTTPS URL. Browser CORS applies. <a href="./http-help.html" target="_blank" rel="noopener">HTTP help ↗</a></p>
     </aside>
   </main>
   <dialog id="model-dialog" aria-labelledby="model-title" aria-describedby="model-description">

app/public/http-help.html

@@ -4,7 +4,7 @@
 <style>html{color-scheme:dark;background:#18181e;color:#e4e4eb;font:17px/1.6 system-ui}body{max-width:760px;margin:40px auto;padding:0 20px}a{color:#a4e2d4}pre{background:#262630;padding:16px;overflow:auto;border-radius:12px}code{font-size:14px}</style>
 <h1>HTTP from the browser shell</h1>
 <p>This experimental copy enables real <code>curl</code> requests. The model and shell run on your device. Requests go directly from your browser to the destination; there is no proxy. Request URLs, headers, and bodies are sent to that service.</p>
-<p>Allowed HTTPS hosts: <code>api.github.com</code>, <code>raw.githubusercontent.com</code>, <code>huggingface.co</code>, <code>jsonplaceholder.typicode.com</code>, and <code>httpbin.org</code>. GET, HEAD, and POST are enabled. Cookies are omitted. Requests have an 8-second deadline and a 1 MiB response limit.</p>
+<p>Any HTTPS URL is allowed, including GitHub Gists; there is no host allowlist. The destination must still permit browser access with CORS. GET, HEAD, and POST are enabled. Cookies are omitted. Requests have an 8-second deadline and a 1 MiB response limit.</p>
 <h2>Try in Shell</h2>
 <pre>curl -fsS https://jsonplaceholder.typicode.com/todos/1 -o todo.json
 jq '{id, title, completed}' todo.json</pre>

app/src/agent.worker.mjs

@@ -14,7 +14,7 @@ When the user explicitly asks you to use a tool, call that tool even if you alre
 Working directory: /workspace. Use read, write, edit, and bash. Keep tool calls and answers concise.
 The bash tool supports cat, ls, find, grep, sed, awk, jq, sort, wc, printf, pipes, redirects, and shell scripts.
 Use jq for JSON and awk for CSV and calculations; python, python3, node, npm, and native binaries are unavailable.
-Use curl -fsS for HTTP requests to the hosts listed by the bash tool. To save a response, use curl -fsS URL -o file.json, then inspect it with jq or read. For pipelines use set -o pipefail so failed downloads are not mistaken for success. CORS and redirect errors are browser limitations: report them accurately; do not invent fetched content or repeatedly retry blocked requests.
+Use curl -fsS for requests to any HTTPS URL; there is no host allowlist. To save a response, use curl -fsS URL -o file.json, then inspect it with jq or read. For pipelines use set -o pipefail so failed downloads are not mistaken for success. CORS and redirect errors are browser limitations: report them accurately; do not invent fetched content or repeatedly retry blocked requests.
 You can create and edit text files, including HTML, CSS, JavaScript, and shell scripts. External URLs may be referenced without fetching them. Only claim an asset was fetched or tested after a successful tool result. HTTP requests send the requested URL and any explicit request data to that external server; model inference stays local.
 Interpret short follow-ups using the conversation and inspect the relevant files when needed.
 Inspect files before editing them. After changing files, verify the result with a tool. Finish with a short factual answer.`;

app/src/browser-http.mjs

@@ -1,23 +1,13 @@
-// just-bash's curl use this browser transport. There is no proxy/server.
-export const HTTP_ORIGINS = Object.freeze([
-  'https://api.github.com',
-  'https://raw.githubusercontent.com',
-  'https://huggingface.co',
-  'https://jsonplaceholder.typicode.com',
-  'https://httpbin.org',
-]);
+// just-bash's curl uses this browser transport. There is no proxy/server.
 export const HTTP_METHODS = Object.freeze(['GET', 'HEAD', 'POST']);
 export const HTTP_LIMIT = 1024 * 1024;
 export const HTTP_TIMEOUT = 8_000;
 
-export function createBrowserFetch({ transport = globalThis.fetch, origins = HTTP_ORIGINS,
+export function createBrowserFetch({ transport = globalThis.fetch,
   timeoutMs = HTTP_TIMEOUT, maxBytes = HTTP_LIMIT } = {}) {
-  const allowed = new Set(origins);
   return async (input, options = {}) => {
     const url = new URL(input);
-    if (url.protocol !== 'https:' || !allowed.has(url.origin)) {
-      throw Error(`HTTP destination is not enabled in this experiment. Allowed hosts: ${[...allowed].map(x => new URL(x).host).join(', ')}.`);
-    }
+    if (url.protocol !== 'https:') throw Error('Only HTTPS URLs are supported. Use https:// instead.');
     if (url.username || url.password) throw Error('Credentials in URLs are unsupported. Use an explicit Authorization header if needed.');
     const method = (options.method ?? 'GET').toUpperCase();
     if (!HTTP_METHODS.includes(method)) throw Error(`HTTP method ${method} is not enabled. Use GET, HEAD, or POST.`);
@@ -33,7 +23,7 @@ export function createBrowserFetch({ transport = globalThis.fetch, origins = HTT
         body: ['GET', 'HEAD'].includes(method) ? undefined : options.body,
         mode: 'cors', credentials: 'omit', redirect: 'manual', referrerPolicy: 'no-referrer', signal });
       // Browsers hide both the Location header and status on manual redirects.
-      // Following automatically would skip the host check at intermediate hops.
+      // Following automatically would skip the HTTPS check at intermediate hops.
       if (response.type === 'opaqueredirect' || (response.status >= 300 && response.status < 400 && response.status !== 304)) {
         throw Error('HTTP redirect cannot be followed safely in this browser shell. Use the final HTTPS URL directly (including exact capitalization).');
       }

app/src/workspace.mjs

@@ -1,6 +1,6 @@
 import { Bash } from 'just-bash/browser';
 import { Type } from '@earendil-works/pi-ai';
-import { createBrowserFetch, HTTP_ORIGINS } from './browser-http.mjs';
+import { createBrowserFetch } from './browser-http.mjs';
 
 // Frozen previous starter data, used only to recognize an unmodified demo.
 export const legacyProductsCsv = 'product,unit_cost,stock\nNotebook,2.50,24\nPencil,0.50,120\nFolder,1.20,18\nEraser,0.35,40\nStapler,2.75,9\nRuler,0.80,30\n';
@@ -52,7 +52,7 @@ export function createWorkspace(files = initialFiles, entries) {
     await bash.fs.writeFile(resolved, content);
   }
   const tools = [{
-    name: 'bash', label: 'Bash', description: `Run a bash command in /workspace. Supports cat, ls, grep, sed, awk, jq, sort, pipes, redirects, shell scripts, and curl. HTTP GET/HEAD/POST is enabled for ${HTTP_ORIGINS.map(x => new URL(x).host).join(', ')}. Use curl -fsS URL; use set -o pipefail before piping HTTP output to jq. Browser CORS applies; redirects require the final URL. HTTP limits: 8 seconds, 1 MiB. No Node.js, Python, npm, or native binaries. Files persist across commands; cd and variables reset each call.`,
+    name: 'bash', label: 'Bash', description: 'Run a bash command in /workspace. Supports cat, ls, grep, sed, awk, jq, sort, pipes, redirects, shell scripts, and curl. HTTP GET/HEAD/POST is enabled for any HTTPS URL; there is no host allowlist. Use curl -fsS URL; use set -o pipefail before piping HTTP output to jq. Browser CORS applies; redirects require the final URL. HTTP limits: 8 seconds, 1 MiB. No Node.js, Python, npm, or native binaries. Files persist across commands; cd and variables reset each call.',
     parameters: Type.Object({ command: Type.String({ description: 'The shell command to execute.' }) }),
     execute: async (_id, { command }, signal) => {
       const r = await bash.exec(command, { signal });

app/tests/browser-http.test.mjs

@@ -15,15 +15,19 @@ test('HTTP preserves bytes/status and omits implicit credentials', async () => {
   assert.equal(request.init.credentials, 'omit'); assert.equal(request.init.redirect, 'manual');
   assert.equal(request.init.body, 'hello'); assert.equal(request.init.mode, 'cors');
 });
-test('Origin/method/URL credential checks happen before any request', async () => {
+test('Any HTTPS host is accepted; protocol/method/URL credentials are checked before requests', async () => {
   let calls = 0;
   const fetch = createBrowserFetch({ transport: async () => { calls++; return new Response(''); } });
-  for (const url of ['http://httpbin.org/get', 'https://httpbin.org.evil.test/', 'https://api.github.com@evil.test/', 'http://127.0.0.1/', 'https://example.com/']) {
-    await assert.rejects(fetch(url), /not enabled/);
+  for (const url of ['http://httpbin.org/get', 'ftp://example.com/file', 'file:///etc/passwd', 'data:text/plain,hello', 'http://127.0.0.1/']) {
+    await assert.rejects(fetch(url), /Only HTTPS/);
   }
   await assert.rejects(fetch('https://user:pass@httpbin.org/get'), /Credentials/);
   await assert.rejects(fetch(endpoint, { method: 'DELETE' }), /method DELETE/);
   assert.equal(calls, 0);
+  for (const url of ['https://example.com/', 'https://gist.github.com/', 'https://gist.githubusercontent.com/', 'https://another-host.test:8443/data']) {
+    assert.equal((await fetch(url)).status, 200);
+  }
+  assert.equal(calls, 4);
 });
 test('Opaque redirects cannot appear as successful empty responses', async () => {
   const fetch = createBrowserFetch({ transport: async () => ({ type: 'opaqueredirect', status: 0 }) });

app/tests/http-live.mjs

@@ -16,6 +16,8 @@ context.on('request', r => { if (!r.url().startsWith(new URL(url).origin)) resul
 const rpc = (action, args) => page.evaluate(({ action, args }) => window.browserPi.rpc(action, args), { action, args });
 const save = () => fs.writeFile(root + '/validation/http-shell-live.json', JSON.stringify(result, null, 2) + '\n');
 const cases = [
+  ['new-host-json', 'curl -fsS https://dummyjson.com/products/1 -o product.json && jq -r .id product.json', r => r.exitCode === 0 && r.stdout.trim() === '1'],
+  ['public-gist', 'curl -fsS https://gist.githubusercontent.com/mbostock/4063269/raw/b111d979d7fcd0fabb554cccd37caf2691f31706/flare.csv -o gist.csv && head -n 1 gist.csv', r => r.exitCode === 0 && r.stdout.trim() === 'id,value'],
   ['json-get-save', 'curl -fsS https://jsonplaceholder.typicode.com/todos/1 -o todo.json && jq -r .title todo.json', r => r.exitCode === 0 && r.stdout.includes('delectus aut autem')],
   ['github-pipeline', "set -o pipefail; curl -fsS https://api.github.com/repos/huggingface/transformers.js | jq -r '.full_name, .license.spdx_id'", r => r.exitCode === 0 && r.stdout.includes('huggingface/transformers.js')],
   ['hf-metadata', 'set -o pipefail; curl -fsS https://huggingface.co/api/models/openbmb/MiniCPM5-2B | jq -r .id', r => r.exitCode === 0 && /openbmb\/MiniCPM5-2B/i.test(r.stdout)],
@@ -24,15 +26,15 @@ const cases = [
   ['post-json', `set -o pipefail; curl -fsS https://httpbin.org/post -H 'Content-Type: application/json' -d '{"demo":"browser-http","value":144}' | jq -c .json`, r => r.exitCode === 0 && r.stdout.includes('browser-http') && r.stdout.includes('144')],
   ['request-headers', "set -o pipefail; curl -fsS https://httpbin.org/headers -H 'X-Demo: browser-http' | jq -r '.headers[\"X-Demo\"]'", r => r.exitCode === 0 && r.stdout.trim() === 'browser-http'],
   ['http-404', 'curl -fsS https://jsonplaceholder.typicode.com/does-not-exist', r => r.exitCode !== 0 && /404/.test(r.stderr)],
-  ['blocked-host', 'curl -fsS https://example.com/', r => r.exitCode !== 0 && /not enabled/.test(r.stderr)],
+  ['http-rejected', 'curl -fsS http://example.com/', r => r.exitCode !== 0 && /Only HTTPS/.test(r.stderr)],
   ['cors-failure', 'curl -fsS https://huggingface.co/', r => r.exitCode !== 0 && /CORS/.test(r.stderr)],
   ['redirect-failure', 'curl -fsSL https://httpbin.org/redirect-to?url=https%3A%2F%2Fhttpbin.org%2Fget', r => r.exitCode !== 0 && /redirect/i.test(r.stderr)],
   ['timeout', 'curl -fsS --max-time 1 https://httpbin.org/delay/5', r => r.exitCode !== 0 && /timed out/i.test(r.stderr)],
-  ['failed-pipeline', 'set -o pipefail; curl -fsS https://example.com/ | jq .', r => r.exitCode !== 0],
+  ['failed-pipeline', 'set -o pipefail; curl -fsS http://example.com/ | jq .', r => r.exitCode !== 0],
 ];
 try {
   await page.goto(url); await page.waitForFunction(() => window.browserPi?.terminal); await page.evaluate(() => window.browserPi.ready);
-  assert.match(await page.locator('.workspace-note').textContent(), /HTTP experiment/);
+  assert.match(await page.locator('.workspace-note').textContent(), /any HTTPS URL/);
   for (const [name, command, check] of cases) {
     const start = Date.now(); const response = await rpc('shell', { command });
     const entry = { name, command, elapsedMs: Date.now() - start, stdout: response.stdout, stderr: response.stderr, exitCode: response.exitCode, passed: check(response) };

app/tests/protocol.test.mjs

@@ -113,8 +113,8 @@ test('real just-bash executes pipelines and scripts; filesystem survives reload
   await workspace.tools.find(t => t.name === 'edit').execute('2', { path: 'hello.sh', oldText: 'Hello', newText: '$& literal' });
   assert.match(await workspace.read('hello.sh'), /\$& literal/);
   const restored = createWorkspace(await workspace.snapshot()); assert.equal(await restored.read('total.txt'), '144\n');
-  const denied = await workspace.bash.exec('curl https://example.com');
-  assert.notEqual(denied.exitCode, 0); assert.match(denied.stderr, /destination is not enabled/);
+  const denied = await workspace.bash.exec('curl http://example.com');
+  assert.notEqual(denied.exitCode, 0); assert.match(denied.stderr, /Only HTTPS/);
   assert.equal((await workspace.bash.exec('node -v')).exitCode, 127);
 });
 test('workspace persistence preserves executable modes, empty directories, symlinks and bytes', async () => {

http-help.html

@@ -4,7 +4,7 @@
 <style>html{color-scheme:dark;background:#18181e;color:#e4e4eb;font:17px/1.6 system-ui}body{max-width:760px;margin:40px auto;padding:0 20px}a{color:#a4e2d4}pre{background:#262630;padding:16px;overflow:auto;border-radius:12px}code{font-size:14px}</style>
 <h1>HTTP from the browser shell</h1>
 <p>This experimental copy enables real <code>curl</code> requests. The model and shell run on your device. Requests go directly from your browser to the destination; there is no proxy. Request URLs, headers, and bodies are sent to that service.</p>
-<p>Allowed HTTPS hosts: <code>api.github.com</code>, <code>raw.githubusercontent.com</code>, <code>huggingface.co</code>, <code>jsonplaceholder.typicode.com</code>, and <code>httpbin.org</code>. GET, HEAD, and POST are enabled. Cookies are omitted. Requests have an 8-second deadline and a 1 MiB response limit.</p>
+<p>Any HTTPS URL is allowed, including GitHub Gists; there is no host allowlist. The destination must still permit browser access with CORS. GET, HEAD, and POST are enabled. Cookies are omitted. Requests have an 8-second deadline and a 1 MiB response limit.</p>
 <h2>Try in Shell</h2>
 <pre>curl -fsS https://jsonplaceholder.typicode.com/todos/1 -o todo.json
 jq '{id, title, completed}' todo.json</pre>

index.html

@@ -21,7 +21,7 @@
   <meta name="twitter:image" content="https://huggingface.co/spaces/Mike0021/MiniCPM5-2B-WebGPU-Pi-HTTP/resolve/main/social-thumbnail-v3.png">
   <meta name="twitter:image:alt" content="A coding agent running entirely in your browser, above the Pi terminal and its pixel cat mascot.">
   <title>Pi · Coding agent in your browser</title>
-  <script type="module" crossorigin src="./assets/index-A7Nq9zGD.js"></script>
+  <script type="module" crossorigin src="./assets/index-BzGjruQ9.js"></script>
   <link rel="stylesheet" crossorigin href="./assets/index-CLisF1Ml.css">
 </head>
 <body>
@@ -67,7 +67,7 @@
         </form>
         <div id="shell-hint"><span>Enter to run · Shift+Enter for newline</span><span>↑↓ history</span></div>
       </details>
-      <p class="workspace-note">HTTP experiment: curl can reach selected APIs. Browser CORS applies. <a href="./http-help.html" target="_blank" rel="noopener">Hosts &amp; examples ↗</a></p>
+      <p class="workspace-note">HTTP: curl can request any HTTPS URL. Browser CORS applies. <a href="./http-help.html" target="_blank" rel="noopener">HTTP help ↗</a></p>
     </aside>
   </main>
   <dialog id="model-dialog" aria-labelledby="model-title" aria-describedby="model-description">

validation/REPORT.md

@@ -1,4 +1,6 @@
-# Browser HTTP experiment — deployed evaluation
+# Browser HTTP experiment — initial host-restricted evaluation
+
+> Historical results: the five-host restriction tested below has since been removed. The current app accepts any HTTPS URL, subject to browser CORS and the existing request limits.
 
 Space: https://huggingface.co/spaces/Mike0021/MiniCPM5-2B-WebGPU-Pi-HTTP