## Current Behavior This branch started as a fix for a flaky `e2e/nx/src/watch.test.ts`. Once we flipped `NX_DAEMON='true'` in the e2e harness, several long-latent daemon issues surfaced; they're fixed in the same PR. **Native watcher.** `nx watch` dropped and duplicated events on Linux/Windows: 1. `watchexec.config.pathset()` registered new directories asynchronously — files written before the real inotify registration were lost. 2. Every raw `notify` event fired a callback (no debounce) so one logical write produced multiple `nx watch` invocations. 3. The daemon called `notifyFileWatcherSockets` twice per event (eagerly for updates/deletes, again post-recompute for creates), doubling invocations. 4. `getProjectsAndGlobalChanges` looked changed files up in `fileMap.projectFileMap`, so brand-new files fell through to `globalFiles` and `--projects=foo` watchers never matched. 5. `nx:test-native` didn't compile on master (walker test used `nix::unistd::mkfifo` behind a feature that wasn't enabled). 6. macOS regressed after the notify migration: `FsEventWatcher` dropped immediately because the closure had no cfg-active references on darwin. **Daemon (surfaced once it ran end-to-end in CI):** 7. `getPluginsSeparated` reused `pendingPluginsPromise` via `??=` after the plugins-config hash changed, serving stale plugins forever after `nx add`. 8. The auto-recompute path updated the in-memory graph but never persisted it; subprocesses reading the disk cache saw stale data (flaky `eslint dependency-checks`). 9. `startInBackground` had an fd race: a concurrent `reset()` could null `this._out`/`this._err` while we awaited `open()`, crashing the spawn with `Cannot read properties of null (reading 'fd')`. 10. The daemon's env-reflection was additive only — `NX_*` vars set by one CLI invocation persisted forever, leaking (e.g., `NX_PREFER_NODE_STRIP_TYPES=true` from a single `nx report` poisoned every later plugin load). Plugin worker's `setWorkerEnv` had the same bug. 11. Several e2e cache-hit assertions matched `[local cache]`, but the daemon-on path emits `[existing outputs match the cache, left as is]` (output kept rather than restored). **Maven (separate but bundled):** 12. Tests asserted on `BUILD SUCCESS` in batch mode, but the resident batch runner's `BatchExecutionListener.sessionEnded` is a no-op — Maven's footer is replaced by `Nx Maven Summary`. Those assertions could never match. ## Solution ### Native watcher #### Architecture ``` Before: Watcher → watchexec (async config loop) → notify → inotify/FSEvents After: Watcher → notify → inotify/FSEvents (direct, synchronous) ``` The watcher is one Rust struct (`packages/nx/src/native/watch/watcher.rs`) that owns: - a `notify::RecommendedWatcher` (the OS-level subscription), - a single `mpsc::Sender` shared by the notify event handler and the napi force-flush method, - a long-running **processing thread** that reads from the corresponding `Receiver`. #### How events are streamed ``` notify (OS) ─► NotifyForwarder ─┐ ├─► mpsc ─► processing thread ─► napi tsfn ─► JS callback JS force_flush_pending() ───────┘ (or sync_channel reply) ``` 1. **Notify side.** `notify::recommended_watcher(NotifyForwarder { tx })` hands every fs event to `NotifyForwarder::handle_event`, which wraps it as `ProcMsg::Notify(event)` and pushes it onto the channel. No work is done on the notify thread beyond the send. 2. **Processing thread.** A dedicated thread runs an infinite loop with three branches: - `ProcMsg::Notify(event)` — filter through gitignore/nxignore (`watch_filterer.rs`), run the new-directory fast path on Linux/Windows if the event is a `Create(Folder)`, transform notify's kinds into our `EventType` (Create/Update/Delete), and merge each resulting `WatchEventInternal` into a per-path `HashMap` accumulator (priority `Delete > Create > Update` so a delete after a create still surfaces correctly). - `ProcMsg::ForceFlush(reply)` — drain whatever notify has buffered (`while let Ok(msg) = rx.try_recv()` so anything in flight at the moment of the request is included), take the accumulator, and send it back to JS via the `SyncSender` reply channel. Used by the daemon to absorb pending events before serving a cached project graph (closes the IDLE_WINDOW race where a 99 ms-old event would otherwise miss the next read). - `Err(RecvTimeoutError::Timeout)` — the wake-up deadline elapsed; emit the accumulator to JS via the napi `ThreadsafeFunction` (`callback_tsfn.call(Ok(events), NonBlocking)`), clear it, and reset. 3. **Trailing-edge debounce.** Each `Notify` ingest updates a single `flush_deadline = min(now + IDLE_WINDOW, burst_start + MAX_WAIT)`: - `IDLE_WINDOW = 100 ms` — flush when the channel goes quiet for this long. Resets on every arriving event, so any burst with gaps <100 ms coalesces into one flush. - `MAX_WAIT = 500 ms` — starvation cap from the first event of the burst. Under pathological continuous streams (events faster than IDLE_WINDOW can ever win), the flush still fires and a new burst starts fresh. - The loop's `rx.recv_timeout(wait)` either picks up the next message or returns `Timeout` exactly at `flush_deadline`. No separate timers, no cross-flush state, no `recent_paths` book-keeping — correctness follows directly from "emit when quiet, cap when busy." - When idle (`flush_deadline = None`) the loop polls on `SHUTDOWN_POLL` so `stop_flag` is observed promptly without `recv()` blocking forever. 4. **napi tsfn handoff.** The processing thread invokes `callback_tsfn.call(Ok(Vec), NonBlocking)` to schedule the JS callback on Node's main thread. `Vec` is converted to the napi-exposed `Vec` (`type, path, origin`) at the moment of flush. The notify thread itself never crosses into JS — only the processing thread does, and only at flush time. 5. **`force_flush_pending` (synchronous drain).** The daemon calls `watcher.force_flush_pending()` from JS before reading the cached project graph: - JS-side napi method creates a `sync_channel::>(1)` reply pair and sends `ProcMsg::ForceFlush(reply_tx)` on the same channel notify events flow through. - The processing thread sees the request in order with any buffered notify events, drains, takes the accumulator, and replies with the events. - JS blocks (with a 500 ms timeout) on `reply_rx.recv_timeout` and returns the events synchronously. - Any event delivered before the JS call — even one sitting unflushed in the IDLE_WINDOW — is visible to the graph read. Returns `[]` if the watcher hasn't started, the thread has exited, or no events are buffered. #### New-directory fast path (Linux/Windows) When notify reports a `Create(Folder)` on inotify/ReadDirectoryChangesW (which only watch the dirs they were given), the processing thread: 1. Calls `watcher.watch(new_dir, NonRecursive)` **synchronously** — the OS watch is active the moment this returns. 2. Re-walks the new directory and merges every existing entry into the accumulator with `EventType::Create`. 3. Recursively registers any subdirectories it found (so a `mkdir -p a/b/c/d` still hooks every level). Because (1) is synchronous, files written between `mkdir` and the `watch()` call are caught by the re-walk in (2). Notify events for those same files arriving later just merge into the same accumulator entry — no duplication. macOS doesn't need this path: `FsEventWatcher` is recursive. `notify_watcher` is held as `Option>>` on the struct so the processing-thread closure can clone the `Arc`; otherwise the macOS `move`-closure (which has no cfg-active references) wouldn't capture the watcher and `FsEventWatcher` would drop the moment `watch()` returned. ### Daemon - **Plugin reload on config change.** `getPluginsSeparated` clears `pendingPluginsPromise` and tears down workers when `nx.json#plugins`'s hash changes. - **Persist project graph after auto-recompute.** New `persistProjectGraphToDisk` helper called from `startAutoRecompute` and `getCachedSerializedProjectGraphPromise`. - **fd race fix.** Open log handles into locals first, assign to `this._out`/`this._err` after both opens resolve, pass the local `fd`s to `spawn`. - **Two-way `NX_*` env reflection.** New `applyDaemonEnvFromClient(newEnv)` helper writes new keys AND deletes any `NX_`-prefixed key the daemon has that the client doesn't (skipping daemon-side exclusions and required settings). Used by both `server.ts` `handleMessage` and the plugin worker's `setWorkerEnv`. Deletion is scoped to `NX_` so unrelated env (PATH, HOME, etc.) is untouched. - **`scheduleTimeoutId` → in-flight promise** for self-documenting recompute scheduling. - **Single `notifyFileWatcherSockets` registration** at daemon startup; one notification per batch. - **Map new files by project root.** `getProjectsAndGlobalChanges` uses `createProjectRootMappings` + `findProjectForPath`, cached by reference identity on `currentProjectGraph`. ### Test suite - `NX_DAEMON='true'` is the default in `e2e/utils/command-utils.ts` so CI exercises the daemon path. - Cache-hit assertions updated to `[existing outputs match the cache, left as is]` for the daemon-on no-op restore path. One assertion in `cache.test.ts:216` reverted to `[local cache]` because that test adds an extra file to `dist/`, invalidating the outputs hash and forcing a real restore. - Maven batch tests assert on `Successfully ran target X` instead of `BUILD SUCCESS`. `verbose: true` dropped where it only added Maven `-X` debug noise. - Watch e2e harness uses `tree-kill` and waits for `close` before reading output. Default wait shortened from 6s/8s → 1s/2s now that debounce is deterministic. - Walker test uses `std::os::unix::net::UnixListener::bind` instead of `nix::unistd::mkfifo` — same `is_hashable_file` rejection, no `nix` feature flag needed. ## Testing - `pnpm nx run e2e-nx:e2e-ci--src/watch.test.ts --skip-nx-cache` — 8/8 passed. - `pnpm nx run e2e-js:e2e-ci--src/js-strip-types.test.ts` — 3/3 (was failing on test 3 before the env-reflection fix). - `pnpm nx run e2e-maven:e2e-ci--src/maven-batch.test.ts` — 3/3 (was failing on test 1 + 3 before the assertion fix). - `pnpm nx run nx:test-native` — 341 tests pass. - macOS path validated end-to-end with the Arc fix. ## Related Issue(s) N/A — flaky-test investigation that grew into a daemon-stability hardening pass.