Skip to content

feat: support allow-insecure HTTP dependencies - #700

Merged
Daniel Meppiel (danielmeppiel) merged 25 commits into
microsoft:mainfrom
arika0093:feat/allow-http-request
Apr 21, 2026
Merged

feat: support allow-insecure HTTP dependencies#700
Daniel Meppiel (danielmeppiel) merged 25 commits into
microsoft:mainfrom
arika0093:feat/allow-http-request

Conversation

@arika0093

@arika0093 arika (arika0093) commented Apr 14, 2026

Copy link
Copy Markdown
Contributor

Description

This PR adds explicit support for HTTP (insecure) APM dependencies behind an opt-in flow, and makes lockfile replays preserve that HTTP source information.

When adding a package, you can explicitly allow HTTP dependencies by providing the --allow-insecure flag.

$ apm install --allow-insecure http://my-server.example.com/owner/repo

# if not provided, it will error like this:
# [*] Validating 1 package...
# [x] http://my-server.example.com/owner/repo -- 'http://my-server.example.com/owner/repo' uses HTTP (insecure). Pass '--allow-insecure' or run 'apm config set allow-insecure true' to allow HTTP dependencies.
# All packages failed validation. Nothing to install.

When restoring packages, you also need to provide the --allow-insecure flag unless the global config is enabled. If there are HTTP URLs in the dependency chain, it will error without --allow-insecure or apm config set allow-insecure true.

$ apm install --allow-insecure

# if not provided, it will error like this:
# [>] Installing dependencies from apm.yml...
# [x] Dependency 'my-server.example.com/owner/repo' uses HTTP (insecure). Pass '--allow-insecure' to apm install, or run 'apm config set allow-insecure true' to allow HTTP dependencies globally.

The apm.yml file will record it in the following format:

dependencies:
  apm:
    - git: http://my-server.example.com/owner/repo
      allow_insecure: true

HTTP dependencies require two explicit signals:

  • allow_insecure: true on the dependency entry in apm.yml
  • either apm install --allow-insecure or apm config set allow-insecure true

The global configuration only removes the need to pass --allow-insecure on the CLI. It does not remove the allow_insecure: true requirement in apm.yml.

apm config set allow-insecure true

Fixes #636

TODO

  • Support for --allow-insecure flag when adding dependencies
  • Support for --allow-insecure flag when installing dependencies
  • Support for allow_insecure: true field in apm.yml and lockfile
  • Support for allow-insecure global config

Type of change

  • Bug fix
  • New feature
  • Documentation
  • Maintenance / refactor

Testing

  • Tested locally
  • All existing tests pass
  • Added tests for new functionality (if applicable)

@arika0093
arika (arika0093) marked this pull request as ready for review April 14, 2026 08:43
Copilot AI review requested due to automatic review settings April 14, 2026 08:43
@arika0093

Copy link
Copy Markdown
Contributor Author

@microsoft-github-policy-service agree

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

This PR introduces an explicit opt-in flow for allowing HTTP (insecure) APM dependencies, including persisting that decision in apm.yml and preserving HTTP metadata during lockfile replays.

Changes:

  • Add --allow-insecure flag + global allow-insecure config to gate installation of http:// dependencies.
  • Persist and replay HTTP dependency metadata (is_insecure, allow_insecure) via manifest and lockfile.
  • Update docs and add unit tests covering HTTP parsing, lockfile round-trip, CLI behaviors, and cloning behavior.

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
tests/unit/test_install_update.py Adds lockfile replay/round-trip tests for insecure HTTP metadata.
tests/unit/test_install_command.py Adds CLI tests for --allow-insecure and install-time security checks.
tests/unit/test_config_command.py Adds config command + config function tests for allow-insecure.
tests/unit/test_canonicalization.py Adds parsing/serialization tests for HTTP dependency handling.
tests/unit/test_auth_scoping.py Ensures HTTP deps don’t fall back to SSH during clone attempts.
src/apm_cli/models/dependency/reference.py Introduces is_insecure/allow_insecure, HTTP canonicalization, manifest serialization, and scheme-aware URLs.
src/apm_cli/drift.py Preserves insecure scheme info during lockfile replay.
src/apm_cli/deps/lockfile.py Persists is_insecure/allow_insecure in lockfile read/write and dependency refs.
src/apm_cli/deps/github_downloader.py Adds HTTP-only clone path and URL building for insecure deps.
src/apm_cli/config.py Adds get_allow_insecure / set_allow_insecure.
src/apm_cli/commands/install.py Adds --allow-insecure, manifest writing behavior, and install-time HTTP enforcement.
src/apm_cli/commands/config.py Generalizes config get/set to support allow-insecure.
src/apm_cli/commands/_helpers.py Propagates HTTP/artifactory/local fields when computing install paths.
src/apm_cli/bundle/plugin_exporter.py Propagates HTTP/artifactory/local fields for exported install paths.
packages/apm-guide/.apm/skills/apm-usage/commands.md Documents --allow-insecure in the command reference table.
docs/src/content/docs/reference/cli-commands.md Documents --allow-insecure and new config key.
docs/src/content/docs/guides/dependencies.md Documents HTTP dependency manifest format + caution guidance.

Comment thread src/apm_cli/models/dependency/reference.py Outdated
Comment thread src/apm_cli/models/dependency/reference.py Outdated
Comment thread src/apm_cli/models/dependency/reference.py Outdated
Comment thread src/apm_cli/commands/config.py
Comment thread src/apm_cli/deps/github_downloader.py Outdated

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Solid feature -- good security model with dual opt-in (dep-level + CLI/config), thorough test coverage (30+ tests), clean lockfile round-trip, and well-written docs. Nice work!

One blocking issue: to_apm_yml_entry() hardcodes allow_insecure = True, which could silently escalate security posture. See inline comment.

Also: this PR silently fixes a bug where artifactory_prefix was missing from DependencyReference construction in _helpers.py, plugin_exporter.py, and lockfile.py -- directly related to #614. Consider referencing that issue in the PR description.

entry["ref"] = self.reference
if self.alias:
entry["alias"] = self.alias
entry["allow_insecure"] = True

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: This hardcodes allow_insecure = True for all HTTP deps, ignoring self.allow_insecure. If this method is reused in a code path that hasn't applied the opt-in check, it silently grants insecure access.

Suggested change
entry["allow_insecure"] = True
entry["allow_insecure"] = self.allow_insecure

@arika0093 arika (arika0093) Apr 14, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

addressed in commit cd8bed0.

ado_project=ado_project,
ado_repo=ado_repo,
artifactory_prefix=artifactory_prefix,
is_insecure=dependency_str.startswith("http://"),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: startswith("http://") is case-sensitive, but URL schemes are case-insensitive per RFC 3986. HTTP://host/repo would bypass insecure detection.

Suggested change
is_insecure=dependency_str.startswith("http://"),
is_insecure=urllib.parse.urlparse(dependency_str).scheme.lower() == "http",

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

addressed in commit 3c0122c.

Comment thread src/apm_cli/deps/github_downloader.py Outdated
return build_ssh_url(host, repo_ref)
elif is_github and github_token:
# Only send GitHub tokens to GitHub hosts
# # Only send GitHub tokens to GitHub hosts

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Double # — typo from the refactor commit.

Suggested change
# # Only send GitHub tokens to GitHub hosts
# Only send GitHub tokens to GitHub hosts

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

addressed in fe7179f.

sub_path = entry.get("path")
ref_override = entry.get("ref")
alias_override = entry.get("alias")
allow_insecure = bool(entry.get("allow_insecure", False))

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: bool(entry.get("allow_insecure", False)) treats any non-empty string (e.g., "false") as True. Since this is a security-relevant field, consider validating the type explicitly:

Suggested change
allow_insecure = bool(entry.get("allow_insecure", False))
allow_insecure = entry.get("allow_insecure", False)
if not isinstance(allow_insecure, bool):
raise ValueError("'allow_insecure' field must be a boolean")

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

addressed in commit 47b87ab.

@arika0093

Copy link
Copy Markdown
Contributor Author

Sergio Sisternes (@sergio-sisternes-epam)
Thanks for the review. I addressed the points you raised, so could you please take another look?

On #614: this PR now explicitly passes artifactory_prefix=dep.registry_prefix when reconstructing DependencyReference, which fixes a related missing-prefix propagation path. But I do not think it resolves the core #614 issue around _parse_artifactory_base_url() ignoring PROXY_REGISTRY_URL; that still appears to be unresolved.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All findings addressed — nice work, arika (@arika0093)! Each fix is clean and well-tested.

Thanks for the clarification on #614 — you're right that the artifactory_prefix propagation fix here is a partial improvement but the core _parse_artifactory_base_url() issue remains separate.

LGTM!

@danielmeppiel Daniel Meppiel (danielmeppiel) added CI/CD Deprecated: use area/ci-cd. Kept for issue history; will be removed in milestone 0.10.0. and removed CI/CD Deprecated: use area/ci-cd. Kept for issue history; will be removed in milestone 0.10.0. labels Apr 19, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First — thanks for tackling this, arika (@arika0093). The use case is real (internal mirrors without HTTPS exist in the wild), the dual opt-in design (per-dep allow_insecure: true + CLI flag) is the right shape, and the test coverage is thorough. Sergio's APPROVE isn't wrong on the implementation quality.

I'm requesting changes from a security/UX lens because APM is a supply-chain tool, and a few aspects of the current design create footguns that I don't think we can ship to the broader community. None of these are deal-breakers — they're scope tightening. If we land them, this becomes a feature I'd be proud to ship.

Blockers

1. Drop apm config set allow-insecure true (the persistent global config)

This is the biggest single risk in the PR and, IMO, the only true blocker.

Per-invocation --allow-insecure is the right friction ceiling. A persistent global setting:

  • Means a developer enables it once for one internal project and then forever silently allows HTTP for every subsequent apm install on that machine, including in unrelated repos.
  • Turns shared CI runners into a network-MITM surface for any repo that lands allow_insecure: true in apm.yml — the runner doesn't need to opt in per-job, the maintainer who set the config months ago already did.
  • Is invisible at install time. There's no per-install reminder. Users forget they set it.
  • Converts a per-action security decision into ambient permission, which is the textbook supply-chain footgun.

The per-invocation flag is the correct model. It's friction, but the friction is the security feature: every HTTP install requires an active, conscious choice. Please drop the apm config set allow-insecure path entirely (and the corresponding get_allow_insecure / set_allow_insecure config functions).

2. Be loud at install time about which URLs are insecure

Today, the user only sees output about HTTP when --allow-insecure is missing (the error path). On the success path there's no equivalent loud diagnostic listing exactly which URLs are about to be fetched over an unauthenticated channel.

Compare to pip's behavior on insecure indexes: a clear warning per fetch. APM should do the same.

Concretely:

  • Print one [!] line per HTTP dependency at install-prepare time, with the full URL: [!] Fetching insecurely (no transport auth): http://internal.example.com/team/foo
  • Make this print whether or not --allow-insecure was passed (i.e. the warning is informational; the flag is the gate).
  • This is what makes Alice notice when Bob's apm.yml grew an unexpected HTTP entry between git pulls.

3. Transitive HTTP dependencies need a separate gate

This is the most subtle one and the one I'd most appreciate your thinking on. Scenario:

  • Bob adds a top-level dep http://internal.bob.example/foo/bar with allow_insecure: true to the project's apm.yml. Alice can see this in the diff and consents by passing --allow-insecure.
  • But Bob's package itself has an apm.yml that pulls a transitive dep http://attacker-controlled.example/lib/x over HTTP, also with allow_insecure: true.
  • When Alice runs apm install --allow-insecure, that transitive dep gets fetched too. Alice never saw it in the project's own apm.yml. Her consent at the root extends silently to a graph she didn't review.

The lockfile makes this auditable after it's pinned, but the first install (and any update) doesn't surface this distinction.

Suggested approach (open to alternatives):

  • Track per-dep whether the HTTP+allow_insecure choice originated in the root apm.yml or in a transitive dep.
  • For transitive HTTP, require an additional explicit acknowledgment: either a separate --allow-insecure-transitive flag, or an interactive prompt listing the transitive HTTP URLs and their introducing parent, with --yes to skip in CI.
  • At minimum, the per-URL warning from #2 should annotate (transitive, introduced by <parent-dep>) so users can see it.

Without this, the dual-opt-in only protects the immediate dep — the transitive surface is a single root flag away from being silently expanded.

Non-blocking but please consider

  • Threat-model docs: the new HTTP section in dependencies.md documents the mechanism but not the why. A short paragraph explaining that HTTP has no transport auth (so an MITM can substitute the package contents — the SHA in the lockfile is itself MITM-able when fetched over HTTP) would help users make informed choices instead of copy-pasting the example.
  • Audit affordance: there's no way to ask "list every insecure dep in my dependency graph" before running install. With the lockfile already preserving is_insecure, an apm deps list --insecure (or apm install --dry-run listing them) would be cheap and high-value. Could be a follow-up.
  • CHANGELOG entry: this is a security-sensitive feature. The CHANGELOG entry should be explicit that this introduces an opt-in HTTP path with the threat-model implications, not just "feat: support http deps."
  • Hostname allowlist (future, not now): a future iteration could let apm.yml declare an insecure_hosts: [...] allowlist so that HTTP is only permitted from explicitly-named hosts. Not for this PR.

What I'd merge

If you land #1 (drop global config) and #2 (loud per-URL output), I'd merge. #3 (transitive gating) is the one I feel strongest about from a supply-chain perspective; if you'd prefer to defer it, I'd accept that as a follow-up issue only if the per-URL warning from #2 explicitly annotates transitive HTTP deps so the information is at least visible.

If you'd rather not drop the global config, I'd respectfully decline this PR and recommend users set up HTTPS on their internal mirrors. APM is a supply-chain tool and persistent ambient "allow insecure" is incompatible with that role.

Thanks again for the thoughtful design and the thorough tests — this is salvageable and worth landing well.

@arika0093

arika (arika0093) commented Apr 20, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the thoughtful review. I think your concerns are reasonable, especially given that APM is a supply-chain tool.

I agree on the first two blockers:

  • I will drop the persistent global allow-insecure config.
  • I will make insecure fetches loud on the success path as well, with one warning per HTTP dependency showing the full URL.

On transitive HTTP dependencies: I agree this is the trickiest part. A root-level opt-in should not silently bless an unreviewed transitive graph.

After thinking more about it, I do not think a simple --allow-insecure-transitive flag is the right long-term shape, because it broadens trust based on graph position rather than which host is actually being trusted.

For this PR, my plan is:

  • a root HTTP dependency allowed via --allow-insecure will also allow transitive HTTP dependencies from the same host
  • transitive HTTP dependencies from a different host will still be blocked by default
  • for non-interactive environments, I will add an explicit host-level opt-in for additional hosts (for example, a repeatable --allow-insecure-host <host> flag)
  • install output will clearly warn for every HTTP dependency, including whether it is transitive and which parent introduced it

I think this is a better narrow fix for the security concern raised here, while also fitting the common internal-repository use case more naturally than a boolean transitive flag.

I will also update the docs and changelog accordingly. The audit affordance (apm deps list --insecure) also makes sense to me.

Thanks again for the careful review.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APM Review Panel -- PR #700

[x] Verdict: REQUEST CHANGES -- one CRITICAL credential-leak finding plus a converging architectural/security issue around HTTP identity. The dual-opt-in design is sound and Sergio's earlier blocker is correctly fixed; the items below are what stand between this and merge.

arika (@arika0093) -- this is a substantial, security-sensitive contribution and the design instincts are right. Five reviewers (python-architect, cli-logging-expert, devx-ux-expert, supply-chain-security-expert; growth-hacker side-channel) converged on a tight set of must-fixes. Thank you for the depth here -- 1338 lines + 30+ tests on a class of feature most projects ship sloppily.

What's excellent

  • Dual opt-in (manifest allow_insecure: true + CLI --allow-insecure) is the right mental model. Matches npm overrides + --legacy-peer-deps. Neither alone is sufficient -- correct.
  • Persistent global config dropped per earlier review feedback. Right call; every install run should be a conscious act.
  • --allow-insecure-host for transitive deps with same-host auto-propagation for direct deps is well-designed. Good FQDN validation prevents flag injection.
  • parse_from_dict() validates allow_insecure is boolean -- prevents YAML type confusion.
  • Doc surface coverage (cli-commands.md, dependencies.md, apm-guide skill resources, CHANGELOG) is exactly the discipline this codebase demands.

Required changes (must land in this PR)

  1. [x] CRITICAL -- HTTP clone leaks credentials in plaintext. github_downloader.py _env_for(use_token=False) strips GIT_ASKPASS and GIT_CONFIG_NOSYSTEM for HTTP attempts, re-enabling system git credential helpers (macOS Keychain, Windows Credential Manager, gh auth). When git clones http://internal.example.com/owner/repo, credential helpers may resolve stored tokens for that host and embed them as plaintext Authorization headers. The existing test test_insecure_http_dep_is_strict_by_default asserts "GIT_ASKPASS" not in env_used -- that test is verifying the vulnerability.

    Fix: HTTP attempts must actively block credential resolution:

    if attempt.scheme == "http":
        env = dict(self.git_env)
        env['GIT_ASKPASS'] = 'echo'
        env['GIT_TERMINAL_PROMPT'] = '0'
        env['GIT_CONFIG_NOSYSTEM'] = '1'
        # Suppress credential helpers explicitly
        env['GIT_CONFIG_COUNT'] = '1'
        env['GIT_CONFIG_KEY_0'] = 'credential.helper'
        env['GIT_CONFIG_VALUE_0'] = ''
        return env

    Update the test to assert credential-helper suppression for HTTP rather than its absence.

  2. [x] BLOCKER -- HTTP/HTTPS identity collision enables transport downgrade. get_unique_key() and get_identity() are scheme-blind, and drift.py documents "Source/host/scheme changes -- not detected." That means https://gitlab.com/team/rules and http://gitlab.com/team/rules collide as the same dep. An attacker submitting a PR that flips the scheme + adds allow_insecure: true produces zero lockfile drift signal. Combined with finding #1, this is a clean transport-downgrade exploit chain.

    Fix: Either (a) get_identity() includes scheme when is_insecure=True, or (b) drift.py treats a is_insecure flip as drift requiring --update. Option (b) is the smaller change and preserves identity stability for non-HTTP deps.

  3. [x] BLOCKER -- Inverted import: install/phases/resolve.py imports from commands/install.py. resolve.py:297-300:

    from apm_cli.commands.install import (
        _check_insecure_dependencies, _collect_insecure_dependency_infos,
        _guard_transitive_insecure_dependencies, _warn_insecure_dependencies,
    )

    Domain code (install/phases/) must never import from CLI command code. Extract the ~200 lines of insecure-policy logic + _InsecureDependencyInfo dataclass into a new src/apm_cli/install/insecure_policy.py. Both commands/install.py and install/phases/resolve.py then import from the new module. Pure move, zero logic change.

  4. [x] BLOCKER -- drift.py propagates is_insecure but not allow_insecure during lockfile replay. Lines +250-252: build_download_ref() copies is_insecure=True but drops allow_insecure. The replayed DependencyReference has is_insecure=True, allow_insecure=False, which _check_insecure_dependencies() rejects. Lockfile replays of HTTP deps are broken (or worse, bypass the check via a different code path).

    if getattr(locked_dep, "is_insecure", False) is True:
        overrides["is_insecure"] = True
        overrides["allow_insecure"] = getattr(locked_dep, "allow_insecure", False)
  5. [x] BLOCKER -- to_canonical() no longer strips http://, breaking the canonical identity contract. Architect + DevX both flagged this. The docstring update silently removes "no https://" and HTTP deps now produce http://host/owner/repo while HTTPS still strips. Every consumer that does dep.to_canonical().split("/") or assumes no :// will silently diverge for HTTP deps. Fix: keep to_canonical() scheme-free. The transport-aware string already exists as to_apm_yml_entry() -- use that for serialization paths and any surface that needs the scheme.

  6. [!] HIGH -- Add LockedDependency.to_dependency_ref() factory. Three sites reconstruct DependencyReference from LockedDependency: _helpers.py:138, plugin_exporter.py:399, lockfile.py:365. This PR already had to fix the missing artifactory_prefix (#614 root cause) AND add is_insecure/allow_insecure at all three. Per the "abstract when 3+ call sites" rule, add a factory method that maps every field once. This is the structural fix Sergio asked about -- the artifactory_prefix patch is a band-aid without it.

  7. [!] HIGH -- Unify the three competing error messages into one canonical recipe. Today a user sees three different errors depending on which gate failed:

    • validation-time ("Pass --allow-insecure"),
    • install-time/manifest-missing ("set allow_insecure: true"),
    • install-time/flag-missing ("Pass --allow-insecure").

    None give the complete recipe. Collapse to one message that always names both requirements:

    [x] http://my-server.example.com/owner/repo -- HTTP dependency (no transport encryption)
        To install:
          1. Set allow_insecure: true on the dep in apm.yml
          2. Pass --allow-insecure to apm install
    

    And use the full URL (with scheme) in every error -- the user needs to see why it's insecure.

  8. [!] HIGH -- Update docs/src/content/docs/enterprise/security.md. This PR introduces an entirely new transport surface and threat model; per the repo rule ("If a code change weakens or contradicts any guarantee in security.md, the doc must be updated in the same PR"), security.md needs an "HTTP (insecure) dependencies" section covering: what allow_insecure does/doesn't protect against, MITM impact on content-hash provenance, credential-helper isolation requirement, the transitive-host boundary model.

  9. [!] HIGH -- Logging routing: dead else: _rich_*() fallback branches. _warn_insecure_dependencies, _guard_transitive_insecure_dependencies, _check_insecure_dependencies repeat if logger: logger.X() else: _rich_X(...) 7 times. Callers always have a logger; the else branches are dead code and bypass DiagnosticCollector if ever triggered. Make logger a required positional arg; drop every fallback branch.

Polish (same PR if cheap; follow-up otherwise)

  1. [i] Library code calls sys.exit(1). _check_insecure_dependencies and _guard_transitive_insecure_dependencies should raise an exception; CLI layer catches and exits. Improves testability and reuse.

  2. [i] _format_insecure_dependency_warning uses jargon ("no transport auth"). Reword:

    [!] Insecure HTTP fetch (unencrypted): http://...
    
  3. [i] Transitive-block error is a policy lecture. Drop the middle sentence; lead with the command. Move the "why" to docs.

  4. [i] apm deps list --insecure column label "HTTP" misleads (values are direct / via <parent>). Rename to Origin.

  5. [i] Unrelated cleanups (config.py refactor, _setup_git_environment SSH cleanup) belong in separate commits for bisectability.

Worth a follow-up issue (don't block this PR)

  • APM_ALLOW_INSECURE=1 env var counterpart for CI ergonomics (mirror APM_ALLOW_PROTOCOL_FALLBACK=1 pattern). Not adding this is fine for v1; tracking issue acceptable.
  • Drift-detection enhancement: warn on first-time HTTP install without lockfile SHA pin (defense-in-depth against the hash-circular-trust limitation).

Merge call

This is solid first-PR work on a security-sensitive feature -- the dual-opt-in design and host scoping are correct. The blockers are concentrated in three areas: credential leakage (#1), identity model (#2, #5), and architectural cleanup (#3, #4, #6). Items #1-#5 are the merge-blockers; #6-#9 should land in the same PR; the rest is polish or follow-up. Please address #1-#9 in this PR and re-request review.

Growth angle

  • Real adoption blocker unlocked: corporate Gitea/Gogs/Artifactory-on-HTTP and air-gapped enterprise users -- the segment that quietly worked around APM's "install from anywhere" claim with manual git clone. Worth a release-notes call-out.
  • First-time NONE-association contributor shipping 1338 lines + 30+ tests on a security feature is the contributor signal worth amplifying once this lands. Suggested release-notes line: "APM now supports private HTTP git hosts with explicit dual-gate opt-in (manifest + CLI). Insecure by default? Never. Insecure when you need it, with a clear audit trail." Repostable hook: "APM doesn't block your infra -- it makes insecure choices visible."
  • Once shipped, security.md becomes the authoritative reference for the threat model -- making the doc update in #8 above doubly load-bearing.

Reviewed via the apm-review-panel skill: python-architect, cli-logging-expert, devx-ux-expert, supply-chain-security-expert; arbitrated by apm-ceo with oss-growth-hacker side-channel.

@arika0093
arika (arika0093) force-pushed the feat/allow-http-request branch from 1796653 to 803a126 Compare April 21, 2026 06:32
arika (arika0093) and others added 4 commits April 21, 2026 07:51
…split)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… 14 stack split)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ck split)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…tack split)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
arika (arika0093) and others added 4 commits April 21, 2026 07:51
…m 10)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@arika0093
arika (arika0093) force-pushed the feat/allow-http-request branch from b2a0a78 to b8a5c5f Compare April 21, 2026 08:00
@arika0093

Copy link
Copy Markdown
Contributor Author

Thanks for the detailed review. I addressed all requested changes from this review, rewrote the stack so the commits are easier to map back to the review items, and kept the split granularity intact.

Review item mapping:

  1. HTTP git attempts now suppress credential helpers: b5ce27e
  2. HTTP/HTTPS transport flips now show up as drift: c67cfde
  3. Insecure policy logic was extracted out of CLI command code into install/insecure_policy.py: 9032b9e
  4. Lockfile replay now preserves allow_insecure for HTTP dependencies: 1fca10b
  5. to_canonical() is scheme-free again: caeb13d
  6. LockedDependency.to_dependency_ref() now centralizes the mapping: 85be6d1
  7. HTTP remediation errors were unified into one canonical recipe: f08a716
  8. docs/src/content/docs/enterprise/security.md was updated for the HTTP threat model: 17e40a3
  9. The insecure policy helpers now require a logger and no longer keep dead fallback branches: 32cb21f
  10. Library-level insecure policy helpers now raise exceptions instead of calling sys.exit(1): b067ab7
  11. The insecure HTTP warning wording was tightened: 2dd90af
  12. The transitive HTTP remediation error was shortened and now leads with the command: 783450f
  13. apm deps list --insecure now uses Origin instead of HTTP: b8a5c5f

For review item 14:

  • The unrelated commands/config.py refactor was split into its own commit for bisectability: 333e8f0
  • The _setup_git_environment() / GIT_SSH_COMMAND SSH cleanup mentioned in that item was not introduced by this PR stack. It predates this branch, so there was no PR-local change to split out for that part.

At this point, all review items from this review have been addressed in the branch.

The combined surface of PR microsoft#700 (allow-insecure CLI flags + policy) and
main's MCP install block (microsoft#810, microsoft#814) pushed commands/install.py over
the 1525 LOC invariant enforced by tests/unit/install/
test_architecture_invariants. Per the test's own guidance, resolved by
extracting -- not trimming.

Conflicts:
- CHANGELOG.md: unioned the PR microsoft#700 Added entry with main's MCP entries.
- packages/apm-guide/.apm/skills/apm-usage/commands.md: merged both flag
  lists into a single --allow-insecure + --mcp row.
- src/apm_cli/commands/install.py: unioned the install() signature and
  docstring examples; kept both the InsecureDependencyPolicyError and
  click.UsageError except branches.
- tests/unit/test_install_command.py: kept TestAllowInsecureFlag and
  TestInstallMcpFlag as sibling classes (each with its own setup/
  teardown).

Architecture (commands/install.py LOC): 1572 -> 1496.
- Extracted F5 SSRF + F7 shell-metachar helpers to new dedicated module
  src/apm_cli/install/mcp_warnings.py (same pattern used in PR microsoft#809).
- commands/install.py re-binds the extracted symbols at module scope
  (warn_ssrf_url -> _warn_ssrf_url, warn_shell_metachars ->
  _warn_shell_metachars, _is_internal_or_metadata_host,
  _SHELL_METACHAR_TOKENS, _METADATA_HOSTS) so existing test patches
  against apm_cli.commands.install._warn_* keep working unchanged.

Tests: 4755/4755 unit + console pass (excludes the known pre-existing
test_user_scope_skips_workspace_runtimes failure on main, unrelated to
this PR or this merge).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
PR microsoft#809 (shell-string MCP command validation) merged to main after the
previous merge commit on this branch. Re-merged to pick up the new tip.

Conflicts:
- src/apm_cli/install/mcp_warnings.py (add/add): both sides added this
  file. Took main's version -- it is the authoritative microsoft#809 copy and
  includes the FU4 extension (command= parameter on warn_shell_metachars)
  that my earlier manual extraction on this branch did not have.
- src/apm_cli/commands/install.py: kept main's re-bind import ordering
  since the two versions were trivially equivalent.

Tests: 4767/4767 unit + console pass.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@danielmeppiel

Copy link
Copy Markdown
Collaborator

APM Review Panel — Round 2 Verdict (PR #700)

Re-review following arika (@arika0093)'s commit-by-item response mapping. Panel: Python Architect, CLI Logging, DevX UX, Supply-Chain Security, OSS Growth. Five specialists, full convergence, no disagreements to arbitrate.

Verdict: APPROVE with minor nits. Ready to merge after one small fix.

All 14 items from the previous round are substantively addressed. Evidence verified commit-by-commit.


Items verified resolved

# Blocker Fix commit Verdict
1 Credential-helper leak on HTTP clone b5ce27eb Defence-in-depth: GIT_ASKPASS=echo, GIT_TERMINAL_PROMPT=0, GIT_CONFIG_NOSYSTEM=1, GIT_CONFIG_COUNT/KEY/VALUE triple. All 4 independent channels closed. Verified at github_downloader.py:505-534 + tests test_auth_scoping.py:277-326, 854-859.
2 HTTP/HTTPS identity drift c67cfded XOR is_insecure comparison at drift.py:100-105 catches both upgrade and downgrade. Tests cover both directions + no-drift case.
3 Inverted imports commands→install 9032b9ef install/insecure_policy.py (209 LOC) with correct import direction. Zero residual duplicates in commands/install.py. Layering clean.
4 allow_insecure dropped on lockfile replay 1fca10b9 drift.py:259-263 replays both is_insecure and allow_insecure from lockfile. Legacy lockfiles fail-closed (missing allow_insecure → False → re-prompt).
5 to_canonical() scheme-leak caeb13d9 Scheme-free invariant restored. All consumers audited: display/identity paths scheme-free; only to_apm_yml_entry() keeps scheme where needed.
6 Missing to_dependency_ref() factory 85be6d14 Single source of truth at lockfile.py:204-218. All 3 call sites migrated. Zero hand-rolled DependencyReference( reconstructions outside the factory.
7 Unified error recipe Canonical _format_insecure_dependency_requirements() used across all three paths. See nit N1 below.
8 HTTP threat-model in security.md 17e40a3b New section (L64-82) covers allow_insecure semantics, MITM exposure, content-hash provenance limitation, credential-helper isolation, transitive host model. Sales-enablement quality.
9 Dead _rich_* fallback branches insecure_policy.py has zero _rich_* imports/calls; all three policy functions call logger.warning/error bare (no if logger: guard, no else: fallback).
10 sys.exit(1) in library code b067ab7a insecure_policy.py raises InsecureDependencyPolicyError(RuntimeError); CLI layer at install.py:1329, 1419 owns the exit. Library-level is pure.
11 Jargon in warning copy Warning now reads Insecure HTTP fetch (unencrypted). See nit N1 — sibling jargon persists in error recipe.
12 Transitive-block error insecure_policy.py:172-178 leads with Re-run with --allow-insecure-host HOST. One-step fix, copy-pasteable.
13 apm deps list --insecure column Renamed to "Origin" in Rich and text-fallback tables. Values direct / via <parent>. Docs match.
14 Commit hygiene / bisectability 20 commits, one-per-review-item, descriptive (review item N) tags. 333e8f00 config.py refactor cleanly isolated.

Architecture invariant (install.py <= 1525 LOC) preserved: 1496 LOC after the merge-commit extraction of mcp_warnings.py. Full unit suite 4767/4767 passing.


Nits (should-fix; not merge-blockers)

N1 — Double URL in validation-time path (CLI Logging, one-line fix)
When apm install <http-url> fails the insecure check, the reason text already starts with {url} -- but is passed to logger.validation_fail(package, reason) which prepends {package} -- . Result: [x] http://server/owner/repo -- http://server/owner/repo -- HTTP dependency.... Fix: call logger.error(reason) directly on the insecure-reject path at install.py:292-297, matching the other two failure paths. One-line change.

N2 — Wording inconsistency between warning and error copy (CLI Logging)
Warning says (unencrypted) (good, post-#11). Error recipe at insecure_policy.py:67 still says (no transport encryption). Unify to (unencrypted) for consistent register across both surfaces.

N3 — Context-aware hint when only one gate is missing (DevX)
_check_insecure_dependencies emits the full two-step recipe even when allow_insecure: true is already set on the dep in apm.yml. Users who already edited their manifest see "1. Set allow_insecure: true" redundantly. Conditionally emit only the missing step. Cosmetic; track as fast-follow.

N4 — _build_noninteractive_git_env intent comment (Security)
The env.pop("GIT_ASKPASS", None) at github_downloader.py:514 unconditionally removes the APM default, then conditionally restores it only when suppress_credential_helpers=True. The intent (allow user keychains for HTTPS/SSH fallback, block them for HTTP) is correct but fragile. Add a short comment to prevent a future refactor from inverting the logic. No behaviour change.

N5 — --insecure sample output missing from cli-commands.md (DevX)
The apm deps list reference doc shows a sample output block for default mode but none for --insecure. Add a second block with the "Origin" column to help users preview the output without installing an HTTP dep.


Tracked follow-ups (out of scope; open as issues)

  • F1 — Advisory SHA-pin for HTTP initial fetch (SRI-like expected-sha: in apm.yml) to break the first-fetch circular-trust loop documented in security.md:81. Hardening item.
  • F2is_insecure field not yet documented in lockfile-spec.md schema table. Doc-writer owns this.
  • F3packages/apm-guide/.apm/skills/apm-usage/dependencies.md doesn't mention allow_insecure or the transitive policy (commands.md does). Makes LLM-agent consumers slightly less informed on the dependencies.md page.
  • F4 — Pre-existing back-references in install/presentation/dry_run.py:15 and install/phases/finalize.py:24 violate the strict "no install/commands/" layering. Inherited from refactor(install): modularize install.py into engine package #764, not this PR's regression. Worth a small follow-up extraction.
  • F5 — Host-only changes without a scheme flip still bypass drift detection (documented limitation at drift.py:41-44). Not a regression; worth an issue.

Growth / positioning (side-channel)

PR #700 converts the README's "Install from anywhere" bullet from aspirational into defensible: corporate Gitea / Gogs / Artifactory / air-gapped Artifactory users can now install. The enterprise/security.md threat-model section is a sales-enablement asset that answers the enterprise-review question before it's asked. The apm deps list --insecure audit surface compounds that.

Contributor-velocity narrative: arika (@arika0093) (first-time external contributor, NONE-association) shipped a security-sensitive 1,300+ LOC feature with 204 new test functions, through two panel reviews, one atomic commit per review item. This is the contributor-funnel story APM should amplify in the release post — it sets the quality bar for future complex contributions.

Recommended release-notes hook:

APM now installs from HTTP mirrors — and makes every insecure choice visible. Corporate Gitea, Gogs, and air-gapped Artifactory users can apm install --allow-insecure with per-host transitive controls, credential-helper suppression, and apm deps list --insecure auditability.

README "Install from anywhere" bullet should be updated in a follow-up to mention HTTP mirrors with explicit opt-in.


Conflict resolution

Merged cleanly against main across two rounds (PR #809 landed mid-review):

Full suite green on HEAD. No functional changes from these merge commits — pure conflict resolution.


Final call

Merge-ready. Single actionable ask before merge: fix N1 (one-line: swap logger.validation_faillogger.error on the insecure-reject path). N2–N5 can ship as a tiny follow-up. F1–F5 → open as issues.

Outstanding work by arika (@arika0093). Thank you for seeing this through two review rounds with the care each commit shows.


Panel: python-architect, cli-logging-expert, devx-ux-expert, supply-chain-security-expert, oss-growth-hacker. Synthesis via apm-review-panel skill. Conflicts resolved in c8bcc700 + 35d64631.

Comment thread tests/unit/test_install_command.py Fixed
Round-2 apm-review-panel findings on PR microsoft#700. Five specialists converged
on APPROVE with minor nits; this commit applies the actionable ones.

N1 (cli-logging): fix double-URL rendering in the validation-time path.
`_format_insecure_dependency_requirements` already embeds the full URL in
the reason string, but `logger.validation_fail(package, reason)` prepends
"{package} -- " and produced output like
"[x] http://host/repo -- http://host/repo -- HTTP dependency...". Switch
to `logger.error(reason)` on the insecure-reject branch to match the
other two failure paths. `invalid_outcomes` still tracks the package so
the later validation summary sees it.

N2 (cli-logging): unify jargon register. Warning copy already says
"(unencrypted)" post-review microsoft#11; align the error recipe from
"(no transport encryption)" to "(unencrypted)" so both surfaces use the
same wording.

N3 (devx): context-aware remediation steps. `_check_insecure_dependencies`
previously emitted the full two-step recipe in both failure branches,
even when the user had already performed step 1 (manifest edit). The
factory now takes `missing_dep_allow` and `missing_cli_flag` keyword
args and only renders the missing step(s). The add-time validation
path keeps the default of both-steps since the dep is not yet in
apm.yml. Tests updated to match the new per-branch assertions.

N4 (security): document the intentional two-stage credential-helper
policy in `_build_noninteractive_git_env`. The pop-then-conditionally-
restore of GIT_ASKPASS is correct but fragile; an explicit docstring
prevents a future refactor from inverting the logic and leaking
credentials over plaintext HTTP or blocking system keychains on
HTTPS/SSH fallback.

N5 (doc-writer): add `apm deps list --insecure` sample output to
cli-commands.md showing the bold-red `Origin` column with `direct`
and `via <parent>` example rows.

F2 (doc-writer): document `is_insecure` and `allow_insecure` fields in
lockfile-spec.md section 4.2, including replay semantics and legacy
fail-closed behaviour.

F3 (doc-writer): add "HTTP dependencies (opt-in)" section to the
apm-guide dependencies.md skill resource so LLM agents consuming the
skill have the dual-opt-in model, example manifest, example CLI
invocation, and cross-references to commands.md and the enterprise
security guide.

Full unit suite: 4767/4767 passing (one deselect unrelated).
`install.py` at 1497 LOC (under the 1525-LOC invariant).

Panel verdict: microsoft#700 (comment 4290979672)

Co-authored-by: arika0093 <arika0093@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@danielmeppiel

Copy link
Copy Markdown
Collaborator

Follow-up on the panel verdict: I've pushed 2c7e81bf addressing N1-N5, F2, and F3 directly on top of your branch so the PR stays a single-author squash-merge under your name. Co-authorship trailers preserve the full attribution.

Nits addressed

  • N1 -- Fixed the double-URL render. install.py now calls logger.error(reason) on the insecure-reject path instead of logger.validation_fail (which prepends the package name, producing pkg -- url -- HTTP dependency...).
  • N2 -- Unified the jargon register: error recipe now says (unencrypted) to match the warning copy.
  • N3 -- _format_insecure_dependency_requirements now takes missing_dep_allow / missing_cli_flag kwargs and only emits the steps the user actually needs. If allow_insecure: true is already on the dep entry, the error only asks for --allow-insecure (and vice versa). The CLI add-time path keeps both steps by default. Unit tests updated to assert the per-branch wording.
  • N4 -- Added an explicit docstring to _build_noninteractive_git_env documenting the intentional two-stage credential-helper policy (pop-then-conditionally-restore of GIT_ASKPASS), preventing a future refactor from inverting the logic.

Follow-ups addressed (doc-writer agent)

  • N5 -- Added an apm deps list --insecure sample output block to docs/.../reference/cli-commands.md showing the Origin column.
  • F2 -- Documented is_insecure and allow_insecure in docs/.../reference/lockfile-spec.md with replay semantics and legacy fail-closed behaviour.
  • F3 -- Added an "HTTP dependencies (opt-in)" section to packages/apm-guide/.apm/skills/apm-usage/dependencies.md so LLM agents consuming the skill know the dual opt-in model.

Validation

  • Full unit suite: 4767/4767 passing (one pre-existing deselect unrelated to this PR).
  • install.py at 1497 LOC -- under the 1525-LOC architecture invariant.

Remaining follow-ups (F1 advisory SHA-pin, F4 install/ -> commands/ inherited back-references, F5 host-only drift) are out of scope and should be filed as separate issues. Ready for maintainer merge whenever CI gates clear.

Thank you for the iteration quality -- the one-commit-per-review-item hygiene made this second round fast and high-confidence.

CodeQL flagged `"http://gitlab.company.internal" in arg` at
test_install_command.py:963 as a potentially unsafe URL substring check:
the hostname could appear at an arbitrary position in the checked URL
(e.g. inside a path segment, query value, or userinfo) rather than as
the actual host, producing a false-positive match.

Replace substring checks with `urllib.parse.urlsplit` and compare
`scheme` and `netloc` explicitly:

- test_explicit_http_generic_host_tries_http_first (the flagged case):
  parse each subprocess-arg URL and require scheme == "http" AND
  netloc == "gitlab.company.internal".
- test_generic_host_falls_back_to_https_when_ssh_fails (sibling with
  the same pattern at line 929): same treatment for the HTTPS arg;
  keep the SSH SCP-form check as `"git@git.example.org:" in arg` since
  SCP-style URLs are not parseable by urlsplit.

Both tests still assert the same contract: explicit `http://` does not
fall back to SSH, and SSH failure falls back to HTTPS on the right host.

Co-authored-by: arika0093 <arika0093@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEATURE] Support installation from git repositories hosted over HTTP only

5 participants