@codacy/tools-opengrep-1

Table of Contents

• Overview
• Rule bundling
• Updating patterns
• Updating rules
• Updating the Opengrep version
• Development
• Notes for maintainers

Overview

Multi-language SAST (Static Application Security Testing) scanner using the Opengrep binary. Supports 30+ languages via YAML rule files. Uses the CLI execution strategy.

Rule bundling

Opengrep uses YAML rule files for detection. The adapter bundles ~2500 rules into src/rules.yaml, sourced from:

1. opengrep/opengrep-rules — the registry rules, pinned at the same commit as the Codacy image
2. GitLab SAST rules — MIT-licensed, from gitlab.com/gitlab-org/security-products/sast-rules
3. Codacy custom rules — docs/codacy-rules*.yaml in codacy/codacy-opengrep

The bundled src/rules.yaml is committed to the repository for consistency and reproducibility.

Keeping it in step with the Codacy image

All three sources, their pinned commits and their file filters are mirrored from codacy/codacy-opengrep's internal/docgen/parsing.go. Keep them that way: the image is the reference implementation, and scripts/fetch-rules.mjs says which upstream function each piece came from.

Three mistakes in that mirroring cost 296 of 2534 patterns (12%), every one of them silent — the pattern stays in patterns.json, init --auto enables it, the config claims it is active, and it simply never fires:

• Rule ids were derived from the file path. GitLab ids are written in the file and must be taken verbatim. Path derivation looks right for the top-level tree (java/crypto/rule-X.yml really does hold java_crypto_rule-X) and is wrong for the licence-tiered one, where rules/lgpl-cc/properties/spring/accesscontrol/rule-Y.yml holds properties_spring_rule-Y. The sharpest case is rule-JwtDecodeWithoutVerify.yml, whose id uses an underscore where the filename uses a hyphen. 265 of 588 GitLab rules were lost this way.
• The registry source was upstream semgrep/semgrep-rules, not Opengrep's fork. The two share most ids, so the output looks plausible while individual rules go missing.
• YAML was read and rewritten line by line. That corrupted multi-line flow scalars — the generated file stopped parsing, and opengrep then reported zero findings for the entire run rather than an error — and it skipped every tests/ directory, where some shipped rules live.

test/rules-coverage.test.ts is the guard: it fails if any catalogued pattern has no rule, and if the generated file does not parse. Run pnpm fetch-rules when it does; the script prints the orphaned ids.

Updating patterns

# Re-fetch pattern metadata from the Codacy API
pnpm prefetch

# Commit the result
git add src/patterns.json

Pattern IDs follow the format Semgrep_python.lang.security.audit.exec-detected.exec-detected (dot-separated rule paths).

Updating rules

# Re-fetch and bundle rules from all three sources
pnpm fetch-rules

# This clones the rule repos, extracts matching rules, and writes src/rules.yaml
# Only rules whose IDs match src/patterns.json are included — and the script now
# warns loudly about the reverse, a catalogued pattern with no rule to run.

# Commit the result
git add src/rules.yaml

The fetch-rules script (scripts/fetch-rules.mjs):

• Clones semgrep/semgrep-rules at the pinned commit
• Clones GitLab SAST rules (latest)
• Clones codacy/codacy-opengrep for custom rules
• Extracts rules, assigns path-based IDs
• Filters to only rules matching src/patterns.json
• Writes concatenated YAML to src/rules.yaml

To update the pinned semgrep-rules commit, edit SEMGREP_RULES_COMMIT in scripts/fetch-rules.mjs.

Updating the Opengrep version

1. Update preferredVersion in src/adapter.ts
2. Run pnpm test to verify compatibility
3. Consider re-running pnpm fetch-rules if the new version supports new rule syntax

Development

pnpm build        # Build with tsup
pnpm test         # Run tests (requires opengrep in PATH)
pnpm fetch-rules  # Re-bundle rules from upstream sources
pnpm prefetch     # Re-fetch pattern metadata from Codacy API

Installing the binary:

The adapter auto-installs opengrep into ~/.codacy/tools/ when run with --install-dependencies (no manual step needed):

codacy-analysis analyze . --tool opengrep --install-dependencies

install() downloads the matching statically-linked binary from GitHub releases (no archive to unpack) and chmod +xes it (not on Windows). The platform/arch/libc → asset mapping mirrors opengrep's official install.sh and install.ps1:

To install manually instead:

curl -fsSL https://raw.githubusercontent.com/opengrep/opengrep/main/install.sh | bash
# Windows (PowerShell)
irm https://raw.githubusercontent.com/opengrep/opengrep/main/install.ps1 | iex

Notes for maintainers

• Preset mappings (src/pattern-mappings.json): prefixes are the upstream rule paths (Semgrep_<lang>.<framework>.…), which already name the framework, so this mapping is prefix-only — Opengrep's tags are topical (security, audit) and would over-match. 22 keys were added, taking ~91 default-on patterns out of repos that lack the framework: Pyramid (16), PyCryptodome (12), Kubernetes/Helm (14, via yaml.kubernetes. + codacy.k8s.; the kubernetes signal matches loose manifests under k8s//kubernetes//manifests/, not just kustomization.yaml, or the most common layout would have lost all 14), GitHub Actions (5), AWS CDK (5), Jinja2 (2), and the smaller Boto3/Twilio/Android/Sequelize/Doctrine families. The YAML-language families matter most: before this they fired in any repo containing a single .yaml file.

  The *.aws-lambda. packs (34 default-on across Python, JavaScript, Ruby and Go) are claimed by seven keys — serverless-framework, cloudformation, aws-cdk, boto3, aws-sdk, aws-sdk-go and lang:Terraform. Handler code carries no manifest of its own; the only evidence is how it is deployed or which SDK it calls, and any one of the seven is enough. Terraform earns a key because deploying a Lambda with aws_lambda_function leaves no other trace, and the SDK telltales had to learn npm's v2 aws-sdk, Go's v1 aws-sdk-go and PyPI's botocore/aws-lambda-powertools — Lambda runtimes preinstall boto3, so it is often absent from requirements.txt entirely.

  Semgrep_python.jinja2. is claimed by jinja2 and flask: Flask hard-depends on Jinja2, so gating on a direct jinja2 declaration alone dropped both Error-severity autoescape rules from most real Flask repos.

  Deliberately still universal, guarded by a test so it stays that way: *.lang.* (the generic per-language packs), csharp.dotnet. (the platform, not a library), terraform.* (already gated by the Terraform language), problem-based-packs.* (cross-language), and apex. / generic.visualforce. — Apex and VisualForce are Salesforce-only languages, so the language gate already scopes them and keying them on salesforce would only add a way to miss them in a Salesforce repo with no sfdx-project.json.

• Tool ID is Semgrep for legacy compatibility with the Codacy platform. The displayName is set to Opengrep for CLI display. Never change the tool ID without coordinating with the Codacy platform team.

• Config materialization: When Codacy patterns are specified, the adapter loads src/rules.yaml, filters to enabled rules, and writes a filtered config to .codacy/generated/opengrep-rules.yaml.

• check_id is the bare rule ID: every invocation passes --no-rewrite-rule-ids. Without it Opengrep prefixes check_id with the config file's directory relative to cwd (.codacy/generated/x.yaml → codacy.generated.), and the adapter used to reproduce that path math to strip the prefix again — never verified on Windows, where a mismatch would silently drop every finding at the enabled-pattern check. With the flag the rule ID maps to a pattern ID directly on every OS.

• Config file search: .semgrep.yaml, .opengrep.yaml, .semgrep.yml, .opengrep.yml (in that order).

• Opengrep errors: Parse errors (syntax errors, partial parsing) from Opengrep's JSON errors array are converted to structured AnalysisError objects with filePath. Stderr (scan status table) is logged at debug level only.

• Rule ID format: Semgrep-rules use {dir}.{filename}.{ruleId} (dots), GitLab rules use {dir}_{filename} (underscores), Codacy custom rules keep their original IDs.

• Install / binary download: install() downloads the bare per-platform binary from https://github.com/opengrep/opengrep/releases/download/v{version}/{asset} and chmod +xes it into ~/.codacy/tools/. The release tag carries a v prefix (v1.30.0) even though preferredVersion does not. On Linux the libc flavour is detected with an ldd /bin/sh probe (musl → musllinux, else manylinux); detection failures fall back to glibc/manylinux. On Windows the binary is saved as opengrep.exe; upstream builds only x64, which install.ps1 also installs on ARM64 (emulated), so we do the same. From opengrep 2.0.0 the Windows asset becomes opengrep_windows_x86.zip (exe + DLLs) — bumping preferredVersion past 2.0 needs an extraction step. The argv chunk budget drops to MAX_ARGV_BYTES (30,000) on Windows, whose command line is capped at 32,767 characters. Unlike install.sh, the Codacy-managed install does not verify the cosign .cert/.sig signatures — if signature verification becomes a requirement, fetch those sibling assets and validate before chmod.

• Argv-size chunking (no directory walk): The adapter always passes Opengrep an explicit file list — the runner's already git-ignored, exclude-filtered target set — so Opengrep never walks the repository tree itself. chunkFilesByArgvBudget() splits that list into the fewest, largest chunks that each fit ARGV_BYTE_BUDGET (200 KB, well under ARG_MAX), and each chunk is scanned in its own opengrep scan invocation; findings and errors are merged across chunks. Chunk by size, not count: every invocation re-compiles rules.yaml, so fewer/larger chunks amortize that fixed cost. The earlier directory/“root” fallback was removed because on huge repos (a single tracked file at the repo root collapsed the minimal-cover dirs to .) it made Opengrep walk the entire tree — re-scanning excluded/generated/vendored files and timing out at the runner's 10-minute limit.

• File-size cap: invocations pass --max-target-bytes 1000000 (~1 MB). This restores Opengrep's default size cap — the adapter previously passed 0, which disables the cap, so giant generated/minified bundles were scanned and dominated runtime. The cap applies to explicitly-listed files (verified on v1.16.4). Note: --exclude-minified-files is not used — that flag is rejected by the bundled binary unless --experimental (a different engine) is also passed, so we rely on the size cap instead. (The runner also enforces a tighter 150 KB per-file limit centrally before files reach the adapter, so this --max-target-bytes is now a secondary defense.)

• Partial results on timeout: the per-chunk loop checks ctx.signal (the runner's timeout abort) between chunks and passes it to spawnTool, so an in-flight scan is killed promptly. On abort the adapter keeps the findings from completed chunks and adds a single Timeout error (→ status: "partial") rather than losing everything. This is why chunking matters beyond ARG_MAX: it's the granularity at which a long scan can be stopped with results preserved.