Skip to main content

Generating Dependency Report

This guide explains how to generate and update the third-party dependency documentation using the VS Code task available in the Step Repository. This automated system tracks all dependencies across the monorepo and maintains up-to-date license compliance information.

Overview​

The Step Repository includes an automated dependency tracking system that:

  • Scans all packages in the monorepo for third-party dependencies
  • Extracts version and license information from package registries
  • Generates a CSV report with comprehensive dependency data
  • Updates the Docusaurus documentation with current dependency tables

When to Run the Task​

You should generate a new dependency report in these situations:

1. After Adding New Dependencies​

Whenever you add a new dependency to any package:

  • Added a new npm package to package.json
  • Added a new Rust crate to Cargo.toml
  • Added a new Maven dependency to pom.xml

2. After Updating Dependency Versions​

When you update existing dependencies to new versions:

  • Running yarn upgrade or cargo update
  • Manually bumping version numbers
  • Merging dependabot or renovate PRs

3. Before Major Releases​

As part of the release checklist:

  • Ensure all dependency information is current
  • Verify license compliance for the release
  • Update documentation for stakeholders

4. During License Audits​

When conducting compliance reviews:

  • Generate fresh license information
  • Review any new dependencies
  • Check for license compatibility issues

5. Periodic Maintenance​

Regular updates (recommended monthly or quarterly):

  • Keep documentation synchronized with actual dependencies
  • Catch any drift between code and documentation
  • Maintain accurate compliance records

How to Execute the Task​

The easiest and most reliable method is through the VS Code task:

  1. Open the Command Palette

    • Press Ctrl+Shift+P (Linux/Windows) or Cmd+Shift+P (macOS)

  2. Select the Task

    • Type Tasks: Run Task and press Enter
    • Search for generate.dependency.report
    • Press Enter to execute

  3. Monitor Progress

    • The task opens in a dedicated terminal panel
    • Watch for completion messages:
      • 🔍 Generating dependency CSV...
      • 📝 Updating markdown documentation...

  4. Verify Results

    • Check the terminal for any errors
    • Review the updated files (see Output Files section)

Using Command Line​

Alternatively, you can run the task manually from the terminal:

# From the repository root
devenv shell bash -- -c ./scripts/tasks/dependencies/generate-dependency-report.sh

Or if you're already in a devenv shell:

./scripts/tasks/dependencies/generate-dependency-report.sh

How It Works​

The dependency report generation system consists of three main components that work together:

1. Dependency Scanner (list_deps.py)​

The scanner examines all packages in the monorepo and extracts dependency information:

What it scans:

  • Rust packages: Reads Cargo.toml files, queries crates.io API for license data
  • TypeScript/JavaScript packages: Reads package.json files, queries npmjs.com API for license data
  • Java packages: Reads pom.xml files, queries Maven Central API for license data

What it extracts:

  • Package name
  • Dependency name and version
  • License information (fetched from package registries)
  • Description (when available)

Output:

  • CSV file at docs/docusaurus/docs/reference/third_party_deps/assets/dependencies.csv
  • Contains columns: Package, Dependency, Version, License, Description

2. Documentation Generator (update_deps_docs.py)​

The generator intelligently updates the markdown documentation:

How it works:

  • Reads the CSV file generated by the scanner
  • Parses the existing markdown documentation
  • Detects package sections automatically (e.g., "Admin Portal", "Voting Portal")
  • Maps CSV package names to markdown section headers
  • Preserves all existing content: descriptions, overviews, license compliance sections
  • Updates only dependency tables with current data from CSV
  • Maintains formatting and structure

Key feature:

  • Safe to run repeatedly without losing manual edits to package descriptions or other content

3. Orchestration Script (generate-dependency-report.sh)​

The wrapper script coordinates the entire process:

Steps:

  1. Creates a Python virtual environment (if not exists)
  2. Activates the virtual environment
  3. Installs required Python dependencies (requests, tomli)
  4. Runs the dependency scanner to generate CSV
  5. Runs the documentation generator to update markdown
  6. Outputs results to docs/docusaurus/docs/reference/third_party_deps/

Environment setup:

  • Uses isolated Python virtualenv at scripts/tasks/dependencies/.venv
  • Installs dependencies from requirements.txt
  • Cleans up after completion

Technical Details​

VS Code Task Configuration:

The task is defined in .vscode/tasks.json (lines 28-51):

{
  "label": "generate.dependency.report",
  "command": "devenv",
  "args": [
    "shell",
    "bash",
    "--",
    "-c",
    "./scripts/tasks/dependencies/generate-dependency-report.sh"
  ],
  "presentation": {
    "reveal": "always",
    "panel": "dedicated"
  },
  "isBackground": false
}

Key aspects:

  • Runs within the devenv shell for consistent environment
  • Uses a dedicated terminal panel for output
  • Synchronous execution (waits for completion)
  • Always reveals output for monitoring

Output Files​

After running the task, you'll find updated files at:

CSV Report​

Location: docs/docusaurus/docs/reference/third_party_deps/assets/dependencies.csv

Format:

Package,Dependency,Version,License,Description
admin-portal,react,18.2.0,MIT,React is a JavaScript library for building user interfaces.
admin-portal,typescript,5.3.3,Apache-2.0,TypeScript is a language for application scale JavaScript development
...

Purpose:

  • Machine-readable dependency data
  • Source for documentation generation
  • Can be used for automated compliance checks

Markdown Documentation​

Location: docs/docusaurus/docs/reference/third_party_deps/third_party_deps.md

Purpose:

  • Human-readable documentation
  • Automatically included in Docusaurus site
  • Available at /docs/reference/third_party_deps/third_party_deps

Structure:

  • Overview section
  • Package sections with dependency tables
  • License compliance information
  • Automatically formatted and sorted

Why This System Exists​

License tracking:

  • Maintains accurate records of all third-party licenses
  • Ensures compliance with open source license requirements
  • Facilitates legal reviews and audits

Attribution:

  • Provides proper attribution to dependency authors
  • Documents license terms for each dependency
  • Supports transparency requirements

Operational Benefits​

Dependency visibility:

  • Single source of truth for all dependencies
  • Easy to review what packages use what dependencies
  • Helps identify duplicate or outdated dependencies

Security:

  • Know what external code is in the system
  • Identify packages that need security updates
  • Track dependency chains for vulnerability management

Maintenance:

  • Understand impact of dependency updates
  • Plan upgrades and migrations
  • Document technical debt

Documentation Quality​

Automation:

  • Reduces manual documentation burden
  • Prevents documentation drift
  • Ensures accuracy through direct source scanning

Consistency:

  • Standardized format across all packages
  • Uniform license information presentation
  • Predictable structure for readers

Best Practices​

1. Run After Dependency Changes​

Always run the task after modifying dependencies:

# After adding/updating dependencies
yarn add some-package
# or
cargo add some-crate

# Then run the task
# VS Code: Ctrl+Shift+P → Tasks: Run Task → generate.dependency.report

2. Review Generated Output​

After running the task:

  • Check the CSV for completeness
  • Review the markdown for formatting
  • Verify license information looks correct
  • Note any "N/A" licenses for manual investigation

3. Commit Both Files​

Always commit both generated files together:

git add docs/docusaurus/docs/reference/third_party_deps/assets/dependencies.csv
git add docs/docusaurus/docs/reference/third_party_deps/third_party_deps.md
git commit -m "docs: update dependency report"

4. Include in PRs​

When your PR adds or updates dependencies:

  • Run the dependency report task
  • Include the updated files in your PR
  • Mention the license changes in PR description

5. Regular Updates​

Schedule regular dependency report updates:

  • Set a calendar reminder (monthly or quarterly)
  • Include in release checklist
  • Part of dependency update sprints

Troubleshooting​

Task Not Found in VS Code​

Problem: Can't find generate.dependency.report in task list

Solutions:

  1. Reload VS Code: Ctrl+Shift+P → Developer: Reload Window
  2. Check .vscode/tasks.json exists in repository root
  3. Ensure you're in the devenv shell

Python Virtual Environment Issues​

Problem: Script fails with Python module errors

Solutions:

  1. Delete the virtual environment: rm -rf scripts/tasks/dependencies/.venv
  2. Run the task again (will recreate virtualenv)
  3. Check Python 3 is available: python3 --version

API Rate Limiting​

Problem: Script fails with "rate limit exceeded" errors

Solutions:

  1. Wait a few minutes and retry
  2. The script queries package registries (crates.io, npmjs.com, Maven Central)
  3. If persistent, run script at a different time
  4. Consider implementing API tokens (for maintainers)

Incomplete License Information​

Problem: CSV shows "N/A" for some licenses

Solutions:

  1. This may be expected for packages without proper license metadata
  2. Manually check the package repository for license information
  3. Update the CSV manually if needed (note: will be overwritten on next run)
  4. Consider filing issues with upstream packages to add license metadata

Merge Conflicts in Generated Files​

Problem: Git merge conflicts in CSV or markdown

Solutions:

  1. Accept incoming changes (from main/master branch)
  2. Re-run the dependency report task
  3. The task will regenerate correct current state
  4. Commit the regenerated files

docs/docusaurus/docs/

Further Information​

For more details about the dependency scanning system architecture, see:

  • scripts/tasks/dependencies/README.md - System architecture and components
  • scripts/tasks/dependencies/list_deps.py - Dependency scanner implementation
  • scripts/tasks/dependencies/update_deps_docs.py - Documentation generator implementation
  • .vscode/tasks.json - Complete task configuration