📥 download_index — Download the Package Index

This example demonstrates how to download the complete Composer package-name index (the full Packagist package list) in one shot and persist the index data to a local file.

🎯 Example Positioning

download_index is the 2nd example in the Packagist API Remote Operations series, following basic_setup. It focuses on one plain but highly practical scenario: getting the names of all packages in the repository.

• 📚 What you'll learn: how to call package-level convenience functions in pkg/repository to pull the raw bytes of the lightweight index endpoint https://packagist.org/packages/list.json into memory, then persist them to a JSON file.
• 🔗 Corresponding SDK methods: repository.DownloadIndex and repository.DownloadIndexToFile.
• 💡 Difference from list_packages: list_packages goes through PackagistClient's typed method (returning a structured PackageListResponse); this example directly downloads raw bytes, suitable for mirror initialization, offline caching, or custom parsing pipelines.

💻 Full Code

package main

import (
	"context"
	"fmt"
	"os"
	"path/filepath"

	"github.com/scagogogo/composer-skills/pkg/repository"
)

func main() {
	// Example 2: Download the Composer package index
	// The index file contains the list of all available packages in the repository.

	// Step 1: Create a context, usable for controlling request timeouts, etc.
	ctx := context.Background()

	// Step 2: Download the index directly into memory
	fmt.Println("Downloading the package index...")
	indexBytes, err := repository.DownloadIndex(ctx)
	if err != nil {
		fmt.Printf("Failed to download the index: %v\n", err)
		return
	}
	fmt.Printf("Index downloaded successfully: %d bytes\n", len(indexBytes))

	// The index data is a JSON string, for example:
	// {"packageNames":["vendor1/package1","vendor2/package2",...]}
	// You can parse the JSON to get all package names

	// Step 3: Save the index to a file
	tempDir, err := os.MkdirTemp("", "composer-index")
	if err != nil {
		fmt.Printf("Failed to create temp directory: %v\n", err)
		return
	}
	defer os.RemoveAll(tempDir) // Clean up when the example ends

	indexPath := filepath.Join(tempDir, "composer-index.json")

	// Use the convenience method to download and save to a file directly
	fmt.Printf("Saving the index to file: %s\n", indexPath)
	err = repository.DownloadIndexToFile(ctx, indexPath)
	if err != nil {
		fmt.Printf("Failed to save the index file: %v\n", err)
		return
	}

	// Get file info to verify the save succeeded
	fileInfo, err := os.Stat(indexPath)
	if err != nil {
		fmt.Printf("Failed to get file info: %v\n", err)
		return
	}
	fmt.Printf("Index file saved successfully, file size: %d bytes\n", fileInfo.Size())

	// Example output:
	// Downloading the package index...
	// Index downloaded successfully: 1234567 bytes
	// Saving the index to file: /tmp/composer-index-123456/composer-index.json
	// Index file saved successfully, file size: 1234567 bytes
}

🧩 Code Walkthrough

• 🧱 Create a context: ctx := context.Background() serves as the root context for API calls; you can derive a timeout-bearing context (context.WithTimeout) on top of it to avoid long blocking during large-file downloads.
• 🌐 Download the raw index: repository.DownloadIndex(ctx) hits the list.json endpoint and returns the raw JSON as []byte. It does not parse the structure, so both memory footprint and CPU overhead are minimal — suitable for persisting first and letting downstream consumers parse as needed.
• 📦 JSON structure hint: the returned byte stream looks like {"packageNames":[...]}; when you need the package names, deserialize it with encoding/json (this example doesn't expand on parsing to keep the "download" semantics front and center).
• 🗂️ Create a temp directory: os.MkdirTemp("", "composer-index") creates a dedicated subdirectory under the system temp directory, and defer os.RemoveAll(tempDir) ensures cleanup after the example ends, avoiding leaking temp files.
• 💾 One-step persistence: repository.DownloadIndexToFile(ctx, indexPath) is the "download + write file" convenience wrapper — it internally reuses DownloadIndex and writes with os.WriteFile at permission os.ModePerm.
• ✅ Verify the result: os.Stat(indexPath) retrieves the file info, and comparing fileInfo.Size() against the in-memory byte count confirms the persisted file is intact.
• 🛡️ Error handling: each step checks err and returns on failure, preventing errors from propagating downward; in production, swap this out for logging/alerting.

▶️ How to Run

Run directly from the example directory:

cd /home/cc11001100/github/scagogogo/composer-skills/examples/download_index
go run main.go

⚠️ This example makes real Packagist API calls; avoid running it at high frequency to avoid burdening the target server. The index file is large (several MB), so ensure adequate network and memory.

📚 SDK Methods Involved

Method Name	Package	Endpoint	Doc Link
DownloadIndex	pkg/repository	GET /packages/list.json	/sdk/packagist/methods/list-packages
DownloadIndexToFile	pkg/repository	GET /packages/list.json + local file write	/sdk/packagist/methods/list-packages

📝 Note: The repository package provides these two convenience methods as package-level functions; the endpoint they hit behind the scenes is identical to PackagistClient.ListPackages (both list.json). The link above points to the typed-method docs for that endpoint, convenient for cross-referencing the field definitions of the structured return value PackageListResponse.

🚀 Going Further

• ⏱️ Add timeout control: replace the bare context.Background() with ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second), defer cancel(), and then call DownloadIndex to prevent network hangs.
• 🔍 Structured parsing: deserialize indexBytes into domain.PackageListResponse, then combine with repository.GetPackage to batch-pull each package's metadata and build a local search index.
• 🪞 Mirror initialization: use the output of DownloadIndexToFile as the "full manifest" starting point for a self-hosted Satis / private mirror, paired with get-package-changes for incremental updates.
• 🧵 Concurrent downloads: after getting the package-name list, use a worker pool (errgroup or a buffered channel) to concurrently pull each package's details — be sure to add time.Sleep or a token-bucket rate limiter to respect Packagist's rate limits.
• 🔐 Integrity verification: after persisting, compute the file's SHA256 and compare it against the previous result to determine whether the index has been updated, avoiding redundant full pulls.