Skip to content
All work

Flagship case study

Quality Engineering · Automation · Performance Engineering · Cloud Infrastructure · AI-Assisted Observability

From Manual Validation to an Automated Security Test System

Designing an automated, infrastructure-aware test system for validating hosted artifact repository scanning under configurable load.

Designed an end-to-end automation workflow that provisions environments, generates realistic repository data, integrates security analysis, orchestrates scanning, applies controlled infrastructure load, monitors execution, and validates results.

  • Jenkins
  • Terraform
  • Gatling
  • AWS
  • Python
  • Boto3
  • APIs
  • AI Agents

The story, in eight moves

  1. Problem
  2. Infrastructure
  3. Data
  4. Integration
  5. Load
  6. Scan
  7. Observe
  8. Validate

Architecture generalized to protect confidential implementation details.

01The problem

Three connected systems. One new scanning capability. A lot of manual work.

A new capability scanned artifacts stored in hosted repositories. Validating it by hand meant stitching three systems together, again and again.

Artifact Repository

Stores software artifacts and dependencies.

Security Analysis

Analyzes artifacts and identifies security and policy information.

Firewall / Audit Layer

Evaluates components entering the repository ecosystem.

  1. 01Provision environments
  2. 02Deploy product builds
  3. 03Configure integrations
  4. 04Create repositories
  5. 05Acquire realistic artifacts
  6. 06Upload artifacts
  7. 07Trigger scans
  8. 08Monitor execution
  9. 09Validate results
  10. 10Repeat per configuration
  11. 11Test under controlled load

02The question

"Can the entire validation workflow be transformed into a repeatable engineering system?"

Manual testing

  1. Provision
  2. Configure
  3. Create repos
  4. Generate data
  5. Upload
  6. Trigger scan
  7. Monitor
  8. Validate
  9. Cleanup

Nine hand-driven steps, repeated for every configuration.

03My approach

I didn't automate the test. I automated the entire environment around the test.

The solution is an end-to-end orchestration system — not a single automated test. Click any component to see what it does and how it connects.
Developer InputJenkins PipelineBuild InputEnv ProvisioningConfigurationArtifactsTerraformTest ParametersTest EnvironmentsRepository SystemSecurity SystemRealistic DataScan ConfigurationLoad ControlScan ExecutionObservabilityValidationBuild Artifacts

Component · exec

Load Control

A feedback loop that raises workload until observed CPU reaches the target, then holds.

Receives from
Realistic Data · Scan Configuration
Feeds into
Scan Execution

04Pipeline journey

Eleven stages, one run

Replay the pipeline or step through each stage.
Stage 1 / 11

Build

Receives the requested branch/build configuration and builds the required artifacts.

Output → Deployable artifacts

05Configuration engine

Configure the experiment

A conceptual view of the parameters every run accepts. Change a value to see which stage it affects — and whether pre-flight lets it through.

Experiment parameters · conceptual

Affected stage

  1. Pre-flight
  2. Provision
  3. Generate Data
  4. Create Repos
  5. Apply Load
  6. Start Scan

Repository layout · 12 repos × 200 artifacts

Pre-flight validation

User configValidation engineValid → provision

Heap 6 GB is compatible with a 16 GB instance. Provisioning may proceed.

Rule shown is illustrative. The real check is an internal resource compatibility calculation.

06Pre-flight validation

Fail before you build the environment

Some configurations are incompatible — heap must suit the selected infrastructure. A pre-flight check rejects invalid combinations before infrastructure cost and execution time are incurred. Try a heap larger than half the instance memory above.

07Realistic data

Real data instead of synthetic noise

The test needs realistic artifact and repository behavior. A controlled catalogue of real artifacts is built, stored, then distributed across many hosted repositories that become the scanning dataset.
Artifact catalogue
Repo A
JARJARJAR
Repo B
JARJARJAR
Repo C
JARJARJAR

Catalogue → distribution → realistic dataset → security scanning

Creating repository activity at scale · 120 RPS (configurable)

  1. Gatling
  2. Request Model
  3. Repository Activity
  4. Hosted Repositories
  5. Scanning Dataset

Gatling simulations generate controlled repository activity. No performance numbers are claimed here.

08Load-aware testing · the key innovation

Testing the feature under controlled infrastructure load

"What if the scan must be validated not just in an idle environment, but under a defined CPU load?" The target is configurable — 40% is just an example. The loop measures → adjusts workload → reaches target → holds → starts the scan, with load generation and scanning running in coordination.
6%target 40%

Load generation

Scanning

Control loop

  1. Target CPU
  2. Load controller
  3. Create activity
  4. Upload artifacts
  5. Metrics (Boto3)
  6. Current CPU
  7. Compare
Below → increase loadTarget met → hold load

Start scan

Controller log · simulated

Idle. Run the loop to watch it converge.

    09AI-assisted observability

    I don't want engineers watching Jenkins

    A local Claude-based agent follows execution logs and flags only the states that need a human. The point isn't "I used Claude" — it's reducing the human attention a long-running workflow demands.

    Engineer watching pipeline

    Nobody should sit and stare at a long-running job.

    1. Raw Logs
    2. AI Agent
    3. Event Understanding
    4. State Detection
    5. Human Attention Only When Needed

    Event stream · generic example

    • Press Stream to replay a run.

    AI is used as an observability assistant; final engineering decisions remain with the engineer.

    10Validation

    The pipeline doesn't stop at execution

    Every scanned repository is checked against expected results. Logs, reports and generated data are retained as build artifacts.
    Scan complete → repository validation
    ExpectedActual
    Validation report + artifacts

    11Preserve the environment

    Automation should not destroy evidence before engineers can investigate it

    Environments are not torn down automatically. Some failures need hands-on inspection, so cleanup is an explicit choice.

    Validation passed

    1. Test complete
    2. Optional cleanup

    Investigation required

    1. Keep environment
    2. Engineer inspection

    12What I automated

    Before and after

    • Manual environment setup
    • Manual configuration
    • Manual repository creation
    • Manual artifact upload
    • Manual scan triggering
    • Manual pipeline monitoring
    • Manual validation
    • Manual cleanup

    13Engineering principles

    Six principles behind the design

    01

    Fail Fast

    Validate configuration before expensive execution.

    02

    Repeatability

    Same inputs should produce repeatable test environments.

    03

    Realistic Data

    Controlled real artifact datasets instead of meaningless synthetic noise.

    04

    Configurability

    Infrastructure, JVM, heap, RPS, repository and load parameters can all change.

    05

    Observability

    Long-running workflows should not require continuous human attention.

    06

    Debuggability

    Never destroy an environment before engineers can investigate it.

    14Why this is different

    What makes this approach interesting?

    End-to-end test engineering system
    AutomationInfrastructureDataPerformanceAIObservabilityValidationDebuggability
    1. 01It automates the environment, not just the test.
    2. 02Infrastructure is provisioned programmatically.
    3. 03Test data comes from controlled realistic artifacts.
    4. 04Repository population is configurable.
    5. 05Resource compatibility is validated before provisioning.
    6. 06Load is dynamically adjusted toward a target utilization.
    7. 07Load generation and scanning are coordinated.
    8. 08Long-running execution is monitored with AI assistance.
    9. 09Build artifacts preserve evidence.
    10. 10Environments can remain available for investigation.

    15Technologies

    What each tool did here

    No proficiency bars — just the role each one played.

    Jenkins

    The orchestration layer that sequences and gates every stage.

    This case study intentionally abstracts product names, internal APIs, implementation details and proprietary business logic. The architecture and engineering methodology are presented to demonstrate problem-solving and system-design thinking without exposing confidential information.

    The core idea

    I transformed a multi-system manual validation process into an automated, configurable and observable engineering system.

    • Automation reduced repetitive manual orchestration.
    • Infrastructure became reproducible.
    • Test data became configurable and realistic.
    • Load became controllable.
    • Long-running execution became observable.
    • AI reduced the need for continuous human monitoring.
    • The environment remained available when investigation was required.
    All work