How to discover APIs in Kubernetes and runtime environments

Modern cloud-native applications are built from dozens or hundreds of loosely coupled services. Each service exposes APIs to other services, creating a dense network of internal communications that may never leave the cluster. While organizations often focus on public-facing APIs, a significant portion of application functionality now exists behind the user interface in service-to-service interactions.

This creates a visibility challenge that is fundamentally different from traditional API discovery.

A security team may have complete visibility into APIs exposed through ingress controllers and API gateways while remaining unaware of APIs that exist only inside the cluster. These internal APIs may process sensitive business data, enforce authorization decisions, manage user state, or expose administrative functionality. Yet they often operate entirely outside the visibility of perimeter-focused security tools.

The problem becomes even harder when Kubernetes’ dynamic nature is added to the equation. Pods scale horizontally, workloads are replaced during deployments, canary releases introduce temporary endpoint variants, and short-lived jobs may exist only for minutes. APIs appear, evolve, and disappear continuously.

As a result, Kubernetes doesn’t just create API sprawl. It creates API visibility gaps that traditional discovery methods were never designed to address.

Runtime API discovery emerged specifically to solve this problem. Rather than relying on documentation or predefined inventories, runtime discovery observes application behavior directly as it occurs inside the cluster. Technologies such as eBPF and service mesh telemetry allow security teams to observe real API communications, identify undocumented services, detect drift between specifications and production behavior, and build a continuously updated picture of the actual API attack surface.

The challenge then becomes turning that visibility into security assurance. Discovering an API is valuable, but security teams still need to know whether that API can be exploited. That is where runtime discovery and dynamic application security testing (DAST) become complementary rather than competing capabilities.

Runtime API discovery in Kubernetes

Runtime API discovery in Kubernetes is the process of identifying API endpoints by observing actual application behavior in running container environments rather than relying on documentation, gateway exports, or source code analysis.

Runtime discovery uses technologies such as eBPF (Extended Berkeley Packet Filter) and service mesh telemetry to observe API communications at the kernel or proxy layer, capturing east-west microservice traffic, ephemeral endpoint activity, and undocumented APIs that other discovery methods may miss.

Why traditional API discovery fails in Kubernetes

Traditional API discovery approaches were designed for relatively stable application environments. Kubernetes introduces architectural characteristics that create visibility challenges these methods were never designed to solve.

The east-west traffic blind spot

In traditional web architectures, most API traffic follows a north-south path:

Client → Load balancer → API gateway → Application

Security and monitoring tools positioned at any point in this path can observe a large percentage of API activity.

Kubernetes changes this model. Microservices communicate extensively with one another through internal service-to-service APIs. These east-west communications occur entirely inside the cluster and often never pass through external gateways or ingress controllers.

This creates a significant visibility gap. Security teams may have complete visibility into public-facing APIs while remaining unaware of a large portion of the internal API attack surface. If an attacker compromises a workload, they may be able to interact with internal APIs that never appear in gateway logs or perimeter monitoring systems.

The ephemeral workload problem

Traditional API discovery assumes stability. Endpoints exist at known locations and remain available long enough to be cataloged and assessed.

Kubernetes workloads often violate these assumptions.

Examples include:

• Auto-scaled workloads that appear and disappear based on demand
• Canary deployments that temporarily expose new API variants
• Scheduled jobs that exist only during execution windows
• Temporary debugging or maintenance containers

Point-in-time API discovery methods produce snapshots. If an API only exists for a short period, there is a good chance it will be absent when the snapshot is taken.

This means organizations can accumulate blind spots even when discovery processes appear comprehensive.

The multi-cluster and multi-namespace complexity

Large Kubernetes environments rarely consist of a single cluster.

Organizations commonly operate:

• Development clusters
• Testing environments
• Staging environments
• Multiple production clusters
• Regional deployments

Each environment may contain different services, namespaces, networking configurations, and service mesh implementations.

Visibility becomes even more complicated when APIs are exposed across cluster boundaries through federation, shared service meshes, or custom networking configurations. Discovery tools scoped to a single cluster may miss APIs that are accessible elsewhere in the environment.

Why gateways don’t solve this

API gateways play an important role in API management and security. They help organizations manage authentication, routing, rate limiting, and traffic control for external APIs.

However, gateways primarily focus on north-south traffic.

While it is technically possible to force internal service communications through a gateway, most organizations avoid doing so because it introduces additional complexity, latency, and operational overhead.

As a result, many internal APIs never traverse a gateway. An organization may have excellent visibility into public APIs while remaining largely blind to a substantial percentage of actual API activity occurring inside the cluster.

eBPF: How kernel-level monitoring solves the Kubernetes visibility problem

Runtime discovery has emerged as one of the most effective approaches for identifying APIs in Kubernetes environments. Among runtime technologies, eBPF has become especially important because it provides visibility at the kernel level.

What is eBPF and why does it work in Kubernetes?

eBPF, or Extended Berkeley Packet Filter, is a Linux kernel technology that allows sandboxed programs to execute in response to kernel-level events.

These programs can observe:

• Network activity
• System calls
• Process execution
• File operations
• Kernel functions

Because eBPF operates at the kernel level, it can observe communications regardless of application language, framework, or architecture.

For Kubernetes API discovery, this provides several important advantages.

eBPF can observe:

• HTTP and HTTPS requests between pods
• Service-to-service communication patterns
• Newly appearing endpoints
• Traffic associated with ephemeral workloads
• API interactions that never reach external gateways

This creates a more complete picture of the runtime API surface than perimeter-focused discovery approaches.

How eBPF-based API discovery works in practice

Many discussions of eBPF stop at "it observes traffic at the kernel level." While technically true, that description doesn’t explain how raw packet observations become a usable API inventory.

The real challenge is not collecting traffic. It is transforming runtime observations into something security teams can actually use.

Collecting runtime observations

In a Kubernetes deployment, eBPF agents are typically deployed as a DaemonSet so that every node in the cluster runs a monitoring component. The agent attaches to network- and process-related kernel events and observes communications generated by containers running on that node.

At this stage, the data is still relatively low level. The agent may see network connections, HTTP requests, TLS sessions, process metadata, and service relationships, but none of that automatically produces a clean API inventory.

Enriching API telemetry

This is where enrichment becomes critical. As traffic is observed, additional context is collected from multiple sources:

• Kubernetes metadata such as namespaces, services, deployments, and labels
• DNS information identifying service names rather than raw IP addresses
• TLS session information that helps associate encrypted traffic with specific applications
• Process ancestry that identifies which workloads initiated communications
• Existing API specifications where available

The goal is to answer questions such as:

• Which service owns this endpoint?
• Which workload is consuming it?
• Is this endpoint already documented?
• Is this traffic associated with an existing API definition or something new?

Reconstructing scan-ready API definitions

Once enough observations have been collected, the discovery platform can begin reconstructing API behavior.

Rather than simply storing raw requests, the platform infers endpoint structures, request methods, parameters, authentication patterns, and response characteristics. These observations are assembled into machine-readable API definitions, typically represented as OpenAPI schemas.

This reconstruction process is particularly important for undocumented and internally developed APIs. Security teams rarely receive complete specifications for these services, yet runtime traffic often provides enough evidence to generate practical, scan-ready definitions.

The final result is not merely a list of observed requests. It is a continuously updated inventory of candidate API assets that can immediately feed downstream security processes.

Turning discovery into validation

The distinction matters because discovery is only useful if something actionable happens next. Many runtime discovery products stop at inventory generation, leaving security teams with another inventory to maintain and investigate manually.

Invicti’s approach extends beyond discovery. Runtime observations are enriched with Kubernetes and traffic context, reconstructed into scan-ready API definitions, and automatically fed into security testing workflows. Rather than requiring teams to manually document, normalize, and onboard newly discovered APIs for testing, the platform turns reconstructed APIs into candidate scan targets that can move directly into validation workflows.

The goal is not simply to identify APIs, but to determine whether those APIs introduce risk.

In practical terms:

• eBPF provides the observations
• Enrichment provides context
• Reconstruction produces usable API definitions
• DAST validates whether those APIs are actually vulnerable

eBPF tools and frameworks in the Kubernetes ecosystem

Several well-known Kubernetes technologies already leverage eBPF.

Cilium and Hubble

Cilium is one of the most widely adopted eBPF-powered networking platforms for Kubernetes.

Its Hubble observability layer provides visibility into:

• Network flows
• DNS activity
• Service communications
• HTTP requests

Organizations already using Cilium often have a strong foundation for runtime visibility.

Falco and Tetragon

Falco and Tetragon use eBPF for runtime security monitoring and threat detection.

These tools focus on identifying suspicious behaviors, policy violations, and security-relevant events occurring within Kubernetes workloads.

Runtime discovery platforms

A growing number of runtime discovery platforms use eBPF to identify APIs and generate schemas from observed traffic.

These tools can provide valuable visibility into API activity, but many stop at inventory generation. Discovery identifies APIs, but discovery alone does not determine whether those APIs are vulnerable.

Invicti extends the workflow by connecting runtime discovery directly to DAST-based security testing, allowing discovered APIs to be automatically assessed for exploitable vulnerabilities.

Service mesh telemetry as an API discovery source

Organizations that have already invested in service meshes often have another valuable source of runtime visibility.

Service meshes deploy sidecar proxies alongside application workloads and intercept service-to-service communications automatically.

This architecture generates detailed telemetry as a byproduct of normal operation.

Service mesh telemetry can reveal:

• Source and destination services
• API paths
• HTTP methods
• Status codes
• Request latency
• Service dependency relationships

This information can be used to build API inventories and service maps without requiring application code changes.

Istio

Istio provides rich telemetry through components such as:

• Prometheus metrics
• Distributed tracing
• Kiali service visualization
• Envoy access logs

These capabilities make Istio a powerful source of API discovery data.

Linkerd

Linkerd provides a lighter-weight approach to service mesh visibility.

While its telemetry capabilities are generally less extensive than Istio’s, many organizations prefer its lower operational complexity.

Envoy

Envoy serves as the proxy foundation for many service mesh deployments.

Its access logs often provide the raw telemetry used to reconstruct API inventories and service relationships.

Service mesh limitations

Service meshes provide an appealing source of API visibility because they already sit in the communication path between services. If every workload is enrolled in the mesh, every request passing through the sidecar proxies can be observed, logged, and analyzed.

In practice, however, few environments achieve complete mesh coverage.

Many organizations adopt service meshes incrementally. New applications may be onboarded first while legacy workloads remain outside the mesh. Third-party software, operational tooling, stateful services, and specialized workloads are frequently excluded because of compatibility concerns, performance considerations, or operational complexity. As a result, the service mesh becomes a visibility boundary rather than a complete visibility solution.

This creates an important distinction between service mesh telemetry and eBPF-based discovery.

A service mesh observes traffic that flows through sidecar proxies. eBPF observes traffic generated by the underlying workloads themselves. If a workload is not enrolled in the mesh, its API activity disappears from mesh telemetry but remains visible through kernel-level monitoring.

Service mesh telemetry excels at showing relationships between managed services, while eBPF provides broader visibility across workloads regardless of mesh participation. 

For this reason, mature Kubernetes environments increasingly treat service mesh telemetry and eBPF as complementary discovery sources rather than competing approaches. The mesh provides rich visibility into managed service interactions, while eBPF closes coverage gaps and captures runtime activity that would otherwise remain invisible.

Kubernetes-specific discovery challenges

Runtime discovery provides powerful visibility, but Kubernetes environments introduce additional complexities that security teams need to understand.

Encrypted east-west traffic (mTLS)

As organizations mature their Kubernetes security programs, many adopt service meshes that enforce mutual TLS (mTLS) between services.

From a security perspective, this is a positive development. mTLS helps ensure that service-to-service communications are encrypted and authenticated, reducing the risk of traffic interception and unauthorized service impersonation.

From a visibility perspective, however, encryption introduces challenges.

Traditional network monitoring approaches often rely on observing packets as they traverse the network. Once traffic is encrypted, these tools can still see connections occurring, but they lose visibility into the API requests themselves. Endpoint paths, request methods, parameters, and response content become inaccessible without decryption.

This limitation becomes particularly significant in Kubernetes environments where a growing percentage of API communications occur inside encrypted service mesh channels.

eBPF-based monitoring addresses this challenge because it operates much closer to the application itself.

Rather than relying solely on network-level packet inspection, modern eBPF-based approaches can, depending on implementation, observe relevant application and TLS library activity before encryption occurs or after decryption takes place. Some implementations achieve this by instrumenting TLS libraries in user space, allowing API-level activity to remain visible even when traffic is encrypted in transit.

The practical result is that organizations do not have to choose between encryption and visibility.

They can enforce mTLS throughout the cluster while still maintaining the visibility needed to discover APIs, reconstruct their behavior, and generate scan-ready definitions for security testing.

This ability to preserve visibility without weakening encryption is one of the reasons runtime monitoring has become an important source of API discovery data in Kubernetes environments.

This capability becomes increasingly important as service meshes such as Istio and Linkerd become standard components of enterprise Kubernetes architectures.

Ephemeral pod API discovery timing

Short-lived workloads can create incomplete visibility.

A pod may process only a subset of its available endpoints before terminating. Runtime observation alone may therefore capture only part of the overall API definition.

Correlating runtime observations with code repository data helps improve completeness by combining observed behavior with documented implementation details.

Kubernetes API server traffic versus application APIs

Clusters generate substantial internal traffic associated with Kubernetes operations.

Examples include:

• Health checks
• Controller activity
• Metrics collection
• Operator workflows

Effective API discovery requires separating infrastructure communications from application APIs to avoid inventory pollution and unnecessary testing.

Multi-cluster visibility

Organizations operating multiple clusters face additional challenges.

Comprehensive visibility often requires:

• Runtime monitoring in every cluster
• Centralized correlation
• Consistent inventory management
• Cross-cluster deduplication

Without this broader perspective, APIs exposed across cluster boundaries can remain undiscovered.

Connecting runtime discovery to DAST security testing

Runtime API discovery solves an important visibility problem. It tells security teams what APIs exist, where they run, which services communicate with them, and which endpoints are actively receiving traffic.

But visibility is not the same as security assurance.

A runtime discovery platform can identify an endpoint, reconstruct its schema, and document its behavior. What it cannot determine is whether that API can actually be exploited. The API risks that matter most in Kubernetes environments require active testing, authenticated workflows, and validation of application behavior under real-world conditions.

This is where DAST becomes the essential next step in the discovery process.

BOLA (Broken Object Level Authorization)

Broken Object Level Authorization (BOLA) remains the most common and most damaging API vulnerability category in the OWASP API Security Top 10.

Runtime discovery can identify that an endpoint exists:

GET /orders/{orderId}

It can observe which services call the endpoint, what authentication mechanisms are used, and how frequently it appears in traffic.

What it cannot determine is whether a user can substitute another customer’s order ID and retrieve data they should never be authorized to access.

That question can only be answered through active testing.

DAST evaluates the API’s actual authorization behavior by making authenticated requests and validating responses. Instead of simply confirming that an endpoint exists, it determines whether access controls are functioning correctly under real operating conditions.

This distinction is particularly important in Kubernetes environments, where authorization decisions are often distributed across dozens of independently developed microservices. A single missing authorization check in one service can expose sensitive data even when the rest of the environment is properly secured.

BFLA (Broken Function Level Authorization)

Microservices architectures often introduce complex role-based access patterns.

Administrative functions, operational controls, internal workflows, and business processes may be distributed across numerous APIs owned by different teams. As services evolve independently, authorization models can become inconsistent.

Consider an endpoint such as:

POST /admin/refunds/approve

Runtime discovery can identify the endpoint and observe who appears to be calling it.

What it cannot determine is whether a lower-privileged user can successfully invoke that administrative function.

That requires authenticated testing using multiple user roles and permission levels.

DAST validates whether function-level authorization controls are enforced correctly and whether users can access actions beyond their intended privileges. In large Kubernetes environments, where hundreds of APIs may participate in a single business process, this type of validation becomes increasingly important.

Exposed secrets and sensitive data in API responses

Discovery platforms observe requests and responses flowing through the environment, but they are not designed to evaluate whether the data being returned is appropriate.

An API may expose:

• Access tokens
• API keys
• Session identifiers
• Internal system information
• Personally identifiable information (PII)
• Sensitive business data

without appearing unusual from a traffic-analysis perspective.

For example, a response may contain debugging information, internal object references, or credentials that developers never intended to expose externally. Runtime discovery can document the endpoint that returned the information, but determining whether the response creates a security risk requires active inspection and analysis.

DAST evaluates actual responses and identifies sensitive data exposures that would otherwise remain hidden inside a seemingly normal API workflow.

Business logic flaws

Some of the most serious API vulnerabilities do not originate from individual endpoints at all.

They emerge from the interaction between multiple APIs and business processes.

Examples include:

• Circumventing approval workflows
• Manipulating transaction sequences
• Bypassing business constraints
• Performing actions in an unintended order
• Combining otherwise legitimate API calls to achieve unauthorized outcomes

These vulnerabilities cannot be identified through inventory analysis or passive traffic observation because every individual request may appear legitimate.

Detecting them requires stateful, context-aware testing of complete workflows.

This is especially relevant in Kubernetes environments, where business processes are often distributed across many independent services communicating through APIs. A flaw may not exist within any individual microservice but instead emerge from the way multiple services interact.

From runtime discovery to continuous validation

These examples highlight the fundamental difference between API inventory and API security assurance.

Runtime discovery identifies what exists. DAST determines whether those APIs can be exploited.

This is why the most effective Kubernetes API security programs treat discovery and testing as complementary capabilities rather than separate initiatives. Visibility alone creates inventory. Continuous validation turns that inventory into actionable security intelligence.

Within the Invicti platform, runtime API discovery is designed as part of a continuous validation workflow rather than a standalone inventory process. Discovered APIs are reconstructed, prepared for testing, and fed directly into DAST assessment pipelines:

1. Kubernetes monitoring observes API activity across the environment.
2. Observed endpoints are enriched with contextual information from Kubernetes metadata, traffic observations, and available specifications.
3. API behavior is reconstructed into scan-ready schemas, allowing newly discovered APIs to be onboarded for testing without requiring manually maintained specifications.
4. Newly discovered APIs are automatically queued for DAST testing.
5. DAST evaluates authorization controls, authentication mechanisms, data exposure risks, and business logic vulnerabilities.
6. Where supported, proof-based scanning safely validates exploitability, helping teams distinguish confirmed vulnerabilities from theoretical findings and prioritize remediation more effectively.
7. Confirmed findings flow into the Invicti Application Security Platform for prioritization, remediation tracking, and reporting.

The result is not simply a continuously updated Kubernetes API inventory. It is a continuous discovery-to-validation pipeline that helps security teams understand both what APIs exist and which of those APIs introduce real risk.

Discovery answers the visibility question, while DAST answers the security question.

Deployment considerations and prerequisites

Successful runtime API discovery requires planning and cross-functional collaboration.

Kernel version requirements

Basic eBPF functionality generally requires Linux kernel 4.4 or newer.

Advanced capabilities commonly used for API discovery and protocol inspection typically require kernel 5.2 or newer with BPF CO-RE support.

Most modern EKS, AKS, and GKE deployments meet these requirements, but validation should be performed before deployment.

Container runtime compatibility

Organizations should verify compatibility with:

• containerd
• CRI-O
• Other supported runtimes

and confirm required kernel modules are available. As with any runtime monitoring technology, organizations should also evaluate resource consumption and operational impact within their own environments before large-scale deployment.

Privileged access requirements

eBPF agents require elevated privileges to access kernel-level telemetry. 

Because these agents operate close to the kernel and often require elevated permissions, deployment typically involves review by both platform engineering and security teams. Organizations should understand the operational and security implications before broad rollout.

Organizations using strict admission controls may also need policy exceptions for monitoring DaemonSets.

Cross-functional deployment requirements

Runtime discovery is not typically a security-only project. Successful deployment often requires participation from:

• Platform teams
• Infrastructure teams
• Security teams
• Networking teams

Aligning stakeholders early helps avoid delays during rollout.

Privacy and data handling

Organizations should review what information is collected and retained.

Some implementations can capture request and response content that may contain regulated or sensitive information.

Data handling requirements should be evaluated before production deployment.

A practical deployment sequence for Kubernetes API discovery

Runtime discovery is most successful when introduced as an incremental visibility program rather than a single tooling deployment.

Organizations that attempt to discover and secure every API at once often become overwhelmed by the volume of newly uncovered services and endpoints. A phased approach allows teams to establish confidence in discovery data before expanding coverage.

Step 1: Assess your current visibility baseline

Before introducing runtime monitoring, document what your existing discovery processes already provide.

Export inventories from API gateways, ingress controllers, developer portals, service catalogs, and existing API management platforms. Review OpenAPI specifications and compare them with known production services.

The goal is not a perfect inventory but a baseline against which runtime discovery results can be compared.

Many organizations are surprised by how many APIs appear once runtime visibility is enabled.

Step 2: Verify eBPF compatibility

Next, validate platform readiness.

Review kernel versions across node pools, identify systems that may lack required eBPF capabilities, and confirm container runtime compatibility. Also review security policies that could affect privileged DaemonSet deployment.

This stage should include platform engineering teams responsible for cluster operations. Early alignment between infrastructure and security teams helps avoid deployment delays later.

Step 3: Deploy runtime monitoring

Begin with a limited deployment rather than an immediate cluster-wide rollout.

A non-production namespace or lower-risk cluster allows teams to validate agent behavior, assess resource consumption, and confirm discovery accuracy.

Compare observed APIs against existing inventories. Endpoints discovered through runtime monitoring but absent from gateway exports or specifications should be investigated.

Some will prove to be legitimate shadow APIs, while others may reveal documentation gaps or previously unknown dependencies.

Step 4: Correlate runtime and code-based discovery

No single discovery method provides complete visibility.

Runtime discovery shows what is active, while repository analysis shows what developers intended to deploy. Comparing these perspectives often produces valuable findings.

Endpoints present in source code but absent from runtime observations may indicate deprecated functionality or potential zombie APIs.

Endpoints observed at runtime but missing from repository-derived specifications may indicate undocumented services, deployment drift, or APIs introduced through third-party components.

The goal is not inventory growth but inventory accuracy.

Step 5: Feed discovered APIs into DAST

The final step is transforming visibility into security validation.

Newly discovered APIs should enter testing workflows automatically rather than waiting for manual review.

For Kubernetes environments, this often means configuring authenticated scan profiles that use service credentials, JWTs, or other authorization mechanisms representative of real application behavior.

Particular attention should be paid to authorization-related findings, especially BOLA and BFLA vulnerabilities, which remain among the most common and highest-impact API security risks in distributed microservices environments.

At this point, runtime discovery becomes part of a continuous feedback loop:

• New APIs are discovered.
• New APIs are tested.
• Validated vulnerabilities are prioritized.
• The inventory remains current while security posture is continuously assessed. 

Conclusion: Closing the Kubernetes API visibility gap, from discovery to security testing

Kubernetes has made API discovery a runtime problem rather than a documentation problem. Organizations that rely solely on gateways, specifications, or periodic inventories will inevitably miss parts of their API attack surface. The challenge is not simply discovering more APIs, but ensuring newly discovered APIs do not become untested APIs.

Kubernetes creates API surface at a speed and scale that traditional discovery tools cannot reliably track, and runtime discovery alone does not make those APIs secure. Kubernetes monitoring as part of Invicti’s API discovery discovers east-west APIs through runtime observation, generates scan-ready schemas automatically, and feeds them directly into continuous DAST testing so your microservices attack surface is continuously discovered and continuously validated.

Read more about setting up runtime API discovery with Invicti and then request a demo to see how Invicti maps your cluster’s internal API surface and feeds it directly into continuous security testing – including APIs your gateway has never seen.