CloudNexus Documentation

Overview

Data collection in CloudNexus is powered by Data Agents, lightweight daemons that run alongside your workloads. Agents handle ingestion, transformation, and secure transmission to centralized storage and analytics services.

📦

Workloads
Containers, VMs, Services

→

🤖

Data Agent
Collection & Transform

→

🌐

CloudNexus Ingest
TLS 1.3, mTLS Auth

→

💾

Storage & Analytics
Logs, Metrics, Traces

Prerequisites

Active CloudNexus account with infrastructure-admin or higher role
CloudNexus CLI v2.4+ installed and authenticated
Network policies configured to allow outbound traffic on ports 443 and 9090
Target workloads deployed and running

Step 1: Enable Data Agents

Deploy the Data Agent DaemonSet across your cluster or virtual machines using the CLI:

bash

cloudnexus agent deploy \
  --cluster prod-eu-west-1 \
  --mode daemonset \
  --resources cpu=200m,memory=256Mi \
  --tls auto \
  --verify

ℹ️ Note

The --tls auto flag automatically provisions and rotates mTLS certificates using CloudNexus's internal PKI. Certificates are valid for 90 days and renew 7 days before expiration.

Step 2: Configure Collection Endpoints

Define what data the agent should collect. Agents support structured logs, Prometheus-style metrics, OpenTelemetry traces, and custom event streams.

yaml

apiVersion: data.cloudnexus.io/v1
kind: DataCollection
metadata:
  name: app-metrics-and-logs
  namespace: monitoring
spec:
  sources:
    - type: prometheus
      scrapeInterval: 15s
      targets: ["*:9090"]
    - type: container-logs
      paths: ["/var/log/containers/*.log"]
      format: json
  buffer:
    size: 1000
    flushInterval: 5s \
    retryPolicy: exponential-backoff

Collection Parameters

Parameter	Type	Description	Status
`scrapeInterval`	`string`	Frequency for metric collection (min: 5s)	Required
`paths`	`string[]`	Filesystem paths or glob patterns for logs	Required
`buffer.size`	`integer`	Maximum events to hold in memory before flushing	Default: 500
`format`	`enum`	Log parsing format: `json`, `regex`, `raw`	Optional

Step 3: Define Routing Rules

Route collected data to appropriate destinations based on environment, severity, or application tags. CloudNexus uses a declarative routing engine.

json

{
  "routing": {
    "rules": [
      {
        "name": "prod-alerts",
        "match": {
          "environment": "production",
          "severity": { "gte": "warning" }
        },
        "destination": "alerting-channel",
        "priority": 1
      },
      {
        "name": "archive-logs",
        "match": {
          "source": "app-server"
        },
        "destination": "s3-compatible-bucket",
        "retention": "90d"
      }
    ]
  }
}

⚠️ Important

Routing rules are evaluated in priority order. Rules with lower priority values are processed first. If no rule matches, data falls back to the default ingestion pipeline.

Step 4: Verify & Monitor Ingestion

After configuration, verify that agents are healthy and data is flowing correctly.

bash

cloudnexus agent status --cluster prod-eu-west-1
cloudnexus ingest verify --endpoint app-metrics-and-logs
cloudnexus telemetry tail --filter "severity:error"

✅ Success Criteria

All agents report Healthy status
Ingestion latency < 200ms (p95)
Zero dropped events in last 5 minutes
Routing rules matching > 95% of expected traffic

Best Practices

Use structured logging: JSON-formatted logs enable faster parsing and richer query capabilities.
Implement sampling: For high-volume debug traces, enable probabilistic sampling to reduce storage costs.
Encrypt at rest: Enable AES-256 encryption for all archived logs and metrics.
Set retention policies: Auto-delete or archive data older than your compliance window to optimize storage tiers.
Monitor agent resources: Keep CPU < 15% and Memory < 60% to prevent collection bottlenecks.

Troubleshooting

Issue	Diagnosis Command	Resolution
Agent stuck in `Pending`	`kubectl describe pod -n monitoring`	Check node resources & taints. Scale cluster if needed.
High ingestion latency	`cloudnexus ingest latency --percentile 99`	Reduce buffer size, increase network bandwidth, or add ingest replicas.
SSL/TLS handshake failure	`cloudnexus agent certs verify`	Rotate certificates or check system clock sync (NTP).