K2I Troubleshooting Guide

This guide covers common issues and their solutions when running K2I.

Related deep dives:

• Schema Registry Protobuf for schema pause and breaking-change behavior.
• Iceberg REST Catalog for REST commit flow and backend caveats.
• DuckDB Iceberg Validation for local Parquet and iceberg_scan checks.
• Production Readiness for current hardening follow-ups.

Diagnostic Commands

Check Health Status

# Via CLI
k2i status --url http://localhost:8080

# Via curl
curl -s http://localhost:8080/health | jq .

# Kubernetes
kubectl exec -it deployment/k2i -n k2i -- curl -s localhost:8080/health

Check Logs

# Systemd
journalctl -u k2i -f

# Docker
docker logs -f k2i

# Kubernetes
kubectl logs -f deployment/k2i -n k2i

Enable Debug Logging

# Via CLI flag
k2i ingest -v      # debug level
k2i ingest -vv     # trace level

# Via environment variable
RUST_LOG=k2i=debug k2i ingest
RUST_LOG=k2i=trace,rdkafka=debug k2i ingest

Check Prometheus Metrics

curl -s http://localhost:9090/metrics | grep k2i

Common Issues

Startup Issues

Config File Not Found

Symptom:

Error: No such file or directory (os error 2)

Cause: Configuration file doesn't exist at the specified path.

Solution:

# Check file exists
ls -la config.toml

# Specify full path
k2i ingest --config /absolute/path/to/config.toml

Invalid TOML Syntax

Symptom:

Error: TOML parse error at line 15, column 3

Cause: Syntax error in configuration file.

Solution:

# Validate config
k2i validate --config config.toml

# Check TOML syntax
cat config.toml | toml-cli lint

Missing Required Fields

Symptom:

Error: missing field `topic` at line 1 column 1

Cause: Required configuration field is not set.

Solution: Add the missing field to your config file. Required fields:

• kafka.bootstrap_servers
• kafka.topic
• kafka.consumer_group
• iceberg.catalog_type
• iceberg.warehouse_path
• iceberg.database_name
• iceberg.table_name

Kafka Connection Issues

Cannot Connect to Brokers

Symptom:

Error: Kafka connection failed: broker not available

Causes:

1. Brokers are not running
2. Network connectivity issue
3. Wrong bootstrap servers

Solutions:

# Test connectivity
nc -zv kafka-broker 9092

# Check broker status
kafka-broker-api-versions --bootstrap-server localhost:9092

# Verify config
cat config.toml | grep bootstrap_servers

SASL Authentication Failed

Symptom:

Error: SASL authentication failed: Invalid credentials

Causes:

1. Wrong username/password
2. Wrong SASL mechanism
3. Missing security configuration

Solution:

[kafka.security]
protocol = "SASL_SSL"
sasl_mechanism = "SCRAM-SHA-256"  # or SCRAM-SHA-512, PLAIN
sasl_username = "correct-username"
sasl_password = "correct-password"

SSL Certificate Errors

Symptom:

Error: SSL handshake failed: certificate verify failed

Solutions:

[kafka.security]
protocol = "SASL_SSL"
ssl_ca_location = "/path/to/ca-cert.pem"

# For self-signed certs (development only)
# ssl_endpoint_identification_algorithm = ""

Consumer Group Rebalance Loops

Symptom:

WARN Consumer rebalance triggered
WARN Consumer rebalance triggered
WARN Consumer rebalance triggered

Cause: max_poll_interval_ms too short for flush time.

Solution:

[kafka]
max_poll_interval_ms = 600000  # 10 minutes

Buffer Issues

Buffer Full / Backpressure

Symptom:

WARN Backpressure engaged: buffer full

Causes:

1. Iceberg writes too slow
2. Buffer too small for throughput
3. Network issues to object storage

Solutions:

[buffer]
# Increase buffer size
max_size_mb = 1000

# Decrease flush interval (write more often)
flush_interval_seconds = 15

# Decrease batch size (smaller files, faster writes)
flush_batch_size = 5000

Out of Memory

Symptom:

Error: memory allocation failed

Causes:

1. Buffer size exceeds available memory
2. Memory leak (report as bug)

Solutions:

[buffer]
# Reduce buffer size to fit in available memory
max_size_mb = 500

# Increase container/process memory limit
docker run -m 4g ...

# Kubernetes
resources:
  limits:
    memory: 4Gi

Iceberg Issues

Catalog Connection Failed

Symptom:

Error: Failed to connect to Iceberg catalog

Causes:

1. Catalog server not running
2. Wrong URI
3. Network issue

Solutions:

# Test REST catalog
curl -s http://localhost:8181/v1/config

# Check Glue access
aws glue get-databases --region us-east-1

# Verify config
cat config.toml | grep -A5 "\[iceberg\]"

Table Not Found

Symptom:

Error: Table not found: analytics.events

Cause: Table doesn't exist in catalog.

Solution:

# Create table via Spark/DuckDB/Trino, or use K2I table commands where appropriate
spark-sql -e "CREATE TABLE analytics.events ..."

# K2I table command help
k2i table --help

# Or use the catalog REST API directly
curl -X POST http://localhost:8181/v1/namespaces/analytics/tables ...

CAS Conflict (Concurrent Writes)

Symptom:

Error: CAS conflict: expected snapshot 123, actual 124

Cause: Another process modified the table.

Solution: K2I automatically retries CAS conflicts. If persistent:

1. Ensure only one K2I instance per table
2. Check for other Iceberg writers

S3 Permission Denied

Symptom:

Error: Access Denied (Service: S3)

Solutions:

# Test S3 access
aws s3 ls s3://your-bucket/warehouse/

# Check IAM permissions
aws sts get-caller-identity

# Verify credentials in config
cat config.toml | grep aws_

Required S3 permissions:

• s3:GetObject
• s3:PutObject
• s3:DeleteObject
• s3:ListBucket

Parquet Write Errors

Symptom:

Error: Failed to write Parquet file

Causes:

1. Schema mismatch
2. Invalid data types
3. Storage full

Solutions:

# Check available storage
aws s3api head-bucket --bucket your-bucket

# Enable trace logging for details
RUST_LOG=k2i::iceberg=trace k2i ingest

Transaction Log Issues

Log Corruption

Symptom:

Error: Transaction log corrupted at position 12345

Cause: Incomplete write, disk failure, or bug.

Solutions:

# Backup corrupted log
cp -r transaction_logs transaction_logs.backup

# Clear log (will replay from Kafka)
rm -rf transaction_logs/*

# Restart (will re-consume from last Kafka offset)
k2i ingest

Checksum Mismatch

Symptom:

Error: Checksum mismatch at entry 5678

Cause: Data corruption in log file.

Solution: Same as log corruption - backup and clear.

Recovery Failed

Symptom:

Error: Failed to recover from transaction log

Solutions:

1. Check disk space
2. Check file permissions
3. Clear transaction log if corrupt
4. Check Iceberg table state matches expected

Performance Issues

High Consumer Lag

Symptom: k2i_kafka_consumer_lag increasing over time.

Causes:

1. Processing too slow
2. Flush taking too long
3. Insufficient resources

Solutions:

[kafka]
# Increase batch size for higher throughput
batch_size = 5000

[buffer]
# Larger flushes are more efficient
flush_batch_size = 50000

[iceberg]
# Faster compression
compression = "lz4"  # instead of zstd

Slow Flush Times

Symptom: k2i_flush_duration_seconds high (> 5s).

Causes:

1. Large batch sizes
2. Slow object storage
3. Network latency

Solutions:

[buffer]
# Smaller, more frequent flushes
flush_batch_size = 10000
flush_interval_seconds = 15

[iceberg]
# Faster compression
compression = "snappy"  # fastest

# Smaller files
target_file_size_mb = 128

High Memory Usage

Symptom: Container/process memory growing.

Solutions:

[buffer]
# Limit buffer size
max_size_mb = 500

# Shorter TTL evicts faster
ttl_seconds = 30

Health Check Issues

Degraded Status

Symptom:

{"status": "degraded", "components": {"catalog": {"status": "degraded"}}}

Cause: Non-critical component experiencing issues but still operational.

Action: Check component logs for warnings, monitor closely.

Readiness Blocked by Schema Evolution

Symptom:

curl -i http://localhost:8080/healthz
# HTTP/1.1 200 OK

curl -i http://localhost:8080/readyz
# HTTP/1.1 503 Service Unavailable

/health shows the schema component as degraded, often with a message about a breaking Protobuf change or schema_evolution.mode=manual.

Cause: K2I has paused ingestion before assigning read LSNs or committing Kafka offsets for incompatible data. This protects the table from type changes, removed fields, required-field additions, or manual-mode additive changes that have not been approved.

Action:

1. Inspect K2I logs for the schema ID and field-level reason.
2. Check the Schema Registry subject configured by [kafka.format].
3. Either roll back or fix the producer schema, or apply the intended table schema migration.
4. Restart K2I after operator action. Do not force Kafka offsets past incompatible records unless a separate recovery plan accounts for the skipped data.

Unhealthy Status

Symptom:

{"status": "unhealthy", "components": {"kafka": {"status": "unhealthy"}}}

Cause: Critical component failed.

Action: Check component-specific errors, restart may be needed.

Health Endpoint Not Responding

Causes:

1. K2I not running
2. Wrong port
3. Network issue

Solutions:

# Check process is running
ps aux | grep k2i
pgrep -f "k2i ingest"

# Check port binding
netstat -tlnp | grep 8080
ss -tlnp | grep 8080

# Check firewall
iptables -L -n | grep 8080

Exit Code Reference

Code	Meaning	Action
0	Success	None
1	Config Error	Fix configuration
2	Kafka Error	Check Kafka connectivity
3	Iceberg Error	Check catalog/storage
4	Storage Error	Check object storage access
5	TxLog Error	Check local disk
6	Health Error	Check all components
10	Runtime Error	Check logs for details
130	Signal Interrupt	Normal shutdown

Getting Help

Collecting Debug Information

When reporting issues, include:

1. Version:

   k2i --version

2. Configuration (redact secrets):

   cat config.toml | sed 's/password.*/password = "REDACTED"/g'

3. Health status:

   curl -s http://localhost:8080/health | jq .

4. Metrics:

   curl -s http://localhost:9090/metrics | grep k2i

5. Logs (last 100 lines with trace):

   RUST_LOG=trace k2i ingest 2>&1 | tail -100

6. Environment:

   uname -a
   rustc --version
   docker --version  # if applicable
   kubectl version   # if applicable

Reporting Issues

Report issues at: https://github.com/osodevops/k2i/issues

Include:

• Description of the problem
• Steps to reproduce
• Expected vs actual behavior
• Debug information (see above)

FAQ

Can I run multiple K2I instances for the same topic?

Treat one K2I process per configured topic and table as the supported first-release deployment pattern. Do not run multiple active writers to the same table until conflict handling, recovery, and multi-partition offset behavior have been validated for your workload.

How do I recover from a crash?

Restart K2I with the same durable transaction_log.log_dir and inspect /health, /readyz, logs, and downstream table visibility. Recovery state is recorded in the transaction log, but startup recovery application and orphan cleanup still need additional hardening before broad production rollout. See Production Readiness.

How do I change the Kafka topic without losing data?

1. Stop K2I gracefully (flushes buffer)
2. Update configuration
3. Start K2I - it will start from the new topic's beginning

How do I migrate to a new Iceberg table?

1. Stop K2I
2. Update database_name and table_name in config
3. Create the new table in your catalog
4. Start K2I

What happens if the buffer fills up?

K2I engages backpressure by slowing or pausing Kafka consumption until the buffer has space. Latency increases while downstream writes catch up.

How do I scale K2I?

K2I is single-process by design. Scale by:

1. Running separate instances for different topics
2. Partitioning your data across multiple topics
3. Increasing resources (CPU, memory) for a single instance