orka.memory_logger module

Memory Logger

The Memory Logger is a critical component of the OrKa framework that provides persistent storage and retrieval capabilities for orchestration events, agent outputs, and system state. It serves as both a runtime memory system and an audit trail for agent workflows.

Modular Architecture

The memory logger features a modular architecture with focused components while maintaining 100% backward compatibility through factory functions.

Key Features

Event Logging

Records all agent activities and system events with detailed metadata

Data Persistence

Stores data in Redis streams or Kafka topics for reliability and durability

Serialization

Handles conversion of complex Python objects to JSON-serializable formats with intelligent blob deduplication

Error Resilience

Implements fallback mechanisms for handling serialization errors gracefully

Querying

Provides methods to retrieve recent events and specific data points efficiently

File Export

Supports exporting memory logs to files for analysis and backup

Multiple Backends

Supports both Redis and Kafka backends with seamless switching

Core Use Cases

The Memory Logger is essential for:

  • Enabling agents to access past context and outputs

  • Debugging and auditing agent workflows

  • Maintaining state across distributed components

  • Supporting complex workflow patterns like fork/join

  • Providing audit trails for compliance and analysis

Modular Components

The memory system is composed of specialized modules:

BaseMemoryLogger

Abstract base class defining the memory logger interface

RedisMemoryLogger

Complete Redis backend implementation with streams and data structures

KafkaMemoryLogger

Kafka-based event streaming implementation

serialization

JSON sanitization and memory processing utilities

file_operations

Save/load functionality and file I/O operations

compressor

Data compression utilities for efficient storage

Usage Examples

Factory Function (Recommended)

from orka.memory_logger import create_memory_logger

# Redis backend (default)
redis_memory = create_memory_logger("redis", redis_url="redis://localhost:6379")

# Kafka backend
kafka_memory = create_memory_logger("kafka", bootstrap_servers="localhost:9092")

Direct Instantiation

from orka.memory.redis_logger import RedisMemoryLogger
from orka.memory.kafka_logger import KafkaMemoryLogger

# Redis logger
redis_logger = RedisMemoryLogger(redis_url="redis://localhost:6379")

# Kafka logger
kafka_logger = KafkaMemoryLogger(bootstrap_servers="localhost:9092")

Environment-Based Configuration

import os
from orka.memory_logger import create_memory_logger

# Set backend via environment variable
os.environ["ORKA_MEMORY_BACKEND"] = "kafka"

# Logger will use Kafka automatically
memory = create_memory_logger()

Backend Comparison

Redis Backend

  • Best for: Development, single-node deployments, quick prototyping

  • Features: Fast in-memory operations, simple setup, full feature support

  • Limitations: Single point of failure, memory-bound storage

Kafka Backend

  • Best for: Production, distributed systems, high-throughput scenarios

  • Features: Persistent event log, horizontal scaling, fault tolerance

  • Limitations: More complex setup, higher resource usage

Implementation Notes

Backward Compatibility

All existing code using RedisMemoryLogger continues to work unchanged

Performance Optimizations

  • Blob deduplication reduces storage overhead

  • In-memory buffers provide fast access to recent events

  • Batch operations improve throughput

Error Handling

  • Robust sanitization handles non-serializable objects

  • Graceful degradation prevents workflow failures

  • Detailed error logging aids debugging

Thread Safety

All memory logger implementations are thread-safe for concurrent access

orka.memory_logger.create_memory_logger(backend: str = 'redisstack', redis_url: str | None = None, bootstrap_servers: str | None = None, topic_prefix: str = 'orka-memory', stream_key: str = 'orka:memory', debug_keep_previous_outputs: bool = False, decay_config: Dict[str, Any] | None = None, enable_hnsw: bool = True, vector_params: Dict[str, Any] | None = None, **kwargs) → BaseMemoryLogger[source]

Enhanced factory with RedisStack as primary backend.

Creates a memory logger instance based on the specified backend. Defaults to RedisStack for optimal performance with automatic fallback.

Parameters:

  • backend – Memory backend type (“redisstack”, “redis”, “kafka”)

  • redis_url – Redis connection URL

  • bootstrap_servers – Kafka bootstrap servers (for Kafka backend)

  • topic_prefix – Kafka topic prefix (for Kafka backend)

  • stream_key – Redis stream key for logging

  • debug_keep_previous_outputs – Whether to keep previous outputs in logs

  • decay_config – Memory decay configuration

  • enable_hnsw – Enable HNSW vector indexing (RedisStack only)

  • vector_params – HNSW configuration parameters

  • **kwargs – Additional parameters for backward compatibility

Returns:

Configured memory logger instance

Raises:

  • ImportError – If required dependencies are not available

  • ConnectionError – If backend connection fails