[heimdex people] If you had to build high availability in KT Connect within 24 hours, what would you trade off?| DevOps

Published at 2026년 8월 7일

Heimdex Infrastructure Engineering Notes — Live Demo Preparation Log

Hello, this is Heimdex. 👋

In July, KT hosted AX Tech Connect 2026 at the KT Umyeon R&D Center in Seocho, Seoul. Organized by KT’s AX Future Technology Institute, the event showcased their latest R&D achievements, including their proprietary AI model 'Mi:dm K', multimodal AI, and Hyper AI Networks.

Heimdex participated in this showcase as a reference application demonstrating KT’s ultra-low latency network. We conducted live demonstrations featuring real-time video search, facial matching, and multi-lingual (Korean, English, Japanese) video commentary generation.

What Heimdex Demonstrated

Natural Language Scene Search: Finding specific video segments using spoken or written queries like "A scene where a blue truck passes by."
Facial Matching: Aggregating and displaying only the video segments where a specific individual appears using facial recognition.
Trilingual Video Commentary Generation: Analyzing video scenes to generate descriptive commentary sequentially in Korean, English, and Japanese.
Real-time Processing Speed Comparison: Processing identical videos with the same AI models over standard public networks vs. KT’s ultra-low latency network, displaying side-by-side processing times on screen.

We did not play pre-rendered videos. Everything ran live directly in front of the attendees, with real-time processing metrics displayed on screen.

The Heimdex system behind this demo was designed and operated by a newly joined junior engineer. Here is a transcript of an interview with Jiseok Park from our engineering team, reflecting on his preparation process.

Note: This article covers the standalone server environment specifically configured for the live demo. It is separate from both our client enterprise deployment environment and the event venue's internal infrastructure.

Q1. Jiseok, please introduce yourself and your role in this project.

Hi, I'm Jiseok Park from the Heimdex engineering team. I handle infrastructure and full-stack development, primarily building and maintaining foundational systems like servers, databases, search engines, and deployment pipelines to keep our service stable.

Before joining Heimdex, I served as an ROTC officer. The tenacity and habit of conducting thorough rehearsals with a Plan B that I developed during military service heavily influence how I work today.

For this project, I was responsible for designing and operating the on-site Heimdex infrastructure—including the query processing servers, the AI video analysis backlog queue, and the search indexing pipeline. It was my first time managing a large-scale live demo, and with the requirement that "even if a server dies during the demo, it must recover instantly without the audience noticing," I was honestly quite nervous. That’s why I conducted repeated rehearsals, simulating almost paranoid worst-case scenarios.

Q2. I heard you changed the infastructure architecture just 2 days before the event. What happened?

While running continuous dry runs of our live scenario, we identified a critical single point of failure.

We were simulating the live workflow sequentially: incoming video feeds → queued analysis tasks in the backlog → search index updates → user search queries → response rendering. During this process, we realized we had no fallback strategy if our main server went down.

For a standard SaaS background process, taking a few minutes to recover is acceptable. But for a live demo, those few minutes do not exist. Since our system directly renders results onto the stage screens, a server halt means the demo stops dead in its tracks.

I immediately raised this with the team, and after an internal discussion, we decided to implement High Availability (HA). This wasn’t requested by external organizers; it was our own operational standard. Calculating our remaining time, we had exactly 24 hours.

Q3. Implementing standard HA in 24 hours seems impossible. What did you trade off?

We sacrificed cloud-managed services and Infrastructure as Code (IaC) automation tools.

In an ideal world, you build high availability using cloud-managed failover services and define all infrastructure as code. That is the gold standard. However, provisioning permission policies, configuring managed clusters, and validating failover behavior alone takes days. Starting from scratch to reach a production-ready state within 24 hours was mathematically impossible.

So, we reframed our objective. Our primary goal was not 'achieving theoretical architectural perfection', but 'demonstrating Heimdex's search and video analysis performance with zero downtime on stage'. With that goal in mind, the technical answer changed.

We pivoted to a script-based Active-Active architecture. Node A (Primary) acted as the authoritative data source, while Node B (Standby) tracked Node A's state in real-time, ready for immediate elevation. We reduced our scope to what was deterministically testable and compensated by running relentless manual rehearsals.

The most critical step here was explicitly documenting every trade-off we made. That document served as our roadmap for what needed proper refactoring after the event.

Q4. You intentionally avoided automated failover. That sounds counterintuitive for an infrastructure engineer. Why?

We deliberately excluded automatic failover. It was the hardest decision we debated.

Automated failover is essential for routine operations. However, in an event venue's unpredictable network environment, the risk calculation changes completely. The damage caused by an automated false positive is far worse than a human operator responding a few seconds later.

What are Split-Brain and Flapping?
These refer to system misbehaviors where temporary network latency causes a monitor to mistake a healthy primary server for dead, leading to premature failovers, split data states, or continuous reboot loops (flapping).

Live demos operate in external venues, not isolated server rooms. Transient network spikes can happen anywhere. If an automated script drops a perfectly healthy primary server live on stage, you lose control in front of the audience.

Instead, we implemented double safety guards: failover triggers required consecutive health check failures and explicit manual confirmation from an engineer.

During rehearsals, we measured that even if the primary server died, traffic completely cut over to Node B after ~10 seconds of latency, achieving full elevation in 87 seconds. This was well within the acceptable margin for our presentation flow. We chose predictable outcomes over flashy automation.

Q5. You also put significant effort into search index synchronization. How was that handled?

Heimdex's core tech lies in chunking video into scene units and indexing them for fast search. If the search index becomes out of sync, search results return wrong data. In a live demo, incorrect results are actually worse than a crash. If a server crashes, you see it and fix it; if a query returns wrong information, nobody catches it instantly.

Our primary database was running real-time streaming replication with near-zero latency, so that was safe. The main challenge was keeping the search engine indexes in sync across nodes.

We implemented periodic zero-downtime re-seeding alongside atomic alias swaps. The Standby node pre-built the latest index state in the background. Once ready, we performed an atomic swap on the query alias pointer. From the user's perspective, query serving was seamless and instant.

We added two extra safeguards:

Three-Tier Fingerprint Validation: Comparing only document counts misses subtle data drift (e.g., if 1 record is added and 1 is deleted, the total count remains identical). We checked total document count, per-video chunk count, and latest timestamp.
Safety Guardrails: If the Primary node (A) experienced a temporary query hiccup, the Standby node (B) froze its current valid state instead of syncing degraded data. We prioritized data consistency over aggressive synchronization.

Thanks to this setup, index discrepancies were caught and reconciled automatically, resulting in zero search freezes or incorrect query outputs throughout the event.

Q6. How did you monitor and operate everything on-site?

I turned a single laptop into an on-site control station.

An automated monitoring script ran short-interval health checks, detected index drift, and pushed updates automatically. My screen was partitioned into multiple tiles displaying server health, resource usage, queue backlog depth, and real-time Slack alerts.

Unplanned conditions always happen at live venues. We iteratively patched edge-case response scripts on the fly. Because we rehearsed heavily, I already knew exactly which system signals indicated actual danger versus temporary noise.

Q7. How was your experience running the workloads on KT's ultra-low latency network?

Personally, this was the most impressive highlight.

Chunking video frames and generating natural language descriptions requires intensive compute and heavy payload data transfers. It is extremely network-sensitive. Running it over KT's dedicated infrastructure felt night-and-day different from our standard dev environments. Waiting times shrank noticeably for identical video inputs and models.

That’s why we placed the standard public network and KT network side-by-side on stage. Viewers didn't need technical explanations—the performance gap was immediately visible on the split screens.

As an engineer, seeing our AI workloads leverage such low-latency infrastructure was a thrilling experience. Opportunities to benchmark workloads on that tier of network infrastructure are rare.

Q8. What were your key takeaways in the post-mortem review?

Our biggest takeaway was strictly decoupling our Live Demo Architecture Requirements from our Production System Requirements in our documentation.

The demo architecture was designed to guarantee a verifiable state within 24 hours. Production systems have entirely different goals. Based on this, we set three sequential priorities for our permanent production stack:

Infrastructure as Code (IaC): Declaratively define all configs to recreate identical environments with a single command.
Managed Backend Services: Retain our custom alias swap and consistency guardrails, but offload underlying cluster replication to managed infrastructure.
Automated Failover: Enable autonomous system detection and failover without human intervention.

There is a clear reason for this order: IaC must come first so that automated failover can be safely simulated and verified in reproducible environments. If you implement auto-failover first, you end up debugging transient false-positive failures in non-reproducible staging environments.

Q9. What does the next phase look like?

We are building an enterprise-ready system architecture suitable for large-scale AI deployments:

Container Orchestration: A multi-node auto-scaling environment supporting zero-downtime rolling deployments.
Resilient AI Pipeline: Dynamic worker scaling based on queue depth to prevent processing bottlenecks during high-throughput requests.
Distributed Tracing: Request-level tracing covering the entire pipeline from video upload and AI inference to search index publication.

Tracing is what I personally missed most. When an anomaly occurred, diagnosing which specific stage caused the latency took time. Full end-to-end tracing will eliminate that overhead.

Q10. Any final closing thoughts?

To summarize in one sentence: "This demo succeeded because human operators guarded a temporary HA system; next time, I want the system to guard itself."

Under a tight 24-hour deadline, we substituted theoretical perfection with custom scripts, rigorous dry runs, and hyper-focused monitoring. Throughout the event, I kept my eyes fixed on the multi-tile dashboard to maintain absolute visibility. Our top priority for the next milestone is automating that surveillance and failover logic into the core infrastructure.

I'm proud that we delivered a lean, effective setup without over-engineering under time pressure. At the same time, it highlighted key areas where I need to deepen my technical expertise. It was a deeply rewarding experience that motivated me to grow further as an engineer.

💡 We are hiring at Heimdex!

Heimdex builds multimodal AI for video search.

Our technology enables natural language search across thousands of hours of video archives—finding exact moments like "a blue truck driving by" or "a specific person entering the frame." We handle facial recognition, scene-level semantic understanding, and automated draft generation for subtitles and audio descriptions. We also support on-premises enterprise deployments.

Key Highlights (Founded 7 months ago):

Selected for NVIDIA Inception and MSS/NVIDIA N-Up Program 2026.
Secured pre-seed investment from Samsung Fire & Marine Insurance (focused on insurance claims processing and fraud detection automation).
Conducting Enterprise PoCs with LG HelloVision, LiveNow, and others.
Executing joint R&D projects with Chung-Ang University PAI Lab (2026).
Incorporated in Gwangju in December 2025, reaching our current scale in just 8 months.

What you'll do as an Infrastructure Engineer

Video AI requires much more than just deep learning models. Large media files must be ingested, GPU workloads queued, indexes continuously updated, and low-latency search APIs served to users. On-premises client deployments introduce additional environmental constraints.

The engineering scope is broad. As Jiseok experienced, even recent hires take full ownership of end-to-end infrastructure design and critical architectural decisions. That is both the challenge and the massive growth opportunity of joining at this stage.

We are actively laying our engineering foundations. IaC, managed migration, and autonomous failover lie directly ahead. Building an infrastructure roadmap from scratch is a fundamentally different experience from operating pre-existing corporate systems. If that excites you, we’d love to chat.

Careers & Inquiries: [Job Board Link] / heimdex@heimdex.co
Enterprise Solutions: https://www.heimdex.co/en/contact