Solving Memory Leaks in Large File Archives

When importing hundreds of files into Dotient, we noticed memory climbing to 1.2GB despite a small 2.2MB SQLite database. Heap snapshots revealed a 100:1 ratio of functions-to-files, indicating massive per-item overhead in our Svelte components. This report details our investigation into the root causes and the engineering solutions we implemented to reduce memory usage by over 75%.

The memory issue manifested most prominently during bulk imports. When a user imported 300 files simultaneously, the application would become unresponsive, with CPU spiking to 70% and memory climbing steadily until the import completed. Even after the import finished, memory remained elevated at approximately 1.2GB, far exceeding what should be required for a file archive containing just a few hundred items.

Our investigation began with Chrome DevTools heap snapshot analysis. We took snapshots before and after importing 300 files and discovered several concerning patterns. The most significant was a delta of +29,298 new Functions and +9,629 node-related objects for only 300 new files, a nearly 100:1 ratio that suggested massive per-item overhead in our Svelte components. The retainer tree showed memory was being held by "reactions" and "deps" within the Svelte reactivity system, indicating that every file in our archive was creating an excessive number of reactive dependencies.

Virtualization Fix

The initial investigation into our masonry grid revealed that virtualization was not functioning as intended. While we had implemented a virtual scrolling system designed to only render items currently visible in the viewport plus a small buffer, a bug in our boundary calculation was causing far too many items to render.

• The problematic code calculated the end index using BUFFER * 100, which translated to loading 2000px worth of items below the viewport
• With our BUFFER value of 20, this meant rendering over 200 items instead of approximately 40-60
• When total height calculation was smaller than expected, the loop condition would never trigger its early exit

We addressed this with two changes. First, we replaced the fixed 2000px buffer with a proportional buffer of viewportHeight * 0.5, which scales appropriately regardless of screen size. Second, we added a hard cap limiting the maximum number of rendered items to 80, ensuring that even under pathological conditions, the DOM would not become overwhelmed.

Event Handler Refactoring

Svelte's reactivity system is powerful but can create performance issues when used carelessly in list contexts. One of the major sources of function object proliferation in our application was the use of inline arrow functions within each blocks.

• Each item had multiple event handlers: click, keyboard, drag-and-drop, waveform hover
• Inline arrows like onclick={() => handleClick(item.id)} create new closures on every render
• With 80 visible items and multiple handlers, this meant hundreds of new function objects per scroll

The fix was to refactor these to named function references. Instead of inline arrows, we created module-level functions that accept the necessary parameters. This allows Svelte to reuse the same function reference across renders, dramatically reducing the number of function objects created.

Waveform Hover Throttling

The most severe performance issue we discovered was in our audio waveform visualization. For each audio file in our archive, we render an SVG waveform that displays the audio data as a series of bars. When a user hovers over this waveform, we calculate which slice of the audio they're pointing to and display a tooltip with volume and brightness information.

• Original implementation updated tooltip on every mousemove event
• It created a new Map object and triggered Svelte reactivity each time
• This was responsible for creating thousands of Map objects per second during interaction

The solution was to add throttling. We implemented a simple throttle that only updates the tooltip position every 50ms, which is more than fast enough for smooth visual feedback while reducing reactive updates by an order of magnitude.

Reactivity Debouncing

Svelte's reactive statements ($:) are incredibly convenient but can be dangerous when attached to frequently-changing values. In our masonry grid, we had a reactive statement that ran whenever archiveItems changed:

• During bulk import of 300 files, archiveItems changed 300 times (once per file)
• Each change triggered computePositions() and runVirtualScrollNow()
• Result: hundreds of unnecessary array allocations during an already stressed period

We resolved this by adding a debounce. Instead of running immediately on each change, we deferred execution by 50ms using setTimeout. This batches rapid changes together during bulk import, all 300 file additions result in only a handful of layout recalculations rather than 300.

Waveform Cache Limits

Our audio waveform generation is computationally expensive. To avoid regenerating waveforms each time they came into view, we cached the results in a Map with a maximum size. However, our cache limit of 50 was too generous, and we had no protection against duplicate generation.

• When users scrolled quickly, multiple instances of the same file could briefly exist in the DOM
• Each triggered generateWaveform() without deduplication
• Same expensive operation ran simultaneously, wasting CPU and memory

We reduced MAX_CACHE_SIZE to 15 and added a Set to track in-progress generations. Now, if a second request comes while generation is running, it waits for the existing promise rather than starting duplicate work.

Results

After implementing these optimizations, we measured significant improvements across all our performance targets:

• Memory Usage: Peak memory during bulk import decreased from 1.2GB to approximately 300MB, a 75% reduction
• DOM Node Count: Rendered masonry-item elements dropped from over 200 to approximately 80
• Array Allocations: Heap snapshot analysis shows allocation decrease of over 90%
• Render Performance: Scrolling is now smooth at 60fps even with 600+ files

Lessons Learned

This investigation reinforced several important principles for building performant Svelte applications:

• Virtualization requires vigilance. Implement virtualization once and verify it's working. Add safeguards like hard caps.
• Inline functions in loops are expensive. Named function references are virtually free by comparison.
• Mousemove is a hot path. Any work done in response to mousemove should be throttled.
• Reactivity has a cost. Debouncing is essential when reacting to list changes.
• Cache aggressively, but not too aggressively. Tune cache limits based on actual usage patterns.