Running more work in parallel means finding room for the work that overlaps. If each worker loads its own models and data, adding another one can use up the available memory before it has contributed much to the calculation.
This came up while I was working on the historical evaluation for Bright Tape, a market-analysis product I am building with AI coding agents. I wanted to make better use of the available CPU cores while keeping the evaluation within a shared 6 GB memory budget.
The evaluation covers more than 200,000 historical starting points, where forecasts are compared with the market movements that followed. It runs in chunks so completed work can be saved and the process can resume after a stop. More than 44,000 of those points already had saved results when we started this investigation, so any change needed to preserve that work and continue from the correct place.
The execution records gave us a reason to look more closely at how the work was organised. The largest recorded memory peak occurred while loading models. A different, particularly expensive chunk spent almost all its CPU time scoring forecasts. These were observations from individual chunks, but they suggested that loading and scoring could benefit from being handled separately.
I asked whether we could split model loading from the rest of the process, so the scoring work would have more of the memory budget available to it.
Separating preparation from scoring
The proposed change gives preparation its own short-lived process. It loads the models, prepares the forecasts for a chunk and saves the information needed for scoring. Once that handoff is complete, the preparation process exits. Scoring can then start without keeping the loaded models in memory.
From there, two scoring processes could work on different forecast paths for the same historical starting point. They would use the existing scoring calculations, and their results would be put back in the original order before the system moved on. The second process would receive only the observations and forecast path it needed for its current task.
That still involves copying some data and running another process. Both have a cost, and the scoring calculations themselves need memory. The trial therefore needs to establish how much room the separation actually creates and whether two scoring processes can use it productively.
I also asked whether loading asynchronously would make the memory peak lower and more predictable. Working through the timing made an important limitation clear: loading the next chunk while the current one is still being scored would make their memory demands overlap. It could save waiting time, but it could also bring back the pressure we were trying to reduce.
For the first version, preparation will finish and exit before scoring starts. Background preparation can be tested later, once we have measured both phases and know whether they fit together with enough room left for the rest of the process.
Checking whether the extra coordination pays off
I wanted to understand whether managing these separate processes would take up the time we hoped to save. Starting a process, preparing the handoff, moving data and checking the returned results all contribute to how long a chunk takes to finish.
We planned a comparison of three versions using the same inputs. The first keeps the existing process. The second separates preparation from scoring but still uses one scoring process. The third uses the same separation with two scoring processes. This should help us see which change affects memory use and which, if either, makes the complete job faster.
The timing needs to include everything from startup to checked and saved results. A shorter scoring phase would not be enough if preparation and data transfer added the same amount of time elsewhere. The 6 GB allowance also has to cover all the processes together, including the coordinator and the data waiting to be passed between them.
Before comparing speed, we need to check that the new versions produce exactly the same evaluation output. Changing when calculations run must not change their results, omit a forecast or alter the saved order. We also need to test what happens if a scoring process fails, because the evaluation must be able to resume from its last verified checkpoint without skipping or repeating saved work.
If the comparison supports the change, the rollout will start with one unfinished chunk, then a few more before continuing through the remaining queue. A local coordinator will manage the routine execution and checks. I do not want an AI agent reviewing every chunk or repeatedly checking in on an otherwise unchanged process.
The trial will determine whether the gain is useful enough to keep the additional coordination, and whether there is room to test background preparation afterward.