[cluster] Mechanism for craft items that play audio longer than 25 seconds
I have been experimenting to see if it is possible to create a craft item in cluster's World Craft that can play audio longer than 25 seconds.
The world where it is actually running
The craft item created using the mechanism introduced here is placed in the following world.
How to overcome the limitations of craft items
It is not difficult to create an item that plays short sound effects, but trying to play a whole song to the end and then loop it as is involved many tedious tasks.
Craft items have several limitations, such as the length and number of audio clips that can be stored, item capacity, and script size. The maximum amount of audio that can be put into a single item is five 5-second clips, for a total of 25 seconds.
Audio longer than that must be split into short clips and distributed across multiple items. This then requires a mechanism to play them in order with as little delay as possible. Increasing the number of items puts pressure on the World Craft capacity, and the upload order must also be planned carefully.
Each task is simple, but as the song gets longer, the number of things to manage increases, making it increasingly difficult to create without errors by hand.
In this article, I will first summarize the mechanism of the craft item I am currently considering and the work required to create it.
Concept: Division of roles between the control item and playback Pods
The method in this article consists of a "control item" that manages the entire song and multiple section audio sources that handle each segment of the audio. Hereinafter, the latter will be referred to as playback Pods. For example, for a 75-second song, the division of roles would look like this.
制御アイテム
├─ 再生Pod 1：曲の前半を担当
├─ 再生Pod 2：曲の中盤を担当
└─ 再生Pod 3：曲の後半を担当Basically, the user only places one control item in the world. The control item automatically prepares the necessary Pods and starts playback. The control item also manages stopping, restarting from the beginning, and looping at the end of the song.
Although multiple items work internally, the intention is to make it treatable as "a single item that plays one song" when in use.
Initial preparation: Preparing the audio source into a playable format
The first thing to check is the audio source format, length, number of channels, sample rate, etc. These are then converted into a state that can be handled by the craft item.
I also check the volume. If the peak of the audio source is too high, it needs to be adjusted, but roughly cutting only the protruding parts will change the original sound. Therefore, only when necessary, I apply the same ratio of adjustment to the entire song to keep the balance as intact as possible.
The positions where the audio is cut cannot simply be "every 5 seconds or so." If you only deal with decimal seconds, the position may gradually shift due to rounding errors. I use the actual number of audio samples as a reference and also provide a safety margin so that it does not reach the upper limit.
It is not just a matter of simply splitting the file at regular intervals.
Planning: Managing the correspondence between clips and playback Pods
The converted audio source is divided into short clips and assigned to several playback Pods. As the song gets longer, both the number of clips and Pods increase.
When splitting, you need to manage the following information so that it remains consistent across the entire song.
At how many milliseconds into the song does this clip start?
Which Pod is responsible for it?
What is the sequence number of that Pod in the overall song?
How many seconds is the entire song, and how many seconds of rest are there between loops?
Do all Pods belong to the same song?
Even if the number or start position is off by just one, audio will drop out or different segments will overlap. If you create it manually, you will end up transcribing the same values over and over while comparing tables and configuration files. This becomes quite tedious as the song gets longer.
Control: Sharing playback time between Pods
Each playback Pod is an independent craft item.
Simply put, it might seem like a good idea for the previous Pod to send a start signal to the next Pod when it finishes playing. However, with this method, slight delays in communication or processing are passed on to the next Pod. As the number of segments increases and loops repeat, there is a possibility that it will gradually drift from the scheduled time.
Therefore, instead of passing signals in order between Pods, I use a method where all Pods are notified of the same playback start time. Each Pod holds information on "how many milliseconds after the start of the song it should play its segment" and plays it by comparing that with the current time.
Even when looping, I do not add to the time the previous loop actually finished. I recalculate the scheduled time for that iteration based on the initial reference time and the length of the song. This is to prevent short delays occurring at one boundary from being carried over to the next segment or the next loop.
Capacity saving: Generating only the necessary playback Pods sequentially
The longer the song, the more Pods you need. However, if you keep all Pods in the world from start to finish, it will easily consume the World Craft capacity.
It would be a problem if you could play long songs but couldn't place other exhibits or gimmicks in the world because of it.
Therefore, I adopt a method of generating and destroying Pods for the necessary segments on demand.
I prepare the segment currently playing and the segment that will come next in advance. Once a Pod has finished its role, it stops the audio and is destroyed, and after confirming that it has disappeared, the next Pod is generated. The control item manages the same flow even when looping back to the beginning of the song.
いま再生するPodを生成
↓
次のPodを先回りして準備
↓
担当区間を再生
↓
役目を終えたPodを停止・破棄
↓
消えたことを確認して、その次のPodを生成What requires caution is that a Pod instructed to be destroyed may not disappear immediately. If you continue generating without confirming that the old Pod has actually disappeared, you might end up with overlapping Pods, which defeats the purpose of the mechanism for reducing capacity.
Through this swapping, even if you use many Pods for the entire song, you can limit the Pods existing during execution to the necessary range. This is a technique to handle long audio sources while leaving world capacity for other expressions.
Generating Pods can take time. If it is too slow, it will not make it in time for the next segment. Therefore, I treat everything from when to start creating, whether preparation is complete, to whether it can join at the scheduled time as part of the playback control.
Experiment: Verifying clip boundaries on actual hardware
Even if all Pods are looking at the same schedule, it is not possible to reserve commands to play audio with sample-level accuracy. Script update intervals and various other factors affect how it actually sounds.
Therefore, short silences or overlaps may occur at clip boundaries.
So, one might wonder if playing the next clip slightly earlier would solve it, but it is not that simple. If you advance the boundary where sounds are already overlapping, double sounds or clicks may become more noticeable. The results also vary depending on whether the sound is a sustained tone, a sharp-attack sound like a drum, or singing or speech.
For this reason, I created test audio sources with set frequencies for investigation, played them in the world, recorded them with OBS, and examined where silences or overlaps occurred. Based on those results, I am experimenting with methods such as slightly overlapping adjacent sections and connecting them smoothly with crossfades.
While numbers on a waveform are important, you ultimately need to check how it actually sounds to the ear. There are boundaries that are noticeable even if the numerical value is small, and some materials sound more natural with a slight overlap.
Note that at this time, we cannot guarantee perfect gapless playback at the sample level. This area is still under investigation.
Upload: Order of Playback Pod -> Control Item
Control items require a confirmed ID to call each playback Pod. However, that ID is not known until the upload of the playback Pod is complete. In other words, you cannot complete the control item alone first.
The order is as follows.
音源を解析・変換する
↓
再生Podを構築・検証する
↓
再生PodをアップロードしてIDを集める
↓
全IDを組み込んだ制御アイテムを構築・検証する
↓
制御アイテムをアップロードするAs the number of Pods increases, so does the number of uploads, completion checks, and ID recordings. If communication is interrupted midway, you must also distinguish between whether it was "not sent" or "transmission finished but result verification failed." You also need records to resume from the middle without pointlessly recreating Pods that have already succeeded.
If done manually, you would have to note down each ID one by one, write them back into the control item without error, and manage how far you have succeeded. This is another part that becomes more troublesome as the song gets longer.
Recovery Control: Stop and Retry Playback in Case of Abnormalities
It is necessary to think not only about playing sound when things are normal, but also about what to do when something goes wrong.
Do not start playback if necessary Pods are missing
Do not proceed if Pods for the same section are duplicated
Do not mix in other songs or old playback instructions
Check if the clocks of the Pods are significantly out of sync
If a Pod disappears during playback, do not continue playing only the remaining sections
Confirm that the old Pod has disappeared before recreating it
When you cannot dispose of them correctly, avoid double playback and do not force an increase
For example, if a Pod in the middle disappears, continuing playback with only the remainder will cause parts of the song to periodically drop out. If you try to automatically recover and overlap the old Pod with a new one, it might play twice instead.
Therefore, if an anomaly is detected, you should stop playback once and resume only after confirming that it is safe to recreate. Considering actual use in a world, this kind of processing is also necessary.
Summary: Until the craft item is complete
The flow up to this point can be summarized as follows.
音源を用意する
↓
形式・長さ・音量を調べる
↓
再生規格へ変換する
↓
クリップへ分け、Podへ割り当てる
↓
曲全体の再生予定表を作る
↓
各Pod用クラフトアイテムを構築・検証する
↓
Podを順番にアップロードする
↓
確定したIDから制御アイテムを作る
↓
制御アイテムをアップロードする
↓
実機で生成・入れ替え・同期・ループ・停止を確かめるThe source is a single music file, but completion requires audio processing, a splitting plan, capacity management, synchronization, craft item generation, uploading, anomaly control, and verification on the actual device.
Future: Making long audio sources easier to handle
When actually trying it out, repeating the detailed preparations without mistakes every time was more difficult than playing long audio sources in cluster itself.
Checking audio limits, splitting them finely, aligning Pod numbers and start times, and managing uploaded IDs. Even if you understand the mechanism once, the same work awaits you every time you create a different song.
Once the audio source and playback method are decided, I am currently creating tools and thinking of various strategies to see if I can simplify the process up to the point where a craft item that can be placed in a world is finished.
I am still in the middle of testing, but if it goes well, I think it might lower the hurdle for using music in cluster's World Craft.
* This article includes mechanisms that are currently under consideration and verification. The configuration and scope of support may change due to cluster specification changes or the results of tests on actual devices. Also, regarding sound boundaries, this does not guarantee perfect gapless operation at the sample level.