My understanding is that it's a cached content. Correct? It's quite heavy to reproduce but it's a derivative data with ttl of a week. or am I mistaking it with something else?

In T426873#11941586, @Ladsgroup wrote:

My understanding is that it's a cached content. Correct? It's quite heavy to reproduce but it's a derivative data with ttl of a week. or am I mistaking it with something else?

Yes, persistent storage of derived output content, refreshed by a maintenance script every week (or maybe every month if load is too high).

In T426873#11941588, @Jdforrester-WMF wrote:

In T426873#11941586, @Ladsgroup wrote:

My understanding is that it's a cached content. Correct? It's quite heavy to reproduce but it's a derivative data with ttl of a week. or am I mistaking it with something else?

Yes, persistent storage of derived output content, refreshed by a maintenance script every week (or maybe every month if load is too high).

Then my strong recommendation is to user maintash instead:

• You get the TTL and clean up for free. No need to write a maint script to clean up
• It has replication factor of two, so if one host goes down or depooled, you still get the data.
• Since it picks two out of three at random (with consistent hashing), it does wear leveling on the infrastructure.
• It is also horizontally scalable. If the load goes really high, we can add more hosts to the rotation and it would work out of the box.

I know maintstash had a lot of issues but it went through a major re-architecture (T383327: Re-architecture mainstash (x2) to allow easier maintenance ) and clean up (T388323: ResourceLoaderModule-dependencies writes the exact same value to database multiple times every second) and it could easily take a lot more load now

Change #1290052 had a related patch set uploaded (by Jforrester; author: Jforrester):

[mediawiki/extensions/WikiLambda@master] AWStorage: Add MainStash-backed implementation of AWArticleStore

https://gerrit.wikimedia.org/r/1290052

Thanks for your input, Amir. However, I am not sure MainStash would work this use case.

Then my strong recommendation is to user maintash instead:

• You get the TTL and clean up for free. No need to write a maint script to clean up

I think there's some mixup between different stored pieces and different storage layers (understandably).
The storage layer referred to in this task is not the ephemeral storage of rendered fragments -- that one exists, too, and is the one that you accurately describe here:

My understanding is that it's a cached content. Correct? It's quite heavy to reproduce but it's a derivative data with ttl of a week. or am I mistaking it with something else?

The storage layer from this task is a different one, a persistent storage of article records of two types:

• Section records (N sections per topic and language), rendered at a given point in time, and
• Article Metadata records (1 metadata row per topic), keeps track of sections, their ordering, last updates and last re-render timestamps, etc.

These records are permanent, they aren't deleted nor they need a TTL, once they are created they are expected to remain in the store.
The maintenance script mentioned by James is a regularly scheduled script that, if possible, updates the section and metadata records with a more recent rendering state.

The existence of this layer as a persistent store has been supported in our conversations with SRE and responds to a heavy constraint:

• no user read request path will ever trigger a re-render; read paths are strictly isolated from the execution layer, which involves the Wikifunctions orchestrator and is performance-heavy

Out product commitment is that:

• a Wikipedia reader visiting an article built from AW content will always see content: no pending state
• content may be outdated; this is fine
• a wikipedia page built from AW content will clearly communicate to the reader the latest render date.

So, as you can see: there's no need of TTL, there's no duplication of records, and records should remain stored until an internally controlled process generates a new render and updates it.

Now, I'm not sure if this is fulfilled by MainStash, but my understanding is that, even if this is DB-backed, MainStash has a cache-like behavior (but please, correct me if I'm wrong); records are ephemeral by design, so misses need to be considered a possible path and handled by a fallback strategy or source of truth.

If we were to change this very important factor, then we need to revisit our whole architecture, which takes us back to square one in terms of what happens when a read request meets a missing section (due to end of ttl, eviction, cold cache...)

• we should not fall back to a rendering process (as per our understanding with SRE)
• we should not serve a pending article (as per our product commitment)

Considering that we are mid quarter, and we have been working on this agreed architecture for months, I'm afraid that changing from a durable to an ephemeral type of storage would likely put the AW integration hypothesis at risk.

I would like to have example of what data you're planning to store there.

In T426873#11944355, @gengh wrote:

Now, I'm not sure if this is fulfilled by MainStash, but my understanding is that, even if this is DB-backed, MainStash has a cache-like behavior (but please, correct me if I'm wrong); records are ephemeral by design, so misses need to be considered a possible path and handled by a fallback strategy or source of truth.

Yes and No. MainStash has cache like interface but from the infrastructure point of view it's somewhere between cache and storage of canonical data. They are not necessarily ephemeral, you can set the TTL to one year and just re-set the value from time to time with maint script or something else (and reset the ttl)

If we were to change this very important factor, then we need to revisit our whole architecture, which takes us back to square one in terms of what happens when a read request meets a missing section (due to end of ttl, eviction, cold cache...)

There is no such thing as eviction in main stash (it's backed by MariaDB so it's a database, not redis or something else), you set the ttl and can make sure it lasts long, cold cache is similar problem as empty table, you'll need jump start the data somewhere and somehow so you just need to change the backend.

• we should not fall back to a rendering process (as per our understanding with SRE)
• we should not serve a pending article (as per our product commitment)

MainStash replicates values in two clusters so if one gets removed, the other one should be able to still serve the data. So the chance of you actually ending up in a situation that mainstash has lost the data should be quite rare (maybe slightly higher than x1 databases but still quite low). In such cases, it's rather simple to queue a job to fill it back and throw a 500 at the user. It's not nice but the chance of it happening should be very low. You can have this via get() instead of using getWithSetCallback()

Considering that we are mid quarter, and we have been working on this agreed architecture for months, I'm afraid that changing from a durable to an ephemeral type of storage would likely put the AW integration hypothesis at risk.

MainStash is not really ephemeral (you can make it ephemeral) and for example, one use case is that we use it stash edits of users in visual editor so if it gets broken, users actually lose their edits.

Also I understand that you're at the middle of it, but we always recommend reaching out to us sooner to avoid having such conversations late but that being said, I don't think main stash is much different than databases.

Thanks, Amir! I now understand a lot better the idea behind MainStash (a great conversation with @RLazarus helped, too!): I agree, this will fit our use case just fine.

The data we are holding is expected to remain because it's costly to re-generate, but it is not critical, and as long as this doesn't happen regularly, it can be regenerated.

I would like to have example of what data you're planning to store there.

For the following configuration settings:

• allowed_topics: a list of allowed article topics that are present in Abstract Wikipedia and ready for their integration in language wikis.
• allowed_langs: a list of language codes for which the articles from the list above can be generated and integrated in their respective language wikis.

Assume:

• allowed_topics = [ Q319 ]
• allowed_languages = [ en, es ]

Also assume:

• the abstract wikipedia article Q319 has two sections: Q001 and Q002 (for simplicity)

The update script will upsert one entry per topic, language, and section with current timestamp and the rendered html value:

And for each article qid, one metadata entry with important timestamps and the sections in the article:

This stored data supports the article read path from language wikis which have articles that are AW powered, so that this read path is fully isolated from any requests to the Wikifunctions code execution services:

You can see the other related paths (e.g. the update script that feeds the storage layer) in here https://miro.com/app/board/uXjVHQYomu8=/

Also I understand that you're at the middle of it, but we always recommend reaching out to us sooner to avoid having such conversations late but that being said, I don't think main stash is much different than databases.

You are right to call this out. We have been working closely with SRE to address all past issues and future requirements regarding our caching and storing, I didn't know you were the person to reach out and I would have done it sooner if I knew, my apologies! 🙏

SO:

• we have a solid interface for our AWArticleStore
• @Jdforrester-WMF has written its MainStash implementation, which I have been testing it and it rocks ✊
• @RLazarus and I have discussed what happens when there's an occasional miss: we will show an error box, communicate expectations to the reader, log, and measure success rate.
• We will agree on a realistic SLO around successful article render.

@Ladsgroup: if you have no concerns, I think we could call this done.

Change #1290052 merged by jenkins-bot:

[mediawiki/extensions/WikiLambda@master] AWStorage: Add MainStash-backed implementation of AWArticleStore

https://gerrit.wikimedia.org/r/1290052

Nothing on my side. Thanks for doing it! And yeah it makes sense to have it as interface and let's say if in a couple of years, we decide to switch to some other backend, we could just move that.