Docs · 06 Feed Generator

Feed Generator · accordion list, filter templates, DBAL streaming, keyset pagination and gzip for large catalogues

Via the Feeds tab you provide automatically updated URL feeds — for Google Shopping, price comparison portals or ERP connections. With built-in templates, your own profiles, filter templates, accordion list, renaming, delete confirmation, streaming performance for large catalogues and token authentication.

What is a feed?

A feed delivers your product data under a unique, tokenised URL — automatically updated, no manual export. The recipient is typically an external system: Google Shopping, idealo, Billiger.de, an ERP system or an analytics pipeline. The URL is tokenised — without the token you see nothing.

Create a feed — template or your own profile

At the top of the Feeds tab you find the "Create feed from template / profile" selector. With one click you create a new feed:

Built-in template — master data, prices & stock, SEO, images, variants, full export. An export profile with the matching field selection is created automatically (named e.g. "Master data (Feed)"), the feed activated, a token generated.
Existing profile — your own export profiles from the export tab can be activated directly as a feed.

Multiple feeds per profile: You can create any number of feeds for the same profile — for example a "Master data" profile as the basis for multiple feeds with different filter templates (one feed per sales channel or target market). Feeds are decoupled from the source profile via an own storage key (exportFeeds).

Feed list as an accordion

All active feeds and all custom export profiles appear as a compact list — each row is a self-contained accordion. Per row, at a glance, you see:

Name — the feed name (renamable)
Status pill — "feed active" (green) or "no feed" (grey)
Feed URL preview — the tokenised URL for active feeds
Filter template chip — the chosen filter template or "no filter"
Product count chip — number of included products, determined live from the catalogue
Pencil icon on the right (opens/closes the editor) and trash icon (deletes with confirmation)

Accordion behaviour

Clicking the row reliably opens and closes the editor — a chevron on the right shows the state and rotates on opening. The trash button reacts separately (with @click.stop) and doesn't accidentally open/close the row.

Editor content

The expanded editor offers all configuration options per feed:

Name field — rename the feed (must be unique; collision with existing feed or built-in profile is rejected)
Activate toggle — turn feed online/offline
Mode — XML, CSV, TSV or JSON
Filter template — selection from saved filter templates
Feed URL — ready to copy (button same height as the token button)
Token — show or regenerate

Filter templates — what's applied, what's deliberately not

Instead of simple checkboxes (with/without variants, only active products) you now pick a saved filter template in the feed — the same one you use in the listing or for bulk-edit.

Criteria applied in the feed (reliably server-side)

Active status of products
Stock (min/max range)
Manufacturer
Category
Sales channel

Criteria deliberately NOT applied in the feed

Quality and completeness criteria from the filter template (e.g. "missing images", "incomplete meta data", "no description", "SEO score < X") are deliberately not applied in the feed — these evaluations are calculated live in the listing and cannot be reliably mapped server-side in the feed context.

Why this separation matters: If the feed silently applied quality criteria, your ERP would receive a silently distorted product set — some products would "disappear" without anyone noticing. With the clear separation, you know exactly: filter template defines which products, the feed delivers those products.

Live product count

Each feed row shows a chip with the number of included products. This value is determined live from the catalogue (via the filter template criteria) and updates when you pick a different filter template.

Orientation value: The number is a DAL count over the server-side filter criteria. It can deviate minimally from the exact CSV row count (e.g. when broken records are skipped on export) — as an orientation it's very close to reality.

Streaming performance for large catalogues

Feeds run through the FeedExportService with multiple performance layers — even catalogues with tens of thousands of products run through without memory issues:

DBAL streaming in 2,000-batches

Feed generation does not load product data all at once into memory. Instead, the service iterates through the catalogue in batches of 2,000 rows and emits them row by row (PHP generator with yield). Memory usage is therefore independent of catalogue size — whether 100 or 100,000 products.

Keyset pagination instead of LIMIT/OFFSET

The batches use seek pagination on the product number (unique, indexed column):

Instead of LIMIT 2000 OFFSET 50000 (gets slower with every page, because the DB has to internally page through until OFFSET)
Uses WHERE product_number > :last ORDER BY product_number LIMIT 2000 — the DB uses the index directly

Output time therefore stays linear with the number of products, whether on page 1 or page 25. For large catalogues this is the most important performance lever — classic OFFSET often becomes drastically slower on deep pages in PostgreSQL/MySQL.

gzip compression over HTTP

If the requesting client sends Accept-Encoding: gzip in the request header (all modern ERP/crawler/browser clients do automatically), the response is server-side compressed via gzencode() and delivered with Content-Encoding: gzip + Vary: Accept-Encoding.

Feed CSVs are pure text and compress very well — typically 80–90 % reduction
A 50 MB CSV is transferred as ~5–10 MB — drastically shorter load time, significantly less bandwidth
The client decompresses transparently — no extra effort on the ERP side
Compression level 5 (default) — good trade-off between CPU and compression

File mode for very large catalogues

Per feed you can choose the mode:

Mode "Live on request" — feed is generated directly from the database on every request. Always up-to-date, suitable for infrequent polling or smaller catalogues.
Mode "File" — a scheduled task pre-generates the file (every 15 min, atomically written to var/staw-pim-feeds/ via Flysystem). On request, the ready file is simply served — no DB load, very fast response. Ideal for ERPs that poll every few minutes.

Graceful fallback: In file mode, if the cron file doesn't exist yet (e.g. directly after creating a new feed), the controller automatically falls back to live generation. You don't have to wait for the cron to run the first time.

Memory limit safety

The feed endpoint defensively sets the PHP memory limit to 1 GB (@ini_set('memory_limit', '1024M')) — even with very wide column selections containing dozens of custom fields or full SEO sets, generation runs cleanly.

Migration of old feeds

Automatic & non-destructive: If you previously had feeds attached directly to an export profile (with token/URL), these are automatically migrated into the new list on first load. Existing feed URLs in your ERP, Google Shopping account or at price comparison portals remain valid — no re-configuration required.

Token and URL format

Each feed gets a unique token. The URL has the format:

https://your-shop.com/staw-pim/feed/{token}.xml
https://your-shop.com/staw-pim/feed/{token}.csv
https://your-shop.com/staw-pim/feed/{token}.json
https://your-shop.com/staw-pim/feed/{token}.tsv

Regenerate token

The regenerate token button creates a new token — the old URL becomes invalid. Useful when a feed link has been leaked publicly or partner access needs to be revoked.

Security

Don't share tokens publicly: Feed URLs are tokenised and not publicly indexable (no entry in sitemap.xml, noindex header). Anyone who knows the token sees the product data of the feed — so don't post in public repositories or tickets. If a leak is suspected: regenerate the token, the old URL is immediately invalid.