Running a complex business with numerous branches or points of sale scattered in different geographical locations requires very well organized data exchange. No less vital here is timely and correct centralized data collecting from all sources and its analysis.
Data feed management (DFM) is the process of collecting, organizing, and delivering data feeds — structured files of data exchanged between systems — so that businesses can keep accurate, up-to-date information flowing across multiple sources. This is a challenging task since, depending on the source and type of data, the mechanisms of its delivery vary. Therefore, businesses need systems that can manage all types of data streams, or feeds, as they are typically called. E-commerce is a sphere with great demand for such software. Contrary to popular belief, gathering information from all sources in a network of e-stores and eCommerce marketplaces is way more than a few clicks, due to the different mechanisms of data retrieval used by various shopping carts.
However, it’s next to impossible to design a system that would equally satisfy the demands of every organization. Data feed management in each company has its own peculiarities and requires a personalized approach to building and configuring a data feed management system.
These are the conditions that make data feed management solutions highly demanded on the market. If you are planning to develop systems for managing data feeds, or are already running this type of business, you should know that organizing it is far from being a piece of cake. There are numerous hidden aspects that might become stumbling blocks if you are unprepared. This is why we've put together an account of the most frequent and common data feed management challenges, to help you get ready for them in your work. This post looks at metadata-related challenges in data feed management — one of several recurring pain points teams run into when building or maintaining a DFMS.
Feed metadata discovery challenges
So, what makes this “data about data” so important, and what pitfalls does it present for the functioning of the whole data feed management system (DFMS)?
Metadata is crucial for the system to correctly interpret the data files. It includes the following information:
- feed file naming code, which can provide the following: timestamp, report generating object, its attributes, file format, etc.
- files arrival patterns, encoding the next data: feed generation frequency, intervals, maximum delay between the event and generation of report about it, etc.
- file format specification - necessary for the system to parse the records
- data semantics - contains interpretation of each data field, like types of data, encodings, domains, etc.
Typically, issues occur because metadata information is unavailable, which makes it harder for the system to correctly interpret feed data files. Below, we go over the key challenges caused by missing metadata.
Metadata incomplete or missing
Aggregated feeds
Often, a source data feed consists of a number of uniform subfeeds with complementary information. Over time, the number of subfeed generators varies, as new ones are added and some become temporarily or permanently unavailable.
What is more, the data feed described above can, in turn, be part of a larger group of feeds with related information. Likewise, the number of feeds in the group tends to change over time, making it a challenging task to discover the metadata for individual feeds.
Insufficient communication with feed sources
This issue is common when source feeds are managed by different organizations, or even different units within a large company. These sources aren't obliged to document the metadata about their feeds, so they rarely spend time on it. The problem deepens when there is little or no communication between company units.
Discovering metadata via feed history browsing
When aggregated data feeds are used extensively and source feed managers provide insufficient information, individual subscribers are left to deal with missing metadata on their own. Traditionally, they try to discover the necessary information by browsing data feed history.
However, even in the relatively simple case of an aggregated feed composed of uniform subfeeds, it is easy to make wrong assumptions — especially when the feed history is scarce. For a group of feeds that are only loosely related to each other, the risk of guessing wrong about the individual data feed structure is much higher. Unfortunately, these wrong assumptions are often discovered only after users of the apps that receive feed data start complaining about incorrect information in the generated reports.
Feed changes
Over their lifetime, data feeds undergo significant changes, which presents a challenge for subscribers relying on the history-browsing method to discover metadata. First, they need to create specific feed definitions that exclude irrelevant files while still working well as the feed evolves.
Feed changes are virtually inevitable and can affect file content, structure, arrival patterns, and more. If the definition isn't robust enough to withstand them, the result is either new files being missed or unwanted files coming in. The longer this issue goes unnoticed, the more incorrect data will be received and processed, leading to undesirable consequences.
Reasons for missing or unwanted files in the feed
If a data feed definition is too specific or too generic, there’s a risk of false negatives or false positives in file classification under the conditions listed below.
- Changes in file naming convention
If the definition was generated from a sample using a certain file naming convention that later changed — for example, due to an update of the software generating the data feed — files with the new names may be recognized as not matching the definition.
- Introduction of new data feed sources
If the list of data sources is hardcoded in the definition, it will remain valid only until new sources not on that list start contributing to the feed.
- Definition based on an invalid data sample
This most commonly occurs with complex data feeds consisting of a number of loosely connected subfeeds. For the definition to be correct, all subfeeds should be present in the data sample. Otherwise, feeds that weren't included in the sample can later be marked as not corresponding to the definition.
- Using wildcards in the feed definition
A wildcard (*) can be used to replace an object name when composing a feed definition. It is mostly used to avoid false negatives in file classification when feeds change. However, this also brings the risk of including many irrelevant files in the feed.
Another approach is to simplify the definition by using wildcards in place of a values list in some fields of the filename. This too makes the definition vulnerable to non-matching files.
Conclusion
Neither feed subscribers nor feed providers can solve the issues described above on their own, since doing so requires communication between both sides. Moreover, even once a problem is detected, it can be challenging to fix without the right mechanism in place. It is therefore up to DFMS designers to build mechanisms for detecting feed metadata issues, along with scenarios for resolving them. Doing so adds real value to their products and supports long-term business growth.
If you're a DFM provider wondering how to streamline integration with shopping carts, consider API2Cart. We offer a unified API to connect to 70+ platforms like Shopify, Bigcommerce, Magento, PrestaShop, OpenCart, and many others. For a closer look at how unified API approaches compare to other integration options, see our breakdown of Cyclr API vs API2Cart. If you have any questions or would like to discuss how the power of the API could fit your business, schedule a call or leave us a message.
Marketing manager, API fan, nerd, perfectionist, British humour enthusiast, literature and history geek.
