Blog
How 365Scores Scaled Up with ironSource Atom

banner (24)Below is a guest post by one of our customers, leading sports app 365Scores, who used Atom DFM to store, manage, and customize their data flow according to their exact needs. What did they need their data to do and how were they able to utilize Atom’s data enrichment offerings while retaining full control of their data? Read on for a personal account on how Atom helps you own and maximize your data’s full potential:

Our Data Needs

Our app launched three years ago, and since then our data-related needs have become increasingly complex. We outgrew Google Analytics and other free platforms, but we were not in the market for a full end-to-end solution for data reporting, storage, and visualization. These end-to-end solutions tend to be expensive and difficult to use, but the bigger problem is that they take the data out of your hands. We wanted to have full control over our data - from the device to our data warehouse - allowing us to run manipulations and transformations as new use cases arose.

atom (1)

Another important factor for us in building our data flow was scalability. At 365Scores our traffic comes in spikes because our notification and data capture system is built around sports events. When Messi scores a goal, for example, we see major jumps in traffic. We had high demands at specific times, so we needed a reporting system with large throughput capacity but one that could also adapt and shift with our needs; we didn’t want to pay for a large capacity when there were traffic lulls.

We could have tried to build our own reporting system that fit our unique demands, but this would have required a considerable investment in terms of development hours and cost. We would also have had to hire an IT team just to monitor and tweak our data services as needed. We were really looking for a solution we could ‘set and forget’, giving us the freedom to focus on developing our own products.

365Scores_1

In our market research, we came across many solutions for both data storage and data reporting. Once we saw that Amazon Redshift was the best solution available for data storage, performance, and scalability, we decided to go with Atom Data Flow Management for data reporting because of its massive threshold for data intake, strong performance, high scalability, and their team’s extensive experience with Amazon Redshift. Atom DFM is the data flow management service developed by ironSource to handle their big data infrastructure.

Ultimately we built our current data flow with Atom for data flow management, Amazon Redshift for analytics and storage, and Tableau for visualization.

Integrating the Atom DFM SDK into our app was simple. Just by copying and pasting a few lines of code we were able to pass all our events through Atom DFM’s data pipelines.

var tracker = new IronSourceAtomTracker({

endpoint: "https://track.atom-data.io/",

apiKey: "4e6f7298a9c2d168935f58c001bad88"

});

var params = JSON.stringify({userid: 'ab353d', level: "5": duration: 34});

tracker.track('analytics', params);
Handling Traffic Spikes

Traffic spikes were previously a big challenge for us. When our traffic increased from 5,000 requests to 200,000 daily, the Elastic Load Balancer that handles our data had to be pre-warmed to be able to handle spikes. But now with Atom DFM, we are part of a big multi-tenant pipeline so this issue is taken care of.

A snapshot of our data being sent to Amazon Redshift.
A snapshot of our data being sent to Amazon Redshift.

 A graph from the Atom DFM console showing the inserts into Amazon Redshift.
A graph from the Atom DFM console showing the inserts into Amazon Redshift.

Infinite Scalability

Atom DFM handles more than 70 billion events a month, with origins ranging from desktop to mobile applications, and the platform has experience in handling huge surges from all origin types. Their infrastructure scales efficiently so we only pay for servers and services we are actually using. And since Atom DFM is integrated with Amazon Redshift, we have room for infinite scalability.

A Joint Partnership: Atom Data Flow Management and Amazon Redshift

While Atom DFM has an extensive range of services that expand Amazon Redshift’s capabilities, here are a few of the features that stood out for us and what our operation needed:

Pause/Play

With Atom DFM we can pause the flow of data into our tables and clusters to vacuum or do maintenance, and then resume the flow without losing any events. While the flow is paused, all of the incoming data is backed up and secured in S3.

Atom DFM allows us to stop the flow of data and resume when we’re ready.
Atom DFM allows us to stop the flow of data and resume when we’re ready.

 

This is great for when we occasionally want to resize a cluster or reformat a table while simultaneously configuring our data warehouse. Since this process can take several hours, it’s a huge benefit that we don’t lose any data in the transition time. Atom DFM performs a full vacuum on our tables once a day, plus their team of experts does additional optimization on the Amazon Redshift cluster.

Replay

After pausing the data flow, we can then use the replay feature to reinsert historical data after we have made adjustments to our cluster. And since we own our data with Atom DFM, we can move our data to different vendors if our requirements change.

Atom DFM’s historical data replay capability.
Atom DFM’s historical data replay capability.

 

Automated Amazon Redshift Encoding

Encoding your Amazon Redshift table is extremely important for optimizing performance and cost. Atom DFM automatically encodes tables, regardless of the encoding type. Here is an example of a table before and after encoding:

Table data before and after Atom DFM encoding.
Table data before and after Atom DFM encoding.

You can see from the graph that encoding the table reduced its size from 2.6T to 860G - a significant improvement with an immediate impact on our speed and bottom line.

Fixer

The Atom DFM dashboard contains a STL_LOAD_ERRORS table that shows all the bad logs for our data. The dashboard identifies the problem and then allows us to easily fix the problematic logs and put them back in the table. We can fix the issue on the fly (or even retrospectively, using the replay tool) without the need to deploy a new client version.

The errors table also shows when the problem occurred, and after we make the adjustment we get all the problematic events reinserted correctly. This way Atom DFM ensures that we can hold on to all our data, even if we make a mistake.

The Atom DFM STL_LOAD_ERRORS table.
The Atom DFM STL_LOAD_ERRORS table.

 

Data Lifecycle Management

As an added benefit, Atom DFM also makes it simple to manage data after it has reached our Amazon Redshift cluster. For example, we can move data from SSD to HDD, and eventually to AWS Glacier, allowing us to reduce storage costs for data we don’t need to query often.

Accessible data lifecycle management helps us save on storage cost.
Accessible data lifecycle management helps us save on storage cost.

Scaling with Atom Data Flow Management

More than anything else, Atom DFM gives 365Scores the peace of mind that all our data is accounted for, and stored safely and smartly. We are now handling billions of events a month, but we’re only charged for exactly what we use, and we pay as we grow. Atom DFM has helped make Amazon Redshift’s powerful services accessible for an operation of our size; it removed the hassle of inserting data, and provided us additional tools for reporting and database management.

What is 365Scores?

365Scores is a leading mobile sports application company whose app, 365Scores, surfaces sports content from across the globe and delivers it in over 25 different languages to sports fans around the world. From tweets to blogs and scores, 365Scores lets you follow the teams, the sports, and the athletes you choose. The mobile app provides real-time score updates and other breaking news via convenient notifications to the fans. Our app has surpassed 10 million downloads and is currently the most rated sports app on the Google Play Store.

feature (25)

Let's put these tips to good use

Monetize with ironSource