banner (24)Below is a guest post by one of our customers, leading sports app 365Scores, who used Atom DFM to store, manage, and customize their data flow according to their exact needs. What did they need their data to do and how were they able to utilize Atom’s data enrichment offerings while retaining full control of their data? Read on for a personal account on how Atom helps you own and maximize your data’s full potential:

Our Data Needs

Our app launched three years ago, and since then our data-related needs have become increasingly complex. We outgrew Google Analytics and other free platforms, but we were not in the market for a full end-to-end solution for data reporting, storage, and visualization. These end-to-end solutions tend to be expensive and difficult to use, but the bigger problem is that they take the data out of your hands. We wanted to have full control over our data – from the device to our data warehouse – allowing us to run manipulations and transformations as new use cases arose.

atom (1)

Another important factor for us in building our data flow was scalability. At 365Scores our traffic comes in spikes because our notification and data capture system is built around sports events. When Messi scores a goal, for example, we see major jumps in traffic. We had high demands at specific times, so we needed a reporting system with large throughput capacity but one that could also adapt and shift with our needs; we didn’t want to pay for a large capacity when there were traffic lulls.

We could have tried to build our own reporting system that fit our unique demands, but this would have required a considerable investment in terms of development hours and cost. We would also have had to hire an IT team just to monitor and tweak our data services as needed. We were really looking for a solution we could ‘set and forget’, giving us the freedom to focus on developing our own products.

365Scores_1

In our market research, we came across many solutions for both data storage and data reporting. Once we saw that Amazon Redshift was the best solution available for data storage, performance, and scalability, we decided to go with Atom Data Flow Management for data reporting because of its massive threshold for data intake, strong performance, high scalability, and their team’s extensive experience with Amazon Redshift. Atom DFM is the data flow management service developed by ironSource to handle their big data infrastructure.

Ultimately we built our current data flow with Atom for data flow management, Amazon Redshift for analytics and storage, and Tableau for visualization.

Integrating the Atom DFM SDK into our app was simple. Just by copying and pasting a few lines of code we were able to pass all our events through Atom DFM’s data pipelines.

var tracker = new IronSourceAtomTracker({

endpoint: "https://track.atom-data.io/",

apiKey: "4e6f7298a9c2d168935f58c001bad88"

});

var params = JSON.stringify({userid: 'ab353d', level: "5": duration: 34});

tracker.track('analytics', params);

Handling Traffic Spikes

Traffic spikes were previously a big challenge for us. When our traffic increased from 5,000 requests to 200,000 daily, the Elastic Load Balancer that handles our data had to be pre-warmed to be able to handle spikes. But now with Atom DFM, we are part of a big multi-tenant pipeline so this issue is taken care of.

A snapshot of our data being sent to Amazon Redshift.

A snapshot of our data being sent to Amazon Redshift.

 A graph from the Atom DFM console showing the inserts into Amazon Redshift.

A graph from the Atom DFM console showing the inserts into Amazon Redshift.

Infinite Scalability

Atom DFM handles more than 70 billion events a month, with origins ranging from desktop to mobile applications, and the platform has experience in handling huge surges from all origin types. Their infrastructure scales efficiently so we only pay for servers and services we are actually using. And since Atom DFM is integrated with Amazon Redshift, we have room for infinite scalability.

A Joint Partnership: Atom Data Flow Management and Amazon Redshift

While Atom DFM has an extensive range of services that expand Amazon Redshift’s capabilities, here are a few of the features that stood out for us and what our operation needed:

Pause/Play

With Atom DFM we can pause the flow of data into our tables and clusters to vacuum or do maintenance, and then resume the flow without losing any events. While the flow is paused, all of the incoming data is backed up and secured in S3.

Atom DFM allows us to stop the flow of data and resume when we’re ready.

Atom DFM allows us to stop the flow of data and resume when we’re ready.

 

This is great for when we occasionally want to resize a cluster or reformat a table while simultaneously configuring our data warehouse. Since this process can take several hours, it’s a huge benefit that we don’t lose any data in the transition time. Atom DFM performs a full vacuum on our tables once a day, plus their team of experts does additional optimization on the Amazon Redshift cluster.

Replay

After pausing the data flow, we can then use the replay feature to reinsert historical data after we have made adjustments to our cluster. And since we own our data with Atom DFM, we can move our data to different vendors if our requirements change.

Atom DFM’s historical data replay capability.

Atom DFM’s historical data replay capability.

 

Automated Amazon Redshift Encoding

Encoding your Amazon Redshift table is extremely important for optimizing performance and cost. Atom DFM automatically encodes tables, regardless of the encoding type. Here is an example of a table before and after encoding:

Table data before and after Atom DFM encoding.

Table data before and after Atom DFM encoding.

You can see from the graph that encoding the table reduced its size from 2.6T to 860G – a significant improvement with an immediate impact on our speed and bottom line.

Fixer

The Atom DFM dashboard contains a STL_LOAD_ERRORS table that shows all the bad logs for our data. The dashboard identifies the problem and then allows us to easily fix the problematic logs and put them back in the table. We can fix the issue on the fly (or even retrospectively, using the replay tool) without the need to deploy a new client version.

The errors table also shows when the problem occurred, and after we make the adjustment we get all the problematic events reinserted correctly. This way Atom DFM ensures that we can hold on to all our data, even if we make a mistake.

The Atom DFM STL_LOAD_ERRORS table.

The Atom DFM STL_LOAD_ERRORS table.

 

Data Lifecycle Management

As an added benefit, Atom DFM also makes it simple to manage data after it has reached our Amazon Redshift cluster. For example, we can move data from SSD to HDD, and eventually to AWS Glacier, allowing us to reduce storage costs for data we don’t need to query often.

Accessible data lifecycle management helps us save on storage cost.

Accessible data lifecycle management helps us save on storage cost.

Scaling with Atom Data Flow Management

More than anything else, Atom DFM gives 365Scores the peace of mind that all our data is accounted for, and stored safely and smartly. We are now handling billions of events a month, but we’re only charged for exactly what we use, and we pay as we grow. Atom DFM has helped make Amazon Redshift’s powerful services accessible for an operation of our size; it removed the hassle of inserting data, and provided us additional tools for reporting and database management.

What is 365Scores?

365Scores is a leading mobile sports application company whose app, 365Scores, surfaces sports content from across the globe and delivers it in over 25 different languages to sports fans around the world. From tweets to blogs and scores, 365Scores lets you follow the teams, the sports, and the athletes you choose. The mobile app provides real-time score updates and other breaking news via convenient notifications to the fans. Our app has surpassed 10 million downloads and is currently the most rated sports app on the Google Play Store.

feature (25)

ironSource Blog

The Source

Ad Monetization and User Acquisition: Q&A with Tapps Games

Ad monetization and user acquisition are two sides of the same coin. For example, UA teams are dedicated to realizing true app growth, whereas monetization teams are driven to generate revenue. Taking this into account, it’s almost logical that gaming..

IndustryChat

What’s the Hype with Hyper-Casual Games? Part 3: Marketing

In our final chapter of LevelUp’s hyper-casual series, Ryan Davies, Digital Marketing Manager at Kwalee, shares his thoughts on running successful user acquisition campaigns for hyper-casual games. Read on for edited highlights from Davies’ podcast: Failing “Failing is critical, especially for hyper-casual games because you’re dealing with such a fast rate of content. You have..

Nordeus Talks Ad Monetization, Brands, and Industry Trends

Nordeus, a Serbian mobile game studio, is far from being the new kid on the block. Their hit title Top Eleven has been live for nearly a decade and has 200 million registered users, across Facebook, iOS, and Android. While Nordeus’ games are more IAP focussed, in-game ads still provide a significant source of revenue..

IndustryChat

The Future of Programmatic Mediation

In this episode of LevelUp, Pieter Kooyman, Chief Advertising Officer at Miniclip, sets the record straight on ‘programmatic mediation’ . He defines the term and predicts what the mobile industry can expect to see in the new year. Read on for edited highlights from Kooyman’s podcast: Many names “Let’s start by listing all the names..

IndustryChat

2018’s Most Popular Level Up Podcasts

2018 was quite the year for ironSource’s LevelUp Podcast. We launched the LevelUp podcast, produced 13 episodes (and 2 seasons!), generated thousands of listens, and made some awesome friends along the way. With the new year upon on us, we thought to curate a list of our most listened to LevelUp podcast episodes with some..

AdChat

The Convergence of Monetization and Design

One of the biggest mistakes a developer can make is only thinking about monetization strategies after designing their game. The choices you make in the initial phases of designing your game will influence the success (or failure) of your app monetization model. We spoke with Matthieu Brossard, the Publishing Director at What(Games) by Gameloft. What(Games)..

아이언소스 팁: 휴가 방식으로 UA하기

1. 여러 개의 창의적인 포맷을 구동하세요 일부 게임의 경우, 플레이어블(Playable)광고가 가장 수익성 높은 광고 포맷입니다. 다른 게임들의 경우, 비디오와 인터렉티브 엔드 카드(IEC)의 결합이 가장 좋은 광고 포맷이지요. 무엇이 여러분들에게 가장 적합한 포맷인지를 알아내는 유일한 방법은 각각 A/B 테스트를 해보는 겁니다. 우리는 보통 플레이어블 광고가 보상형 비디오 공급에서 가장 좋은 퍼포먼스를 발휘하고, IEC 방식은 중간광고에서 가장..

ironSource Rock: UA the Holiday Way

Reading the same decks, talking about the same best practices time after time gets a bit boring. We thought to share our best user acquisition tips in a completely different format. So we recorded a music video. Now sit back, relax, and enjoy our Christmas gift to you! 1. Run multiple creative formats For some..

IndustryChat

How to Make Hyper-Casual Games, Part 2: Trends

In this episode of LevelUp, Tom Kinniburgh, a mobile consultant for game studios, who also runs the popular blog Mobile Free to Play, talks about the history of hyper-casual, monetization and user-acquisition best practices, working with publishers, and more. This episode is part two of a three part series focused on designing, monetizing, and marketing..

IndustryChat

Highlights from IAB’s Opt-in Value Exchange Advertising Playbook for Brands

Opt-in ads The IAB just released the ultimate playbook for brand advertisers. The first guide of its kind, it gives brand advertisers all the information they need when it comes to rewarded or opt-in ads. According to the IAB, “opt-in value exchange advertising is an ad format that is growing in importance … numerous studies..

Interested in becoming an ironSource partner?

We’d love to hear from you.