Running Ad Exchange at 100K+ QPS?—?System Design

This blog is second in a series of blogs written on building a highly scalable, fault tolerant and robust Ad Exchange. In this blog I will focus on overall System Design and Architecture of InMobi Exchange (IX). If you got directly here, please do refer to the first blog of the series focusing on the Infrastructure aspect of the InMobi Exchange?Running Ad Exchange at 100K+ QPS — Infrastructure.

First let us start with the basics of an Ad Exchange the Wikipedia the definition of an Ad Exchange is :?“An?ad exchange?is a technology platform that facilitates the buying and selling of media advertising inventory from multiple ad networks. Prices for the inventory are determined through bidding. The approach is technology-driven as opposed to the historical approach of negotiating price on media inventory. This represents a field beyond ad networks as defined by the?Interactive Advertising Bureau (IAB).”?From the perspective of IX we get supply from our publishers and based on the attributes of the supply (User profile, OS, location, Ad-format etc.) we choose one or more partners and send them the request in Open RTB (Real Time Bidding) format. Our partners are either other exchanges or DSP (Demand Side Platforms). After getting an Ad Request they will either bid with a price and creative or choose not to bid at all. IX waits for a fixed time for all the partners to respond and then conducts a first/second price auction to choose a winner which is send back to the publisher. Now let discuss the various components which make an Ad Exchange.

Admin Portal:?As discussed above not all of our partners are interested in getting all of our supply, and they are very choosy in their bidding as they have to match the supply with their pool of demand. The Admin Portal is UI written on?Django?and helps our partner managers in configuring the right supply for our partners. This portal is also used for configuring PMP (Private Marketplace) and deals with our partners.

Microservices:?We are the advocates of microservice architecture and follow it diligently within InMobi. We primarily use?Apache Thrift?to design our microservices and are write it in Java, but the consumers of these services use clients in any language they prefer. The set of microservices Exchange owns is to power the Admin Portal, Creative Management, Client Mailers, VAST Validation API’s and many more. Admin Portal and microservices?reside only in one main data center of InMobi. Most of our microservices uses Spring MVC and Hibernate.

Data Store:?The Data Store serves as the persistence layer for the finite sets of data we have. This DB connects to the microservices and holds config and metadata about partners, publishers, billing and many more. The database is a?PostgreSQL?DB and follows Master-Slave architecture for being fault tolerant and load balancing. This Data Store is one of the components which is?replicated in all our data centers?for Ad Exchange servers to connect to. The Ad Servers never performs any write operations in this DB. Even the reads happen via local caches in Ad Server JVM. These caches are updated via an InMobi proprietary library called as Repositories. The cross colo and master slave replication happens via Slony. Slony is an asynchronous master-slave replication system for the PostgreSQL DBMS, providing support for cascading and failover. Asynchronous means that when a database transaction has been committed to the master server, it is not yet guaranteed to be available in slaves. Cascading means that replicas can be created (and updated) via other replicas, i.e. they needn’t directly connect to the master.

User Store:?InMobi has an user base of over 1.5 Billion. Storing, lookup and updates in such a huge audience personas in relational Databases is not scalable. So we chose to use?Aerospike?to do this job. Each data center at InMobi has its own Aerospike cluster for the Ad Exchange Servers to connect to. The asynchronous data sync between data centers is taken care by aerospike. Aerospike is a next-gen NoSQL database built from scratch in ‘C’ to scale out on commodity hardware. Aerospike is the database of choice for Ad-tech companies and is:

• An?operational?NoSQL database with simple real-time analytics capabilities built in.
• A fast?key value store?with support for complex objects, user defined functions and distributed queries.
• The first?in-memory?NoSQL database optimised for flash (indexes in DRAM and data in DRAM or natively stored on flash devices) with dramatic price/performance benefits.
• A?distributed?database with a shared-nothing,?clustered?architecture and single row?ACID?properties. Aerospike ensures high availability, replicating data synchronously within clusters and asynchronously across data centers.

Ad Exchange Platform:?The hero of the story, all other components are here to help the Ad Exchange Platform do its job without any hiccups. The Exchange is a Netty based NIO server having its presence in all our DC. And each of our DCs hold hundreds of Exchange containers (yes, you guessed it right all our components are deployed as?Dockers). The Exchange is an Real Time Bidding Platform for Demand (RTBD). This platform enables?DSPs?and other?Exchanges?to connect with InMobi’s supply and participate in real time bidding. Following are the primary functions of the Ad Exchange Platform:

• Enrichment of supply with user, geo etc. contexts. We use Aerospike user store to do a fast lookup. The exchange also finds if a supply is eligible for PMP (Private Marketplace). We use?CQEngine in?package and deals selection. CQEngine (Collection Query Engine) is a high-performance Java collection which can be searched with SQL-like queries, with extremely low latency.
• Load all publisher and partner level rules from PostgreSql in local cache (repositories) and do a supply to partner/s matching. The supply is then forwarded to all the selected partners as an?Ad Request?in?OpenRTB 2.5?format.
• Exchange runs First/Second Price auction for all the Ad Responses received within about a quarter of a second and send back the response to the publisher. In cases where none of our partners respond, we send a No Fill back to the publishers.
• Post response sending the exchange logs all data for reporting, winning creatives for post-serving validations against fraud. Exchange also does an important jobs of abstraction between the supply and demand, keeping away supply/demand jazz with either sides.

Alerting and Monitoring:?Discussed in the first blog of the series. Refer to this link:?Running Ad Exchange at 100K+ QPS — Infrastructure.

Data Pipelines:?Every ad which gets served and renders on the device will send events (render, click, download, video play, video quartiles completed etc.) to InMobi?Event Servers.?These events flows to?HDFS?via?Kafka.?From there we run various?Spark?and?Pig?jobs joining it with the Exchange Server logs (emitted by Exchange server on to HDFS via Kafka). The output of these jobs are enriched summaries which is also stored back to HDFS. These jobs run on local as well as global clusters.

Reporting:?The HDFS summaries generated above are exposed to consumers via?Hive.?Apache Hive is a data warehouse software project built on top of Apache Hadoop for providing data summarization, query and analysis. Hive gives an SQL-like interface to query data stored in various databases and file systems that integrate with Hadoop. Traditional SQL queries must be implemented in the MapReduce Java API to execute SQL applications and queries over distributed data. Hive provides the necessary SQL abstraction to integrate SQL-like queries (HiveQL) into the underlying Java without the need to implement queries in the low-level Java API. As you might have guessed getting data via hive is slow as they spin up MapReduce jobs to fetch the data. So for data which is more frequently used, which will be exposed via reporting API and which will power internal dashboards we load the HDFS summaries in another OLAP data warehouses —?Vertica?and?Druid.?Druid is a column-oriented, open-source, distributed data store written in Java. Druid is designed to quickly ingest massive quantities of event data, and provide low-latency queries on top of the data. At InMobi Vertica/Hive is not exposed to end users directly. We use one of InMobi incubated Apache project called?Lens?to expose data. Lens aims to cut the Data Analytics silos by providing a single view of data across multiple tiered data stores and optimal execution environment for the analytical query. It seamlessly integrates with Hadoop, Hive, columnar and traditional DWH and provides an Unified Analytics interface.

Reporting API:?This module provides the Reporting API to our demand partners (DSPs, Exchanges). It is a?NodeJS?server, which fetches data from Lens, uses?Redis?to cache those queries & data and exposes it to consumers via?GraphQL.?Reporting API internally uses authentication services to authenticate the API gateway. The consumers of reporting api can integrate directly from?exchange.inmobi.com/api?or can login to their dashboards at?exchange.inmobi.com?. Following is a screen grab from the exchange portal which is powered by?React?and?Apollo Meteor.

Demand/Ad Quality:?One of the biggest challenge we face in exchange world is about Ad Quality. Since we do not own the creative nor are they ingested in our systems prior to serving it gets very nasty sometime when the served ad auto redirects to some malicious website, or generates auto click and popups. Even violations WRT to ad policies can happen many times. To handle these situations we log all the winning creatives from Exchange Servers and these creatives undergo following checks. And while serving along with the status of the creative we consider the partner score to decide which creative should dropped.

• Load each creative in a headless Firefox browser connected to a virtual display using?Selenium.?Let the whole DOM load and extract all possible data out from there. Also block all creatives which show some malicious intentions.
• Verify all the domains which the creative connects to via?Google Safe Browsing API. If any of the domains is flagged here, we block all the creatives having that domain.
• All the creatives is sent to?GeoEdge?for 3rd party verification and blocked if flagged by GeoEdge.
• Whatever creative is not blocked by above methods goes through InMobi Adops team. This team is a 24x7 team which manually checks each and every creative and then approve/reject it.
• All the creative parameters along with results of above jobs are used as training set for a?Machine Learning?model build solely to predict a bad ad. We run?Random Forest and?Logistic Regressions?algorithms in the ML Model. And the creatives flagged by these models are also blocked before manually checking it.

I hope you enjoyed reading this blog if you have made it till here. Leave behind your comments and valuable suggestions. Also do mention which component/s you would would like me to dig deeper into.