Datacratic CEO James Prudhomme & Co-founder & CTO Jeremy Barnes, Breakdown Their Open-Source Bidder, RTBkit
<a href="http://www.exchangewire.com/blog/2013/04/02/james-prudhomme-ceo-jeremy-barnes-founder-cto-datacratic-breakdown-their-open-source-software-RTBkit/datacratic-6/” rel=”attachment wp-att-25129″>
It has long been noted that technology itself is no longer a differentiator; but strategy and execution is what paves the way to competitive advantage. To that end, Datacratic recently unveiled the first phase of their vision to help create an open-sourced ad tech environment.
ExchangeWire caught up with James Prudhomme, CEO & Jeremy Barnes, CTO to determine what inspired the project to bring to market an open source bidder solution, how companies can begin integrating and where businesses should be looking to provide value in the ecosystem as underlying technology becomes increasingly commodotised.
What is an open source bidder project? What is RTBkit?
RTBkit is a software framework which is meant to make the task of engineering and building a real-time bidder a lot faster and easier. RTBkit includes approximately 65,000 lines of code and includes most of the major components required to assemble a buy-side DSP stack. Open source, in this case, effectively means free software. There are no license fees or costs of any kind. Anyone can download the code and set about building their own bidder.
So far we have offered a technology preview release. A 0.9 release of RTBkit with sufficient functionality for full-scale production use is targeted for mid-April. We’ll be adding more and more features and functions over the next several months and expect to have a 1.0 release in place by mid-summer.
We refer to it as a project because we hope and expect that a community will evolve around RTBkit. As companies adopt the framework as the core of their RTB stack we hope that they will contribute back to the project and make it better. We also hope that wide-scale adoption encourages third-parties to offer additional products and services to complement the core code base.
What drove Datacratic to launch this as an open source, community-driven project?
There were a number of factors, but I think three stand out. First, we had predicted that in the next two to three years there would be hundreds, if not a thousand, new bidders connected to various exchanges around the world. We wanted to accelerate that and we wanted to take the focus off of the relatively mundane and mechanical parts of the RTB stack and redirect the focus to data-driven optimisation and algorithmic bidding strategies.
Next, we wanted to make it easy for our customers to work with us. Datacratic’s primary focus is to abstract the optimisation layer from the real-time marketing ecosystem and to make big-data and machine-learning accessible to as many customers as possible. RTBkit is a way to make it easy for customers to integrate our Real Time Bid Optimisation product. They can either build their entire RTB stack on the open source framework, or if they already have a bidder, RTBkit can serve as the connector between their stack and our optimisation platform.
Finally, and in a very altruistic sense, we wanted to accelerate the pace of innovation within ad tech, and specifically within RTB. We felt the best way to do that was via an open source strategy which we hope will draw in engineers and developers from around the world to contribute to the project. This means Datacratic can focus entirely on productising our machine learning platform while ensuring RTBkit gets stronger and that innovation can flourish around the edges.
Do you believe that this will bring a degree of disruption to the space?
I’m not sure if disruption is the word I would use, though I’m sure a lot of people will see it that way. We prefer to think of it as acceleration. We see the proliferation of advertising exchanges and real-time bidders as inevitable; I don’t think anything can stop that.
By releasing RTBkit we have simply made RTB technology more accessible and hopefully have inspired more companies to build their own bidder and connect directly to the exchanges. That may cause some of the more established players in the space to modify their business plans somewhat, and may cause them to accelerate certain aspects of their product roadmap, but all-in-all, RTBkit is not designed to compete directly with the established platforms, but rather to complement them by encouraging a larger ecosystem of bidders, many of whom will connect to inventory sources through some of the larger platforms.
In the end, I do not think RTBkit will dramatically alter the path of RTB, it would have continued to march on with or without us. That said, we would like to think that we have widened the path somewhat and made room for more players to join the game.
Can you outline the mechanical structure of RTBkit? How does it connect to other components, owned or otherwise, by Datacratic?
The image below illustrates a very high level architecture of the framework and how it connects to the outside world. We have already seen great traction with the exchanges and SSPs building connectors into the RTBkit core. For example, Rubicon Project started working on theirs immediately after launch. Many of the leading exchanges and SSPs from both North America and Europe have since made commitments to supporting RTBkit and have started building connectors which they intend to open source over the coming weeks.
We have also seen keen interest from data providers, analytics and reporting solutions, as well as systems integrators and custom software developers. The primary objective for companies that become part of the ecosystem is to make it extremely easy for people who adopt RTBkit to begin licensing the third-party products and services which are connected to it.
At Datacratic we offer a Real Time Bid Management and Optimisation product, which seamlessly plugs into RTBkit. Our primary business objective with RTBkit is to create lots and lots of customers for our optimisation product. Our system handles all aspects of RTB campaign management and delivers unique data-driven bidding strategies, powered by sophisticated machine-learning algorithms. Our product includes a set of APIs which allow our customers to push campaign-level information (i.e. from their ad server) like budgets, flight dates, creative tags, filtering criteria etc. into the system.
One of the most unique and innovative aspects of RTBkit is the augmenter component. An augmenter can be built easily and can be used to append data directly to the bid request in real time. This allows Datacratic’s Bid Optimisation system to use that data as a feature into the algorithmic model and bring the data directly to the bid decision in real-time.
The sky is the limit here: bid requests can be augmented with user-segment data, contextual data at the url level, geographical data… just about anything you can imagine. Everyone seems to be talking about weather-triggered advertising; well this is how to make it happen. An augmenter can truly offer a bridge between both online and offline data and RTB.
We expect that DMPs, data providers and other third-parties will build and maintain augmenters in order to make their products and data easily available to RTBkit users.
Most importantly, this architecture allows for users of RTBkit to add value via their own first-party data assets and deep domain knowledge, without requiring them to build a full bidder stack and campaign optimisation logic. Buyers can bring their audience and their deep understanding of their unique characteristics. Datacratic’s optimisation system lets them run with the real-time targeting and scale of RTB.
We feel that this truly enables a ‘bring your own optimisation’ approach by allowing differentiation based on domain knowledge, and on a scientific approach to data-driven bidding, all powered by our Bid Management and Optimisation product.
On the other side of the equation, we have the data logger interface. These APIs can be used to push data into any sort of reporting and analytics system. We have seen good uptake here. For example, Ad Ternity has already begun to develop an interface to their Data Haptics system.
Ultimately we hope to see lots of new and innovative ideas grow up around RTBkit. That is the true mark of success for an open source project.
If the process of building a bidder is now made somewhat easier, how should companies be thinking about infrastructure and ongoing upkeep costs? How easy will it be to scale?
We are using RTBkit within Datacratic to support some fairly large deployments, and so it has been designed to scale. We will be rolling out improvements over the next few months that support horizontal scalability, both within and across data centres, as well as high-availability features. The underlying architecture is designed to scale out very naturally: as soon as you configure an RTBkit machine to point to an installation, that machine will automatically join the others and begin processing bids. It is also possible to scale out different parts of the stack independently: you can scale out exchange listening capacity, augmentation and bid-calculation capacity separately to support a given use case.
In terms of infrastructure, RTBkit has been designed to maximise ROI by running on high-end commodity servers and will make full use of all available CPU power. As a rule of thumb, a high-end server can deal with about 20,000 bid requests per second, under a reasonable campaign load; the exact number depends upon the number of campaigns and creatives, the complexity of the filtering logic and the number of bids that are received per bid request. It has also been designed to gracefully shed load when it is overloaded, which means that it is not necessary to commission extra idle capacity as a contingency against infrequent peaks in supply.
RTBkit does not come with a bundled monitoring system; this is something that we leave up to the user so as to allow for tight integration with existing systems. We do, however,log very comprehensive health statistics to Graphite; at Datacratic we use these statistics to feed our monitoring system.
If the logic of the bidder becomes more configurable via RTBkit, can you outline certain examples of how different data inputs can be leveraged that would make instances of RTBkit bespoke to the end user(s)?
Sure. Earlier on I mentioned the idea of weather data. Let’s take that a step further and consider the broader notion of contextual location information. In the example of a mobile bid request, you often see the latitude/longitude of the user who initiated the request.
Let’s say you are a mobile data provider and you have developed an innovative system where you can map latitude/longitude to a specific location; let’s say a sports arena. RTBkit makes it very easy for you to take that unique and innovative data processing capability and deliver it directly to the RTB stream. In this example you could chose to maintain that as a proprietary system or you could make the enhanced geolocation data easily available to other users of RTBkit.
There are an endless number of ways with which RTBkit can be used to develop proprietary systems that rely on unique data flows. This is precisely our goal with RTBkit. We have offered the core mechanics of an RTB stack under an open source license so that companies can focus entirely on unique, data-driven ways of buying ads from the exchange.
How easy will this be to integrate to inventory sources? Previously, it has been suggested, that the ongoing management and costs involved with managing integrations (when owning a bespoke bidding platform) with supply sources is non sustainable.
We have seen really keen adoption of RTBkit by the supply side. Many of the large exchanges, for example Rubicon Project, have built connectors into the system and will maintain these going forward. Ultimately, inventory suppliers should appear as an a la carte menu to RTBkit users. Keep in mind that RTBkit is fully compatible with the Open RTB protocol.
Previously, everyone who had built a bidder would need to build their own connections into each of the exchanges with which they wanted to work. Now the tables are turned, and the SSPs themselves can build and maintain a single piece of software which can be used by anyone who uses RTBkit. This dramatically reduces the friction in the process of getting buyers connected to exchanges.
What businesses do you expect to contribute to this project? Anyone who has previously adopted open RTB specs?
We hope that anyone who adopts or benefits from RTBkit in any way sees their way toward contributing back to the community. RTBkit supports OpenRTB, but it is not tied to that specification: we expect to see contribution from a much larger community than just those who have adopted OpenRTB.
Do you believe this will create a larger trend around open source ad tech much like we see in the infrastructure and data processing space? If ad technology continues to become commoditised, where will value continue to be added?
We certainly hope this is the start of a much larger trend toward open source within ad tech and digital marketing software. As you pointed out, many ad tech companies are already built on open source foundations like Hadoop, Apache, MySQL, PostgreSQL, Python and Ruby on Rails.
As the core software components become commoditised, and the barrier to entry into RTB is lowered, we fully expect all of the value to be created at the data-driven bid strategy layer. Our Augmenter and Bidding Agent interfaces were designed as a natural injection point for this kind of differentiated logic.
Where does mobile play a role in this?
OpenRTB has been very successful in the mobile space, and due to its support for OpenRTB, RTBkit is fully compatible with web, mobile and video exchanges. It can also be connected fairly easily to just about any flavour of mobile ad-server. We have seen a lot of interest from players in the mobile space and we expect there to be strong uptake.
What is next on the roadmap for RTBkit? Integrations with cloud-based providers such as Amazon?
One thing to remember is that RTBkit is not an out-of-the-box bidding solution. It’s a set of tools and parts that you can fit together to build one. To use RTBkit, you need to have exchange seats, an ad server, campaign management software, a strategy for bidding on those campaigns, a user data store, an ETL system, reporting technology and infrastructure on which to run it all. (Not to mention advertisers!)
That being said, our goal for RTBkit 1.0 is to package up and integrate enough open source technology (including RTBkit) to create an Amazon-hosted bidder box that can have you bidding in an afternoon with a test campaign on a popular exchange. As the ecosystem forms around RTBkit, we would expect that many providers (including Datacratic) will release branded Amazon images of RTBkit-compatible components that can be plugged together to create a fully featured, scalable bidder.
For more details, the RTBkit roadmap is available <a href="https://github.com/RTBkit/RTBkit-preview/wiki/Roadmap” target=”_blank”>here.
