If you don't find an answer, please click here to post your question.
Engineering Blog

Flume in Analytics Realtime Pipeline

by Community Manager ‎05-30-2017 10:34 AM - edited ‎05-30-2017 10:35 AM (736 Views)

Written by Swapnesh Gandhi 5/30/17

 

Why?

In early 2016, we were building a JSON API to send data to IQ, we have a realtime pipeline which powers to IQ dashboard page. We'd receive Thrift and JSON pings in our logger machines, we had a process called livelogsd pushing data to Kafka for thrift data, Livelogsd would tail the player log file and send everything in that file to Kafka. 

To support JSON API in IQ dashboard, we have two choices:

1. Make changes to Livelogsd to support JSON API.

2. Revisit the Livelogsd model and use an open source project for this purpose instead of Livelogsd.

Using an open source project like Flume has its advantages. We do not have to maintain our own code for a use-case that can be done with open source project which is better maintained.

Read more...

Code Coverage Dashboard

by Community Manager on ‎04-13-2017 08:13 AM (2,733 Views)

Written by Anupama Shetty 4/13/17

 

Code coverage as defined by Wikipedia refers to a measure used to describe the degree to which the source code of a program is executed when a particular test suite runs. Thus, serving as a metric to track the percentage of code lines having a corresponding test to validate its functionality. While code coverage itself is not a self sufficient metric and may not always indicate a healthy or well tested product, it is still a quick way to get a sense of the quality of a product.

At Ooyala, many of our products such as Analytics and Discovery are built using many micro services. As such, it is important for us to ensure each of these services are well tested for every new product release. We use code coverage metric as one of the basic metric to ensure these services are well tested by setting a minimum of 70% code coverage at unit test level. As each of these services were built for a specific need and requirement, they were developed using different programming languages such as Scala, Golang, Javascript or Ruby. Most of the programming languages used today, offer a code coverage tool that can be easily enabled for your code repository. We use Jenkins as our continuous integration environment and have jobs setup to trigger unit test runs for every new code change. These jobs will generate and publish the code coverage report showing the latest code coverage numbers for every Jenkins test run.

Read more...

Transcoding Media files using Cloud Brokering

by Community Manager on ‎03-13-2017 02:40 PM (825 Views)

Written by Maheshwaran G 3/12/17

 

Overview

Media streaming is taking the internet by storm by dominating the internet traffic and will continue doing so in the future too!  Hope everybody would agree with this statement without any element of doubt. Advancement in telecommunication, broadband internet access and enhanced coding technique that aids in video streaming services are becoming increasingly popular. The video files are generally transcoded before being delivered to the end users using different devices like desktop, tablet, smart phone, etc for consuming the content. The process of transcoding primarily involves a two step process for the content. One, decoding in which video data is converted to uncompressed format. Second, re-encoding where the data is converted to desired format that enables the content to be rendered across various devices. Traditionally, the video transcoding software is supposed to be installed in one system for transforming the video files which is more of a centralized approach where powerful server with higher bandwidth is deployed to perform the video/audio transcoding.

Read more...

Experience Building Mobile Test Automation Framework for Playback Mobile SDK (Android & IOS) Part 2

by Community Manager ‎02-08-2017 12:12 PM - edited ‎02-08-2017 12:36 PM (634 Views)

Written by Bindu Sondur 1/10/17

Software Development

 

 

Building the Test Framework and Test Scripts

 

 

Wrapping TESTNG  and MAVEN around Appium

 

The Appium test framework provides the flexibility to choose the test development ecosystem of our choice.  We opted for the Java ecosystem. The web automation framework for playback was already using TESTNG and Maven, and it was working quite well for our needs.

 

TESTNG provides the testing framework to manage test cases, test scripts, and test suites effectively (goal 3 from the previous blog) (http://testng.org/doc/index.html). MAVEN is the build automation tool which helps in managing test frameworks build and reporting (https://maven.apache.org/).

Read more...

Experience Building Mobile Test Automation Framework for Playback Mobile SDK (Android & IOS) Part 1

by Community Manager on ‎02-08-2017 11:41 AM - last edited on ‎02-08-2017 12:18 PM by Community Manager (551 Views)

Written by Bindu Sondur 1/10/17

Software Development

 

The first project I worked on at Ooyala was to build a robust, sustainable and scalable mobile test automation framework for playback SDK (Android and iOS) as part of the playback automation team. Because we were starting with a clean slate, we initially came up with a few goals of what we expect the framework to do.

Read more...

A Rock Band of Engineers

by Community Manager on ‎02-08-2017 11:00 AM (384 Views)

Written by Marcus Hamrin, published 7/12/16

Software Development

I’ve played music for about 25 years, which is most of my life. While playing some pretty obscure genres, I still consider myself musically agnostic; if it sounds good, it is good, regardless of genre. Period. When I’ve been part of creating music in a band, it has mostly been a very collaborative process. Depending on the people in the band, the process has been wildly different. Some groups of people prefer to have a starting point, a reference to guide them towards a finished song, a rough idea they can work on. Sometimes it ends up completely different than anticipated by the one providing the draft, but everyone has agreed it turned out to the better. Other groups go for complete collaborate improvisation; jamming all night until something sticks, and eventually there is a song that everybody considers complete. Yet another group of people goes for creating an entire set of songs, then rearrange them into new ones.

Read more...

Experience building an OLAP engine on Spark and Cassandra

by Community Manager ‎02-08-2017 10:54 AM - edited ‎02-08-2017 10:56 AM (1,228 Views)

Written by Tiong Lee 6/1/2016

Systems and Architecture

 

 

Summary

 

Recently Ooyala has launched its new OVP analytics product, IQ, as a standalone product offering. The product is powered by our in-house OLAP system which is built on top of Spark and Cassandra technology. This article discusses the experience of developing the system and some lessons learned from the development of the OLAP engine for IQ.

Read more...

Mixing up Vagrant in Test-Kitchen

by Community Manager on ‎02-08-2017 10:43 AM (485 Views)

Written by Karen Bruner 3/31/2015

Systems and Architecture

 

 

The Problem

 

We had a use case in test-kitchen where one of our end-to-end cookbook tests installed so many packages and support tools, it was filling up the 10Gb root disk of our custom VirtualBox images. Rather than make larger images, since most groups were not having the same issue, it made more sense to add more disk space when needed to the Vagrant/VirtualBox instances.

Potential use cases other than simply needing more disk space include testing recipes that create RAID devices, logical volumes, filesystem creation and manipulation, and so on.

Read more...

Ooyala-Sponsored Robotics Team Wins Championship

by Community Manager on ‎02-08-2017 10:38 AM (422 Views)

Written by Patrick Fairbank 8/20/2014

Culture

 

 

At Ooyala, we believe that investing in the next generation of science, technology, engineering and math leaders is important. For this reason, we sponsor a high school robotics team, Team 254, “The Cheesy Poofs” from San Jose.

Read more...

Keeping Secrets with Chef

by Community Manager on ‎02-08-2017 10:30 AM (1,036 Views)

Written by Karen Bruner 6/26/2014

Systems and Architecture

 

 

Chef @Ooyala

 

Here at Ooyala, we are managing more and more of our server and application configurations with Chef. Chef is a platform which allows users to manage their servers programmatically, thus providing great power and flexibility in customizing configurations on the fly, depending on various attributes of each target server.

Far beyond simply writing configuration files, Chef can easily configure the operating system, install and configure applications, start and stop them in given situations, and more, using "recipes" or specialized scripts written in the Chef DSL, an extension of Ruby. The recipes are organized into "cookbooks," collections of recipes generally related to a single service. When a client server contacts its chef server, the server authenticates via the client's key and sends the client its run list, the recipes it needs to execute. Each client's specific attributes then determine how each recipe behaves on that client.

Chef's power and flexibility allows all Ooyala systems, both physical and virtual, across multiple OS platforms, application roles, development environments, and geographical regions to be managed using the same set of cookbooks. Configuration becomes increasingly automated as we write or modify cookbooks to manage our applications.

Read more...

Chrome Debugging Extension for the Ooyala Player

by Community Manager on ‎02-08-2017 10:26 AM (307 Views)

Written by Dustin Preuss April 8th, 2014

Software Development

 

 

The Trouble

 

Here at Ooyala, we’ve developed a highly configurable and extensible media player. It’s an intricate system, and intricate systems can be intricate to debug at times. Every player can have a myriad of modules and settings configured, and a customer can set up as many players as they like. As you can imagine, this can lead to issues only reproducing in very specific conditions. So during a hackathon at Ooyala, a team put together a shiny new debugging tool. What they made is a Google Chrome extension that takes a lot of the complexity out of what’s needed for basic debugging and allows us to tune what it provides to make our triage of the problem faster and more efficient.

Read more...

Using Parquet and Scrooge with Spark

by Community Manager ‎02-08-2017 10:21 AM - edited ‎02-08-2017 10:22 AM (612 Views)

Written by Evan Chan 2/28/2014

Software Development

 

 

Parquet is an exciting new columnar HDFS file format with built-in dictionary encoding and compression, as well as the ability to only read the columns you care about. These features make it very suitable for high-performance OLAP workloads with query engines such as Cloudera Impala or Facebook Presto.

Parquet translates each row into an object when reading the file, and it natively supports Thrift and Avro schemas for your data. Scrooge is Twitter's Scala class generator for Thrift, making it much more convenient and idiomatic to work with Thrift structs in Scala.

This blog post will lay down the steps for reading Parquet files using Scrooge-generated Scala classes for Thrift schemas, in Apache Spark.

Read more...

Comparing Scala JSON Libraries

by Community Manager on ‎02-08-2017 10:15 AM (2,686 Views)

Written Evan Chan 1/17/2014

Software Development

 

We were working on a Spark job to read JSON files out of HDFS, and it seemed to be running way too slowly. It turns out it was the JSON parsing library. So, here are some notes to help others navigate the Scala JSON parsing landscape, where there are at least 6 different libraries -- on both performance and correctness.

Read more...

Open sourcing our Spark Job Server

by Community Manager on ‎02-08-2017 10:03 AM (552 Views)

Written by Evan Chan 12/19/2013

Software Development

 

 

Here at Ooyala, we have been investing heavily into Spark. We wrote a REST server for submitting, running, and managing Spark jobs and contexts. From the beginning, we thought about the job server as a generic infrastructure piece that could be open sourced, that seems like many organizations could benefit from. At the just concluded Spark Summit 2013, we happily announced that we had started the open sourcing of our job server (video of our talk). The pull request is available for anyone to review.

Read more...

Stop Bashing on Tarballs in Chef

by Community Manager on ‎10-15-2015 09:55 AM (955 Views)

Written by karen Bruner
Systems and Architecture

 

Stop Bashing on Tarballs in Chef

 

 

The Problem

 

Tarball handling in Chef recipes can lead to multiple problems. Sometimes it becomes a gateway drug to bad practices. "We had to use a bash block to call the system tar because there's no native tar resource. And that worked well, so we just decided to put more stuff in bash blocks." Next thing you know, the entire recipe is one big bash 'everything' do ... end. Or a recipe will untar some files, then try to modify them using the Chef file resource only to throw an error during the client run about how the file doesn't exist, because when Chef went to create the resource at the beginning of the run, the file really didn't exist.

Read more...

Mixing up Vagrant in Test-Kitchen

by Community Manager ‎10-15-2015 09:43 AM - edited ‎10-15-2015 09:45 AM (503 Views)

Written by Karen Bruner
Systems and Architecture

 

 

The Problem

 

We had a use case in test-kitchen where one of our end-to-end cookbook tests installed so many packages and support tools, it was filling up the 10Gb root disk of our custom VirtualBox images. Rather than make larger images, since most groups were not having the same issue, it made more sense to add more disk space when needed to the Vagrant/VirtualBox instances.

Potential use cases other than simply needing more disk space include testing recipes that create RAID devices, logical volumes, filesystem creation and manipulation, and so on.

 

Read more...

Fast Spark Queries on In-Memory Datasets

by Community Manager on ‎07-11-2013 09:39 AM (971 Views)

Written by Evan Chan
Systems and Architecture


Here at Ooyala, we process over two billion video events a day, all of which are archived in a big Cassandra cluster. The challenge is, how do we turn the mountain of raw events into small, actionable nuggets of truth? And how do we make the development and querying of these insights quick, scalable, and easy?

Evan Chan
  • Systems and Architecture

Here at Ooyala, we process over two billion video events a day, all of which are archived in a big Cassandra cluster. The challenge is, how do we turn the mountain of raw events into small, actionable nuggets of truth? And how do we make the development and querying of these insights quick, scalable, and easy?

- See more at: http://engineering.ooyala.com/blog/fast-spark-queries-memory-datasets#sthash.OxBB9c9V.dpuf
Read more...

Open sourcing metrics-storm

by Community Manager on ‎05-08-2013 07:15 AM (870 Views)

Evan Chan
Software Development
 

Here at Ooyala, we use Twitter Storm to compute real-time video metrics for our Ooyala Now product. When working with a high-performance distributed system, gaining visibility and metrics into the production system without significantly impacting performance is crucial to troubleshooting and operations.

Read more...

How to create XKCD-style charts using Javascript and D3

by Community Manager on ‎04-02-2013 09:27 AM (737 Views)

Noah Gibbs
Software Development

 

XKCD is a great web comic about “romance, sarcasm, math, and language.” It also features some great writing (and stick figures).

 

 

Read more...

Staying Up When Amazon Web Services Isn’t

by Community Manager ‎01-31-2013 07:15 AM - edited ‎01-31-2013 09:40 AM (998 Views)

Over the holidays you may have noticed that many of your favorite websites experienced outages, due to incidents impacting Amazon Web Services. Ooyala has been hosting much of our infrastructure on Amazon's platform since we won the first ever AWS Startup Challenge in 2007.
Read more...

Storm and Other Awesomeness

by Community Manager ‎01-03-2013 07:31 AM - edited ‎01-03-2013 07:33 AM (710 Views)

We hosted a Storm meet up a bit ago here at Ooyala HQ to discuss real time computation with the open source community. Nathan Marz, the creator of Storm, gave a talk and was kind enough to let us interview him too.

 

Ooyala also just got recognized as one of the top 5 startups to work for in Silicon Valley.

Read more...

Safe Deletion in Amazon S3

by Community Manager ‎12-06-2012 09:40 AM - edited ‎12-07-2012 03:01 PM (1,810 Views)

Written by Patrick Fairbank
 
 

At Ooyala, we store a lot of large files in Amazon S3. When a customer wants to host a video with us, they upload it to our servers and we store a copy in S3. Our distributed transcoding system converts the source video into the various formats, resolutions, and bitrates required for streaming, and all the resulting files are also sent to S3. In total, we have on the order of several hundred terabytes of video data stored.

Read more...

Ooyala's Culture of Innovation

by Community Manager ‎12-06-2012 09:38 AM - edited ‎02-08-2017 09:53 AM (763 Views)

Written by Evan Chan
Culture
 

The innovative culture here in Silicon Valley has received extensive press, especially amongst startups. In particular, hackathons are popping up everywhere. Award-winning software company Atlassian has a "FedEx" day where, once a year, teams cobble together innovative projects for 24 hours.

Yes, Ooyala has a Hackathon too. It goes for 30 hours, has great prizes, and it is a huge amount of fun for everyone involved. Themed hackathons can even be used to generate ideas about saving COGs, for example; but what happens when it is over?

What is innovation anyways? Has it become an annual event? I'd like to take you on a tour of innovation at Ooyala.

Read more...

Ooyala Discovery Leveraging Five Year Investment in Big Data

by Community Manager ‎12-06-2012 09:35 AM - edited ‎03-09-2017 10:33 AM (1,051 Views)

Written by Zhichen Xu
Systems and Architecture
 

With a mission to personalize every screen, we have released the Ooyala Discovery product. At the core of this release is a set of recommendation algorithms. Recommending video content poses some significant technical challenges: Unlike for text-based content, video keyword generation is manual and therefore quantity and quality can vary widely across providers. Techniques based keywords alone are not sufficient. In the online world, content can quickly become popular, and then just as quickly cool down.

Read more...