Customer Case Study

Crowdworks Chooses FlyData to Support Their Growing Data Analysis Operation

“The fact that the data can be shown almost in real time in the analysis infrastructure was a very important reason why we chose FlyData” – Koichiro Ohba, CTO of Crowdworks

Crowdworks is one of the largest crowdsourcing service platforms in Japan. More than 700,000 companies have used the service to find on-demand workers, including many listed on the Tokyo Stock Exchange. In 2012, Crowdworks was named to the list, “100 Next Generation Ventures that will Save Japan” by Nikkei Business.

Crowdworks relies on FlyData — along with Amazon Redshift and Domo — to power their robust Business Intelligence operation. We spoke with Crowdworks CTO, Koichiro Ohba, to learn more.

# Challenge: Rapidly-Growing Volumes of Data

As a marketplace for talent, Crowdworks generates massive amounts of user-generated data. The company’s Business Intelligence (BI) team models and analyzes that data to help executives make better business decisions.

As Crowdworks grew rapidly, so did the volume of data that needed to be stored, secured, and analyzed. Soon, Ohba realized they needed a change.

“In view of increases in data volume, it was felt that the limit had been reached, unless the data analysis [operation] could be expanded,” said Ohba.

In order to continue their expansion, Crowdworks would need a new big data solution.

# Build vs. Buy: Choosing Off-the-Shelf Big Data Tools

Ohba made his shopping list. Crowdworks would need three tools: Data storage, BI visualization, and an ETL pipeline.

Ohba and his team first considered a homegrown big data solution using an open source platform like Hadoop. However, after an in-depth analysis, Ohba determined they could accomplish their goals with off-the-shelf tools. This decision would free up his team for business-critical work.

“We wanted to concentrate on developing and providing a crowdsourcing service, rather than developing and operating an analysis infrastructure,” said Ohba.

# Crowdworks’ New Stack: Amazon Redshift, Domo, and FlyData

Crowdworks first decided on two tools: Amazon Redshift as their data warehouse, and Domo for BI visualization.

“By storing data on Amazon Redshift and using Domo for real-time analysis, we designed it so that batch processing could be carried out, separately,” said Ohba. “This remedied the problem we previously had of batch processing taking too much time for the service and for MySQL aggregate processing.”

However, Crowdworks still needed an ETL pipeline solution. That’s when Ohba and his team discovered FlyData.

“The fact that the data can be shown in near real-time in the analysis infrastructure was a very important reason why we chose [FlyData],” said Ohba.

With the help of Redshift, Domo, and FlyData, Crowdworks has continued their rapid expansion. The company has grown from 225,000 company clients in 2018 to over 700,000 today.

They have continued working with Japan’s largest companies, including those listed on the Tokyo Stock Exchange. The only difference: Crowdworks is now listed on the Exchange as well.

More case studies
Crowdsourcing, Internet
When needed a new ETL-as-a-Service provider, Nick Smith and his team launched an in-depth review of top options, including FlyData and Fivetran. After an extensive search, they chose FlyData for its reliability, ease-of-use, and excellent technical support. And they haven't looked back.
Brand Reviews, News
“We have made massive improvements to our business thanks to FlyData enabling us to have our data available on demand in Redshift.” – Ron Timoshenko, Director of Architecture of ConsumerAffairs
E-Commerce, Crowdsourcing
"We needed to move data from MySQL to Redshift and we needed to move it around as fast as possible. FlyData does that, and it does that well." – Gareth Tilley, Head of Data Analytics at 99Designs
View all case studies
Get started. Try FlyData.
Quick setup. No credit card required. Unlimited sync during trial.
Fast and secure Enterprise-grade security and near real-time sync.
World-class support Proactive monitoring from technical experts, 24/7.
Straightforward pricing Pay for the rows you use, and nothing you don’t.