Crowdworks is one of the largest crowdsourcing service platforms in Japan. More than 700,000 companies have used the service to find on-demand workers, including many listed on the Tokyo Stock Exchange. In 2012, Crowdworks was named to the list, “100 Next Generation Ventures that will Save Japan” by Nikkei Business.
Crowdworks relies on FlyData — along with Amazon Redshift and Domo — to power their robust Business Intelligence operation. We spoke with Crowdworks CTO, Koichiro Ohba, to learn more.
# Challenge: Rapidly-Growing Volumes of Data
As a marketplace for talent, Crowdworks generates massive amounts of user-generated data. The company’s Business Intelligence (BI) team models and analyzes that data to help executives make better business decisions.
As Crowdworks grew rapidly, so did the volume of data that needed to be stored, secured, and analyzed. Soon, Ohba realized they needed a change.
“In view of increases in data volume, it was felt that the limit had been reached, unless the data analysis [operation] could be expanded,” said Ohba.
In order to continue their expansion, Crowdworks would need a new big data solution.
# Build vs. Buy: Choosing Off-the-Shelf Big Data Tools
Ohba made his shopping list. Crowdworks would need three tools: Data storage, BI visualization, and an ETL pipeline.
Ohba and his team first considered a homegrown big data solution using an open source platform like Hadoop. However, after an in-depth analysis, Ohba determined they could accomplish their goals with off-the-shelf tools. This decision would free up his team for business-critical work.
“We wanted to concentrate on developing and providing a crowdsourcing service, rather than developing and operating an analysis infrastructure,” said Ohba.
# Crowdworks’ New Stack: Amazon Redshift, Domo, and FlyData
Crowdworks first decided on two tools: Amazon Redshift as their data warehouse, and Domo for BI visualization.
“By storing data on Amazon Redshift and using Domo for real-time analysis, we designed it so that batch processing could be carried out, separately,” said Ohba. “This remedied the problem we previously had of batch processing taking too much time for the service and for MySQL aggregate processing.”
However, Crowdworks still needed an ETL pipeline solution. That’s when Ohba and his team discovered FlyData.
“The fact that the data can be shown in near real-time in the analysis infrastructure was a very important reason why we chose [FlyData],” said Ohba.
With the help of Redshift, Domo, and FlyData, Crowdworks has continued their rapid expansion. The company has grown from 225,000 company clients in 2018 to over 700,000 today.
They have continued working with Japan’s largest companies, including those listed on the Tokyo Stock Exchange. The only difference: Crowdworks is now listed on the Exchange as well.