“The fact that the data can be shown almost in real time in the analysis infrastructure was a very important reason why we chose Flydata”

– Koichiro Ohba, CTO of Crowdworks

Crowdworks uses FlyData Sync to replicate MySQL on Amazon Redshift for near real-time marketing analysis.

Crowdworks Service

Crowdworks is one of the largest crowd sourcing services company in Japan, with a mission to provide work styles for the 21st century. They assist businesses to expand based on three themes: regional revitalization; new work style support for women; and overseas development. More than 35,000 companies, many listed on the Tokyo Stock Exchange, have used Crowdworks’s services. This has resulted in jobs in excess of 12 billion yen (as of July 2014).

Crowdworks has achieved significant growth in the expanding crowd sourcing market. They were selected in 2012 for Nikkei Business’s “100 Next Generation Ventures that will Save Japan”. They have actively connected local governments and businesses in cooperative frameworks, working with Gifu Prefecture, Minamisoma City, and Fukushima Prefecture, as well as forming alliances with Yahoo, Benesse Corporation, and TV Tokyo, among others.

Challenges Manifested

Crowdworks has put into practice the various indices it has visualized and shared, with an emphasis on data-driven services since it began operations. “But in order to implement management system functionality which matches our services, analysis and services had to be interconnected,” says Koichiro Ohba, CTO of Crowdworks. “In view of increases in data volume and corresponding increases in indices, it was felt that the limit had been reached, unless the data analysis could be expanded so that it would not affect service.”

Considering a Solution

While looking into solutions necessary to create a loosely coupled analysis infrastructure, the management of Crowdworks realized that their service needed three elements: data gathering, data storage, and BI visualization.

“First of all, for storage, a crowd service that is capable of expanding, while verifying effectiveness and is able to start with minimal structure like that of Amazon Redshift, would have a high affinity to a venture business like ours,” says Ohba. The question regarding the use of Amazon Redshift was how to gather data from Amazon Redshift for the service.

“A solution was hit upon to use FlyData Sync’s function which synchronizes data from MySQL with Amazon Redshift,” explains Ohba. “Additionally, we decided to use Domo for visualizing how to provide data company-wide in nearly real time. We considered Hadoop and other analytical solutions we had developed to handle large scale data, but we wanted to concentrate on developing and providing a crowd service, rather than developing and operating an analysis infrastructure.”


Noticeable Results

Ohba points out that “by storing data on Amazon Redshift and using Domo for real-time analysis, we designed it so that batch processing could be carried out, separately. This remedied the problem we previously had of batch processing taking too much time for the service and for MySQL aggregate processing”

With FlyData Sync, from the DB update on the service side, the data is synced with Amazon Redshift after at most a five-minute delay. “The fact that the data can be shown almost in real time in the analysis infrastructure was a very important reason why we chose it,” says Ohba “Also, the marketing person in charge can analyze the data directly using Amazon Redshift since it uses MySQL.”

Amazon Redshift is admirably suited for aggregate processing. It’s able to simulate data collection and analysis using actual data and quickly connect it to the next processes.

FlyData handles real-time replication for Amazon RDS and Aurora, MySQL and PostgreSQL.

Get set up in minutes. Start uncovering data to make faster, better business decisions today.

By using this website you agree to accept our Privacy Policy and Terms & Conditions