Google Cloud Dataflow is a cloud-based data-processing service for both batch and real-time data streaming applications.It allows developers to set up processing pipelines for the integrating, preparing and analyzing large data sets, for example in Web analytics or big data analytics applications.
On earlier Google parallel processing projects the Cloud Dataflow software expanded , including MapReduce, which originated at the company. Cloud Dataflow is designed to bring to entire analytics pipelines the style of fast parallel execution that MapReduce brought to a single type of computational sort for batch processing jobs. It’s based partly on MillWheel and FlumeJava, these two Google-developed software frameworks focused at large-scale data ingestion and low-latency processing.
Google Cloud Dataflow overlaps with competitive software frameworks and services.At the same time, Cloud Dataflow was made available on a limited basis as part of a controlled beta program. The first version of Cloud Dataflow is supported by a Java software development kit, with other language support to follow.
Publish-and-subscribe mode is supported in Cloud Dataflow which is from Google Cloud Pub/Sub middleware feeds or, in batch mode, from any database or file system.Using the format called PCollections you can customize the sizes and structures of data. PCollections is short for “parallel collections.” The Google Cloud Dataflow service also includes a library of parallel transforms, or PTransforms, which allow high-level programming of often-repeated tasks using basic templates; in addition, it supports developer customization of data transformations. The service optimizes processing tasks — for example, by reducing multiple tasks into single execution passes. And it supports SQL queries via Google BigQuery, a cloud-based analytics service.
Image reference :- https://cloud.google.com/blog/topics/developers-practitioners/dataflow-backbone-data-analytics
How does data processing work?
Three steps involved in data processing : You read the data from a source, transform it and write the data back into a sink.
- The data is read from the source into a PCollection. The ‘P’ stands for “parallel” because a PCollection is designed to be distributed across multiple machines.
- Transforms is the other operation performed on the PCollection. Each time it runs a transform, a new PCollection is created. That’s because PCollections are immutable.
- After all of the transforms are executed, the pipeline writes the final PCollection to an external sink.
Using Java or Python language, create your pipeline using Apache beam SDK. You can use Dataflow to deploy and execute that pipeline which is called a Dataflow job.
This Dataflow then assigns the worker virtual machines to execute the data processing, you can customize the shape and size of these machines. Dataflow automatically increases or decreases the number of worker instances required to run your job if your traffic pattern is spiky. Dataflow streaming engine separates compute from storage and moves parts of pipeline execution out of the worker VMs and into the Dataflow service backend. This improves autoscaling and data latency.
Dataflow governance
When using Dataflow, all the data is encrypted at rest and in transit. For the further secure data processing environment there will be some process like :
- Turn off public IPs to restrict access to internal systems.
- Leverage VPC Service Controls that help mitigate the risk of data exfiltration
- Use your own custom encryption keys customer-managed encryption key (CMEK)
Conclusion
Dataflow is a great choice for batch or stream data that needs processing and enrichment for the downstream systems such as analysis, machine learning or data warehousing. Dataflow brings streaming events to Google Cloud’s Vertex AI and TensorFlow Extended (TFX) to enable predictive analytics, fraud detection, real-time personalization, and other advanced analytics use cases.
For more details contact info@vafion.com
Follow us on Social media : Twitter | Facebook | Instagram | Linkedin
Similar Posts:
- No similar blogs
