What is the difference between “calculate” and “compute”? I assure you, we are not going to discuss such quintessential terms related to computing world, which might bore some of us, as it might have given the impression 😀 But this is something out of curiosity about the crux of what we are going to go through. So, the calculation involves an arithmetic process. Computation is involved in the implementation of non-arithmetic steps of the algorithm which actually brings things up to the calculation. You got the idea where I am going with this right? We can try to visualize every aspect of data processing stages from data collection, cleansing, processing and then transforming it through mathematical operations to map data into something which makes more sense i.e. “Insight“. But the intelligence used for such meaningful transformation used to be the human intervention which now can be “Artificial” as…
Tag: #clairvoyantsoft
Creating Custom Origin for Streamsets
Streamsets Data Collector: StreamSets Data Collector is a lightweight and powerful engine that streams data in real time. It allows you to build continuous data pipelines, each of which consumes record-oriented data from a single origin, optionally operates on those records in one or more processors and writes data to one or more destinations. Streamsets Origin Stage: To define the flow of data for Data Collector, you configure a pipeline. A pipeline consists of stages that represents the origin and destination of the pipeline and any additional processing that you want to perform. An origin stage represents the source for the pipeline. For example, this pipeline, based on the SDC taxi data tutorial https://streamsets.com/documentation/datacollector/latest/help/#Tutorial/Overview.html which uses the Directory origin, four processors and the Hadoop File System destination: Stremsets comes bundled with many origin stage components to connect with almost all commonly used data sources and if you don’t find one for your source system, don’t worry Streamsets APIs are…