![]() ![]() For this example, we need to add 1 day, 12 hours, 5 minutes, and 2 seconds to some date value we have. ![]() This Interval is used to determine a future date and comes into the pipeline as dd.hh:mm:ss (e.g. We have a Data Pipeline that has an Interval parameter (string) being passed in. Part 1: How to convert a time interval (dd.hh:mm:ss) into seconds Scenario By the end of this series, we will cover all of the tips and tricks needed to achieve these performance gains. The first two parts of this series are designed to ease you into two technical processes while the third and final part will bring everything together. Using a ForEach Activity (Sequential = False, Batch Count = 50) and iterating over each range, I was able to execute many copy activities in parallel, taking processing time down from 6.5 hours to under 8 minutes. To provide some context, I’ve used this on a large SQL Table that had a datetime column by taking the Min and Max date then creating sub-time ranges. This method can also be extended to any scenario when provided with some boundary condition. We can improve performance by creating many sub-time ranges based on the StartIndex, EndIndex, and Interval, then for each sub-time range, we call a child pipeline containing a Copy Activity, allowing multiple Copy Activities to execute at the same time, all handling a subset of data. In the current design, the pipeline will take a considerable amount of time using a single Copy Activity if we provide a StartIndex and EndIndex spanning multiple years with a small Interval and Pagecount.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |