You can read it from numerous headlines: your company’s future depends on data. Collecting more of it and understanding it better, whether for customer insight or monitoring your business processes, is the new normal. Many are predicting AI and machine learning will only amplify this development in the future.
However, in practice data rarely ”plays nice” and is conveniently in one place for all analysis work. Data tends to reside on multiple clouds, on-premise, and increasingly behind various APIs (which is good) – or just accessed via plain traditional file transfers (not so…). Maybe you have an IoT system deployed where sensor data is stored within your IoT provider’s platform.
For consolidation with data analysis in mind, what is your best path forward then? Every data project is unique, here are some thoughts that may help you.
- Plan ahead as much as you can. It may sound obvious, but rushing for new deployments is a sure way for data challenges in the future. If you have your eyes set on one particular cloud platform, e.g. Azure or Google, try to find ways to consolidate as much under it as possible – but note that while this certainly helps, it alone is not a guarantee for seamless data migration in the future.
- Those who expect terabytes or even petabytes of data over time and especially if its unstructured, consider using big data stores like Azure Data Lake. Anything less, often times a typical SQL setup will serve you well. There are several pro’s and con’s on this decision so it pays off to study it well, not to mention they can also co-exist.
- Better to divide and conquer, don’t try to do it all at once. Instead work on the data sources one by one and accept that getting it right may take time. Can you split the work into multiple smaller projects? Ideally you want an automated long term solution also for your future needs.
- Make informed decisions and seek assistance when needed, technology in the cloud progresses very rapidly and keeping up with all the options can be exhausting.
- Lastly, if you find out you have chosen the wrong approach, then “fail fast” and try different path. Proof-of-Concepts with outside help are a great way to verify different approaches before starting a larger project. They can be quick and very cost efficient. If a PoC fails, you have a lot of new insight into your system and data.
Need help with your data? Drop us a note.