Well, another autumn is here, and our data grow bigger every year (Based on a prediction from the IDC report, the global data volume will grow exponentially from 4.4 zettabytes to 44 zettabytes (44×1021 bytes) between 2013 and 2020). We use it to feed ever-hungry sales, marketing, accounting, management, and day-to-day operations, then there is the stuff we are obliged to pile up because of the law, more stuff stored “just-in-case” and also lots of weeds between the rows. So, do we need another harvester? Someone said that big data machines are all the rage this season. But wait; what shall we do with them?
- We are a bank. We process millions of transaction every day. Our sales and marketing want to use transaction logs for up-to-date analytics, our credit department wants to use them for better credit scoring and risk mitigation, and our security department wants them for fraud detection. That can’t be done with the current systems.
- We are a telco. We’d like to actively use billions of records on calls, messages, and data transfers to identify families and other interest groups so we can provide better services, offer relevant products, and reduce customer defection to competitors. Our current logs are not suitable for up-to-date deep analytics.
- We are a rapidly expanding security company. We collect tens of thousands of sensor signals per second from smart homes and businesses. We need a new system capable of storing and processing them with very low latency.
These are just a few examples Profinit has had the opportunity to participate in devising a solution. Drafting big data solution architecture, designing data storage and processes, is just one part of the solution. Then there is the question of getting the information the user needs. Everything about your clients could be hidden somewhere in all those terabytes – their status, their needs, their preferences, even their personality traits, habits and relationships. To get this information in a usable form, we often rely on data science methods.
When selecting tools for our technology toolbox, we are pragmatic. We adapt to the technologies most commonly employed by our customers. In big data, this means the Hadoop + Spark technology stack (we are partners of both Hortonworks and Cloudera), but we also have experience with cloud computing architecture (AWS, Azure). In data science, we build on open source technologies from both R and Python universe because of their rapid progress and widespread use. But if your company standard is, let’s say, Oracle + Excel, that’s no problem for us.
If you want to hear more about the possibilities of big data and data science, contact us to arrange a meeting or a call. You can also meet with us at one of the following autumn conferences:
W-JAX, 5. – 9. November 2018, München
FinTech Connect, 5. – 6. December 2018, London. You can register with a special Profinit discount.