A year ago, we started looking at ideas of how to solve some data related problems we had. At that point, our current environment was struggling at managing our Point of Sales (POS) feed. We get millions of messages that the POS generates in order to do analytic on them. The second problem we had was associating the POS data with other data sources which ultimately could help to predict sales in stores while getting the right amount of products from the distributors. We wanted to be able to correlate all these sources easily and fast, but we were far away of doing it effectively, yet it was being done with a lot of efforts by our organization. Our Hadoop Journal We got the idea of trying Hadoop out from another team within the company that already had invested time investigating about it since they had a similar problem. Our first Proof of Concept (POC) took few weeks to be completed and my opinion, it was the first step to get into the Hadoop world. This POC was done using . We had a bunch of data stored at and we could dynamically spin up clusters and process these data using . It was just a POC and because of that, we only demonstrated some key features of the solution. We need to know how to process data stored somewhere, how to secure it, and how to do analytic using Business Intelligence (BI) tools that non technical people could use such as Amazon Elastic Mapreduce (AWS EMR) Amazon Simple Storage (S3) Apache Spark Microstrategy After completing our first approach, we basically got shot down because we had the in-house infrastructure to do the same, and a cloud based solution should not be our focus, but instead implement a similar solution on premises. Even though we, in fact, have the infrastructure power to host a solution like the one proposed, we knew it will take time to implement what we need, yet we all committed ourselves to solve our organization problems. Now Comes Hortonworks The first vendor we tried was but at that moment we were just starting and the process of learning was slow and painful. Hortonworks (Horton), A team from did join to us every week for some weeks to check the progress we were making on their platform. Very knowledgeable guys, might I say, but something was still missing. Horton is based on the open source community and some of the components we needed were not updated and we just could not use them. Horton The business model Horton uses is also based on the open source ideas, they give you the entire platform for free, yet there will not be any support engagement until some money is on the table. For us that was a big deal, we were exploring in a complete new environment and we did not have anyone to ask for help. This is madness a colleague of mine says from time to time while trying to figure out why was not working on our Hadoop Cluster. Hive Yes, we had a lot of configuration problems, our cluster was not set up correctly and some of the latest software components were not present. Also, we could not find a nice way to automate the creation and addition of virtual machines to the cluster using the recommended Horton tools. We had problems, and that was fine, but we were expecting some help from them and we never got it. A Small Design Session From our initial test using the platform, we could define a design that will become on our future implementations. In order to everyone to follow, let’s review our data workflow. Hortonworks the design We have a constant flow of messages coming through a web service and those messages are written in a general purpose, distributed, and high performance log queue designed at LinkedIn. From here, an job those messages into two different flows. One that aggregates them and writes them down to and another flow which writes them in a relation fashion to our environment. Apache Kafka , Apache Spark streams Hadoop Distributed File System (HDFS) Teradata Now, we have our messages stored at HDFS yet most of the time we don’t know how to correlate those messages, or how to look at them. We need a way to explore this data, we need a self BI environment. Why Self BI? A common problem in the BI word is the one related to the time required to develop ETL jobs. Yes, there are very skilled people whose work is exactly this, and they do it right and fast. However, what we have seen recently has been that at the moment developers end creating a solution (ETL jobs) the business questions have already changed so the previous jobs are now useless. On the other hand, developers could expend a lot of time creating jobs that will be used only a few times and then will be deprecated since they do not respond to the business inquiries any more. You might wonder why is this happening. Why business people don’t get an agreement about what they really want? There might be many answers, but in our world, it happens because we don’t know all the data we have. We need to explore it, make sense of it and then create the relations that will allow us to get reports out of it. Today, we want certain reports and right away ETL developers start working on them. Tomorrow, we discover that a new data could be related to the one that ETL developers are working with and by the time they finish, the original requirements have changed so much that the results of their work is entire useless. Now, we need to start everything again. Wouldn’t be nice if business people have the ability to explore their own data by themselves? Now that we explained why we need self BI environments, let’s go back to our design session. How we give to the business a self BI environment At this point, we have hundred of Gigabytes written in but we need to access them efficiently. HDFS, There are several tools we could use. Hortonworks provides Apache Hive out of the box. Hive works as an SQL engine that runs on top of HDFS, but wait, our business guys don’t need to know SQL! Based on this, we put Microstrategy at their disposition so they drag and click while Microstrategy generates SQL statements that are executed over our data through Hive on HDFS. Because the business guys will use microstrategy as a front end tool, we could change Hive (our back end query engine) to any other processing engine such as Spark SQL, and they will still perform the same tasks on their end without problems, hopefully. Performance With the Hortonworks platform we experimented a lot of performance issues when using Hive. That was one of the main arguments while moving to the next vendor we tried out. Apache Spark worked well, but some team members were not happy to code some Scala code every time a new data feed was added (even when all this process can be automated which implies the argument is not valid any longer). Apache Spark has become one the most popular data processing frameworks out there, we definitely have to come back to it. Cloudera, a Leader Hadoop Vendor One day, the guys from Cloudera appeared at office and we got introduced to their platform. In order to be short talking about our experience we Cloudera, I have to say that the platform it truly at the enterprise level, something we were looking for. All components were updated, we get their customer support since day one, and they worked with us in order to set up the cluster from the beginning. For us that was just impressive after our experience with our previous vendor. right The management tools were easy to use and understand. The Cloudera tools allowed us to work fast, to discover the advantages of their platform with easy. Cloudera has the same components that Hortonworks and also adds a new one that we were quite interested in, Apache Impala . Impala is the faster SQL compliance tools we have seen so far. It was quite faster than Hive. Impala integrates with Microstrategy just fine, so our front end does not have to change at all. The Common Problem In both platforms, Hortonworks and Cloudera, we face one common challenge. How do we expose new data that we haven’t defined the schema for it? The process is the following: Open the feed file (xml, json, csv, …) manually Define a table that points to the location where the feed is. Indicate that our query engine (Hive, Impala) use the defined table. Start querying the new data. The question is who is going to do this? No one wants to do it, it should be automated, but sometimes the business wants to look at some files that are completely new for everyone (extractions from different sources they manually generate and use). There was no way to automate that. The business depends on developers to follow the previous process every time it wants to do these operations. Our idea was not possible, it was incompleted and these two platforms were not enough for us. Self BI The simplicity of MapR The truth is that we were skeptical when the guys from showed up_._ However, by the end of their talk, I think we were kind of convince that they have our solution. MapR Even though this vendor is not the most popular one in the Hadoop world, what they showed us matched our requirements perfectly, so we decided to give it a try. Setting the cluster up was as easy as in Cloudera, I don’t think there were any problems. However, the console management tool was not as good as the one present in the Cloudera cluster. It was a little more difficult to navigate and understand. The Advantages of Being Compliance POSIX MapR is POSIX compliance! That means that our data path get significant simplified. Having a POSIX platform implies we can create a volume inside the cluster and it in our Linux or Window environment. The volume will appear as any other drive we have on our computer. We can read / write from / to it as we do with any other attached storage. We in fact, are reading/writing to a distributed storage (our MapR cluster). mount Now we can remove Kafka, we can remove the Spark job that aggregates messages and writes them to HDFS and with that we also remove the complexity of all those moving parts in the middle of our data path. Now we have less components to worry about and less hardware requirements. Part of our initial design was streaming messages to Teradata. We still can do that by changing our Spark streaming source to be the file system instead of Kafka and everything else just works fine. In an initial data movement test, we executed a command between a network share and a mounted drive that pointed to the MapR file system. We got a 100 Mb / s transfer rate between our environment and the MapR cluster. For us, it was more than enough. rsync How MapR solves our Self BI Problems MapR brings a new tool to the table as an addition to the ones found in other vendors. a tool that auto discover schemas from different sources and exposes them without any previous registration. This is the tool we were looking for. Apache Drill , Yes, we had a lot of problems with it, mostly related to performance. Some team members dedicated hours to work it out, and they did. We can get results back as fast as in Impala with the addition of auto discovering schemas. Drill exposes data to Microstrategy in the same way that Hive and Impala do it so it can be used in the same fashion. Because we can mount different volumes, now business people can have their our space in the MapR cluster, so they can copy and paste the data they want to explore (streaming and automatic feeds are processed by automatic processes such as Spark jobs). If a guy creates an extracted file and wants to correlate it to another that comes within the automatic feed, he only needs to: copy this file to MapR using via mounted volume. copy / paste open Microstrategy and query the new data. Behind the scene, Drill will pick the file up. will infer the schema so it can be exposed to Microstrategy. Microstrategy will see this file and the user will it to other ones already in the cluster. join Small Files Problem is a big deal in Hadoop. In Hortonworks and Cloudera it was something we really need to take care of. It was the main reason behind the aggregation process we had. We aggregated a lot of small file into a big file so we can avoid the by reducing the number of files written to HDFS. The small files problem small files problem MapR does not use HDFS, instead, they have implemented MapR-FS, a file system that exposes a HDFS interface so it can be used by any tools designed to work with HDFS. It can be also accessed through POSIX commands, adding the possibility of work as any other file system compatible with most operating systems. In MapR-FS, we can have millions of small files without problem. The internal specifications of this file system and how to work with it can be found MapR has done an incredible job in this area and we all can benefit from it. here . Apache Spark MapR, in the same way Cloudera does, has relied, heavily, in the use of Spark as one of the main component of their solution. Spark is used extensively as a data processing framework and sometimes as the engine that powers Impala and Drill. Spark is by far, one the most used components in any big data solution because its versatility while working on any task. Ending thoughts about MapR As we can see, MapR not only simplified our solution, but also offer a different way to attack the problems we have. MapR can be seen as an extension of our environment instead of a separated environment that we need to architect in order to move and process data within it. Even though MapR management tools are a little behind of the ones offered by Cloudera, they work as they should, they keep our environment stable, always working. MapR helps us to eliminate unnecessary processes so we can focus on the data and how to work with it, and that is what we really need. If you enjoyed this peace you might also like Apache Spark as a Distributed SQL Engine This blog doesn’t represent the thoughts, plans or strategies of my employer. is how hackers start their afternoons. We’re a part of the family. We are now and happy to opportunities. Hacker Noon @AMI accepting submissions discuss advertising &sponsorship To learn more, , , or simply, read our about page like/message us on Facebook tweet/DM @HackerNoon. If you enjoyed this story, we recommend reading our and . Until next time, don’t take the realities of the world for granted! latest tech stories trending tech stories
