Material Big Data

Lanzados ppts informativos de tecnologías BigData: Hadoop, Hbase, Hive, Zookeeper...

Apuntate al Curso de PowerBI. Totalmente práctico, aprende los principales trucos con los mejores especialistas

Imprescindible para el mercado laboral actual. Con Certificado de realización!!

Pentaho Analytics. Un gran salto

Ya se ha lanzado Pentaho 8 y con grandes sorpresas. Descubre con nosotros las mejoras de la mejor suite Open BI

Aprende gratis Analytics OLAP sobre Pentaho

La solución open source para business intelligence y Big Data sobre Pentaho, no te lo pierdas!!

5 may. 2016

Hadoop Hive y Pentaho: Business Intelligence con Big Data (Caso Practico)



Cuando los profesionales del Business Intelligence (BI) oímos hablar de Big Data existe una cuestión que nos suele venir a la cabeza casi de forma natural ¿Es posible usar Big Data para el desarrollo de aplicaciones de BI típicas como el análisis OLAP o la generación de informes? 

A continuación, si la respuesta es afirmativa, con seguridad nos surgirán algunas dudas más:

  •   Posibilidades de conexión con las herramientas de BI. Por ejemplo con Pentaho, la suite BI Open Source más conocida y líder del mercado
  •   Rendimiento con aplicaciones de generación de informes y OLAP
  •   Carga de datos relacionales, procesos de ETL con Big Data, automatización de los procesos…

Para intentar dar respuesta a algunas de las cuestiones anteriores, hemos llevado a cabo un conjunto de pruebas para el desarrollo de aplicaciones BI haciendo uso del binomio Hadoop – Pentaho

La prueba ha consistido en replicar un Data Warehouse  generado en un proyecto real sobre una BBDD Oracle y con una alta volumetría, a un nuevo Data Warehouse en el clúster Hadoop usando la herramienta Apache Hive

Gracias a la conexión JDBC, conectaremos Hive con las aplicaciones disponibles en la suite Pentaho para evaluar la viabilidad de la generación de informes o reporting con Big Data, siendo esta una de las aplicaciones BI más demandadas en la actualidad por las empresas.
 
Para llevar a cabo la prueba hemos usado las últimas versiones 6.X de las herramientas de Pentaho y disponemos de un clúster Hadoop con las siguientes características:

·         Distribución 2.4 de Hortonworks
·         2 máquinas o hosts
·         2 procesadores por máquina (total 4)
·         16 Gb Ram por máquina (total 32 Gb)


Generación de informes

En la siguiente imagen puede verse la arquitectura de la aplicación BI desarrollada:


Para la carga de las tablas del DW de Oracle en Hadoop hemos usado tanto Sqoop como Pentaho Data Integration, gracias a la integración de esta última con HDFS y Sqoop. 

Tras esto, para la creación del Data Warehouse en Hadoop hemos usado la herramienta Apache Hive. Esta herramienta soporta consultas en lenguaje SQL y usa como almacenamiento el sistema de archivos distribuido de Hadoop HDFS. 
Además, recientemente se ha incorporado en la distribución Hortonworks el nuevo motor de ejecución Apache Tez, que mejora en gran medida el rendimiento de Hive haciendo uso intensivo de la memoria RAM del clúster y evitando el uso de IO a disco siempre que sea posible.

Por último, hemos usado la conexión JDBC disponible en Hive para conectar con las siguientes aplicaciones de la suite Pentaho, con el objetivo de dar soporte a la generación de informes sobre el Data Warehouse creado Hive:

 * Pentaho Reporting Designer: Usada para la generación de informes estáticos y parametrizables. Para las pruebas hemos creado 3 informes con consultas de distinta complejidad.

* Pentaho Metadata Editor: Creación de un modelo de metadatos que es usado por aplicaciones como STReport para la generación de informes AdHoc, herramienta incluida en la suite Lince BI, desarrollada por el equipo de StrateBI a partir de Saiku Reporting. Con STReport  generaremos 3 informes con consultas similares a las de los 3 informes estáticos generados con Pentaho Reporting Designer.

* Pentaho BA Analytics (Servidor de BI): Servidor de BI de Pentaho, donde ejecutaremos los informes creados con Pentaho Reporting Designer y crearemos nuevos informes sobre el modelo de metadatos usando la herramienta STReport



Dado que los entornos Big Data están preparados para procesar volúmenes de datos mucho más grandes que los de nuestro DW de ejemplo, hemos creado dos tablas de hechos adicionales a la original de 1.240.361 filas, las cuales tienen 5.161.444 filas (x4) y 25.807.220 (x20) respectivamente. De esta forma hemos creado versiones de los 6 informes (3 estáticos con PRD y 3 adhoc con PME + STReport) que se ejecutan sobre las 3 tablas de hechos de distinta volumétrica. 

Tras la ejecución, en la siguiente tabla se muestran los tiempos de generación medidos:



Conclusiones

Esta prueba nos demuestra que es posible la generación de informes sobre datos que están almacenados en una plataforma Big Data como Apache Hadoop, gracias a las capacidades de la herramienta Apache Hive y su conectividad JDBC. 

También hemos comprobado como las herramientas de la suite Pentaho, gracias a su conectividad con Hadoop, son el complemento ideal para el desarrollo de aplicaciones BI que hacen uso del Big Data
No obstante es necesario tener en cuenta los tiempos de respuesta en la generación de informes, los cuales hacen que la generación de informes sobre Hive sea recomendada en casos los que el tiempo respuesta instantánea no sea un requisito indispensable. A cambio, obtenemos la posibilidad de generar informes sobre datos de tipo Big Data (Volumen, Variedad y Velocidad). 

En cualquier caso, nuestro clúster de pruebas tiene unas prestaciones muy reducidas, siendo habitual el despliegue de clúster que cuentan con más 5 máquinas y un cantidad de memoria RAM en conjunto superior a los 100 Gb. Es precisamente el uso intensivo de la RAM por Apache Hive (sobre el motor de ejecución Tez), lo que seguramente está penalizando nuestros tiempos en respuesta en más de 10-15 segundos.

Dado que existen más herramientas y aplicaciones BI susceptibles de ser desarrollados con la tecnología Big Data, en pruebas posteriores nos proponemos comprobar las capacidades de Apache Impala para la generación de informes en una distribución de Cloudera o el análisis OLAP usando el novedoso Kylin sobre Hadoop

Esperamos que os sea útil

·  


56 comentarios:

Giri Mani 2 dijo...

This blog giving the details of technology. This gives the details about working with the business processes and change the way. Here explains think different and work different then provide the better output. Thanks for this blog.
Hadoop Training in Chennai

Akalya dijo...

Its Nice Article about "Hadoop Hive y Pentaho: Business Intelligence con Big Data (Caso Practico)"

Akalya from Digital Marketing Training in Chennai

Reena S dijo...

Thanks for sharing this informative blog keep updating more ..

Core Java Training in chennai

vishali dijo...

Hadoop Training

SKARtec SEO dijo...

Its Nice Article about "Hadoop Hive y Pentaho: Business Intelligence con Big Data (Caso Practico)"

digital marketing course

mohd Sameer dijo...

https://stitchclothes.com here we stitch clothes as per your design and selection

Apsara G dijo...


Nice Blog... RPA Training in Chennai | Blue Prism Training in Chennai

hari raj dijo...

The blog, you presented on Big data and business intelligence is clear to understand the basics of the software. Really good blog. Best Software Training Institute in Chennai

hari raj dijo...

The blog, you presented on Big data and business intelligence is clear to understand the basics of the software. Really good and nice blog. RPA Training in Chennai|Blue Prism Training in Chennai

shalinipriya dijo...

This is a nice article here with some useful tips for those who are not used-to comment that frequently. Thanks for this helpful information I agree with all points you have given to us. I will follow all of them.
Data Science Training in Chennai
Data science training in bangalore
Data science online training
Data science training in pune
Data science training in kalyan nagar

Ram priya dijo...

Hi, Great. The tutorial is just awesome.It is really helpful for a newbie like me.. I am a regular follower of your blog. The really very informative post you shared here. Kindly keep blogging.

Devops training in Chennai
Devops training in Bangalore
Devops Online training
Devops training in Pune

amilu B dijo...

Very nice post here and thanks for it .I always like and such a super contents of these post.Excellent and very cool idea and great content of different kinds of the valuable information's.java training in chennai | java training in bangalore

java online training | java training in pune

selenium training in chennai

selenium training in bangalore

ajay. chinna dijo...

Nice post keep do posting The Info was too good, for more information regarding the technology Click
Amazon web Services Training
Professional Salesforce CRM Training
Devops training

Nithin dijo...

we repair and service all models of oneplus Service in Chennai, We rely on customer satisfaction. We can service all problems faced by oneplus customers in chennai.


Oneplus Service Center in Chennai | Oneplus Service Centre | Oneplus Service Center | Oneplus Service Center near me | Oneplus Service Center in velachery | Oneplus Service Chennai

Priya Rajesh dijo...

Wonderful post. Thanks for taking time to share this information with us.
RPA courses in Chennai | RPA Training in Chennai | Blue Prism Training Institute in Chennai | UiPath Courses in Chennai

Mouni yoga dijo...

A very nice guide. I will definitely follow these tips. Thank you for sharing such detailed article. I am learning a lot from you.
python training in pune
python training institute in chennai
python training in Bangalore

gowsalya dijo...

Thank you for allowing me to read it, welcome to the next in a recent article. And thanks for sharing the nice article, keep posting or updating news article.
Online DevOps Certification Course - Gangboard
Best Devops Training institute in Chennai

Sri Nithya dijo...

Wonderful post. Thanks for taking time to share this information with us.
Big Data Analytics Courses in Chennai
Data Analytics Training in Chennai
Blue Prism Training near me
Blue Prism Training Chennai
ccna Training institute in Chennai
AWS Certification in Chennai

Priya Rajesh dijo...

Thanks for sharing this information admin, it helps me to learn new things. Continue sharing more like this.
DevOps certification Chennai
DevOps Training in Chennai
RPA Training in Chennai
RPA courses in Chennai
R Training in Chennai
UiPath Training in Chennai

Teju Teju dijo...

It is nice blog Thank you provide important information and I am searching for the same information to save my time Big Data Hadoop Online Training Bangalore

Robotic Process Automation Tutorial dijo...

Very informative article.Thank you admin for you valuable points.Keep Rocking
rpa training chennai | rpa training in velachery | best rpa training in chennai

Anónimo dijo...

I simply wanted to write down a quick word to say thanks to you for those wonderful tips and hints you are showing on this site.
angularjs Training in marathahalli

angularjs interview questions and answers

angularjs Training in bangalore

angularjs Training in bangalore

angularjs Training in chennai

automation anywhere online Training

LindaJasmine dijo...

Interesting Post. Looking for this information for a while. Thanks for Posting.

Node JS Training in Chennai
Node JS Course in Chennai
Node JS Advanced Training
Node JS Training Institute in chennai
Node JS Training Institutes in chennai
Node JS Course

Roja Priya dijo...

Thank you for sharing your article. Great efforts put it to find the list of articles which is very useful to know, Definitely will share the same to other forums.
Data Science Training in chennai at Credo Systemz | data science course fees in chennai | data science course in chennai quora | data science with python training in chennai

Vicky Ram dijo...

It is a great post. Keep sharing such kind of useful information.

Education
Technology

Aruna ram dijo...

Really thank you!!!This is one marvelous blog article. I appreciate your sharing this post.
Best Etical Hacking Training in Bangalore
Ethical Hacking Training Institute in Bangalore
Ethical Hacking Training in Perambur
Ethical Hacking Course in Nolambur
Ethical Hacking Training in Aminjikarai
Ethical Hacking Course in Vadapalani

Meera Kumar dijo...

It's really a nice experience to read your post. Thank you for sharing this useful information. If you are looking for more about R Programming Training in Chennai | R Programming Training in Chennai with Placement | R Programming Interview Questions and Answers | Trending Software Technologies in 2018

Shadeep shree dijo...

The blog which you have posted is outstanding. Thanks for your Sharing.
Ethical Hacking Course in Coimbatore
Hacking Course in Coimbatore
Ethical Hacking Training in Coimbatore
Ethical HackingTraining Institute in Coimbatore
Ethical Hacking Training

kavinilavu G dijo...

Great post! I am actually getting ready to across this information, It's very helpful for this blog.Also great with all of the valuable information you have Keep up the good work you are doing well.Trending Software Technologies in 2018 | Hadoop Training in Chennai | big data Hadoop training and certification in Chennai

sathyaramesh dijo...

thanks for sharing such a nice info.I hope you will share more information like this. please keep on sharing!
Selenium Training in Chennai
Best Selenium Training Institute in Chennai
ios developer training in chennai
Digital Marketing Training in Chennai
.Net coaching centre in chennai
PHP Institutes in Chennai
PHP Training Center in Chennai

aruna ram dijo...

I appreciate you sharing this post. Thanks for your efforts in sharing this information in detail. kindly keep continuing the good job.
Robotics Courses in Bangalore
Automation Courses in Bangalore
RPA Training in Bangalore
Robotics Classes in Bangalore
Robotics Training in Bangalore
RPA Courses in Bangalore

mercyroy dijo...

I really thank you for your innovative post.I have never read a creative ideas like your posts.here after i will follow your posts which is very much help for my career.
Selenium training near me
Selenium Training in Chennai
Selenium Training in T nagar
Selenium Certification Training in T nagar

Nadeem Malik dijo...

nice topic which you have choose.
second is, the information which you have provided is better then other blog.
so nice work keep it up. And thanks for sharing. digital marketing company in delhi

PLC Training Chennai dijo...

PLC Training in Chennai | PLC Training Institute in Chennai | PLC Training Center in Chennai | PLC SCADA Training in Chennai | PLC SCADA DCS Training in Chennai | Best PLC Training in Chennai | Best PLC Training Institute in Chennai | PLC Training Centre in Chennai | PLC SCADA Training in Chennai | Automation Training Institute in Chennai

PLC Training Chennai dijo...

PLC Training in Chennai | PLC Training Institute in Chennai | PLC Training Center in Chennai | PLC SCADA Training in Chennai | PLC SCADA DCS Training in Chennai | Best PLC Training in Chennai | Best PLC Training Institute in Chennai | PLC Training Centre in Chennai | Embedded System Training in Chennai | Embedded Training in Chennai | VLSI Training in Chennai | VLSI Training Institute in Chennai

Embedded System Training dijo...

Embedded Training in Chennai | Best Embedded Training in Chennai | Embedded System Training in Chennai | Embedded System Training Institute in Chennai | Best Embedded System Training Institute in Chennai | Embedded Course in Chennai | Embedded System Training Institutes in Chennai | Embedded System Training Center in Chennai | Best Embedded System Training in Chennai | Embedded Systems Training in Chennai | VLSI Training in Chennai | VLSI Training Institute in Chennai

VLSI Training Chennai dijo...

VLSI Training in Chennai | Best VLSI Training in Chennai | VLSI Training Centres in Chennai | VLSI Courses in Chennai | VLSI Training Courses in Chennai | VLSI Training Institute in Chennai | VLSI Training Institutes in Chennai | Best VLSI Training Institute in Chennai

Mechanical Projects dijo...

Mechanical Projects | Mechanical Projects in Chennai | Mechanical Project Centers in Chennai | Mechanical Project Centre in Chennai | Mechanical Final Year Project in Chennai | Final Year Projects Mechanical | Final Year Projects Mechanical in Chennai

Pooja Rao dijo...

fridge repair in gurgaon
refrigerator repair in gurgaon
lg fridge repair in gurgaon

Pooja Rao dijo...

cattle feed bags supplier

mercyroy dijo...

feeling so good to read your information's in the blog.
thanks for sharing your ideas with us and add more info.
best devops training in bangalore
devops course in bangalore
devops Training in anna nagar
devops Training in Chennai Anna Nagar

Digital Marketing Agency dijo...

Thank you for sharing this post Website Development Company in Rohini

gowthunan dijo...

My rather long internet look up has at the end of the day been compensated with pleasant insight to talk about with my family and friends.
iosh course in chennai

Nila shri dijo...

This is a nice article here with some useful tips for those who are not used-to comment that frequently. Thanks for this helpful information I agree with all points you have given to us. I will follow all of them.
Data Science training in Chennai | Data Science Training Institute in Chennai
Data science training in Bangalore | Data Science Training institute in Bangalore
Data science training in pune | Data Science training institute in Pune
Data science online training | online Data Science certification Training-Gangboard
Data Science Interview questions and answers

aruna ram dijo...

Thanks for your sharing the knowledge to needed ones. Waiting for your updates. Keep it up.
Digital Marketing Classes in Bangalore
Best Digital Marketing Course in Bangalore
Digital Marketing Course Bangalore
Digital Marketing Training in Chennai Velachery
Digital Marketing Training in Tnagar
Digital Marketing Training in Navalur
Digital Marketing Training in Kelambakkam

Akash Thakur dijo...

nice work keep it up thanks for sharing the knowledge.Thanks for sharing this type of information, it is so useful.

Epoxy Grout manufacturer in delhi

sathyaramesh dijo...

Thanks for splitting your comprehension with us. It’s really useful to me & I hope it helps the people who in need of this vital information.
Web Designing Course in chennai
web designing training in chennai
Web Development courses in Chennai
PHP Training Chennai
PHP Course Chennai
PHP Training Institute in Chennai

Vikash Kumar dijo...

hi, nice information is given in this blog. Thanks for sharing this type of information, it is so useful for me. nice work keep it up. Laminated Doors manufacturer in hubli

aruna ram dijo...

It was really great! I learn lot of information from your post. I want more updates.....
Machine Learning Course in Chennai
Machine Learning Classes near me
Machine Learning Training in Chennai Velachery
Machine Learning Course in Tnagar
Machine Learning Training in Nungambakkam
Machine Learning Course in Saidapet

Amase LED Lighting dijo...

hello sir,
thanks for giving that type of information. led lawn lights in delhi

aruna ram dijo...

Wonderful idea! It's very impress to me and very nice concept. Thank you so much sharing with us. Please keeping....
Best PHP Training in Bangalore
PHP Coaching in Bangalore
PHP Course in Annanagar
PHP Course in Perambur
PHP Course in Tnagar
PHP Training Institute in Velachery
PHP Course in Omr
PHP Training in Kandanchavadi

sathyaramesh dijo...

Nice article. I liked very much. All the informations given by you are really helpful for my research. keep on posting your views.
Cloud computing Training in Chennai
Cloud computing Training
Cloud computing Training near me
Salesforce Developer Training
Salesforce crm Training in Chennai
Salesforce administrator training in chennai

stemians dijo...

Hi , thanks for sharing your information.The insights are really helpful and informative.
Robotics in Coimbatore
Learn robotics online

Deekshana dijo...

For long days, I am waiting to see this kind of most valuable thought about Hadoop. Indeed, this is so much helpful to every Hadoop admin developers. I welcome your good work.

SEO services in Madurai

john brito dijo...

This is an awesome post.Really very informative and creative contents. These concept is a good way to enhance the knowledge.I like it and help me to development very well.Thank you for this brief explanation and very nice information.Well, got a good knowledge.

rpa training in bangalore | best rpa training in bangalore | rpa course in bangalore | rpa training institute in bangalore | rpa training in bangalore | rpa online training | rpa training in Chennai

sunshineprofe dijo...

Whoa! I’m enjoying the template/theme of this website. It’s simple, yet effective. A lot of times it’s very hard to get that “perfect balance” between superb usability and visual appeal. I must say you’ve done a very good job with this.
iosh safety course in chennai