Material Big Data

Lanzados ppts informativos de tecnologías BigData: Hadoop, Hbase, Hive, Zookeeper...

Apuntate al Curso de PowerBI. Totalmente práctico, aprende los principales trucos con los mejores especialistas

Imprescindible para el mercado laboral actual. Con Certificado de realización!!

Pentaho Analytics. Un gran salto

Ya se ha lanzado Pentaho 8 y con grandes sorpresas. Descubre con nosotros las mejoras de la mejor suite Open BI

LinceBI, la mejor solución Big Data Analytics basada en Open Source

LinceBI incluye Reports, OLAP, Dashboards, Scorecards, Machine Learning y Big Data. Pruébala!!

2 nov. 2016

Analysis Big Data OLAP sobre Hadoop con Apache Kylin


Caso de estudio que presentamos, en el que hacemos uso de las herramientas Apache Kylin y STPivot para dar soporte al análisis interactivo OLAP de un almacén de datos (Data Warehouse, DW) que contiene datos con características Big Data (Volumen, Velocidad y Variedad)

Se trata de un gran Volumen de datos académicos, relativos a los últimos 15 años de una universidad de gran tamaño. A partir de esta fuente de datos, se ha diseñado un modelo multidimensional para el análisis del rendimiento académico. En él contamos con unos 100 millones de filas con medidas cómo los créditos relativos a asignaturas aprobadas, suspendidas o matriculadas. Estos hechos se analizan en base a distintas dimensiones o contextos de análisis, como el Sexo o la Calificación y la siempre presente componente temporal, el Año Académico.

Dado que este Volumen de datos es demasiado grande para analizarlo con un rendimiento aceptable con los sistemas OLAP (R-OLAP y M-OLAP) tradicionales, hemos decidido probar la tecnología Apache Kylin, la cual promete tiempos de respuesta en consultas de unos pocos segundos para Volúmenes superiores a los 10 billones de filas.

Las tecnologías del entorno Hadoop fundamentales para Kylin son Apache Hive y Apache HBase.
El almacén de datos (Data Warehouse, DW) se crea en forma de modelo estrella y se mantiene en Apache Hive. 

 
A partir de este modelo y mediante la definición de un modelo de metadatos del cubo OLAP, Apache Kylin, mediante un proceso offline crea un cubo multidimensional (MOLAP) en HBase. 




Ver Big Data-OLAP en funcionamiento
 
A partir de este momento, Kylin permite hacer consultas sobre el mismo a través de su interfaz SQL, también accesible a través de conectores J/ODBC.



Por último, para hacer posible el análisis OLAP mediante consultas MDX y las tablas o vistas multidimensionales correspondientes, hacemos uso de la herramienta STPivot. 
 
STPivot es un visor OLAP desarrollado por StrateBI como parte de la suite STTools.
STPivot usa Mondrian como servidor OLAP y puede desplegarse sobre un servidor BI como Pentaho BA Server, ambos open source. De esta forma, STPivot permite crear y explorar vistas o tablas multidimensionales, cómo las de esta demo, que hacen uso del cubo OLAP creado con Apache Kylin.        





Desarrollada por eBay y posteriormente liberada como proyecto Apache open source, Kylin es una herramienta de código libre que da soporte al procesamiento analítico en línea (OLAP) de grandes volúmenes de datos con las características del Big Data (Volumen, Velocidad y Variedad).

Sin embargo, hasta la llegada de Kylin, la tecnología OLAP estaba limitada a las bases de datos relacionales o, en el mejor de los casos, con optimizaciones para el almacenamiento multidimensional, tecnologías con importantes limitaciones para enfrentarse al Big Data.

Apache Kylin, construida sobre la base de distintas tecnologías del entorno Hadoop, proporciona una interfaz SQL que permite la realización de consultas para el análisis multidimensional de un conjunto de datos, logrando unos tiempos de consulta muy bajos (segundos) para hechos de estudio que pueden llegar hasta los 10 billones de filas o más.
Las tecnologías del entorno Hadoop fundamentales para Kylin son Apache Hive y Apache HBase

El almacén de datos (Data Warehouse, DW) se crea en forma de modelo estrella y se mantiene en Apache Hive. A partir de este modelo y mediante la definición de un modelo de metadatos del cubo OLAP, Apache Kylin, mediante un proceso offline, crea un cubo multidimensional (MOLAP) en HBase. Se trata de una estructura optimizada para su consulta a través de la interfaz SQL proporcionada por Kylin.

De esta forma cuando Kylin recibe una consulta SQL, debe decidir si puede responderla con el cubo MOLAP en HBase (en milisegundos o segundos), o sí por el contrario, no se ha incluido en el cubo MOLAP, y se ha ejecutar una consulta frente al esquema estrella en Apache Hive (minutos), lo cual es poco frecuente.

Por último, gracias al uso de SQL y la disponibilidad de drivers J/ODBC podemos conectar con herramientas de Business Intelligence como Tableau, Apache Zeppelin o incluso motores de consultas MDX como Pentaho Mondrian, permitiendo el análisis multidimensional en sus formas habituales: vistas o tablas multidimensionales, cuadros de mando o informes.      

Ver Big Data-Dashboard en funcionamiento


STPivot es un visor OLAP potente a la par que fácil de usar, desarrollado por StrateBI y que forma parte de la suite de aplicaciones Business Intelligence, STTools

El objetivo de este visor es mejorar la experiencia de usuario haciendo tan sencillo el análisis OLAP como arrastrar y soltar las medidas y contextos del análisis en un lienzo, de forma que la vista OLAP se genera de forma transparente al usuario.

Además, la incorporación de asistentes de consulta, gráficos novedosos además de las propias tablas multidimensionales, un editor de fórmulas avanzado o la exportación para la publicación de las vistas en distintos formatos, son algunas de las características más destacadas de STPivot y que diferencian nuestra herramienta de otros visores OLAP existentes.

En cuanto a su arquitectura, STPivot funciona sobre el motor de ejecución MDX, Mondrian.
Es por ello, qué STPivot puede usarse como aplicación del servidor de Business Intelligence open source Pentaho BA Server (CE), el cual ya incluye Mondrian como parte del mismo. 



Gracias a la conectividad JDBC es posible la conexión de Mondrian con Apache Kylin y, de esta forma, el uso de esta fuente de datos OLAP y Big Data con STPivot.      

Como fuente datos Big Data de esta demo, disponemos de un gran Volumen de datos académicos ficticios, relativos a los últimos 15 años de una universidad de gran tamaño y por la que han pasado más de un millón de alumnos en este tiempo. A partir de esta fuente de datos, se ha diseñado un modelo multidimensional para el análisis del rendimiento académico



En él contamos con unos 100 millones de filas con medidas cómo la suma de los créditos relativos a asignaturas aprobadas, suspendidas o matriculadas.

Además también nos encontramos con otras medidas derivadas de las anteriores y, por tanto, más complejas como son la Tasa de rendimiento y Tasa de éxito, calculadas a partir de la relación entre Créditos Superados y Créditos Matriculados y de la relación entre Créditos Superados y Créditos Presentados.

No menos importantes son las dimensiones o contextos de análisis en base a los que se analizan las medidas anteriores. Como dimensiones de un solo nivel tenemos el Sexo, la Calificación, el Rango de Edad y la siempre presente componente temporal, el Año Académico. Además, hemos incorporado dos dimensiones complejas, con jerarquías de dos niveles y una mayor cardinalidad, siendo frecuente encontrarnos con dimensiones de esta naturaleza.


Con la dimensión Estudio, podemos analizar los datos agrupados al nivel de Tipo de Estudio (Grado, Máster, Doctorado,...) o profundizar (operación Drill Down sobre la vista OLAP) hasta los distintos Planes de Estudio, esto es, las distintas titulaciones, como "315-Grado en Biología".              

73 comentarios:

Blogger dijo...

BlueHost is definitely the best web-hosting company with plans for all of your hosting requirements.

Apsara G dijo...

Excellent Article ...thank u for sharing, such a valuable content Learners to get good knowledge after read this article.. Big Data Training Institute in Chennai

raju v dijo...

Awesome,
Thank you so much for sharing such an awesome blog...
big data training services
big data certification services

Janvi Sanvi dijo...

It is nice blog Thank you provide important information and I am searching for same information to save my timeBig data Hadoop online training India

Teju Teju dijo...

Really nice blog post.provided a helpful information.I hope that you will post more updates like thisBig data hadoop online Course Bangalore

SRI dijo...

Wow it is really wonderful and awesome thus it is very much useful for me to understand many concepts and helped me a lot. it is really explainable very well and i got more information from your blog.

rpa Training in Chennai

rpa Training in bangalore

rpa Training in pune

rpa online training

blueprism Training in Chennai

blueprism Training in bangalore

blueprism Training in pune

Nila shri dijo...

Wow it is really wonderful and awesome thus it is very much useful for me to understand many concepts and helped me a lot. it is really explainable very well and i got more information from your blog.

Data Science Training in Chennai
Data science training in bangalore
Data science online training
Data science training in pune
Data science training in kalyan nagar
selenium training in chennai

simbu dijo...

Wow it is really wonderful and awesome thus it is very much useful for me to understand many concepts and helped me a lot. it is really explainable very well and i got more information from your blog.
java training in chennai | java training in bangalore

java online training | java training in pune

java training in chennai | java training in bangalore

Mouni yoga dijo...

This is a good post. This post give truly quality information. I’m definitely going to look into it. Really very useful tips are provided here. thank you so much. Keep up the good works.
Python training in pune
AWS Training in chennai

shethal dijo...

This is most informative and also this post most user friendly and super navigation to all posts... Thank you so much for giving this information to me.. 
Devops Training in Chennai
Devops Training in Bangalore

param jothi dijo...

Really very nice blog information for this one and more technical skills are improve,i like that kind of post.
Blueprism training in Chennai

Blueprism online training

Blue Prism Training in Pune

Nadeem Malik dijo...

hello sir,
thanks for giving that type of information.
digital marketing company in delhi

Nadeem Malik dijo...

Offshore Anonymous Hosting

Nadeem Malik dijo...

nice work keep it up thanks for sharing the knowledge.Thanks for sharing this type of information, it is so useful.
block adhesive manufacturer in delhi

Vikash Kumar dijo...

Laminated Doors manufacturer in hubli

shivam enterprises dijo...

outdoor led wall lights in delhi

Sachin Sinhmar dijo...

refrigerator repair in gurgaon

Vikky cmd dijo...

I am really happy with your blog because your article is very unique and powerful for new reader.
Click here:
selenium training in chennai
selenium training in bangalore
selenium training in Pune
selenium training in pune
Selenium Online Training

Nadeem Malik dijo...

digital marketing company in delhi

Customer Help Care dijo...

Your blog was really amazing and informative. Thanks for sharing with us. Visit for
Web Designing Company in Delhi
SEO Service in Delhi

Apsara G dijo...


The strategy you have posted on this technology helped me to get into the next level and had lot of information in it...
AngularJs Training in Chennai OMR | ReactJs Training in Chennai OMR | Node.Js Training in Chennai OMR

Pankaj Singh dijo...

Thanks for given such informative and valuable blog. Visit for
SEO Service in Delhi

sachin.ogeninfo dijo...

animal feed bags supplier

sachin.ogeninfo dijo...

washing machine repair in Gurgaon

مروة محمد dijo...

شركة عزل اسطح بحائل

رجب البرنس dijo...


بسم الله الرحمن الرحيم عميلنا العزيز نحن نقدم افضل خدمات تنظيف المجالس والشقق بافضل انواع

التنظيفات المشهود لها عالميا
شركة تنظيف مجالس بالطائف
شركة تنظيف مجالس بجازان
شركة تنظيف مجالس بحائل
ونحن فى خماتكم 24 ساعة على مدار الشهر

Pooja Rao dijo...

Bali Honeymoon Packages From Delhi
Bali Honeymoon Packages From Chennai
Hong Kong Packages From Delhi

Pooja Rao dijo...

fridge repair in faridabad
Bosch Fridge Repair in Faridabad
Godrej Fridge Repair in Faridabad
Haier Fridge Repair in Faridabad
Panasonic Fridge Repair in Faridabad

Pooja Rao dijo...

we have provide the best ppc service in Gurgaon.
ppc company in gurgaon
website designing company in Gurgaon

Pooja Rao dijo...

Rice Packaging Bags Manufacturers
Pouch Manufacturers

Cara Mengobati Campak Pada Anak dijo...

There are times when you suffer the disease being treated immediately, before things got worse. Thank you for the information:)

Pengobatan Penyumbatan Pembuluh Darah Di Otak
Pengobatan Atasi Benjolan Di Leher
Obat Hepatitis Alami
Pengobatan Kencing Manis Secara Alami
Obat Atasi Daya Tahan Tubuh Menurun Dan Mudah Lemah
Pengobatan Stroke Pada Wajah

cynthia williams dijo...

Excellent stuff and this is really helpful for all. Do share more like this.
ReactJS Training in Chennai
ReactJS Training
AngularJS Training in Chennai
AngularJS course in Chennai
R Training in Chennai
R Programming Training in Chennai
ReactJS Training in Velachery
ReactJS Training in Tambaram

jaya devan dijo...

You are doing a great job. I would like to appreciate your work for good accuracy
Regards,
Selenium Training Institute in Chennai | Selenium Testing Training in chennai

tamilsasi dijo...

The post is written in very a good manner and it entails many useful information for me. I am happy to find your distinguished way of writing the post. Now you make it easy for me to understand and implement the concept.
devops online training

aws online training

data science with python online training

data science online training

rpa online training

Sai Bk dijo...

I really like the dear information you offer in your articles. I’m able to bookmark your site and show the kids check out up here generally. Im fairly positive theyre likely to be informed a great deal of new stuff here than anyone

Microsoft Azure online training
Selenium online training
Java online training
Python online training
uipath online training

Aaditya dijo...

Really nice and informative post. This kind of informative posts are very few. Keep posting. Thanks for sharing.

Data Science Bangalore

zaintech99 dijo...

I have to search sites with relevant information on given topic and provide them to teacher our opinion and the article.
date analytics certification training courses
for more :click here

Data Science Course dijo...

Actually I read it yesterday but I had some thoughts about it and today I wanted to read it again because it is very well written.

Big Data Course

istiaq ahmed dijo...

wow, great, I was wondering how to cure acne naturally. and found your site by google, learned a lot, now i’m a bit clear. I’ve bookmark your site and also add rss. keep us updated.
Data Science Course in Pune

Unknown dijo...

Good information has been given. Thanks for posting it.
Node JS Training in Chennai
Node JS Course
TOEFL Training in Chennai
french courses in chennai
Best Spoken English Class in Chennai
IELTS Training in Chennai
Node JS Training in Tambaram
German classes in anna nagar
spoken english class anna nagar

unknown dijo...
Este comentario ha sido eliminado por el autor.
unknown dijo...

Yes, Java is everywhere, it's on the desktop, it's on mobile, it's on the card, almost everywhere and so is Java programmers. I think Java programmer outnumber any other programming language professional. I have learned java from the best java training institute in Chennai.Java training in chennai

Chris Hemsworth dijo...

The article is so informative. This is more helpful for our
software testing training courses
selenium testing training
software testing training institute
Thanks for sharing.

Chris Hemsworth dijo...

The article is so informative. This is more helpful for our
software testing training courses
selenium testing training
software testing training institute
Thanks for sharing.

magento 2 training in chennai dijo...

Thanks for this blog!!!
web designing training and placement
php training with placement in chennai

Naveen S dijo...

Thanks For blog!!!

Learn Digital Marketing Course Training and Placement Institute in Chennai

sasireka dijo...

Woah this blog is wonderful i like studying your posts. Keep up the great work! You understand, lots of persons are hunting around for this info, you could help them greatly.

ReactJS Online Training

Unknown dijo...

After reading your article I was amazed. I know that you explain it very well. And I hope that other readers will also experience how I feel after reading your article.
pmp certification malaysia

seocom dijo...

سباك بجدة
يقوم سباك بجدة بتقديم كافة الأعمال المتعلقة بالسباكة ومد مواسير المياه والأنابيب الخاصة بشبكة المياه والصرف الصحي، وجميع قطع الغيار التي يستخدمها سباك بجدة أصلية وعمرها الافتراضي طويل، وأسعار سباك بجدة مناسبة لجميع أيدي فئات المجتمع لذلك لا داعي للقلق حيال هذه المشكلة لأن أسعارنا من أرخص الأسعار الموجودة في السوق، كما أن سباك بجدة يمتلك كافة الأجهزة التي تساعد على الكشف على تسربات المياه ومن ضمن الأجهزة التي يتم استخدامها هي أجهزة الاستشعار عن بعد التي تساعد على وصول أماكن التسربات من خلال إصدار بعض الأصوات والذبذبات، لذلك عند الكشف عن مكان التسرب لا يحتاج الأمر إلى تكسير أو تخريب أي شيء.

Rigid Box dijo...

Excellent Blog. I really want to admire the quality of this post. I like the way of your presentation of ideas, views and valuable content. No doubt you are doing great work. I’ll be waiting for your next post. Thanks .Keep it up!
Kindly visit us @
Luxury Boxes
Premium Packaging
Luxury Candles Box
Earphone Packaging Box
Wireless Headphone Box
Innovative Packaging Boxes
Wedding gift box
Leather Bag Packaging Box
Cosmetics Packaging Box
Luxury Chocolate Boxes

KIT dijo...

Nice blog, it's so knowledgeable, informative, and good looking site. I appreciate your hard work. Good job. Thank you for this wonderful sharing with us. Keep Sharing.
Kindly visit us @
100% Job Placement
Best Colleges for Computer Engineering
Biomedical Engineering Colleges in Coimbatore
Best Biotechnology Colleges in Tamilnadu
Biotechnology Colleges in Coimbatore
Biotechnology Courses in Coimbatore
Best MCA Colleges in Tamilnadu
Best MBA Colleges in Coimbatore
Engineering Courses in Tamilnadu
Engg Colleges in Coimbatore

Benish dijo...

awesome blog.. thank you for sharing your good information..
AngularJS interview questions and answers/angularjs interview questions/angularjs 6 interview questions and answers/mindtree angular 2 interview questions/jquery angularjs interview questions/angular interview questions/angularjs 6 interview questions/angularjs interview questions and answers for 3 years experience

venkatesh@CS dijo...

Excellent Blog. Thank you so much for sharing.
best react js training in chennai
react js training in Chennai
react js workshop in Chennai
react js courses in Chennai
react js tutorial
reactjs training Chennai
react js online training
react js training course content
react js online training india
react js training courses
react js training topics
react js course syllabus
react js course content
react js training institute in Chennai

al ghadeer dijo...

شركة عزل فوم بحائل
شركة عزل فوم بالقصيم
شركة عزل فوم بمكة
شركة عزل فوم بجدة
شركة عزل فوم بالطائف
شركة عزل فوم بالباحة
شركة عزل فوم بتبوك
شركة عزل فوم بينبع
نتيجة الثانوية العامة 2019

jose dijo...

Really nice post. Thank you for sharing amazing information.
Java Training in Chennai/Java Training in Chennai with Placements/Java Training in Velachery/Java Training in OMR/Java Training Institute in Chennai/Java Training Center in Chennai/Java Training in Chennai fees/Best Java Training in Chennai/Best Java Training in Chennai with Placements/Best Java Training Institute in Chennai/Best Java Training Institute near me/Best Java Training in Velachery/Best Java Training in OMR/Best Java Training in India/Best Online Java Training in India/Best Java Training with Placement in Chennai

gokul dijo...

Thank you for this informative blog
Top 5 Data science training in chennai
Data science training in chennai
Data science training in velachery
Data science training in OMR
Best Data science training in chennai
Data science training course content
Data science certification in chennai
Data science courses in chennai
Data science training institute in chennai
Data science online course
Data science with python training in chennai
Data science with R training in chennai

Aruna Ram dijo...

Excellent blog, I read your great blog and it is one of the best explanation about this content. Keep doing the new posts...


Pega Training in Chennai
Pega Course
Primavera Training in Chennai
Tableau Training in Chennai
Unix Training in Chennai
Job Openings in Chennai
Placement Training in Chennai
Linux Training in Chennai
JMeter Training in Chennai
Pega Training in T Nagar
Pega Training in Anna Nagar

adhi dijo...

thanks for your information really good and very nice web design company in velachery

manjuragul dijo...

Superb! Your blog is incredible. I am delighted with it. Thanks for sharing with me more information.
RPA Training in anna nagar
RPA Training in Chennai
RPA Training in OMR
Python Training in anna nagar
Software testing training in T Nagar
RPA Training in T Nagar
AngularJS Training in Anna Nagar
Spoken English Classes in Tnagar

شركة المثالي سوبر dijo...

شركة تنظيف افران بالجبيل

شركة تنظيف أفران بالخبر

شركة تنظيف أفران بالقطيف

شركة صيانة حمامات بالخبر

shemadevan dijo...

nice article...keep updating like this..
LoadRunner Training in Chennai
Loadrunner course in Chennai
Best Loadrunner training institute in chennai
Loadrunner Training in Velachery
Loadrunner Training in Tambaram
QTP Training in Chennai
core java training in chennai
C C++ Training in Chennai
Mobile Testing Training in Chennai
Manual Testing Training in Chennai

Riya Raj dijo...

Good Blog!!! The way you have conveyed your blog is more impressive...
JAVA Training in Chennai
Best JAVA Training institute in Chennai
best java course in chennai
JAVA J2EE Training in Chennai
Best JAVA Training in Chennai
java training in OMR
JAVA Training in Annanagar
Big data training in chennai
Selenium Training in Chennai
Android Training in Chennai

Durai Raj dijo...

Great blog!!! The information was more useful for us... Thanks for sharing with us...
Python Training in Chennai
Python course in Chennai
Python Training Institute in Chennai
Best Python Training in Chennai
Python Training in Tambaram
Python training in Guindy
Hadoop Training in Chennai
Big data training in chennai
SEO training in chennai
JAVA Training in Chennai

venkatesh@CS dijo...

Excellent Blog. Thank you so much for sharing.
best react js training in chennai
react js training in Chennai
react js workshop in Chennai
react js courses in Chennai
react js training institute in Chennai
reactjs training Chennai
react js online training
react js online training india
react js course content
react js training courses
react js course syllabus
react js training
react js certification in chennai
best react js training

shwethapriya dijo...

Thanks for posting this information. Keep updating.
Spoken English Classes in Chennai
Spoken English in Chennai
German Classes in Chennai
Japanese Classes in Chennai
TOEFL Coaching in Chennai
Informatica Training in Chennai
Spoken English Classes in Adyar
Spoken English Classes in Velachery

gokul dijo...

Thank you for this informative blog
Top 5 Data science training in chennai
Data science training in chennai
Data science training in velachery
Data science training in OMR
Best Data science training in chennai
Data science training course content
Data science certification in chennai
Data science courses in chennai
Data science training institute in chennai
Data science online course
Data science with python training in chennai
Data science with R training in chennai

diya shivanya dijo...

Really a awesome blog for the freshers. Thanks for posting the information.
Blockchain Training in Chennai
Blockchain Training in Chennai
IELTS Coaching in Chennai
Japanese Classes in Chennai
Spoken English Classes in Chennai
TOEFL Coaching in Chennai
Blockchain Training in Anna Nagar
Blockchain Training in T Nagar 

shivam dijo...

Flying Shift - Packers & Movers in Bhopal

sasi dijo...

Great experience for me by reading this blog. Nice article.
Angularjs Training in Chennai
Angularjs Training in Bangalore
angularjs training institute in bangalore
Angular 2 Training in Chennai
best angularjs training in bangalore
Angularjs course in Chennai
dot net training institutes in bangalore
php course in coimbatore


seoexpert dijo...

Nice Post...I have learn some new information.thanks for sharing.
ExcelR data analytics course in Pune | business analytics course | data scientist course in Pune

divi dijo...

Pretty post...! I got more useful information about this topic and I like a unique post. Please updating...thanks for your information really good and very nice web design company in velachery

virgoacp dijo...

Awesome post. I am a normal visitor of your blog and appreciate you taking the time to maintain the excellent site. I’ll be a frequent visitor for a long time.

ACP Sheet
ACP Sheet Price
Aluminium Composite Panel

verdure skills india pvt ltd dijo...

nice blog
get best placement at VSIPL

get digital marketing services
seo network point