大数据学习笔记.pdf下载

目录 第一部分 Spark学习 ....................................................................................................................... 6 第1章 Spark介绍 ................................................................................................................... 7 1.1 Spark简介与发展 ...................................................................................................... 7 1.2 Spark特点 .................................................................................................................. 7 1.3 Spark与Hadoop集成 ................................................................................................ 7 1.4 Spark组件 .................................................................................................................. 8 第2章 Spark弹性分布数据集 ............................................................................................... 9 2.1 弹性分布式数据集 .................................................................................................... 9 2.2 MapReduce数据分享效率低..................................................................................... 9 2.3 MapReduce进行迭代操作 ........................................................................................ 9 2.4 MapReduce进行交互操作 ...................................................................................... 10 2.5 Spark RDD数据分享 ............................................................................................... 10 2.6 Spark RDD 迭代操作 .............................................................................................. 10 2.7 Spark RDD交互操作 ............................................................................................... 10 第3章 Spark安装 ................................................................................................................. 11 第4章 Spark CORE编程 ....................................................................................................... 13 4.1 Spark Shell ................................................................................................................ 13 4.2 RDD Transformations ............................................................................................... 13 4.3 Actions ...................................................................................................................... 16 4.4 用RDD编程 ........................................................................................................... 17 4.5 UN-Persist存储 ........................................................................................................ 18 第5章 Spark调度与高级编程 ............................................................................................. 20 5.1 Spark应用程序例子 ................................................................................................ 20 5.2 Spark-submit语法 .................................................................................................... 22 5.3 Spark变量 ................................................................................................................ 23 5.4 数字类型 RDD操作 .............................................................................................. 23 第二部分 ZOOKEEPER学习 .......................................................................................................... 24 第6章 zookeeper介绍 ......................................................................................................... 25 6.1 zookeeper简介 ......................................................................................................... 25 6.2 分布式应用程序 ...................................................................................................... 25 6.3 Apache Zookeeper意味着什么？ ............................................................................ 26 第7章 zookeeper基本组成与工作流程 ............................................................................. 27 第8章 zookeeper的leader节点选择 ................................................................................. 31 第9章 zookeeper安装 ......................................................................................................... 33 第10章 zookeeper 命令行接口 .......................................................................................... 35 第11章 zookeeper应用程序接口 ....................................................................................... 39 第12章 zookeeper应用 ....................................................................................................... 40 第三部分 KAFKA学习 ................................................................................................................... 48 第12章 KAFKA介绍 ............................................................................................................. 49 12.1 KAFKA简介 .......................................................................................................... 49 12.2信息系统 ................................................................................................................. 49 12.3 KAFKA是什么？ .................................................................................................. 50 第13章 KAFKA基本组成与集群架构 ................................................................................. 51 13.1 KAFKA的基本组成 .............................................................................................. 51 13.2 KAFKA集群架构 .................................................................................................. 52 第14章 KAFKA工作流程 ..................................................................................................... 53 14.1 PUB-SUB信息工作流 ........................................................................................... 53 14.2 队列信息工作流/消费者组 ................................................................................... 53 14.3 Zookeeper在KAFKA中扮演的角色 ................................................................... 54 第15章 KAFKA安装 ............................................................................................................. 55 第16章 KAFKA基本操作 ..................................................................................................... 56 16.1 启动zookeeper服务 ............................................................................................. 56 16.2 单个单节点中间件配置 ........................................................................................ 56 16.3 Topics列表 ............................................................................................................. 56 16.4 启动生产者发送信息 ............................................................................................ 57 16.5 启动消费者接收信息 ............................................................................................ 57 16.6 单个多节点中间件配置 ........................................................................................ 57 16.7 创建一个topic ....................................................................................................... 58 16.8 启动生产者发送信息 ............................................................................................ 59 16.9 启动消费者接收信息 ............................................................................................ 59 16.10 基本Topic操作 ................................................................................................... 59 16.11 删除Topic ............................................................................................................ 59 第17章 KAFKA 生产者与消费者群实例 ............................................................................ 60 17.1 生产者实例 ............................................................................................................ 60 17.2 简单消费者实例 .................................................................................................... 63 17.3 消费者群例子 ........................................................................................................ 65 第18章 KAFKA与SPARK集成 ............................................................................................. 67 18.1 Kafka与spark集成 ............................................................................................... 67 18.2 SparkConf API ........................................................................................................ 67 18.3 StreamingContext API ............................................................................................ 67 18.4 KafkaUtils API ........................................................................................................ 67 18.5 建立脚本 ................................................................................................................ 69 18.6 编译/打包 ............................................................................................................... 69 18.7 提交到Spark ......................................................................................................... 69 第四部分HIVE学习 ....................................................................................................................... 70 第19章 HIVE介绍 ................................................................................................................ 71 19.1 HIVE是什么？ ...................................................................................................... 71 19.2 HIVE特点 .............................................................................................................. 71 19.3 HIVE架构 .............................................................................................................. 71 19.5 HIVE工作流 .......................................................................................................... 72 第20章 HIVE 安装 ............................................................................................................... 74 20.1 Hadoop安装 ........................................................................................................... 74 20.2 HIVE安装 .............................................................................................................. 77 20.3 Derby安装与设置 .................................................................................................. 78 第21章 HIVE 数据类型 ....................................................................................................... 80 21.1列类型(Column Type) ............................................................................................ 80 21.2文本类型(Literals) .................................................................................................. 81 21.3 Null 值.................................................................................................................... 81 21.4 复杂类型 ................................................................................................................ 81 第22章 数据库操作 ............................................................................................................. 82 22.1 创建数据库 ............................................................................................................ 82 22.2 删除数据库 ............................................................................................................ 82 第23章 数据表操作 ............................................................................................................. 83 23.1 创建数据表 ............................................................................................................ 83 23.2 load数据(插入数据) .............................................................................................. 85 23.3 修改数据表(Alter table) ........................................................................................ 86 23.4 删除表(Drop table) ................................................................................................ 90 第24章 分区 ......................................................................................................................... 92 24.1 添加分区(Adding a Partition) ................................................................................ 93 24.2 重命名分区(Renaming a Partition) ....................................................................... 93 24.3 删除分区(Droping a Partition) .............................................................................. 93 第25章 内置运算符 ............................................................................................................. 94 25.1 关系运算符 ............................................................................................................ 94 25.2 算术运算符 ............................................................................................................ 96 25.3 逻辑运算符 ............................................................................................................ 97 25.4 复杂运算符(Complex Operators) .......................................................................... 97 第26章 内置函数 ................................................................................................................. 98 26.1 内置函数 ................................................................................................................ 98 26.2 聚合函数(Aggregate Functions) ............................................................................ 99 第27章 视图与索引 ........................................................................................................... 100 27.1 创建视图（Creating a View） ............................................................................ 100 27.2 删除视图(Dropping a View) ................................................................................ 100 27.3 创建索引(Creating an Index)............................................................................... 101 27.4 删除索引(Dropping an Index) ............................................................................. 101 第28章 HIVEQL ................................................................................................................... 102 28.1 查询语句(SELECT ...WHERE) .......................................................................... 102 28.1.1 查询语句实例 ................................................................................................... 102 28.1.2 JDBC查询语句实例 ......................................................................................... 102 28.2 查询语句(SELECT...ORDER BY) ...................................................................... 103 28.2.1 ORDER BY查询语句实例 ............................................................................... 103 28.2.2 JDBC ORDER BY 查询语句实例 ................................................................... 104 28.3 查询语句(GROUP BY) ....................................................................................... 104 28.3.1 GROUP BY查询语句实例 ............................................................................... 104 28.3.2 JDBC GROUP BY查询语句实例 .................................................................... 105 28.4 查询语句(JOIN) .................................................................................................. 106 28.4.1 JOIN查询语句实例 .......................................................................................... 106 第五部分HBASE学习 .................................................................................................................. 111 第29章 HBASE介绍 ........................................................................................................... 112 29.1 HBase是什么？ ................................................................................................... 112 29.2 HBase与HDFS .................................................................................................... 113 29.3 HBase存储机制 ................................................................................................... 113 29.4 面向列和面向行 .................................................................................................. 113 29.5 HBase与RDBMS ................................................................................................ 114 29.6 HBase优点 ........................................................................................................... 114 29.7 HBase使用场景 ................................................................................................... 114 29.8 HBase的应用 ....................................................................................................... 114 第30章 HBASE架构与安装 ............................................................................................... 115 30.1 HBase架构 ........................................................................................................... 115 30.1.1主服务器(Master Server) ................................................................................... 115 30.1.2 分区及分区服务器 ........................................................................................... 115 30.1.3 Zookeeper ........................................................................................................... 116 30.2 HBase安装 ........................................................................................................... 116 第31章 HBASE的SHELL .................................................................................................... 121 31.1 一般命令 .............................................................................................................. 121 31.2 数据定义语言(DDL) ........................................................................................... 121 31.3数据操纵语言(DML) ........................................................................................... 121 第32章 HBASE普通命令 ................................................................................................... 122 32.1 status...................................................................................................................... 122 32.2 version ................................................................................................................... 122 32.3 table_help .............................................................................................................. 122 32.4 whoami .................................................................................................................. 122 第33章 HBASE ADMIN API ................................................................................................. 123 33.1 HBaseAdmin类 .................................................................................................... 123 33.2 Class Descriptor .................................................................................................... 123 33.3 Constructors(构造函数) ........................................................................................ 123