数据库

ToKuDB简要

介绍TokuDB存储引擎是Tokutek2009年发布的一个数据库存储引擎,于2013年4月开源。支持MySQL/MariaDB。它和InnoDB一样支持事务、MVCC。

特色: Franctal Tree而不是B-Tree 内部结点不仅有指向父子的指针还有Buffer区,数据写入先写buffer区,FIFO结构,写入只需要顺序添加到Buffer区就可返回,后续满时一次性刷新到下面的子树中,插入数据基本上是一个顺序添加的过程。可轻松应对随机IO,减少空间碎片。 出色的压缩性能 块大小默认是4MB 在线DDL
数据结构: 一、Buffered Tree:类B-Tree,写时直接写到Root结点,如果Root结点满了,就把数据刷新到它子结点上,如果子结点满了就继续刷新到子子结点,一直这样下去。因此,只有当结点满时才有Disk Seek产生。Node大小可设置很大,比如4MB,为提高读,需要对Node做更细划分,分成小块,随机读IO复杂度也为O(logN)。 写: 写时不产生disk seek,因为总是先写Root,由于经常被操作,可认为树顶部结点一直在Page Cache中。 节点数据紧凑,大Node压缩有优势 天然适合做事务,没有undo log,叶子结点上做mvcc
二、Fractal-Tree(Buffer-Tree的变种)buffered(4,16)-tree,OMT结构维护结点数据,大小4MB,nonleaf节点OMT结构,leaf节点多个OMT(4MB/64KB)

OMT:Order Maintenance Tree元素用数组表示,具有父子关系的元素尽量相邻存储,cpu cache line(如果一个节点的周边节点能在文件中紧邻的被存储,当读取其中一个的时候,其他节点被prefetching出来,io数则可以减少。这就是vEB layout)

结点刷新:Node完成写时检查是否满足flush条件,满足加到flush队列,后台线程并行处理从队列中读取的任务。

CheckPoint:60秒一次,sharp checkpoint无Fuzz Checkpoint,会对所有索引加只读锁,其它线程写node时,会clone一个Node。做完后检查LSN,以清理无用的log。基本上这个操作不影响前台读写操作,但是因为会进行数据压缩和clone node、写磁盘的操作,会造成一定的性能波动。

Cache:LRU,写cache时添加node到cache链表,然后检查cache状态,设置了四个水平位(低水平位,低警示点,高警示点,高水平位),高于高水平位,客户端线程进入等待状态,超过低警示点,开始收集逐出数据。

三、读: KEY读:每次读数据都要从ROOT到LEAF,做数据合并,才能得到完整的ROW数据。 范围读:对树做深度优先遍历,在LEAF结点返回,痛苦。
四、Schema Changes列修改:Broadcast类型的Message,从Root广播到每一节点,最后到达LEAF结点。(Mysql对列修改会先关闭表,再打开表,关闭时Tokudb会把脏Node写盘,有些性能消耗),腾讯的游戏运维部门在InnoDB上实现了类似的功能。

索引:两种索引,cst_(clustering index)和cvr_(covering index),每个都是一个单独的F-tree文件 offline方式:启动多个线程,遍历记录生成索引,速度快,但创建过程中写操作不可用。 hot方式:速度慢,创建过程读写不受影响
log:log manager来管理log文件,无重做日志组的概念,当日志写满后重新生成一个文件继续写,checkpoint后检查数据都被刷新到磁盘后会删除,达到InnoDB类似的效果;分in buffer和out buffer(都是16MB),在两块内存上来回倒,多线程下锁控制方便。

优点 高压缩比,写性能高,DDL速度超快 缺点 cpu usr态消耗,响应时间变长 场景 插入效率高、压缩效率好、可在线DDL,适合写性能要求高,数据量过亿,记录不是太大的场合。
大压缩比所以CPU的USR态有消耗 相同QPS时读IOPS差别不大,写IOPS InnoDB平均多消耗些 TokuDB数据经过压缩,但会有解压消耗,所以结果不好确定,测试上看响应时间高于InnoDB
IOPS (Input/Output Operations Per Second),即每秒进行读写(I/O)操作的次数。QPS:Queries Per Second意思是“每秒查询率”,是一台服务器每秒能够相应的查询次数,是对一个特定的查询服务器在规定时间内所处理流量多少的衡量标准。

未分类

go语言开发环境配置

这篇Blog的代码的高亮显示真心不错,花花绿绿的… ‘;’

1、编译vimgdb

下载vimgdb73和vim73

1
2
3
4
5
6
7
mkdir -p ./tmp
cd tmp
tar zxvf ../vim-7.3.tar.gz
unzip ../vimgdb-for-vim7.3-master.zip
mv vimgdb-for-vim7.3-master vimgdb-for-vim7.3
patch -p0 < vimgdb-for-vim7.3/vim73.patch
cd vim73

安装依赖

1
2
3
sudo apt-get install build-essential
sudo apt-get build-dep vim-gtk
sudo apt-get install libncurses5-dev

安装

1
2
3
// 这里直接执行make的操作
make
sudo make install

安装vimgdb runtime

1
2
cd ../vimgdb-for-vim7.3
cp vimgdb_runtime  ~/.vim/bundle

打开vim

1
:helptags ~/.vim/bundle/vimgdb_runtime/doc " 生成doc文件

添加配置.vimrc

1
2
" vimgdb插件
run macros/gdb_mappings.vim

在vim中执行gdb时,报 “Unable to read from GDB pseudo tty” 的错误,因为没有安装 gdb ,所以安装gdb

1
sudo apt-get install gdb

2、安装vundle

1
2
set up vundle
$ git clone https://github.com/gmarik/vundle.git ~/.vim/bundle/vundle

Configure Plugins
在.vimrc文件的开头添加下面的内容,有些不是必须的,可以注掉

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
set nocompatible              " be iMproved, required
filetype off                  " required
 
" set the runtime path to include Vundle and initialize
set rtp+=~/.vim/bundle/vundle/
call vundle#rc()
" alternatively, pass a path where Vundle should install plugins
"let path = '~/some/path/here'
"call vundle#rc(path)
 
" let Vundle manage Vundle, required
Plugin 'gmarik/vundle'
 
" The following are examples of different formats supported.
" Keep Plugin commands between here and filetype plugin indent on.
" scripts on GitHub repos
Plugin 'tpope/vim-fugitive'
Plugin 'Lokaltog/vim-easymotion'
Plugin 'tpope/vim-rails.git'
" The sparkup vim script is in a subdirectory of this repo called vim.
" Pass the path to set the runtimepath properly.
Plugin 'rstacruz/sparkup', {'rtp': 'vim/'}
" scripts from http://vim-scripts.org/vim/scripts.html
Plugin 'L9'
Plugin 'FuzzyFinder'
" scripts not on GitHub
Plugin 'git://git.wincent.com/command-t.git'
" git repos on your local machine (i.e. when working on your own plugin)
Plugin 'file:///home/gmarik/path/to/plugin'
" ...
 
filetype plugin indent on     " required
" To ignore plugin indent changes, instead use:
"filetype plugin on
"
" Brief help
" : PluginList          - list configured plugins
" : PluginInstall(!)    - install (update) plugins
" : PluginSearch(!) foo - search (or refresh cache first) for foo
" : PluginClean(!)      - confirm (or auto-approve) removal of unused plugins
"
" see :h vundle for more details or wiki for FAQ
" NOTE: comments after Plugin commands are not allowed.
" Put your stuff after this line
Install Plugins
Launch vim and run
 
: PluginInstall
或者
In command line:

vim +PluginInstall +qall
3、官方vim-lang插件

1
2
3
4
5
6
7
8
9
10
11
Config vim file .vimrc,Add content bellow in bottom of the file
" 官方的插件
" Some Linux distributions set filetype in /etc/vimrc.
" Clear filetype flags before changing runtimepath to force Vim to
"   reload them.
filetype off
filetype plugin indent off
set runtimepath+=$GOROOT/misc/vim
filetype plugin indent on
syntax on
autocmd FileType go autocmd BufWritePre  Fmt

4、代码补全的插件gocode

配置go的环境变量,比如我的配置,GOPATH变量是必须要配置的,PATH中必须把GOPATH的bin也添加进去,否则没有自动提示,会提示找不到模式

1
2
3
export GOROOT=/usr/local/go
export GOPATH=/data/app/gopath
export PATH=$PATH:$GOROOT/bin:$GOPATH/bin

Set up gocode
Then you need to get the appropriate version of the gocode, for 6g/8g/5g compiler you can do this:
Then you need to get the appropriate
go get -u github.com/nsf/gocode (-u flag for “update”)

Configure vim in .vimrc file

1
Plugin 'nsf/gocode', {'rtp': 'vim/'}

Install Plugins
Launch vim and run

1
: PluginInstall

或者
In command line:

1
vim +PluginInstall +qall

写一个helloword程序,输入fmt后按如果能看到函数的声明展示出来,说明安装是正确的。

4、代码跳转提示godef
Set up godef

1
2
3
go get -v code.google.com/p/rog-go/exp/cmd/godef
go install -v code.google.com/p/rog-go/exp/cmd/godef
git clone https://github.com/dgryski/vim-godef ~/.vim/bundle/vim-godef

Configure vim in .vimrc file

1
Bundle 'dgryski/vim-godef'

Install Plugins
Launch vim and run

1
: PluginInstall

或者
In command line:

1
vim +PluginInstall +qall

5、代码结构提示gotags
Set up gotags

1
go get -u github.com/jstemmer/gotags

Put the following configuration in your vimrc:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
Bundle 'majutsushi/tagbar'
nmap  :TagbarToggle
let g:tagbar_type_go = {
    \ 'ctagstype' : 'go',
    \ 'kinds'     : [
        \ 'p:package',
        \ 'i:imports:1',
        \ 'c:constants',
        \ 'v:variables',
        \ 't:types',
        \ 'n:interfaces',
        \ 'w:fields',
        \ 'e:embedded',
        \ 'm:methods',
        \ 'r:constructor',
        \ 'f:functions'
    \ ],
    \ 'sro' : '.',
    \ 'kind2scope' : {
        \ 't' : 'ctype',
        \ 'n' : 'ntype'
    \ },
    \ 'scope2kind' : {
        \ 'ctype' : 't',
        \ 'ntype' : 'n'
    \ },
    \ 'ctagsbin'  : 'gotags',
    \ 'ctagsargs' : '-sort -silent'
\ }

命令模式下按在右边就会显示当前文件下的函数名,结构体名等等,光标放到相应的tag上,按回车可以快速跳到程序中的相应位置。
再次按会关闭tag窗口。

PS:本地的.vimrc的配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
" 插件管理器 vundle
set nocompatible              " be iMproved, required
filetype off                  " required
 
" set the runtime path to include Vundle and initialize
set rtp+=~/.vim/bundle/vundle/
call vundle#rc()
" alternatively, pass a path where Vundle should install plugins
"let path = '~/some/path/here'
"call vundle#rc(path)
 
" let Vundle manage Vundle, required
Plugin 'gmarik/vundle'
 
" The following are examples of different formats supported.
" Keep Plugin commands between here and filetype plugin indent on.
" scripts on GitHub repos
" Plugin 'tpope/vim-fugitive'
" Plugin 'Lokaltog/vim-easymotion'
" Plugin 'tpope/vim-rails.git'
" The sparkup vim script is in a subdirectory of this repo called vim.
" Pass the path to set the runtimepath properly.
" Plugin 'rstacruz/sparkup', {'rtp': 'vim/'}
" scripts from http://vim-scripts.org/vim/scripts.html
" Plugin 'L9'
" Plugin 'FuzzyFinder'
" scripts not on GitHub
" Plugin 'git://git.wincent.com/command-t.git'
" git repos on your local machine (i.e. when working on your own plugin)
" Plugin 'file:///home/gmarik/path/to/plugin'
" ...
"
filetype plugin indent on     " required
" To ignore plugin indent changes, instead use:
" filetype plugin on
"
" Brief help
" : PluginList          - list configured plugins
" : PluginInstall(!)    - install (update) plugins
" : PluginSearch(!) foo - search (or refresh cache first) for foo
" : PluginClean(!)      - confirm (or auto-approve) removal of unused plugins
"
" see :h vundle for more details or wiki for FAQ
" NOTE: comments after Plugin commands are not allowed.
" Put your stuff after this line
 
syntax on
 
" ********************************************************************
 
" 这里省略了其它不相关的插件
 
" vimgdb插件
run macros/gdb_mappings.vim
 
" 官方的插件
" Some Linux distributions set filetype in /etc/vimrc.
" Clear filetype flags before changing runtimepath to force Vim to
"   reload them.
filetype off
filetype plugin indent off
set runtimepath+=$GOROOT/misc/vim
filetype plugin indent on
syntax on
autocmd FileType go autocmd BufWritePre <buffer> Fmt
 
" 代码补全的插件
Bundle 'Blackrush/vim-gocode'
 
" 代码跳转提示
Bundle 'dgryski/vim-godef'
 
" 代码结构提示
Bundle 'majutsushi/tagbar'
nmap <F8> :TagbarToggle<CR>
let g:tagbar_type_go = {
    \ 'ctagstype' : 'go',
    \ 'kinds'     : [
        \ 'p:package',
        \ 'i:imports:1',
        \ 'c:constants',
        \ 'v:variables',
        \ 't:types',
        \ 'n:interfaces',
        \ 'w:fields',
        \ 'e:embedded',
        \ 'm:methods',
        \ 'r:constructor',
        \ 'f:functions'
    \ ],
    \ 'sro' : '.',
    \ 'kind2scope' : {
        \ 't' : 'ctype',
        \ 'n' : 'ntype'
    \ },
    \ 'scope2kind' : {
        \ 'ctype' : 't',
        \ 'ntype' : 'n'
    \ },
    \ 'ctagsbin'  : 'gotags',
    \ 'ctagsargs' : '-sort -silent'
    \ }

<-EOF->