# 表格存储

# 功能介绍

column-store-orm 是一个列式存储 orm 框架，封装了阿里云 tablestore、hbase、mongo，为应用提供了多云环境下对列式存储的一致性访问方式。 注意：由于 column-store-orm-mongo 使用到了 spring-data-mongodb，因此与 spring 存在版本兼容性问题。

column-store-orm-mongo版本	spring-data-mongodb	spring版本	spring-boot-dependencies版本
1.0.0 ~ 1.0.2	1.10.6.RELEASE	4.3.10.RELEASE	1.5.6.RELEASE
1.0.2-SCG.RELEASE ~ 1.0.4-SCG.RELEASE	3.0.1.RELEASE	5.2.7.RELEASE	2.3.1.RELEASE

# 使用流程

# 1.maven settings.xml 添加仓库地址

<!-- scg maven仓库 -->
<repository>
    <id>scg-private</id>
    <name>maven-scg-private</name>
    <url>http://packages.glodon.com/artifactory/maven-scg-private/</url>
</repository>

1
2
3
4
5
6

# 2.项目中添加 pom 依赖

<!-- 添加column-store-orm starter依赖 -->
<dependencies>
    <dependency>
        <groupId>com.glodon.cloud</groupId>
        <artifactId>column-store-orm-spring-boot-starter</artifactId>
        <version>1.0.1</version>
    </dependency>
</dependencies>

1
2
3
4
5
6
7
8

如果结合spring-cloud-glodon使用，则通过引入spring-cloud-glodon-dependencies后管理依赖版本即可

<!-- 添加column-store-orm starter依赖 -->
<dependencies>
    <dependency>
        <groupId>com.glodon.cloud</groupId>
        <artifactId>column-store-orm-spring-boot-starter</artifactId>
    </dependency>
</dependencies>

<!-- 导入SCG依赖 -->
<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>com.glodon.cloud</groupId>
            <artifactId>spring-cloud-glodon-dependencies</artifactId>
            <version>2.3.2.RELEASE</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

# 3.添加配置项

application.yaml

# 推荐使用方式，公共配置提取出来
# 公共配置都放在application.yaml，如各个实体类和它对应的要生成的bean名称
# application.yaml
glodon:
  column-store-orm:
    dao:
      entities:
        - name: "bookDAO"
          entity: "com.glodon.columnstoreorm.entity.BookEntity"
        - name: "elementIndexDAO"
          entity: "com.glodon.columnstoreorm.entity.ElementIndexEntity"
        - name: "elementPropertyDAO"
          entity: "com.glodon.columnstoreorm.entity.ElementPropertyEntity"

1
2
3
4
5
6
7
8
9
10
11
12
13

application-aliyun.yaml

# application-aliyun.yaml
glodon:
  column-store-orm:
  	aliyun:
      endpoint: https://xxx.your-region.ots.aliyuncs.com
      accessKey: xxxxxx
      accessSecret: xxxxxx
      instance: xxx
      maxConnections: 300
      connectionTimeoutInMillisecond: {15 * 1000}
      socketTimeoutInMillisecond: {15 * 1000}

1
2
3
4
5
6
7
8
9
10
11

application-hbase.yaml

# application-hbase.yaml
glodon:
  column-store-orm:
    hbase:
      zookeeperQuorum: host-application
      port: 2181
      timeoutMillisecond: {15 * 1000}

1
2
3
4
5
6
7

application-mongo.yaml

glodon:
  column-store-orm:
    mongo:
      hosts: host-application:27017
      database: mongo-test
      username: root
      password: 123456

1
2
3
4
5
6
7

application-dynamo.yaml

glodon:
  column-store-orm:
    dynamo:
      accessKey: xxxxxx
      accessSecret: xxxxxx

1
2
3
4
5

注意：只用配置项目中使用到的组件即可

# 4.使用

初始化仅需创建一个 TableStoreDao 对象，传入 tableStore 的 config 和一个实体类。

@Resource(name = "bookDAO")
private TableStoreDAO<BookEntity> bookEntityDAO;

// put
BookEntity entity = new BookEntity("f1", 12345, "Jon");
entity.setReaderIds(new ArrayList<>(Arrays.asList("100", "101")));
bookEntityDAO.put(entity);

1
2
3
4
5
6
7

实体类到 TableStore 的 PrimaryKeyName 和 ColumnName 的自动映射是通过注解实现的，BookEntity 的代码如下：

@Data
@TSTable(name = "ut_book")
public class BookEntity {

    @TSPKColumn
    private String floor = "";

    @TSAttrColumn
    private int ibsn;

    @TSAttrColumn
    private String author = "";

    @TSAttrColumn(dynamicColumnNames = true)
    private List<String> readerIds = new ArrayList<>(10);

    public BookEntity() {
    }

    public BookEntity(String floor, int ibsn, String author) {
        this.floor = floor;
        this.ibsn = ibsn;
        this.author = author;
    }

}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

主要用到了三个注解，TSTable 和 TSPKColumn、TSAttrColumn。

TSTable用于实体类到 TableStore 中表的映射：

name，指定表名；

TSPKColumn用于实体类中属性到 TableStore 中 PrimaryKey 的映射：

name，主键名
order，当有多个属性做主键时，安装 order 顺序拼接

TSAttrColumn用于实体类中属性到 TableStore 中 Column 的映射：

name，属性名；
dynamicColumnNames，是否为动态属性列，默认 false；表格存储中列数与列名是不固定的，把此选项设为 true 来定义变化的列名，暂时不支持列值的定义；

# 4.1 单记录操作

单条记录有put、get、update、delete四种：

get和delete有两种调用方式，第一种为按照顺序依次传入primaryKeyValue，第二种先用buildPrimaryKey方法构造出PrimaryKey，再传入get、delete方法；

// put
BookEntity entity = new BookEntity("f1", 12345, "Jon");
entity.setReaderIds(new ArrayList<>(Arrays.asList("100", "101")));
dao.put(entity);

// get
PrimaryKey primaryKey = PrimaryKey.builder()
        .addColumn("floor", "f1")
        .build();
BookEntity bookEntityGet = dao.get(primaryKey);
Assert.assertEquals("Jon", bookEntityGet.getAuthor());
Assert.assertEquals(2, bookEntityGet.getReaderIds().size());

// update
entity.setAuthor("Json_updated");
entity.getReaderIds().add("102");
dao.update(entity);
BookEntity bookEntityUpdate = dao.get(primaryKey);
Assert.assertEquals("Json_updated", bookEntityUpdate.getAuthor());
Assert.assertEquals(3, bookEntityUpdate.getReaderIds().size());

// delete
dao.delete(primaryKey);
BookEntity bookEntityDelete = dao.get(primaryKey);
Assert.assertNull(bookEntityDelete);

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

# 4.2 批量操作：

批量操作有put、get、update、delete四种：

List<Integer> ids = Arrays.asList(1, 2, 3, 4, 5, 6, 7, 8, 9, 10);
// putRow
dao.batchPut(ids.stream()
        .map(id -> {
            BookEntity bookEntity = new BookEntity("f" + id + "_batch", id, "author" + id + "_batch");
            for (int i = 1; i <= id; i++) {
                bookEntity.getReaderIds().add(String.valueOf(10000 + i));
            }
            return bookEntity;
        })
        .collect(Collectors.toList()));

// get
List<PrimaryKey> pks = ids.stream()
        .map(id -> PrimaryKey.builder()
                .addColumn("floor", "f" + id + "_batch")
                .build())
        .collect(Collectors.toList());
dao.batchGet(pks).forEach(bookEntity -> {
    String floor = bookEntity.getFloor();
    int start = "f".length();
    int end = floor.indexOf("_batch");
    int index = Integer.valueOf(floor.substring(start, end));
    Assert.assertEquals("author" + index + "_batch", bookEntity.getAuthor());
    Assert.assertEquals(index, bookEntity.getReaderIds().size());
});

// update
dao.batchUpdate(ids.stream()
        .map(id -> {
            BookEntity bookEntity = new BookEntity("f" + id + "_batch", id + 9000, "author" + id + "_batch_updated");
            bookEntity.getReaderIds().add("999");
            return bookEntity;
        })
        .collect(Collectors.toList()));
dao.batchGet(pks).forEach(bookEntity -> {
    String floor = bookEntity.getFloor();
    int start = "f".length();
    int end = floor.indexOf("_batch");
    int index = Integer.valueOf(floor.substring(start, end));
    Assert.assertEquals("author" + index + "_batch_updated", bookEntity.getAuthor());
    Assert.assertEquals(index + 1, bookEntity.getReaderIds().size());
});

// update delete dynamic columnValues
dao.batchUpdate(ids.stream()
        .map(id -> {
            BookEntity bookEntity = new BookEntity("f" + id + "_batch", id + 8000, "author" + id + "_batch_updated_2");
            bookEntity.getReaderIds().add("999");
            return bookEntity;
        })
        .collect(Collectors.toList()), true);
dao.batchGet(pks).forEach(bookEntity -> {
    String floor = bookEntity.getFloor();
    int start = "f".length();
    int end = floor.indexOf("_batch");
    int index = Integer.valueOf(floor.substring(start, end));
    Assert.assertEquals("author" + index + "_batch_updated_2", bookEntity.getAuthor());
    Assert.assertEquals(index, bookEntity.getReaderIds().size());
});

// delete
dao.batchDelete(pks);
List<BookEntity> ret = dao.batchGet(pks);
Assert.assertTrue(ret == null || ret.isEmpty());

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65

range操作有range get和range consume类接口，前者返回结果，后者直接指定对结果的操作； RangePrimaryKey有四种指定方式：

Fixed，固定的值，一般位于最开始的主键列；
FullRange，全部范围，一般位于最后的主键列；
SubRangeString，String类型的指定范围的主键列，一般位于Fixed和FullRange之间；

# 4.3 范围查找操作

// put first
List<Integer> ids = new ArrayList<>(100);
for (int i = 1; i <= 100; i++) {
    ids.add(i);
}

// putRow first
dao.batchPut(ids.stream()
        .map(id -> {
            BookEntity bookEntity = new BookEntity("f" + id + "_range", id, "author" + id + "_range");
            for (int i = 1; i <= id; i++) {
                bookEntity.getReaderIds().add(String.valueOf(20000 + i));
            }
            return bookEntity;
        })
        .collect(Collectors.toList()));

RangePrimaryKey rangePrimaryKey = RangePrimaryKey.builder()
        .addSubRangeStringValue("floor", "f20", "f50")
        .build();
Assert.assertEquals(dao.rangeGet(rangePrimaryKey).size(), 33);

dao.rangeConsume(RangePrimaryKey.builder().addFullRangePrimaryKeyColumn("floor").build(), bookEntity -> {
    System.out.println(bookEntity.getAuthor());
    dao.delete(PrimaryKey.builder().addColumn("floor", bookEntity.getFloor()).build());
});

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

# 4.4column get操作

column get为按照一个或多个列值进行匹配，操作有两个接口columnGet(SearchColumn searchColumn) 与columnGet(SearchColumn searchColumn, int offset, int count, Boolean asc)，前者默认升序列出满足条件的前十条，后者可以自定义。

// 查询ibsn为12345，author为Jon的数据
List<BookEntity> bookEntities = dao.columnGet(SearchColumn.builder()
        .addColumn("ibsn", "12345")
        .addColumn("author", "Jon").build());

1
2
3
4

List<BookEntity> bookEntities = dao.columnGet(SearchColumn.builder()
        .addColumn("ibsn", "12345")
        .addColumn("author", "Jon").build(), 0, 10, true);

1
2
3

HBase与MongoDB可以直接使用
阿里云TableStore需要额外在类要查询的列上声明@TSAttrColumn注解，每次启动都会重新建立索引，建立多元索引需要30秒左右时间。

@Data
@TSTable(name = "ut_book")
public class BookEntity {

    @TSPKColumn
    private String floor = "";

    @TSAttrColumn
    @TSIndexColumn
    private int ibsn;

    @TSAttrColumn
    @TSIndexColumn
    private String author = "";

    @TSAttrColumn(dynamicColumnNames = true)
    private List<String> readerIds = new ArrayList<>(10);
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

# 5.切换配置

如上所示配置时，启动程序，可以通过指定配置不同来切换不同的存储

$ java -Dspring.profiles.active=aliyun/hbase/mongo/dynamo -jar xxx.jar

← 消息队列短信网关 →