温馨提示×

温馨提示×

您好,登录后才能下订单哦!

密码登录×
登录注册×
其他方式登录
点击 登录注册 即表示同意《亿速云用户服务条款》

Influx DB first explore

发布时间:2020-02-25 22:47:36 来源:网络 阅读:264 作者:zhanjia 栏目:大数据

Key characters:

   Store time series data:  
  • Data volume is large
  • High number of concurrent reads/writes
  • Optimized for (C)reate, (R)ead:    TSM Tree
  • Limited support for (U)pdate, (D)elet  
Serial data is more important than single data point:
  • No traditional id, data identified by series and timestamp   
  • Good support for aggregation
Schemaless
  • New tags can be added to new data record on demand

Key concept:

Measurement:
  • Similar to SQL table
Tag:
  •       Searchable
  •       Indexed
  •       Value must be string  
  •       Cardinality limitation: (memory)
  •       Optional
  •       Support group by
Field:
  •       Searchable (full scan)
  •       Not indexed
  •       Value can be string, float, integer, boolean
  •       Can apply aggregation function
  •       At least one field in measurement (table)
  •       Support aggregation functions
Retention policy:
  • Two key parts:
    Duration: how long data is kept  
    Replication: replication factor
  • One measurement can have many retention policies  
  • Default retention policy: duration = infinite and replication factor = 1
  • Can be altered at runtime
Series:

The collection of data in the InfluxDB data structure that share a measurement, tag set, and retention policy. (not including fields)

Shard:
  • Mapped to a TSM file
  • Belongs to a single shard group
  • A shard group can have multiple shards, each belongs to different series
  • A shard group has shard duration, determine the duration of data span
  • Default shard group duration can be deduced from retention policy duration

Special Features:

Continuous query:

Recommendation:

  • Adding data in time ascending order is significantly more performant.
  • Limit tag cardinality for optimal memory usage
  • Use SSD for storage  
    https://docs.influxdata.com/influxdb/v1.7/guides/hardware_sizing/#general-hardware-guidelines-for-a-single-node
  • Use long shard duration for better query performance
  • Use short shard duration for more efficient data deletion
  • Shard groups should be twice as long as the longest time range of the most frequent queries
  • Shard groups should each contain more than 100,000 points per shard group
  • Shard groups should each contain more than 1,000 points per series
  • When writing historical data, it is recommended to temporarily setting a longer shard group duration to avoid lots of shard

Limitations:

  • Unable to store duplicated data.
  • Query might not see most recently written data  (trade off consistency for performance)
  • Only able to update field value with same measure + tag set + timestamp
  • Unable to update tag value
  • Unable to delete tag value
  • Cannot delete a single point (workaround: delete base on time), can only drop/delete series
  • No join table is supported

Installation:

version: '3.3'

services:
influxdb:
image: 'influxdb:1.7.8-alpine'
environment:
INFLUXDB_DB: monitordb
INFLUXDB_HTTP_AUTH_ENABLED: 'true'
INFLUXDB_ADMIN_USER: admin
INFLUXDB_ADMIN_PASSWORD: password
INFLUXDB_USER: influxuser
INFLUXDB_USER_PASSWORD: password

ports:
- "8086:8086"
- "8088:8088"

volumes:
- /opt/volumes/influxdb/data:/var/lib/influxdb

networks:
layer0:
aliases:
- 'influxdb'

labels:
- "dev.description=transaction monitoring influx db"

networks:
layer0:

Commandline:  
./usr/bin/influx -username 'admin' -password 'password'

Issues:

If data point's time is before retention policy's valid time range, inserting it will show error:  "partial write: points beyond retention policy dropped=1"

If data point is inserted with retention policy, querying the data should also set retention policy. Eg  "select * from ninetyday.test1", otherwise, no record will be found 

Batch processing uses a separate thread pool and data is send to server if data is accumulated until flush duration (1s) or buffer limit exceeded (10000 records).   Need to call InfluxDB.close after batch processing to ensure proper resource reclamation 

max-values-per-tag limit exceeded (100000/100000):   /etc/influxdb/influxdb.conf  max-values-per-tag = 0    or docker environment variable:  INFLUXDB_DATA_MAX_VALUES_PER_TAG: 0

Performance: 

Insertion: 
  • 100000 records with sequential timestamp takes about 4 seconds
  • 100000 records with random timestamp takes about 4.4 seconds
Query:
  • select * from ninetyday.test1 where time < now() limit 100  takes about 1.1 seconds
  • select count(eventtype) from ninetyday.test1 where time < now()  takes about 1.2 seconds
  • select count(eventtype) from ninetyday.test1 where time < now() group by userId  takes about 1.5 seconds
  • select * from ninetyday.test1 where time < now() and transactionId = 'transaction1'  takes about 106 ms
  • select * from ninetyday.test1 where time < now() and userId = 'user1'    takes about 130ms
Insertion:
  •     1000000 records with random timestamp takes about 37 seconds
Query: 
  • select * from ninetyday.test1 where time < now() limit 100  takes about 8.3 seconds
  • select count(eventtype) from ninetyday.test1 where time < now()  read timeout
  • select count(eventtype) from ninetyday.test1 where time < now() group by userId  read timeout
  • select * from ninetyday.test1 where time < now() and transactionId = 'transaction1'  takes about 70 ms
  • select * from ninetyday.test1 where time < now() and userId = 'user1'   takes about 130ms
向AI问一下细节

免责声明:本站发布的内容(图片、视频和文字)以原创、转载和分享为主,文章观点不代表本网站立场,如果涉及侵权请联系站长邮箱:is@yisu.com进行举报,并提供相关证据,一经查实,将立刻删除涉嫌侵权内容。

AI