metadata

datahub-frontend / datahub-gms / datahub-ingestion / datahub-actions


I. storage backend

    Cassandra ---> DataHub GMS can use Cassandra as an alternate storage backend.
    mariadb
    mysql (default)
    postgres

II. Neo4j

    DataHub uses Neo4j as graph db in the backend to serve graph queries.

III. Elasticsearch & Kibana

    DataHub uses Elasticsearch as a search engine.

IV. Kafka, Zookeeper and Schema Registry

    DataHub uses Kafka as the pub-sub message queue in the backend. 

acryldata/datahub-ingestion
acryldata/datahub-gms
acryldata/datahub-frontend-react
acryldata/datahub-mae-consumer
acryldata/datahub-mce-consumer
acryldata/datahub-upgrade
acryldata/datahub-kafka-setup
acryldata/datahub-elasticsearch-setup
acryldata/datahub-mysql-setup
acryldata/datahub-postgres-setup
acryldata/datahub-actions
## Do not use acryldata/acryl-datahub-actions as that is deprecated and no longer used.

## Dependencies:
Kafka, Zookeeper, and Schema Registry
Elasticsearch
MySQL
(Optional) Neo4j

docker pull acryldata/datahub-frontend-react:v1.0.0
docker pull acryldata/datahub-gms:v1.0.0

# https://github.com/datahub-project/datahub/blob/master/docker/docker-compose.yml

spark.sql.catalog.spark_catalog的子属性主要用于配置Spark的目录服务，特别是当使用Iceberg或其他类型的目录时。以下是一些常见的子属性：

子属性列表
type：指定目录的类型，如hive、hadoop、rest、glue、jdbc或nessie。


spark.sql.catalog.spark_catalog.type=hive
uri：用于指定目录的URI，例如Hive Metastore的Thrift URL或REST API的URL。


spark.sql.catalog.spark_catalog.uri=thrift://localhost:9083
warehouse：指定数据仓库的路径，通常用于Hadoop类型的目录。


spark.sql.catalog.spark_catalog.warehouse=hdfs://nn:8020/warehouse/path
default-namespace：设置目录的默认命名空间。


spark.sql.catalog.spark_catalog.default-namespace=default
catalog-impl：用于指定自定义的目录实现类。


spark.sql.catalog.spark_catalog.catalog-impl=org.apache.iceberg.rest.RESTCatalog
io-impl：指定自定义的文件IO实现类。


spark.sql.catalog.spark_catalog.io-impl=org.apache.iceberg.io.FileIO
metrics-reporter-impl：指定自定义的指标报告实现类。


spark.sql.catalog.spark_catalog.metrics-reporter-impl=org.apache.iceberg.metrics.ConsoleMetricsReporter
cache-enabled：是否启用目录缓存。


spark.sql.catalog.spark_catalog.cache-enabled=true
cache.expiration-interval-ms：缓存过期时间（毫秒）。


spark.sql.catalog.spark_catalog.cache.expiration-interval-ms=30000
table-default.propertyKey：设置默认的表属性值。


spark.sql.catalog.spark_catalog.table-default.write-format-version=2
table-override.propertyKey：设置强制覆盖的表属性值。


spark.sql.catalog.spark_catalog.table-override.write-format-version=2
view-default.propertyKey：设置默认的视图属性值。


spark.sql.catalog.spark_catalog.view-default.write-format-version=2
use-nullable-query-schema：是否在创建表时保留字段的可空性。


spark.sql.catalog.spark_catalog.use-nullable-query-schema=true
这些子属性根据具体的目录类型和需求进行配置，以实现灵活的元数据管理和数据访问。

A Spark TableCatalog implementation that wraps an Iceberg Catalog.
This supports the following catalog configuration options:

type - catalog type, "hive" or "hadoop" or "rest". To specify a non-hive or hadoop catalog, use the catalog-impl option.
uri - the Hive Metastore URI for Hive catalog or REST URI for REST catalog
warehouse - the warehouse path (Hadoop catalog only)
catalog-impl - a custom Catalog implementation to use
io-impl - a custom FileIO implementation to use
metrics-reporter-impl - a custom MetricsReporter implementation to use
default-namespace - a namespace to use as the default
cache-enabled - whether to enable catalog cache
cache.case-sensitive - whether the catalog cache should compare table identifiers in a case sensitive way
cache.expiration-interval-ms - interval in millis before expiring tables from catalog cache. Refer to CatalogProperties.CACHE_EXPIRATION_INTERVAL_MS for further details and significant values.
table-default.$tablePropertyKey - table property $tablePropertyKey default at catalog level
table-override.$tablePropertyKey - table property $tablePropertyKey enforced at catalog level