Skip to main content

This week in Databend #70

· 5 min read
PsiACE

Databend is a powerful cloud data warehouse. Built for elasticity and efficiency. Free and open. Also available in the cloud: https://app.databend.com .

What's Changed​

Below is a list of some major changes that we don't want you to miss.

Exciting New Features ✨​

format

  • better checking of format options (#8981)
  • add basic schema infer for parquet (#9043)

query

  • QualifiedName support 'db.table.' and 'table.' (#8965)
  • support bulk insert without exprssion (#8966)

storage

  • add cache layer for fuse engine (#8830)
  • add system table system.memory_statistics (#8945)
  • add optimize statistic ddl support (#8891)

Code Refactor 🎉​

base

  • remove common macros (#8936)

format

  • TypeDeserializer get rid of FormatSetting (#8950)

planner

  • refactor extract or predicate (#8951)

processors

  • optimize join by merging build data block (#8961)

new expression

  • allow sparse column id in chunk, redo #8789 with a new approach. (#9008)

Documentation 📔​

Thoughtful Bug Fix 🔧​

base

  • try fix lost tracker (#8932)

meta

  • fix share db bug, create DatabaseIdToName if need (#9006)

mysql handler

  • fix mysql conns leak (#8894)

processors

  • try fix update list memory leak (#9023)

storage

  • read and write block in parallel when compact (#8921)

News​

Let's take a look at what's new at Datafuse Labs & Databend each week.

Preview of Infer Schema​

To load data from a stage or location, users need to create table first. However, sometimes users don't know the file schema, or the schema is too complex / too simple to be input by users.

Allowing infer schema from existing files makes our users' lives easier. Also, this feature will unlock the databend from implementing select * from @my_stage.

INFER 's3://mybucket/data.csv' FILE_FORMAT = ( TYPE = CSV );
+-------------+---------+----------+
| COLUMN_NAME | TYPE    | NULLABLE |
|-------------+---------+----------|
| CONTINENT   | TEXT    | True     |
| COUNTRY     | VARIANT | True     |
+-------------+---------+----------+

#9043 has added basic schema infer for parquet. We will be moving forward with support for select from stage (#7211) on this basis.

Learn More

Issues​

Meet issues you may be interested in and try to solve it.

Add tls support for mysql handler​

The just released opensrv-mysql v0.3.0 includes support for tls and it is time to introduce it for Databend.

let (is_ssl, init_params) = opensrv_mysql::AsyncMysqlIntermediary::init_before_ssl(
    &mut shim,
    &mut r,
    &mut w,
    &Some(tls_config.clone()),
)
.await
.unwrap();

opensrv_mysql::secure_run_with_options(shim, w, ops, tls_config, init_params).await

Issue 8983: Feature: tls support for mysql handler

If you find it interesting, try to solve it or participate in discussions and PR reviews. Or you can click on https://link.databend.rs/i-m-feeling-lucky to pick up a good first issue, good luck!

Changlogs​

You can check the changelogs of Databend nightly to learn about our latest developments.

Contributors​

Thanks a lot to the contributors for their excellent work this week.

andylokandyariesdevilb41shBohuTANGdantengskydrmingdrmer
andylokandyariesdevilb41shBohuTANGdantengskydrmingdrmer
everpcpcflaneur2020leiyskylichuangmergify[bot]PsiACE
everpcpcflaneur2020leiyskylichuangmergify[bot]PsiACE
sandfleesoyeric128sundy-liTCeasonTracyZYJXuanwo
sandfleesoyeric128sundy-liTCeasonTracyZYJXuanwo
xudong963youngsofunyufan022zhang2014zhyass
xudong963youngsofunyufan022zhang2014zhyass

Meet Us​

Please join the DatafuseLabs Community if you are interested in Databend.

We are looking forward to seeing you try our code. We have a strong team behind you to ensure a smooth experience in trying our code for your projects. If you are a hacker passionate about database internals, feel free to play with our code.

You can submit issues for any problems you find. We also highly appreciate any of your pull requests.