Repository navigation
Commit 7d13bb1
Squashed commit of the following(only BE):
commit e5ead33dd247a6b8e301209b86f19a2089103551
Author: plat1ko <[email protected]>
Date: Mon Jun 20 15:37:21 2022 +0800
temporarily avoid src/io conflicts with doris/master
commit 2ffee9d8906af61c40203a71fd0c30db14fbb47a
Author: plat1ko <[email protected]>
Date: Fri Jun 17 20:59:41 2022 +0800
(cold_on_s3) Report remote data size in tablets (apache#195)
* be report tablet remote size
* update TTabletStat
* fe report tablet remote data size
commit ee0d856685c1f089f585d032f26203d4397dc3e7
Author: deardeng <[email protected]>
Date: Fri Jun 17 16:06:23 2022 +0800
(cold_on_s3) support modify partition add storage policy
commit bbf81a905ff3bda38f80f629755ad983c6f6d556
Author: deardeng <[email protected]>
Date: Fri Jun 17 10:24:14 2022 +0800
(cold_on_s3) support alter table add storage policy
commit 1c662b90d59ef8a870ebb7cc89801433fdab1f8a
Author: plat1ko <[email protected]>
Date: Thu Jun 16 16:01:02 2022 +0800
(cold_on_s3) directly upload rowset exceeded ttl (apache#186)
* directly upload rowset exceed ttl
* clang-format
commit f17709819a2a51b2783cceee2b32451740f95256
Author: plat1ko <[email protected]>
Date: Thu Jun 16 11:39:08 2022 +0800
(cold_on_s3) Implement and rewrite some filesystem methods (apache#183)
* remove temporary test code
* no need to send fs to LocalFileWriter ctor
* rewrite some fs methods
* fix opened fd leak
commit 1f7a09d90e38a16574f841701fa04b8164c5155a
Author: Luwei <[email protected]>
Date: Tue Jun 14 18:07:52 2022 +0800
(cold_on_s3) fix log printing (apache#176)
commit 41b2ca65c6340e9432e58d53816b380bdfa9e9b7
Author: Luwei <[email protected]>
Date: Tue Jun 14 17:02:48 2022 +0800
(cold_on_s3) fix rowset write timestamp bug (apache#175)
commit f987fdffdc9d3245e44f226ea19fa489a3951652
Author: deardeng <[email protected]>
Date: Tue Jun 14 16:14:56 2022 +0800
(cold_on_s3) fix be core in file cache and fix be report remote capacity error (apache#174)
commit 4190cd3784a7c308afdc0f29878a40bc476999f5
Author: deardeng <[email protected]>
Date: Tue Jun 14 11:17:23 2022 +0800
(cold_on_s3) fix compile error and thread pool core
commit f76186517b65b8bf70dc2fa456f2ec32d791e4e7
Author: plat1ko <[email protected]>
Date: Mon Jun 13 19:15:37 2022 +0800
(cold_on_s3) Revert FilePathDesc (apache#136)
* change cooldown pick rowset strategy; add/modify some log
* revert FilePathDesc
* return unsupport error to avoid undefined behavior
* patch to reduce path copy
* fix be ut
commit 9ddac6074840417b7a73a4eb8c3f4da00f97a4d1
Author: Luwei <[email protected]>
Date: Mon Jun 13 16:07:36 2022 +0800
(cold_on_s3) Add ut for s3 read (apache#147)
commit d325739ab1ba4e13539ec0b6401ebc13a2dad851
Author: Luwei <[email protected]>
Date: Mon Jun 13 16:07:21 2022 +0800
(cold_on_s3) Add metrics for file system operation (apache#146)
* Add metrics for file system operation
* fix status code
commit b42bb65cc24870f78f3f4a5c7682d0983eb7ac8b
Author: deardeng <[email protected]>
Date: Mon Jun 13 12:53:10 2022 +0800
(cold_on_s3) support show proc backends return remote used capacity
commit fdffe69455fe867d3d95a85311684664abaaf5bb
Author: deardeng <[email protected]>
Date: Fri Jun 10 16:08:09 2022 +0800
(cold_on_s3) support storage policy use default policy
commit 1921e75bce6b85bac61f278b3077a47c916aab00
Author: deardeng <[email protected]>
Date: Thu Jun 9 10:40:46 2022 +0800
(cold_on_s3) create table support different partition use their own storage policy
commit ebb0b585a1dbc8fae146cc742f8000b1b4033507
Author: plat1ko <[email protected]>
Date: Fri Jun 10 22:28:40 2022 +0800
(cold_on_s3) Implement some fs methods (apache#163)
* implement some fs methods
* add more log info
commit 89e820d3f1a9f6b4680bf445ff60a825c315bcab
Author: Lei Zhang <[email protected]>
Date: Thu Jun 9 23:12:58 2022 +0800
[fix](be) fix asan be set_storage_medium core (apache#9986) (apache#9987)
apache#9986
commit ffe2504e24c9afb06b7e843b455f341f1e8864d7
Author: plat1ko <[email protected]>
Date: Thu Jun 9 16:17:06 2022 +0800
[hot-fix](cold_on_s3) do not skip 0 segment rowset (apache#143)
commit f8cd4a12a838d2ee25d0ec72acc3d346f8c24db9
Author: Luwei <[email protected]>
Date: Wed Jun 8 16:09:35 2022 +0800
(cold_on_s3) Optimize cooldown policy (apache#113)
* Optimize cooldown policy
* fix ut
* remove useless log
* add ut
* add ut
* rename var
* fix typo and ut
commit 08aae82daae19d9d7dc4bc0dd294def6d1ea6157
Author: plat1ko <[email protected]>
Date: Wed Jun 8 16:02:41 2022 +0800
provide quick_cooldown switch for regression test (apache#132)
commit 704af00135ff3aacfe746ff617efebfe19e7c53a
Author: deardeng <[email protected]>
Date: Tue Jun 7 18:05:01 2022 +0800
(cold_on_s3) fix multiple fe synchronization resource storage policy exit
commit 58f09c47d4128d6cfe4959a61a0d0e3bde0cbc82
Author: platoneko <[email protected]>
Date: Thu Jun 2 11:38:46 2022 +0800
(cold_on_s3) fix move bug
commit cb4d4d766828103355c47dff8cc37c018c767eb5
Author: deardeng <[email protected]>
Date: Thu Jun 2 11:24:16 2022 +0800
(cold_on_s3) fix clang report '_exec_env' is not used
commit 6b8f6f2b62e01abfdd30b5c42b9c72d554ec5f4e
Author: platoneko <[email protected]>
Date: Thu Jun 2 00:05:42 2022 +0800
(cold_on_s3) reformat be code to make ci happy
commit 4168d9fb1c2ddad57cff7afb9da20272f1a0d389
Author: platoneko <[email protected]>
Date: Wed Jun 1 23:44:56 2022 +0800
(cold_on_s3) fix be ut
commit 110fc7421ceba32a8cf46b237e65309d94a60752
Author: platoneko <[email protected]>
Date: Wed Jun 1 22:36:37 2022 +0800
(cold_on_s3) fix fe compile error
commit a74f84a7f5980c5c7cc5d5c80b0bfd379cfcf11d
Author: plat1ko <[email protected]>
Date: Thu May 26 11:15:31 2022 +0800
[Enhancement](cold_on_s3) refresh fs map (apache#67)
* refresh fs map
* fetch cooldown ttl from storage policy
commit 71a4f364133e1686d0e86a04ab785a64c47ae701
Author: plat1ko <[email protected]>
Date: Tue May 24 21:28:06 2022 +0800
(cold_on_s3) Do not compact remote rowsets; modify pick cooldown rowset strategy (apache#53)
* time from the beginning of upload
* compact to appropriate state before upload
* only do compaction on local rowsets
commit b434d2c9a9c4f4d9d81b4fd3bbabb6cd913e7262
Author: plat1ko <[email protected]>
Date: Tue May 24 19:22:25 2022 +0800
(cold_on_s3) use noncopy iostream for s3 get object (apache#54)
commit 606b98217eb261b9001da77bf766b52c16efb426
Author: plat1ko <[email protected]>
Date: Tue May 24 18:42:09 2022 +0800
(cold_on_s3) copy remote rowset meta to new tablet meta (apache#52)
commit e218f051e685e27a1392befb40ee7fef832a1515
Author: deardeng <[email protected]>
Date: Tue May 24 12:58:43 2022 +0800
(cold_on_s3) support storage policy change fix thrift merge redefine
commit 152fa1856fb3034afcf0b3dc4272b0bafac8f630
Author: deardeng <[email protected]>
Date: Tue May 24 11:45:13 2022 +0800
(cold_on_s3) support storage policy change and create table add storage policy name (apache#39)
* [feature](cold-hot)support storage policy change and create table add storage policy name
* support storage policy change and create table add storage policy name, check resource policy exist when create table.
* when create table use storage policy check it exist
* support storage policy change and create table add storage policy name and when create table use storage policy check it exist
* support storage policy change fix some bug.
* fix some code style
commit 77101dd7f563e1d0ddd03fbe6b35b4196811a87e
Author: deardeng <[email protected]>
Date: Sat May 21 07:37:43 2022 -0700
(cold_on_s3) support storage policy (apache#28)
* [feature](cold-hot)support storage policy
* fix review
* fix review v1
commit 9923d59e97c25e6d6fb54bf09ed6e0a00a5704c3
Author: plat1ko <[email protected]>
Date: Fri May 20 16:22:36 2022 +0800
(cold_on_s3) Batch upload to improve upload speed (apache#38)
* test upload duration
* use s3 transfer
* batch upload
* some fix in s3 filesystem
commit 29656e73f33e94bb85a954eb31dca9e6eea33cf4
Author: Luwei <[email protected]>
Date: Wed May 18 15:08:40 2022 +0800
(cold_on_s3) Implement periodic uploading of rowsets to remote storage (apache#18)
* Implement periodic uploading of rowsets to remote storage
* add queue size check
* add test config for cooldwon
* modify read path
commit c35ed8bc87d212d442f1b57be0340230de4fb619
Author: plat1ko <[email protected]>
Date: Tue May 17 17:22:19 2022 +0800
(cold_on_s3) fe distinguish drop partition (apache#26)
commit e216b0320d71d04a4fde829f13fd8c8bfa94854b
Author: deardeng <[email protected]>
Date: Mon May 16 21:02:51 2022 -0700
(cold_on_s3) support clone task don't download file in s3 (apache#15)
* apache#12 support clone task don't download file in s3
* apache#12 add clone ut
* fix code format
* fix commit apache#12
commit be7521900726c9cd401e3a6503177e0365fc8a9a
Author: plat1ko <[email protected]>
Date: Tue May 17 12:01:32 2022 +0800
[Enhancement](cold_on_s3) FE distinguish drop table or drop replica; disk capacity distinguish local or remote (apache#24)
* disk capacity differentiates between local and remote
* fe drop table
commit 414a1cb0c5d9777b6eb0b2ad5d7557dc1bf4a243
Author: plat1ko <[email protected]>
Date: Mon May 16 11:55:42 2022 +0800
[Enhancement](cold_on_s3) Physically remove remote rowsets when drop table/partition (apache#14)
* update is_local logical
* remove remote rowsets when drop table/partition
commit ed7cf258051716ff99c37812a7fa63ac1fc69f3f
Author: Luwei <[email protected]>
Date: Mon May 16 11:17:30 2022 +0800
(cold_on_s3) Implement reading rowset data from remote storage (apache#11)
* adjust io stack
* support s3
* refine code
* fix ut
* remove useless code
* rename rblock
* fix err msg
commit da6ca91f22d30e9d16e4dd70dbcad30714d321c9
Author: plat1ko <[email protected]>
Date: Wed May 11 16:29:54 2022 +0800
(cold_on_s3) Implement tablet cooldown (apache#10)
* tablet cooldown
* add global_local_filesystem to keep compatible
* test tablet cooldown
* revert engine_storage_migration_task_test.cpp
commit 83f4a3bfcd70cd747941338cd89148fb69d272ae
Author: platoneko <[email protected]>
Date: Sun May 1 12:06:41 2022 +0800
(cold_on_s3) Rowset write path using fs (apache#7)
* replace WritableBlock with FileWriter
* local file system
* set fs when load dir
* fix be ut
commit 3585fe6
Author: Xinyi Zou <[email protected]>
Date: Tue Jun 7 11:31:49 2022 +0800
[fix][mem tracker] Fix logout load task mem tracker dcheck fail (apache#9943)
* fix tracker 0602
* fix format
commit 429a41d
Author: Xinyi Zou <[email protected]>
Date: Tue May 31 19:12:42 2022 +0800
[bugfix] Fix BE core about vectorized join build thread memtracker switch, and FileStat duplicate
commit 89d3a36
Author: Pxl <[email protected]>
Date: Thu May 26 16:51:01 2022 +0800
[Bug] [Vectorized] add padding when load char type data (apache#9734)
Co-authored-by: Luwei <[email protected]>
Co-authored-by: deardeng <[email protected]>1 parent 006283c commit 7d13bb1
140 files changed
Lines changed: 3694 additions & 1352 deletions
File tree
- be
- src
- agent
- common
- env
- http/action
- io
- fs
- olap
- rowset
- segment_v2
- task
- runtime
- service
- util
- test
- olap
- rowset
- segment_v2
- test_data
- runtime
- tools
- gensrc
- proto
- thrift
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
550 | 550 | | |
551 | 551 | | |
552 | 552 | | |
| 553 | + | |
553 | 554 | | |
554 | 555 | | |
555 | 556 | | |
| |||
743 | 744 | | |
744 | 745 | | |
745 | 746 | | |
| 747 | + | |
746 | 748 | | |
747 | 749 | | |
748 | 750 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
39 | 39 | | |
40 | 40 | | |
41 | 41 | | |
42 | | - | |
| 42 | + | |
43 | 43 | | |
44 | 44 | | |
45 | 45 | | |
| |||
89 | 89 | | |
90 | 90 | | |
91 | 91 | | |
92 | | - | |
| 92 | + | |
| 93 | + | |
93 | 94 | | |
94 | 95 | | |
95 | 96 | | |
| |||
152 | 153 | | |
153 | 154 | | |
154 | 155 | | |
155 | | - | |
156 | | - | |
| 156 | + | |
| 157 | + | |
157 | 158 | | |
158 | 159 | | |
159 | 160 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
82 | 82 | | |
83 | 83 | | |
84 | 84 | | |
85 | | - | |
86 | | - | |
| 85 | + | |
| 86 | + | |
87 | 87 | | |
88 | 88 | | |
89 | 89 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
35 | 35 | | |
36 | 36 | | |
37 | 37 | | |
| 38 | + | |
38 | 39 | | |
39 | 40 | | |
40 | 41 | | |
41 | 42 | | |
42 | 43 | | |
43 | 44 | | |
44 | 45 | | |
45 | | - | |
46 | 46 | | |
47 | 47 | | |
48 | 48 | | |
| |||
193 | 193 | | |
194 | 194 | | |
195 | 195 | | |
196 | | - | |
| 196 | + | |
197 | 197 | | |
198 | | - | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
199 | 204 | | |
200 | 205 | | |
201 | 206 | | |
| |||
361 | 366 | | |
362 | 367 | | |
363 | 368 | | |
| 369 | + | |
364 | 370 | | |
365 | 371 | | |
366 | 372 | | |
| |||
437 | 443 | | |
438 | 444 | | |
439 | 445 | | |
| 446 | + | |
| 447 | + | |
| 448 | + | |
| 449 | + | |
| 450 | + | |
440 | 451 | | |
441 | 452 | | |
442 | 453 | | |
| |||
854 | 865 | | |
855 | 866 | | |
856 | 867 | | |
857 | | - | |
858 | | - | |
| 868 | + | |
| 869 | + | |
| 870 | + | |
| 871 | + | |
| 872 | + | |
| 873 | + | |
| 874 | + | |
| 875 | + | |
| 876 | + | |
859 | 877 | | |
860 | 878 | | |
861 | 879 | | |
| |||
1036 | 1054 | | |
1037 | 1055 | | |
1038 | 1056 | | |
1039 | | - | |
1040 | | - | |
| 1057 | + | |
| 1058 | + | |
1041 | 1059 | | |
1042 | 1060 | | |
1043 | 1061 | | |
| |||
1171 | 1189 | | |
1172 | 1190 | | |
1173 | 1191 | | |
1174 | | - | |
| 1192 | + | |
1175 | 1193 | | |
1176 | 1194 | | |
1177 | 1195 | | |
1178 | | - | |
| 1196 | + | |
| 1197 | + | |
1179 | 1198 | | |
1180 | 1199 | | |
1181 | | - | |
| 1200 | + | |
1182 | 1201 | | |
1183 | 1202 | | |
1184 | 1203 | | |
| |||
1565 | 1584 | | |
1566 | 1585 | | |
1567 | 1586 | | |
1568 | | - | |
1569 | 1587 | | |
1570 | 1588 | | |
1571 | 1589 | | |
| |||
1683 | 1701 | | |
1684 | 1702 | | |
1685 | 1703 | | |
1686 | | - | |
| 1704 | + | |
| 1705 | + | |
| 1706 | + | |
| 1707 | + | |
| 1708 | + | |
| 1709 | + | |
| 1710 | + | |
| 1711 | + | |
| 1712 | + | |
| 1713 | + | |
| 1714 | + | |
| 1715 | + | |
| 1716 | + | |
| 1717 | + | |
| 1718 | + | |
| 1719 | + | |
| 1720 | + | |
| 1721 | + | |
| 1722 | + | |
| 1723 | + | |
| 1724 | + | |
| 1725 | + | |
| 1726 | + | |
| 1727 | + | |
| 1728 | + | |
| 1729 | + | |
| 1730 | + | |
| 1731 | + | |
| 1732 | + | |
| 1733 | + | |
| 1734 | + | |
| 1735 | + | |
| 1736 | + | |
| 1737 | + | |
| 1738 | + | |
| 1739 | + | |
| 1740 | + | |
| 1741 | + | |
| 1742 | + | |
| 1743 | + | |
| 1744 | + | |
| 1745 | + | |
| 1746 | + | |
| 1747 | + | |
| 1748 | + | |
| 1749 | + | |
| 1750 | + | |
| 1751 | + | |
| 1752 | + | |
| 1753 | + | |
| 1754 | + | |
| 1755 | + | |
| 1756 | + | |
| 1757 | + | |
| 1758 | + | |
1687 | 1759 | | |
1688 | 1760 | | |
| 1761 | + | |
1689 | 1762 | | |
1690 | 1763 | | |
1691 | 1764 | | |
| |||
1696 | 1769 | | |
1697 | 1770 | | |
1698 | 1771 | | |
| 1772 | + | |
1699 | 1773 | | |
1700 | 1774 | | |
1701 | | - | |
1702 | | - | |
1703 | | - | |
1704 | | - | |
1705 | | - | |
1706 | | - | |
1707 | | - | |
1708 | | - | |
1709 | | - | |
1710 | | - | |
1711 | | - | |
1712 | | - | |
1713 | | - | |
1714 | | - | |
1715 | | - | |
1716 | | - | |
1717 | | - | |
1718 | | - | |
1719 | | - | |
1720 | | - | |
1721 | | - | |
1722 | | - | |
1723 | | - | |
1724 | | - | |
1725 | | - | |
1726 | | - | |
1727 | | - | |
1728 | | - | |
1729 | | - | |
1730 | | - | |
1731 | | - | |
1732 | | - | |
1733 | | - | |
1734 | | - | |
1735 | | - | |
1736 | | - | |
1737 | | - | |
1738 | | - | |
1739 | | - | |
1740 | | - | |
1741 | | - | |
1742 | | - | |
1743 | | - | |
1744 | | - | |
1745 | | - | |
1746 | | - | |
1747 | | - | |
1748 | | - | |
1749 | | - | |
1750 | | - | |
1751 | | - | |
1752 | | - | |
1753 | | - | |
1754 | | - | |
1755 | | - | |
1756 | | - | |
1757 | | - | |
1758 | | - | |
1759 | | - | |
1760 | | - | |
1761 | | - | |
1762 | | - | |
1763 | | - | |
1764 | | - | |
1765 | | - | |
1766 | | - | |
1767 | | - | |
1768 | | - | |
1769 | | - | |
1770 | 1775 | | |
1771 | | - | |
1772 | | - | |
1773 | | - | |
1774 | | - | |
1775 | | - | |
1776 | | - | |
1777 | | - | |
1778 | | - | |
1779 | | - | |
1780 | | - | |
1781 | | - | |
1782 | | - | |
1783 | | - | |
1784 | | - | |
1785 | | - | |
1786 | | - | |
1787 | | - | |
1788 | | - | |
1789 | | - | |
1790 | | - | |
1791 | | - | |
1792 | | - | |
1793 | | - | |
1794 | | - | |
1795 | | - | |
1796 | | - | |
1797 | | - | |
1798 | | - | |
1799 | | - | |
1800 | | - | |
1801 | | - | |
1802 | | - | |
1803 | | - | |
| 1776 | + | |
| 1777 | + | |
| 1778 | + | |
| 1779 | + | |
| 1780 | + | |
| 1781 | + | |
| 1782 | + | |
| 1783 | + | |
| 1784 | + | |
| 1785 | + | |
| 1786 | + | |
| 1787 | + | |
| 1788 | + | |
| 1789 | + | |
| 1790 | + | |
| 1791 | + | |
| 1792 | + | |
| 1793 | + | |
| 1794 | + | |
| 1795 | + | |
| 1796 | + | |
1804 | 1797 | | |
1805 | | - | |
1806 | | - | |
1807 | | - | |
1808 | 1798 | | |
1809 | 1799 | | |
1810 | 1800 | | |
0 commit comments