Skip to content

Commit b8bae84

Browse files
Ma77Ballchenlicakunwp1aglinxinyuan
authored
feat(ui): add pagination and search for wide-column tables (apache#4086)
<!-- Thanks for sending a pull request (PR)! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: [Contributing to Texera](https://github.com/apache/texera/blob/main/CONTRIBUTING.md) 2. Ensure you have added or run the appropriate tests for your PR 3. If the PR is a work in progress, mark it as a draft on GitHub. 4. Please write your PR title to summarize what this PR proposes. We are following the Conventional Commits style for PR titles as well. 5. Be sure to keep the PR description updated to reflect all changes. --> ### What changes were proposed in this PR? <!-- Please clarify what changes you are proposing. The purpose of this section is to outline the changes. Here are some tips for you: 1. If you propose a new API, clarify the use case for a new API. 2. If you fix a bug, you can clarify why it is a bug. 3. If it is a refactoring, could you explain what has been changed? 3. It would be helpful to include a before-and-after comparison using screenshots or GIFs. 4. Please consider writing helpful notes for better and faster reviews. --> This PR adds a feature that enables Texera to efficiently handle tables with vast numbers of columns in the result panel. This PR adds UI features that enable Texera to efficiently handle tables with large numbers of columns in the result viewer and related data preview components. Specifically, this PR introduces: 1. Horizontal Column Pagination 1. "Next Columns" and "Previous Columns" buttons have been added. 2. Columns are now loaded in column windows (configurable size, default 25). 3. Prevents UI freezing or overflow when dealing with tables containing hundreds or thousands of columns. 2. Column Search Bar 1. A new search box allows users to filter or jump directly to column names. 2. When a column is found, that column window is automatically loaded and highlighted. 3. Useful for wide schemas such as: 1. large scientific datasets 2. logs with hundreds of attributes 3. denormalized tables or wide joins 3. Improvements to rendering performance 1. The frontend now only renders the visible subset of columns. 2. Reduces DOM load and improves React change detection speed. ### Any related issues, documentation, or discussions? <!-- Please use this section to link to other resources if not mentioned already. 1. If this PR fixes an issue, please include `Fixes apache#1234`, `Resolves apache#1234.` or `Closes apache#1234`. If it is only related, mention the issue number. 2. If there is design documentation, please add the link. 3. If there is a discussion on the mailing list, please add the link. --> Fixes: apache#3825 ### How was this PR tested? <!-- If tests were added, say so here. Or mention that if the PR is tested with existing test cases. Please include/update test cases that check the changes thoroughly, including negative and positive cases if possible. If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future. If tests were not added, please describe why they were not added and/or why it was difficult to add. --> 1. Ran a couple of CSV scan operators that produced a wide output table 2. Clicked through column windows in both directions 3. Using search to jump to: - first column - random middle column - last column ### Was this PR authored or co-authored using generative AI tooling? <!-- If generative AI tooling has been used in the process of authoring this PR, please include the phrase: 'Generated-by: ' followed by the name of the tool and its version. If no, write 'No'. Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details. --> No ### New Layout: <img width="1905" height="733" alt="image" src="https://github.com/user-attachments/assets/18d14c2d-c134-422f-a5d1-2c826cf3a8e9" /> --------- Co-authored-by: Chen Li <[email protected]> Co-authored-by: Chris <[email protected]> Co-authored-by: Xinyuan Lin <[email protected]>
1 parent fffbddf commit b8bae84

18 files changed

Lines changed: 185 additions & 24 deletions

File tree

‎amber/src/main/scala/org/apache/texera/web/model/websocket/request/ResultPaginationRequest.scala‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,5 +23,8 @@ case class ResultPaginationRequest(
2323
requestID: String,
2424
operatorID: String,
2525
pageIndex: Int,
26-
pageSize: Int
26+
pageSize: Int,
27+
columnOffset: Int = 0,
28+
columnLimit: Int = Int.MaxValue,
29+
columnSearch: Option[String] = None
2730
) extends TexeraWebSocketRequest

‎amber/src/main/scala/org/apache/texera/web/service/ExecutionResultService.scala‎

Lines changed: 18 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -437,12 +437,25 @@ class ExecutionResultService(
437437

438438
storageUriOption match {
439439
case Some(storageUri) =>
440+
val (document, schemaOption) = DocumentFactory.openDocument(storageUri)
441+
val virtualDocument = document.asInstanceOf[VirtualDocument[Tuple]]
442+
443+
val columns = {
444+
val schema = schemaOption.get
445+
val allColumns = schema.getAttributeNames
446+
val filteredColumns = request.columnSearch match {
447+
case Some(search) =>
448+
allColumns.filter(col => col.toLowerCase.contains(search.toLowerCase))
449+
case None => allColumns
450+
}
451+
Some(
452+
filteredColumns.slice(request.columnOffset, request.columnOffset + request.columnLimit)
453+
)
454+
}
455+
440456
val paginationIterable = {
441-
DocumentFactory
442-
.openDocument(storageUri)
443-
._1
444-
.asInstanceOf[VirtualDocument[Tuple]]
445-
.getRange(from, from + request.pageSize)
457+
virtualDocument
458+
.getRange(from, from + request.pageSize, columns)
446459
.to(Iterable)
447460
}
448461
val mappedResults = convertTuplesToJson(paginationIterable)

‎common/config/src/main/resources/gui.conf‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -108,5 +108,9 @@ gui {
108108
# whether AI copilot feature is enabled
109109
copilot-enabled = false
110110
copilot-enabled = ${?GUI_WORKFLOW_WORKSPACE_COPILOT_ENABLED}
111+
112+
# the limit of columns to be displayed in the result table
113+
limit-columns = 15
114+
limit-columns = ${?GUI_WORKFLOW_WORKSPACE_LIMIT_COLUMNS}
111115
}
112116
}

‎common/config/src/main/scala/org/apache/texera/config/GuiConfig.scala‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,4 +69,6 @@ object GuiConfig {
6969
conf.getInt("gui.workflow-workspace.active-time-in-minutes")
7070
val guiWorkflowWorkspaceCopilotEnabled: Boolean =
7171
conf.getBoolean("gui.workflow-workspace.copilot-enabled")
72+
val guiWorkflowWorkspaceLimitColumns: Int =
73+
conf.getInt("gui.workflow-workspace.limit-columns")
7274
}

‎common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/DocumentFactory.scala‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -87,8 +87,8 @@ object DocumentFactory {
8787
overrideIfExists = true
8888
)
8989
val serde: (IcebergSchema, Tuple) => Record = IcebergUtil.toGenericRecord
90-
val deserde: (IcebergSchema, Record) => Tuple = (_, record) =>
91-
IcebergUtil.fromRecord(record, schema)
90+
val deserde: (IcebergSchema, Record) => Tuple = (schema, record) =>
91+
IcebergUtil.fromRecord(record, IcebergUtil.fromIcebergSchema(schema))
9292

9393
new IcebergDocument[Tuple](
9494
namespace,
@@ -144,8 +144,8 @@ object DocumentFactory {
144144

145145
val amberSchema = IcebergUtil.fromIcebergSchema(table.schema())
146146
val serde: (IcebergSchema, Tuple) => Record = IcebergUtil.toGenericRecord
147-
val deserde: (IcebergSchema, Record) => Tuple = (_, record) =>
148-
IcebergUtil.fromRecord(record, amberSchema)
147+
val deserde: (IcebergSchema, Record) => Tuple = (schema, record) =>
148+
IcebergUtil.fromRecord(record, IcebergUtil.fromIcebergSchema(schema))
149149

150150
(
151151
new IcebergDocument[Tuple](

‎common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/model/ReadonlyLocalFileDocument.scala‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -54,7 +54,11 @@ private[storage] class ReadonlyLocalFileDocument(uri: URI)
5454
override def get(): Iterator[Nothing] =
5555
throw new NotImplementedError("get is not supported for ReadonlyLocalFileDocument")
5656

57-
override def getRange(from: Int, until: Int): Iterator[Nothing] =
57+
override def getRange(
58+
from: Int,
59+
until: Int,
60+
columns: Option[Seq[String]] = None
61+
): Iterator[Nothing] =
5862
throw new NotImplementedError("getRange is not supported for ReadonlyLocalFileDocument")
5963

6064
override def getAfter(offset: Int): Iterator[Nothing] =

‎common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/model/ReadonlyVirtualDocument.scala‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -52,9 +52,10 @@ trait ReadonlyVirtualDocument[T] {
5252
* Get an iterator of a sequence starting from index `from`, until index `until`.
5353
* @param from the starting index (inclusive)
5454
* @param until the ending index (exclusive)
55+
* @param columns optional sequence of column names to retrieve. If None, retrieves all columns.
5556
* @return an iterator that returns data items of type T
5657
*/
57-
def getRange(from: Int, until: Int): Iterator[T]
58+
def getRange(from: Int, until: Int, columns: Option[Seq[String]] = None): Iterator[T]
5859

5960
/**
6061
* Get an iterator of all items after the specified index `offset`.

‎common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/model/VirtualDocument.scala‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -55,9 +55,10 @@ abstract class VirtualDocument[T] extends ReadonlyVirtualDocument[T] {
5555
* get an iterator of a sequence starting from index `from`, until index `until`
5656
* @param from the starting index (inclusive)
5757
* @param until the ending index (exclusive)
58+
* @param columns the columns to be projected
5859
* @return an iterator that returns data items of type T
5960
*/
60-
def getRange(from: Int, until: Int): Iterator[T] =
61+
def getRange(from: Int, until: Int, columns: Option[Seq[String]] = None): Iterator[T] =
6162
throw new NotImplementedError("getRange method is not implemented")
6263

6364
/**

‎common/workflow-core/src/main/scala/org/apache/texera/amber/core/storage/result/iceberg/IcebergDocument.scala‎

Lines changed: 18 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -105,8 +105,8 @@ private[storage] class IcebergDocument[T >: Null <: AnyRef](
105105
/**
106106
* Get records within a specified range [from, until).
107107
*/
108-
override def getRange(from: Int, until: Int): Iterator[T] = {
109-
getUsingFileSequenceOrder(from, Some(until))
108+
override def getRange(from: Int, until: Int, columns: Option[Seq[String]] = None): Iterator[T] = {
109+
getUsingFileSequenceOrder(from, Some(until), columns)
110110
}
111111

112112
/**
@@ -150,8 +150,13 @@ private[storage] class IcebergDocument[T >: Null <: AnyRef](
150150
*
151151
* @param from start from which record inclusively, if 0 means start from the first
152152
* @param until end at which record exclusively, if None means read to the table's EOF
153+
* @param columns columns to be projected
153154
*/
154-
private def getUsingFileSequenceOrder(from: Int, until: Option[Int]): Iterator[T] =
155+
private def getUsingFileSequenceOrder(
156+
from: Int,
157+
until: Option[Int],
158+
columns: Option[Seq[String]] = None
159+
): Iterator[T] =
155160
withReadLock(lock) {
156161
new Iterator[T] {
157162
private val iteLock = new ReentrantLock()
@@ -259,9 +264,13 @@ private[storage] class IcebergDocument[T >: Null <: AnyRef](
259264

260265
while (!currentRecordIterator.hasNext && usableFileIterator.hasNext) {
261266
val nextFile = usableFileIterator.next()
267+
val schemaToUse = columns match {
268+
case Some(cols) => tableSchema.select(cols.asJava)
269+
case None => tableSchema
270+
}
262271
currentRecordIterator = IcebergUtil.readDataFileAsIterator(
263272
nextFile.file(),
264-
tableSchema,
273+
schemaToUse,
265274
table.get
266275
)
267276

@@ -281,7 +290,11 @@ private[storage] class IcebergDocument[T >: Null <: AnyRef](
281290

282291
val record = currentRecordIterator.next()
283292
numOfReturnedRecords += 1
284-
deserde(tableSchema, record)
293+
val schemaToUse = columns match {
294+
case Some(cols) => tableSchema.select(cols.asJava)
295+
case None => tableSchema
296+
}
297+
deserde(schemaToUse, record)
285298
}
286299
}
287300
}

‎config-service/src/main/scala/org/apache/texera/service/resource/ConfigResource.scala‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,7 @@ class ConfigResource {
5656
),
5757
"activeTimeInMinutes" -> GuiConfig.guiWorkflowWorkspaceActiveTimeInMinutes,
5858
"copilotEnabled" -> GuiConfig.guiWorkflowWorkspaceCopilotEnabled,
59+
"limitColumns" -> GuiConfig.guiWorkflowWorkspaceLimitColumns,
5960
// flags from the auth.conf if needed
6061
"expirationTimeInMinutes" -> AuthConfig.jwtExpirationMinutes
6162
)

0 commit comments

Comments
 (0)