Skip to content

Rename TextChunker from SemanticTextPartitioner - #683

Merged
Adrian Bonar (adrianwyatt) merged 2 commits into
microsoft:mainfrom
lemillermicrosoft:426_text_chunker_dotnet
Apr 26, 2023
Merged

Adrian Bonar (adrianwyatt) merged 2 commits into
microsoft:mainfrom
lemillermicrosoft:426_text_chunker_dotnet

Conversation

@lemillermicrosoft

@lemillermicrosoft Lee Miller (lemillermicrosoft) commented Apr 26, 2023 •

Copy link
Copy Markdown
Member

Motivation and Context

To match changes in #450

Description

This pull request renames the SemanticTextPartitioner class to TextChunker, and updates all references to this class in the code and documentation. The TextChunker class is responsible for splitting text into smaller chunks based on tokens and paragraphs, which is useful for processing large texts or code files. The new name TextChunker reflects the functionality of the class more clearly and avoids confusion with the SemanticPartitioner class, which is a different concept that splits text into semantic segments based on embeddings.

Details:

  • Rename SemanticTextPartitioner.cs to TextChunker.cs and update the class name and namespace accordingly. The TextChunker class is now in the Text namespace, which is more consistent with its purpose and the other classes in the namespace.
  • Update all references to SemanticTextPartitioner in the code and documentation to use TextChunker instead. This includes the ConversationSummarySkill and the DocumentImportController classes, which use the TextChunker class to split text into sentences and paragraphs for summarization and import.
  • Update the README.md file for the GitHub Repo Q&A Bot sample to explain how the TextChunker class works with memory and embeddings.
  • Make some minor formatting changes to the TextChunker class, such as adding line breaks and spaces for readability, and removing an unused using directive.

Contribution Checklist

This commit refactors the SemanticTextPartitioner class, which splits text into chunks based on punctuation and new lines, and renames it to TextChunker. The new name better reflects the functionality of the class and avoids confusion with the SemanticText and SemanticPartitioner classes, which are different abstractions for text analysis. The TextChunker class is also moved from the SemanticFunctions.Partitioning namespace to the Text namespace, as it is a utility class rather than a semantic function. The commit updates the references to the class in the ConversationSummarySkill and the DocumentImportController classes, as well as the documentation.
@lemillermicrosoft Lee Miller (lemillermicrosoft) added the PR: ready to merge PR has been approved by all reviewers, and is ready to merge. label Apr 26, 2023
@adrianwyatt
Adrian Bonar (adrianwyatt) enabled auto-merge (squash) April 26, 2023 23:37
@adrianwyatt Adrian Bonar (adrianwyatt) removed the PR: ready for review All feedback addressed, ready for reviews label Apr 26, 2023
@adrianwyatt
Adrian Bonar (adrianwyatt) merged commit 5b1ed2f into microsoft:main Apr 26, 2023
Devis Lucato (dluc) pushed a commit that referenced this pull request Apr 29, 2023
### Motivation and Context
To match changes in #450 

### Description
This pull request renames the SemanticTextPartitioner class to
TextChunker, and updates all references to this class in the code and
documentation. The TextChunker class is responsible for splitting text
into smaller chunks based on tokens and paragraphs, which is useful for
processing large texts or code files. The new name TextChunker reflects
the functionality of the class more clearly and avoids confusion with
the SemanticPartitioner class, which is a different concept that splits
text into semantic segments based on embeddings.

Details:
- Rename SemanticTextPartitioner.cs to TextChunker.cs and update the
class name and namespace accordingly. The TextChunker class is now in
the Text namespace, which is more consistent with its purpose and the
other classes in the namespace.
- Update all references to SemanticTextPartitioner in the code and
documentation to use TextChunker instead. This includes the
ConversationSummarySkill and the DocumentImportController classes, which
use the TextChunker class to split text into sentences and paragraphs
for summarization and import.
- Update the README.md file for the GitHub Repo Q&A Bot sample to
explain how the TextChunker class works with memory and embeddings.
- Make some minor formatting changes to the TextChunker class, such as
adding line breaks and spaces for readability, and removing an unused
using directive.
Desmond Howard (dehoward) pushed a commit to lemillermicrosoft/semantic-kernel that referenced this pull request Jun 1, 2023
### Motivation and Context
To match changes in microsoft#450 

### Description
This pull request renames the SemanticTextPartitioner class to
TextChunker, and updates all references to this class in the code and
documentation. The TextChunker class is responsible for splitting text
into smaller chunks based on tokens and paragraphs, which is useful for
processing large texts or code files. The new name TextChunker reflects
the functionality of the class more clearly and avoids confusion with
the SemanticPartitioner class, which is a different concept that splits
text into semantic segments based on embeddings.

Details:
- Rename SemanticTextPartitioner.cs to TextChunker.cs and update the
class name and namespace accordingly. The TextChunker class is now in
the Text namespace, which is more consistent with its purpose and the
other classes in the namespace.
- Update all references to SemanticTextPartitioner in the code and
documentation to use TextChunker instead. This includes the
ConversationSummarySkill and the DocumentImportController classes, which
use the TextChunker class to split text into sentences and paragraphs
for summarization and import.
- Update the README.md file for the GitHub Repo Q&A Bot sample to
explain how the TextChunker class works with memory and embeddings.
- Make some minor formatting changes to the TextChunker class, such as
adding line breaks and spaces for readability, and removing an unused
using directive.
golden-aries pushed a commit to golden-aries/semantic-kernel that referenced this pull request Oct 10, 2023
### Motivation and Context
To match changes in microsoft#450 

### Description
This pull request renames the SemanticTextPartitioner class to
TextChunker, and updates all references to this class in the code and
documentation. The TextChunker class is responsible for splitting text
into smaller chunks based on tokens and paragraphs, which is useful for
processing large texts or code files. The new name TextChunker reflects
the functionality of the class more clearly and avoids confusion with
the SemanticPartitioner class, which is a different concept that splits
text into semantic segments based on embeddings.

Details:
- Rename SemanticTextPartitioner.cs to TextChunker.cs and update the
class name and namespace accordingly. The TextChunker class is now in
the Text namespace, which is more consistent with its purpose and the
other classes in the namespace.
- Update all references to SemanticTextPartitioner in the code and
documentation to use TextChunker instead. This includes the
ConversationSummarySkill and the DocumentImportController classes, which
use the TextChunker class to split text into sentences and paragraphs
for summarization and import.
- Update the README.md file for the GitHub Repo Q&A Bot sample to
explain how the TextChunker class works with memory and embeddings.
- Make some minor formatting changes to the TextChunker class, such as
adding line breaks and spaces for readability, and removing an unused
using directive.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

.NET Issue or Pull requests regarding .NET code PR: ready to merge PR has been approved by all reviewers, and is ready to merge.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants