Skip to content

Commit 89cee3e

Browse files
mnriemCopilot
andcommitted
fix: complete Unicode feature naming across script variants
Replace the initial proposed sanitizer with Unicode-aware name generation across Bash, PowerShell, and Python. Keep UTF-8 branch names within GitHub byte limits, preserve existing punctuation-only warnings, and update parity tests and documentation. Refs #4574. Assisted-by: GitHub Copilot (model: GPT-6 Sol, autonomous) Co-authored-by: Copilot App <[email protected]>
1 parent eb4c9dc commit 89cee3e

5 files changed

Lines changed: 201 additions & 79 deletions

File tree

‎docs/reference/core.md‎

Lines changed: 9 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -65,20 +65,22 @@ specify init my-project --integration copilot --preset compliance
6565
## Naming Features with the Helper Scripts
6666

6767
When calling the bundled `create-new-feature` helper scripts directly, generated
68-
names retain only ASCII letters and digits. A description entirely in a non-Latin
69-
script, or made only of punctuation, can therefore produce an empty suffix such
70-
as `001-`. The scripts warn on stderr when this happens, including during a dry
71-
run; JSON output remains parseable.
68+
names retain Unicode letters and digits in UTF-8, so a description such as
69+
`添加用户` produces `001-添加用户`. Descriptions made only of punctuation can still
70+
produce an empty suffix such as `001-`; the scripts warn on stderr when this
71+
happens, including during a dry run. JSON output remains parseable.
7272

73-
Keep the original description and supply a readable ASCII short name:
73+
To choose a different name, keep the original description and supply a short name:
7474

7575
```bash
76-
bash .specify/scripts/bash/create-new-feature.sh --json --short-name user-auth "添加用户"
76+
bash .specify/scripts/bash/create-new-feature.sh --json --short-name 用户管理 "添加用户"
7777
```
7878

7979
The Python helper also accepts `--short-name`; the PowerShell helper uses
8080
`-ShortName`. A supplied short name is cleaned by the same rules, so it must
81-
contain at least one ASCII letter or digit.
81+
contain at least one letter or digit. The Bash helper needs an installed UTF-8
82+
locale to recognize Unicode letters and digits. ASCII capitals are lowercased;
83+
non-ASCII letter casing is preserved across the script variants.
8284

8385
## Check Installed Tools
8486

‎scripts/bash/create-new-feature.sh‎

Lines changed: 41 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -146,16 +146,35 @@ spec_prefix_exists() {
146146

147147
# Function to clean and format a branch name
148148
#
149-
# Three details keep this byte-identical to the Python and PowerShell twins:
150-
# * LC_ALL=C -- the character classes below only replace separators, leaving
151-
# UTF-8 letters and digits intact without depending on locale collation.
149+
# Three details keep this consistent with the Python and PowerShell twins:
150+
# * A UTF-8 locale makes [:alnum:] recognize Unicode letters and digits.
152151
# * `--*` instead of the GNU-only `\+`, which POSIX/BSD sed reads as a literal
153152
# '+', leaving repeated separators uncollapsed on macOS.
154153
# * printf instead of echo, so a name of "-n"/"-e"/"-E" is text, not options.
154+
UNICODE_LOCALE=""
155+
for candidate in C.UTF-8 C.utf8 en_US.UTF-8 en_US.utf8 "${LC_ALL:-${LC_CTYPE:-${LANG:-}}}"; do
156+
if [ -n "$candidate" ] && [ "$(printf 'é。' | LC_ALL="$candidate" sed 's/[^[:alnum:]]/-/g' 2>/dev/null)" = 'é-' ]; then
157+
UNICODE_LOCALE="$candidate"
158+
break
159+
fi
160+
done
161+
162+
if [ -z "$UNICODE_LOCALE" ]; then
163+
UNICODE_LOCALE=C
164+
if printf '%s' "${SHORT_NAME:-$FEATURE_DESCRIPTION}" | LC_ALL=C grep -q '[^ -~]'; then
165+
echo "Error: A UTF-8 locale is required to create a Unicode feature name" >&2
166+
exit 1
167+
fi
168+
fi
169+
155170
clean_branch_name() {
156171
local name="$1"
157-
local -x LC_ALL=C
158-
printf '%s\n' "$name" | tr '[:upper:]' '[:lower:]' | sed 's/[[:space:][:punct:]]/-/g' | sed 's/--*/-/g' | sed 's/^-//' | sed 's/-$//'
172+
local -x LC_ALL="$UNICODE_LOCALE"
173+
printf '%s\n' "$name" | LC_ALL=C tr '[:upper:]' '[:lower:]' | sed 's/[^[:alnum:]]/-/g' | sed 's/--*/-/g' | sed 's/^-//' | sed 's/-$//'
174+
}
175+
176+
branch_byte_count() {
177+
printf '%s' "$1" | wc -c | tr -d '[:space:]'
159178
}
160179

161180
# Fit a feature prefix and suffix within GitHub's branch-name limit.
@@ -164,11 +183,16 @@ fit_branch_name() {
164183
local branch_suffix="$2"
165184
local branch_name="${feature_num}-${branch_suffix}"
166185

167-
if [ ${#branch_name} -gt $MAX_BRANCH_LENGTH ]; then
186+
if [ "$(branch_byte_count "$branch_name")" -gt "$MAX_BRANCH_LENGTH" ]; then
168187
local prefix_length=$(( ${#feature_num} + 1 ))
169188
local max_suffix_length=$((MAX_BRANCH_LENGTH - prefix_length))
170189
local truncated_suffix
171-
truncated_suffix=$(printf '%s' "$branch_suffix" | cut -c "1-$max_suffix_length" | sed 's/-$//')
190+
local -x LC_ALL="$UNICODE_LOCALE"
191+
truncated_suffix="${branch_suffix:0:$max_suffix_length}"
192+
while [ "$(branch_byte_count "$truncated_suffix")" -gt "$max_suffix_length" ]; do
193+
truncated_suffix="${truncated_suffix%?}"
194+
done
195+
truncated_suffix="${truncated_suffix%-}"
172196
branch_name="${feature_num}-${truncated_suffix}"
173197
fi
174198

@@ -208,27 +232,25 @@ generate_branch_name() {
208232
# Common stop words to filter out
209233
local stop_words="^(i|a|an|the|to|for|of|in|on|at|by|with|from|is|are|was|were|be|been|being|have|has|had|do|does|did|will|would|should|could|can|may|might|must|shall|this|that|these|those|my|your|our|their|want|need|add|get|set)$"
210234

211-
# Convert to lowercase and split into words. LC_ALL=C keeps ASCII
212-
# punctuation handling stable while non-ASCII words remain intact, and so the `grep -qw`
213-
# acronym probe below uses ASCII word boundaries like the Python twin's
214-
# (?<![0-9A-Za-z_]) lookarounds.
215-
local -x LC_ALL=C
216-
local clean_name=$(printf '%s' "$description" | tr '[:upper:]' '[:lower:]' | sed 's/[[:space:][:punct:]]/ /g')
235+
# Use a UTF-8 locale for Unicode character classes. Lowercase only ASCII
236+
# so results do not depend on the platform's multibyte case conversion.
237+
local -x LC_ALL="$UNICODE_LOCALE"
238+
local clean_name=$(printf '%s\n' "$description" | LC_ALL=C tr '[:upper:]' '[:lower:]' | sed 's/[^[:alnum:]]/ /g')
217239

218240
# Filter words: remove stop words and words shorter than 3 chars (unless they're uppercase acronyms in original)
219241
local meaningful_words=()
220242
for word in $clean_name; do
221243
# Skip empty words
222244
[ -z "$word" ] && continue
223245

224-
# Keep words that are NOT stop words AND (length >= 3 OR are potential acronyms)
246+
# Retain non-ASCII words even when shorter than three characters.
225247
if ! echo "$word" | grep -qiE "$stop_words"; then
226-
if [ ${#word} -ge 3 ]; then
248+
if [ ${#word} -ge 3 ] || printf '%s' "$word" | LC_ALL=C grep -q '[^ -~]'; then
227249
meaningful_words+=("$word")
228250
# Keep short words that appear as an uppercase acronym in the original.
229251
# Uppercase via tr and match with grep -w (both portable) rather than
230252
# bash's 4+ "^^" case expansion (breaks on macOS bash 3.2) and \b (non-POSIX).
231-
elif printf '%s' "$description" | grep -qw -- "$(printf '%s' "$word" | tr '[:lower:]' '[:upper:]')"; then
253+
elif printf '%s' "$description" | LC_ALL=C grep -qw -- "$(printf '%s' "$word" | LC_ALL=C tr '[:lower:]' '[:upper:]')"; then
232254
meaningful_words+=("$word")
233255
fi
234256
fi
@@ -265,7 +287,7 @@ else
265287
fi
266288

267289
if [ -z "$BRANCH_SUFFIX" ]; then
268-
echo "[specify] Warning: Feature name is empty after removing unsupported characters. Use --short-name with ASCII letters or digits (for example, user-auth)." >&2
290+
echo "[specify] Warning: Feature name is empty after removing unsupported characters. Use --short-name with letters or digits (for example, user-auth)." >&2
269291
fi
270292

271293
# Warn if --number and --timestamp are both specified
@@ -338,8 +360,8 @@ ORIGINAL_BRANCH_NAME="${FEATURE_NUM}-${BRANCH_SUFFIX}"
338360
BRANCH_NAME=$(fit_branch_name "$FEATURE_NUM" "$BRANCH_SUFFIX")
339361
if [ "$BRANCH_NAME" != "$ORIGINAL_BRANCH_NAME" ]; then
340362
>&2 echo "[specify] Warning: Branch name exceeded GitHub's 244-byte limit"
341-
>&2 echo "[specify] Original: $ORIGINAL_BRANCH_NAME (${#ORIGINAL_BRANCH_NAME} bytes)"
342-
>&2 echo "[specify] Truncated to: $BRANCH_NAME (${#BRANCH_NAME} bytes)"
363+
>&2 echo "[specify] Original: $ORIGINAL_BRANCH_NAME ($(branch_byte_count "$ORIGINAL_BRANCH_NAME") bytes)"
364+
>&2 echo "[specify] Truncated to: $BRANCH_NAME ($(branch_byte_count "$BRANCH_NAME") bytes)"
343365
fi
344366

345367
FEATURE_DIR="$SPECS_DIR/$BRANCH_NAME"

‎scripts/powershell/create-new-feature.ps1‎

Lines changed: 38 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -83,10 +83,32 @@ function Test-SpecPrefixInUse {
8383
Select-Object -First 1)
8484
}
8585

86+
function ConvertTo-AsciiLower {
87+
param([string]$Name)
88+
89+
return [regex]::Replace($Name, '[A-Z]', { param($match) $match.Value.ToLowerInvariant() })
90+
}
91+
92+
function ConvertTo-UnicodeWords {
93+
param([string]$Name, [string]$Separator)
94+
95+
$lowerName = ConvertTo-AsciiLower -Name $Name
96+
return [regex]::Replace($lowerName, '[\uD800-\uDBFF][\uDC00-\uDFFF]|[^\p{L}\p{N}]', {
97+
param($match)
98+
if ($match.Length -eq 2) {
99+
$category = [System.Globalization.CharUnicodeInfo]::GetUnicodeCategory($match.Value, 0)
100+
if ($category.ToString() -match '(Letter|Number)$') {
101+
return $match.Value
102+
}
103+
}
104+
return $Separator
105+
})
106+
}
107+
86108
function ConvertTo-CleanBranchName {
87109
param([string]$Name)
88110

89-
return $Name.ToLower() -replace '[^\p{L}\p{N}]', '-' -replace '-{2,}', '-' -replace '^-', '' -replace '-$', ''
111+
return (ConvertTo-UnicodeWords -Name $Name -Separator '-') -replace '-{2,}', '-' -replace '^-', '' -replace '-$', ''
90112
}
91113

92114
function Get-FittedBranchName {
@@ -96,10 +118,15 @@ function Get-FittedBranchName {
96118
)
97119

98120
$fittedName = "$FeatureNum-$BranchSuffix"
99-
if ($fittedName.Length -gt $maxBranchLength) {
121+
if ([System.Text.Encoding]::UTF8.GetByteCount($fittedName) -gt $maxBranchLength) {
100122
$prefixLength = $FeatureNum.Length + 1
101123
$maxSuffixLength = $maxBranchLength - $prefixLength
102-
$truncatedSuffix = $BranchSuffix.Substring(0, [Math]::Min($BranchSuffix.Length, $maxSuffixLength))
124+
$bytes = [System.Text.Encoding]::UTF8.GetBytes($BranchSuffix)
125+
$bytesToUse = $maxSuffixLength
126+
while ($bytesToUse -gt 0 -and ($bytes[$bytesToUse] -band 0xC0) -eq 0x80) {
127+
$bytesToUse--
128+
}
129+
$truncatedSuffix = [System.Text.Encoding]::UTF8.GetString($bytes, 0, $bytesToUse)
103130
$truncatedSuffix = $truncatedSuffix -replace '-$', ''
104131
$fittedName = "$FeatureNum-$truncatedSuffix"
105132
}
@@ -132,8 +159,8 @@ function Get-BranchName {
132159
'want', 'need', 'add', 'get', 'set'
133160
)
134161

135-
# Convert to lowercase and extract Unicode words.
136-
$cleanName = $Description.ToLower() -replace '[^\p{L}\p{N}\s]', ' '
162+
# Lowercase ASCII and extract Unicode words, matching the shell variant.
163+
$cleanName = ConvertTo-UnicodeWords -Name $Description -Separator ' '
137164
$words = $cleanName -split '\s+' | Where-Object { $_ }
138165

139166
# Filter words: remove stop words and words shorter than 3 chars (unless they're uppercase acronyms in original)
@@ -142,8 +169,9 @@ function Get-BranchName {
142169
# Skip stop words
143170
if ($stopWords -contains $word) { continue }
144171

145-
# Keep words that are length >= 3 OR appear as uppercase in original (likely acronyms)
146-
if ($word.Length -ge 3) {
172+
# Keep Unicode words even when short; ASCII words still need three
173+
# characters or an uppercase acronym in the original.
174+
if ($word.Length -ge 3 -or $word -match '[^\x00-\x7F]') {
147175
$meaningfulWords += $word
148176
} elseif ($Description -cmatch "(?<![0-9A-Za-z_])$($word.ToUpper())(?![0-9A-Za-z_])") {
149177
# Keep short words only if they appear as uppercase in original (likely
@@ -180,7 +208,7 @@ if ($ShortName) {
180208
}
181209

182210
if (-not $branchSuffix) {
183-
[Console]::Error.WriteLine("[specify] Warning: Feature name is empty after removing unsupported characters. Use -ShortName with ASCII letters or digits (for example, user-auth).")
211+
[Console]::Error.WriteLine("[specify] Warning: Feature name is empty after removing unsupported characters. Use -ShortName with letters or digits (for example, user-auth).")
184212
}
185213

186214
# Treat an explicit empty string as omitted, matching the bash and Python twins.
@@ -252,8 +280,8 @@ $originalBranchName = "$featureNum-$branchSuffix"
252280
$branchName = Get-FittedBranchName -FeatureNum $featureNum -BranchSuffix $branchSuffix
253281
if ($branchName -ne $originalBranchName) {
254282
[Console]::Error.WriteLine("[specify] Warning: Branch name exceeded GitHub's 244-byte limit")
255-
[Console]::Error.WriteLine("[specify] Original: $originalBranchName ($($originalBranchName.Length) bytes)")
256-
[Console]::Error.WriteLine("[specify] Truncated to: $branchName ($($branchName.Length) bytes)")
283+
[Console]::Error.WriteLine("[specify] Original: $originalBranchName ($([System.Text.Encoding]::UTF8.GetByteCount($originalBranchName)) bytes)")
284+
[Console]::Error.WriteLine("[specify] Truncated to: $branchName ($([System.Text.Encoding]::UTF8.GetByteCount($branchName)) bytes)")
257285
}
258286

259287
$featureDir = Join-Path $specsDir $branchName

‎scripts/python/create_new_feature.py‎

Lines changed: 14 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,9 @@ def _json_line(payload: object) -> str:
4343

4444
_MAX_BRANCH_LENGTH = 244
4545
_MAX_FEATURE_NUMBER = 2**63 - 1
46+
_ASCII_LOWER = str.maketrans(
47+
"ABCDEFGHIJKLMNOPQRSTUVWXYZ", "abcdefghijklmnopqrstuvwxyz"
48+
)
4649

4750

4851
def _int64_from_digits(value: str) -> int | None:
@@ -167,18 +170,18 @@ def _parse_args(argv: list[str], argv0: str) -> Args:
167170

168171

169172
def _clean_branch_name(name: str) -> str:
170-
cleaned = re.sub(r"[^\w]|_", "-", name.lower(), flags=re.UNICODE)
173+
cleaned = re.sub(r"[^\w]|_", "-", name.translate(_ASCII_LOWER))
171174
cleaned = re.sub(r"-+", "-", cleaned)
172175
return cleaned.strip("-")
173176

174177

175178
def _generate_branch_name(description: str) -> str:
176-
clean = re.sub(r"[^\w]|_", " ", description.lower(), flags=re.UNICODE)
179+
clean = re.sub(r"[^\w]|_", " ", description.translate(_ASCII_LOWER))
177180
meaningful: list[str] = []
178181
for word in clean.split():
179182
if word in _STOP_WORDS:
180183
continue
181-
if len(word) >= 3:
184+
if len(word) >= 3 or not word.isascii():
182185
meaningful.append(word)
183186
# Keep short words that appear as an uppercase acronym in the original,
184187
# mirroring the bash twin's case-sensitive `grep -qw` check.
@@ -217,11 +220,13 @@ def _get_highest_from_specs(specs_dir: Path) -> int:
217220
def _fit_branch_name(feature_num: str, branch_suffix: str) -> str:
218221
"""Fit a feature prefix and suffix within GitHub's branch-name limit."""
219222
branch_name = f"{feature_num}-{branch_suffix}"
220-
if len(branch_name) <= _MAX_BRANCH_LENGTH:
223+
if len(branch_name.encode("utf-8")) <= _MAX_BRANCH_LENGTH:
221224
return branch_name
222225

223226
max_suffix_length = _MAX_BRANCH_LENGTH - (len(feature_num) + 1)
224-
truncated_suffix = re.sub(r"-$", "", branch_suffix[:max_suffix_length])
227+
truncated_suffix = branch_suffix.encode("utf-8")[:max_suffix_length].decode(
228+
"utf-8", errors="ignore"
229+
).rstrip("-")
225230
return f"{feature_num}-{truncated_suffix}"
226231

227232

@@ -268,7 +273,7 @@ def main(argv: list[str] | None = None) -> int:
268273
if not branch_suffix:
269274
print(
270275
"[specify] Warning: Feature name is empty after removing unsupported characters. "
271-
"Use --short-name with ASCII letters or digits (for example, user-auth).",
276+
"Use --short-name with letters or digits (for example, user-auth).",
272277
file=sys.stderr,
273278
)
274279

@@ -363,11 +368,12 @@ def main(argv: list[str] | None = None) -> int:
363368
)
364369
print(
365370
f"[specify] Original: {original_branch_name} "
366-
f"({len(original_branch_name)} bytes)",
371+
f"({len(original_branch_name.encode('utf-8'))} bytes)",
367372
file=sys.stderr,
368373
)
369374
print(
370-
f"[specify] Truncated to: {branch_name} ({len(branch_name)} bytes)",
375+
f"[specify] Truncated to: {branch_name} "
376+
f"({len(branch_name.encode('utf-8'))} bytes)",
371377
file=sys.stderr,
372378
)
373379

0 commit comments

Comments
 (0)