Skip to content
This repository was archived by the owner on Apr 22, 2023. It is now read-only.

Commit ac2857b

Browse files
srl295trevnorris
srl295
authored and
trevnorris
committed
build, i18n: improve Intl build, add "--with-intl"
The two main goals of this change are: - To make it easier to build the Intl option using ICU (particularly, using a newer ICU than v8/Chromium's version) - To enable a much smaller ICU build with only English support The goal here is to get node.js binaries built this way by default so that the Intl API can be used. Additional data can be added at execution time (see Readme and wiki) More details are at #7719 In particular, this change adds the "--with-intl=" configure option to provide more ways of building "Intl": - "full-icu" picks up an ICU from deps/icu - "small-icu" is similar, but builds only English - "system-icu" uses pkg-config to find an installed ICU - "none" does nothing (no Intl) For Windows builds, the "full-icu" or "small-icu" options are added to vcbuild.bat. Note that the existing "--with-icu-path" option is not removed from configure, but may not be used alongside the new option. Wiki changes have already been made on https://github.com/joyent/node/wiki/Installation and a new page created at https://github.com/joyent/node/wiki/Intl (marked as provisional until this change lands.) Summary of changes: * README.md : doc updates * .gitignore : added "deps/icu" as this is the location where ICU is unpacked to. * Makefile : added the tools/icu/* files to cpplint, but excluded a problematic file. * configure : added the "--with-intl" option mentioned above. Calculate at config time the list of ICU source files to use and data packaging options. * node.gyp : add the new files src/node_i18n.cc/.h as well as ICU linkage. * src/node.cc : add call into node::i18n::InitializeICUDirectory(icu_data_dir) as well as new --icu-data-dir option and NODE_ICU_DATA env variable to configure ICU data loading. This loading is only relevant in the "small" configuration. * src/node_i18n.cc : new source file for the above Initialize.. function, to setup ICU as needed. * tools/icu : new directory with some tools needed for this build. * tools/icu/icu-generic.gyp : new .gyp file that builds ICU in some new ways, both on unix/mac and windows. * tools/icu/icu-system.gyp : new .gyp file to build node against a pkg-config detected ICU. * tools/icu/icu_small.json : new config file for the "English-only" small build. * tools/icu/icutrim.py : new tool for trimming down ICU data. Reads the above .json file. * tools/icu/iculslocs.cc : new tool for repairing ICU data manifests after trim operation. * tools/icu/no-op.cc : dummy file to force .gyp into using a C++ linker. * vcbuild.bat : added small-icu and full-icu options, to call into configure. * Fixed toolset dependencies, see #7719 (comment) Note that because of a bug in gyp {CC,CXX}_host must also be set. Otherwise gcc/g++ will be used by default for part of the build. Reviewed-by: Trevor Norris <[email protected]> Reviewed-by: Fedor Indutny <[email protected]>
1 parent 95726b0 commit ac2857b

16 files changed

Lines changed: 1605 additions & 20 deletions

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,7 @@ ipch/
4545
/test/addons/doc-*/
4646
email.md
4747
deps/v8-*
48+
deps/icu
4849
./node_modules
4950
.svn/
5051

‎Makefile‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -406,7 +406,7 @@ CPPLINT_EXCLUDE += src/queue.h
406406
CPPLINT_EXCLUDE += src/tree.h
407407
CPPLINT_EXCLUDE += src/v8abbr.h
408408

409-
CPPLINT_FILES = $(filter-out $(CPPLINT_EXCLUDE), $(wildcard src/*.cc src/*.h src/*.c))
409+
CPPLINT_FILES = $(filter-out $(CPPLINT_EXCLUDE), $(wildcard src/*.cc src/*.h src/*.c tools/icu/*.h tools/icu/*.cc))
410410

411411
cpplint:
412412
@$(PYTHON) tools/cpplint.py $(CPPLINT_FILES)

‎README.md‎

Lines changed: 32 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -19,17 +19,6 @@ make
1919
make install
2020
```
2121

22-
With libicu i18n support:
23-
24-
```sh
25-
svn checkout --force --revision 214189 \
26-
http://src.chromium.org/svn/trunk/deps/third_party/icu46 \
27-
deps/v8/third_party/icu46
28-
./configure --with-icu-path=deps/v8/third_party/icu46/icu.gyp
29-
make
30-
make install
31-
```
32-
3322
If your python binary is in a non-standard location or has a
3423
non-standard name, run the following instead:
3524

@@ -47,7 +36,9 @@ Prerequisites (Windows only):
4736

4837
Windows:
4938

50-
vcbuild nosign
39+
```sh
40+
vcbuild nosign
41+
```
5142

5243
You can download pre-built binaries for various operating systems from
5344
[http://nodejs.org/download/](http://nodejs.org/download/). The Windows
@@ -92,6 +83,35 @@ make doc
9283
man doc/node.1
9384
```
9485

86+
### To build `Intl` (ECMA-402) support:
87+
88+
*Note:* more docs, including how to reduce disk footprint, are on
89+
[the wiki](https://github.com/joyent/node/wiki/Intl).
90+
91+
#### Use existing installed ICU (Unix/Macintosh only):
92+
93+
```sh
94+
pkg-config --modversion icu-i18n && ./configure --with-intl=system-icu
95+
```
96+
97+
#### Build ICU from source:
98+
99+
First: Unpack latest ICU
100+
[icu4c-**##.#**-src.tgz](http://icu-project.org/download) (or `.zip`)
101+
as `deps/icu` (You'll have: `deps/icu/source/...`)
102+
103+
Unix/Macintosh:
104+
105+
```sh
106+
./configure --with-intl=full-icu
107+
```
108+
109+
Windows:
110+
111+
```sh
112+
vcbuild full-icu
113+
```
114+
95115
Resources for Newcomers
96116
---
97117
- [The Wiki](https://github.com/joyent/node/wiki)

‎configure‎

Lines changed: 136 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -241,6 +241,11 @@ parser.add_option('--with-icu-path',
241241
dest='with_icu_path',
242242
help='Path to icu.gyp (ICU i18n, Chromium version only.)')
243243

244+
parser.add_option('--with-intl',
245+
action='store',
246+
dest='with_intl',
247+
help='Intl mode: none, full-icu, small-icu (default is none)')
248+
244249
parser.add_option('--with-perfctr',
245250
action='store_true',
246251
dest='with_perfctr',
@@ -686,13 +691,138 @@ def configure_winsdk(o):
686691
print('ctrpp not found in WinSDK path--using pre-gen files '
687692
'from tools/msvs/genfiles.')
688693

689-
690-
def configure_icu(o):
694+
def glob_to_var(dir_base, dir_sub):
695+
list = []
696+
dir_all = os.path.join(dir_base, dir_sub)
697+
files = os.walk(dir_all)
698+
for ent in files:
699+
(path, dirs, files) = ent
700+
for file in files:
701+
if file.endswith('.cpp') or file.endswith('.c') or file.endswith('.h'):
702+
list.append('%s/%s' % (dir_sub, file))
703+
break
704+
return list
705+
706+
707+
def configure_intl(o):
708+
# small ICU is off by default.
709+
# always set icu_small, node.gyp depends on it being defined.
710+
o['variables']['icu_small'] = b(False)
711+
712+
with_intl = options.with_intl
691713
have_icu_path = bool(options.with_icu_path)
692-
o['variables']['v8_enable_i18n_support'] = int(have_icu_path)
693-
if have_icu_path:
714+
if have_icu_path and with_intl:
715+
print 'Error: Cannot specify both --with-icu-path and --with-intl'
716+
sys.exit(1)
717+
elif have_icu_path:
718+
# Chromium .gyp mode: --with-icu-path
719+
o['variables']['v8_enable_i18n_support'] = 1
720+
# use the .gyp given
694721
o['variables']['icu_gyp_path'] = options.with_icu_path
695-
722+
return
723+
# --with-intl=<with_intl>
724+
if with_intl == 'none' or with_intl is None:
725+
o['variables']['v8_enable_i18n_support'] = 0
726+
return # no Intl
727+
elif with_intl == 'small-icu':
728+
# small ICU (English only)
729+
o['variables']['v8_enable_i18n_support'] = 1
730+
o['variables']['icu_small'] = b(True)
731+
elif with_intl == 'full-icu':
732+
# full ICU
733+
o['variables']['v8_enable_i18n_support'] = 1
734+
elif with_intl == 'system-icu':
735+
# ICU from pkg-config.
736+
o['variables']['v8_enable_i18n_support'] = 1
737+
pkgicu = pkg_config('icu-i18n')
738+
if not pkgicu:
739+
print 'Error: could not load pkg-config data for "icu-i18n".'
740+
print 'See above errors or the README.md.'
741+
sys.exit(1)
742+
(libs, cflags) = pkgicu
743+
o['libraries'] += libs.split()
744+
o['cflags'] += cflags.split()
745+
# use the "system" .gyp
746+
o['variables']['icu_gyp_path'] = 'tools/icu/icu-system.gyp'
747+
return
748+
else:
749+
print 'Error: unknown value --with-intl=%s' % with_intl
750+
sys.exit(1)
751+
# Note: non-ICU implementations could use other 'with_intl'
752+
# values.
753+
754+
# ICU mode. (icu-generic.gyp)
755+
byteorder = sys.byteorder
756+
o['variables']['icu_gyp_path'] = 'tools/icu/icu-generic.gyp'
757+
# ICU source dir relative to root
758+
icu_full_path = os.path.join(root_dir, 'deps/icu')
759+
o['variables']['icu_path'] = icu_full_path
760+
if not os.path.isdir(icu_full_path):
761+
print 'Error: ICU path is not a directory: %s' % (icu_full_path)
762+
sys.exit(1)
763+
# Now, what version of ICU is it? We just need the "major", such as 54.
764+
# uvernum.h contains it as a #define.
765+
uvernum_h = os.path.join(icu_full_path, 'source/common/unicode/uvernum.h')
766+
if not os.path.isfile(uvernum_h):
767+
print 'Error: could not load %s - is ICU installed?' % uvernum_h
768+
sys.exit(1)
769+
icu_ver_major = None
770+
matchVerExp = r'^\s*#define\s+U_ICU_VERSION_SHORT\s+"([^"]*)".*'
771+
match_version = re.compile(matchVerExp)
772+
for line in open(uvernum_h).readlines():
773+
m = match_version.match(line)
774+
if m:
775+
icu_ver_major = m.group(1)
776+
if not icu_ver_major:
777+
print 'Could not read U_ICU_VERSION_SHORT version from %s' % uvernum_h
778+
sys.exit(1)
779+
icu_endianness = sys.byteorder[0]; # TODO(srl295): EBCDIC should be 'e'
780+
o['variables']['icu_ver_major'] = icu_ver_major
781+
o['variables']['icu_endianness'] = icu_endianness
782+
icu_data_file_l = 'icudt%s%s.dat' % (icu_ver_major, 'l')
783+
icu_data_file = 'icudt%s%s.dat' % (icu_ver_major, icu_endianness)
784+
# relative to configure
785+
icu_data_path = os.path.join(icu_full_path,
786+
'source/data/in',
787+
icu_data_file_l)
788+
# relative to dep..
789+
icu_data_in = os.path.join('../../deps/icu/source/data/in', icu_data_file_l)
790+
if not os.path.isfile(icu_data_path) and icu_endianness != 'l':
791+
# use host endianness
792+
icu_data_path = os.path.join(icu_full_path,
793+
'source/data/in',
794+
icu_data_file)
795+
# relative to dep..
796+
icu_data_in = os.path.join('icu/source/data/in',
797+
icu_data_file)
798+
# this is the input '.dat' file to use .. icudt*.dat
799+
# may be little-endian if from a icu-project.org tarball
800+
o['variables']['icu_data_in'] = icu_data_in
801+
# this is the icudt*.dat file which node will be using (platform endianness)
802+
o['variables']['icu_data_file'] = icu_data_file
803+
if not os.path.isfile(icu_data_path):
804+
print 'Error: ICU prebuilt data file %s does not exist.' % icu_data_path
805+
print 'See the README.md.'
806+
# .. and we're not about to build it from .gyp!
807+
sys.exit(1)
808+
# map from variable name to subdirs
809+
icu_src = {
810+
'stubdata': 'stubdata',
811+
'common': 'common',
812+
'i18n': 'i18n',
813+
'io': 'io',
814+
'tools': 'tools/toolutil',
815+
'genccode': 'tools/genccode',
816+
'genrb': 'tools/genrb',
817+
'icupkg': 'tools/icupkg',
818+
}
819+
# this creates a variable icu_src_XXX for each of the subdirs
820+
# with a list of the src files to use
821+
for i in icu_src:
822+
var = 'icu_src_%s' % i
823+
path = '../../deps/icu/source/%s' % icu_src[i]
824+
o['variables'][var] = glob_to_var('tools/icu', path)
825+
return # end of configure_intl
696826

697827
# determine the "flavor" (operating system) we're building for,
698828
# leveraging gyp's GetFlavor function
@@ -717,7 +847,7 @@ configure_libuv(output)
717847
configure_v8(output)
718848
configure_openssl(output)
719849
configure_winsdk(output)
720-
configure_icu(output)
850+
configure_intl(output)
721851
configure_fullystatic(output)
722852

723853
# variables should be a root level element,

‎node.gyp‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -103,6 +103,7 @@
103103
'src/node_stat_watcher.cc',
104104
'src/node_watchdog.cc',
105105
'src/node_zlib.cc',
106+
'src/node_i18n.cc',
106107
'src/pipe_wrap.cc',
107108
'src/signal_wrap.cc',
108109
'src/smalloc.cc',
@@ -134,6 +135,7 @@
134135
'src/node_version.h',
135136
'src/node_watchdog.h',
136137
'src/node_wrap.h',
138+
'src/node_i18n.h',
137139
'src/pipe_wrap.h',
138140
'src/queue.h',
139141
'src/smalloc.h',
@@ -164,6 +166,17 @@
164166
],
165167

166168
'conditions': [
169+
[ 'v8_enable_i18n_support==1', {
170+
'defines': [ 'NODE_HAVE_I18N_SUPPORT=1' ],
171+
'dependencies': [
172+
'<(icu_gyp_path):icui18n',
173+
'<(icu_gyp_path):icuuc',
174+
],
175+
'conditions': [
176+
[ 'icu_small=="true"', {
177+
'defines': [ 'NODE_HAVE_SMALL_ICU=1' ],
178+
}]],
179+
}],
167180
[ 'node_use_openssl=="true"', {
168181
'defines': [ 'HAVE_OPENSSL=1' ],
169182
'sources': [

‎src/node.cc‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,10 @@
3535
#include "node_crypto.h"
3636
#endif
3737

38+
#if defined(NODE_HAVE_I18N_SUPPORT)
39+
#include "node_i18n.h"
40+
#endif
41+
3842
#if defined HAVE_DTRACE || defined HAVE_ETW
3943
#include "node_dtrace.h"
4044
#endif
@@ -135,6 +139,11 @@ static node_module* modpending;
135139
static node_module* modlist_builtin;
136140
static node_module* modlist_addon;
137141

142+
#if defined(NODE_HAVE_I18N_SUPPORT)
143+
// Path to ICU data (for i18n / Intl)
144+
static const char* icu_data_dir = NULL;
145+
#endif
146+
138147
// used by C++ modules as well
139148
bool no_deprecation = false;
140149

@@ -2953,6 +2962,14 @@ static void PrintHelp() {
29532962
" --trace-deprecation show stack traces on deprecations\n"
29542963
" --v8-options print v8 command line options\n"
29552964
" --max-stack-size=val set max v8 stack size (bytes)\n"
2965+
#if defined(NODE_HAVE_I18N_SUPPORT)
2966+
" --icu-data-dir=dir set ICU data load path to dir\n"
2967+
" (overrides NODE_ICU_DATA)\n"
2968+
#if !defined(NODE_HAVE_SMALL_ICU)
2969+
" Note: linked-in ICU data is\n"
2970+
" present.\n"
2971+
#endif
2972+
#endif
29562973
"\n"
29572974
"Environment variables:\n"
29582975
#ifdef _WIN32
@@ -2964,6 +2981,12 @@ static void PrintHelp() {
29642981
"NODE_MODULE_CONTEXTS Set to 1 to load modules in their own\n"
29652982
" global contexts.\n"
29662983
"NODE_DISABLE_COLORS Set to 1 to disable colors in the REPL\n"
2984+
#if defined(NODE_HAVE_I18N_SUPPORT)
2985+
"NODE_ICU_DATA Data path for ICU (Intl object) data\n"
2986+
#if !defined(NODE_HAVE_SMALL_ICU)
2987+
" (will extend linked-in data)\n"
2988+
#endif
2989+
#endif
29672990
"\n"
29682991
"Documentation can be found at http://nodejs.org/\n");
29692992
}
@@ -3054,6 +3077,10 @@ static void ParseArgs(int* argc,
30543077
} else if (strcmp(arg, "--v8-options") == 0) {
30553078
new_v8_argv[new_v8_argc] = "--help";
30563079
new_v8_argc += 1;
3080+
#if defined(NODE_HAVE_I18N_SUPPORT)
3081+
} else if (strncmp(arg, "--icu-data-dir=", 15) == 0) {
3082+
icu_data_dir = arg + 15;
3083+
#endif
30573084
} else {
30583085
// V8 option. Pass through as-is.
30593086
new_v8_argv[new_v8_argc] = arg;
@@ -3410,6 +3437,18 @@ void Init(int* argc,
34103437
}
34113438
}
34123439

3440+
#if defined(NODE_HAVE_I18N_SUPPORT)
3441+
if (icu_data_dir == NULL) {
3442+
// if the parameter isn't given, use the env variable.
3443+
icu_data_dir = getenv("NODE_ICU_DATA");
3444+
}
3445+
// Initialize ICU.
3446+
// If icu_data_dir is NULL here, it will load the 'minimal' data.
3447+
if (!i18n::InitializeICUDirectory(icu_data_dir)) {
3448+
FatalError(NULL, "Could not initialize ICU "
3449+
"(check NODE_ICU_DATA or --icu-data-dir parameters)");
3450+
}
3451+
#endif
34133452
// The const_cast doesn't violate conceptual const-ness. V8 doesn't modify
34143453
// the argv array or the elements it points to.
34153454
V8::SetFlagsFromCommandLine(&v8_argc, const_cast<char**>(v8_argv), true);

0 commit comments

Comments
 (0)