forked from noaa-ocs-modeling/CoastalApp
-
Notifications
You must be signed in to change notification settings - Fork 0
/
HOWTO
655 lines (487 loc) · 26.2 KB
/
HOWTO
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
Modified: Panagiotis Velissariou - 12/05/2020
==================================
Coastal Act Team - 9/13/2019
Thanks to all the contributors.
==================================
Forked latest version of the NEMS software program from the
official NEMS repository at https://github.com/NOAA-EMC/NEMS
git log:
commit e05bf2a554544093f02bf177658ad432cbafbaa8
Merge: c8e05a8 95c63cb
Author: RatkoVasic-NOAA <37597874+RatkoVasic-NOAA@users.noreply.github.com>
Date: Fri Sep 13 15:13:28 2019 -0400
The purpose of this document is to setup, integrate, compile and run ESMF aware
applications, also known as NUOPC CAP using the NEMS framework which is also an
application of ESMF framework. The idea is to show, with an example, how to add
a new model into a couple system, using the NEMS program.
To put it simply NEMS consists of a main program and a driver where different
models can integrate into and exchange data at runtime in a standard manner.
Figure below, depicts simplified relations among the ESMF-NUOPC & NEMS & MODELS
Model Coupling Main Exec. Models + Interface(CAP)
Framework & Driver to ESMF & Makefile
Fragment for NEMS
============== ========== ======================
ESMF & NUOPC --> NEMS --> Mode1(1), Model(2), Model(3), ...
| | |
| | |
V V V
libraries of unifier to models participating
callable functions bring models to be coupled together
designed to couple together as an through calls to ESMF & NUOPC(a.k.a CAP)
Models executable through makefile & manual setup to NEMS
Note:
A CAP is a piece of program written in Fortran extending the model source
code through calling the appropriate functions from ESMF & NUOPC frameworks.
The models can communicate with one another through CAPs.
A makefile fragment is a section of a makefile that is cut from a makefile
and is written into a different file. A makefile fragment is a part of a
whole. So all the parts must fit together like pieces of a puzzle in order
to complete the entire makefile.
*****************************************************************
In rush to use NEMS - below are the required steps for your model
and your computing system and your machine name and your compiler
*****************************************************************
Assumptions:
NWM - is the model to be added to the NEMS system - <MODEL>
Hera - is the user computer, machine name - <COMPUTER>
rdhpcs - is the computing system where hera is a node of - <COMPUTING>
intel - is the compiler of choice in hera computer - <COMPILER>
ROOTDIR - is the directory of your choice in hera where you
clone the NEMS in
cd ROOTDIR
1) git clone --recursive https://github.com/noaa-ocs-modeling/CoastalApp
2) mkdir conf
3) mkdir modulefiles
4) mkdir modulefiles/<COMPUTER>
5) vi modulefiles/<COMPUTER>/ESMF_NUOPC -- see TABLE 1
6) vi conf/configure.nems.<COMPUTER>.<COMPILER> -- see TABLE 2
7) mkdir NEMS/src/incmake/env/<COMPUTING> -- if it doesn't exist
8) vi NEMS/src/incmake/env/<COMPUTING>/<COMPUTER>.<COMPILER>.mk -- see step 4 item c below
9) vi NEMS/src/incmake/env/<COMPUTING>/detect.mk -- see TABLE 3
10) vi NEMS/src/incmake/buildenv.mk
include $(call locate_incmake_file,env/<COMPUTER>/detect.mk) -- add this line
include $(call locate_incmake_file,env/uname/detect.mk) -- must be the last in line
11) cp NEMS/src/incmake/example_component_FOO.mk NEMS/src/incmake/component_<COMPUTER>.mk
12) vi NEMS/src/incmake/component_<MODEL>.mk -- see TABLE 4
13) vi NEMS/src/module_EARTH_GRID_COMP.F90 -- copy lines 171 and 3869 and change model name
14) Adjust your model makefile -- see TABLE 5 and component_<MODEL>.mk (step 12)
TABLE 0 depicts above steps in form of tree structure
************************************************************************************
Below instrutions show how to add a new model for the coupling with other models and
how to compile and integrate with the NEMS framwork.
************************************************************************************
NEMS is designed to work with different earth system models developed in variety
of Linux based computer systems. There are two parts for the user to acheive
to integrate into NEMS program. Apart from developing a NUOPC CAP for the model
to participate in a coupling project, user should write a makefile fragment in
the model's makefile to comply with NEMS workflow for a successful integration.
The second part of the user's action requires configuration, setup, and coding
into NEMS source files. We start with the sencond part first. The first part is
described on step 7.
Assumptions:
NWM - is the model to be added to the NEMS system
Hera - is the user computer, machine name
rdhpcs - is the computing system where hera is a node of
intel - is the compiler of choice in hera computer with the version of interest
ROOTDIR - is the directory of your choice in hera
1) cd <ROOTDIR>
2) Get the source codes into <ROOTDIR>
a) git clone https://github.com/NOAA-EMC/NEMS
The above command only gets the NEMS code from the official repository.
Most likely, users are required permission from the NEMS administrator
to access this repo.
Or get the development (a.k.a dev) version of the NEMS code to follow
the instruction easier.
b) git clone --recursive https://github.com/noaa-ocs-modeling/CoastalApp
The development version includes the NEMS code and few model codes
such as ADCIRC, WW3, ATMESH. This version is located in a public
github repository at https://github.com/noaa-ocs-modeling .
To clone from dev you should have an account on github repository
with a valid email, username, and a password. You also required to
request a collaborator access to this repo. by sending an email to
saeed.moghimi@noaa.gov.
During cloning from dev, github asks for a username and password,
which are those you use to login to your github account.
After the first clone, you should use "git pull" command to access
the latest changes from dev repo.
For user who used 2a), contents are put into TABLEs and they are located
at the end of instructions.
3) Setup and configure user environment required by NEMS
a) Create 4 directories: compsets\ conf\ modulefiles\ parm\ in <ROOTDIR> if
they do not exist.
Note: These 4 directories are user directories and may or may not be saved as part
of the NEMS source codes. However, there are name dependencies in NEMS code
to these directories.
You should have 5 directories, including NEMS, if cloned by way of 2a).
You should have 5 directories, including NEMS, and many models that are
integrated into this development version of NEMS (github version), if
cloned by way of 2b).
Since this document is describing the addition of NWM, the final product is
also included in development version. As such you should also see NWM final
product & to checkout each step of the instruction as you arrive to them.
b) mkdir modulefiles/hera -- this should match the machine name
c) vi modulefiles/hera/ESMF_NUOPC -- could be any name
Note: This file is to set the compilation environments for this version of NEMS and
ESMF and all the participant coupled models. Examples are MPI libraries, NETCDF,
HDF, SZIP, ESMF, Compilers, and more. See <ROOTDIR>/modulefiles/hera/ESMF_NUOPC
The module file name can be any name, however, the directory name "modulefiles"
is expected. See <ROOTDIR>/NEMS/src/incmake/configure_vars.mk for detail.
During runtime the contents of this modulefile will be copied into "modules.nems"
located at the <ROOTDIR>/NEMS/src/conf/modules.nems
d) vi conf/configure.nems.hera.intel
Note: This makefile fragment is the common part of a makefile that applies to all the
participant coupled models including NEMS itself in Hera computer using intel
compiler. See the contents in <ROOTDIR>/conf/configure.nems.hera.intel
The name of this makefile is constructed by appending the machine name, "hera",
dotted with compiler selected, "intel", to the constant "configure.nems", formula
is configure.nems.{COMPUTER}.{COMPILER}
We suggest to copy contents of another makefile into this file and modify according
to your system. Make sure CC, FC and F77 compiler options are set to the intended
compilers.
e) Place holder for the <ROOTDIR>/compsets and <ROOTDIR>/parm - TODO
4) Setup, configuration, and integration of the machine and user modulefile into NEMS system
a) ls NEMS/src/incmake/env
Look for directory named after your computing system (i.e. rdhpcs).
Note: There are two types of computing systems that NEMS is concerned with.
case 1) The system contains many nodes(computers) such as "rdhpcs" with nodes known as
"hera", "gaea", "jet", ...
case 2) The system is an stand alone system such as "wcoss"
b) mkdir NEMS/src/incmake/env/rdhpcs -- if it doesn't exist
In our case "hera" is a machine belong to the "rdhpcs" where other nodes such as "gaea" and
"jet" were setup but "hera" was not. So we need to implement "hera" under its parent system
"rdhpcs".
In case 1), such as our case example, user should create the parent directory in <ROOTDIR>/
NEMS/src/incmake/env, if it doesn't exist.
In case 2), machine name is also the parent and user should create the directory with the node
name in <ROOTDIR>/NEMS/src/incmake/env, if it doesn't exist.
c) vi NEMS/src/incmake/env/rdhpcs/hera.intel.mk
Under parent directory you must create a makefile fragment with this formula, defined by NEMS
{COMPUTER}.{COMPILER}.mk
Note: The parent and the node names are only "strings" and NEMS program does not care what they
are called as long as you are consistant on using the same name in the right place.
Add below contents to this file or copy from another similar file and then edit
MACHINE_ID=hera
FULL_MACHINE_ID=hera
USE_MODULES=YES
DEFAULT_MODULE=$(FULL_MACHINE_ID)/ESMF_NUOPC
BUILD_TARGET=$(FULL_MACHINE_ID).$(NEMS_COMPILER)
NEMS_COMPILER=intel
MODULE_LOGIC=$(call ULIMIT_MODULE_LOGIC,200000)
Note: The first and 2nd lines are the machine name, the third line is a toggle set to "YES".
This means that NEMS program expects a user defined module file set on the fourth line.
This the user modulefile is the same modulefile we created previousely on step 3 item b
called <ROOTDIR>/modulefiles/hera/ESMF_NUOPC.
See the file NEMS/src/incmake/configure_rules.mk for more information on the options.
d) vi NEMS/src/incmake/env/rdhpcs/detect.mk
The purpose of this file is to locate the machine and find the modules and model codes, etc.
In order to do so, especially in case 2) where there are many computer nodes under one parent
computing system, the task is not that trivial. That is why selection of computing system name
and computer name are important to pay attention to.
In our case example, file systems /scratch1 and /scratch2 are mounting points on "Hera" node.
As a result we added the following statement to the "detect.mk" file to find our machine.
ifneq (,$(and $(wildcard /scratch1),$(wildcard /scratch2)))
NEMS_COMPILER?=intel
$(call add_build_env,hera.$(NEMS_COMPILER),env/rdhpcs/hera.$(NEMS_COMPILER).mk)
...
After finding the computer, the code also calls the "add_build_env" function of the NEMS code
and passes "hera.intel" and "NEMS/src/incmake/env/rdhpcs/hera.intel.mk", created on step 4c
as arguments to this function in order to build the required environment.
In summary, you have to add such statements to each detect.mk file where each node under one
parent can be identified uniquely. There is one such detect.mk file per computer name.
e) vi NEMS/src/incmake/buildenv.mk
Add the newely created "detect.mk" file to the "buildenv.mk" above the "uname/detect.mk" line.
include $(call locate_incmake_file,env/rdhpcs/detect.mk)
include $(call locate_incmake_file,env/uname/detect.mk) -- comes last
Note: It is important to make sure uname/detect.mk line is always the last line after all other
detect.mk lines. Otherwise, uname/detect.mk, if executed, assumes sets the computer name
to the result of linux command "uname" which may or may not equate to the machine name you
defined.
If you do not have your detect.mk file included, you will get an error from systems file
NEMS/src/incmake/buildenv.mk.
5) Setup, configuration, and integration of the model specific compiled files into NEMS system
a) cp NEMS/src/incmake/example_component_FOO.mk NEMS/src/incmake/component_NWM.mk
Note: Each model is required to have a makefile fragment using formula component_{model_name}.mk
There is a system "example_component_FOO.mk" file that is for this purpose, to copy from it.
The selected model_name is significant because this string is like a keyword and is shared
between the model and the NEMS system. More on this later in this document.
b) vi NEMS/src/incmake/component_NWM.mk
Through this file NEMS dynamically communicates information about the model's compiled modules location
as well as source files location.
1) Edit the file by changing the word "FOO" to "NWM" and "foo" to "nwm" without double quotes.
2) Further edit this file to match your model's compilation and linkage process. It is best to look
at this 3 files at the same time to see the connections.
- NEMS/src/incmake/component_NWM.mk
- $(ROOTDIR)/NWM_INSTALL/nwm.mk
- <ROOTDIR>/NWM/trunk/NDHMS/CPL/NUOPC_cpl/Makefile around line 200, rule $(CAP_MK)
OR checkout ADCIRC
- NEMS/src/incmake/component_ADCIRC.mk
- $(ROOTDIR)/ADCIRC_INSTALL/adcirc.mk
- <ROOTDIR>/ADCIRC/cpl/nuopc/makefile.adc_cap.nuopc lines 73-95
6) Integration of the model CAP into NEMS system
a) vi NEMS/src/module_EARTH_GRID_COMP.F90
Edit to add below statements tailored to your model name into two places in this file:
#ifdef FRONT_NWM
use FRONT_NWM, only: NWM_SS => SetServices
#endif
And further down the file also add:
#ifdef FRONT_NWM
call NUOPC_DriverAddComp(driver, trim(prefix), NWM_SS, &
petList=petList, comp=comp, rc=rc)
if (ESMF_LogFoundError(rcToCheck=rc, msg=ESMF_LOGERR_PASSTHRU, &
line=__LINE__, file=trim(name)//":"//__FILE__)) return
#else
write (msg, *) "Model '", trim(model), "' was requested, "// &
"but is not available in the executable!"
call ESMF_LogSetError(ESMF_RC_NOT_VALID, msg=msg, line=__LINE__, &
file=__FILE__, rcToReturn=rc)
return ! bail out
#endif
7) Model makefile specific fragment for NEMS
Each model's makefile has to provide following information to the NEMS system:
ESMF_DEP_FRONT = nwm_nuopc_cap
ESMF_DEP_INCPATH = /scratch2/COASTAL/coastal/save/NAMED_STORMS/NEMS_APP/NWM/trunk/NDHMS/CPL/NUOPC_cpl
/scratch2/COASTAL/coastal/save/NAMED_STORMS/NEMS_APP/NWM_INSTALL
ESMF_DEP_CMPL_OBJS =
ESMF_DEP_LINK_OBJS = /scratch2/COASTAL/coastal/save/NAMED_STORMS/NEMS_APP/NWM_INSTALL/libnwm_nuopc.a
ESMF_DEP_SHRD_PATH =
ESMF_DEP_SHRD_LIBS =
These information are writen to the file $(ROOTDIR)/NWM_INSTALL/nwm.mk from the model's makefile. Related
section of NWM makefile is shown below:
CAP_DEP_FRONT := nwm_nuopc_cap # for NEMS, must be same name as its module
CAP_VERS := VERSION
CAP_MK := nwm.mk # for NEMS
$(CAP_MK):
@echo $(HR)
@echo "Generating NUOPC Makefile Fragment"
@echo
@echo "# ESMF self-describing build dependency makefile fragment" > $(CAP_MK)
@echo "" >> $(CAP_MK)
@echo "ESMF_DEP_FRONT = $(CAP_DEP_FRONT)" >> $(CAP_MK)
@echo "ESMF_DEP_INCPATH = $(CAP_DIR)" >> $(CAP_MK)
@echo "ESMF_DEP_CMPL_OBJS = " >> $(CAP_MK)
@echo "ESMF_DEP_LINK_OBJS = $(CAP_DIR)/$(CAP_LIB)" >> $(CAP_MK)
@echo "ESMF_DEP_SHRD_PATH = " >> $(CAP_MK)
@echo "ESMF_DEP_SHRD_LIBS = " >> $(CAP_MK)
Note: The value of the ESMF_DEP_FRONT is the name of the model's CAP module name and not the model's
file name.
8) Model source code locations
a) cp -rf nwm <ROOTDIR>/NWM
NEMS system expects source codes of all the model participant to be available in the same level
directory as NEMS source code. This location is the <ROOTDIR> where we cloned the NEMS. As the last
step before compilation, copy your model into <ROOTDIR>
9) There are 3 files located in <ROOTDIR> worth noting
a) model_configure
b) nems.configure
c) build.sh
d) nems.job
***************************
To build model(s) with NEMS
***************************
1) got to: <ROOTDIR>/NEMS
clean and/or distclean before compiling
2) make -f GNUmakefile distclean_ADCIRC COMPONENTS="ADCIRC"
make -f GNUmakefile distclean_NWM COMPONENTS="NWM"
make -f GNUmakefile distclean_WW3DATA COMPONENTS="WW3DATA"
make -f GNUmakefile distclean_ATMESH COMPONENTS="ATMESH"
make -f GNUmakefile distclean_NEMS COMPONENTS="NEMS"
Note: You have to make sure your component_<model_name>.mk and your model's
makefile has distclean rule build into it.
build one or more model
3) make -f GNUmakefile build COMPONENTS="ADCIRC WW3DATA ATMESH NWM"
make -f GNUmakefile build COMPONENTS="ADCIRC"
OR simply, add the new model to <ROOTDIR>/build.sh and while in
<ROOTDIR> type: ./build.sh
*********************************
TO run NEMS: TODO - workflow is in progress
*********************************
1) create a <RUNDIR> (better be outside of NEMSDIR!!)
2) copy all the input files/environ files/config files/etc/
required by all the coupled models into <RUNDIR>
3) copy <ROOTDIR>nsem.job into <RUNDIR>
4) edit nsem.job file to adjsut the <ROOTDIR> and the NODE/TASK options
per model_configure and nems.configure files
5) help yourself to edit any other variables in nsem.job file as needed
6) then run the command > sbatch nsem.job
7) check the process > squeue -u $USER
Note: There are two run time configuration files, required by NEMS.
Example of these two files are located in this directory for convenience.
To use them, you should copy and rename them to exclude the ".sample".
These two files are hard coded into the NEMS source codes and expected files
names are:
- model_configure and
- nems.configure
The file <ROOTDIR>/nsem.job.sample is an example of SLURM job batch file.
******************************************************************
TABLE 0 - Schematic workflow of files involved in pre compilation
******************************************************************
<ROOTDIR>
|
|--- NEMS
| |
| |--- src/ --- module_EARTH_GRID_COMP.F90
| |
| |--- incmake/ --- buildenv.mk, component_NWM.mk, example_component_FOO.mk
| | |
| | |--- env/ --- rdhpcs/ --- detect.mk, hera.intel.mk
| |
| |--- conf/ --- modules.nems
|--- ModelA
|
|--- ModelA_INSTALL/ --- libmodelA.a, modelA.mk, modelA.mod
|
|--- ModelB
|
|--- ModelB_INSTALL/ --- libmodelB.a, modelB.mk, modelB.mod
|
|--- ModelN
|
|--- ModelN_INSTALL/ --- libmodelN.a, modelN.mk, modelN.mod
|
|--- modulefiles/ --- hera/ --- ESMF_NUOPC
|
|--- conf/ --- configure.nems.hera.intel
|
|--- NEMS_RUN
*************************************************
TABLE 1 - Contents of modulefiles/hera/ESMF_NUOPC
*************************************************
#%Module
#
# NEMS FV3 Prerequisites: Hera/Intel
# load programming environment
# this typically includes compiler, MPI and job scheduler
#
module load intel/18.0.5.274
module load szip/2.1
module load hdf5/1.10.4
module load impi/2018.0.4
module load netcdf/4.6.1
# environment for ESMF v8.0.0 beta snapshot 48g
module use /home/emc.nemspara/SOFT-hera/modulefiles
module load esmf/8.0.0bs48g
# add others for each model as needed
****************************************************
TABLE 2 - Contents of conf/configure.nems.hera.intel
****************************************************
# NEMS configuration file
#
# Platform: Hera
# Compiler: Intel with IntelMPI
SHELL = /bin/sh
# Include the common configuration parts
include $(TOP)/conf/configure.nems.NUOPC
# Other settings
NETCDF_INC = -I$(NETCDF)/include
NETCDF_LIB = -L$(NETCDF)/lib -lnetcdf
NEMSIO_INC = -I${LIBDIR}/incmod/nemsio
NEMSIO_LIB = -L${LIBDIR} -lnemsio
SYS_LIB =
EXTLIBS = $(NEMSIO_LIB) \
$(NETCDF_LIB) \
$(ESMF_LIB) \
$(SYS_LIB) -lm
EXTLIBS_POST = $(NEMSIO_LIB) \
$(ESMF_LIB) \
$(NETCDF_LIB) \
$(SYS_LIB)
#
FC = mpiifort -g -qopenmp -mkl=sequential -align array32byte -lmkl_intel_lp64 -lmkl_core -lmkl_sequential -lpthread -qopenmp -convert big_endian -assume byterecl -mkl=sequential
F77 = mpiifort -g -qopenmp -mkl=sequential -align array32byte -lmkl_intel_lp64 -lmkl_core -lmkl_sequential -lpthread -qopenmp -convert big_endian -assume byterecl -mkl=sequential
FREE = -free
FIXED = -fixed
R8 = -r8
FINCS = $(ESMF_INC) $(NEMSIO_INC) $(NETCDF_INC)
#TRAPS = -g -fno-inline -no-ip -traceback -ftrapuv -fpe0 -ftz -check all -check noarg_temp_created -fp-stack-check
FFLAGS = $(TRAPS) $(FINCS) -fp-model strict
OPTS_NMM = -g -fno-inline -no-ip -traceback -ftrapuv -fpe0 -ftz -check all -check noarg_temp_created -fp-stack-check $(FREE)
FFLAGM_DEBUG =
FFLAGS_NMM = $(MACROS_NWM) $(OPTS_NMM) $(FFLAGS)
FPP = -fpp
CPP = cpp -P -traditional
CPPFLAGS = -DENABLE_SMP -DCHNK_RRTM=8
AR = ar
ARFLAGS = -r
RM = rm
***********************************************************
TABLE 3 - Contents of NEMS/src/incmake/env/rdhpcs/detect.mk
***********************************************************
ifneq (,$(and $(wildcard /scratch),$(wildcard /scratch3)))
NEMS_COMPILER?=intel
$(call add_build_env,theia.$(NEMS_COMPILER),env/rdhpcs/theia.$(NEMS_COMPILER).mk)
else
ifneq (,$(and $(wildcard /scratch1),$(wildcard /scratch2)))
NEMS_COMPILER?=intel
$(call add_build_env,hera.$(NEMS_COMPILER),env/rdhpcs/hera.$(NEMS_COMPILER).mk)
else
ifneq (,$(and $(wildcard /lfs1),$(wildcard /lfs3)))
NEMS_COMPILER?=intel
$(call add_build_env,jet.$(NEMS_COMPILER),env/rdhpcs/jet.$(NEMS_COMPILER).mk)
else
ifneq (,$(shell hostname | grep -i gaea))
NEMS_COMPILER?=intel
$(call add_build_env,gaea.$(NEMS_COMPILER),env/rdhpcs/gaea.$(NEMS_COMPILER).mk)
endif
endif
endif
endif
********************************************************
TABLE 4 - Contents of NEMS/src/incmake/component_NWM.mk
********************************************************
# Location of the ESMF makefile fragment for this component:
nwm_mk = $(NWM_BINDIR)/nwm.mk
all_component_mk_files+=$(nwm_mk)
# Location of source code and installation
NWM_SRCDIR?=$(ROOTDIR)/NWM/trunk/NDHMS
NWM_BINDIR?=$(ROOTDIR)/NWM_INSTALL
# Make sure the expected directories exist and are non-empty:
$(call require_dir,$(NWM_SRCDIR),NWM source directory)
# Rule for building this component:
build_NWM: $(nwm_mk)
env_file=$(NWM_SRCDIR)/setEnvar.sh
comp_opt=3
# HOW to source env here??
#
NWM_ALL_OPTS= \
COMP_SRCDIR="$(NWM_SRCDIR)" \
COMP_BINDIR="$(NWM_BINDIR)" \
MACHINE_ID="$(MACHINE_ID)"
$(nwm_mk): configure $(CONFDIR)/configure.nems
@echo ""
+$(MODULE_LOGIC) ; cd $(NWM_SRCDIR) ; exec ./compile_nuopc_NoahMP.sh $(env_file) $(esmf_env) $(comp_opt)
+$(MODULE_LOGIC) ; cd $(NWM_SRCDIR)/CPL/NUOPC_cpl ; exec $(MAKE) -f Makefile nuopcinstall \
$(NWM_ALL_OPTS) DESTDIR=/ "INSTDIR=$(NWM_BINDIR)"
@echo ""
test -d "$(NWM_BINDIR)"
@echo ""
test -s $(nwm_mk)
# Rule for cleaning the SRCDIR and BINDIR:
clean_NWM:
@echo ""
+-cd $(NWM_SRCDIR) ; exec $(MAKE) -f Makefile.nuopc nuopcclean
distclean_NWM: clean_NWM
@echo ""
rm -rf $(NWM_BINDIR)
distclean_NUOPC:
@echo ""
+-cd $(NWM_SRCDIR) ; exec $(MAKE) -C CPL/NUOPC_cpl -f Makefile nuopcclean
rm -rf $(NWM_BINDIR)
***********************************************************
TABLE 5 - <ROOTDIR>/NWM/trunk/NDHMS/CPL/NUOPC_cpl/Makefile
***********************************************************
CAP_DEP_FRONT := nwm_nuopc_cap # for NEMS, must be same name as its module
CAP_VERS := VERSION
CAP_MK := nwm.mk # for NEMS
$(CAP_MK):
@echo $(HR)
@echo "Generating NUOPC Makefile Fragment"
@echo
@echo "# ESMF self-describing build dependency makefile fragment" > $(CAP_MK)
@echo "" >> $(CAP_MK)
@echo "ESMF_DEP_FRONT = $(CAP_DEP_FRONT)" >> $(CAP_MK)
@echo "ESMF_DEP_INCPATH = $(CAP_DIR)" >> $(CAP_MK)
@echo "ESMF_DEP_CMPL_OBJS = " >> $(CAP_MK)
@echo "ESMF_DEP_LINK_OBJS = $(CAP_DIR)/$(CAP_LIB)" >> $(CAP_MK)
@echo "ESMF_DEP_SHRD_PATH = " >> $(CAP_MK)
@echo "ESMF_DEP_SHRD_LIBS = " >> $(CAP_MK)
=====================================================
Please add contents as needed into <ROOTDIR>.gitignore file: