{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# pyclmuapp on HPC: Command line\n", "\n", "Junjie Yu, 2024-06-19\n", "\n", "before running the script, you need to load the following modules:\n", "`pip install pyclmuapp`\n", "\n", "For now, HPC is more favoribale for `apptainer/singularity`. \n", "\n", "We provide `pyclmuapp_hpc` bash script to make more lightwiegt usage of pyclmuapp\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "test pyclmuapp" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "pyclmuapp 0.0.0. Author: Junjie Yu. Email: yjj1997@live.cn\n" ] } ], "source": [ "! pyclmuapp -v" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "here we use the provide `pyclmuapp_hpc` script to quickly run the `pyclmuapp` command on the HPC." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Start pyclmuapp\n", "container_type: singularity\n", "task: init\n", "Start time: Wed 19 Jun 2024 10:06:13 PM BST\n", "Namespace(case_name='usp_case', clean=False, container_type='singularity', end_month=12, end_year=2012, forcing_action=0, forcing_var=None, has_container=True, hist_mfilt=1000000000, hist_nhtfrq=1, hist_type='GRID', iflog=True, init=True, input_path=None, lat=None, log_path=None, logfile='pyclmuapp.log', lon=None, output_path=None, output_prefix='_clm.nc', outputname='surfdata.nc', pct_urban=[0, 0, 100.0], pwd='/mnt/lustre/a2fs-work2/work/n02/n02/junjieyu/pyclmuapp', pyclmuapp_mode='usp', run_refcase='None', run_refdate='None', run_startdate=None, run_type='coldstart', script=None, scripts_path=None, soildata=None, start_month=1, start_year=2012, stop_n='1', stop_option='ndays', surf_action=0, surf_var=None, urbsurf=None, usr_domain=None, usr_forcing=None, usr_surfdata=None, zbot=30)\n", "Folder 'inputfolder' created successfully!\n", "Folder 'outputfolder' created successfully!\n", "Folder 'logfolder' created successfully!\n", "Folder 'scriptsfolder' created successfully!\n", "Folder '/mnt/lustre/a2fs-work2/work/n02/n02/junjieyu/pyclmuapp/inputfolder/usp' created successfully!\n", "The pyclmuapp is initialized.\n" ] } ], "source": [ "! sh pyclmuapp_hpc --task init\n", "# this is equivalent to the following command\n", "# container_type=singularity\n", "# pyclmuapp --has_container False --container_type $container_type --init True\n", "# folders will be created in the current directory\n", "# the container will be downloaded and the image will be created" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Start pyclmuapp\n", "container_type: singularity\n", "task: usp\n", "Start time: Wed 19 Jun 2024 10:06:16 PM BST\n", "Namespace(case_name='pyclmuapp', clean=False, container_type='singularity', end_month=12, end_year=2012, forcing_action=0, forcing_var=None, has_container=True, hist_mfilt=1000000000, hist_nhtfrq=1, hist_type='GRID', iflog=True, init=False, input_path=None, lat=None, log_path=None, logfile='pyclmuapp.log', lon=None, output_path=None, output_prefix='_clm.nc', outputname='surfdata.nc', pct_urban=[0, 0, 100.0], pwd='/mnt/lustre/a2fs-work2/work/n02/n02/junjieyu/pyclmuapp', pyclmuapp_mode='usp', run_refcase='None', run_refdate='None', run_startdate='2012-01-01', run_type='coldstart', script=None, scripts_path=None, soildata=None, start_month=1, start_year=2012, stop_n='15', stop_option='ndays', surf_action=0, surf_var=None, urbsurf=None, usr_domain=None, usr_forcing='era5_forcing_51.5_0.12_30.0_2012_1_2012_12.nc', usr_surfdata='surfdata.nc', zbot=30)\n", "Folder 'inputfolder' already exists.\n", "Folder 'outputfolder' already exists.\n", "Folder 'logfolder' already exists.\n", "Folder 'scriptsfolder' already exists.\n", "Folder '/mnt/lustre/a2fs-work2/work/n02/n02/junjieyu/pyclmuapp/inputfolder/usp' already exists.\n", "Copying the file era5_forcing_51.5_0.12_30.0_2012_1_2012_12.nc to the /mnt/lustre/a2fs-work2/work/n02/n02/junjieyu/pyclmuapp/inputfolder/usp\n", "The case is: pyclmuapp\n", "The log file is: pyclmuapp.log\n", "The output file is: {'original': ['/mnt/lustre/a2fs-work2/work/n02/n02/junjieyu/pyclmuapp/outputfolder/lnd/hist/pyclmuapp_clm0_2024-06-19_22-13-01_clm.nc']}\n" ] } ], "source": [ "! sh pyclmuapp_hpc\n", "# revise the parameters in the pyclmuapp_hpc before running the command\n", "# 1. pwd or input_path, output_path, log_path, scripts_path\n", "# 2. usr_forcing, usr_surfdatab\n", "# 3. RUN_STARTDATE, STOP_OPTION, STOP_N, RUN_TYPE" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "the `pyclmuapp_hpc` is a shell script that contains the following commands:" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "#! bin/bash\n", "\n", "# Junjie Yu, Manchester, UK, 2024-06-19\n", "# This is an example script to run pyclmuapp\n", "\n", "# =================================================================================================\n", "# Before running this script, please make sure you have installed pyclmuapp and docker/singularity\n", "# You can install pyclmuapp by running: pip install pyclmuapp\n", "# then you can run the following command to check if pyclmuapp is installed successfully:\n", "# $ pyclmuapp -h\n", "\n", "# =================================================================================================\n", "# before running this script, you also need to revise the following parameters:\n", "# 1. pwd or input_path, output_path, log_path, scripts_path\n", "# 2. usr_forcing, usr_surfdata\n", "# 3. RUN_STARTDATE, STOP_OPTION, STOP_N, RUN_TYPE,\n", "# 4. lat, lon, zbot, start_year, end_year, start_month, end_month\n", "# 5. pct_urban, urbsurf, soildata\n", "\n", "# =================================================================================================\n", "# also you need to download the urban surface data file and soil data file\n", "# The download link is provided in the script\n", "# How to run this script:\n", "# $ bash example.sh --container_type docker --task usp # or get_forcing, get_surfdata, init\n", "# $ bash example.sh --container_type singularity --task usp # or get_forcing, get_surfdata, init\n", "# =================================================================================================\n", "\n", "container_type=singularity\n", "task=usp\n", "\n", "while [[ $# -gt 0 ]]; do\n", " case \"$1\" in\n", " --container_type)\n", " container_type=$2\n", " shift 2\n", " ;;\n", " --task)\n", " task=$2\n", " shift 2\n", " ;;\n", " *)\n", " echo \"Unknown option: $1\"\n", " exit 1\n", " ;;\n", " esac\n", "done\n", "\n", "echo \"Start pyclmuapp\"\n", "echo \"container_type: $container_type\"\n", "echo \"task: $task\"\n", "echo \"Start time: $(date)\"\n", "\n", "if [ $task == \"init\" ]; then\n", " pyclmuapp --has_container False --container_type $container_type --init True\n", "fi\n", "\n", "if [ $task == \"usp\" ]; then\n", "\n", " # must revise the \n", " # 1. pwd or input_path, output_path, log_path, scripts_path\n", " # 2. usr_forcing, usr_surfdata\n", " # 3. RUN_STARTDATE, STOP_OPTION, STOP_N, RUN_TYPE,\n", " pyclmuapp \\\n", " --container_type $container_type \\\n", " --iflog True \\\n", " --logfile \"pyclmuapp.log\" \\\n", " --usr_forcing \"era5_forcing_51.5_0.12_30.0_2012_1_2012_12.nc\" \\\n", " --usr_surfdata \"surfdata.nc\" \\\n", " --RUN_STARTDATE \"2012-01-01\" \\\n", " --STOP_OPTION \"ndays\" \\\n", " --STOP_N \"15\" \\\n", " --RUN_TYPE \"coldstart\" \\\n", " --hist_type \"GRID\" \\\n", " --hist_nhtfrq \"1\" \\\n", " --hist_mfilt \"1000000000\" \\\n", " --output_prefix \"_clm.nc\" \\\n", " --CASE_NAME \"pyclmuapp\" \\\n", " --run_type \"coldstart\" \\\n", " --run_refcase \"None\" \\\n", " --run_refdate \"None\" \n", "fi\n", "\n", "if [ $task == \"get_forcing\" ]; then\n", " # revise the lat, lon, zbot, start_year, end_year, start_month, end_month\n", " pyclmuapp \\\n", " --pyclmuapp_mode get_forcing \\\n", " --lat 51.5074 --lon 0.1278 --zbot 30 \\\n", " --start_year 2012 --end_year 2012 \\\n", " --start_month 1 --end_month 12\n", "fi\n", "\n", "\n", "if [ $task == \"get_surfdata\" ]; then\n", " # revise the lat, lon, pct_urban, urbsurf, soildata\n", " pyclmuapp \\\n", " --pyclmuapp_mode get_surfdata \\\n", " --lat 51.5074 --lon 0.1278 \\\n", " --outputname \"surfdata.nc\" \\\n", " --pct_urban 0,0,100.0 \\\n", " --urbsurf \"./inputfolder/mksrf_urban_0.05x0.05_simyr2000.c120601.nc\" \\\n", " --soildata \"./inputfolder/mksrf_soitex.10level.c010119.nc\"\n", "fi\n", "# how to download the urban surface data file: \n", "#wget --no-check-certificate -O {soildata_name} \\\n", "# https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata/lnd/clm2/rawdata/mksrf_soitex.10level.c010119.nc\n", "#wget --no-check-certificate -O {urbsurf_name} \\\n", "# https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata/lnd/clm2/rawdata/mksrf_urban_0.05x0.05_simyr2000.c120601.nc\n", "\n", "\n", "# pyclmuapp -h\n", "#usage: pyclmuapp [-h] [--version] [--init INIT] [--pwd PWD] [--container_type CONTAINER_TYPE] [--input_path INPUT_PATH]\n", "# [--output_path OUTPUT_PATH] [--log_path LOG_PATH] [--scripts_path SCRIPTS_PATH] [--pyclmuapp_mode PYCLMUAPP_MODE]\n", "# [--has_container HAS_CONTAINER] [--usr_domain USR_DOMAIN] [--usr_forcing USR_FORCING] [--usr_surfdata USR_SURFDATA]\n", "# [--output_prefix OUTPUT_PREFIX] [--case_name CASE_NAME] [--run_startdate RUN_STARTDATE] [--stop_option STOP_OPTION]\n", "# [--stop_n STOP_N] [--run_type RUN_TYPE] [--run_refcase RUN_REFCASE] [--run_refdate RUN_REFDATE] [--iflog IFLOG]\n", "# [--logfile LOGFILE] [--hist_type HIST_TYPE] [--hist_nhtfrq HIST_NHTFRQ] [--hist_mfilt HIST_MFILT] [--clean CLEAN]\n", "# [--surf_var SURF_VAR] [--surf_action SURF_ACTION] [--forcing_var FORCING_VAR] [--forcing_action FORCING_ACTION] [--script SCRIPT]\n", "# [--urbsurf URBSURF] [--soildata SOILDATA] [--pct_urban PCT_URBAN] [--lat LAT] [--lon LON] [--outputname OUTPUTNAME] [--zbot ZBOT]\n", "# [--start_year START_YEAR] [--end_year END_YEAR] [--start_month START_MONTH] [--end_month END_MONTH]\n", "#\n", "#pyclmuapp command line tool.\n", "#\n", "#options:\n", "# -h, --help show this help message and exit\n", "# --version, -v show program's version number and exit\n", "# --init INIT Init pyclmuapp, default is False\n", "# --pwd PWD Param for usp and script. Current working directory, default is pwd, can be none. If is not none, the the input_path,\n", "# output_path, log_path, scripts_path will be used in pwd or be created. If none, the input_path, output_path, log_path,\n", "# scripts_path should be provided.\n", "# --container_type CONTAINER_TYPE\n", "# Param for usp and script. Container type, default is docker, can be singularity\n", "# --input_path INPUT_PATH\n", "# Param for usp and script. CTSM input path, default is None. The path will be binded to \"inputdata\" in container\n", "# --output_path OUTPUT_PATH\n", "# Param for usp and script. CTSM output path, default is None. The path will be binded to \"Archive\" in container\n", "# --log_path LOG_PATH Param for usp and script. CTSM log path, default is None. The path will be binded to \"CaseOutputs\" in container\n", "# --scripts_path SCRIPTS_PATH\n", "# Param for usp and script. CTSM scripts path, default is None. The path will be binded to \"/p/scripts\" in container\n", "# --pyclmuapp_mode PYCLMUAPP_MODE\n", "# pyclmuapp mode, default is usp, can be script, pts, get_forcing, get_surfdata\n", "# --has_container HAS_CONTAINER\n", "# Param for usp and script. Has container, default is True\n", "# --usr_domain USR_DOMAIN\n", "# Param for usp. User domain file, default is None\n", "# --usr_forcing USR_FORCING\n", "# Param for usp. User forcing file, default is None\n", "# --usr_surfdata USR_SURFDATA\n", "# Param for usp. User surface data file, default is None\n", "# --output_prefix OUTPUT_PREFIX\n", "# Param for usp. Output file name prefix, default is _clm.nc, is used to generate the output file of pyclmuapp\n", "# --case_name CASE_NAME\n", "# Param for usp. Case name, default is usp_case\n", "# --run_startdate RUN_STARTDATE\n", "# Param for usp. Start date, default is None\n", "# --stop_option STOP_OPTION\n", "# Param for usp. Stop option, default is ndays, can be nyears, nmonths, ndays\n", "# --stop_n STOP_N Param for usp. Stop number, default is 1\n", "# --run_type RUN_TYPE Param for usp. Run type, default is coldstart, can be branch\n", "# --run_refcase RUN_REFCASE\n", "# Param for usp. Reference case, default is None\n", "# --run_refdate RUN_REFDATE\n", "# Param for usp. Reference date, default is None\n", "# --iflog IFLOG Param for usp and script. If log, default is True\n", "# --logfile LOGFILE Param for usp and script. Log file, default is pyclmuapp.log\n", "# --hist_type HIST_TYPE\n", "# Param for usp. ouput type. Can be GRID, LAND, COLS, default is GRID\n", "# --hist_nhtfrq HIST_NHTFRQ\n", "# Param for usp. History file frequency, default is 1 (ouput each time step)\n", "# --hist_mfilt HIST_MFILT\n", "# Param for usp. each history file will include mfilt time steps, default is 1000000000\n", "# --clean CLEAN Param for usp. Clean, default is False\n", "# --surf_var SURF_VAR Param for usp. Surface variable, default is None. Can be one/some (use ','(withou space to seperate each)) of\n", "# 'CANYON_HWR', 'HT_ROOF','THICK_ROOF','THICK_WALL','WTLUNIT_ROOF','WTROAD_PERV','WIND_HGT_CANYON','NLEV_IMPROAD','TK_ROOF',\n", "# 'TK_WALL','TK_IMPROAD','CV_ROOF','CV_WALL','CV_IMPROAD','EM_IMPROAD','EM_PERROAD','EM_ROOF','EM_WALL','ALB_IMPROAD_DIR','A\n", "# LB_IMPROAD_DIF','ALB_PERROAD_DIR','ALB_PERROAD_DIF','ALB_ROOF_DIR','ALB_ROOF_DIF','ALB_WALL_DIR','ALB_WALL_DIF','T_BUILDIN\n", "# G_MIN'\n", "# --surf_action SURF_ACTION\n", "# Param for usp. Surface action, default is None. The number is same as surf_var with \",\" seperated (not \", \")\n", "# --forcing_var FORCING_VAR\n", "# 'Param for usp. Forcing variable, default is None. Can be one/some (use ','(withou space to seperate each)) of\n", "# 'Prectmms','Wind','LWdown','PSurf','Qair','Tair','SWdown'\n", "# --forcing_action FORCING_ACTION\n", "# Param for usp. Forcing action, default is None. The number is same as forcing_var with \",\" seperated (not \", \")\n", "# --script SCRIPT Param for script. Script file in container, default is None\n", "# --urbsurf URBSURF Param for get_surfdata. Urban surface data file, default is None. Here to download the urban surface data file:\n", "# https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata/lnd/clm2/rawdata/mksrf_urban_0.05x0.05_simyr2000.c120601.nc\n", "# --soildata SOILDATA Param for get_surfdata. Soil data file, default is None. Here to download the soil data file: https://svn-ccsm-\n", "# inputdata.cgd.ucar.edu/trunk/inputdata/lnd/clm2/rawdata/mksrf_soitex.10level.c010119.nc\n", "# --pct_urban PCT_URBAN\n", "# Param for get_surfdata. Percentage of urban land use in each density class, sum should be 100, default is [0,0,100.0]\n", "# --lat LAT Param for get_surfdata and get_forcing. Latitude of the urban area, default is None\n", "# --lon LON Param for get_surfdata and get_forcing. Longitude of the urban area, default is None\n", "# --outputname OUTPUTNAME\n", "# Param for get_surfdata. Output file name, default is surfdata.nc\n", "# --zbot ZBOT Param for get_forcing. zbot, default is 30 meters\n", "# --start_year START_YEAR\n", "# Param for get_forcing. Start year, default is 2012\n", "# --end_year END_YEAR Param for get_forcing. End year, default is 2012\n", "# --start_month START_MONTH\n", "# Param for get_forcing. Start month, default is 1\n", "# --end_month END_MONTH\n", "# Param for get_forcing. End month, default is 12\n", "#\n", "#For any question, contact yjj1997@live.cn" ] } ], "source": [ "! cat pyclmuapp_hpc " ] } ], "metadata": { "kernelspec": { "display_name": "pyclmuapp", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.19" } }, "nbformat": 4, "nbformat_minor": 2 }