pyclmuapp package

Submodules

pyclmuapp.clmu module

class pyclmuapp.clmu.cesm_run(CASESCRIPT_local, CASEROOT_local, DOUT_S_ROOT, caseconfig: str | dict | DataFrame | Series)

Bases: object

This class is used to create CESM2/CTSM case. Run the CESM2/CTSM case should be done by running the generated shell script

using terminal or python subprocess or container (docker or singularity with CESM2/CTSM installed)

Parameters:

  • CASESCRIPT_local (str) – The path of the case script in the local machine.

  • CASEROOT_local (str) – The path of the case root in the local machine.

  • DOUT_S_ROOT (str) – The path of the output root in the local machine.

  • caseconfig (Union[str, dict, pd.DataFrame, pd.Series]) – The configuration of the case to be built.

CASESCRIPT

The path of the case script.

Type:

str

CASEROOT

The path of the case root.

Type:

str

DOUT_S_ROOT

The path of the output root.

Type:

str

case_lat

The latitude of the case.

Type:

float

case_lon

The longitude of the case.

Type:

float

case_name

The name of the case.

Type:

str

json_file_path

The path of the JSON file.

Type:

str

config

The configuration data.

Type:

dict

Methods

create_case(scriptpath)

Create the case scripts for single point modeling.

modify_case_config(scriptpaht)

Modify the case config file.

modify_forcing(var, action, forcing_location)

Modify the forcing file.

modify_surf(var, action[, numurbl])

Modify the surface data file.

nc_view([ds])

View the netcdf file.

read_json_config()

Read the JSON config file.

recover_forcing(forcing_location, ...)

Recover the forcing file.

reset_case([password])

Reset the case.
create_case(scriptpath) → str

Create the case scripts for single point modeling.

Parameters:

filepath (str) – File path of the config file.

Returns:

The modified script file.

Return type:

script (str)

modify_case_config(scriptpaht) → str

Modify the case config file.

Parameters:

scriptpath (str) – The path of the script file.

Returns:

The modified script file.

Return type:

script (str)

modify_forcing(var, action, forcing_location) → None

Modify the forcing file.

Parameters:

  • var (str) – The variable to be modified.

  • action (float, np.ndarray, dict) – The action to be taken.

  • forcing_location (str) – The location of the forcing file. - if action is a dict, the key is the variable name, the value is the action. - if action is a float, the action will be added to the variable. - if action is a np.ndarray, the variable will be replaced by the action.

Returns:

None

modify_surf(var, action, numurbl=None) → str

Modify the surface data file.

Parameters:

  • var (str) – The variable to be modified.

  • action (float, np.ndarray, dict) – The action to be taken. - if action is a dict, the key is the variable name, the value is the action. - if action is a float, the action will be added to the variable. - if action is a np.ndarray, the variable will be replaced by the action.

  • numurbl (int, optional) – The number of urban land units. Defaults to None. None means the action will be implemented to all the urban land units. numurbl 0 –> TBD urban, numurbl 1 –> HD urban, numurbl 2 –> MD urban

Returns:

The modified surface data file path.

Return type:

str

nc_view(ds: str = 'None') → Dataset

View the netcdf file.

Parameters:

  • xarray.Dataset – The xarray dataset.

  • ds (str, optional) – The path of the netcdf file. Defaults to “None”.

Returns:

The xarray dataset.

Return type:

xarray.Dataset

read_json_config() → dict

Read the JSON config file.

Returns:

The configuration data.

Return type:

config (dict)

recover_forcing(forcing_location, backup_location) → None

Recover the forcing file.

Parameters:

  • forcing_location (str) – The location of the forcing file.

  • backup_location (str) – The location of the backup file.

Returns:

None

reset_case(password=None) → CompletedProcess

Reset the case. (delete the case folders and files)

Parameters:

password (str, optional) – The password of the server. Defaults to None.

Returns:

The result of the command.

Return type:

subprocess.CompletedProcess

pyclmuapp.clmu.copy_file_if_not_exists(source_path, destination_path)
pyclmuapp.clmu.copy_file_if_not_exists2(source_path, destination_path, lon, lat, res='1.25*0.9') → None

Copy the file if the destination file does not exist.

Parameters:

  • source_path (str) – The source file path.

  • destination_path (str) – The destination file path.

  • lon (str) – The longitude of the point.

  • lat (str) – The latitude of the point.

  • res (str) – The resolution of the file. Defaults to “1.25*0.9”.

pyclmuapp.clmu.get_forcing(start_year, end_year, start_month, end_month, lat, lon, zbot, source='cds')

get the forcing data from the era5 dataset

Parameters:

  • start_year (_type_) – the start year

  • end_year (_type_) – the end year

  • start_month (_type_) – the start month

  • end_month (_type_) – the end month

  • lat (_type_) – latitude of interest point

  • lon (_type_) – longitude of interest point

  • zbot (_type_) – the bottom level height

  • source (_type_) – the source of the data, cds or arco-era5

Returns:

the forcing dataset

Return type:

_type_

pyclmuapp.clmu.get_soil_params(ds: Dataset | DataArray | str, lat: float = 51.508965, lon: float = -0.118092) → tuple

Get the soil parameters.

Parameters:

  • ds (_type_) – the soil dataset

  • lat (_type_) – latitude of interest point

  • lon (_type_) – longitude of interest point

Returns:

sand and clay content from the soil dataset

Return type:

tuple

pyclmuapp.clmu.get_urban_params(urban_ds: Dataset | str, soil_ds: Dataset | str, lat: float, lon: float, template: Dataset | str = '/home/runner/work/pyclmuapp/pyclmuapp/pyclmuapp/usp/surfdata.nc', PTC_URBAN: list = [0, 0, 100], outputname: str = 'surfdata.nc') → Dataset

Get the urban parameters.

Parameters:

  • urban_ds (_type_) – the urban dataset

  • soil_ds (_type_) – the soil dataset

  • template (_type_) – the template dataset

  • lat (_type_) – latitude of interest point

  • lon (_type_) – longitude of interest point

  • PTC_URBAN (list, optional) – The percentage of urban. Defaults to [0,0,100]. 0. TBD urban, 1. HD urban, 2. MD urban

  • outputname (_type_, optional) – the output file name. Defaults to “surfdata.nc”.

Returns:

the modified template dataset

Return type:

_type_

pyclmuapp.clmu.getconfig(caseconfig: str | dict | DataFrame | Series = 'None') → dict

Read the configuration of the case to be built.

Parameters:

caseconfig (Union[str, dict, pd.DataFrame, pd.Series]) – The configuration of the case to be built.

Returns:

The configuration data.

Return type:

dict

pyclmuapp.clmu.now_time()

Get the current time

pyclmuapp.clmu.run_command(command, password='None', logname='None', iflog=True) → None

Run the command. There are two ways to run the command, with or without password. The loges will be saved in the log_ppo.txt file.

Parameters:

  • command (str) – The command to be run.

  • password (str, optional) – The password of the server.

  • logname (str, optional) – The name of the log file. Defaults to “cmdlog.txt”.

  • iflog (bool, optional) – If log is needed. Defaults to True.

Returns:

None

pyclmuapp.container module

class pyclmuapp.container.clumapp(pwd='/home/runner/work/pyclmuapp/pyclmuapp/docs/workdir', input_path: str = 'inputfolder', output_path: str = 'outputfolder', log_path: str = 'logfolder', scripts_path: str = 'scriptsfolder', container_type: str = 'docker')

Bases: object

The class for running CLMU-App.

Parameters:

  • pwd (str) – The path to the working directory. The default is “wokdir”.

  • input_path (str) – The path to the folder for the input of CLMU-App.

  • output_path (str) – The path to the folder for the output of CLMU-App.

  • log_path (str) – The path to the folder for the log of CLMU-App.

  • scripts_path (str) – The path to the folder for the scripts of CLMU-App.

  • container_type (str) – The type of the container for CLMU-App. The supported types are: - docker: The docker container. - singularity: The singularity container.

- input_path

The path to the folder for the input of CLMU-App.

Type:

str

- output_path

The path to the folder for the output of CLMU-App.

Type:

str

- log_path

The path to the folder for the log of CLMU-App.

Type:

str

- image_name

The name of the image for CLMU-App.

Type:

str

- container_name

The name of the container for CLMU-App.

Type:

str

- caseconfig

The configuration of the case to be built.

Type:

dict

- fsurdat_path

The path to the surface input file.

Type:

str

- save_name

The name of the output file.

Type:

str

- caseconfig

The configuration of the case to be built.

Type:

dict

- clmu_run

The object for running CLMU-App.

Type:

cesm_run

Methods

case_clean([case_name])

Clean the case artifacts.

case_scripts([mode])

Copy the scripts to the input folder.

create_folder(folder_path)

Create folders for the scripts ,input, output and log of CLMU-App if it does not exist.

docker([cmd, iflog, password, cmdlogfile, ...])

Run the docker command.
case_clean(case_name: str | None = None) → None

Clean the case artifacts. :param case_name: The name of the case to be cleaned. :type case_name: str

case_scripts(mode: str = 'usp') → None

Copy the scripts to the input folder.

Parameters:

mode (str) – The mode of the scripts. The supported modes are:

create_folder(folder_path) → str

Create folders for the scripts ,input, output and log of CLMU-App if it does not exist.

Parameters:

folder_path (str) – The path to the folder to be created.

Returns:

The path to the folder created.

Return type:

folder_path (str)

docker(cmd: str = 'run', iflog: bool = True, password: str = 'None', cmdlogfile: str | None = None, dockersript: str = 'docker.sh') → None

Run the docker command.

Parameters:

  • cmd (str) – The docker command to be run. The supported commands are: - pull: Pull the docker image. - chmod: Change the mode of a file. - run: create a docker container. - exec: Execute the docker command.

  • iflog (bool, optional) – If log is needed. The default is True.

  • password (str, optional) – The password for the sudo command. The default is “None”.

  • cmdlogfile (str, optional) – The name of the log file. The default is “dockercmd.log”.

  • dockersript (str, optional) – The name of the docker script. The default is “docker.sh”.

pyclmuapp.getcity module

pyclmuapp.getcity.adjust_longitude(longitude)
pyclmuapp.getcity.calculate_polygon_area(latitude_longitude_coords)
pyclmuapp.getcity.closest(data, v)

find the nearest urban grid cell in CESM using haversine distance reference: https://stackoverflow.com/questions/41336756/find-the-closest-latitude-and-longitude

Parameters:
datadict

a dict of cities’ lat and lon, from get_mask_cities

vdict

a dict of a city’s lat and lon that we are interested in, e.g., {‘lat’: 40.1164, ‘lon’: -88.2434}

Returns:
dict

lat and lon of the nearest grid cell in Earth System Model

pyclmuapp.getcity.get_mask_cities(df_mask)

This is a function for getting urban mask and a list of cities’ lat and lon

Parameters:
maskxarray.DataArray

urban mask

Returns:
dict

a dict of cities’ lat and lon

pyclmuapp.getcity.haversine_dist(a, b)

This is a function for getting the haversine distance

Parameters:
a: list

[lat, lon] e.g., (45.7597, 4.8422)

b: list

[lat, lon] e.g., (48.8567, 2.3508)

er:

Earth radius, default is 6371.0088 km

Returns:
haversine_dist

the haversine distance between a and b

pyclmuapp.pts module

class pyclmuapp.pts.pts_clmu(pwd='/home/runner/work/pyclmuapp/pyclmuapp/docs/workdir', input_path: str = 'inputfolder', output_path: str = 'outputfolder', log_path: str = 'logfolder', scripts_path: str = 'scriptsfolder', container_type: str = 'docker')

Bases: clumapp

Methods

case_clean([case_name])

Clean the case artifacts.

case_scripts([mode])

Copy the scripts to the input folder.

create_folder(folder_path)

Create folders for the scripts ,input, output and log of CLMU-App if it does not exist.

docker([cmd, iflog, password, cmdlogfile, ...])

Run the docker command.

modify_forcing(var, action[, ...])

Modify the forcing input file.

modify_surf(var, action[, numurbl, caseconfig])

Modify the surface input file.

nc_view([ds])

View the netcdf file.

run([caseconfig, ouptname, iflog, password, ...])

Run workflow for a case of CLMU-App.
modify_forcing(var: str, action: str, forcing_location: str = 'forcing location', caseconfig: str | dict | DataFrame | Series = 'None') → dict

Modify the forcing input file.

Parameters:

  • var (str) – The variable to be modified.

  • action (str) – The action to be taken.

  • param_location (str) – The location of the parameter to be modified.

  • caseconfig (Union[str, dict, pd.DataFrame, pd.Series]) – The configuration of the case to be built.

modify_surf(var: str, action: dict | float | ndarray, numurbl: int | None = None, caseconfig: str | dict | DataFrame | Series = 'None') → dict

Modify the surface input file.

Parameters:

  • var (str) – The variable to be modified.

  • action (Union[dict, float, np.ndarray]) – The action to be taken. - if action is a float, the action will be added to the variable. - if action is a dict, the key is the variable name, the value is the action. - if action is a np.ndarray, the variable will be replaced by the action.

  • numurbl (int, optional) – The number of urban land units. Defaults to None. The type of urban, 0 is TBD, 1 is HD, 2 is MD.

  • caseconfig (Union[str, dict, pd.DataFrame, pd.Series]) – The configuration of the case to be built.

nc_view(ds: str = 'None')

View the netcdf file. The netcdf file should be in the DOUT_S_ROOT folder.

Parameters:

ds (xarray.Dataset) – The xarray dataset.

run(caseconfig: str | dict | DataFrame | Series = 'None', ouptname: str = '_clm.nc', iflog: bool = True, password: str = 'None', cmdlogfile: str = 'dockercmd.log') → list

Run workflow for a case of CLMU-App.

Parameters:

  • caseconfig (Union[str, dict, pd.DataFrame, pd.Series], optional) – The configuration of the case to be built. The default is “None”. “None” means the default configuration is used.

  • ouptname (str, optional) – The name of the output file. The default is “_clm.nc”.

  • iflog (bool, optional) – If log is needed. The default is True.

  • password (str, optional) – The password for the sudo command. The default is “None”.

  • cmdlogfile (str, optional) – The name of the log file. The default is “dockercmd.log”.

Returns:

The list of the saved output files names.

Return type:

savename_list (list[str])

pyclmuapp.usp module

class pyclmuapp.usp.usp_clmu(pwd='/home/runner/work/pyclmuapp/pyclmuapp/docs/workdir', input_path: str = 'inputfolder', output_path: str = 'outputfolder', log_path: str = 'logfolder', scripts_path: str = 'scriptsfolder', container_type: str = 'docker')

Bases: clumapp

Methods

case_clean([case_name])

Clean the case artifacts.

case_scripts([mode])

Copy the scripts to the input folder.

check_domain([usr_domain, domain_name])

The function to get the domain data for the urban surface parameters.

check_forcing([usr_forcing])

The function to get the forcing data for the urban surface parameters.

check_surf([usr_surfdata, surfata_name, ...])

The function to get the surface data for the urban surface parameters.

clean_usp()

Clean the usp folder.

create_folder(folder_path)

Create folders for the scripts ,input, output and log of CLMU-App if it does not exist.

docker([cmd, iflog, password, cmdlogfile, ...])

Run the docker command.

modify_forcing([usr_forcing, action, mode, ...])

The function to revise the forcing data for the urban surface parameters.

modify_surf([usr_surfdata, action, mode, ...])

The function to revise the surface data for the urban surface parameters.

nc_view([ds])

View the netcdf file.

run([output_prefix, case_name, ...])

The function to run the CLMU-App for the urban single point.
check_domain(usr_domain: str | None = None, domain_name: str = 'domain.nc') → None

The function to get the domain data for the urban surface parameters.

Parameters:

  • usr_domain (str) – The path to the user-defined domain data file. The default is None, which means using the default domain data.

  • domain_name (str) – The name of the domain data file. The default is “domain.nc”.

Returns:

The dictionary of the domain data for the urban surface parameters.

Return type:

dict

check_forcing(usr_forcing: str | None = None) → None

The function to get the forcing data for the urban surface parameters.

Parameters:

usr_forcing (str) – The path to the user-defined forcing data file. The default is None.

check_surf(usr_surfdata: str | None = None, surfata_name: str = 'surfdata.nc', urban_type: int = 2) → dict

The function to get the surface data for the urban surface parameters.

Parameters:

  • usr_surfdata (str) – The path to the user-defined surface data file. The default is None.

  • surfata_name (str) – The name of the surface data file. The default is “surfdata.nc”.

  • urban_type (int) – The type of the urban surface. The default is 2. 0 is for TBD urban, 1 is for HD urban, and 2 is for MD urban.

Returns:

The dictionary of the surface data for the urban surface parameters.

Return type:

dict

clean_usp() → None

Clean the usp folder.

modify_forcing(usr_forcing: str | None = None, action: dict | None = None, mode: str = 'add', forcing_name: str = 'forcing.nc') → None

The function to revise the forcing data for the urban surface parameters.

Parameters:

  • usr_forcing (str) – The path to the user-defined forcing data file. The default is None.

  • action (dict) – The dictionary of the revised forcing data for the urban surface parameters. The default is None, which means no action.

  • mode (str) – The mode for the revision. The default is “add”.

  • forcing_name (str) – The name of the revised forcing data file. The default is “forcing.nc”.

modify_surf(usr_surfdata: str | None = None, action: dict | None = None, mode: str = 'replace', surfata_name: str = 'surfdata.nc', urban_type: int = 2) → dict

The function to revise the surface data for the urban surface parameters.

Parameters:

  • usr_surfdata (str) – The path to the user-defined surface data file. The default is None.

  • action (dict) – The dictionary of the revised surface data for the urban surface parameters. The default is None, which means no action.

  • mode (str) – The mode for the revision. The default is “replace”.

  • surfata_name (str) – The name of the revised surface data file. The default is “surfdata.nc”.

  • urban_type (int) – The type of the urban surface. The default is 2. 0 is for TBD urban, 1 is for HD urban, and 2 is for MD urban.

Returns:

The dictionary of the revised surface data for the urban surface parameters.

Return type:

dict

nc_view(ds: str = 'None') → Dataset

View the netcdf file. The netcdf file should be in the DOUT_S_ROOT folder.

Parameters:

ds (xarray.Dataset) – The xarray dataset.

Returns:

The xarray dataset.

Return type:

xarray.Dataset

run(output_prefix: str = '_clm.nc', case_name: str = 'usp_case', RUN_STARTDATE: str = '2012-08-08', START_TOD: str = '00000', STOP_OPTION: str = 'ndays', STOP_N: str = '10', ATM_DOM: str | None = None, SURF: str | None = None, FORCING: str | None = None, RUN_TYPE: str = 'coldstart', RUN_REFCASE: str = 'None', RUN_REFDATE: str = 'None', RUN_REFTOD: str = '00000', password: str = 'None', iflog: bool = True, logfile: str | None = None, hist_type: str = 'GRID', hist_nhtfrq: int = 1, hist_mfilt: int = 1000000000, urban_hac: str = 'ON_WASTEHEAT', crun_type: str = 'usp') → list

The function to run the CLMU-App for the urban single point.

Parameters:

  • output_prefix (str) – The output file name. The default is “_clm.nc”. if the output_prefix is none, the output file name will not be changed.

  • case_name (str) – The case name. The default is “usp_case”.

  • RUN_STARTDATE (str) – The start date of the run. The default is “2012-08-08”.

  • START_TOD (str) – The start time of the day. The default is “00000”.

  • STOP_OPTION (str) – The stop option. The default is “ndays”.

  • STOP_N (str) – The number of days to run. The default is “10”.

  • ATM_DOM (str) – The path to the domain data file. Will use the domain data provided by the user. The default is None.

  • SURF (str) – The path to the surface data file. Will use the surface data provided by the user. The default is None.

  • FORCING (str) – The path to the forcing data file. Will use the forcing data provided by the user. The default is None.

  • RUN_TYPE (str) – The type of the run. The default is “coldstart”. The other option is “branch”.

  • RUN_REFCASE (str) – The reference case. The default is “None”. Need to be provided when the RUN_TYPE is “branch”.

  • RUN_REFDATE (str) – The reference date. The default is “None”. Need to be provided when the RUN_TYPE is “branch”.

  • RUN_REFTOD (str) – The reference time of the day. The default is “00000”. Need to be provided when the RUN_TYPE is “branch”.

  • password (str) – The password for the docker. The default is “None”. Need to be provided when server is needed.

  • iflog (bool) – The flag to log the output. The default is True.

  • logfile (str) – The log file name. The default is pwd+”log.log”.

  • urban_hac (str) – The flag to turn on the urban HAC. The default is “ON_WASTEHEAT”. valid_values=”OFF”,”ON”,”ON_WASTEHEAT”.

  • crun_type (str) – The type of the run. The default is “usp”.

Returns:

The list of the output files names.

Return type:

list

Module contents