Log in to a password protected FTP

Some password protected FTP sites can be visited by anonymous users  but if you don’t input user name and password the protected contents won’t show up.

In web browser, use

ftp://(user name):(password)@(ftp_site_address)

In shell,

wget --user=XXX --password='XXX' ftp://ftp.XXX.edu/*
Advertisements

More NetCDF commands

1. NetCDF file compression

ncks -L 2 in.nc out.nc    # 2 is the compression ratio, could be 2-9

2. Split file by dimension (e.g. time, lat, lon)

ncks -d time,0,30 in.nc out.nc     
ncks -F -d time,1,31 in.nc out.nc  (equivalent)

3. Merge time using NCO

ncks --mk_rec_dmn time in.nc out.nc
ncrcat *.nc out.nc

Damn BASH

Spent hours debugging a piece of simple script in which I pass more than 9 arguments from outside. Everything’s fine for arguments 1-9 but bash couldn’t recognize argument 10. Turns out you need to add these damn curly braces on the numbers. Otherwise it will just treat it as $1

Something like this:

#!/bin/bash
 PWD=$1
 BI=$2
 DS=$3
 WS=$4
 EXP=$5
 D2=$6
 D3=$7
 DSMAX=$8
 Inimois2=$9
 Inimois3=${10}

Matlab meets Unix

1. running Matlab .m script in terminal without showing desktop (for batch running).

> know where the Matlab is installed. For example in my Mac the Matlab is installed in

/Applications/MATLAB_R2012b.app/bin

> call Matlab from anywhere that stores the script (test_script.m) you want to

run/Applications/MATLAB_R2012b.app/bin/matlab -nodesktop -nojvm -nosplash -r "test_script(arguments)"

2. compile Matlab .m scripts to executables.

> in Matlab GUI use Apps->Matlab complier
> in terminal use mcc -m test_script.m
to generate an executable file (test_script) and a shell script (run_test_script.sh) which is used to run the executable.
Then in terminal:

./run_test_script.sh /PATH/TO/MATLAB/ arguments

Note that the arguments will be passed to Matlab as strings. That means if they are numbers you need to add “str2num” in the code to convert them from strings to the numbers.

Useful Hyak Commends

Hyak WIKI:

https://sig.washington.edu/itsigs/WIKI_for_Hyak_users

Show jobs in queues:

showq                 #show all jobs
showq -w qos=hpc      #show jobs based on group (hpc)
showq -w class=bf     #show jobs on backfill list

Managing jobs:

mjobctl -c <job_id>   #cancel a job
showbf -q hpc         #show what resources are available for a given group (hpc)
checkjob <job_id>     #check the job status

Serial job scripts:
If running single thread job like VIC, it is required to use GNU parallel to bundle the job together and then submit to the node.

> first create an ascii file (jobargs) containing the input arguments for the program (e.g. VIC)

> then in the script add

cat /path/to/jobargs | parallel -j $HYAK_SLOTS --joblog paralleljobs.log --resume /path/to/executable/program {}

to activate parallel to bundle the job based on the jobargs