Part 10. Checking the output
70.
Once the job has been completed, you should see the following items in the day directory:
(env1) [ec2-user@ip-10-0-5-43 20181105] $ ls
181105.edf pipe-slurm.ip-xxx.2.err sessioneye
mountains pipe-slurm.ip-xxx.2.out
P11_5.edf session01
You should see the following items in the session01 directory:
(env1) [ec2-user@ip-10-0-5-43 20181105] $ ls session01
181105_Block1.nev binData.hdf slist.txt
181105_Block1.ns5 eyelink_24d5.hkl unity_71bf.hkl
array01 logs.txt unityfile_eyelink.csv
array02 missingData.csv VirtualMazeBatchLog.txt
array03 RawData_T1-400
array04 rplparallel_d41d.hkl
In total, we expect the following .hkl files to be created:
session01: rplparallel, unity, eyelink
8 channel directories: rplraw, rpllfp, rplhighpass
sessioneye: rplparallel, eyelink
8 channel directories: rplraw, rpllfp, rplhighpass
which adds up to 53. There will also be some spiketrain .hkl files, but the number returned for each channel is not predictable.
So we can do the following to find all the .hkl files:
(env1) [ec2-user@ip-10-0-5-43 20181105] $ find . -name "*.hkl"
The find command is a very useful function that can help find files using different parameters. In this case, we are asking it to start from the current directory (the . in the second argument) and look for files with names ending in .hkl.
In order to leave out the spiketrain .hkl files, we can send the output of the above command to the grep function using the pipe feature | in the shell. The grep function looks for lines containing matching strings (specified by the -e argument) inside a file or from the output of another function sent to it via a pipe. However, when we specify the -v argument, it will instead look for lines that do not contain the matching strings.
So in the command below, by specifying -v -e spiketrain -e mountains, the grep function will only return lines that do not contain “spiketrain” nor “mountains”, which will allow us to select only the files returned by the find function that do not contain spiketrain or mountains in their names:
(env1) [ec2-user@ip-10-0-5-43 20181105] $ find . -name "*.hkl" | grep -v -e spiketrain -e mountains
We can count the number of files by again piping the output above to the wc command to make sure they were all created properly:
(env1) [ec2-user@ip-10-0-5-43 20181105] $ find . -name "*.hkl" | grep -v -e spiketrain -e mountains | wc -l
53
We can also do the following to make sure that the file sizes look correct:
(env1) [ec2-user@ip-10-0-5-43 20181105] $ find . -name "*.hkl" | grep -v -e spiketrain -e mountains | xargs ls -hl
-rw-rw-r-- 1 ec2-user ec2-user 129M Aug 10 17:20
./session01/eyelink_24d5.hkl
-rw-rw-r-- 1 ec2-user ec2-user 61K Aug 10 13:55
./session01/rplparallel_d41d.hkl
-rw-rw-r-- 1 ec2-user ec2-user 12M Aug 10 17:20
unity_71bf.hkl
-rw-rw-r-- 1 ec2-user ec2-user 630M Aug 11 02:28
./session01/array01/channel009/rplhighpass_b59f.hkl
-rw-rw-r-- 1 ec2-user ec2-user 22M Aug 11 02:27
./session01/array01/channel009/rpllfp_6eca.hkl
-rw-rw-r-- 1 ec2-user ec2-user 630M Aug 11 02:18
./session01/array01/channel009/rplraw_d41d.hkl
Task
Include a screenshot of your Terminal window with the file sizes above in your lab report. Make sure you increase the size of your Terminal window so that the size of all 53 files can be captured in the screenshot.
The xargs function used in the command above takes the output of the grep function, and appends them as arguments to the end of the call to the ls function. For instance, if the grep function returns:
$ find . -name "*.hkl" | grep -v -e spiketrain -e mountains
./session01/eyelink_24d5.hkl
./session01/rplparallel_d41d.hkl
Using xargs will be the equivalent of:
$ ls -hl ./session01/eyelink_24d5.hkl ./session01/rplparallel_d41d.hkl
For session01, the rplraw and rplhighpass files are typically over 600 MB, while the rpllfp files are around 20 MB. The unity files are typically around 10 MB, and the eyelink files are typically over 100 MB. The files in the sessioneye directory are typically quite a bit smaller.
In order to check the output of the spike sorting, we would expect one of these files to be created for each channel:
(env1) [ec2-user@ip-10-0-5-43 20181105] $ find mountains -name "firings.mda" | wc -l
8
Task
Include a screenshot of your terminal window with the output above in your lab report.
71.
You can check the time taken to complete the job by looking for the printouts of time in the output file for the job:
(env1) [ec2-user@ip-10-0-5-43 20181105] $ tail pipe-slurm*.out
Task
Include the output of the command above in your lab report, and convert the time taken for the job to hours, minutes, and seconds so it is easy to understand. Extrapolate from the time taken for 8 channels to estimate how long it will take to process all 110 channels.