Working with files¶
If you have already initiated a session, authentication, and have a storage system setup then you start working with files.
Listing files and directories¶
Before we even get started, we should know what it is we are going to be
working with.
To see any files or directories in your remote storage system, let’s say this
is called data-tacc-user
.
>>> agave.files_list("/data-tacc-user")
./ .slurm/ apps/ benchmarks/
bigfile cptests/ users/ myotherfile
old-sd2e-apps/ rpmbuild/ sd2e-apps/ sd2e-data/
singularity-test/ somefile SP1-copy.fq SP1.fq
tests/
For more details, such as permissions and size:
>>> agave.files_list("data-tacc-user/", long_format=True)
drwx 4096 Oct 25 14:47 ./
drwx 4096 May 25 16:13 .slurm/
drwx 4096 Jun 5 10:14 apps/
drwx 4096 Jun 9 18:39 benchmarks/
-rw- 1073741824 Oct 25 14:48 bigfile
drwx 4096 Jul 13 09:17 cptests/
drwx 4096 Aug 1 14:26 users/
-rw- 24 Oct 25 14:42 myotherfile
drwx 4096 Jul 10 11:27 old-sd2e-apps/
drwx 4096 Jun 29 10:40 rpmbuild/
drwx 4096 Jul 13 15:41 sd2e-apps/
drwx 4096 Jul 7 20:43 sd2e-data/
drwx 4096 Jul 9 16:50 singularity-test/
-rw- 24 Oct 25 14:20 somefile
-rw- 22471 Aug 22 14:40 SP1-copy.fq
-rw- 22471 Jul 7 20:42 SP1.fq
drwx 4096 Aug 24 13:44 tests/
Download a file¶
Let’s say you have a file named SP1.fq
on your remote storage system,
data-tacc-user
.
You can download the file, SP1.fq
, to your host and specify the location of
where you want to store the file (i.e., /opt/data/SP1.fq
) as such:
>>> agave.files_download("data-tacc-user/SP1.fq", "/opt/data/SP1.fq")
Upload a file¶
Similarly, if you want to upload a file, data.ext
, and you
want to save it on your stoorage system named, data-tacc-user
under
the name cool_data.bin
.
>>> agave.files_upload("./data.ext", "data-tacc-user/cool.ext")
Make a copy of a file on a remote system¶
So now, you have a cool.ext
file on your remote storage
system, data-tacc-user
. Let’s make a copy of it!
>>> agave.files_copy("data-tacc-user/cool.ext", "data-tacc-user/copy.ext")
Move files¶
To move files around on a remote storage system, you can do so as follows:
>>> agave.files_move("data-tacc-user/file.ext", "data-tacc-user/dir/file.ext")
Delete a file¶
On the other hand, if there is a file or directory that you want to get rid off:
>>> agave.files_delete("data-tacc-user/somefile-or-directory")
Import files from ther systems¶
It may be useful to import data from other storage systems, e.g. from the
community data space to your private data space.
The files_import
method can be used for that purpose.
agave.files_import("agave://data-community/test.txt", "system-id/")
Please also note that even though you are able to import files from other Agave
storage systems, you may not always need to import those files.
Also, note that the source, the first argument, must be an agave compliant uri
by prefixing the system if and path convination with the string agave://
.
Making a directory¶
AgavePy
also provides mkdir
-like functionality.
To create a directory on a remote storage system:
>>> agave.files_mkdir("data-tacc-user/new_dir")