GTDB Forum

How to run interrupted job from checkpoint instead of starting over?

The pplacer part finally finished but the FastANI was still running. If I rerun this command in the same directory, will that part of the pipeline pick up again?

Here is my command and log:

(base) -bash-4.2$ conda activate gtdbtk_env
(gtdbtk_env) -bash-4.2$ gtdbtk classify_wf --genome_dir mags_output/ --out_dir gtdbtk_output --cpus 1 -x fa
[2021-06-08 08:02:26] INFO: GTDB-Tk v1.5.0
[2021-06-08 08:02:26] INFO: gtdbtk classify_wf --genome_dir mags_output/ --out_dir gtdbtk_output --cpus 1 -x fa
[2021-06-08 08:02:26] INFO: Using GTDB-Tk reference data version r202: /usr/local/scratch/METAGENOMICS/jespinoz/db/gtdbtk_db/release202/
[2021-06-08 08:02:28] INFO: Identifying markers in 529 genomes with 1 threads.
[2021-06-08 08:02:28] TASK: Running Prodigal V2.6.3 to identify genes.
[2021-06-08 08:03:46] INFO: Completed 529 genomes in 1.30 minutes (407.14 genomes/minute).
[2021-06-08 08:03:46] WARNING: Prodigal skipped 529 genomes due to pre-existing data, see warnings.log
Traceback (most recent call last):
  File "/usr/local/devel/ANNOTATION/jespinoz/anaconda3/envs/gtdbtk_env/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/local/devel/ANNOTATION/jespinoz/anaconda3/envs/gtdbtk_env/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/devel/ANNOTATION/jespinoz/anaconda3/envs/gtdbtk_env/lib/python3.8/multiprocessing/managers.py", line 616, in _run_server
    server.serve_forever()
  File "/usr/local/devel/ANNOTATION/jespinoz/anaconda3/envs/gtdbtk_env/lib/python3.8/multiprocessing/managers.py", line 182, in serve_forever
    sys.exit(0)
SystemExit: 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/devel/ANNOTATION/jespinoz/anaconda3/envs/gtdbtk_env/lib/python3.8/multiprocessing/util.py", line 300, in _run_finalizers
    finalizer()
  File "/usr/local/devel/ANNOTATION/jespinoz/anaconda3/envs/gtdbtk_env/lib/python3.8/multiprocessing/util.py", line 224, in __call__
    res = self._callback(*self._args, **self._kwargs)
  File "/usr/local/devel/ANNOTATION/jespinoz/anaconda3/envs/gtdbtk_env/lib/python3.8/multiprocessing/util.py", line 133, in _remove_temp_dir
    rmtree(tempdir)
  File "/usr/local/devel/ANNOTATION/jespinoz/anaconda3/envs/gtdbtk_env/lib/python3.8/shutil.py", line 718, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/usr/local/devel/ANNOTATION/jespinoz/anaconda3/envs/gtdbtk_env/lib/python3.8/shutil.py", line 675, in _rmtree_safe_fd
    onerror(os.unlink, fullname, sys.exc_info())
  File "/usr/local/devel/ANNOTATION/jespinoz/anaconda3/envs/gtdbtk_env/lib/python3.8/shutil.py", line 673, in _rmtree_safe_fd
    os.unlink(entry.name, dir_fd=topfd)
OSError: [Errno 16] Device or resource busy: '.nfs00000003ea37a2e8000002c2'
[2021-06-08 08:03:47] TASK: Identifying TIGRFAM protein families.
[2021-06-08 08:04:18] INFO: Completed 529 genomes in 30.85 seconds (17.15 genomes/second).
[2021-06-08 08:04:18] WARNING: TIGRFAM skipped 529 genomes due to pre-existing data, see warnings.log
[2021-06-08 08:04:18] TASK: Identifying Pfam protein families.
[2021-06-08 08:04:23] INFO: Completed 529 genomes in 5.23 seconds (101.22 genomes/second).
[2021-06-08 08:04:23] WARNING: Pfam skipped 529 genomes due to pre-existing data, see warnings.log
[2021-06-08 08:04:23] INFO: Annotations done using HMMER 3.1b2 (February 2015).
[2021-06-08 08:04:23] TASK: Summarising identified marker genes.
[2021-06-08 08:04:41] INFO: Completed 529 genomes in 17.35 seconds (30.48 genomes/second).
[2021-06-08 08:04:41] INFO: Done.
[2021-06-08 08:04:42] INFO: Aligning markers in 529 genomes with 1 CPUs.
[2021-06-08 08:04:42] INFO: Processing 529 genomes identified as bacterial.
[2021-06-08 08:05:04] INFO: Read concatenated alignment for 45,555 GTDB genomes.
[2021-06-08 08:05:04] TASK: Generating concatenated alignment for each marker.
[2021-06-08 08:05:16] INFO: Completed 529 genomes in 11.09 seconds (47.72 genomes/second).
[2021-06-08 08:05:17] TASK: Aligning 120 identified markers using hmmalign 3.1b2 (February 2015).
[2021-06-08 08:33:57] INFO: Completed 120 markers in 28.67 minutes (4.19 markers/minute).
[2021-06-08 08:33:57] TASK: Masking columns of bacterial multiple sequence alignment using canonical mask.
[2021-06-08 08:36:22] INFO: Completed 46,084 sequences in 2.41 minutes (19,149.48 sequences/minute).
[2021-06-08 08:36:22] INFO: Masked bacterial alignment from 41,084 to 5,037 AAs.
[2021-06-08 08:36:22] INFO: 0 bacterial user genomes have amino acids in <10.0% of columns in filtered MSA.
[2021-06-08 08:36:22] INFO: Creating concatenated alignment for 46,084 bacterial GTDB and user genomes.
[2021-06-08 08:36:24] INFO: Creating concatenated alignment for 529 bacterial user genomes.
[2021-06-08 08:36:25] INFO: Done.
[2021-06-08 08:36:29] TASK: Placing 529 bacterial genomes into reference tree with pplacer using 1 CPUs (be patient).
[2021-06-08 08:36:29] INFO: pplacer version: v1.1.alpha19-0-g807f6f3
[2021-06-09 01:26:12] INFO: Calculating RED values based on reference tree.
[2021-06-09 01:26:32] TASK: Traversing tree to determine classification method.
[2021-06-09 01:26:34] INFO: Completed 529 genomes in 1.08 seconds (488.35 genomes/second).
[2021-06-09 01:26:38] TASK: Calculating average nucleotide identity using FastANI (v1.32).
==> Processed 1914/34854 comparisons (5%) |▊              | [ 1.61s/comparison, ETA 14:42:20]client_loop: send disconnect: Broken pipe

Hi. No, unfortunately GTDB-Tk does not have any capability to restart a partially complete run at this point.