Repository navigation
"invalid shape in fixed-type tuple" when saving adata after ranking genes #832
Description
I'm using Scanpy with the following software versions:
python==3.7
scanpy==1.4.4
numpy==1.17.2
anndata==0.6.22.post1
on Ubuntu 18.04. I am able to save my AnnData object just fine with
sc.write(results_file, adata)and to load it again with
adata = sc.read(results_file)however if I save it after I run the command
sc.tl.rank_genes_groups(adata, 'louvain12_lab', method='wilcoxon')the AnnData object will save but when I try to reload it, I get an error message:
ValueError Traceback (most recent call last)
<ipython-input-141-159082f1696f> in <module>
1 results_file = os.path.join(adir, '{project}.count_{count}.gene_{gene}.mito_{mito}.HVGs_{nhvgs}.TPT.{log}.scale.TEST.h5ad'.format(project=project_name, count=count_thresh, gene=gene_thresh, mito=mitothresh, nhvgs=nhvgs, log=logstatus))
2 print(results_file)
----> 3 adata = sc.read(results_file)
/opt/miniconda3/envs/py37/lib/python3.7/site-packages/scanpy/readwrite.py in read(filename, backed, sheet, ext, delimiter, first_column_names, backup_url, cache, **kwargs)
95 filename, backed=backed, sheet=sheet, ext=ext,
96 delimiter=delimiter, first_column_names=first_column_names,
---> 97 backup_url=backup_url, cache=cache, **kwargs,
98 )
99 # generate filename and read to dict
/opt/miniconda3/envs/py37/lib/python3.7/site-packages/scanpy/readwrite.py in _read(filename, backed, sheet, ext, delimiter, first_column_names, backup_url, cache, suppress_cache_warning, **kwargs)
497 if ext in {'h5', 'h5ad'}:
498 if sheet is None:
--> 499 return read_h5ad(filename, backed=backed)
500 else:
501 logg.debug(f'reading sheet {sheet} from file {filename}')
/opt/miniconda3/envs/py37/lib/python3.7/site-packages/anndata/readwrite/read.py in read_h5ad(filename, backed, chunk_size)
445 else:
446 # load everything into memory
--> 447 constructor_args = _read_args_from_h5ad(filename=filename, chunk_size=chunk_size)
448 X = constructor_args[0]
449 dtype = None
/opt/miniconda3/envs/py37/lib/python3.7/site-packages/anndata/readwrite/read.py in _read_args_from_h5ad(adata, filename, mode, chunk_size)
484 d[key] = None
485 else:
--> 486 _read_key_value_from_h5(f, d, key, chunk_size=chunk_size)
487 # backwards compat: save X with the correct name
488 if 'X' not in d:
/opt/miniconda3/envs/py37/lib/python3.7/site-packages/anndata/readwrite/read.py in _read_key_value_from_h5(f, d, key, key_write, chunk_size)
508 d[key_write] = OrderedDict() if key == 'uns' else {}
509 for k in f[key].keys():
--> 510 _read_key_value_from_h5(f, d[key_write], key + '/' + k, k, chunk_size)
511 return
512
/opt/miniconda3/envs/py37/lib/python3.7/site-packages/anndata/readwrite/read.py in _read_key_value_from_h5(f, d, key, key_write, chunk_size)
508 d[key_write] = OrderedDict() if key == 'uns' else {}
509 for k in f[key].keys():
--> 510 _read_key_value_from_h5(f, d[key_write], key + '/' + k, k, chunk_size)
511 return
512
/opt/miniconda3/envs/py37/lib/python3.7/site-packages/anndata/readwrite/read.py in _read_key_value_from_h5(f, d, key, key_write, chunk_size)
542 return key, value
543
--> 544 key, value = postprocess_reading(key, value)
545 d[key_write] = value
546 return
/opt/miniconda3/envs/py37/lib/python3.7/site-packages/anndata/readwrite/read.py in postprocess_reading(key, value)
539 new_dtype = [((dt[0], 'U{}'.format(int(int(dt[1][2:])/4)))
540 if dt[1][1] == 'S' else dt) for dt in value.dtype.descr]
--> 541 value = value.astype(new_dtype)
542 return key, value
543
ValueError: invalid shape in fixed-type tuple.Any idea what is going on or what I can do to make it past this error? It only started happening after I updated my operating system to Ubuntu 18 and my Python to 3.7 and reinstalled scanpy from conda.
Activity
gokceneraslan commentedon Sep 11, 2019
Hi Dylan,
This is an issue with the new h5py package, which @ivirshup already fixed on master (928d475). For now, you can downgrade your h5py package to 2.9.0 using
pip install h5py==2.9.0as a workaround.bkmartinjr commentedon Sep 11, 2019
@gokceneraslan - thanks for the fast response. This broke our (cellxgene) travis pipeline as well. Do you have any info on eta for a fix/workaround other than pinning the module version? TY!
ivirshup commentedon Sep 12, 2019
LuckyMD commentedon Oct 22, 2019
I'm having the same error with
h5py==2.9.0. Cellxgene doesn't seem to be working with the object that I created the object with scanpy1.4.3+116.g0075c62. I can however load it again with that version. But when I downgrade to 1.3.7 (recommendation from @mbuttner who had the same cellxgene issue) I can no longer load the object and get the above error.Back in the 1.4.3 dev version scanpy it no longer writes the object after loading, and gives me the following error:
Everything however seems to work fine when I throw out the rank_genes_groups results from
adata.unsEdit: actually cellxgene still isn't working, but I could at least save again.
ivirshup commentedon Oct 23, 2019
@LuckyMD
I can replicate that with:
Some context around this, and my current thinking on a solution:
np.str_type arrays.LuckyMD commentedon Oct 23, 2019
Sorry for the lack of minimal reproducible example... and thanks for creating one :).
flying-sheep commentedon Oct 23, 2019
Where might bytestrings be useful? If you say text, I have a very strong opposing opinion as I’m a survivor of Python 2 and don’t want to see an UnicodeDecodeError in my life again 😉
ivirshup commentedon Oct 23, 2019
flying-sheep commentedon Oct 24, 2019
It is, too bad numpy has no good variable-length string array type.
When would bytes make sense? Bytes just mean “data, but I don’t know its structure or am about to write it to disk”