[Storage][Jumbo]Remove super#12314
Merged
xiafu-msft merged 14 commits intoAzure:feature/storage-stg73from Jul 7, 2020
Merged
Conversation
xiafu-msft
requested review from
amishra-dev,
annatisch,
danieljurek,
mitchdenny,
rakshith91,
scbedd,
weshaggard and
zezha-msft
as code owners
xiafu-msft
requested review from
chlowell,
lmazuel,
shurd and
southpolesteve
as code owners
rakshith91
reviewed
Jul 2, 2020
Contributor
There was a problem hiding this comment.
curious - why are we doing this?
Contributor
Author
There was a problem hiding this comment.
don't worry I'm trying to run live test, this is just for test
make the test account location for tags to central canada
xiafu-msft
commented
Jul 7, 2020
| # when the returned data is in avro format, keep the raw bytes instead of parsing it to str | ||
| if not body.startswith(b'Obj'): | ||
| response['body']['string'] = body.decode('utf-8') | ||
| response['body']['string'] = body.decode('utf-8') |
Contributor
Author
There was a problem hiding this comment.
this is just to revert the previous workaround for qq recording
xiafu-msft
commented
Jul 7, 2020
| # If the body is not in str format, no need to scrub the data. This case is for when data in avro format | ||
| if not isinstance(response['body']['string'], bytes): | ||
| response['body']['string'] = response['body']['string'].replace(old, new) | ||
| response['body']['string'] = response['body']['string'].replace(old, new) |
Contributor
Author
There was a problem hiding this comment.
this is just to revert the previous workaround for qq recording
xiafu-msft
commented
Jul 7, 2020
|
|
||
| def serialize_blob_tags_header(tags=None): | ||
| # type: (Optional[Dict[str, str]]) -> str | ||
| if tags is None: |
Contributor
Author
There was a problem hiding this comment.
this will avoid constructing empty headers like this x-ms-tags: ''
xiafu-msft
commented
Jul 7, 2020
|
|
||
| class LargeStream(BytesIO): | ||
| def __init__(self, length, initial_buffer_length=1024 * 1024): | ||
| super().__init__() |
Contributor
Author
There was a problem hiding this comment.
python2 doesn't allow empty super().init()
xiafu-msft
commented
Jul 7, 2020
| @pytest.mark.live_test_only | ||
| @record | ||
| def test_upload_large_stream_without_network(self): | ||
| if not TestMode.is_playback(self.test_mode): |
Contributor
Author
There was a problem hiding this comment.
this test failed with Linux Pypy3 on this line:
self.assertEqual(self.payload_dropping_policy.append_counter, 2)
append_counter is 1 somehow. I'm still setting up environment on Ubuntu to reproduce the problem
kasobol-msft
approved these changes
Jul 7, 2020
| def _parse_datetime_from_segment_path(segment_path): | ||
| path_tokens = segment_path.name.split("/") | ||
| path_tokens = segment_path.split("/") | ||
| return datetime(int(path_tokens[2]), int(path_tokens[3]), int(path_tokens[4]), int(path_tokens[5][:2])) |
Contributor
There was a problem hiding this comment.
I'd introduce some local variables to make this code more self descriptive.
xiafu-msft
commented
Jul 7, 2020
| content = await (await dest_blob.download_blob()).readall() | ||
| self.assertEqual(self.source_blob_data, content) | ||
|
|
||
| @pytest.mark.playback_test_only |
Contributor
Author
There was a problem hiding this comment.
it has a vid returned, the current test framework doesn't support enable versioning for an account. A pr would be generated later so we can remove the playback_test_only mark on every vid related test
xiafu-msft
commented
Jul 7, 2020
| await self._create_block_blob(tags=tags, blob_name="blob4", container_name=container_name3) | ||
|
|
||
| if self.is_live: | ||
| sleep(10) |
Contributor
Author
There was a problem hiding this comment.
tags needs some time to take effect. Without this delay, filter find_blobs_by_tags will return empty blob list
skip upload large file test address comment
xiafu-msft
added a commit
that referenced
this pull request
Jul 7, 2020
* [Storage]STG73 * [Blob][Swagger]Update Swagger (#10943) * [Blob][Swagger]Regenerate Swagger Code * fix container test failure caused by list_containers include type change * [Storage] Internal avro parser. (#10764) * initial avro parser * try fixing test... * falling in love with python compatibility... * make linter happy. * raise StopIteration when there is no more bytes instead of tracking file length. * async avro parser * fix syntax for Python 3.5 * get rid of 'readers_schema' as we only honor schema that has been written to file ('writer_schema'). * pr feedback * trim unused code. * pr feedback. * simplify skip sync in next. * move avro tests from _shared. * Jumbo blob support (#11176) * wip * initial test coverage. * wip. * wip * single upload. * add async tests. * disable 50k block tests. * datalake append. * async datalake * disable tests that send large payload over network. * pr feedback. * Undelete share (#11394) * Undelete container (#11339) * [Storage][Blob] Added support for Object Replication (#11525) * Blob versioning (#11154) * [Blob][QuickQuery]Add Quick Query Support (#10946) * [Blob][STG73]Blob Tags (#11418) * regenerate code (#11964) * fix the bug which caused only showing fatal error (#11997) * [Storage][STG73]Address API Review Comments (#12111) * [Storage][STG73]Address API Review Comments * [Storage][STG73]dict<policy, rules> -> list(ObjectReplicationPolicy) * fix blob tag_value test * expose ObjectReplicationPolicy and ObjectReplicationRule, fix test * fix test * Changefeed (#10755) * [ChangeFeed]Add ChangeFeed Package * test_avro failure * update dev_requirement.txt * change namespace to azure.storage.blob.changefeed * address comments * optimize memory when reading changefeed events * namespace change * set up package change * fix failed tests * readme and kwargs * Update sdk/storage/azure-storage-blob-changefeed/azure/storage/blob/changefeed/_change_feed_client.py Co-authored-by: Rakshith Bhyravabhotla <[email protected]> * address comments * 'azure-storage-blob>=12.3.0' which does not match the frozen requirement 'azure-storage-blob~=1.3' Co-authored-by: Rakshith Bhyravabhotla <[email protected]> * [Storage-Blob] Quick Query API (#11991) * Renamed query error * Renamed query reader * Updated config models * Updated format request params * Updated iterator * fix the bug which caused only showing fatal error * Updated Error message * Fixed query helper * Started test conversion * small fix * Fixed tests * Updated error handling + json model * Updated recordings * Removed old recording * Added iter tests * Iter test recordings * Fix test * Remove extra recording * Fix pylint * Some docs cleanup * Renamed iter_records -> iter_stream * Review feedback * Updated tests * Missing commas * Fix syntax * Fix pylint Co-authored-by: xiafu <[email protected]> * tag sas (#12258) * tag sas * disable undelete_container * pylint * skip undelete_container tests * [Blob][Versioning]Disable Versioning Live Test (#12281) * [Blob][QQ]Default output_format to input_format (#12283) * [Storage][Jumbo]Remove super (#12314) * [Storage][JumboBlob]remove empty super() * pypy3 * change sas version to latest * set tags account location to central canada * re-recording queue * changefeed paths generator * mark tests for vid as playback only * fix changefeed * fix pylint make the test account location for tags to central canada * add a delay before calling find_blobs_by_tags * remove tags header * mark a large file test playback only * revert "mark a large file test playback only" skip upload large file test address comment * move tag permission and filter_by_tags permission to kwargs Co-authored-by: Kamil Sobol <[email protected]> Co-authored-by: Ze Qian Zhang <[email protected]> Co-authored-by: Rakshith Bhyravabhotla <[email protected]> Co-authored-by: annatisch <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.