Skip to content

Add async Writer to support pub&mpub#25

Merged
mreiferson merged 2 commits intonsqio:masterfrom
felinx:master
Apr 30, 2013
Merged

Add async Writer to support pub&mpub#25
mreiferson merged 2 commits intonsqio:masterfrom
felinx:master

Conversation

@felinx
Copy link
Copy Markdown
Contributor

@felinx felinx commented Apr 26, 2013

Add async Writer to support pub&mpub and fix JSONDecodeError

nsq/Writer.py Outdated
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i dont think we need this for writing

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding the tests, this stray line is the last item

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  •    conn.id = conn_id
    
  •    conn.last_recv_timestamp = time.time()
    
  •    conn.last_msg_timestamp = time.time()
    

Do you mean check_last_recv_timestamps etc?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I meant last_msg_timestamp, it's not used here.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated, please review it, thanks!

@mreiferson
Copy link
Copy Markdown
Member

I had started writing some code in #11, can you port over the low-level tests I added there to this pull request (and I'll close #11).

I think eventually (not now) this can benefit from some re-structuring for the shared code in Reader and Writer for managing connections, data flow, and liveness to reduce some of the duplication, but I'm fine with this for now.

Appreciate the contribution, nice work.

@felinx
Copy link
Copy Markdown
Contributor Author

felinx commented Apr 26, 2013

Yes, there are some duplicated code now, I just want to implement this functional at first and I will do my best to do more contribution later.

@mreiferson
Copy link
Copy Markdown
Member

Looks good, you mind squashing down to two commits (one for JSON fix, one for everything else)?

@felinx
Copy link
Copy Markdown
Contributor Author

felinx commented Apr 27, 2013

It's ok for me,thanks!

@felinx
Copy link
Copy Markdown
Contributor Author

felinx commented Apr 27, 2013

Squashing is done, it's ready for merging, thanks!

nsq/Writer.py Outdated
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just noticed this... conn.last_pub(1) should be conn.last_pub[1], but I have an alternative suggestion.

Even though we have async heartbeat messages that the Writer has to handle... every PUB or MPUB command can expect exactly one FRAME_TYPE_RESPONSE or FRAME_TYPE_ERROR _in the order they were sent_.

So, if instead we maintained a callback_queue instead of just a response_callback_queue, we can actually respond to each specific PUB or MPUB command rather than a generic "this failed callback" like you've currently implemented.

This callback can be optional, but the API would look like this:

def handler(self):
    self.writer.pub("log", "Hello world", callback=self._finish_pub)

def _finish_pub(self, data):
    if data.frame_type_id == nsq.FRAME_TYPE_ERROR:
        logging.error("blablabla")

what do you think?

@mreiferson mreiferson mentioned this pull request Apr 27, 2013
nsq/Writer.py Outdated
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rename the nsqds variable to nsqd_tcp_addresses for consistency with Reader

@felinx
Copy link
Copy Markdown
Contributor Author

felinx commented Apr 28, 2013

Updated code follow your comments, and I also renamed variable data to msg in pub&mpub func because it is a little confused with response's data. Squashed.

nsq/Writer.py Outdated
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we update the example w/ the new (optional) API?

@felinx
Copy link
Copy Markdown
Contributor Author

felinx commented Apr 28, 2013

Refined callback and updated example.

nsq/Writer.py Outdated
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should send response when frame == nsq.FRAME_TYPE_RESPONSE xor error when frame == nsq.FRAME_TYPE_ERROR, in the latter case perhaps as an exception we define?

@felinx
Copy link
Copy Markdown
Contributor Author

felinx commented Apr 30, 2013

Committed this change and squashed :)

@mreiferson
Copy link
Copy Markdown
Member

thanks

mreiferson added a commit that referenced this pull request Apr 30, 2013
Add async Writer to support pub&mpub
@mreiferson mreiferson merged commit 9ad4f30 into nsqio:master Apr 30, 2013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants