Skip to content

Enable timezone-aware datetime instances for Quotes by utilizing dateutil package.#310

Closed
claytonEtillman wants to merge 4 commits intofacioquo:mainfrom
claytonEtillman:add_tz_aware
Closed

Enable timezone-aware datetime instances for Quotes by utilizing dateutil package.#310
claytonEtillman wants to merge 4 commits intofacioquo:mainfrom
claytonEtillman:add_tz_aware

Conversation

@claytonEtillman
Copy link

@claytonEtillman claytonEtillman commented Jul 31, 2023

Description

An error is produced in the current version when trying to export result.date if the input Quote.date was timezone-aware. The problem is that, according to the Python docs, prior to Python 3.11 the function datetime.datetime.fromisoformat "does not support parsing arbitrary ISO 8601 strings - it is only intended as the inverse operation of datetime.isoformat(). A more full-featured ISO 8601 parser, dateutil.parser.isoparse is available in the third-party package dateutil." While this is fixed in Python 3.11, I put in the fix in this pull request for backwards-compatibility. Most of the libraries I use have not upgraded to Python 3.11 yet.

Below is a screenshot of the failing tests I added before putting in the fix. These originate from calling result.date after using an 'aware' datetime.datetime instance as input.

image

CAUTION I also ran benchmarks on the aware instances with a few indicators, and there is a significant decrease in performance when using 'aware' datetimes. See screenshot below.

image

Fixes #-
I did not put an issue in as I had already fixed this on my local installation. I figured I might as well share this so that others can use it if they are on a Python lower than 3.11.

Checklist

  • My code follows the existing style, code structure, and naming taxonomy
  • I have commented my code, particularly in hard-to-understand areas
  • I have performed a self-review of my own code and included any verifying manual calculations
  • I have added or updated unit tests that prove my fix is effective or that my feature works, and achieves sufficient code coverage. New and existing unit tests pass locally and in the build (below) with my changes
  • My changes generate no new warnings and running code analysis does not produce any issues
  • I have added or run the performance tests that depict optimal execution times
  • I have made corresponding changes to the documentation

@claytonEtillman
Copy link
Author

Haha, it looks like the server is on UTC time and my machine is on EDT so we have a mismatch on output although the times are the same. I should be doing a timezone-aware comparison! How ironic. I'll make the necessary changes sometime this week.

@DaveSkender
Copy link
Member

DaveSkender commented Jul 31, 2023

If it helps, dates passed through this library are merely data labels, so converting to a time zone aware format isn’t helpful for anything it does; however, I so see how someone sending in with TZ aware offset formats would want it to come back in that same format. Few financial market quote sources use this more expensive format, so am not sure this’ll be worth the added cost. @LeeDongGeon1996 any thoughts here?

@LeeDongGeon1996
Copy link
Collaborator

I'll have a look tonight.

@LeeDongGeon1996
Copy link
Collaborator

Hi, first of all, thanks for your contribution :)

@DaveSkender Do you mean that datetime data in financial markets is usually provided in not timezone-aware form?
I think it's worth it in terms of backward-compatibility. If there's someone having timezone-aware data, they will not care about whatever their date format is and struggle with the error from timezone-aware data.

@DaveSkender
Copy link
Member

DaveSkender commented Jul 31, 2023

Hi, first of all, thanks for your contribution :)

@DaveSkender Do you mean that datetime data in financial markets is usually provided in not timezone-aware form? I think it's worth it in terms of backward-compatibility. If there's someone having timezone-aware data, they will not care about whatever their date format is and struggle with the error from timezone-aware data.

Yes, it'll usually comes from sources without timezone offset information, it comes at intrinsic regional market time or UTC time, without any offset information. For example, most U.S- based market data is in Eastern Time Zone, but someone could encode it with timezone offest information as Z-5 if they wanted to.

This is really a matter of how much cost (e.g., to performance and extra features) do we want to bear to accommodate its use in quotes people provide. I don't think I'll change the .NET library from DateTime to DateTimeOffset either, it's way too expensive and can get really goofy, with little value. It's possible that I'm overthinking this as there might be some simple tweak here that just works.

@claytonEtillman
Copy link
Author

Hi @DaveSkender & @LeeDongGeon1996, thank you for the great library! I appreciate the feedback. Please see below how I encountered this error.

stoch_rsi_results = si.get_stoch_rsi(quotes_list, R, S, G, M).remove_warmup_periods()
rsi_df = pd.DataFrame([ (r.date, r.stoch_rsi, r.signal) for r in stoch_rsi_results ], columns=['TimeStamp','stoch_rsi','rsi_signal'])

The TradeStation API returns data in UTC time with a 'Z' marker to indicate the timezone: TradeStation API specification. I convert this datetime to Eastern for New York exchanges before calling the stock-indicators functions. If someone is using this library for other exchanges, they may use that exchange's local timezone. I then load the results to a pandas DataFrame to merge back in with my other data based on datetime, which I've named TimeStamp. I convert both DataFrames that I'm merging to Eastern time for New York exchanges. As long as they are both timezone aware, this works. It's okay if it changes timezones as long as it knows which timezone it has changed to.

The errors in the GitHub build are due to trying to compare them as strings in the tests I added. I'll put a fix in for those shortly.

@claytonEtillman
Copy link
Author

I don't think I'll change the .NET library from DateTime to DateTimeOffset either, it's way too expensive and can get really goofy, with little value. It's possible that I'm overthinking this as there might be some simple tweak here that just works.

It doesn't seem to need a tweak in the C# code. It comes back timezone aware from what I can tell. It just has one too many zeros in the fraction of seconds. The use of the Python dateutils isoparse function compensates for that adequately.

@claytonEtillman
Copy link
Author

Concerning the computational cost, it doesn't seem to increase unless someone uses a timezone aware datetime. The dateutils function seems comparable to the datetime function in computation speed. Also, I give 2000 rows for the tz aware benchmark test, which is four times the data in the default dataset. Maybe this is the cause of the increased computation time.

@claytonEtillman
Copy link
Author

I fixed the tests: just needed to compare them as dates.

I didn't push the added benchmarks I ran because the docs say not to update the Excel file, but I have those stashed if you want me to. Just let me know.

@claytonEtillman
Copy link
Author

I have yet one more set of fixes to make. While I normally put standard library imports before third-party with clearer separation, I tried to follow the format of the library. I like 'import' statements together and 'from' statements together. I'll fix that quickly.

@LeeDongGeon1996
Copy link
Collaborator

Sorry for late reply, I'll have a look tonight.

@LeeDongGeon1996
Copy link
Collaborator

@claytonEtillman I think that your words are reasonable and this feature will be helpful to users. One thing that I couldn't fully understand, could you explain more about this?

It just has one too many zeros in the fraction of seconds.

@LeeDongGeon1996
Copy link
Collaborator

LeeDongGeon1996 commented Aug 3, 2023

Ohh.. now I get your words. I'm not 100% sure, but think that this is not a problem about Python datetime parsing. It's a problem of parameter for C-Sharp datetime "o" .
Please refer to https://learn.microsoft.com/en-us/dotnet/api/system.datetime.tostring?view=net-7.0#code-try-6
I'll look into more on this weekend.

EDIT:
The problem is from where I split a datetime string, whick comes from C# datetime. It originally comes with well-formed value, 2008-06-15T21:15:07.0000000, but I remove the last zero from the code. That makes it incorrect-form of datetime string, eventually failed to parse on Python datetime

datetime.fromisoformat('2008-06-15T21:15:07.0000000'[-1])

Traceback (most recent call last):
ERROR!
File "<string>", line 15, in <module>
ValueError: Invalid isoformat string: '0'

Please let me know if there's sth wrong.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jan 31, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants