Skip to content

Conversation

@mhvk
Copy link
Contributor

@mhvk mhvk commented Oct 22, 2013

Beyond magnitude, there are two other logarithmic units commonly used in astronomy: dex (e.g., metallicity [Fe/H], surface gravity logg) and dB (radio astronomy, cable losses, etc.) This PR adds them, writing mag and dB as derived units of dex. The latter causes a small problem for the cds and vounit formats, since these know mag and, in the case of vounit, dB, but do not know dex. Hence, mag.decompose().to_string('vounit') fails. This PR simply skips the corresponding tests, as is done for the whole decompose test in FITS; better suggestions welcome.

@mdboom
Copy link
Contributor

mdboom commented Oct 22, 2013

A couple of things come to mind about this:

  1. Are there plans to allow conversions to linear units? For example dB to W or Pa (depending on context, so probably needs to go through an equivalency). Given that equivalencies are more or less required (at least for the case of dB), this lets us put the logic in the equivalency itself, along with providing a place for specifying an offset (if necessary). We should make sure we can handle that, though, because that's where (I believe) this gets really powerful.

  2. The FITS and VOUnit formats have the syntax log(x) to represent logarithmic units. Therefore, I think dex should decompose into that, don't you think? So then some unit like dB/s would represent as 0.1 * log(s) (or something).

@mhvk
Copy link
Contributor Author

mhvk commented Oct 22, 2013

  1. Are there plans to allow conversions to linear units?

Yes, indeed! That is what I've been spending (far too much of) my time on. I'm still adding documentation and tests for the LogQuantity part, but the LogUnit is more or less done. Indeed, if you are able to, it would be very nice to get some feedback on my design [1].

  1. The FITS and VOUnit formats have the syntax log(x)

Indeed, I should try to use that; as I'm already using, e.g., lu.DecibelUnit('W').__str__() -> dB(W), this would be quite natural.

For the dimensionless case (relevant for this PR), would it be log() or log('')?

[1] https://github.com/mhvk/astropy/compare/units-logarithmic?expand=1

@eteq
Copy link
Member

eteq commented Oct 22, 2013

FYI @mhvk, you may have a hard sell (at least to me) on LogQuantity. My first impression is that it will be very confusing on users (and developers) to have two separate types of quantities. So you might consider issuing a partial PR and wait for some discussion before you spend more time on something that might not get accepted.

That said, this PR seems fine to me, and I agree with Mike that equivalencies provide a natural way to move between log and linear units.

@mhvk
Copy link
Contributor Author

mhvk commented Oct 22, 2013

@eteq - I guess ultimately it will be a question of what tools you think are important to provide -- as noted by @mdboom above, it would be really good for astropy to provide both magnitude, decibels, and dex with physical units, since these are used regularly. Since these are essentially the same type of logarithmic quantities, it would be insane to have them use different implementations.

As said in #1577, I do think that what I'm doing is the logical extension of the magnitude idea by @pllim and you, but one where you can already almost anything you'd ever want with it in a way that is physically correct -- and gives dB, dex, etc., as a bonus.

What I have so far provides little documentation on the API, but the usage would be very similar to that outlined in #1577, where one would typically initalise with <some-number> * mag or with Magnitude(<number>) -- the LogQuantity is just the class that defines everything. Of course, the resulting object is not a normal quantity, as it isn't physically either.

p.s. As I'm essentially implementing the API described in #1577 is lacking, please let me know if you think it is lacking.
p.s.2 Of course, we'll still need a magnitudes package, but it will be for the really hard stuff -- different magnitude systems.

@eteq
Copy link
Member

eteq commented Oct 23, 2013

Lets leave this open until we see how LogQuantity goes, then. Another issue that occurred to me here is the following current behavior in master:

>>> log10(1 * u.km)
TypeError: Can only apply 'log10' function to dimensionless quantities
>>> log10(1 * u.km/u.m)
<Quantity 3.0 >

that is the correct behavior, but the thing that comes out is dimensionless, rather than having units of dex or something like that. So I think it'll be confusing that most arithmetic operations only yield dimensionless quantities, even though there's a dex available as a unit. But it may be that your plan for LogQuantity will fix all that. But we should have the discussion there, and it will either render this moot, or we can revisit this if it doesn't happen.

@mhvk
Copy link
Contributor Author

mhvk commented Oct 24, 2013

I think the current behaviour is correct, since here the user explicitly asks to actually take the log10, not to construct a logarithmic unit. Mathematically, it is also different; 3*dex could not represent the log10(), but rather 1*u.km/u.m itself ( as could 7.5*mag; remember "dex" is short for decimal exponent). In a similar vein, 1*u.km would be 3*dex(km) /EDIT: should of course be 3*dex(m) EDIT/. Or more related to actual usage, 100*u.mW would be 20.*dB(mW) (normally called dBm [1], but I'm definitely not advocating following that nomenclature -- although a user would have to do nothing more than dBm = DecibelUnit(u.mW) if they rather do have it).

[1] http://en.wikipedia.org/wiki/Decibel

@mhvk
Copy link
Contributor Author

mhvk commented Oct 24, 2013

Lets leave this open until we see how LogQuantity goes, then.

Just to clarify: this PR does not depend on LogQuantity, which is why I split it off. Hence, there is no need to wait; e.g., someone calculating RG-58 cable losses for a radio receiver could already do (11.2*u.dB/(100.*u.feet) * 1.*u.km (Yes, these awful units are really used [1]; good that we have add_enabled_units(imperial)...)

[1] http://www.w4rp.com/ref/coax.html

@mhvk
Copy link
Contributor Author

mhvk commented Oct 28, 2013

Added an equivalency, so one can now do the example above as in:

import astropy.units as u, astropy.units.imperial as imperial
with imperial.enable(), u.set_enabled_equivalencies(u.logarithmic()):
    rg58_loss = -11.2*u.dB/(100.*u.imperial.ft)
    total_loss = rg58_loss * 100.*u.m
    print("RG-58 cable loss over 100 m: {0:6.2f} or a factor {1:7.5f}"
        .format(total_loss.to(u.dB), total_loss.to(1)))

which yields

RG-58 cable loss over 100 m: -36.75 dB or a factor 0.00021

p.s. @mdboom - I still haven't gotten around to looking how vounit and fits deal with dimensionless for their log unit, but if that's easy, I would definitely like to add it to this PR.

@pllim
Copy link
Member

pllim commented Oct 30, 2013

Between this and https://github.com/mhvk/astropy/compare/units-logarithmic , I see a lot of overlap with #1577 . Should I hold off my Flux and Magnitude work until they are merged? Or perhaps only Flux needs implementation after that.

@mhvk
Copy link
Contributor Author

mhvk commented Oct 30, 2013

@pllim - yes, I think so; we definitely need a Flux, but I believe my functional units are a more general and hopefully preferable way to deal with magnitudes. I am actually more or less done, though moved to a sub-directory units/functional rather than a single huge file [1]. I hope to do a PR on this next week (after 0.3 is released). If you want a peek preview of how I hope it can be used, see http://www.astro.utoronto.ca/~mhvk/logq.py

[1] https://github.com/mhvk/astropy/tree/units-functional

@mhvk
Copy link
Contributor Author

mhvk commented Dec 9, 2013

@mdboom - I rebased this old PR to introduce dex and dB, as I hope to get to the follow-up with full functional units. It still has a coverage error that I do not understand; do you have a suggestion?

Otherwise, the one outstanding issue is whether/how to deal with the link between dex and the log() unit in FITS; as you will have seen, I've sent another request for clarification about that. In the meantime, might it be an idea for now to raise a separate issue about that, and proceed here?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you find this in the FITS spec somewhere? That's kind of an odd unit...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this unit will ever occur (and if it did, I'm not sure what I would make of it!), but it is defined by the prefixes, and thus is tested, and with dB being present, we have to make the short form of decibyte is not being created. Would happily implement another way to circumvent it (maybe in this case, just omitting it from the tests is better than cluttering the code?).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The way that this is handled elsewhere is to use the exclude_prefixes kwarg when the unit is defined.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did that in the definition of byte -- so in our unit system dbyte cannot exist. But in FITS in principle it can, and that is the case I'm trying to capture here.

Maybe it is good to step back, and consider what is the actual problem: if I remove these special_cases, I get an error any time I do <unit>.to_string('fits'):

In [2]: u.a.to_string('fits')
ERROR: AttributeError: 'module' object has no attribute 'dbyte' [astropy.units.format.fits]

The difference with other excluded prefixes seems to be that one usually excludes them since the other unit exists. Hence, when, e.g., the code tries to get 'P'+'a', it will just overwrite names['Pa']=u.Pa, which is fine. But in my case, it will work for dB but not for dbyte since that particular unit does not exist. So, really I want to exclude the prefix d only for B, not also for byte.

Which perhaps suggests a solution: if I define B as a derived unit from byte, I can exclude the prefix d only where it matters. This works except that it breaks another test:


    def test_megabit():
        """See #1543"""
        assert u.Mbit is u.Mb
        assert u.megabit is u.Mb

>       assert u.Mbyte is u.MB
E       assert Unit("Mbyte") is Unit("MB")
E        +  where Unit("Mbyte") = u.Mbyte
E        +  and   Unit("MB") = u.MB

/data/mhvk/venv/astropy-dev/lib/python3.3/site-packages/astropy-0.4.dev6748-py3.3-linux-x86_64.egg/astropy/units/tests/test_units.py:516: AssertionError

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wow, what a pain. I wonder if we shouldn't rework how SI prefixes are handled. Rather than making prefixes= a boolean option when defining units, makes prefixes accept a dictionary mapping unit names/abbreviations to a list of allowed prefixes. For example, in the definition for byte:

prefixes = {'B': ['k', 'M', 'G', 'T', 'P', 'Y', 'Z', etc....],
            'byte': ['k', 'M', 'G', etc...]}

There could be shortcuts available if all spellings of the unit take the same prefixes. There should also be a few constants defined like a list named SI_PREFIXES so that one can easily assign prefixes=SI_PREFIXES for units that accept all standard SI prefixes. sigh

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is a pain, but then again, we have a mostly working system, so I think it doesn't hurt that much to just put in a hack for a few corner cases. Perhaps mine is not so bad? At least it is extensible. Alternatively, we could just replace

if hasattr(u, key):
    names[key] = getattr(u, key)

and not unreasonably assume that nobody will ever have a fits file with units of dbyte.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another possibility is to do what was done for cds and define units specific to the format in their own module. As there is only one of those here, that seems like a bit of overkill, though...

I'm fine with this for now, now that I understand why it's here. Let's not put too much effort into a single unit that is very unlikely to ever be seen in the wild.

@mdboom
Copy link
Contributor

mdboom commented Dec 9, 2013

My concern is with introducing a unit syntax that ultimately is incompatible with the FITS standard. I hadn't expected it to be so difficult to get a response... This feature is implemented in wcslib, so perhaps we just look there at what it's doing -- that's probably as "authoritative" as we can expect, but I suspect many people simply aren't using this feature in FITS.

If we proceed without that, I'd say we need a way to flag this as a "discretionary" feature in the meantime. Maybe that's to raise a warning when the log keyword is encountered.

(EDIT: I'm out today with weather-related school closures, or I'd look into this myself...)

@mhvk
Copy link
Contributor Author

mhvk commented Dec 9, 2013

@mdboom - fair enough. At least Dr Chiappetti replied; hopefully his suggestion of posting to fitsbits will lead somewhere.

@mhvk
Copy link
Contributor Author

mhvk commented Dec 10, 2013

@mdboom - ok, I'll add a comment the the fits part so it is clearer why this test is there.

Aside: do you have any idea why the coverage test failed?

@mdboom
Copy link
Contributor

mdboom commented Dec 10, 2013

It's not clear why the coverage test is failing here and not on master. It's because Unit.__repr__ returns a Unicode string on Python 2, which it shouldn't. Fix in a PR against this one.

@mhvk
Copy link
Contributor Author

mhvk commented Dec 10, 2013

@mdboom - thanks for another fix. I was indeed very puzzled that the test seemed problematic on my branch but not on master, although it was a change in the cds test that seems to trigger it.

@mdboom
Copy link
Contributor

mdboom commented Dec 11, 2013

Here's what wcslib "says" in comments and code on the subject:

*   4: Function types log(), ln() and exp() may only occur at the start of the
*      units specification.  The scale and units[] returned for these refers
*      to the string inside the function "argument", e.g. to "MHz" in log(MHz)
*      for which a scale of 1e6 will be returned.

So there can only be one "function" per unit specification.

When converting between units, (in wcsunitse), it does the following:

  • If both units don't have a function, it's a standard scaling operation
  • If both units have log, it performs an offset of log10(scale1 / scale2)
  • If both units have ln, likewise for ln
  • If it's a mix of log and ln, it can apply a scale of ln(10) or 1 / ln(10) in addition to the offset
  • If both units have exp, it's a standard scaling operation.

The VOUnit adds sqrt to this.

@mhvk
Copy link
Contributor Author

mhvk commented Dec 11, 2013

@mdboom - you're fix for the coverage problem actually causes an enormous number of new problems... Could you check?

@mhvk
Copy link
Contributor Author

mhvk commented Dec 12, 2013

@mdboom - tests now failed for python3; I made a commit in which __repr__ acts slightly differently for python2 and 3; it seemed the only way. Hopefully it will work everywhere.

@mhvk
Copy link
Contributor Author

mhvk commented Dec 12, 2013

The __repr__ change seems OK. I added one commit, which adds a test that u.mag ** 0.5 works as expected. Since this is -0.4 * u.dex, decomposing it gives a complex scale. This was the reason for the change in is_effectively_unity and the addition of sanitize_scale, but I had omitted to add a test, and it turned out that a ** p automatically returns a complex number only on python3. So, more changes. All for u.mag ** 0.5... Anyway, I think it works.

@mdboom
Copy link
Contributor

mdboom commented Dec 12, 2013

Looks good, aside from my comments on the (-neg) ** (frac) handling.

@mdboom
Copy link
Contributor

mdboom commented Feb 12, 2014

Is this obsolete in light of #1894?

@mhvk
Copy link
Contributor Author

mhvk commented Feb 12, 2014

@mdboom - #1894 is a (much) further development of this (and is written assuming this exists).

@mdboom mdboom closed this Feb 13, 2014
@mhvk
Copy link
Contributor Author

mhvk commented Feb 13, 2014

Hmm, perhaps I wasn't clear: #1894 develops this further, but this PR is provides what I think is useful functionality independent of whether people like my approach to logarithmic units. So, I'll reopen for now, unless you are sure enough that #1894 will go in...

@mhvk mhvk reopened this Feb 13, 2014
@mhvk
Copy link
Contributor Author

mhvk commented Mar 24, 2014

@mdboom - as @wkerzendorf mentioned in #1894 that it would be quite useful to have dex, I rebased to ensure this can be merged. To summarize briefly:

  • This introduces dex and dB as units without an associated physical unit. It should be useful on its own (like mag), e.g., for metallicities.
  • While it is needed for the functional logarithmic units in Function units: mag, dB, and dex #1894, it does not depend on it.
  • The one issue from the discussion above that remains is that, in principle, fits could represent dex as log(1) (and, I think, CDS as [1]), but at present the code does not implement that. I think this is better done as part of Function units: mag, dB, and dex #1894.

Let me know if you think there is anything missing.

@mhvk
Copy link
Contributor Author

mhvk commented Mar 24, 2014

Travis throws an error, but it seems unrelated. Restart to be sure?

@mdboom
Copy link
Contributor

mdboom commented Mar 24, 2014

Looks good. Like you say, we should get the output formatting right, too... I'm not sure why you suggest we delay that. Won't merging this pull request allow for units that when written to FITS won't be FITS compliant?

@mhvk
Copy link
Contributor Author

mhvk commented Mar 24, 2014

No, this PR will not write non-compliant FITS; instead, mag is still OK, but it will give an error when trying to write dex (or dB) to fits (or CDS):

In [1]: import astropy.units as u

In [2]: u.mag.to_string('fits')
Out[2]: 'mag'

In [3]: u.dex.to_string('fits')
ERROR: ValueError: Unit 'dex' is not part of the FITS standard [astropy.units.format.fits]

I think that is fine in principle, since the standard is so unclear anyway.

But I do want to implement it returning log(1) eventually; however, this implies getting the reader to understand what is between the parentheses, and I think that is much more logically done as part of #1894.

@mdboom
Copy link
Contributor

mdboom commented Mar 24, 2014

I see. That makes sense.

@mhvk
Copy link
Contributor Author

mhvk commented Mar 24, 2014

Most bizarre, the same travis setup fails (one but last, numpy 1.6, python 3.2), and for a similar reason (EOF when importing module), but in a different package (was table last time). Any ideas?

ERROR collecting astropy/io/votable/setup_package.py

@mhvk
Copy link
Contributor Author

mhvk commented Mar 24, 2014

The travis error does seem to be spurious -- #1894, which is based on this PR, does pass all tests.

mdboom added a commit that referenced this pull request Mar 24, 2014
@mdboom mdboom merged commit db648c3 into astropy:master Mar 24, 2014
@astrofrog
Copy link
Member

@mhvk - this needs an entry in CHANGES.rst

@mhvk mhvk deleted the units-dex-dB branch March 24, 2014 21:58
mhvk added a commit to mhvk/astropy that referenced this pull request Mar 24, 2014
mhvk added a commit that referenced this pull request Mar 24, 2014
@mhvk
Copy link
Contributor Author

mhvk commented Mar 24, 2014

@astrofrog - OK, sorry to have forgotten that; now done (#2231)

ktchrn pushed a commit to ktchrn/astropy that referenced this pull request Oct 28, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants