Codecov Report

Merging #241 (f1d8d4a) into main (2f9144b) will decrease coverage by 0.87%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main     #241      +/-   ##
==========================================
- Coverage   77.30%   76.44%   -0.87%     
==========================================
  Files         111       99      -12     
  Lines        6170     5518     -652     
  Branches      576      576              
==========================================
- Hits         4770     4218     -552     
+ Misses       1384     1284     -100     
  Partials       16       16              

see 12 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

This looks great! The ability to cite multiple sources is very useful as well.
Re: structured citations @cguedes we might want to look at langchain for this - https://python.langchain.com/docs/use_cases/question_answering/#adding-in-sources

I also found a tutorial demonstrating similar functionality - https://github.com/gkamradt/langchain-tutorials/blob/cc89c780e6484bdc3b15a967c1be2a5bfa4934dc/data_generation/Custom%20Files%20Question%20%26%20Answer.ipynb#L9

This is a great step forward working with the chat, but my expectation was that we would get structured data reply with the reference(s) in a field outside of the text. Can we have it @gjreda?

@cguedes updated this to include structured citations. see the updated top comment on this PR for examples

I'm sorry about the back-and-forth regarding this feature, but I'm a bit concerned about this update, which feels like regression to me.

The reply we were getting before for the question about hidden feedback loops was the following:
Hidden feedback loops in machine learning refer to situations where two systems indirectly influence each other through the world. One example of this is when two systems independently determine different aspects of a web page, such as selecting products to show and selecting related reviews. Improving one system may lead to changes in behavior in the other, as users react to the changes by clicking more or less on the other components (Hidden Technical Debt in Machine Learning Systems.pdf, p4). These hidden feedback loops can exist between completely disjoint systems and pose a statistical challenge for ML researchers to investigate (Machine Learning - The High-Interest Credit Card of Technical Debt.pdf, p3).

The new reply is:
Hidden feedback loops are systems that influence each other indirectly through the world, often resulting in changes in behavior in reaction to changes in one system. These loops may exist between completely disjoint systems.

Although we now get citations with a proper schema, I think the reply contains far less details. And another issue is that we don't know which part of the answer corresponds to which citation.
Maybe we should rollback to the previous solution and parse content between parenthesis with a regex to determine which references are being used?
What are your thoughts on that?

I'm ok with reverting to the other solution and regex/parse the reference for now and work on the improved solution in a different PR.

Yes, requiring JSON output means less available token space in the response, and it also means eating up a lot more of the input context space due to requiring more directions in the prompt. It took a lot of trial and error to have a prompt that resulted in JSON output and also left enough of the input context space for the relevant text chunks.

I don't love the idea of reworking this again and trying to regex out a somewhat non-deterministic result from the LLM, but can give it a shot if we think it's worth the time.

I think the best approach for now would be to move forward with how this PR previously was (with inline citations) and then I can add a follow-up PR to regex out the citations if we think it's valuable.

What do y'all think about that? @cguedes @sehyod

@gjreda we see your point. Ok. Push a revert to this branch and let me know when it's ready for a new review.

@cguedes Sound good, should be good to go now and I'll work on a separate regex PR.